RU2803488C2 - Method and server for waveform generation - Google Patents

Abstract

FIELD: computer science.

SUBSTANCE: method includes obtaining a trained flow-based vocoder including reversible blocks and an untrained feed-forward vocoder including irreversible blocks that form a teacher-student network, performing a learning process on the teacher-student network, during which the server generates (i) a teacher-related waveform by a trained flow-based vocoder using the first spectrogram and the first input noise, (ii) a student-related waveform by an untrained feed-forward vocoder using the first spectrogram and the first input noise, and (iii) a loss value for a particular training iteration using the teacher-related waveform and the student-related waveform. The server then trains the untrained feed-forward vocoder to generate a waveform. The trained feed-forward vocoder is used instead of the trained flow-based vocoder to generate waveforms based on spectrograms and input noise.

EFFECT: improved efficiency of generating realistic audio representations of text.

17 cl, 7 dwg

Description

TECHNICAL FIELD

[01] The present technology relates to generative modeling and, in particular, to a method and server for generating a waveform.

BACKGROUND OF THE ART

[02] Generative modeling is an unsupervised learning task in machine learning that involves automatically detecting and learning regularities or patterns in input data so that the model can be used to generate or infer new examples that could be plausibly extracted from original data set. During the learning process, a large body of data in a particular domain can be collected (eg millions of images, sentences or sounds, etc.) and can be used to train a generative model to generate or synthesize such data.

[03] For example, generative adversarial networks (GANs) use a technique for training a generative model by framing the problem as a supervised learning problem with two submodels: a generator model that learns to generate new examples, and a discriminator model that attempts to classify examples as " real" (from a given area), or "fake" (generated). The two models are trained together in a zero-sum adversarial game until the discriminator model fails to fool about half the time, meaning the generator model generates plausible examples.

[04] In another example, normalizing flow (NF) models are a family of generative models with easily processed distributions, where both sampling and density estimation can be efficient and accurate. Applications include image generation, noise modeling, video generation, audio generation, graph generation, reinforcement learning, computer graphics, and physics.

[05] Speech generation is used in many different applications, from smart assistants to synthetic voices for people unable to speak independently. Various methods for generating artificial voices have been developed, but it has been difficult to effectively create realistic and stylized voices.

[06] Image generation is used in many different applications, including upscaling algorithms and image restoration. Various methods for generating artificial images have been developed, but it has been difficult to generate realistic image data in an efficient manner.

[07] US Patent Application No. 2020/402497 discloses a student NF network system for learning the distribution of teacher CNN spectrograms.

SUMMARY OF THE INVENTION

[08] The purpose of the present technology is to eliminate at least some of the problems present in the prior art. Embodiments of the present technology may provide and/or expand the scope of approaches and/or methods for achieving the goals and objectives of the present technology.

[09] Some aspects of the present technology provide a process of distillation of knowledge between the teacher model and the student model to form a teacher-student network. More specifically, the teacher model is instantiated as a pretrained feed-forward model having reversible blocks, and the student model is instantiated as an untrained feed-forward model having irreversible blocks.

[10] The developers of this technology have developed a teacher-student network in which the student architecture in a sense “sacrifices” the reversibility of the teacher architecture in exchange for a more compact and efficient model compared to the teacher architecture. Using more compact and efficient models instead of corresponding flow-based models is useful for a number of different computer-implemented applications such as speech synthesis and super-resolution.

Flow-based models

[11] Generally speaking, a flow-based generative model is a generative model that explicitly models a probability distribution using a probabilistic law of variable variation to transform a simple distribution into a complex one. Direct likelihood modeling can provide many benefits. For example, the negative log-likelihood can be directly calculated and minimized as a loss function. Additionally, new samples can be generated by sampling from the original distribution and applying a stream transformation. In comparison, many alternative generative modeling techniques, such as variational autoencoder (VAE) and generative adversarial network (GAN), do not explicitly represent the likelihood function.

[12] It should be noted that flow-based models can use different classes of flow transformations, such as autoregressive and bipartite transformations. Autoregressive transformations include autoregressive flow (AF) and inverse autoregressive flow (IAF). AFs are similar to autoregressive models, which allow parallel density estimation but perform sequential synthesis. In comparison, IAF allows parallel synthesis but performs sequential density estimation, making likelihood learning very slow. The second group includes bipartite transformations that enable efficient likelihood learning and parallel synthesis. However, bipartite flows are usually less expressive than AF. Therefore, they require more layers and more parameters to achieve comparable capacity.

[13] Generally speaking, normalizing flow (NF) is a transformation of a simple probability distribution (such as the standard normal) into a more complex distribution using a sequence of invertible and differentiable mappings. The sampling density can be estimated by transforming it back to the original simple distribution and then calculating the product of i) the density of the inversely transformed sample under that distribution and ii) the associated change in volume caused by the sequence of inverse transformations. The change in volume is the product of the absolute values of the Jacobian determinants for each transformation, as required by the change of variables formula. The result of this approach is a mechanism for constructing new families of distributions by choosing an initial density and then combining a number of parameterized, invertible and differentiable transformations. The new density can be obtained by sampling (by sampling from the initial density and applying transformations), and the density in the sample (ie the likelihood) can be calculated.

[14] The developers of the present technology realized that NF models are a powerful class of generative models that demonstrate good performance on several speech- and vision-related problems. As mentioned above, these models use a change of variables formula that provides a "normalized" density after applying invertible transformations. The reversible transformations of such models can be combined with each other, and in a "flow" sense, to create more complex reversible transformations. Unlike other generative models, NF models are invertible, have easily processed likelihoods, and provide stable learning. Since NF models allow exact likelihood calculation using a variable substitution formula, they can be trained using likelihood maximization techniques. As mentioned above, this property makes NF models more desirable in some situations than, for example, GAN models, which require adversarial optimization and may suffer from vanishing gradients, mode collapse, oscillations, or cyclic behavior.

[15] Some NFs can be configured to generate output data based on input noise; however, other NFs can be further "conditioned" based on conditioning inputs. Conditional NFs are gaining popularity in various practical applications related to speech and vision, in particular for waveform synthesis, image super-resolution, and point cloud generation. Developers of this technology have developed methods and systems that use conditional flow-based models for image super-resolution and speech synthesis.

Super resolution apps

[16] Super-resolution (SR) is one of the fundamental problems in image processing, which aims to improve the quality of low-resolution (LR) images by upscaling them to high-resolution (HR) images with natural high-frequency details. The developers of the present technology realized that most image SR approaches use GAN models, which can be difficult to train due to the instability of adversarial optimization. To overcome this shortcoming, the SRFlow model can be used. How the SRFlow model can be implemented is described in a paper entitled “SRFlow: Learning the Super-Resolution Space with Normalizing Flow,” published in 2020 by Lugmayr et al., the contents of which are incorporated herein in full by reference.

[17] Generally speaking, the SRFlow model has a flow-based architecture for estimating the full conditional distribution of natural HR images corresponding to a specific LR image, whose parameters are trained by likelihood maximization. Specifically, the SRFlow model architecture consists of an LR encoder that provides the characteristic features of LR images to a reversible flow network. The streaming network accepts the corresponding LR encoding along with a sample from the initial distribution and converts them into SR image. The LR encoder is a popular forward propagation SR architecture based on Residual in Residual Density Blocks (RRDB). This block applies multiple residual and dense skip connections without any batch normalization layers. To form a multi-level LR representation, the activations after each RRDB block are concatenated. The streaming network is organized into L levels, each of which operates at a resolution , Where - level index, and - HR permission. Each layer first performs a compression operation (squeeze), which reduces the spatial resolution by half. The compression operation is followed by a transition stage in which linear reversible interpolation between adjacent pixels is learned. A sequence of conditional flow stages is then applied. At the end of each level, 50% of the channels are separated to evaluate the likelihood. Each flow stage consists of four layers. First, an activation normalization layer (actnorm) and a 1x1 convolution are applied. This is followed by the affine injector and conditional affine connection layers.

[18] In some embodiments of the present technology, the developers of the present technology have developed a teacher-student network in which the teacher model is an SRFlow model and the student model is a feedforward model trained for use in SR applications in place of the corresponding SRFlow model.

Speech synthesis applications

[19] Speech synthesis can be achieved using deep neural models, sometimes called "neural vocoders". It should be noted that the neural vocoder can synthesize time-domain waveforms that can be driven by mel spectrograms from the text-to-spectrogram model.

[20] Most modern neural vocoders are autoregressive models, which provide good performance but suffer from a slow sequential generation process. The runtime requirements of autoregressive vocoders limit their use in real-time applications. The developers of this technology realized that there was a need for techniques aimed at speeding up the inference of such models. There are also GAN-based vocoders, but they are usually inferior to autoregressive vocoders in terms of speech reproduction accuracy.

[21] The developers of this technology realized that flow-based models could be successfully used to synthesize waveforms in parallel with fidelity comparable to autoregressive models. Such flow-based vocoders include the WaveGlow model. How the WaveGlow model can be implemented is described in the paper entitled “WaveGlow: A Flow-Based Generative Network For Speech Synthesis,” published on October 31, 2019, by Prenger et al., the contents of which are incorporated herein in their entirety by reference.

[22] Generally speaking, a WaveGlow vocoder contains a sequence of bipartite streaming stages. Each flow stage consists of a 1x1 invertible convolution followed by a conditional affine link layer. The WaveGlow vocoder is driven by upsampled mel spectrograms through affine coupling layers to WaveNet blocks. How a WaveNet block can be implemented is described in the article entitled "Wavenet: A Generative Model For Raw Audio", published on September 19, 2016 by Aäron van den Oord et al., the contents of which are incorporated herein in their entirety by reference.

[23] In some embodiments of the present technology, the developers of the present technology have developed a teacher-student network in which the teacher model is a WaveGlow vocoder and the student model is a feedforward vocoder trained for use in speech synthesis applications in place of the corresponding WaveGlow vocoder.

[24] To enable accurate likelihood computation, NF architectures must consist of invertible modules that also support efficient computation of their Jacobian determinant. Recently, a large number of such modules have been developed, including autoregressive, bipartite, linear and residual transformations. Although some modules are more efficient than others, NFs are generally inferior to their direct propagation counterparts in terms of inference time. In particular, autoregressive streams use a slow sequential generation procedure, while bipartite streams may require a large number of submodules with low expressive power. Moreover, reversibility limits the size of activation maps between modules, leading to bottlenecks and hence impractical deeper models.

[25] However, in many practical applications, explicit density estimation may not be necessary during inference. As mentioned earlier, the developers of this technology realized that the reversibility of NF could be sacrificed to improve execution time and memory consumption. Developers realized that the training stage can benefit from robust explicit likelihood learning from NF, and that in deployment, efficient inference can be obtained from more compact feedforward architectures.

Distillation of knowledge

[26] In some aspects of the present technology, developers have invented a process for distilling knowledge from pre-trained flow-based models into compact and more efficient feed-forward architectures without the reversibility property of NF architectures.

[27] Knowledge distillation is one of the most popular compression and acceleration techniques for large models and ensembles of neural networks. Generally speaking, the idea of knowledge distillation is to train an efficient student on predictions produced by a computationally intensive teacher. It can be said that the knowledge distillation technique disclosed in this paper eliminates the computational inefficiency of flow-based models and allows knowledge to be transferred from NFs to feed-forward networks that do not necessarily need to be reversible.

[28] In at least some embodiments of the present technology, learning methods and learner structures are designed that allow the distillation of knowledge from conditional NFs into a feed-forward model with faster output and/or less memory consumption. It is believed that in at least some implementations of the present technology, the distillation processes disclosed herein may result in faster output without noticeable loss of quality.

Distillation procedure

[29] Referring to Figure 4, a representation of a teacher-student network 400 provided in some embodiments of the present technology is depicted. The teacher-student network 400 includes a teacher model representing the teacher-related portion 402 of the network 400 and a student model representing the student-related portion 404 of the network 400. It should be noted that the representation of the network 400 illustrated in Figure 4 is a simplified representation of the actual teacher network. -student, which can be used in various implementations of this technology. How the teacher-student network 400 can be implemented in some embodiments of the present technology is described in the article entitled "Distilling the Knowledge from Normalizing Flows", which is attached hereto.

[30] The teacher associated portion 402 comprises flow stage blocks 412-1, 412-2,..., 412-3 and 412-4. It should be noted that blocks 412-1 to 412-4 of flow stages are reversible blocks. Therefore, the teacher-related portion 402 can be said to be a reversible (flow-based) model. The student-related portion 404 contains forward propagation blocks 424-1,... and 414-2. Therefore, it can be said that the student-related part 404 is an irreversible (forward propagation) model.

[31] Both the teacher portion 402 and the student portion 404 are provided with an input data set 410 including a noise sample 406 (input noise) and context information 408 (conditioner). The developers of the present technology realized that using the same input data set 410 for the teacher portion 402 and the student portion 404 allows a non-reversible, feed-forward student model to be trained from a reversible, flow-based teacher model.

[32] Thus, input data set 410 may be used by teacher associated portion 402 to generate teacher output sample 422, as well as student associated portion 404 to generate student output sample 424. Teacher output sample 422 and student output sample 424 may be used to calculating the loss function 460. In the example illustrated in Figure 4, feature extraction model 450 may be used to extract features from teacher output sample 422 and student output sample 424 for further use by loss function 460.

[33] How the teacher associated portion 402 and the student associated portion 404 may be designed by one skilled in the art familiar with the principles set forth herein will now be discussed. However, architectures other than those not exhaustively described herein may be used to form a teacher-student network having a flow-based teacher model and a feedforward student model.

Teacher structure

[34] The teacher portion 402 may be implemented in the form of a pre-trained conditional flow-based model that determines a deterministic bijective mapping from the sample noise and context information to the output sample The developers of the present technology realized that this mapping could be approximated by a more efficient model in a controlled manner using an objective loss function. In particular, a combination of "recovery" losses and "feature" losses may be used by loss function 460. In some implementations, the training loss function 460 may be defined as:

Where represents a hyperparameter that can be used to balance the corresponding loss terms. It is envisaged that a combination of reconstruction loss and feature loss can be used for a number of applications, such as speech synthesis applications and super-resolution applications.

[35] For the recovery loss, one can use, for example, the average L1-norm between the student-related sample and the teacher-related sample:

Where And correspond, respectively, to the teacher and student models from the teacher-student network 400.

[36] For feature loss, for example, the distance between the set of feature representations of the generated samples can be used.

[37] In implementations where the teacher-student network 400 is to be used for SR applications, such as during distillation of the SRFlow model, the feature loss can be an example of the perceptual distance between the generated images, which is calculated using the trained perceptual similarity metric of fragments images (LPIPS). The feature extraction model 450 may be implemented as a pre-trained model VGG16 (convolutional neural network model) for computing LPIPS. In this case, the perceptual loss can be defined as follows:

[38] In implementations where the teacher-student network 400 is to be used for speech synthesis applications such as WaveGlow model distillation, an example would be the multi-resolution short-time Fourier transform (STFT) loss, which is the sum of the losses per based on STFT with different analysis parameters (i.e. FFT size, window size and frame shift).

[39] For example, one STFT-based loss could be the sum of the spectral convergence term ( ) and the logarithmic absolute value term STFT ( ). Multi-resolution STFT-based loss for permissions can be defined as:

Where And denote the Frobenius and L1 norms respectively, and denotes the absolute values of STFT.

[40] The developers of the present technology realized that after the distillation process, feedforward student models are able to extract samples from the latent distribution of the flow-based teacher model, and this allows them to generate different output samples for certain inputs and/or use latent geometry to perform editing operations .

Student Structure

[41] Referring to Figure 5, depicts a representation of a first learner architecture 502 that can be used for SR applications. For example, the first student architecture 502 may be used to design the SRFlow student model while distilling the SRFlow teacher model. However, it should be noted that other architectures for student and teacher models may be used in other implementations for SR applications without going beyond the scope of the present technology.

[42] Generally speaking, the SRFlow learner model consists of L layers. At each level, activations from the previous level are concatenated with the corresponding vector noise and LR representation and are passed through a sequence of RRDB blocks. Then the transition stage and the unqueeze operation are applied. The decompression operation doubles the spatial resolution of feature maps and reduces the dimension of their channels by 4 times.

[43] Figure 5 also illustrates a representation of a second learner architecture 504 that can be used for speech synthesis applications. For example, a second student architecture 502 may be used to design a WaveGlow student model while distilling a WaveGlow teacher model. However, it should be noted that other architectures for student and teacher models may be used in other implementations for speech synthesis applications without departing from the scope of the present technology.

[44] Generally speaking, a WaveGlow learner model is a sequence of WaveNet conditional blocks. Each WaveNet block is organized into several residual layers, each of which consists of a dilated convolution, followed by a guided activation block and a 1x1 convolution. Upsampled chalk spectrograms are added to the intermediate activations before the guided activation block. Unlike the WaveGlow teacher model, the WaveGlow student model does not introduce noise between the WaveNet intermediate blocks, but instead receives z at the very beginning.

[45] As mentioned above, regardless of the specific implementation of the flow-based teacher model and feed-forward student model, the developers of the present technology have designed a teacher-student network in which the student architecture in some sense “sacrifices” the reversibility of the teacher architecture in exchange for a more compact and effective model. The architecture of the student model (such as the first student architecture 502 and the second student architecture 504) may have some similarities with the architecture corresponding to the flow-based teacher model, however, in some implementations of the present technology, there may be noticeable differences.

[46] First, unlike the flow-based teacher model, the student layers can be organized in reverse order, since the student model can represent the "backward pass" (sequence of blocks) of the teacher.

[47] Second, stream layers are replaced by widely used feedforward modules for their respective tasks. In those implementations where the teacher-student network 400 is to be used for SR applications, such as during distillation of an SRFlow model, the flow stages of the SRFlow teacher model may, by way of example, be replaced by bundled RRDBs in the corresponding SRFlow student model. In those implementations where the teacher-student network 400 is to be used for speech synthesis applications, such as during distillation of the WaveGlow model, the flow stages of the WaveGlow teacher model can be replaced by WaveNet blocks in the WaveGlow student model, as an example.

[48] Third, since the feedforward learner model no longer needs to be invertible, the number of covert channels between modules of the learner model can change. Changing the number of covert channels can reduce the number of bottlenecks in the feedforward learner model. The developers of the present technology realized that such student architectures may require fewer blocks than the corresponding teacher architecture to achieve similar performance.

[49] Fourth, in conditional NF models, contextual information is continuously input through conditional flow layers. After replacing these layers, this information must be entered into the network. In those implementations where the teacher-student network 400 is to be used for SR applications, such as during distillation of the SRFlow model, the LR encodings in the SRFlow student model may, by way of example, be concatenated with intermediate activations at the beginning of each level. In those implementations where the teacher-student network 400 is to be used for speech synthesis applications, such as during WaveGlow model distillation, the WaveGlow student model can retain its original conditioning mechanism through WaveNet blocks.

[50] In a first broad aspect of the present technology, a method for generating a waveform based on a spectrogram and input noise is provided. The spectrogram was generated from the text. A waveform is a digital audio representation of text. The method is executable via a server. The method comprises receiving by a server a trained flow-based vocoder including a plurality of reversible blocks. The trained flow-based vocoder was trained to generate a waveform based on the spectrogram and input noise. The method comprises receiving by the server an untrained forward propagation vocoder including a plurality of irreversible blocks. Moreover, the trained flow-based vocoder and the untrained feedforward vocoder form a teacher-student network. The method comprises performing a learning process in a teacher-student network. During a particular training iteration of the learning process, the method comprises: the server generating a teacher-related waveform via a trained flow-based vocoder using a first spectrogram and a first input noise, the server generating a student-related waveform via an untrained feed-forward vocoder using the first spectrogram and a first input noise. input noise, the server generating a loss value for a given training iteration using the teacher-related waveform and the student-related waveform, and training the server to generate an untrained feedforward vocoder to generate a waveform using the loss value to approximate the mapping between the first input noise and the teacher-related waveform. waveform of a stream-based vocoder. The method comprises receiving a spectrogram and input noise by the server. The method comprises using a server to use a trained forward propagation vocoder instead of a trained flow-based vocoder to generate a waveform based on a spectrogram and input noise.

[51] In some embodiments of the method, the spectrogram is a chalk-scaled spectrogram.

[52] In some embodiments of the method, the input noise is a normal distribution of noise values.

[53] In some embodiments of the method, the method further comprises storing the trained feedforward vocoder by the server in storage.

[54] In some embodiments of the method, the loss value is a combination of a recovery-based loss value and a feature-based loss value, wherein the recovery-based loss value represents the difference between a teacher-related waveform and a student-related waveform, and the value-based loss on loss features represents the difference between teacher-related waveform features and student-related waveform features.

[55] In some embodiments of the method, the trained flow-based vocoder is a conditional flow-normalizing vocoder.

[56] In some embodiments of the method, the trained feedforward vocoder is a convolutional neural network (CNN)-based vocoder.

[57] In a second broad aspect of the present technology, a method is provided for generating output data using input conditioning data and input noise. The method is executable via a server. The method comprises receiving by a server a trained conditional flow-based model including a plurality of invertible blocks, wherein the trained conditional flow-based model has been trained to generate output based on input conditioning data and input noise. The method comprises receiving by the server an untrained feedforward model including a plurality of irreversible blocks, wherein the trained conditional flow-based model and the untrained feedforward model form a teacher-student network. The method comprises performing a learning process in a teacher-student network. During a particular training iteration of the learning process, the method comprises: the server generating teacher-related outputs through a trained conditional flow-based model using the first input conditioning data and the first noise input, the server generating student-related outputs through an untrained feedforward model using the first input conditioning data and the first input noise, the server generating a loss value for a given training iteration using the teacher-related output and the student-related output, and the server training an untrained feedforward model by using the loss value to approximate the mapping between the first input noise and the associated with the supervised output of a conditional flow-based model. The method includes the server receiving input conditioning data and input noise. The method comprises using the server to use a trained feedforward model instead of a trained conditional flow-based model to generate output data based on input conditioning data and input noise.

[58] In some embodiments of the method, the trained conditional flow-based model is a trained conditional flow-based vocoder, wherein the trained feedforward model is a trained feedforward vocoder, wherein the input conditioning data is a spectrogram and the output data is a waveform. form.

[59] In some embodiments of the method, the trained conditional flow-based model is a trained conditional flow-based image enhancement model, wherein the trained forward propagation model is a trained forward propagation image enhancement model, wherein the input conditioning data is a first image, the output the data is a second image, the second image having a higher resolution than the first image.

[60] In a third broad aspect of the present technology, a waveform generation server is provided based on a spectrogram and input noise. The spectrogram was generated from the text. A waveform is a digital audio representation of text. The server is configured to receive a trained flow-based vocoder including a plurality of reversible blocks, wherein the trained flow-based vocoder has been trained to generate a waveform based on a spectrogram and input noise. The server is configured to receive an untrained feedforward vocoder including a plurality of irreversible blocks, wherein the trained flow-based vocoder and the untrained feedforward vocoder form a teacher-student network. The server is configured to carry out the learning process in a teacher-student network. During a particular training iteration of the learning process, the server is configured to: generate a teacher-related waveform through a trained flow-based vocoder using the first spectrogram and a first input noise, generate a student-related waveform through an untrained feedforward vocoder using the first spectrogram and the first input noise, generating a loss value for a given training iteration using the teacher-related waveform and the student-related waveform, and training the untrained feedforward vocoder to generate a waveform using the loss value to approximate the mapping between the first input noise and the teacher-related waveform stream-based vocoder. The server is configured to receive a spectrogram and input noise. The server is configured to use a trained feedforward vocoder instead of a trained flow-based vocoder to generate a waveform based on the spectrogram and input noise.

[61] In some server embodiments, the spectrogram is a chalk-scaled spectrogram.

[62] In some server embodiments, the input noise is a normal distribution of noise values.

[63] In some server embodiments, the server is further configured to store the trained feedforward vocoder in storage.

[64] In some server embodiments, the loss value is a combination of a recovery-based loss value and a feature-based loss value, wherein the recovery-based loss value represents the difference between a teacher-related waveform and a student-related waveform, and the value-based on loss features represents the difference between teacher-related waveform features and student-related waveform features.

[65] In some server embodiments, the trained flow-based vocoder is a conditional flow-normalizing vocoder.

[66] In some server embodiments, the trained feedforward vocoder is a convolutional neural network (CNN)-based vocoder.

[67] As used herein, a “server” is a computer program that runs on suitable hardware and is capable of receiving requests (eg, from devices) over a network and executing or causing those requests to be executed. The hardware may be one physical computer or one physical computer system, but neither is required for this technology. In the present context, the use of the expression "server" does not imply that every task (for example, received instructions or requests) or any particular task will be accepted, executed or called for execution by the same server (i.e. the same software software and/or hardware); this expression implies that any number of software elements or hardware devices may be involved in receiving/sending, executing or invoking the performance of any task or request, or the consequences of any task or request; and all of this software and hardware may be a single server or multiple servers, both of which are included in the expression “at least one server.”

[68] As used herein, a “device” is any computer hardware that is capable of executing software suitable for the relevant task at hand. Thus, some (non-limiting) examples of devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, and network equipment such as routers, switches, and gateways. It should be noted that a device acting as a device in a given context is not prohibited from acting as a server for other devices. The use of the term "device" does not exclude the use of multiple devices in receiving/sending, performing or causing any task or request to be performed, or the consequences of any task or request, or the steps of any method described herein.

[69] As used herein, a “database” is any structured collection of data, regardless of its specific structure, database administration software, or computer hardware on which data is stored, implemented, or otherwise made available for use. . The database may reside on the same hardware as the process that stores or uses the information stored in the database, or it may reside on separate hardware, such as a dedicated server or multiple servers.

[70] As used herein, the expression “information” includes information of any nature or kind that is capable of being stored in a database in any manner. Thus, information includes, but is not limited to, audiovisual works (images, films, sound recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments , questions, messages, etc.), documents, spreadsheets, word lists, etc.

[71] As used herein, unless specifically stated otherwise, the term "component" is intended to include software (corresponding to a particular hardware context) that is both necessary and sufficient to implement the particular function(s) on which is referred to.

[72] As used herein, the term “computer-usable storage medium” is intended to include media of any nature or kind, including RAM, ROM, disks (CD-ROM, DVD, floppy disks, hard disk drives, etc.) .etc.), USB keys, solid state drives, tape drives, etc.

[73] As used herein, the words “first”, “second”, “third”, etc. are used as adjectives only to enable the nouns they modify to be distinguished from each other, and not to describe any special relationship between such nouns. Thus, for example, it should be understood that the use of the terms "first server" and "third server" does not imply any particular order, type, chronology, hierarchy or ranking (for example) of such servers, nor does their use (in itself ) does not mean that any "second server" must necessarily exist in any given situation. Moreover, as discussed in other contexts in this document, reference to a “first” element and a “second” element does not preclude those two elements from actually being the same real-world element. Thus, for example, in some cases the “first” server and the “second” server may be the same software and/or hardware, in other cases they may be different software and/or hardware.

[74] Each of the implementations of the present technology has at least one of the above-mentioned aspects and/or objectives, but does not necessarily have all of them. It should be understood that some aspects of the present technology that have arisen in an attempt to achieve the above purpose may not satisfy this purpose and/or satisfy other purposes that are not expressly described herein.

[75] Additional and/or alternative features, aspects and advantages of implementations of the present technology will become apparent from the following description, the accompanying drawings and the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[76] For a better understanding of the present technology, as well as other aspects and additional features thereof, reference is made to the following description to be used in conjunction with the accompanying drawings, in which:

[77] Figure 1 illustrates a system suitable for implementing non-limiting embodiments of the present technology.

[78] Figure 2 illustrates a representation of a speech generation engine hosted on the server of Figure 1, as provided in some implementations of the present technology.

[79] Figure 3 illustrates a representation of a server-hosted image generation engine, as provided in some implementations of the present technology.

[80] Figure 4 illustrates a representation of a teacher-student network hosted on a server, as provided in some embodiments of the present technology.

[81] Figure 5 illustrates a representation of a first learner architecture that is used by a speech generation engine and a second learner architecture that is used by an image generation engine, as provided in some implementations of the present technology.

[82] Figure 6 illustrates an iteration of training a student model based on a teacher model from a teacher-student network, as provided in some implementations of the present technology.

[83] Figure 7 is a schematic representation of a method executed by a server, in accordance with at least some non-limiting embodiments of the present technology.

[84] Appendix A is provided at the end of this specification. Appendix A includes a copy of a not-yet-published article entitled "Distilling the Knowledge from Normalizing Flows." This article provides additional background information, descriptions of implementations of non-limiting embodiments of the present technology, and some additional examples. This article is incorporated herein by reference in its entirety in all jurisdictions where such incorporation by reference is permitted.

DETAILED DESCRIPTION

[85] With reference to Figure 1, a schematic representation of a system 100 is illustrated, the system 100 being suitable for implementing non-limiting embodiments of the present technology. It should be clearly understood that the illustrated system 100 is only an illustrative implementation of the present technology. Therefore, the following description is intended only to be used as a description of illustrative examples of the present technology. This description is not intended to define the scope or boundaries of this technology. In some cases, what are considered useful examples of modifications to the system 100 may also be set forth below. This is simply to aid understanding and, again, not to define the scope or set the boundaries of the present technology.

[86] These modifications are not an exhaustive list and, as one skilled in the art will appreciate, other modifications are possible. In addition, those cases where this has not been done (i.e., where no examples of modifications have been provided) should not be interpreted to mean that no modifications are possible and/or that what is described is the only way to implement such an element in the present technology. One skilled in the art will appreciate that this is probably not the case. In addition, it should be understood that in some cases the system 100 may provide simple implementations of the present technology, and that in such cases they are presented to aid understanding. Those skilled in the art will appreciate that various implementations of the present technology can be highly complex.

[87] Generally speaking, system 100 is configured to provide visual and/or audio guidance to user 102. For example, audio guidance 152 (speech utterances or "machine-generated utterances") may be provided by electronic device 104 (or simply "device 104"). to user 102. In another example, visual guidance 154 (a visual representation of a digital image or "machine-generated" digital image) may be provided by device 104 to user 102. Various components of system 100 and how those components may be configured to provide audio guidance 152 and visual instructions 154 will now be described in turn.

User device

[88] As mentioned previously, the system 100 includes a device 104. The implementation of the device 104 is not particularly limited, but, as an example, the device 104 may be implemented as a personal computer (desktops, laptops, netbooks, etc.), device wireless communications (such as smartphones, cell phones, tablets, smart speakers, and the like); and networking equipment (such as routers, switches, and gateways). As such, device 104 may sometimes be referred to as an “electronic device,” an “end user device,” a “client electronic device,” or simply a “device.” It should be noted that the fact that the device 104 is associated with the user 102 does not necessarily suggest or imply any mode of operation - for example, the need to log in, the need to register, or the like.

[89] Device 104 is contemplated to include hardware and/or software and/or firmware (or a combination thereof), as is known in the art, for providing or reproducing audio guidance 152. For example, device 104 may include one or more microphones for detecting or capturing sound and one or more speakers for providing or reproducing audio guidance 152.

[90] Device 104 is contemplated to include hardware and/or software and/or firmware (or a combination thereof) as known in the art for providing or reproducing audio guidance 154. For example, device 104 may have a screen or another display component for rendering and displaying the visual indication 154.

[91] In some embodiments, device 104 may include hardware and/or software and/or firmware (or a combination thereof) as known in the art for running an intelligent personal assistant (IPA) application (not illustrated). Generally speaking, the purpose of an IPA application, also known as a "chatbot", is to allow the user 102 to make requests in the form of spoken utterances and, in response, provide the user 102 with responses in the form of spoken utterances (e.g., audio guidance 152). .

[92] Providing requests and providing responses can be accomplished by the IPA application through a natural language-based user interface. Generally speaking, a natural language user interface in an IPA application can be any type of computer-human interface in which linguistic phenomena such as words, phrases, sentences and the like act as user interface controls for retrieval, selection, modification or otherwise generating data in an IPA application.

[93] For example, when the spoken utterances of the user 102 are detected (i.e., captured) by the device 104, the IPA application may use its natural language-based user interface to analyze the spoken utterances of the user 102 and extract data from them that indicate the user's queries. In addition, data indicating the responses received by device 104 is parsed by a natural language user interface in the IPA application to provide or reproduce spoken utterances (eg, audio guidance 152) indicating those responses.

[94] In at least some embodiments of the present technology, as will be apparent from the description below, electronic device 104 may be configured to receive data to reproduce audio guidance 152 from server 106. This means that in some embodiments, server 106 may be configured to synthesize waveforms in real time and transmit data representing those waveforms (e.g., in segments) to device 104, which in turn is configured to use that data to reproduce audio guidance 152 to user 102 .

[95] In other embodiments, device 104 may comprise hardware and/or software and/or firmware (or a combination thereof), as is known in the art, for performing an image enhancement application. Generally speaking, the purpose of an image enhancement application is to allow a user 102 to provide a poor quality image and, in response, provide the user 102 with an improved version of that image (eg, visual cue 154).

[96] In at least some embodiments of the present technology, as will be apparent from the description below, electronic device 104 may be configured to receive data to display audio guidance 154 from server 106. This means that in some embodiments, server 106 may be configured to use a super resolution (SR) process to increase resolution and/or enhance detail in a low quality digital image, and transmit data representing that SR image to device 104, which in turn is configured to use that data to display visual guidance 154 for the user 102.

Communication network

[97] In an illustrative example of system 100, device 104 is communicatively coupled to communications network 110 to access and transmit data packets to/from server 106 and/or other web resources (not shown). In some non-limiting embodiments of the present technology, communications network 110 may be implemented as the Internet. In other non-limiting embodiments of the present technology, the communications network 110 may be implemented in other ways, such as a wide area communications network, a local communications network, a private communications network, or the like. How a communication link (not specifically numbered) between device 104 and communications network 110 is implemented will depend on, among other things, how device 104 is implemented.

[98] Merely by way of example and not as limitation, in those embodiments of the present technology in which device 104 is implemented as a wireless communication device (such as a smartphone), the communication link may be implemented as a wireless communication link (such as but not restrictions, 3G network line, 4G network line, Wireless Fidelity or WiFi® for short, Bluetooth® and the like). In those examples in which device 104 is implemented as a laptop, the communication link may be either wireless (such as Wireless Fidelity or WiFi® for short, Bluetooth® or the like) or wired (such as an Ethernet connection).

Server

[99] As mentioned previously, system 100 also includes a server 106, which may be implemented as a conventional computer server. In an example embodiment of the present technology, server 106 may be implemented as a Dell™ PowerEdge™ server running the Microsoft™ Windows Server™ operating system. Of course, the server 106 may be implemented in any other suitable hardware, software and/or firmware, or a combination thereof. In the illustrated non-limiting embodiment of the present technology, server 106 is the only server. In alternative, non-limiting embodiments of the present technology, the functionality of the server 106 may be distributed and implemented by multiple servers.

[100] Generally speaking, the server 106 is configured to synthesize waveforms to provide audio guidance 152 and/or to generate SR digital images to display visual guidance 154. It can be said that the server 106 can be used for speech synthesis applications and/or super resolution applications in various implementations of this technology.

[101] In some embodiments, server 106 may receive text and respond by generating a waveform representing the spoken text. For this purpose, the server 106 may have a speech generation engine 130. Generally speaking, speech generation engine 130 contains one or more machine learning algorithms that enable server 106 to synthesize “audio output” representing text. As will be described in more detail below with reference to Figure 2, the speech generation engine 130 may include a first model configured to generate a mel spectrogram based on input text data, and a second model configured to use the mel spectrogram to synthesize a waveform. We can say that the first model is a “text to spectrogram” model. The second model can also be said to be a neural vocoder capable of synthesizing time-domain waveforms that can be driven by mel spectrograms from the text-to-spectrogram model.

[102] In other embodiments, server 106 may receive a low quality digital image and generate a high quality digital image in response. For this purpose, the server 106 may have an image generation engine 140. Generally speaking, image generation engine 140 contains one or more machine learning algorithms that enable server 106 to perform SR processing on digital images. As will be described in more detail below with reference to Figure 3, speech generation engine 130 may include a model configured to generate an SR image by upscaling the LR image to a higher resolution.

[103] Server 106 may have access to a memory device (not illustrated). The server 106 is configured to use a memory device to store data processed by at least some components of the speech generation engine 130 and/or the image generation engine 140. In some embodiments, the memory device may be an integral part of the server 106. However, it is contemplated that the memory device may be a memory device remote from the server 106 without departing from the scope of the present technology.

[104] In at least one embodiment of the present technology, the memory device may be a "graphics processing unit" (GPU) device. Generally speaking, a GPU device contains a specialized processor with dedicated memory that typically performs the floating point operations required for graphics rendering. GPU-type memory devices can be optimized for deep learning models because they can process multiple calculations simultaneously. In other words, deep learning models can be trained faster using GPU-type memory devices, as opposed to central processing unit (CPU)-type devices, by performing operations in parallel (at the same time) rather than sequentially (one after the other).

[105] Indeed, GPU-type memory devices can have a large number of cores, which allows for better computation of multiple parallel processes. In addition, deep learning computations must process enormous amounts of data, making the throughput of GPU-type memory devices most suitable.

[106] In fact, several parameters can make GPUs preferable to CPUs for deep learning applications. Bandwidth is one of the main reasons why GPU is faster for computing than CPU. With large datasets, CPUs use a lot of memory when training a model. On the one hand, computing huge and complex problems uses many clock cycles in a memory device such as a CPU. CPUs execute tasks sequentially and have comparatively fewer cores. On the other hand, GPUs come with dedicated VRAM (Video RAM). Additionally, training a model using deep learning requires a large dataset, hence the computational overhead in terms of memory. For efficient data computation, a GPU type memory device may be a more suitable choice - the larger the number of calculations, the greater the advantage that a GPU type memory device may have over a CPU type memory device.

Database system

[107] The server 106 is communicatively coupled to the database system 120 . Generally speaking, the database system 120 is configured to store information retrieved and/or generated by the server 106 during processing. For example, database system 120 may receive data from server 106 that was retrieved and/or generated by server 106 during processing for temporary and/or permanent storage, and may provide the stored data to server 106 for future use.

[108] The database system 120 may be configured to store spectrogram data and waveform data. Generally speaking, the spectrogram data and waveform data may be used by the server 106 to train at least some components of the speech generation engine 130. For example, server 105 may be configured to generate a specific spectrogram based on a specific waveform, thereby generating a spectrogram-waveform pair. Server 106 may generate and store in database 120 a large number of such spectrogram-waveform pairs for vocoder training. In at least some embodiments, a particular spectrogram-waveform pair may be used to generate a group of training sets for a vocoder. For example, a spectrogram and a waveform from a particular pair may be divided into a number of corresponding parts, wherein the pair from the first spectrogram part and the first waveform part can be used together during the first training iteration, and the pair from the second spectrogram part and the second waveform part can be used together during the second iteration of training.

[109] The database system 120 may be configured to store image data. Generally speaking, the image data may be used by the server 106 to train at least some components of the image generation engine 140. For example, server 106 may be configured to generate an LR image based on a particular HR image, thereby generating an LR-HR image pair. Server 106 may use various downsampling techniques to generate a particular LR image. The server 106 may generate and store in a database 120 a large number of such LR-HR image pairs for training SR algorithms. In at least some embodiments, a particular LR-HR image pair may be used to generate a group of training sets for the SR algorithm. For example, an LR image and an HR image from a particular pair may be divided into a number of corresponding parts, and wherein the pair from the first LR image part and the first HR image part may be used together during the first training iteration, and the pair from the second LR image part and the second part HR images can be used together during the second training iteration.

[110] The database system 120 may be configured to store noise data. Generally speaking, the noise data may be used by the server 106 to train and/or use at least some components of the speech generation engine 130 and the image generation engine 140. For example, database system 120 may store multiple noise distributions, such as normal noise distributions. These noise distributions can be used during the use phase of the flow-based vocoder. These noise distributions can be used during the training phases and use of the feedforward vocoder. These noise distributions can be further used during the use phase of the flow-based SR algorithm. These noise distributions can also be used during the training phases and use of the feedforward SR algorithm.

Speech generation mechanism

[111] Referring to Figure 2, depicts a representation of how speech generation engine 130 may be configured to use input text data 200 and input noise 225 to generate waveform 230. Server 106 may receive input text data 200 in various ways. In one non-limiting example, input text data 200 may represent content to be provided to user 102.

[112] Regardless of how the server 106 receives and/or generates the text input data 200, the goal is to process the text input data 200 by the speech generation engine 130 to generate a waveform 230 (available in the audio representation of the text input data 200) so that it can be provided to the user 102 as a machine-generated utterance. To this end, the speech generation engine 130 includes a text-to-spectrogram model 210 configured to generate a spectrogram 215 based on input text data 200, and a vocoder 220 configured to synthesize a waveform 230 based on the spectrogram 215 and input noise 225.

[113] How the text-to-spectrogram model 210 is implemented is not particularly limited. In one non-limiting example, server 106 may be configured to use a specific machine learning algorithm that has been trained to generate spectrogram data based on input text data. In one non-limiting implementation of the present technology, the text-to-spectrogram model 210 may be implemented through a "Tacotron", which is a sequence-to-sequence machine learning architecture for generating amplitude spectrograms from a sequence of symbols. In some embodiments, the spectrogram 215 may be a chalk spectrogram, which means that one of the axes on the spectrogram has a chalk scale.

[114] In the context of the present technology, vocoder 220 is implemented as a forward propagation vocoder. The feedforward vocoder 220 can be said to be a feedforward generative model configured to synthesize a waveform based on the spectrogram 215 and the input noise 225. The feedforward vocoder 220 can be trained by performing a distillation process of a corresponding flow-based vocoder. The process of distilling a particular flow-based vocoder to train the feedforward vocoder 220 has been described above with reference to Figure 4. In at least some implementations of the present technology, the feedforward vocoder 220 may have the first learner architecture 502 described above with reference to Figure 5. In addition Moreover, in some embodiments of the present technology, feedforward vocoder 220 may be trained and used in a manner similar to that disclosed in the article accompanying this document.

Image generation mechanism

[115] Referring to Figure 3, depicts a representation of how an image generation engine 140 may be configured to use an LR image 300 and input noise 325 to generate an HR image 330. The server 106 may obtain the LR image 300 in various ways.

[116] Regardless of how the server 106 receives the LR image 300, the goal is to process the HR image 300 through the image generation engine 140 to synthesize the SR pixels so that they can be provided to the user 102 as a machine-generated image. To this end, the image generation engine 140 includes a forward propagation SR model 320 . The feedforward SR model 320 can be said to be a generative feedforward model configured to synthesize an HR image based on the LR image 300 and input noise 325. The feedforward model 320 can be trained by performing a distillation process of the corresponding flow-based SR model. The process of distilling a certain flow-based SR model to train the feedforward SR model 320 was described above with reference to Figure 4. In at least some implementations of the present technology, the feedforward SR model 320 may have a second learner architecture 504 described above with reference to the Figure. 5. Additionally, in some embodiments of the present technology, feedforward vocoder 220 may be trained and used in a manner similar to that disclosed in the article accompanying this document.

[117] In some embodiments of the present technology, the server 106 may be configured to execute the method 700 illustrated in Figure 7. The various steps of the method 700 will now be discussed in more detail with reference to Figure 6 and Figure 7.

STEP 702: Obtaining a trained flow-based vocoder including a plurality of reversible blocks

[118] Method 700 begins at step 702, where server 106 receives a trained flow-based vocoder 602. The trained flow-based vocoder includes a plurality of invertible blocks 612. A trained flow-based vocoder was trained to generate a waveform based on a pair of spectrogram (conditioner) and input noise.

[119] In some embodiments, it is contemplated that server 106 may be configured to first train flow-based vocoder 602. For example, server 106 may retrieve spectrogram-waveform pairs stored in database 120 and use them to generate multiple sets training data. The server 106 may then be configured to perform a large number of training iterations based on the multiple training data sets.

[120] In at least some embodiments of the present technology, the spectrograms used during training and use of the conditional flow-based vocoder may be fine-scaled spectrograms. It should be noted that a particular input noise during the use phase of a conditional flow-based vocoder may represent a normal distribution of noise values from which it may be sampled to generate an appropriate waveform. The conditional flow-based vocoder can be implemented as a normalizing flow-based vocoder.

STEP 704: Receive an untrained feedforward vocoder including a plurality of irreversible blocks

[121] The method 700 proceeds to step 704, where the server 106 receives an untrained feedforward vocoder 604. The untrained forward propagation vocoder 604 includes a plurality of 614 irreversible blocks. It is contemplated that feedforward vocoder 604 may be implemented as a specific convolutional neural network (CNN) vocoder.

[122] The server 106 may be configured to form a teacher-student network 600 including a trained flow-based vocoder 602 as a teacher model and an untrained feedforward vocoder 604 as a student model. The server 106 may be configured to perform a distillation process to distill knowledge from the trained flow-based vocoder 602 to the untrained feedforward vocoder 604.

STEP 706: Execution of teaching process in teacher-student network

[123] The method 700 proceeds to step 706, where the server 106 is configured to execute a learning process in the teacher-student network 600 in which the student model is trained based on the teacher model. The learning process includes a number of learning iterations. It can be said that a particular training iteration of the teacher-student network 600 includes an iteration of using the teacher model and an iteration of training the student model.

[124] As illustrated in Figure 6, server 106 may use a trained flow-based vocoder 602 to generate a teacher-associated waveform 622 using the first spectrogram 608 and the first input noise 606, which together form a conditioning element-input noise pair 610.

[125] In some embodiments, a particular flow stage (eg, a reversible block) of the trained flow-based vocoder 602 may be said to receive a conditioner (the first spectrogram 608) to determine the parameters of the affine relation representation. These parameters can then be applied to a noise sample from said noise (first input noise 225). When a particular waveform is synthesized from a conditioner in this manner, the flow-based vocoder can be said to have a "degree of freedom" to generate that waveform, since a number of waveforms can be generated for the same conditioner. Thus, providing input noise during the use phase of the flow-based vocoder allows, in a sense, to lock in this degree of freedom so that the vocoder synthesizes a specific waveform that matches the conditioning element and the input noise. A stream-based vocoder can sample from input noise while generating a specific waveform.

[126] Also, as illustrated in Figure 6, server 106 may use conditioner-input noise pair 610 as input to untrained feedforward vocoder 604 to generate learner-related waveform 624. It should be noted that, in the context of the present technology, the server 106 is configured to use the same conditioner-input noise pair to generate a teacher-related waveform 622 by a trained flow-based vocoder 602 and to generate a student-related waveform 624 by an untrained feedforward vocoder 624.

[127] The server 106 is then configured to generate a loss value 650 for a given training iteration using a teacher-related waveform 622 and a student-related waveform 624. As an example, the loss value 650 may be a combination of (i) a recovery-based value a loss value representing the difference between the teacher-related waveform 622 and the student-related waveform 624, and (ii) a feature-based loss value representing the difference between the attributes of the teacher-related waveform 622 and the student-related waveform 624.

[128] The server 106 is also configured to train an untrained feedforward vocoder 604 using a loss value 650. For example, training of the untrained forward propagation vocoder 604 may be performed in a controlled manner, including a backpropagation loop 670 performed based on the loss value 650. It may be said that the server 106 may thus train the untrained feedforward vocoder 604 to approximate the mapping between at least the first input noise 606 and the teacher associated waveform 622. It may also be said that the server 106 may thus train the untrained feedforward vocoder 604 to approximate the mapping between conditioner-input noise pair 610 and teacher-related waveform 622.

[129] It should be noted that a large number of such training iterations can occur during the training process in the teacher-student network 600 without going beyond the scope of the present technology.

STEP 708: Obtain Spectrogram and Input Noise

[130] The method 700 proceeds to step 708, where the server 106 is configured to obtain a certain spectrogram and a certain input noise. For example, server 106 may be configured to generate a spectrogram 215 based on text 200. Additionally, server 106 may extract input noise 225 from database 120 and/or generate input noise 225 during an iteration of the now trained feedforward vocoder.

STEP 710: Using a trained feedforward vocoder instead of a trained flow-based vocoder to generate a waveform based on the spectrogram and input noise

[131] Method 700 proceeds to step 710, where server 106 is configured to use the now trained feedforward vocoder 604 instead of the trained flow-based vocoder 602 to generate waveform 230 based on spectrogram 215 and input noise 225.

[132] Modifications and improvements to the above-described implementations of the present technology may become apparent to those skilled in the art. The foregoing description is intended to be exemplary and not limiting. It is therefore intended that the scope of the present technology be limited only by the scope of the appended claims.

Claims (51)

1. A method for generating a waveform based on a spectrogram and input noise, wherein the spectrogram was generated from text, the waveform being a digital audio representation of the text, wherein the method is executable by a server and comprises: receiving by the server a trained flow-based vocoder including a plurality of reversible blocks, wherein the trained flow-based vocoder has been trained to generate a waveform based on the spectrogram and the input noise; the server receives an untrained forward propagation vocoder, which includes many irreversible blocks, wherein the trained flow-based vocoder and the untrained feedforward vocoder form a teacher-student network; execution of the learning process in the teacher-student network, and during a certain iteration of the learning process: generating a teacher-related waveform by the server via the trained flow-based vocoder using the first spectrogram and the first input noise; generating a student-related waveform by the server via an untrained feedforward vocoder using the first spectrogram and the first input noise; generating by the server a loss value for a given training iteration using the teacher associated waveform and the student associated waveform; And teaching the untrained feedforward vocoder server to generate a waveform using the loss value to approximate the mapping between the first input noise and the teacher-associated waveform of the flow-based vocoder; the server receives the spectrogram and input noise; And The server uses a trained forward propagation vocoder instead of a trained flow-based vocoder to generate a waveform based on the spectrogram and input noise. 2. The method of claim 1, wherein the spectrogram is a chalk-scaled spectrogram. 3. The method according to claim 1, wherein the input noise is a normal distribution of noise values. 4. The method of claim 1, wherein the method further comprises storing the trained feedforward vocoder in storage by the server. 5. The method of claim 1, wherein the loss value is a combination of the recovery-based loss value and the feature-based loss value, the value of the recovery-based loss represents the difference between the teacher-related waveform and the student-related waveform, and The value of the attribute-based losses represents the difference between the attributes of the teacher-related waveform and the attributes of the student-related waveform. 6. The method of claim 1, wherein the trained flow-based vocoder is a conditional flow normalizing vocoder. 7. The method of claim 1, wherein the trained feedforward vocoder is a convolutional neural network (CNN)-based vocoder. 8. A method for generating output data using input conditioning data and input noise, wherein the method is an executable server and contains: the server receiving a trained conditional flow-based model including a plurality of invertible blocks, wherein the trained conditional flow-based model has been trained to generate output based on the input conditioning data and the input noise; the server receives an untrained forward propagation model, which includes many irreversible blocks, wherein the trained conditional flow-based model and the untrained feedforward model form a teacher-student network; execution of the learning process in the teacher-student network, and during a certain iteration of the learning process: generating teacher-related output by the server via the trained conditional flow-based model using the first conditioning inputs and the first input noise, generating the student-related output by the server via an untrained feedforward model using the first conditioning input and the first noise input, generating by the server a loss value for a given training iteration using the teacher-related output and the student-related output; And training the server on an untrained feedforward model by using a loss value to approximate a mapping between the first input noise and the teacher-associated output of the conditional flow-based model; the server receiving input conditioning data and input noise; And The server uses a trained feedforward model instead of a trained conditional flow model to generate output based on the input conditioning data and the input noise. 9. The method of claim 8, wherein the trained conditional flow-based model is a trained conditional flow-based vocoder, wherein the trained feedforward model is a trained feedforward vocoder, wherein the input conditioning data is a spectrogram, the output data is wave form. 10. The method of claim 8, wherein the trained conditional flow-based model is a trained conditional flow-based image enhancement model, wherein the trained forward propagation model is a trained forward propagation image enhancement model, wherein the input conditioning data is a first image , the output is a second image, with the second image having a higher resolution than the first image. 11. A server for generating a waveform based on a spectrogram and input noise, wherein the spectrogram was generated based on text, the waveform being a digital audio representation of the text, wherein the server is configured to: obtaining a trained flow-based vocoder including a plurality of reversible blocks, wherein the trained flow-based vocoder has been trained to generate a waveform based on the spectrogram and the input noise; obtaining an untrained feed-forward vocoder including many irreversible blocks, wherein the trained flow-based vocoder and the untrained feedforward vocoder form a teacher-student network; performing the learning process in a teacher-student network, and during a certain iteration of the learning process, the server is configured to: generating a teacher-associated waveform via the trained flow-based vocoder using the first spectrogram and the first input noise; generating a student-related waveform through an untrained feedforward vocoder using the first spectrogram and the first input noise; generating a loss value for a given training iteration using the teacher-related waveform and the student-related waveform; And teaching the untrained feedforward vocoder to generate a waveform using a loss value to approximate the mapping between the first input noise and the teacher-associated waveform of the flow-based vocoder; obtaining a spectrogram and input noise; And using a trained feedforward vocoder instead of a trained flow-based vocoder to generate a waveform based on the spectrogram and input noise. 12. The server of claim 11, wherein the spectrogram is a chalk-scaled spectrogram. 13. The server according to claim 11, wherein the input noise is a normal distribution of noise values. 14. The server according to claim 11, wherein the server is additionally configured to store the trained direct propagation vocoder in storage. 15. The server of claim 11, wherein the loss value is a combination of the recovery-based loss value and the feature-based loss value, the value of the recovery-based loss represents the difference between the teacher-related waveform and the student-related waveform, and The value of the attribute-based losses represents the difference between the attributes of the teacher-related waveform and the attributes of the student-related waveform. 16. The server of claim 11, wherein the trained flow-based vocoder is a conditional flow-normalizing vocoder. 17. The server of claim 11, wherein the trained feedforward vocoder is a convolutional neural network (CNN)-based vocoder.

Images