KR20230017286A - Generate conditional output through data density gradient estimation - Google Patents

Abstract

Methods, systems and apparatus are provided that include a computer program encoded on a computer storage medium for using a neural network to generate an output conditioned on a network input. In one aspect, a method includes obtaining a network input; initializing the current network output; Generating a final network output by updating the current network output in each of a plurality of iterations, each iteration corresponding to a respective noise level, the update at each iteration: to generate a noise output. processing a model input for an iteration comprising (i) a current network output and (ii) a network input using a noise estimation neural network configured to process the model input, wherein the noise output is for each value of the current network output including each noise estimate; and updating the current network output using the noise estimate and the noise level for the iteration.

Description

Generate conditional output through data density gradient estimation

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims priority to US Application No. 63/073,867, filed on September 2, 2020, the disclosure of which is incorporated herein by reference.

The present invention relates to using a machine learning model to generate outputs conditioned on network inputs.

A machine learning model receives inputs and produces outputs, for example predicted outputs, based on the received inputs. Some machine learning models are parametric models and generate outputs based on received inputs and the model's parameter values.

Some machine learning models are deep models that use multiple layers of models to generate outputs for received inputs. For example, a deep neural network is a deep learning model that includes an output layer and one or more hidden layers that each apply a nonlinear transformation to received inputs to generate an output.

This specification describes a system implemented as a computer program on one or more computers at one or more locations that generates network outputs conditioned on network inputs.

According to a first aspect of the present invention there is provided a method for generating a final network output comprising a plurality of outputs conditioned on a network input, the method comprising: obtaining the network input; initializing the current network output; Generating a final network output by updating the current network output in each of a plurality of iterations, each iteration corresponding to a respective noise level, the update at each iteration: to generate a noise output. processing a model input for an iteration comprising (i) a current network output and (ii) a network input using a noise estimation neural network configured to process the model input, wherein the noise output is for each value of the current network output including each noise estimate; and updating the current network output using the noise estimate and the noise level for the iteration.

In some implementations, the network input is a spectrogram of an audio segment and the final network output is a waveform for the audio segment.

In some implementations, the audio segment is a speech segment.

In some implementations, the spectrogram is generated from a text segment or linguistic features of a text segment by a text-to-speech model.

In some embodiments, the spectrogram is a mel spectrogram or a log mel spectrogram.

In some implementations, updating the current network output using the noise estimate and the noise level for the iteration includes generating an update for the iteration from at least the noise estimate and the noise level corresponding to the iteration; and subtracting the update from the current network output to produce an initial updated network output.

In some implementations, updating the current network output further comprises modifying the initial updated network output based on the noise level for the iteration to produce a modified initial updated network output.

In some implementations, for a last iteration, the modified initially updated network output is an updated network output after the last iteration, and for each iteration before the last iteration, after the last iteration. The updated network output of is produced by adding noise to the modified initial updated network output.

In some implementations, initializing the current network output includes sampling each of a plurality of initial values for the current network output from a corresponding noise distribution.

In some implementations, the model input for each iteration includes different iteration-specific data for each iteration.

In some implementations, the model input for each iteration includes a noise level corresponding to the iteration.

In some implementations, the model input for each iteration includes a total noise level for an iteration generated from noise levels corresponding to that iteration and any iteration after the iteration of the plurality of iterations. do.

In some implementations, the noise estimation neural network includes a noise generating neural network comprising a plurality of noise generating neural network layers and configured to process the network input to map the network input to a noise output; and a network output processing neural network comprising a plurality of network output processing neural network layers configured to process the current network output to generate an alternate representation of the current network output, wherein at least one noise generating neural network layer comprises (i) noise It receives input derived from the output of another one of the generating neural network layers, (ii) the output of that network output processing neural network layer, and (iii) iteration specific data for the iteration.

In some implementations, the final network output has a higher dimension than the network input, and the alternate representation has the same dimension as the network input.

In some implementations, the noise estimation neural network includes each feature-wise linear modulation (FiLM) module corresponding to each of at least one noise generating neural network layer, and the FiLM module corresponding to a predetermined noise generating neural network layer comprises: To generate the input to the noise-producing neural network layer, (i) the output of another one of the noise-producing neural network layers, (ii) the output of that network output processing neural network layer, (iii) processing the iteration-specific data for the iteration. is configured to

In some implementations, the FiLM module corresponding to the given noise generating neural network layer generates scale vectors and bias vectors from (ii) the output of that network output processing neural network layer, and (iii) iteration specific data for the iteration. do; (i) generate an input to a noise-producing neural network layer by applying an affine transformation to an output of another one of the noise-producing neural network layers;

In some implementations, at least one of the noise generating neural network layers includes an activation function layer that applies a non-linear activation function to an input to the activation function layer.

In some implementations, another of the noise generating neural network layers corresponding to the activation function layer is a residual connection layer or a convolutional layer.

In some implementations, a method of training a noise estimation neural network is provided, the method comprising: obtaining a training network input and a corresponding training network output; selecting iteration specific data from a set comprising iteration specific data for all iterations of a plurality of iterations; sampling a noisy output comprising a respective noise value for each value of the training network output; generating a modified training network output from the noisy output and the corresponding training network output; processing (i) a modified training network output, (ii) a training network input, and (iii) a model input comprising iteration specific data to generate a training noise output using the noise estimation neural network; and determining updates to network parameters of the noise estimation neural network from a gradient of an objective function that measures an error between the sampled noisy output and the training noise output.

In some implementations, the objective function measures the distance between the sampled noisy output and the training noise output.

In some embodiments, the distance is an L1 distance.

Certain embodiments of the subject matter described herein may be implemented to realize one or more of the following advantages.

The described technique produces a network output conditioned on network inputs in a non-autoregressive manner. In general, autoregressive models have been shown to produce high-quality network outputs but require a large number of iterations, resulting in high latency and high resource consumption (eg, memory and processing power). This is because an autoregressive model produces one for each given output in the network output, each conditioned on all outputs that precede the given output in the network output.

On the other hand, the described technique starts with an initial network output, e.g., a noisy output comprising values sampled from a noise distribution, and iterates the network output through a gradient-based sampler conditioned on the network input. refine That is, an iteration noise cancellation process may be used. As a result, this approach is not autoregressive and requires only a certain number of generative steps during inference. For example, in the case of audio synthesis conditioned on the spectrogram, the described technique uses very few computational resources while greatly reducing latency, in very few iterations (e.g., 6 iterations or less). It can produce high-fidelity audio samples, comparable to or even superior to those produced by existing auto-regressive models. Additionally, the described technique may produce samples with higher quality (eg, higher fidelity) than those produced by existing non-autoregressive models.

The details of one or more embodiments of the subject matter in this specification are set forth in the accompanying drawings and the description below. Other features, aspects and advantages of the present invention will become apparent from the description, drawings and claims.

1 is a block diagram of an exemplary conditional output generating system.
2 is a flow diagram of an exemplary process for generating outputs conditioned on network inputs.
3 is a block diagram of an exemplary noise estimation neural network.
4 is a block diagram of an example network output processing neural network block.
5 is a block diagram of an exemplary Feature-wise Linear Modulation (FiLM) module.
6 is a block diagram of an example noise generating neural network block.
7 is a flow diagram of an example process for training a noise estimation neural network.
Like reference numbers and designations in the various drawings indicate like elements.

1 shows an example conditional output generating system 100 . Conditional output generation system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations where the systems, components, and techniques described below are implemented.

A conditional output generation system (100) generates a final network output (104) conditioned on a network input (102).

The conditional output generation system 100 of the present invention is broadly applicable and is not limited to one specific implementation. However, for purposes of explanation, a few example implementations are described below.

For example, the system can be configured to generate a spectrogram, eg, a mel-spectrogram or a waveform of audio conditioned to the spectrogram, where the frequencies are different scales of the audio. As a specific example of this, the spectrogram can be the spectrogram of a speech segment and the waveform can be the waveform for the speech segment. For example, the spectrogram may be the output of a text-to-speech machine learning model that transforms text or linguistic features of text into a spectrogram for an utterance of text being spoken.

As another example, a system can be configured to perform image processing tasks on network inputs to generate network outputs. For example, the network input can be a class of object specifying the class of image object to be created (e.g., represented as a one-hot vector), and the network output can be a generated image of the class of object (e.g., represented as a one-hot vector). It can be expressed as a set of intensity values or RGB values for each pixel in the image.

As another specific example, the task can be conditional image generation and the network input can be a sequence of text and the network output can be an image reflecting the text. For example, a text sequence may include a sequence of sentences or adjectives describing a scene within an image.

In another particular example, the task may be image embedding generation, the network input may be an image and the network output may be a numeric embedding of the input image characterizing the image.

As another specific example, the task can be object detection, the network input can be an image, and the network output can identify locations within the input image at which certain types of objects are depicted, eg, depictions of objects. You can specify the bounding boxes in the containing input image.

As another specific example, a task can be an image segment, a network input can be an image, and a network output can be a segment output that assigns each of a plurality of pixels of the input image to a category of a category set, e.g. For example, each pixel may be assigned a score for each of the categories representing the probability that the pixel belongs to the category.

More generally, a task can be any task that outputs continuous data conditioned on network inputs.

To generate a final network output (104) conditioned on a network input (102), the conditional output generation system (100) obtains the network input (102) and initializes the current network output (114). For example, system 100 samples each value of the current network output from a corresponding noise distribution (e.g., a Gaussian distribution such as N(0,I), where I is the identity matrix) to obtain the current network output ( 114) (ie, create a first instance of the current network output 114). That is, the initial current network output 114 includes the same number of values as the final network output 104, but each value is sampled from a corresponding noise distribution.

Next, system 100 generates final network output 104 by updating current network output 114 in each of multiple iterations. In other words, the last network output 104 is the current network output 114 since the last iteration of multiple iterations.

In some cases, the number of iterations is fixed.

In other cases, system 100 or other systems may adjust the number of iterations based on latency requirements for generation of the final network output. That is, the system 100 can select the number of iterations so that the final network output 104 can be generated while meeting latency requirements.

In still other cases, system 100 or other systems may adjust the number of iterations based on computational resource consumption requirements for generation of final network output 104 . That is, system 100 or another system can choose the number of iterations so that the final network output 104 can be generated while meeting these requirements. For example, this requirement may be the maximum number of floating operations (FLOPS) to be performed as part of generating the final network output.

At each iteration, the system uses the noise estimation neural network 300 to specify (i) the current network output 114, (ii) the network input 112, and optionally (iii) the iteration specific for the iteration. Process model input for iterations containing data. Iteration specific data is generally derived from noise levels 106 (eg, each noise level corresponds to a particular iteration). The system may update the current network output using the noise level 106 as a scale for each iteration of the update. That is, each noise level of noise levels 106 may correspond to a particular iteration, and each noise level for an iteration guides the scale of updates to the current network output 114 in that iteration. can do.

The noise estimation neural network 300 is a neural network having parameters ("network parameters"), which process the model input according to the current values of the network parameters to obtain a respective noise for each value of the current network output 114. and generate a noise output 110 that includes an estimate. The details of the noise estimation neural network are discussed in more detail next with respect to FIG. 3 .

In general, the noise estimate for a given value of the current network output is an estimate of the noise added to the corresponding actual value of the actual network output for the network input to produce that given value. That is, the noise estimate defines how the actual value (if known) needs to be modified to produce that given value at the current network output, given the noise level corresponding to the current iteration. In other words, the predetermined value may be generated by applying a noise estimate to the actual value according to the noise level for the current iteration.

Since this noise estimate can be interpreted as a gradient estimate of the data density, the generation process can be seen as a process of iteratively generating the network output through data density estimation.

Next, system 100 uses update engine 112 to update current network output 114 in the direction of the noise estimate.

In particular, update engine 112 updates current network output 114 using the noise estimate and the corresponding noise level for the iteration. That is, update engine 112 uses the corresponding noise estimate of noise output 110 and the corresponding noise level in the iteration to update each value of current network output 114, which is related to FIG. discussed in more detail.

After the final iteration, the conditional output generation system 100 outputs the updated network output 114 as the final network output 104 . For example, in an implementation where the final network output 104 represents an audio waveform, the system can use a speaker to play the audio or send the audio for playback, and the like. In other implementations where the final network output 104 represents an image, the system may show the image on a user display, send the image for display, and the like. In some implementations, system 100 can store final network output 104 in a data store or transmit final network output 104 to be stored remotely.

Before system 100 uses noise estimation neural network 300 to generate the final network output, system 100 or another system uses training data to train noise estimation neural network 300. In implementations that include a range-affected neural network, the system can also use the training data to train the range-affected neural network as well. Training will be described later with reference to FIG. 7 .

2 is a flow diagram of an exemplary process 200 for generating outputs conditioned on network inputs. For convenience, process 200 will be described as being performed by one or more computer systems located at one or more locations. For example, a conditional output generation system suitably programmed in accordance with this disclosure, such as conditional output generation system 100 of FIG. 1 , may perform process 200 .

The system obtains a network input 202 to condition the final network output. For example, if the network output is an audio waveform, the network input could be a spectrogram, a mel-spectrogram, or linguistic features of a body of text reflected by the audio waveform.

The system initializes the current network output (204). For a final network output containing multiple values, the system can sample from the noise distribution each value of the initial current network output having the same number of values as the final network output. For example, the system can initialize the current network output using a noise distribution denoted by y _N ~ N(0, I) (e.g., a Gaussian noise distribution), where I is the identity matrix and N of y _N is Indicates the intended number of iterations. The system may update the initial current network output during N iterations from iteration N to iteration 1 in descending order.

The system then updates the current network output in each of multiple iterations. In general, the current network output of each iteration can be interpreted as the final network output with additional noise. That is, the current network output is a noisy version of the final network output. For example, for the initial current network output y _N , where N denotes the number of iterations, the system removes the estimate of the noise corresponding to the iteration, so that at each iteration from N to 1 the current network The output can be updated. That is, the system can refine the current network output at each iteration by determining an estimate for the noise and updating the current network output according to the estimate. The system can use descending order through iterations until it outputs the final network output y ₀ .

At each of several iterations, the system uses the noise estimation neural network to process (1) the current network output, (2) the network input, and optionally (3) the model input, including iteration-specific data for the iteration. to generate a noise output for the iteration (206). Iteration-specific data is generally derived from noise levels over the iterations, where each noise level corresponds to a specific iteration. The noise output may include a noise estimate for each value of the current network output. For example, each noise estimate for a particular value of the current network output may represent an estimate of noise added to that actual value of the actual network output for the network input to produce that particular value. That is, the noise estimate for a particular value indicates how the actual value (if known) would need to be modified to produce that particular value, given the corresponding noise level.

At each of several iterations, the system updates the current network output equal to the current iteration using the noise output for the current iteration and the noise level corresponding to the current iteration (208). The system can update each value of the current network output using the corresponding noise estimate of the noise output and the noise level for the current iteration. The system can generate an update for an iteration from the noise estimate and noise level for the iteration, and can subtract the update from the current network output to produce an initial updated network output. The system can then modify the initial updated network output based on the noise level over the iteration, producing the following modified initial updated network output as

(1)

(One)

은 이터레이션 n 에 대한 총 잡음 레벨을 나타내고(예를 들어, 현재 이터레이션 및 현재 이터레이션 이후의 임의의 이터레이션에서의 잡음 레벨들로부터 생성됨),

는 파라미터 θ를 구비한 잡음 추정 신경망에 의해 생성된 잡음 출력을 나타낸다. 잡음 레벨 a_n 및 총 잡음 레벨

은 잡음 스케줄

로부터 결정될 수 있다(예를 들어, 최소값에서 최대값까지 선형 범위의 선형 잡음 스케줄, 피보나치-기반 스케줄 또는 데이터-유도 기반 또는 휴리스틱 방법에서 생성된 커스텀 스케줄). 잡음 레벨

= 1 - βn 이고, 총 잡음 레벨

은 다음과 같이 균일한 분포에서 샘플링될 수 있다. where n indexes the iterations, y _n denotes the current network output in iteration n, y _n-1 denotes the modified initial updated network output, x denotes the network input, and a _n denotes the iteration denotes the noise level for n,

denotes the total noise level for iteration n (e.g., generated from the noise levels in the current iteration and any iterations after the current iteration);

denotes the noise output generated by the noise estimation neural network with parameter θ. Noise level a _n and total noise level

silver noise schedule

(e.g., a linear noise schedule with a linear range from minimum to maximum, a Fibonacci-based schedule, or a data-derived based or custom schedule generated from a heuristic method). noise level

= 1 - βn, and the total noise level

can be sampled from a uniform distribution as

(2)

(2)

이다. 수학식 (2)에서와 같은 샘플링

은, 시스템이 서로 다른 잡음 스케일들에 기초하여 업데이트들을 생성할 수 있게 한다. 각각의 이터레이션 n 에 대한 잡음 레벨 a_n 및 총 잡음 레벨

은 모델 입력의 일부로서 시스템에 의해서 미리결정되고 그리고 획득될 수 있다. where n indexes the iterations,

am. Sampling as in Equation (2)

, allows the system to generate updates based on different noise scales. Noise level a _n and total noise level for each iteration n

may be predetermined and obtained by the system as part of the model input.

For the last iteration, the modified initial updated network output is the updated network output since the last iteration, and for each iteration before the last iteration, by adding noise to the modified initial updated network output, An updated network output since the last iteration is generated. That is, if the iteration is not the final iteration (i.e. n > 1), the system further updates the modified initial updated network output as

(3)

(3)

은 잡음 스케줄

또는 다른 방법(예컨대, 잡음 스케줄의 함수로서, 또는 경험적 실험을 사용하여 하이퍼-파라미터 튜닝을 통해 결정됨)으로부터 결정될 수 있으며,

이다. 다중 모달 분포를 모델링할 수 있도록

가 포함되어 있다.where n indexes the iterations,

silver noise schedule

or determined from other methods (e.g., as a function of noise schedule, or determined through hyper-parameter tuning using empirical experiments);

am. To be able to model multimodal distributions

is included.

The system determines whether the exit criterion has been met (210). For example, the termination criterion may include performing a certain number of iterations (eg, determined to meet a minimum performance metric, a maximum latency requirement, or a maximum computational resource requirement such as a maximum number of FLOPS). If the specified number of iterations are not performed, the system may start again at step 206 and perform another update to the current network output.

If the system determines that the termination criterion has been met, the system outputs the last network output 212, which is the updated network output since the last iteration.

Process 200 can be used to generate network outputs in a non-autoregressive manner conditioned on network inputs. In general, auto-regressive models have been known to produce high-quality network outputs but require a large number of iterations, resulting in high latency and resource (eg, memory and processing power) consumption. This is because auto-regressive models generate each given output in the network output, one at a time, each conditioned on all outputs preceding the given output in the network output. On the other hand, process 200 starts with an initial network output, e.g., a noisy output comprising values sampled from a noise distribution, and iterates the network output through a gradient-based sampler conditioned on the network input. refined with As a result, this approach is non-autoregressive and requires only a certain number of generative steps during inference. For example, in the case of audio synthesis conditioned on a spectrogram, the described technique can generate high fidelity audio samples in very few iterations (e.g., 6 iterations or less). can be generated, which can match or even exceed those produced by state-of-the-art autoregressive models, with significantly reduced latency while using far less computational resources.

3 shows an exemplary architecture of a noise estimation network 300 .

The exemplary noise estimation network 300 includes convolutional neural network layers, noise generating neural network blocks (e.g., each neural network block includes several neural network layers), feature-wise linear modulation: FiLM) module neural network block and network output processing neural network block.

The noise estimation network 300 provides iteration-specific data comprising (1) current network output 114, (2) network input 102, and (3) total noise level 306 corresponding to the current iteration. Processing the model input, including , to produce a noise output (110). The network output 114 has a higher dimension than the network input 102, and the noise output 110 has the same dimension as the current network output 114. For example, if the current network output represents an audio waveform at 24 kHz, the network input may include an 80 Hz mel-spectrogram signal corresponding to the audio waveform (eg predicted by another system during inference).

The noise estimation network 300 includes a number of network output processing blocks for processing the current network output 114 to generate respective alternative representations of the current network output 114 .

Noise estimation network 300 also includes a network output processing block 400 that processes the current network output 114 to produce an alternative representation of the current network output, where the alternative representation has a smaller dimension than the current network output. have

The noise estimation network 300 has an additional network output processing block (e.g. , network output processing blocks 318, 316, 314, 312 (e.g., network 318 processes the replacement representation from block 400 to have a smaller dimension than the output of block 400). generate an alternative representation, block 316 processes the alternative representation from block 318 to generate an alternative representation having a smaller dimension than the output of block 318, and so forth). An alternative representation of the current network output generated from the last network output processing block (eg 312 ) has the same dimensions as the network input 102 .

For example, for a current network output containing an audio waveform at 24 kHz and a network input containing a mel-spectrogram at 80 Hz, the network output processing blocks will run when the alternative representation generated by the last layer 312 is at 80 Hz. (i.e., reduced by a factor of 300 to match the mel-spectrogram), we can “downsample” (i.e., reduce the dimension) by a factor of 2, 2, 3, 5, and 5. (eg, by network output processing blocks 400, 318, 316, 314 and 312 respectively). This architecture of an exemplary network output processing block is discussed in more detail with respect to FIG. 4 .

The noise estimation block 300 processes the iteration-specific data corresponding to the current iteration (e.g., the total noise level 306) and replacement representations from the network output processing neural network blocks to determine the noise generating neural network blocks. The FiLM module contains a number of neural network blocks that generate inputs. Each FiLM module processes the total noise level 306 and a substitute representation from each network output processing block to produce an input to a respective noise generation block (e.g., FiLM module 500 outputs the network output The replacement representation from processing block 400 is processed to generate an input to noise generation block 600, and the FiLM module 328 processes the replacement representation from network output processing block 318 to generate noise block 338. ), and so on). In particular, each FiLM module generates a scale vector and a bias vector as inputs to each noise generation block (e.g., as inputs to an affine transform neural network layer within each noise generation block), see FIG. are discussed in more detail.

The noise estimation network 300 includes a number of noise generating neural network blocks that process the network input 102 and the output from the FiLM modules to produce a noise output 110 . The noise estimation network 300 can include a convolution layer 302 that processes the network input 102 to generate an input to a first noise generation block 332, and from the final noise generation block 600 and a convolutional layer 304 that processes the output of x to generate noise output 110. Each noise generating block produces an output having a higher dimension than the network input 102 . In particular, each noise generating block after the first generates an output having a higher dimension than the output of the previous noise generating block. The final noise generation block produces an output having the same dimensions as the current network output (114).

The noise estimation network 300 includes a noise generation block 332 comprising the output from the convolution layer 302 (i.e., the convolution layer processing the network input 102) and the FILM module The output from 322 is processed to produce an input to noise generation block 334. In addition, the noise estimation network 300 further includes noise generation blocks 336, 338, and 600. Each of the noise generation blocks 334, 336, 338, 600 processes the output of each of the previous noise generation blocks (e.g., block 334 processes the output from block 332, and block 336 processes the output from block 334). processing the output of FILM module 324, etc.) and processing the output from each FiLM module (e.g., noise generation block 334 processes the output from FILM module 324, noise generation block 336 processes the output from the FILM module 326, etc.), and generates input for the next neural network block. Noise generation block 600 generates an input to convolution layer 304, which processes this input to produce noise output 110. The architecture of an exemplary noise generation block (eg, noise generation block 600 ) is discussed in more detail with respect to FIG. 6 .

Each noise-generating block before the last noise-generating block may produce an output having the same dimensions as the corresponding alternative representation of the current network output. For example, noise generation block 332 generates an output having the same dimensions as the alternate representation produced by network output processing block 314, and noise generation block 334 outputs output of network output processing block 316. produce an output with dimensions equal to , and so forth.

For example, if the current network output includes an audio waveform at 24 kHz and the network input includes a mel-spectrogram at 80 Hz, the noise generation blocks are the output of the final noise generation block (e.g., noise generation block 600). 24 KHz (i.e., increased by a factor of 300 to match the current network output of 114), "upsampling" the dimension by a factor of 5, 5, 3, 2, and 2 (i.e., increasing the dimension (e.g., by noise generation blocks 332, 334, 336, 338, and 600, respectively).

4 shows an exemplary architecture of network output processing block 400 .

The network output processing block 400 processes the current network output 114 to generate an alternative representation 402 of the current network output 114 . The alternative representation has a smaller dimension than the current network output. Network output processing block 400 includes one or more neural network layers. One or more neural network layers may include a downsampling layer (e.g., to “downsample” or reduce the dimensionality of the input), an activation layer with a nonlinear activation function (e.g., a fully- It may include various types of neural network layers, including connected layers), convolutional layers, and residual connected layers.

For example, a downsampling layer can be a convolutional layer with a necessary stride to reduce the dimensionality of the input (“downsampling”). In one specific example, a stride X can be used to reduce the dimension of an input by a factor of X (e.g., a stride of 2 can be used to reduce the dimension of an input by a factor of 2, and a stride of 5 can reduce the dimension of an input by a factor of 5). can be used to reduce by a factor of ).

The left branch of the residual linking layer 420 includes a convolution layer 402 and a downsampling layer 404 . The convolution layer 402 may process the current network output 114 to generate an input to the downsampling layer 404 . The downsampling layer 404 processes the output of the convolution layer 402 to produce an input to the residual concatenation layer 420 . The output of the downsampling layer 404 has a reduced dimension compared to the current network output 114 . For example, the convolution layer 402 can include filters of size 1x1 with a stride of 1 (i.e., to maintain dimensionality) and the downsampling layer 404 downsamples the dimensionality of the input by a factor of 2. You can include a filter of size 2x1 with a stride of 2 to sample.

The right branch of the residual concatenation layer 420 includes three blocks of an activation layer followed by a downsampling layer 406 and a convolution layer (e.g., an activation layer 408, a convolution layer 410, activation layer 412, convolutional layer 414, activation layer 416 and convolutional layer 418). The downsampling layer 406 processes the current network output 114 to generate input for the activation layer and the next three blocks of the convolutional layer. The output of the downsampling layer 406 has a smaller dimension compared to the current network output 114 . The next three blocks process the output of the downsampling layer 406 to generate the input to the residual concatenation layer 420 . For example, the downsampling layer 406 may include filters of size 2x1 with a stride of 2 to reduce the dimensionality of the input by a factor of 2 (e.g., matching the downsampling layer 404 as appropriate). to do). Activation layers (eg, 408, 412 and 416) may be fully-connected layers with leaky ReLU activation functions. The convolutional layers (eg, 410, 414, and 418) may include filters of size 3x1 with a stride of 1 (ie, to preserve dimensionality).

Residual linking layer 420 combines the output from the left branch with the output from the right branch to produce alternative representation 402 . For example, the residual linking layer 420 can sum the output from the left branch with the output from the right branch to produce the alternative representation 402 (eg, elementwise addition).

5 shows an example of a Feature-wise Linear Modulation (FiLM) module 500 .

The FiLM module 500 processes the alternative representation 402 of the current network output and the total noise level 306 corresponding to the current iteration to produce a scale vector 512 and a bias vector 516 . The scale vector 512 and the bias vector 516 are specific to a particular layer (e.g., an affine transform) in each noise generation block (e.g., the noise generation block 600 of the noise estimation network 300 in FIG. layer). The FiLM module 500 includes a positional encoding function and one or more neural network layers. The one or more neural network layers may include several types of neural network layers, including residual connected layers, convolutional layers, and activation layers with nonlinear activation functions (eg, fully connected layers with leaky ReLU activation functions).

The left branch of the residual linking layer 508 includes the position encoding function 502. A positional encoding function 502 processes the total noise level 306 to produce a positional encoding for the noise level. For example, the total noise level 306 is a position encoding function that is a combination of the sine function for even dimension indices and the cosine function for odd dimension indices, as in pre-processing for the transformer model. It can be multiplied by (502).

The right branch of the residual connection layer 508 includes a convolution layer 504 and an activation layer 506 . The convolutional layer 504 processes the replacement representation 402 to generate an input to the activation layer 506. The activation layer 506 processes the output of the convolution layer 504 to produce an input to the residual concatenation layer 508 . For example, the convolutional layer 504 can include a filter of size 3x1 with a stride of 1 (i.e., to maintain dimensionality), and the activation layer 506 can include a fully-connected filter with a leaky ReLU activation function. can be hierarchical.

The residual linking layer 508 combines the output from the left branch (e.g., the output from the position encoding function 502) and the output from the right branch (e.g., the output from the activation layer 506) to , can generate inputs to both convolutional layer 510 and convolutional layer 514. For example, the residual linking layer 508 can sum the output from the left branch branch and the output from the right branch to generate inputs for two convolutional layers (e.g., 510 and 514). For example, element-by-element summation).

The convolution layer 510 processes the output from the residual concatenation layer 508 to generate a scale vector 512 . For example, convolutional layer 510 may include a filter of size 3×1 with a stride of 1 (to maintain dimensionality). The convolution layer 514 processes the output from the residual concatenation layer 508 to generate a bias vector 516 . For example, convolutional layer 514 may include a filter of size 3x1 with a stride of 1 (to maintain dimensionality).

6 shows an example noise generation network 600 . Noise generation network 600 is an example of a system implemented as a computer program on one or more computers at one or more locations where the systems, components, and techniques described below are implemented.

Noise generation block 600 is an example neural network architecture of a noise generation block used in a noise estimation neural network, such as noise estimation neural network 300 of FIG. 3 .

Noise generation block 600 processes input 602 and output from FiLM module 500 to produce output 310 . Input 602 may be network input processed by one or more previous neural network layers (e.g., noise generation blocks 338, 336, 334, 332, and convolution layer 302 of FIG. 3). Output 310 may be an input to a subsequent convolution layer, which may process the output 310 to produce a noise output 110 (e.g., convolution layer 304 in FIG. 3). there is. Noise generation block 600 includes one or more neural network layers. One or more neural network layers may be an activation layer with nonlinear activation functions (e.g., a fully connected layer with a leaky ReLU activation function), an upsampling layer (e.g., a layer that “upsamples” or increases the dimensionality of the input), convolutional It may include several types of neural network layers, including layers, affine transformation layers, and residual connection layers.

For example, an upsampling layer can be a neural network layer that “upsamples” (ie increases) the dimensionality of the input. In other words, the upsampling layers produce outputs with a higher dimension than the inputs to these layers. In one specific example, the upsampling layer may produce an output with X copies of each value in the input, increasing the dimensionality of the output relative to the input by a factor X (e.g., input(2, 7, - For 4), it produces an output with 2 copies of each value, such as (2,2,7,7,-4,-4) or (2,2,2,2,2,7,7, 7,7,7,-4,-4,-4,-4,-4), and so on). In general, an upsampling layer can fill each extra spot in the output with a value closest to the input.

The left branch of the residual concatenation layer 618 includes an upsampling layer 602 and a convolution layer 604. Upsampling layer 602 processes input 602 to generate input for convolution layer 604 . The input to the convolutional layer has a higher dimension than input 602 . The convolution layer 604 processes the output from the upsampling layer 602 to produce an input to a residual concatenation layer 618 . For example, the upsampling layer can increase the dimensionality of the input by a factor of two, by producing an output with two copies of each value in input 602 . The convolutional layer 604 may include a filter of dimension 3×1 with a stride of 1 (eg, to preserve dimensionality).

The right branch of the residual connected layer 618 is the activation layer 606 (e.g., a fully-connected layer with a leaky ReLU activation function), an upsampling layer 608, a convolutional layer 610 (e.g. . eg with a 3x1 filter size and a stride of 1) in that order.

Activation layer 606 processes input 602 to generate input for upsampling layer 608 . Upsampling layer 608 increases the dimensionality of the output from activation layer 606 to produce an input to convolutional layer 610, which has a higher dimension than input 602 (e.g., Matching with sampling layer 602 increased by factor 2). The convolution layer 610 processes the output from the upsampling layer 608 to generate an input to an affine transformation layer 612 (e.g., using a filter of dimension 3x1 and stride 1 to preserve dimensionality). So). In addition, the activation layer 614 and the convolution layer 616 further process the output from the affine transformation layer 612 to generate an input to the residual link layer 618 (e.g., the network 614 ) using a leaky ReLU function and a filter of dimension 3x1 and stride 1 for network 616).

For example, the affine transform function may process the output from the previous neural network layer (eg, the convolution layer 610 of the noise generation block 600) and the output from the FiLM module to generate an output. For example, the FiLM module can generate scale vectors and bias vectors. The affine transform layer may add a bias vector to the result of scaling the output of the previous neural network layer using the FiLM module's scale vector (eg, using Hadamard multiplication or elementwise multiplication).

The affine transformation layer 612 may process the output from the convolutional layer 610 and the output from the FiLM module 500 to generate an input to the activation layer 614 (e.g., the FiLM module 500 by adding the bias vector from the FiLM module 500 to the result of scaling the output from the convolutional layer 610 using the scale vector from ).

Residual connection layer 618 combines the output from the left branch (e.g., output from convolution layer 604) and the output from the right branch (e.g., output from convolution layer 616). to generate the output. For example, the residual linking layer 618 can sum the output from the left branch and the output from the right branch to produce an output.

The left branch of residual linking layer 632 includes the output from residual linking layer 618 . The left branch can be interpreted as the identity function of the output from the residual linking layer 618.

The right branch of the residual link layer 632 processes the output of the residual link layer 618 and generates the input to the residual link layer 632, in order the affine transform layer, the activation layer and the two convolution layers. contains sequential blocks. In particular, the first block includes an affine transformation layer 620 , an activation layer 622 and a convolution layer 624 . The second block includes an affine transformation layer 626 , an activation layer 628 and a convolution layer 630 .

For example, for each block, each affine transformation layer may process the output from the FiLM module 500 and the output from each previous neural network layer to generate a respective output (e.g., Affine transformation layer 620 can process the output from residual linking layer 618, and affine transformation layer 626 can process the output from convolution layer 624). Each affine transformation layer may generate each output by scaling the output of the previous neural network layer with a scale vector from the FiLM module 500 and summing the scaling result with the bias vector of the FiLM module 500. Each activation layer (eg, 620 and 628) can be a respective fully-connected layer with a leaky ReLU activation function. Each convolutional layer may include filters of dimension 3x1 and stride 1, respectively (eg, to preserve dimensionality).

The residual linking layer 632 is the output from the left branch (e.g., the identity of the output from the residual linking layer 618) and the output from the right branch (e.g., the output from the convolutional layer 630). may be combined to produce output 310 . For example, residual linking layer 632 can sum the output from the left branch and the output from the right branch to produce output 310 . Output 310 may be an input to a convolution layer (eg, convolution layer 304 of FIG. 3 ), which convolution layer will produce noise output 110 .

The noise generation block 600 may include several channels. Each noise generating block (eg, 600, 338, 336, 334, and 332) of FIG. 3 may include several channels. For example, the noise generating blocks 600, 338, 336, 334, and 332 may include 128, 128, 256, 512, and 512 channels, respectively.

7 is a flow diagram of an example process for training a noise estimation neural network. For convenience, process 700 will be described as being performed by one or more computer systems located at one or more locations.

The system may perform process 700 in each of multiple training iterations to iteratively update the parameter values of the noise estimation neural network.

This system obtains (702) a batch of training network input-training network output pairs. For example, the system can randomly sample training pairs from a data store. For example, each training network output can be an audio waveform and each network input can be a ground-true Mel-spectrogram computed from the corresponding audio waveform.

For each training pair in the batch, the system selects the iteration-specific data from the set containing the iteration-specific data for all iterations (704). For example, the system can sample a particular iteration from a discrete uniform distribution containing integers from 1 to the final iteration, and based on the particular iteration sampled from the distribution, iterate-specific data is generated. You can choose. The iteration specific data may include the noise level, the total noise level (e.g. determined in Equation (2)) or the number of iterations itself. Thus, the system can condition the noise estimation network on a discrete index or on a continuous scalar representing the noise level. Conditioning on a continuous scalar representing the noise level can be advantageous, since once the noise estimation network is trained, the inference can use a different number of refinement steps (i.e., iterations) when generating the final network output. am.

For each training pair in the batch, the system samples 708 the noisy output, which includes a respective noise value for each value of the training network output. For example, the system can sample the noisy output from a noise distribution. In a specific example, the noise distribution may be a Gaussian noise distribution (e.g., N(0, I), where I is an identity matrix with dimensions n×n, and n are numeric values of the training network output).

For each training pair in the batch, the system generates a modified training network output from the noisy output and the corresponding training network output (708). The system may combine the noisy output and the corresponding training network output to produce a modified training network output. For example, the system can produce a modified training network output as follows.

(4)

(4)

는 이터레이션 특정 데이터(예: 총 잡음 레벨)를 나타낸다. where y' denotes the modified training network output, y0 denotes the corresponding training network output, ε denotes the noisy output,

denotes iteration specific data (eg total noise level).

을 포함할 수 있다. For each training pair in the batch, the system uses the noise estimation neural network according to the current values of the network parameters, (1) modified training network output, (2) training network input, and (3) iteration specific The model input comprising the data is processed to produce a training noise output (710). The noise estimation neural network may process the model input to generate a training noise output as described in the process of FIG. 2 . For example, an iteration specific criterion is the total noise level.

can include

The system determines an update to the network parameters of the noise estimation network from the gradient of the objective function over the batch of training pairs (712). The system can determine the gradient of the objective function with respect to the neural network parameters of the noise estimation network for each training pair, and then use a variety of suitable optimization methods, such as Stochastic Gradient Descent with Momentum or ADAM, to determine the neural network parameters. Current values can be updated with gradients (eg, a linear combination of gradients, such as the mean of the gradients).

The objective function may measure the error between the training noise output and the noisy output generated by the noise estimation network for each training pair. For example, for a particular training pair, the objective function may include a loss term that measures the L1 distance between the noisy output and the training noise output as

(5)

(5)

는 손실 함수를 나타내고, ε는 잡음성 출력을 나타내고,

는 파라미터 θ를 사용하여 잡음 추정 신경망에 의해 생성된 트레이닝 잡음 출력을 나타내고, y' 는 수정된 트레이닝 네트워크 출력을 나타내고, x 는 트레이닝 네트워크 입력을 나타내고,

는 이터레이션별 데이터(예: 총 소음 레벨)를 나타낸다. here,

denotes the loss function, ε denotes the noisy output,

denotes the training noise output generated by the noise estimation neural network using parameters θ, y' denotes the modified training network output, x denotes the training network input,

represents data per iteration (eg, total noise level).

The system may repeatedly perform steps 702 - 712 for multiple batches (eg, multiple batches of training network input-training network output pairs).

This specification uses the term "configured" in relation to systems and computer program components. In the case of a system consisting of one or more computers configured to perform certain operations or actions, this means having installed on the system software, firmware, hardware or a combination thereof which, when operated, causes the system to perform the operations or actions. When one or more computer programs are configured to perform particular actions or actions, it is meant that the one or more programs, when executed by a data processing device, contain instructions that cause the device to perform the actions or actions.

Embodiments of subject matter and functional operation described herein may be implemented in digital electronic circuitry, tangibly implemented computer software or firmware, computer hardware, or a combination of one or more of these, including the structures disclosed herein and their structural equivalents. It can be. Embodiments of the subject matter described herein may be implemented as one or more computer programs, ie, one or more computer program instruction modules encoded in a tangible, non-transitory storage medium to control the operation or execution by a data processing device. A computer storage medium may be a machine readable storage device, a machine readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be artificially generated radio signals, e.g., machine generated electrical, optical or electromagnetic generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus. can be encoded into the signal.

The term “data processing apparatus” refers to data processing hardware and includes all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The device may also be or further include a special purpose logic circuit, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, such as code that constitutes a processor firmware, protocol stack, database management system, operating system, or a combination of one or more of these.

A computer program, which may also be referred to as or described as program, software, software application, app, module, software module, script, or code, is in any form of programming language, including compiled or interpreted language, or declarative or procedural language. can be written; may be distributed in any form, including stand-alone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A program can, but does not have to, correspond to a file on a file system. A program may be stored in another program or part of a file that holds data, eg one or more scripts stored in a markup language document, a single file dedicated to the program in question, or several control files, eg one or more scripts. It can be stored in files that store modules, sub-programs or parts of code. A computer program may be distributed to be executed on one computer or multiple computers, and multiple computers may be located in one place or distributed in several places and connected by a communication network.

The term "engine" is used herein broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases more than one computer will be dedicated to a particular engine; In other cases, multiple engines can be installed and run on the same computer.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may be performed by special purpose logic circuitry, such as, for example, FPGAs or ASICs, or a combination of special purpose logic circuitry and one or more programmed computers.

A computer suitable for executing a computer program may be based on a general purpose or special purpose microprocessor, or both, or some other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. The essential elements of a computer are a central processing unit that carries out or executes instructions and one or more memory devices that store instructions and data. The central processing unit and memory may be supplemented or integrated by special purpose logic circuitry. Generally, a computer will include, or be operably coupled to, receive data from, transmit data to, or both from one or more mass storage devices (e.g., magnetic, magneto-optical disks, or optical disks) for storing data. . However, a computer does not necessarily need such a device. In addition, a computer may include another device such as a cell phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or portable storage device (eg, Universal Serial Bus (USB) flash drive). Can be built into the device.

Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disk; and CD-ROM and DVD-ROM disks.

To provide interaction with a user, embodiments of the subject matter described herein may be implemented in a computer, which is a display device for displaying information, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). ) monitor, and also a keyboard and a pointing device such as a mouse or trackball through which the user and the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. there is; Input from the user may be received in any form including acoustic, voice, or tactile input. In addition, the computer can interact with the user by exchanging documents with the device used by the user. For example, a web page may be sent to a web browser of a user device in response to a request received from the web browser. The computer may also interact with the user by sending a text message or other type of message to the personal device, for example by sending a message to a smartphone running a messaging application and receiving a response message from the user. can

Data processing apparatus for implementing machine learning models may also include special purpose hardware accelerator units for processing common and computationally intensive parts of machine learning training, ie inference, workloads, for example.

The machine learning model may be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

Embodiments of the subject matter described herein may be implemented in a computing system, which includes a backend component, such as a data server, or a middleware component, such as an application server, or, for example, a graphics server. Include a front-end component, such as a user interface, a web browser, or a client computer having an app through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back-ends, middleware, or front-end components. can include Components of the system may be interconnected by any form or medium of digital data communication, such as, for example, a communication network. Examples of communication networks include Local Area Networks (LANs) and Wide Area Networks (WANs), such as the Internet.

A computing system may include a client and a server. Clients and servers are usually remote from each other and usually interact through a communication network. The relationship of client and server arises by virtue of computer programs running on each computer and having a client-server relationship with each other. In some embodiments, a server sends data, eg, an HTML page, to a user device, eg, to display the data to a user interacting with the device acting as a client and to receive user input therefrom. Data generated by the user device, for example the result of user interaction, may be received at the server from the device.

Although this specification contains many specific implementation details, they should not be construed as limitations on the scope of any invention or what may be claimed, but rather as a description of features that may be particular to particular embodiments of a particular invention. should be interpreted Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in a particular combination, and even initially claimed to do so, one or more features from a claimed combination may be excluded from the combination as the case may be, and the claimed combination may be a subcombination or subcombination. It may be about a variation of the combination.

Similarly, while actions are shown in the drawings and disclosed in the claims in a particular order, this requires that these acts be performed in the particular order shown or in a sequential order or that all of the illustrated acts be performed to obtain a desired result. It should not be understood as Multitasking and parallel processing can be advantageous in certain circumstances. Further, the separation of various system modules and components in the foregoing embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems are generally a single software product or multiple software products. packaged with

Specific embodiments of the subject matter of the present invention have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still obtain desired results. As an example, the processes depicted in the attached figures do not necessarily require the specific order or sequential order shown to achieve desired results. In some cases, multitasking and parallel processing can be advantageous.

Claims (24)

A method of generating a final network output comprising a plurality of outputs conditioned on a network input, comprising:
obtaining network input;
initializing the current network output;
generating a final network output by updating the current network output in each of a plurality of iterations, each iteration corresponding to a respective noise level, the update at each iteration:
processing a model input for an iteration comprising (i) a current network output and (ii) a network input using a noise estimation neural network configured to process the model input to generate a noise output, the noise output being the current network output includes a respective noise estimate for each value of the output; and
and updating the current network output using the noise estimate and the noise level for the iteration. According to claim 1,
wherein the network input is a spectrogram of an audio segment and the final network output is a waveform for the audio segment. According to claim 2,
The method of claim 1, wherein the audio segment is a speech segment. According to claim 3,
wherein the spectrogram is generated from a text segment or linguistic features of a text segment by a text-to-speech model. According to any one of claims 2 to 4,
Wherein the spectrogram is a mel spectrogram or a log mel spectrogram. According to claim 1,
network input is a class of object specifying the class of image object to be created, and network output is a generated image of that class of object, or
The network input is a sequence of text, and the network output is an image reflecting the text, or
The network input is an image and the network output is a numeric embedding of the input image characterizing the image, or
The network input is an image, and the network output identifies locations in the input image where objects of particular types are depicted, or
wherein the network input is an image and the network input is a segment output that assigns each of a plurality of pixels of the input image to a category from a set of categories. In any preceding claim,
Updating the current network output using the noise estimate and the noise level for the iteration comprises:
generating an update to the iteration from at least the noise estimate and the noise level corresponding to the iteration; and
and subtracting said update from a current network output to produce an initial updated network output. According to claim 7,
Updating the current network output comprises:
and modifying the initial updated network output based on the noise level for the iteration to produce a modified initial updated network output. According to claim 8,
For the last iteration, the modified initial updated network output is the updated network output after the last iteration, and for each iteration before the last iteration, the updated network output after the last iteration. wherein is generated by adding noise to the modified initial updated network output. In any preceding claim,
wherein the step of initializing the current network output comprises sampling each of a plurality of initial values for the current network output from a corresponding noise distribution. In any preceding claim,
Wherein the model input for each iteration includes different iteration specific data for each iteration. According to claim 11,
Wherein the model input for each iteration includes a noise level corresponding to the iteration. According to claim 11,
The model input for each iteration is
and a total noise level for an iteration generated from noise levels corresponding to the iteration of the plurality of iterations and any iteration after the iteration. According to any one of claims 11 to 13,
The noise estimation neural network,
a noise generating neural network comprising a plurality of noise generating neural network layers and configured to process a network input to map the network input to a noise output; and
a network output processing neural network comprising a plurality of network output processing neural network layers configured to process the current network output to generate an alternative representation of the current network output;
At least one of the noise generating neural network layers is (i) the output of another of the noise generating neural network layers, (ii) the output of that network output processing neural network layer, and (iii) an input derived from iteration specific data for the iteration. A method characterized in that for receiving. According to claim 14,
wherein the final network output has a higher dimension than the network input, and wherein the alternative representation has the same dimension as the network input. The method of claim 14 or 15,
The noise estimation neural network includes each feature-wise linear modulation (FiLM) module corresponding to each of at least one noise generating neural network layer, and the FiLM module corresponding to a predetermined noise generating neural network layer is included in the noise generating neural network layer. configured to process (i) the output of another one of the noise generating neural network layers, (ii) the output of that network output processing neural network layer, and (iii) iteration specific data for the iteration to generate an input for the iteration. How to. According to claim 16,
The FiLM module corresponding to the predetermined noise generating neural network layer,
generate scale vectors and bias vectors from (ii) the output of the corresponding network output processing neural network layer, and (iii) iteration specific data for the iteration;
(i) generate an input to a noise-producing neural network layer by applying an affine transformation to the output of another one of the noise-producing neural network layers. According to any one of claims 14 to 17,
The method of claim 1 , wherein at least one of the noise generating neural network layers comprises an activation function layer that applies a non-linear activation function to an input to the activation function layer. According to claim 18,
Another of the noise generating neural network layers corresponding to the activation function layer is a residual connection layer or a convolutional layer. A method for training the noise estimation neural network of any one of claims 11 to 19, the method comprising:
obtaining a training network input and a corresponding training network output;
selecting iteration specific data from a set comprising iteration specific data for all iterations of a plurality of iterations;
sampling a noisy output comprising a respective noise value for each value of the training network output;
generating a modified training network output from the noisy output and the corresponding training network output;
processing (i) a modified training network output, (ii) a training network input, and (iii) a model input comprising iteration specific data to generate a training noise output using the noise estimation neural network; and
determining an update to network parameters of a noise estimation neural network from a gradient of an objective function that measures an error between a sampled noisy output and a training noise output;
A method for training a noise estimation neural network, characterized in that it repeatedly performs the steps including. According to claim 20,
wherein the objective function measures a distance between a sampled noisy output and a training noise output. According to claim 21,
The method of training a noise estimation neural network, characterized in that the distance is an L1 distance. A system comprising one or more computers and one or more storage devices storing instructions, which when executed by the one or more computers cause the one or more computers to perform the method of any one of the preceding claims. A system characterized by doing so. A computer readable storage medium encoded with instructions, which when executed by one or more computers causes the one or more computers to perform the method of any one of the preceding claims. available storage media.

Images