DE102021105291A1 - WEIGHT DEMODULATION FOR A GENERATING NEURAL NETWORK - Google Patents

Abstract

A style-based generating network architecture enables scale-specific control of synthesized output data, such as B. Pictures. During the training, the style-based generating neural network (neural generator network) has a mapping network and a synthesis network. During a prediction, the mapping network can be omitted, replicated or evaluated several times. The synthesis network can be used to generate very different, high-quality output data with a large number of attributes. When generating images of people's faces, the following attributes can vary, for example: age, ethnicity, camera position, pose, face shape, glasses, colors (eyes, hair, etc.), hairstyle, lighting, background, etc. Depending on the task, the Output data includes images, audio, video, three-dimensional (3D) objects, text, and so on.

Description

TECHNICAL AREA

The present invention relates to neural networks and, more particularly, to a generator architecture for synthesizing data with scale specific controls.

BACKGROUND

The resolution and quality of images generated by Generative Adversarial Network (GAN) has improved recently. However, GANs still operate as black boxes and despite recent efforts, there is still a lack of understanding of various aspects of the image synthesis process, e.g. B. the origin of stochastic features. The properties of latent space are also poorly understood, and the generally demonstrated interpolations of latent space do not provide a quantitative way of comparing different GANs. There is a need to solve these problems and / or other problems associated with the prior art.

SUMMARY

A style-based generative network architecture enables a scale-specific or scale-specific control of the synthesized output data, such as B. Pictures. During the training, the style-based generative or generating neural network (neural generator network) has a mapping network and a synthesis network. During the prediction, the mapping network can be omitted, replicated or evaluated several times. The synthesis network can be used to generate very different, high-quality output data with a large number of attributes. When creating images of people's faces, the following attributes can vary, for example: age, ethnicity, camera position, pose, face shape, glasses, colors (eyes, hair, etc.), hairstyle, lighting, background, etc. Depending on the task, the The generated output data include images, audio, video, three-dimensional (3D) objects, text, and so on.

A method, computer readable medium and system for synthesizing output data using a neural mapping network and a neural synthesis network are disclosed. A latent code defined in an input space is processed by the neural mapping network to produce an intermediate latent code defined in a latent space. The intermediate latent code is converted into a first style signal. The first style signal is used in a first layer of the neural synthesis network in order to modify first intermediate data in accordance with the first style signal and to generate modified first intermediate data. In one embodiment, the latent intermediate code is a vector which is converted into the first style signal via an affine transformation. The modified first intermediate data is processed to generate second intermediate data, and a second style signal is applied in a second layer of the neural synthesis network to modify the second intermediate data in accordance with the second style signal to generate second modified intermediate data. In one embodiment, the intermediate latent code is a combination of the first and second style signals and a portion of the intermediate latent code is extracted to generate the first and / or second style signals. In one embodiment, the latent intermediate code is converted into the second style signal via an affine transformation. In one embodiment, a second latent code defined in the input space is processed by the neural mapping network to generate a second intermediate latent code defined in the latent space and the second intermediate latent code is converted to the second style signal. In one embodiment, the modified first intermediate data is passed through subsequent layers, such as e.g. A 3x3 convolutional layer, is processed to generate the second intermediate data. The second intermediate data is processed to Generate output data comprising content corresponding to the second intermediate data.

A method, computer readable medium and system are disclosed for synthesizing output data using a neural synthesis network. A first set of spatial noise is applied in a first layer of the neural synthesis network to generate modified first intermediate data that includes content that corresponds to the first intermediate data that is modified depending on the first set of spatial noise. The modified first intermediate data is processed to generate second intermediate data, and a second set of spatial noise is applied in a second layer of the neural synthesis network to generate modified second intermediate data comprising content corresponding to the second intermediate data dependent on the second set of spatial noise. The modified second intermediate data is processed to generate output data comprising content corresponding to the second intermediate data.

Figure list

• 1A FIG. 10 illustrates a block diagram of a style-based generator system in accordance with an embodiment.
• 1B Figure 12 shows images generated by the style-based generator system according to an embodiment.
• 1C shows a flowchart of a method for style-based generation according to an embodiment.
• 2A represents a block diagram of the in 1A illustrated neural mapping network according to an embodiment.
• 2 B represents a block diagram of the in 1A shown neural synthesis network according to an embodiment.
• 2C FIG. 10 illustrates a flow diagram of a method for applying spatial noise using the style-based generator system, according to an embodiment.
• 2D FIG. 10 illustrates a block diagram of a GAN system according to an embodiment.
• 3 illustrates a parallel processing unit according to an embodiment.
• 4A represents a general processing cluster within the parallel processing unit of 3 according to one embodiment.
• 4B FIG. 13 shows a memory partition unit of the parallel processing unit of FIG 3 according to one embodiment.
• 5A turns off the streaming multiprocessor 4A according to one embodiment.
• 5B FIG. 13 is a conceptual diagram of a processing system implemented using the PPU of FIG 3 is implemented according to one embodiment.
• 5C FIG. 10 depicts an exemplary system in which the various architectures and / or functions of the various previous embodiments may be implemented.
• 5D FIG. 10 illustrates components of an exemplary system that may be used to train and use machine learning for use in implementing some embodiments of the present disclosure.
• 6A illustrates artifacts in images generated by the style-based generator system according to one embodiment.
• 6B represents a block diagram of the in 2 B processing blocks shown according to an embodiment.
• 6C FIG. 10 illustrates a block diagram of a style block for use in implementing some embodiments of the present disclosure.
• 6D Figure 12 shows a block diagram of another style block for use in implementing some embodiments of the present disclosure.
• 6E FIG. 3 shows a flow diagram of a method for demodulating weights applied by a neural generator network, according to an embodiment.
• 6F Figure 12 shows images and feature maps generated using demodulated weights according to one embodiment.
• 7A Figure 12 shows images generated by the style-based generator system having high perceptual path length (PPL) values, according to one embodiment.
• 7B illustrates images generated with the style-based generator system having low PPL values, according to one embodiment.
• 7C Figure 12 shows a graph of PPL values for a set of images according to one embodiment.
• 7D Figure 12 shows a graph of PPL values for a set of images generated using smoothing regularization, according to one embodiment.
• 7E Figure 12 shows a conceptual diagram of paths without regularization, according to an embodiment.
• 7F FIG. 13 shows a conceptual diagram of paths with regularization, according to an embodiment.
• 8A FIG. 13 shows a block diagram of a neural synthesis network that is generated using the style block from 6D for use in implementing some embodiments of the present disclosure.
• 8B FIG. 10 shows a block diagram of a training system of a neural generator network according to an embodiment.
• 8C illustrates a flow diagram of a method for smoothing regularization for use in a neural generator network according to an embodiment.

DETAILED DESCRIPTION

A style-based generating network architecture enables scale-specific control of the synthesized output. A style-based generator system has a mapping network and a synthesis network. In one embodiment, feature maps (containing spatially varying information representing the content of the output data, each feature map being a channel of intermediate activations) generated by different layers of the synthesis network are modified depending on style control signals provided by the mapping network will. The style control signals for different layers of the synthesis network can be generated by the same or different latent codes. A latent code can be a random N-dimensional vector, e.g. B. is drawn from a Gaussian distribution. The style control signals for different layers of the synthesis network can be generated from the same or from different mapping networks. In addition, spatial noise can be injected into any layer of the synthesis network.

1A Figure 3 shows a block diagram of a style-based generator system 100 according to one embodiment. The style-based generator system 100 exhibits a neural mapping network 110 , a style conversion unit 115 and a neural synthesis network 140 on. After the neural synthesis network 140 trained, the neural synthesis network 140 without the neural mapping network 110 be used if the latent intermediate code (s) and / or that of the style conversion unit 115 generated style signals are precalculated. In one embodiment, additional style conversion units 115 be present to that of the neural mapping network 110 to convert generated intermediate latent code into a second style signal or to convert another intermediate latent code into the second style signal. One or more additional neural mapping networks 110 can in the style-based generator system 100 be present to generate additional intermediate latent codes from the latent code or additional latent codes.

The style-based generator system 100 can be implemented by a program, a custom circuit, or a combination of a custom circuit and a program. For example, the style-based generator system 100 may be implemented using a GPU (graphics processing unit), CPU (central processing unit), or any processor capable of performing the operations described herein. In addition, those skilled in the art will understand that any system that can perform the operations of the style-based generator system 100 is within the scope and spirit of embodiments of the present invention.

Conventionally, a latent code is provided by an input layer, such as e.g. B. the first layer of a neural feedforward network, provided on a generator. In contrast, in one embodiment, instead of receiving the latent code, the neural synthesis network goes 140 from a learned constant, and the latent code is entered into the neural mapping network 110 entered. In one embodiment, the first intermediate data is the learned constant. With a latent code z in the latent input space Z, a non-linear mapping network f: Z → W first generates a latent intermediate code w ∈ W. The neural mapping network 110 can be configured to implement the nonlinear mapping network. In one embodiment, the dimensions of the input and output activations in the latent input space Z and the latent space W are the same (e.g. 512 ). In one embodiment, the mapping function f is implemented with an 8-layer MLP (multilayer perceptron, ie a neural network that only consists of fully linked layers).

While the conventional generator only guides the latent code through the input layer of the generator, the neural mapping network forms 110 the input latent code z onto the latent space W to generate the intermediate latent code w. The style conversion unit 115 converts the intermediate latent code w into a first style signal. One or more intermediate latent codes w are converted into spatially immutable styles including the first style signal and a second style signal. In contrast to conventional style transfer methods, the spatially invariable styles are calculated from a vector, namely the latent intermediate code w, and not from an example image. The one or more intermediate latent codes w may be from one or more neural mapping networks 110 for one or more corresponding latent codes z are generated. The neural synthesis network 140 processes the first intermediate data (e.g. a learned constant which is coded as a feature map) in accordance with the style signals, with e.g. B. increases the density of the first intermediate data from 4x4 to 8x8 and continues until the density of the output data is reached.

In one embodiment, the style conversion unit performs 115 perform an affine transformation. The style conversion unit 115 can be trained to perform an affine transformation during training of the neural synthesis network 140 learns. The first style signal controls operations in a first layer 120 of the neural synthesis network 140 to generate modified first intermediate data. In one embodiment, the first style signal controls an adaptive instance normalization (AdalN) operation in the first layer 120 of the synthesis network 140 . In one embodiment, the AdaIN operation receives a set of content feature maps and a style signal and modifies the first order statistics (ie, "style") of the content feature maps to match the first order statistics defined by the style signal. The modified first intermediate data obtained from the first layer 120 are issued by the processing layer (s) 125 processed to produce second intermediate data. In one embodiment, the processing layer (s) 125 a 3x3 convolutional layer. In one embodiment, the processing layer (s) 125 has a 3x3 convolutional layer followed by an AdaIN operation that receives an additional style signal that is in 1A is not explicitly shown.

The second intermediate data is in a second layer 130 of the neural synthesis network 140 entered. The second style signal controls operations in the second layer 130 to generate modified second intermediate data. In one embodiment, the first style signal modifies a first attribute encoded in the first intermediate data and the second style signal modifies a second attribute encoded in the first intermediate data and the second intermediate data. For example, the first intermediate data are rough data compared to the second intermediate data, and the first style becomes rough feature maps in the first layer 120 transferred while the second style to feature maps with higher density in the second layer 130 is transmitted.

In one embodiment, the second layer takes 130 an upsampling of the second intermediate data and has a 3x3 convolutional layer followed by an AdalN operation. In one embodiment, the second style signal controls an AdaIN operation within the second layer 130 of the synthesis network 140 . The one from the second shift 130 The modified second intermediate data output is processed by the processing layer (s) 135 processed to generate output data having a content corresponding to the second intermediate data. In one embodiment, multiple (e.g., 32, 48, 64, 96, etc.) channels of features in the modified second intermediate data are converted to the output data encoded as color channels (e.g., red, green, blue).

In one embodiment, the processing layer (s) 135 a 3x3 convolutional layer. In one embodiment, the output data is an image that has first attributes corresponding to a first scale or a first scale and second attributes corresponding to a second scale or a second scale, the first scale being coarser compared to the second scale. The first scale can correspond to a scale of the feature maps used by the first layer 120 can be processed, and the second scale can correspond to a scale of the feature maps used by the second layer 130 are processed. Correspondingly, the first style signal modifies the first attributes according to the first scale and the second style signal modifies the second attributes according to the second scale.

Further illustrative information is now set forth in relation to various optional architectures and functions that can be used to implement the framework described above as desired by the user. It should be noted that the following information is only It is intended to be illustrative and in no way limiting. Each of the following features may optionally be present with or without excluding the other features described.

1B shows pictures taken by the style-based generator system 100 can be generated according to one embodiment. The images are generated in a resolution of 1024 ^2. In other designs, the images can be generated with a different resolution. Two different latent codes are used to denote the styles of the style-based generator system 100 to control generated images. In particular, a first part of the styles is created by the neural mapping network 110 and a style conversion unit 115 generated from the latent "source" codes in the top row. A second part of the styles is created by the same or an additional neural mapping network 110 and a corresponding style conversion unit 115 generated from the latent "target" codes in the leftmost column. The style-based generator system 100 assumes a learned constant input to the neural synthesis network 140 and adapts the “style” of the image for each convolution layer depending on the latent code, whereby the strength of image attributes, which are coded in feature maps, is directly controlled in different scales. In other words, a given set of styles from the “source” data is copied to the “destination” data. The copied styles (coarse, medium or fine) are taken from the “source” data, while all other styles are retained exactly as in the “target” data.

The first part of the styles (target) is from the neural synthesis network 140 used to process the learned constant, a first subset of the first part of the styles being replaced by a corresponding second subset of the second part of the styles (source). In one embodiment, the learned constant is a 4x4x512 constant tensor. In the second, third and fourth row of pictures in 1B the second part of the styles (source) replaces the first part of the styles (target) in the coarse layers of the neural synthesis network 140 . In one embodiment, the coarse layers correspond to the coarse spatial densities 4 ² - 8 ² . In one embodiment, high level attributes such as pose, general hairstyle, face shape and glasses are copied from the source while other attributes such as all colors (eyes, hair, lighting) and finer facial features of the target are retained.

In the fifth and sixth row of pictures in 1B the second part of the styles (source) replaces the first part of the styles (target) in the middle layers of the neural synthesis network 140 . In one embodiment, the middle layers correspond to the spatial densities of 16 ² - 32 ² . Minor facial features, hairstyle, open / closed eyes are taken from the source while the pose, general face shape and glasses are retained from the target. In the last row of pictures in 1B finally, the second part of the styles (source) replaces the first part of the styles (target) in the high-density (fine) layers of the neural synthesis network 140 . In one embodiment, the fine layers correspond to the spatial densities of 64 ² - 1024 ² . Using the styles from the second part of the styles (source) for the fine layers inherits the color scheme and microstructure from the source while maintaining the pose and general face shape from the target.

The architecture of the style-based generator system 100 enables the control of the image synthesis via scale-specific modifications of the styles. The mapping network 110 and the affine transformations performed by the style conversion unit 115 can be viewed as a way of extracting patterns for each style from a learned distribution, and the synthesis network 140 provides a mechanism to create a novel image based on a collection of styles. The effects of each style are in the synthesis network 140 localized, that is, the modification of a certain subset of the styles is expected to affect only certain attributes of the image.

The use of style signals from at least two different latent codes, as in 1B shown is called style blending or blend regularization. Mixing styles during training decorrelates neighboring styles and allows for finer-grain control over the images generated. In one embodiment, a percentage of the images are generated with two random latent codes instead of one during training. When such an image is generated, a random location (e.g. a crossover point) in the neural synthesis network 140 at which the processing switches from the use of style signals generated with a first latent code to style signals generated with a second latent code. In one embodiment, two latent codes z ₁ , z _{2 are} generated by the neural mapping network 110 processed, and the respective intermediate latent codes w ₁ , w ₂ control the styles so that w _{1 is applied} before the crossover point and w ₂ after the crossover point. The method of mixture regularization prevents the neural synthesis network 140 assumes that neighboring styles are correlated.

TABLE 1 shows how enabling mix regularization during training can greatly improve style localization, as indicated by improved (lower is better) Frechet Inception Distances (FIDs) in scenarios where multiple latent codes are at test time be mixed. In the 1B The images shown are examples of images synthesized by mixing two latent codes at different scales. Each subset of styles controls meaningful high-level attributes of the image. Table 1 FIDs for various mix regularization ratios Mixing ratio (training time) Number of latent codes (test time) 1 2 3 4th 0% 4.42 8.22 12.88 17.41 50% 4.41 6.10 8.71 11.61 90% 4.40 5.11 6.88 9.03 100% 4.83 5.17 6.63 8.40

The mix ratio indicates the percentage of training samples for which mix regularization is enabled. During the test period, a maximum of four different latent codes were selected at random and the crossover points between the different latent codes were also selected at random. The mixture regularization significantly improves the tolerance towards these opposing operations.

As the FIDs confirm, the average quality is that of the style-based generator system 100 generated images, and even accessories such as glasses and hats are successfully synthesized. For the in 1B Sampling from the extreme areas of W is avoided by the so-called cut-off trick, which can be performed in W instead of Z. It should be noted that the style-based generator system 100 can be implemented so that clipping can only be selectively applied at low resolutions so that high resolution details are not affected.

Im Fall eines Datensatzes menschlicher Gesichter (z. B. FFHQ, Flickr-Faces-HQ) repräsentiert der Punkt eine Art Durchschnittsgesicht (ψ=0). Die Abweichung eines gegebenen w wird vom Zentrum her als w' = w̅ + ψ(w - w̅) skaliert, wobei ψ <1 ist. In konventionellen erzeugenden Modellierungssystemen ist nur eine Teilmenge der neuronalen Netze für eine solche Abschneidung zugänglich, selbst wenn eine orthogonale Regularisierung verwendet wird, scheint die Abschneidung im W-Raum auch ohne Änderungen an der Verlustfunktion zuverlässig zu funktionieren.If one looks at the distribution of the training data, areas with a low density are poorly represented and therefore for the style-based generator system 100 probably difficult to learn. Uneven distributions of training data are a significant open problem in all generating modeling techniques. However, it is known that pulling latent vectors from a truncated or otherwise reduced scan space tends to improve average image quality, although some amount of variation is lost. In one embodiment, to improve the training of the style-based generator system 100 a center of mass of W calculated as $\overline{w}={\mathbb{E.}}_{z~\mathrm{P.}\left(z\right)}\left[f\left(z\right)\right].$

In the case of a data set of human faces (e.g. FFHQ, Flickr-Faces-HQ), the point represents a kind of average face (ψ = 0). The deviation of a given w is scaled from the center as w '= w̅ + ψ (w - w̅), where ψ <1. In conventional generating modeling systems, only a subset of the neural networks is accessible for such a clipping, even if an orthogonal regularization is used, the clipping in W-space seems to work reliably even without changes to the loss function.

1C shows a flow diagram of a method 150 for style-based generation, according to one embodiment. The procedure 150 can be executed by a program, a custom circuit, or a combination of a custom circuit and a program. For example, the method 150 executed by a GPU (graphics processing unit), CPU (central processing unit), or any processor capable of performing the operations of the style-based generator system 100 to execute. In addition, those skilled in the art understand that any system that incorporates the method 150 is within the scope and spirit of embodiments of the present invention.

In step 155 processes the neural mapping network 110 a latent code defined in an input space to generate an intermediate latent code defined in a latent space. In step 160 becomes the intermediate latent code by the style conversion unit 115 converted into a first style signal. In step 165 becomes the first style signal on a first layer 120 of the neural synthesis network 140 applied to the first intermediate data corresponding to the first style signal modify to produce modified first intermediate data. In step 170 the modified first intermediate data are processed by the processing layer (s) 125 processed to generate the second intermediate data. In step 175 will add a second style signal to the second layer 130 of the neural synthesis network 140 applied to modify the second intermediate data in accordance with the second style signal to generate modified second intermediate data. In step 180 the modified second intermediate data are processed by the processing layer (s) 135 processed to generate output data having a content corresponding to the second intermediate data.

There are different definitions of unbundling, but one common goal is a latent space made up of linear subspaces, each of which controls a factor of variation. However, the sampling probability of each combination of factors in the latent space Z must match the corresponding density in the training data.

A big advantage of the style-based generator system 100 is that the latent space W need not support a fixed distribution scan; the sampling density for the style-based generator system 100 is caused by the learned piece-wise continuous mapping f (z). The mapping can be adjusted to "equalize" W so that the factors of variation become more linear. The style-based generator system 100 can of course tend to equalize W, since it should be easier to generate realistic images based on an unwoven representation than on the basis of an interwoven representation. As such, training can give a less intertwined W in an unsupervised environment, that is, when the factors of variation are not known in advance.

2A shows a block diagram of the in 1A shown neural mapping network 110 according to one embodiment. A combination of attributes may be missing in a distribution of the training data, e.g. B. Children wearing glasses. A distribution of the variation factors in the combination of glasses and age becomes more linear in the latent space W compared to the latent space Z.

In one embodiment, the neural mapping network 110 a normalization layer 205 and several fully connected or fully connected layers 210 on. In one embodiment, there are eight fully-connected layers 210 connected in series to generate the latent intermediate code. The parameters (e.g. weights) of the neural mapping network 110 are learned during training and the parameters are used to process the latent input codes when the style-based generator system 100 is used to generate the output data. In one embodiment, the neural mapping network generates 110 one or more intermediate latent codes generated by the neural synthesis network 140 can be used at a later point in time to generate the output data.

There are many attributes in human portraits that can be considered stochastic, such as: B. the exact placement of hair, stubble, freckles or skin pores. Any of these features can be randomized without affecting the perception of the image, as long as the randomization follows the correct distribution. The artificial omission of noise in the image generation leads to images with a characteristic "painterly" appearance. Particularly when creating human portraits, coarse noise can cause the hair to curl over a large area and the background features appear larger, while the fine noise emphasizes the finer curls of the hair, finer background details and skin pores.

A conventional generator can only produce a stochastic variation based on the input to the neural network as provided by the input layer. During training, the conventional generator may be forced to learn to generate spatially varying pseudorandom numbers from previous activations whenever the pseudorandom numbers are needed. In other words, the generation of pseudorandom numbers has not been intentionally built into the conventional generator. Instead, the generation of pseudo-random numbers occurs automatically during training so that the conventional generator can achieve the training goal. The generation of the pseudo-random numbers consumes the capacity of the neural network and the hiding of the periodicity of the generated signal is difficult - and not always successful, as the frequently seen repeating patterns in the generated images show. In contrast, the style-based generator system 100 designed to avoid these limitations by adding noise per pixel after each convolution.

In one embodiment, the style-based generator system is 100 designed with a direct means of generating stochastic details by introducing explicit noise inputs. In one embodiment, the noise inputs are single-channel images made up of uncorrelated Gaussian noise, and a dedicated noise image is placed in one or more layers of the synthesis network 140 entered. With the aid of learned scaling factors per feature, the noise image can be transferred to all feature maps and then added to the output of the corresponding convolution.

2 B shows a block diagram of the in 1A shown neural synthesis network 140 according to one embodiment. The neural synthesis network 140 has a first processing block 200 and a second processing block 230 on. In one embodiment, the processing block processes 200 Feature maps with a resolution of 4x4 and the processing block 230 Feature maps with a resolution of 8x8. One or more additional processing blocks can follow the processing blocks 200 and 230 , in front of them and / or between them in the neural synthesis network 140 to be available.

The first processing block 200 receives the first intermediate data, the first spatial noise, and the second spatial noise. In one embodiment, the first spatial noise is scaled with a learned scaling factor per channel before it is combined with (e.g. added to) the first intermediate data. In one embodiment, the first spatial noise, the second spatial noise, the third spatial noise, and the fourth spatial noise is independent Gaussian noise per pixel.

The first processing block 200 also receives the first style signal and the second style signal. As already explained, the style signals can be obtained by processing the intermediate latent code in accordance with a learned affine transformation. Learned affine transformations specialize w in styles y = (y _s , y _b ) that control the adaptive instance normalization (AdaIN) operations performed by the modules 220 in the neural synthesis network 140 implemented. Compared to more general feature transformations, AdalN is particularly good for implementation in the style-based generator system due to its efficiency and compact representation 100 suitable.

wobei jede Merkmalskarte x_i separat normalisiert und dann mit den entsprechenden Skalarkomponenten aus dem Stil y skaliert und verzerrt wird. Somit ist die Dimensionalität von y doppelt so groß wie die Anzahl der Merkmalskarten im Vergleich zu der Eingabe der Schicht. In einer Ausführungsform ist die Dimension des Stilsignals ein Vielfaches einer Anzahl der Merkmalskarten in der Schicht, an der das Stilsignal anliegt. Im Gegensatz zur konventionellen Stilübertragung wird der räumlich invariante Stil y aus dem Vektor w und nicht aus einem Beispielbild berechnet.The AdaIN operation is defined as $$\text{AdaIN}\left({\text{x}}_i,\text{y}\right)={\text{y}}_{s,i}\frac{{\text{x}}_i-\mu \left({\text{x}}_i\right)}{\sigma \left({\text{x}}_i\right)}+{\text{y}}_{b,i}$$

where each feature _{map x i is} normalized separately and then scaled and distorted with the corresponding scalar components from style y. Thus, the dimensionality of y is twice as large as the number of feature maps compared to the input of the layer. In one embodiment, the dimension of the style signal is a multiple of a number of the feature maps in the layer to which the style signal is applied. In contrast to the conventional style transfer, the spatially invariant style y is calculated from the vector w and not from an example image.

The effects of each style signal are in the neural synthesis network 140 localized, ie it is to be expected that a modification of a certain subset of the style signals affects only certain attributes of an image represented by the output data. The reason for the localization is that the AdalN operation (Eq. 1) carried out by the module 220 is implemented, first normalizes each channel to a mean value of zero and a unit variance and only then applies scales and distortions depending on the style signal. The new per-channel statistics given by the style change the relative importance of features for the subsequent convolution operation, but the new per-channel statistics do not depend on the original statistics due to normalization. Thus, each style signal controls only a predefined number of folds 225 before it is overwritten by the next AdalN operation. In one embodiment, after each convolution and before processing by a further module 220 a scaled spatial noise added to the features.

On every module 220 can be a convolutional layer 225 follow. In one embodiment, the convolutional layer applies 225 apply a 3 x 3 convolution kernel to the input. Within the processing block 200 will be the second intermediate data received from the convolutional layer 225 are output, combined with the second spatial noise and placed in a second module 220 which applies the second style signal to an output of the processing block 200 to create. In one embodiment, the second spatial noise is scaled with a learned scaling factor per channel before it is combined with (e.g., added to) the second intermediate data.

The processing block 230 receives feature maps from the processing block 200 are output, and the feature maps are passed through an upsampling layer 235 increased with respect to their sampling rate or subjected to an upsampling. In one embodiment, 4x4 feature maps are upsampled by the layer 235 increased in sampling rate to generate 8x8 feature maps. The feature maps, which have been increased in terms of their sampling rate, are placed in a further convolution layer 225 is input to generate third intermediate data. Within the processing block 230 the third intermediate data is combined with the third spatial noise and converted into a third module 220 which applies the third style signal via an AdaIN operation. In one embodiment, the third spatial noise is scaled with a learned scaling factor per channel before it is combined with (e.g., added to) the third intermediate data. The output of the third module 220 is made up of another folding layer 225 processed to produce fourth intermediate data. The fourth intermediate data is combined with the fourth spatial noise and into a fourth module 220 which applies the fourth style signal via an AdaIN operation. In one embodiment, the fourth spatial noise is scaled with a learned scaling factor per channel before it is combined with (e.g., added to) the fourth intermediate data.

In one embodiment, the resolution of the output data is 1024 ² and the neural synthesis network 140 has 18 layers - two for each power of two of the resolution (4 ² - 1024 ² ). The output of the last layer of the neural synthesis network 140 can be converted to RGB with a separate I x I convolution. In one embodiment, the neural synthesis network 140 a total of 26.2 M trainable parameters, compared to 23.1 M for a conventional generator with the same number of layers and feature maps.

The introduction of spatial noise only affects the stochastic aspects of the output data, so the overall composition and high-level attributes such as identity remain intact. Separate noise inputs for the neural synthesis network 140 allow a stochastic variation to be applied to different subsets of layers. The application of spatial noise generated by a specific layer of the neural synthesis network 140 is entered leads to a stochastic variation on a scale that corresponds to the scale of the respective layer.

The effect of the noise appears in the neural synthesis network 140 closely localized. At any point in the neural synthesis 140 there is a need to introduce new content as quickly as possible and the easiest way for the neural synthesis network 140 To create a stochastic variation is to rely on the spatial noise inputs. For each layer in the neural synthesis network 140 a fresh set of spatial noise is available so there is no incentive to create the stochastic effects from previous activations, resulting in a localized effect. Therefore, the noise only affects insignificant stochastic variations (differently combed hair, beard, etc.). In contrast, changes to the style signals have global effects (change in pose, identity, etc.).

If the output data is the neural synthesis network 140 If it is an image, the style signals affect the entire image, since complete feature maps are scaled and distorted with the same values. Therefore, global effects such as pose, lighting or background style can be controlled coherently. Meanwhile, the spatial noise is added to each pixel independently and is therefore ideal for controlling a stochastic variation. Would the neural synthesis network 140 try z. B. to control the pose with the help of the noise, this would lead to spatially inconsistent decisions that would be punished during training. Therefore, the neural synthesis network learns 140 to use the global and local channels appropriately without explicit guidance.

2C shows a flow diagram of a method 250 on the application of spatial noise using the style-based generator system 100 according to one embodiment. The procedure 250 can be executed by a program, a custom circuit, or a combination of a custom circuit and a program. For example, the method 250 executed by a GPU (graphics processing unit), CPU (central processing unit), or any processor capable of performing the operations of the style-based generator system 100 to execute. In addition, those skilled in the art understand that any system that incorporates the method 250 is within the scope and spirit of embodiments of the present invention.

In step 255 becomes a first set of spatial noise on a first layer of the neural synthesis network 140 applied to generate the first intermediate data comprising content which corresponds to the original data modified based on the first set of spatial noise. In one embodiment, the original data is the first intermediate data and the first layer is a layer that comprises the module 220 and / or the folding layer 225 includes. In step 258 the modified first intermediate data are processed by the processing layer (s) 125 processed to generate the second intermediate data. In step 260 becomes a second set of spatial noise on a second layer of the neural synthesis network 140 is applied to generate second intermediate data including content corresponding to the first intermediate data modified based on the second set of spatial noise. In one embodiment, the first intermediate data are at least provided by the module 220 modified to generate the second intermediate data. In step 265 the second intermediate data is processed to generate output data having contents corresponding to the second intermediate data. In one embodiment, the second intermediate data is from another module 220 and the block 230 processed to produce the output data.

Noise can enter the layers of the neural synthesis network 140 be fed in to bring about a synthesis of stochastic variations on a scale corresponding to the layer. It is important that the noise is fed in both during training and during generation or generation. In addition, the strength of the noise can be changed during generation in order to further control the "appearance" of the output data. Providing style signals instead of entering the latent code directly into the neural synthesis network 140 in combination with directly in the neural synthesis network 140 Noise fed in leads to an automatic, unsupervised separation of high-level attributes (e.g. pose, identity) from a stochastic variation (e.g. freckles, hair) in the generated images and enables intuitive, scale-specific mixing and interpolation operations.

In particular, the style signals directly match the strength of the image attributes at different scales in the neural synthesis network 140 at. During generation, the style signals can be used to modify selected image attributes. In addition, the neural mapping network 110 be designed during training in such a way that it carries out a mix of styles regularization in order to improve a localization of the styles.

The neural mapping network 110 embeds the entered latent code in the latent space, which has a great influence on how the factors of variation in the neural synthesis network 140 being represented. The latent space entered follows the probability density of the training data, and this is likely to result in some degree of inevitable entanglement. The latent space is exempt from this restriction and can therefore be disentangled. Compared to a conventional generator architecture, the style-based generator system allows 100 a more linear, less intertwined representation of several factors of variation. In one embodiment, replacing a conventional generator with the style-based generator may not require any other component of the training framework to be changed (loss function, discriminator, optimization method, or the like).

The style-based generating neural network 100 can e.g. B. can be trained with the GAN (generative adversarial networks), VAE (variational autoencoder) framework, flow-based framework or similar. 2D shows a block diagram of the training framework GAN 270 according to one embodiment. The GAN 270 can be implemented by a program, a custom circuit, or a combination of a custom circuit and a program. For example, the GAN 270 be implemented with a GPU, CPU, or any processor capable of performing the operations described herein. In addition, it will be understood by those skilled in the art that any system capable of performing the operations of the GAN 270 performs falls within the scope and spirit of the embodiments of the present invention.

The GAN 270 assign a generator, like the style-based generator system 100 , a discriminator (neural network) 275 and a loss of training session 280 on. The topologies of both the generator 110 as well as the discriminator 275 can be modified during training. The GAN 270 can work in an unsupervised setting or in a dependent setting. The style-based generator system 100 receives input data (e.g. at least one latent code and / or noise inputs) and generates output data. Depending on the task, the output data can be an image, audio, video or other types of data (configuration setting). The discriminator 275 is an adaptive loss function used when training the style-based generator system 100 is used. The style-based generator system 100 and the discriminator 275 are trained simultaneously with a training data set that has sample output data that is used by the style-based generator system 100 generated output data match should. The style-based generator system 100 generates output data depending on the input data, and the discriminator 275 determines whether the output data is similar to the sample output data included in the training data. Based on the determination, the parameters of the discriminator 275 and / or the style-based generating neural network 100 customized.

In the unsupervised setting, the discriminator gives 275 outputs a continuous value that indicates how closely the output data matches the sample output data. In one embodiment the discriminator gives 275 For example, a first training stimulus (e.g. a high value) if it is determined that the output data matches the sample output data, and a second training stimulus (e.g. a low value) if it is determined that the Output data does not match the sample output data. The training loss session 280 fits the parameters (weights) of the GAN 270 based on the output of the discriminator 275 at. If the style-based generator system 100 is trained for a specific task, e.g. B. the generation of images of human faces, the discriminator outputs a high value if the output data is an image of a human face. The ones from the style-based generator system 100 generated output data do not have to be identical to the sample output data so that the discriminator 275 determines the consistency of the output data with the sample output data. In the context of the following description, the discriminator determines 275 indicates that the output data will match the example output data if the output data is similar to any of the example output data.

In the dependent setting, the input of the style-based generating neural network 100 have other data, such as B. an image, a classification label, segmentation contours and other (additional) types of data (distribution, audio, etc.). The additional data can be provided in addition to the random latent code, or the additional data can replace the random latent code entirely. The training data set can have input / output data pairs and the task of the discriminator 275 can be based on the exemplary input / output pairs that the discriminator 275 saw in the training data to determine if the output of the style-based generating neural network 100 appears consistent with the entry.

Parallel processing architecture

3 illustrates a parallel processing unit (PPU) 300 according to one embodiment. The PPU 400 can be used to build the style based generator system 100 to implement. The PPU 400 can be used to implement one or more of the following: the neural mapping network 110 , the style conversion unit 115 , the neural synthesis network 140 , the generating opposing networks 270 , the style block 640 , the style block 645 and the neural synthesis network 840 . In one embodiment, a processor such as the PPU 400 be designed so that it implements a model of a neural network. The neural network model can be implemented as software instructions that are executed by the processor, or in other implementations, the processor can include an array of hardware elements configured to receive a set of inputs (e.g., electrical signals that Represent values) to produce a set of outputs that can represent activations of the neural network model. In other embodiments, the neural network model may be implemented as a combination of software instructions and processing by an array of hardware elements. The implementation of the neural network model may include determining a set of parameters for the neural network model, e.g. By supervised or unsupervised training of the neural network model and, or alternatively, performing inference using the set of parameters to process new sets of inputs.

In one embodiment, the PPU is 300 a multi-thread processor implemented on one or more integrated circuit devices. The PPU 300 is a latency-hiding architecture designed to process many threads in parallel. A thread (ie a thread of execution) is an instantiation of a set of instructions that are to be executed by the PPU 300 are designed. In one embodiment, the PPU is 300 a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) graphics data to produce two-dimensional (2D) image data for display on a display device, such as B. a liquid crystal display (LCD) to generate. In another embodiment, the PPU is 300 designed to the neural network system 100 to implement. For other versions, the PPU 300 used for performing general purpose calculations. While an exemplary parallel processor is provided herein for purposes of illustration, it is expressly understood that such a processor is set forth for illustrative purposes only and that any processor may be used to supplement and / or replace it.

One or more PPUs 300 can be designed to accelerate thousands of high performance computing (HPC), data center, and machine learning applications. The PPU 300 can be designed to accelerate numerous deep learning systems and applications, including autonomous vehicle platforms, deep learning, high-precision voice, image and text recognition systems, intelligent video analysis, molecular simulations, drug discovery, disease diagnosis, weather forecasting, big data Analytics, astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time voice translation, online search optimization and personalized user recommendations and the like.

As in 3 shown, the PPU 300 an input / output (I / O) unit 305 , a front-end unit 315 , a sequence control unit or scheduler unit 320 , a work distribution unit 325 , a network node 330 , a switching matrix (Xbar) 370 , one or more general processing clusters or general processing clusters (GPCs) 350 and one or more partition units 380 on. The PPU 300 can be with a host processor or other PPUs 300 connected via one or more high-speed NVLink 310 connections. The PPU 300 can interconnect with a host processor or other peripheral equipment 302 be connected. The PPU 300 can also use a local store 304 be connected, which comprises a number of storage devices. In one embodiment, the local memory may comprise a number of DRAM (Dynamic Random Access Memory) devices. The DRAM devices can be designed as an HBM (high-bandwidth memory) subsystem, with several DRAM dies being stacked in each device.

The NVLink 310 connection enables systems to scale and assigns one or more PPUs 300 in combination with one or more CPUs, supports cache coherence between the PPUs 300 and the CPUs as well as the CPU mastering. Data and / or commands can be sent via the NVLink 310 through the network node 330 to / from other units of the PPU 300 transmitted, such as B. one or more copy engines, a video encoder, a video decoder, a power management unit, etc. (not explicitly shown). The NVLink 310 will be used in conjunction with 5B described in more detail.

The I / O base 305 is designed to receive communications (ie commands, data, etc.) from a host processor (not shown) over the link 302 sends and receives. The I / O base 305 can connect to the host processor directly 302 or via one or more intermediary devices such as B. communicate a memory bridge. In one embodiment, the I / O device 305 with one or more other processors, such as B. one or more PPUs 300 , through the interconnection 302 communicate. In one embodiment, implements the I / O device 305 a Peripheral Component Interconnect Express (PCIe) interface for communication over a PCIe bus, and the interconnect 302 is a PCIe bus. In other versions, the I / O base can 305 implement other types of known interfaces for communication with external devices.

The I / O base 305 decodes packets passing through the connection 302 be received. In one embodiment, the packets represent commands that are designed to be used by the PPU 300 cause various operations to be performed. The I / O base 305 transmits the decoded commands to various other units of the PPU 300 as the commands can dictate. For example, some commands can be sent to the front-end unit 315 be transmitted. Other commands can be sent to the network node 330 or other units of the PPU 300 transmitted, such as B. one or more copy engines, a video encoder, a video decoder, a power management unit, etc. (not explicitly shown). In other words, the I / O base 305 is designed in such a way that it enables communication between and among the various logical units of the PPU 300 routes.

In one embodiment, a program executed by the host processor encodes an instruction stream in a buffer called the PPU 300 Provision workloads for processing. A workload can contain multiple statements and data to be processed by those statements. The buffer is an area in memory shared by both the host processor and the PPU 300 be able to access (ie read and write). For example, the I / O base can 305 be configured to access the buffer in a system memory associated with the connection 302 is connected through storage requests made through the connection 302 be transmitted. In one embodiment, the host processor writes the instruction stream to the buffer and then transfers a pointer to the beginning of the instruction stream to the PPU 300 . The front-end unit 315 receives pointers to one or more instruction streams. The front end unit 315 manages the one or more streams, reads commands from the streams and forwards commands to the various units of the PPU 300 Further.

The front-end unit 315 is with a scheduler unit 320 coupled that the various GPCs 350 designed to process the tasks defined by the one or more streams. The scheduler unit 320 is designed to provide status information related to the various, from the scheduler unit 320 managed tasks tracked. The state can indicate which GPC 350 a task is assigned, whether the task is active or inactive, a priority level associated with the task, and so on. The scheduler unit 320 manages the execution of a large number of tasks on the one or more GPCs 350 .

The scheduler unit 320 is with a labor distribution unit 325 coupled, which is designed so that it can execute tasks on the GPCs 350 distributed. The work distribution unit 325 can keep track of a number of scheduled tasks performed by the scheduler unit 320 were received. In one embodiment, the work distribution unit manages 325 a pool of outstanding tasks and a pool of active tasks for each of the GPCs 350 . The pending task pool may include a number of slots (e.g. 32 slots) that contain tasks that are to be processed by a particular GPC 350 are assigned. The pool of active tasks may include a number of slots (e.g. 4 slots) for tasks that are active by the GPCs 350 to be edited. When a GPC 350 When a task finishes, that task is removed from the pool of active tasks for the GPC 350 removed and one of the other tasks from the pool of pending tasks is selected and scheduled to run on the GPC 350 planned. If there is an active task on the GPC 350 is idle, e.g. B. while waiting for the resolution of a data dependency, then the active task can be taken from the GPC 350 removed and returned to the pending task pool, while another task in the pending task pool is selected and executed on the GPC 350 is scheduled.

The work distribution unit 325 communicates with the one or more GPCs 350 via the switching matrix 370 . The coupling matrix 370 is an interconnection network that connects many of the units of the PPU 300 with other units of the PPU 300 couples. For example, the switching matrix 370 be designed so that it is the work distribution unit 325 with a specific GPC 350 couples. Although not explicitly shown, one or more other units of the PPU 300 via the network node 330 with the switching matrix 370 be connected.

The tasks are carried out by the scheduler unit 320 managed and by the labor distribution unit 325 to a GPC 350 forwarded. The GPC 350 is designed in such a way that it processes the task and generates results. The results can be obtained from other tasks within the GPC 350 recorded, via the switching matrix 370 to another GPC 350 forwarded or in memory 304 be filed. The results can be obtained through the partition units 380 into memory 304 which is a memory interface for reading and writing data to / from memory 304 to implement. The results can be viewed via the NVLink 310 to another PPU 300 or CPU. In one embodiment, the PPU 300 a number U of partition units 380 based on the number of separate and distinct storage devices used by the PPU 300 coupled storage 304 is equivalent to. A partition unit 380 will be discussed below in conjunction with 4B described in more detail.

In one embodiment, a host processor executes a driver core that implements an application programming interface (API) that allows one or more applications running on the host processor to operate on the PPU 300 to plan. In one embodiment, multiple computing applications are run concurrently by the PPU 300 executed and the PPU 300 provides isolation, quality of service (QoS), and independent address spaces for the multiple computing applications. An application can generate instructions (ie, API calls) that cause the driver core to perform one or more tasks for the PPU to perform 300 to generate. The driver kernel issues tasks to one or more streams that the PPU 300 are processed. Each task can include one or more groups of contiguous threads, referred to herein as a warp. In one embodiment, a warp includes 32 related threads that can be executed in parallel. Cooperating threads can refer to a variety of threads that have instructions for performing the task and can exchange data over a shared memory. Threads and cooperating threads are associated with 5A described in more detail.

4A shows a GPC 350 the PPU 300 the end 3 according to one embodiment. As in 4A each has GPC 350 a number of hardware units for processing tasks. In one embodiment, each has GPC 350 a pipeline manager 410 , a Pre-Raster Operations Unit (PROP) 415 , a raster engine 425 , a work distribution crossbar (WDX) 480 , a memory management unit (MMU) 490 and one or more data processing clusters or data processing clusters (DPCs) 420 on. It is clear that the GPC 350 the end 4A instead of the in 4A units shown or in addition to these may have other hardware units.

In one embodiment, the operation of the GPC 350 through the pipeline manager 410 controlled. The pipeline manager 410 manages the configuration of one or more DPCs 420 for processing tasks assigned to the GPC 350 are assigned. In one embodiment, the pipeline manager 410 at least one of the one or more DPCs 420 configure to implement at least a portion of a graphics rendering pipeline. For example, a DPC 420 be designed so that it can run a vertex shader program on the programmable streaming multiprocessor (SM) 440 executes. The pipeline manager 410 can also be designed so that it is from the work distribution unit 325 received packets to the appropriate logical units within the GPC 350 forwards. For example, some packages of hardware units with a fixed function in the PROP 415 and / or in the raster engine 425 while other packets are forwarded to the DPCs 420 for processing by the primitive engine 435 or the SM 440 can be forwarded. In one embodiment, the pipeline manager 410 may include at least one of the one or more DPCs 420 configure it to implement a neural network model and / or a computational pipeline.

The PROP unit 415 is designed to be used by the raster engine 425 and the DPCs 420 forwards generated data to a Raster Operations (ROP) unit, which in conjunction with 4B is described in more detail. The PROP unit 415 can also be designed in such a way that it carries out optimizations for the color mixing, organizes pixel data, carries out address translations and the like.

The raster engine 425 comprises a number of fixed function hardware units designed to perform various raster operations. In one embodiment, the raster engine 425 a setup engine, a coarse grid engine, a culling engine, a clipping engine, a fine grid engine and a tile coalescing engine. The setup engine receives transformed vertices and generates plane equations associated with the geometric primitive defined by the vertices. The plane equations are passed to the coarse raster engine to generate coverage information (e.g., an x, y coverage mask for a tile) for the primitive. The output of the coarse-raster engine is forwarded to the culling engine, where fragments connected to the primitive that fail a z-test are sorted out, and passed to a clipping engine, where fragments that are outside a cone of view to be cut off. The fragments that survive the clipping and sorting can be passed to the fine grid engine to generate attributes for the pixel fragments based on the plane equations generated by the setup engine. The output of the raster engine 425 includes fragments z. B. from one in a DPC 420 implemented fragment shaders are processed.

Everyone in the GPC 350 included DPC 420 assigns an M-Pipe control (MPC) 430 , a primitive engine 435 and one or more SMs 440 on. The MPC 430 controls the operation of the DPC 420 and directs those from the pipeline manager 410 received packets to the corresponding units in the DPC 420 Further. For example, packets that are assigned to a vertex can be sent to the primitive engine 435 are forwarded, which is designed so that they vertex attributes, which are assigned to the vertex, from the memory 304 retrieves. In contrast, packets that are connected to a shader program can be sent to the SM 440 be transmitted.

The SM 440 comprises a programmable streaming processor which is designed to process tasks represented by a number of threads. Every SM 440 is multi-thread capable and designed in such a way that it can execute a large number of threads (e.g. 32 threads) from a specific group of threads at the same time. In one embodiment, the SM 440 a single-instruction, multiple-data (SIMD) architecture in which each thread in a group of threads (ie, a warp) is designed to process a different set of data based on the same set of instructions. All threads in the group of threads execute the same instructions. In another embodiment, the SM 440 a SIMT (single instruction, multiple thread) architecture in which each thread in a group of threads is designed to process a different data set based on the same instruction set, but with individual threads in the group of threads during execution may diverge. In one embodiment, a program counter, a call stack, and an execution status are maintained for each warp, indicating simultaneity between warps and a Enables serial execution within warps when threads diverge within the warp. In another embodiment, a program counter, a call stack and an execution status are kept for each individual thread, which enables equal simultaneity between all threads within and between warps. By keeping the execution status for each individual thread, threads executing the same instructions can be merged and run in parallel for maximum efficiency. The SM 440 will be discussed below in conjunction with 5A described in more detail.

The MMU 490 forms an interface between the GPC 350 and the partition unit 380 . The MMU 490 can handle the translation of virtual addresses into physical addresses, memory protection and the cleanup of memory requests. In one embodiment, the MMU 490 one or more translation lookaside buffers (TLBs) ready to handle the translation of virtual addresses into physical addresses in memory 304 perform.

4B represents a storage partition unit 380 the PPU 300 the end 3 according to one embodiment. As in 4B is shown, has the memory partition unit 380 a raster operation unit (ROP) 450 , a level 2 cache (L2) 460 and a memory interface 470 on. The storage interface 470 is with the memory 304 coupled. The storage interface 470 can implement 32-, 64-, 128-, 1024-bit data buses or the like for high-speed data transfer. In one embodiment, the PPU has 300 via U memory interfaces 470 , a memory interface 470 per pair of partition units 380 , where each pair of partition units 380 with a corresponding setup of the memory 304 connected is. For example, the PPU 300 with up to Y storage devices 304 be connected, such as B. Memory stacks with high bandwidth or graphics memory with double data rate, version 5 , synchronous dynamic random access storage, or other types of persistent storage.

In one embodiment, implements the memory interface 470 an HBM2 memory interface and Y is half of U. In one embodiment, the HBM2 memory stack is in the same physical chassis or assembly as the PPU 300 which enables considerable energy and space savings compared to conventional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack has four memory chips and Y is equal to 4, the HBM2 stack having two 128-bit channels per chip for a total of 8 channels and a data bus width of 1024 bits.

In one embodiment, the memory supports 304 the Single-Error Correcting Double-Error Detecting (SECDED) Error Correction Code (ECC) to protect the data. ECC provides higher reliability for data processing applications that are sensitive to data corruption. Reliability is especially important in large cluster computing environments where the PPUs 300 process very large data sets and / or run applications for long periods of time.

In one embodiment, the PPU implements 300 a tiered storage hierarchy. In one embodiment, the memory partitioning unit supports 380 a unified memory to provide a single unified virtual address space for the CPU and PPU 300 memory, which enables the sharing of data between virtual memory systems. In one embodiment, the frequency of accesses by a PPU 300 on memory that is on other processors is tracked to ensure that pages are in the physical memory of the PPU 300 that accesses the pages more often. In one embodiment, the NVLink supports 310 Address translation services that make it the PPU 300 allow direct access to the side tables of a CPU and the PPU 300 allow full access to the CPU memory.

In one embodiment, copy engines transfer data between multiple PPUs 300 or between PPUs 300 and CPUs. The copy engines can generate page faults for addresses that are not mapped in the page tables. The storage partitioning unit 380 can then process the page faults and map the addresses in the page table, whereupon the copy engine can perform the transfer. In a conventional system, for multiple copy operations, memory is pinned between multiple processors (that is, it cannot be paged out), which significantly reduces available memory. Hardware page faulting allows addresses to be passed to the copy engines without worrying about whether the memory pages are resident, and the copying process is transparent.

Data from memory 304 or other system memory can be used by the memory partition unit 380 retrieved and in L2 cache 460 stored on the chip and by the various GPCs 350 is shared. As shown, each memory partition unit has 380 a portion of the L2 cache 460 on that of a corresponding memory 304 assigned. Caches the lower level can then be in different units within the GPCs 350 be implemented. For example, each of the SMs 440 implement a level one (L1) cache. The L1 cache is private storage that is used for a specific SM 440 is determined. Data from the L2 cache 460 can be retrieved and stored in each of the L1 caches for processing in the functional units of the SMs 440 get saved. The L2 cache 460 is with the memory interface 470 and the switching matrix 370 coupled.

The ROP unit 450 performs graphics raster operations related to pixel color, such as B. color compression, pixel blending and the like. The ROP unit 450 also implements an in-depth check in conjunction with the raster engine 425 by getting a depth for a sample location associated with a pixel fragment from the raster engine's culling engine 425 receives. The depth is tested against a corresponding depth in a depth buffer for a sample position associated with the fragment. If the fragment passes the scan position depth test, the ROP unit updates 450 the depth buffer and transmits a result of the depth test to the raster engine 425 . It becomes clear that the number of partition units 380 on the number of GPCs 350 may differ and therefore every ROP unit 450 with each of the GPCs 350 can be coupled. The ROP unit 450 tracked by the various GPCs 350 received packets and determines which GPC 350 one from the ROP unit 450 generated result via the switching matrix 370 is forwarded. Although the ROP unit 450 in 4B within the storage partition unit 380 is shown, it can also be outside of the memory partition unit in other versions 380 lie. For example, the ROP unit can be 450 in the GPC 350 or another unit.

5A illustrates the streaming multiprocessor 440 the end 4A according to one embodiment. As in 5A is shown, the SM 440 an instruction cache 505 , one or more scheduler units 510 , a register file 520 , one or more processing cores 550 , one or more special functional units (SFUs) 552 , one or more load / storage units (LSUs) 554 , an interconnection network 580 and a shared memory / L1 cache 570 on.

As described above, the work distribution unit distributes 325 Tasks to be executed on the GPCs 350 the PPU 300 . The tasks are assigned to a specific DPC 420 within a GPC 350 and, if the task is linked to a shader program, the task can be assigned to an SM 440 be assigned to. The scheduler unit 510 receives the tasks from the work distribution unit 325 and manages command scheduling for one or more thread blocks assigned to the SM 440 are assigned. The scheduler unit 510 plans thread blocks for execution as warps of parallel threads, with at least one warp assigned to each thread block. In one embodiment, each warp executes 32 threads. The scheduler unit 510 can manage a multitude of different thread blocks by assigning the warps to the different thread blocks and then during each clock cycle instructions from the multitude of different cooperative groups to the different functional units (i.e. cores 550 , SFUs 552 and LSUs 554 ) distributed.

Cooperative Groups is a programming model for organizing groups of communicating threads that allows developers to express the granularity with which threads communicate, enabling richer, more efficient parallel decompositions. The Cooperative Start APIs support synchronization between thread blocks for running parallel algorithms. Conventional programming models offer a single, simple construct for synchronizing cooperating threads: a lock over all threads of a thread block (i.e. the syncthreads () function). However, programmers often want to define groups of threads with a smaller granularity than thread blocks and synchronize them within the defined groups in order to enable higher performance, design flexibility and software reuse in the form of collective group-wide functional interfaces.

Cooperative groups allow programmers to explicitly define groups of threads on sub-block (i.e., as small as a single thread) and multi-block granularity, and to perform collective operations such as synchronization on the threads in a cooperative group. The programming model supports a clean composition across software boundaries so that libraries and utility functions can safely synchronize within their local context without having to make assumptions about convergence. The cooperative group primitives enable new patterns of cooperative parallelism, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across a network of thread blocks.

A dispatch unit 515 is designed to transmit instructions to one or more of the functional units. In the embodiment, the scheduler unit 510 two dispatch units 515 that allow two different instructions from the same warp to be processed in each clock cycle. In alternative embodiments, each scheduler unit 510 a single dispatch unit 515 or additional dispatch units 515 exhibit.

Every SM 440 assigns a register file 520 containing a set of registers for the functional units of the SM 440 provides. In one embodiment, the register file is 520 split between the individual functional units so that each functional unit has its own section of the register file 520 assigned. In another embodiment, the register file 520 split between the various warps used by the SM 440 are executed. The register file 520 serves as a buffer for operands that are connected to the data paths of the functional units.

Every SM 440 includes L processing cores 550 . In one embodiment, the SM 440 a large number (e.g. 128 etc.) of various processing cores 550 on. Every core 550 may include a single precision, double precision, and / or mixed precision processing unit that is fully sequential and includes a floating point arithmetic logic unit and an integer arithmetic logic unit. In one embodiment, the floating point arithmetic logic units implement the IEEE 754-2008 standard for floating point arithmetic. In one embodiment, the cores have 550 64 single-precision (32-bit) floating-point cores, 64 integer cores, 32 double-precision (64-bit) floating-point cores, and 8 tensor cores.

Tensor kernels are designed to perform matrix operations, and in one embodiment one or more tensor kernels are in the kernels 550 contain. In particular, the tensor cores are designed so that they perform deep learning matrix arithmetic, such as B. GEMM (Matrix-Matrix-Multiplication) for convolution operations during training and inference of neural networks. In one embodiment, each tensor kernel operates on a 4x4 matrix and performs a matrix multiply and accumulate operation D = AB + C, where A, B, C and D are 4x4 matrices.

In one embodiment, the matrix multiplication inputs A and B can be integer, fixed point, or floating point matrices, while the accumulation matrices C and D can be integer, fixed point, or floating point matrices of equal or greater bit width. In one embodiment, tensor cores operate on one-, four-, or eight-bit integer input data with 32-bit integer accumulation. The 8-bit integer matrix multiplication requires 1024 operations and gives a full precision product which is then accumulated with 32-bit integer addition with the other intermediate products for an 8x8x16 matrix multiplication. In one embodiment, the tensor cores operate on 16-bit floating point input data with 32-bit floating point accumulation. The 16-bit floating point multiplication requires 64 operations and gives a product with full precision, which is then accumulated using 32-bit floating point addition with the other intermediate products for a 4x4x4 matrix multiplication. In practice, tensor kernels are used to perform much larger two-dimensional or higher-dimensional matrix operations built from these smaller elements. An API such as B. the CUDA 9 C ++ API, provides specialized matrix load, matrix multiplication and accumulation and matrix memory operations to use tensor cores efficiently from a CUDA-C ++ program. At the CUDA level, the interface at the warp level assumes 16x16 matrices that span all 32 threads of the warp.

Every SM 440 also includes M SFUs 552 that perform special functions (e.g. attribute evaluation, reciprocal square root, and the like). In one embodiment, the SFUs 552 comprise a tree traversing unit which is configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 552 have a texture unit which is configured to perform filter operations for the texture mapping. In one embodiment, the texture units are designed such that they can get texture maps (e.g. a 2D array of texels) from the memory 304 load and scan the texture maps to retrieve scanned texture values for use in by the SM 440 to generate executed shader programs. In one embodiment, the texture maps are stored in shared memory / L1 cache 470. The texture units implement texture operations such as B. Filter operations using mip maps (ie texture maps with different levels of detail). In one embodiment, each SM 340 two texture units.

Every SM 440 also includes N LSUs 554 , the load and store operations between the shared memory / L1 cache 570 and the register file 520 carry out. Every SM 440 assigns an interconnection network 580 that each of the functional units with the register file 520 and the LSU 554 with the register file 520 and the shared memory / L1 cache 570 connects. In one embodiment, the interconnection network is 580 a switching matrix that can be configured to accommodate any of the Functional units with each of the registers in the register file 520 connects and the LSUs 554 with the register file and locations in the shared memory / L1 cache 570 connects.

The shared memory / L1 cache 570 is an array of on-chip memory that provides data storage and communication between the SM 440 and the primitive engine 435 as well as between threads in the SM 440 enables. In one embodiment, the shared memory includes / L1 cache 570 128 KB of storage space and is located in the path of the SM 440 to the partition unit 380 . The shared memory / L1 cache 570 can be used for caching reads and writes. One or more of the shared memory / L1 cache 570 , the L2 cache 460 and memory 304 are backup storage.

The combination of data cache and shared memory functionality in a single memory block provides the best overall performance for both types of memory access. The capacity can be used as a cache for programs that do not use shared memory. If z. For example, if the shared memory is designed so that half of the capacity is used, texture and load / store operations can use the remaining capacity. By integrating with the shared memory / L1 cache 570 can use shared memory / L1 cache 570 Act as a high-throughput line for streaming data while providing high-bandwidth, low-latency access to frequently reused data.

If it is designed for general parallel computation, a simpler configuration can be used compared to graphics processing. In particular, the in 3 The graphic processing units shown with fixed functions are bypassed, resulting in a much simpler programming model. In the configuration for general parallel calculations, the work distribution unit 325 Blocks of threads and distributes them directly to the DPCs 420 . The threads in a block run the same program, using a unique thread ID in the calculation to ensure that each thread produces unique results, with the SM 440 for executing the program and performing calculations, the shared memory / L1 cache 570 for communication between the threads and the LSU 554 for reading and writing the global memory via the shared memory / L1 cache 570 and the storage partition unit 380 be used. When designed for general parallel computations, the SM 440 also write commands that the scheduler unit 320 can use to do new work on the DPCs 420 to start.

The PPUs 300 can each have one or more processing cores and / or components thereof and / or be designed so that they perform the functions of one or more processing cores and / or components thereof, such as. B. Tensor Cores (TCs), Tensor Processing Units (TPUs), Pixel Visual Cores (PVCs), Ray Tracing (RT) Cores, Vision Processing Units (VPUs), Graphics Processing Clusters (GPCs), Texture Processing Clusters ( TPCs), Streaming Multiprocessors (SMs), Tree Traversal Units (TTUs), Artificial Intelligence Accelerators (AlAs), Deep Learning Accelerators (DLAs), Arithmetic Logic Units (ALUs), Application Specific Integrated Circuits (ASICs), Floating Point Units ( FPUs), input / output elements (I / Os), Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) elements and / or the like.

The PPU 300 may be in a desktop computer, a laptop computer, a tablet computer, servers, supercomputers, a smartphone (e.g. a wireless, hand-held device), a personal digital assistant (PDA), a digital camera, a Vehicle, a head-mounted display or a head-mounted display, a hand-held electronic device and the like may be present. In one embodiment, the PPU is 300 executed on a single semiconductor substrate. In another embodiment, the PPU is 300 in a system-on-a-chip (SoC) together with one or more other devices, such as B. Additional PPUs 300 , the memory 304 , a RISC-CPU (Reduced Instruction Set Computer), an MMU (Memory Management Unit), a DAC (Digital-to-Analog Converter) and the like.

In one embodiment, the PPU 300 on a graphics card that has one or more storage devices. The graphics card can be designed so that it can be connected to a PCIe slot on a motherboard of a desktop computer. In another embodiment, the PPU 300 an integrated graphics processing unit (iGPU) or a parallel processor residing in the chipset on the motherboard. In another embodiment, the PPU 300 be implemented in reconfigurable hardware. In a further embodiment, parts of the PPU 300 be implemented in reconfigurable hardware.

Exemplary computing system

Systems with multiple GPUs and CPUs are used in a wide variety of industries as developers have more parallelism in applications such as B. Uncover and exploit artificial intelligence. Powerful, GPU-accelerated systems with tens to many thousands of computing nodes are used in data centers, research facilities and supercomputers to solve ever greater problems. As the number of processing devices within the high performance systems increases, the communication and data transfer mechanisms must be scaled to support the increased bandwidth.

5B is a conceptual illustration of a processing system 500 that is done using the PPU 300 the end 3 was implemented according to one embodiment. The exemplary system 565 can be designed to accommodate the in 1C shown procedures 150 and / or the in 2C shown procedures 250 to implement. The processing system 500 assigns a CPU 530 , a switch 510 and multiple PPUs 300 as well as corresponding memory 304 on. The NVLink 310 provides high speed communications links between each of the PPUs 300 ready. Although in 5B a certain number of NVLink 310 - and interconnect 302 Connections is shown, the number of connections to each PPU 300 and the CPU 530 vary. The switch 510 forms the interface between the connection 302 and the CPU 530 . The PPUs 300 who have favourited Memory 304 and the NVLinks 310 can be arranged on a single semiconductor platform to accommodate a parallel processing module 525 to build. In one embodiment, the switch supports 510 two or more protocols to mediate between different connections and / or links.

In another embodiment (not shown) the NVLink 310 one or more high speed communications links between each of the PPUs 300 and the CPU 530 ready, and the switch 510 forms an interface between the connection 302 and each of the PPUs 300 . The PPUs 300 who have favourited Memory 304 and the connection 302 can be arranged on a single semiconductor platform to accommodate a parallel processing module 525 to build. In a further embodiment (not shown) the connection 302 one or more communication links between each of the PPUs 300 and the CPU 530 ready, and the switch 510 forms an interface between each of the PPUs 300 using the NVLink 310 to establish one or more high speed communications links between the PPUs 300 provide. In another embodiment (not shown) the NVLink 310 one or more high-speed communication links between the PPUs 300 and the CPU 530 over the switch 510 ready. In a further embodiment (not shown) the connection 302 one or more communication links directly between the individual PPUs 300 ready. One or more of the NVLink 310 -High speed communications links can be implemented as a physical NVLink link or either an on-chip or on-die link using the same protocol as the NVLink 310 used.

In the context of the present description, a single semiconductor platform can refer to a single unitary semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single semiconductor platform can also refer to multi-chip modules with increased connectivity that simulate on-chip operation and provide significant improvements over the use of a conventional bus implementation. Of course, the various circuits or devices can also be arranged separately or in various combinations of semiconductor platforms according to the wishes of the user. Alternatively, the parallel processing module 525 be implemented as a printed circuit board substrate and each of the PPUs 300 and / or memory 304 can be devices with housing. In one embodiment, the CPU is located 530 , the switch 510 and the parallel processing module 525 on a single semiconductor platform.

In one embodiment, the signaling rate is each NVLink 310 20 to 25 gigabits / second and any PPU 300 has six NVLink 310 Interfaces (as in 5B shown are five NVLink 310 -Interfaces for each PPU 300 available). Any NVLink 310 provides a data transfer rate of 25 gigabytes / second in each direction, with six links 300 Provide gigabytes / second. The NVLinks 310 can only be used for PPU-to-PPU communication, as described in 5B or for a combination of PPU-to-PPU and PPU-to-CPU if the CPU 530 also has one or more NVLink 310 interfaces.

In one embodiment, the NVLink enables 310 direct load / store / atomic access from the CPU 530 on the memory 304 every PPU 300 . In one embodiment, the NVLink supports 310 Coherence operations so that from the memories 304 read data in the cache hierarchy of the CPU 530 can be saved what the cache access latency for the CPU 530 reduced. In one embodiment instructs the NVLink 310 support for address translation services (ATS), which makes the PPU 300 directly on page tables within the CPU 530 can access. One or more of the NVLinks 310 can also be designed to operate in a power-saving mode.

5C shows an exemplary system 565 in which the various architectures and / or functions of the various previous embodiments may be implemented. The exemplary system 565 can be designed to accommodate the in 1C shown procedures 150 and / or the in 2C shown procedures 250 to implement.

As it is shown is a system 565 provided that at least one central processing unit 530 having that with a communication bus 575 connected is. The communication bus 575 can directly or indirectly couple one or more of the following devices: a main memory 540 , a network interface 535 , (a) CPU (s) 530 , (a) display device (s) 545 , (an) input device (s) 560 , a switch 510 and a parallel processing system 525 . The communication bus 575 can be implemented using any suitable protocol and can represent one or more links or buses, such as B. an address bus, a data bus, a control bus or a combination thereof. The communication bus 575 may have one or more bus or connection types, such as B. an ISA bus (Industry Standard Architecture), an EISA bus (Extended Industry Standard Architecture), a VESA bus (Video Electronics Standards Association), a PCI bus (Peripheral Component Interconnect), a PCIe bus (Peripheral Component Interconnect Express), HyperTransport, and / or some other type of bus or connection. In some embodiments, there are direct connections between the components. For example, the CPU (s) can 530 directly to the main memory 540 be connected. In addition, the CPU (s) can 530 directly with the parallel processing system 525 be connected. In the case of a direct or point-to-point connection between components, the communication bus 575 have a PCIe connection to establish the connection. In these examples there is no need for a PCI bus in the system 565 to be available.

Although the various blocks in 5C than via the communication bus 575 are shown connected with lines, this is not intended as a restriction and is only for the sake of clarity. In some embodiments, for example, a presentation component, such as the display device (s) 545 , be viewed as an I / O component, like the input device (s) 560 (e.g. if the display is a touchscreen). As a further example, the CPU (s) can 530 and / or the parallel processing system 525 have a memory (e.g. the main memory 540 in addition to the parallel processing system 525 , the CPUs 530 and / or other components represent a storage device). In other words, the computing device of the 5C is only illustrative. No distinction is made between categories such as “workstation”, “server”, “laptop”, “desktop”, “tablet”, “client device”, “mobile device”, “handheld device”, “game console”, “electronic control unit” (ECU) ”,“ Virtual Reality System ”and / or other types of equipment or systems, since all of them are part of the computing equipment of the 5C be considered.

The system 565 also has a main memory 540 on. Control logic (software) and data are in the main memory 540 that may take the form of a variety of computer readable media. The computer readable media can be any available media that the system 565 can access. The computer readable media can include both volatile and non-volatile media, as well as removable and non-removable media. By way of example and without limitation, the computer readable media can include computer storage media and communication media.

The computer storage media can include both volatile and non-volatile media and / or removable and non-removable media that can be used in any method or technology for storing information, such as, e.g. B. computer-readable instructions, data structures, program modules and / or other data types are implemented. For example, the main memory 540 store computer readable instructions (e.g., representing one or more programs and / or one or more program elements, such as an operating system). Computer storage media can include RAM, ROM, EEPROM, flash memory, or other storage technology, CD-ROM, Digital Versatile Disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage device, or any other medium that can be used to store the information you want and on which the system 565 can access, but are not limited to. As used herein, the term computer storage medium does not include signals per se.

The computer storage media can contain computer readable instructions, data structures, program modules and / or other types of data in a modulated data signal, such as e.g. B. a carrier wave or other transport mechanism embody and have any information transmission media. The term “modulated data signal” can refer to a signal in which one or more of its properties are adjusted or changed in such a way that information is encoded in the signal. By way of example and without limitation, the computer storage media may include wired media such as A wired network or a direct wired connection, and wireless media such as B. acoustic, RF, infrared and other wireless media. Combinations of the above media should also be included in the scope of computer readable media.

Computer programs, when executed, enable the system 565 to perform various functions. The CPU (s) 530 may be configured to execute at least some of the computer readable instructions to one or more components of the system 565 to control in order to carry out one or more of the methods and / or processes described here. The CPU (s) 530 each may have one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a plurality of software threads simultaneously. The CPU (s) 530 can have any type of processor and depending on the type of system implemented 565 have different processor types (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). Depending on the type of system 565 For example, the processor can be an Advanced RISC Machines (ARM) processor implemented with Reduced Instruction Set Computing (RISC) or an x86 processor implemented with Complex Instruction Set Computing (CISC). The system 565 can be one or more CPUs 530 have, in addition to one or more microprocessors or additional co-processors, such as. B. mathematical co-processors.

In addition to or as an alternative to the CPU (s) 530 can the parallel processing module 525 be configured to execute at least some of the computer readable instructions to one or more components of the system 565 to control in order to carry out one or more of the methods and / or processes described here. The parallel processing module 525 can from the system 565 be used to render graphics (such as 3D graphics) or perform general-purpose calculations. For example, the parallel processing module 525 be used for general purpose computing on GPUs (GPGPU). In embodiments, the CPU (s) can 530 and / or the parallel processing module 525 discreetly or collectively execute any combination of the methods, processes and / or portions thereof.

The system 565 also shows the input device (s) 560 , the parallel processing system 525 and the display device (s) 545 on. The display device (s) 545 may include a display (e.g., monitor, touchscreen, television screen, heads-up display (HUD), other types of displays, or a combination thereof), speakers, and / or other presentation components. The display device (s) 545 can / can data from other components (e.g. the parallel processing system 525 , the CPU (s) 530 etc.) and output the data (e.g. as an image, video, sound, etc.).

The network interface 535 can it to the system 565 allow to be logically coupled to other devices, including the input devices 560 , the display device (s) 545 and / or other components, some of which are in the system 565 can be built-in (e.g. integrated). Input devices shown 560 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, and so on. The input devices 560 may provide a natural user interface (NUI) that processes aerial gestures, speech, or other physiological input generated by a user. In some cases, the inputs can be transmitted to an appropriate network element for further processing. A NUI can be any combination of speech recognition, pen recognition, face recognition, biometric recognition, both on and off screen gesture recognition, aerial gestures, head and eye tracking, and touch recognition (as described in more detail below) in conjunction with a display of the system 565 to implement. The system 565 can depth cameras such as B. stereoscopic camera systems, infrared camera systems, RGB camera systems, a touchscreen technology and combinations thereof, for gesture recognition and capture. In addition, the system can 565 Have accelerometers or gyroscopes (e.g. as part of an Inertial Measurement Unit (IMU)) that allow motion detection. In some examples, the output from the accelerometers or gyroscopes may be from the system 565 can be used to represent immersive augmented reality or virtual reality.

In addition, the system can 565 via a network interface 535 be connected to a network (e.g. a telecommunications network, a local area network (LAN), a wireless network, a wide area network (WAN) such as the Internet, a peer-to-peer network, a cable network or the like) for communication purposes. The system 565 may have a distributed network and / or a cloud computing environment.

The network interface 535 may have one or more receivers, transmitters, and / or transceivers that allow the system 565 enable communication with other computing devices over an electronic communication network, including wired and / or wireless communication. The network interface 535 may have components and functions to enable communication over any number of different networks, such as E.g. wireless networks (e.g. Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee etc.), wired networks (e.g. communication via Ethernet or InfiniBand), wide area networks with low power (e.g. LoRaWAN, SigFox etc.) and / or the Internet.

The system 565 can also have a secondary memory (not shown). The secondary storage has, for example, a hard disk drive and / or a removable storage drive, which is a floppy disk drive, a magnetic tape drive, a compact disk drive, a DVD drive (digital versatile disk), a recording device or a USB flash memory. The removable storage drive reads from and / or writes to removable storage in a known manner. The system 565 may also include a hardwired power supply, battery power supply, or a combination thereof (not shown). The power supply can be provided by the system 565 provide power to the operation of the components of the system 565 to enable.

Each of the aforementioned modules and / or devices can even be arranged on a single semiconductor platform around the system 565 to train. Alternatively, the different modules can also be arranged separately or in different combinations of semiconductor platforms according to the wishes of the user. While various embodiments have been described above, it should be understood that they are presented as an example and not as a limitation. Therefore, the breadth and scope of a preferred embodiment should not be limited by any of the exemplary embodiments described above, but should only be defined in accordance with the following claims and their equivalents.

Exemplary network environments

Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and / or other types of devices. The client devices, servers, and / or other types of devices (e.g., any device) may be on one or more instances of the processing system 500 the 5A and / or the example system 565 the 5B be implemented - e.g. E.g., each device may have similar components, features, and / or functionality of the processing system 500 and / or the example system 565 exhibit.

Components of a network environment can communicate with one another via one or more networks, which can be wired, wireless, or both. The network can comprise multiple networks or a network of networks. For example, the network can have one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or a public switched telephone network (PSTN) and / or one or more private networks . If the network comprises a wireless telecommunications network, components such as a base station, communication tower, or even access points (as well as other components) can provide a wireless connection.

Compatible network environments can be one or more peer-to-peer network environments - in which case a server cannot be included in a network environment - and one or more client-server network environments - in which case one or more servers can be included in a network environment - exhibit. In peer-to-peer network environments, the functionality described here can be implemented in relation to one or more servers on any number of client devices.

In at least one embodiment, a network environment can include one or more cloud-based network environments, a distributed computing environment, a combination thereof, and so on. exhibit. A cloud-based network environment can have a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. A framework layer can have a framework for supporting software of a software layer and / or one or more application (s) of an application layer. The software or application (s) may each include web-based service software or applications. In embodiments, one or more of the client devices may utilize the web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be some type of free and open source software web application framework, e.g. For example, a distributed file system is used for processing large amounts of data (e.g. "big data"), but is not limited to this.

A cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Each of these various functions may be distributed across multiple locations of central or core servers (e.g., one or more data centers, which may be distributed across a state, region, country, globe, etc.). When a connection to a user (e.g., a client device) is relatively close to one or more edge servers, one or more core servers can assign at least some of the functionality to the edge server (s). A cloud-based network environment can be private (e.g. limited to a single organization), public (e.g. available to many organizations) and / or a combination thereof (e.g. a hybrid cloud environment).

The client device (s) may at least some of the components, features, and functions of the exemplary processing system 500 the 5B and / or the exemplary system 565 the 5C exhibit. As an embodiment and without restriction, a client device can be configured as a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet computer, a smart watch, a portable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or a device, a video player, a video camera, a surveillance device or a surveillance system, a vehicle, a boat, an airship, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, a device, a consumer electronics device, a workstation, an edge device, a any combination of these described devices or any other suitable device.

Machine learning

Deep Neural Networks (DNNs) that run on processors like the PPU 300 are used for a variety of applications, from self-driving cars to faster drug development, from automatic captioning in online image databases to intelligent real-time voice translation in video chat applications. Deep learning is a technology that models the neural learning process of the human brain and thereby continuously learns, becomes more and more intelligent and, over time, delivers increasingly more accurate results. A child initially learns from an adult to correctly identify and classify various shapes and eventually is able to recognize shapes without tutoring. Similarly, a deep learning or neural learning system needs to be trained in object recognition and classification so that it becomes more intelligent and efficient in identifying basic objects, hidden objects, etc., and at the same time assigning a context to the objects.

At the simplest level, neurons in the human brain look at various inputs that are being received, each of those inputs being assigned levels of importance, and the output passed on to other neurons to respond to. An artificial neuron or perceptron is the simplest model of a neural network. In one example, a perceptron may receive one or more inputs representing various features of an object that the perceptron is trained to recognize and classify, and each of these features is assigned a weight based on the importance of that feature in defining the shape of an object.

A deep neural network (DNN) model has multiple layers with many connected nodes (e.g., perceptrons, Boltzmann machines, radial basis functions, convolution layers, etc.) that can be trained with enormous amounts of input data to solve complex problems quickly and with high Solving accuracy. In one example, a first layer of the DNN model breaks down an input image of a car into different sections and looks for basic patterns such as lines and angles. The second layer puts the lines together to look for higher-level patterns like wheels, windshields, and mirrors. The next layer identifies the type of vehicle, and the last few layers create a label on the input image that identifies the model of a particular make of car.

Once the DNN is trained, it can be deployed and used to identify and classify objects or patterns in a process called inference. Examples of inference (the process by which a DNN extracts useful information from a given input) include handwritten numbers on checks deposited in ATMs, identifying pictures of friends in photos, providing movie recommendations to over fifty million users, identifying and classifying them different types of cars, pedestrians and road hazards in driverless cars or translate human speech in real time.

During training, the data flows through the DNN in a forward propagation phase until a prediction is generated indicating a label that corresponds to the input. If the neural network does not correctly label the input, the errors between the correct label and the predicted label are analyzed and the weights for each feature are adjusted in a backward propagation phase until the DNN correctly labels the input and other inputs in a training dataset. Training complex neural networks requires enormous amounts of parallel computing power, which have floating point multiplications and additions provided by the PPU 300 get supported. Inferencing is less computationally intensive than training because it is a latency-dependent process in which a trained neural network is applied to new input that it has not seen before to classify images, translate speech, and generally derive new information .

Neural networks rely heavily on mathematical matrix operations, and complex multilayer networks require enormous amounts of floating point performance and bandwidth to be both efficient and fast. With thousands of cores optimized for matrix math operations and delivering tens to hundreds of TFLOPS performance, the PPU 300 a computing platform capable of delivering the performance required for artificial intelligence and machine learning applications based on deep neural networks.

In addition, images generated using one or more of the techniques disclosed herein can be used to train, test, or certify DNNs that are used to recognize objects and environments in the real world. Such images can include scenes of streets, factories, buildings, urban settings, rural settings, people, animals, and other physical objects or real world environments. Such images can be used to train, test or certify DNNs that are used in machines or robots to manipulate, handle or change physical objects in the real world. In addition, such images can be used to train, test or certify DNNs that are used in autonomous vehicles to navigate and move the vehicles through the real world. In addition, images generated using one or more of the techniques disclosed herein can be used to convey information to users of such machines, robots, and vehicles.

5D shows components of an exemplary system 555 which, according to at least one embodiment, can be used to train and apply machine learning. As will be discussed, various components may be provided by various combinations of computing devices and resources, or by a single computing system that may be under the control of a single device or multiple devices. In addition, the aspects can be triggered, initiated or requested by different units. In at least one embodiment, the training of a neural network can be instructed by a provider who is familiar with a provider environment 506 while, in at least one embodiment, the training may be requested from a customer or other user via a client device 502 or some other such resource has access to a provider environment. In at least one embodiment, the training data (or data to be analyzed by a trained neural network) can be from a provider, a user, or a content provider 524 provided to a third party. In at least one embodiment, the client device 502 z. B. be a vehicle or an object that is to be navigated on behalf of a user and that can make inquiries and / or receive instructions that help navigate a facility.

In at least one embodiment, requests can be made over at least one network 504 Submitted to by a provider environment 506 to be received. In at least one embodiment, a client device 502 any suitable electronic device and / or computing device that enables a user to generate and send such requests, e.g. B., but not limited to, a desktop computer, a notebook computer, a computer server, smartphones, a tablet computer, game consoles (portable or not), computer processors, computational logic and set-top boxes. The network (s) 504 may have any suitable network for the transmission of a request or other such data, e.g. Such as the Internet, an intranet, a cellular network, a local area network (LAN), a wide area network (WAN), a personal network (PAN), an ad hoc network of direct wireless connections between peers, and so on.

In at least one embodiment, the requests may be made at an interface layer 508 are received, which in this example data to a training and inference manager 532 can forward. In at least one embodiment, the training and inference manager 532 receive a request to train a neural network and data for a request to a training module 512 provide, being the training and inference manager 532 may have a system or a service that has hardware and software for managing requirements and for providing corresponding data or content. In at least one embodiment, the training module 512 select an appropriate model or neural network to be used if not specified by the requirement, and train a model using relevant training data. In at least one embodiment, the training data can be a set of data stored in a training data store 514 is stored by the client device 502 received or from a provider 524 a third party. In at least one embodiment, the training module 512 be responsible for the training data. A neural network can be any suitable network, such as e.g. B. a feedback neural network (RNN) or a folding neural network (CNN). As soon as a neural network has been trained and successfully evaluated, a trained neural network can, for example, be stored in a model memory 516 which can store various models or networks for users, applications or services, etc. In at least one embodiment, there may be multiple models for a single application or entity that can be used based on a number of different factors.

In at least one embodiment, a request from the client device can be made at a later date 502 (or another such device) for content (e.g. path determinations) or data that are at least partially determined or influenced by a trained neural network. This request can, for example, have input data that are to be processed with a neural network in order to obtain one or more conclusions or other output values, classifications or predictions. In at least one embodiment, the input data can be from the interface layer 508 received and to the inference module 518 forwarded, although another system or service may be used. In at least one embodiment, the inference module 518 a suitable trained network, such as B. a trained deep neural network (DNN), as described here, from the model memory 516 refer if it is not already locally in the inference module 518 is stored. The inference module 518 can provide data as input to a trained network, which can then produce one or more conclusions as output. This can e.g. B. Include a classification of an instance of input data. In at least one embodiment, the conclusions can then be passed to the client device 502 for display or other communication with a user. In at least one embodiment, context data for a user can also be stored in a user context data store 522 which may include data about a user that may be useful as input to a network in drawing conclusions or determining data that will be returned to a user after instances have been obtained. In at least one embodiment, relevant data including at least some of the input or inference data can also be in a local database 534 stored for the processing of future inquiries. In at least one embodiment, a user can use account information or other information to access resources or features of a provider environment. In at least one embodiment, if permissible and available, user data can also be collected and used for further training of models in order to enable more precise conclusions for future inquiries. In at least one embodiment, requests can be made through a user interface to a machine learning application 526 that is set up on the client 502 will be received and the results will be displayed through the same interface. A client device can have resources such as a processor 528 and a memory 562 for generating a request and for processing results or a response, as well as at least one data storage element 552 to store data for machine learning application 526 exhibit.

In at least one embodiment, the processor is 528 (or a processor of the training module 512 or the inference module 518 ) a central processing unit (CPU). However, as mentioned earlier, resources in such environments can use GPUs to process data for at least certain types of requests. With thousands of cores, GPUs such as B. the PPU 300 , designed to handle significant parallel workloads, and has become very popular in deep learning for training neural networks and making predictions. While the use of GPUs for offline builds has enabled faster training of larger and more complex models, making predictions offline implies that either input features cannot be used at request time or predictions for all permutations of features are generated and in a look-up table must be saved in order to serve real-time inquiries. If a deep learning framework supports a CPU mode and a model is small and simple enough to feed forward on a CPU with reasonable latency, then a service could host a model on one CPU instance. In this case, the training can be carried out offline on a GPU and the inference in real time on a CPU. If a CPU approach is impractical, then a service can run on a GPU instance. However, since GPUs have different performance and cost properties than CPUs, it may be necessary to design a service that outsources a runtime algorithm to a GPU differently from a CPU-based service.

In at least one embodiment, video data from the client device 502 for enrichment in the provider environment 506 to be provided. In at least one embodiment, the video data can be enhanced on the client device 502 are processed. In at least one embodiment, video data may be from a content provider 524 streamed to a third party and from the content provider 524 a third party, the provider environment 506 or the client device 502 be improved. In at least one embodiment, video data from the client device 502 for use as training data in the provider environment 506 to be provided.

In at least one embodiment, monitored and / or unsupervised training can be provided by the client device 502 and / or the provider environment 506 be performed. In at least one embodiment, a set of training data 514 (e.g. classified or labeled data) is provided as input to act as training data. In one embodiment, the set of training data can be used in a generating adversarial training configuration to train a generating neural network.

In at least one embodiment, the training data can include images of at least one human object, an avatar, a character, an animal, an object or the like for which a neural network is to be trained. In at least one embodiment, the training data can have instances of at least one object type for which a neural network is to be trained, as well as information that identifies this object type. In at least one embodiment, the training data can have a set of images that each contain a representation of an object type, each image also having an identifier, metadata, a classification or other information for identifying an object type that is represented in a corresponding image, or associated with it. Various other types of data can also be used as training data, such as B. text data, audio data, video data, etc. In at least one embodiment, the training data 514 as training input for a training module 512 provided. In at least one embodiment, the training module 512 be a system or service comprising hardware and software, e.g. B. one or more computing devices running a training application to train a neural network (or other model or algorithm, etc.). In at least one embodiment, a model can be any suitable statistical model, network, or algorithm useful for such purposes as e.g. B. an artificial neural network, a deep learning algorithm, a learning classifier, a Bayesian network and so on. In at least one embodiment, the training module 512 an initial model or another untrained model from a suitable repository 516 select and training data 514 to train a model, thereby generating a trained model (e.g., trained deep neural network) that can be used to classify similar types of data or generate other such inferences. In at least one embodiment in which no training data is used, a suitable initial model can nevertheless be used for training on input data via a training module 512 to be selected.

In at least one embodiment, a model can be trained in a number of ways, depending in part on the type of model chosen. In at least one embodiment, a machine learning algorithm can be provided with a set of training data, wherein a model is a model artifact that is created by a training process. In at least one embodiment, each instance of training data contains a correct answer (e.g., a classification) that can be referred to as a goal or a goal attribute. In at least one embodiment, a learning algorithm finds patterns in the training data that map the attributes of the input data to a goal, an answer to be predicted, and a machine learning model is output that captures these patterns. In at least one embodiment, a machine learning model can then be used to obtain predictions for new data for which a goal is not specified.

In at least one embodiment, the training and inference manager 532 choose from a set of machine learning models that includes binary classification, multi-class classification, generating models, and regression models. In at least one embodiment, the type of model to be used may depend, at least in part, on the type of target to be predicted.

Images generated using one or more of the techniques disclosed herein can be displayed on a monitor or other device. In some embodiments, the display device can be coupled directly to the system or processor that generates or displays the images. In other embodiments, the display device may be indirectly coupled to the system or processor, e.g. B. over a network. Examples of such networks include the Internet, mobile telecommunications networks, a WIFI network, and any other wired and / or wireless network system. When the display device is indirectly coupled, the images generated by the system or processor can be streamed over the network to the display device. Such streaming enables, for example, video games or other applications to render images running on a server, data center, or cloud-based computing environment, and the rendered images to one or more user devices (such as a computer, video game console, smartphone, other mobile device, etc.), and displaying them there, which are physically separate from the server or data center. Therefore, the techniques disclosed herein can be used to enhance the streamed images and to improve services that stream images such as NVIDIA GeForce Now (GFN), Google Stadia, and the like.

Weight demodulation for a neural generator network

The style-based GAN architecture (StyleGAN), which is in the style-based generator system 100 is implemented, delivers impressive results in data-driven independent generating image modeling. However, the style-based GAN architecture can synthesize images that have undesirable artifacts. As is also described here, several of the characteristic artifacts can be avoided or reduced by changing the structure of the neural synthesis network and / or the training methods. In one embodiment, the normalization operation is restructured and the style-based generator system can be adjusted to provide good conditioning in mapping latent vectors to output data such as B. Images, to promote. The improvement of the style-based GAN architecture to avoid and / or reduce artifacts significantly improves the results in the unconditioned or independent image modeling, both with regard to the existing distribution quality metrics and the perceived quality of the output data.

In the following, various optional architectures and functions are presented with which the above framework can be implemented according to the wishes of the user. It is expressly pointed out that the following information is for illustrative purposes only and should in no way be regarded as restrictive. Each of the following features can optionally be used with or without excluding other features described.

6A shows artifacts in images and feature maps generated by the style-based generator system 100 can be generated according to one embodiment. An artifact 601 appears as a blob-shaped anomaly that resembles a drop of water. Even if the artifact 601 not evident in the final picture is the artifact 602 present in the intermediate feature maps generated by the neural synthesis network. In one embodiment, the anomaly begins to appear at a resolution of 64 x 64, is present in all subsequent feature maps, and grows stronger at higher resolutions. Another example is an artifact 603 visible in a final image of a vehicle and a corresponding artifact 604 is present in a feature map that is used during the synthesis of the final output image of the vehicle is generated. Another example is an artifact 605 visible in a final output image of a horse and a corresponding artifact 606 is present in a feature map that is generated during the synthesis of the final output image of the horse. The presence of such a consistent artifact is puzzling as a discriminator should be able to detect it during training.

The “blob” or “blob” or “water drop” artifacts in images generated by the neural synthesis network 140 appear to be caused by the normalization operation taking place in the processing layers of the neural synthesis network 140 is too destructive and completely removes the mean and scale of intermediate activations for each feature map. Indeed, the style-based GAN is likely to generate the artifacts to circumvent a design flaw in its architecture.

direkt als Eingabe in das neuronale Synthesenetzwerk 140 einzuspeisen, transformiert das neuronale Abbildungsnetzwerk 110 zunächst den eingegebenen latenten Code in einen latenten Zwischencode $w\in \mathcal{W}.$

Die von der Stilkonvertierungseinheit 115 durchgeführten affinen Transformationen erzeugen dann die Stilsignale, die über die adaptive Instanznormalisierung (AdaIN) die Schichten des neuronalen Synthesenetzwerks 140 steuern. Im Kontext der folgenden Beschreibung bezieht sich die Instanznormalisierung auf die Durchführung der Normalisierung separat oder unabhängig für jede Instanz oder jedes Muster (Bild), ohne Interaktion zwischen den Mustern. Im Gegensatz dazu werden bei der Batch-Normalisierung alle Muster in einem Minibatch gemeinsam normalisiert. Zusätzlich wird in einer Ausführungsform die stochastische Variation dadurch erleichtert, dass dem neuronalen Synthesenetzwerk 140 zusätzliche zufällige Rauschkarten zugeführt werden. Die Rauschkarten können in Zwischendaten eingefügt werden, die von jeder Faltungsschicht des neuronalen Synthesenetzwerks 140 ausgegeben werden. Es wurde gezeigt, dass der latente Zwischenraum W aufgrund des unkonventionellen Designs viel weniger verflochten ist als der latente Eingangsraum Z. Im Rahmen der folgenden Beschreibung konzentriert sich die Analyse ausschließlich auf W, da dies der relevante latente Raum aus der Sicht des neuronalen Synthesenetzwerks 140 ist.A special feature of the style-based generator system 100 is the unconventional architecture of the generator. Instead of the latent code $z\in \mathcal{Z}$

directly as input into the neural synthesis network 140 feed transforms the neural mapping network 110 first converts the entered latent code into an intermediate latent code $w\in \mathcal{W.}.$

The one from the style conversion unit 115 The affine transformations carried out then generate the style signals that the layers of the neural synthesis network via the adaptive instance normalization (AdaIN) 140 steer. In the context of the following description, instance normalization refers to performing normalization separately or independently for each instance or pattern (image), with no interaction between the patterns. In contrast, with batch normalization, all patterns in a minibatch are normalized together. In addition, in one embodiment, the stochastic variation is facilitated by the fact that the neural synthesis network 140 additional random noise maps can be supplied. The noise maps can be inserted into intermediate data obtained from each convolutional layer of the neural synthesis network 140 are issued. It was shown that the latent space W is much less intertwined than the latent entrance space Z due to the unconventional design. In the context of the following description, the analysis focuses exclusively on W, as this is the relevant latent space from the point of view of the neural synthesis network 140 is.

The artifacts are introduced by the AdalN operation which normalizes the mean and variance of each feature map separately, potentially destroying any information found in the sizes of the features in relation to one another. The droplet artifact can be a result of the style-based generator system 100 intentionally smuggling signal strength information past instance normalization. More specifically, by generating a strong, localized peak that dominates the statistics, the style-based generator system 100 effectively scale the signal elsewhere at will. This hypothesis is supported by the finding that, in one embodiment, the droplet artifacts completely disappear when the normalization step is removed from the neural synthesis network 140 removed as described below.

6B shows a block diagram of the in 2 B processing blocks shown 200 according to one embodiment. In the in the 1A and 2 B shown neural synthesis network 140 Style signals may be active for each processing stage to apply attributes specific to that processing stage. In one embodiment, the style conversion unit applies 115 applies a learned affine transform on W to generate each style signal. Conceptually, feature maps (which represent the content of an image) are generated by different processing layers 125 , 135 and or 220 of the neural synthesis network 140 are generated depending on style signals modified by the neural mapping network 110 to be provided. In other words, the first order statistics are replaced by style-specific attributes for each level. Compared to the 1A and 2 B becomes the operation of adaptive instance normalization (AdaIN) within the layers 120 and 130 and the modules 220 broken down into its two components: a normalization 620 followed by a modulation 625 , both of which act on the mean and the standard deviation per feature map. The normalization 620 is performed for each style application in order to avoid the accumulation of the respective style in the subsequent stages in which different styles with different scales can be applied.

Before normalization 620 can through an element 610 spatial noise can be inserted into the first intermediate data. In one embodiment, the first intermediate data is the learned constant or the input pattern. In one embodiment (not shown), a bias for each scale can also be applied along with the spatial noise. Learned parameters (weights) are used by each convolutional layer 225 applied. A style block 600 , of at least the folding layer 225 who have favourited normalization 620 , the introduction of spatial noise by the element 610 and demodulation 625 having, can be restructured to remove artifacts by demodulating the weights (instead of normalizing the activations). Interestingly, the neural synthesis network is turning 140 the spatial noise within the processing block 200 which results in a relative impact of the spatial noise being inversely proportional to a size of the style signal delivered to the processing block 200 is applied. More predictable results can be obtained if the insert operation is outside the processing block 200 takes place so that the insertion works on normalized data instead.

6C Figure 3 illustrates a block diagram of a style block 640 that can be used in implementing some embodiments of the present disclosure. Compared to the in 6B processing block shown 200 is the element's introduction of spatial noise (and bias) 610 outside of the style block 640 arranged. In other words, the spatial noise and the bias are applied regardless of any style encoded in a style signal. In one embodiment, after moving the insert operation, it is outside of the style block 600 to create the style block 640 sufficient that the normalization and modulation are based on the standard deviation only (ie the mean is not needed). In one embodiment, the application of the bias, the spatial noise and the normalization to the input pattern (e.g. first intermediate data) can also be removed safely and without significant disadvantages.

Compared to normalization 620 and modulation 625 in the style block 600 the modulation of the mean value is removed and the modulation unit 630 and the normalization unit 635 only work with the standard deviation. The modulation unit 630 modulates the first intermediate data based on the style signal to produce modulated features. In one embodiment, the style block 640 in addition to the modulation unit 630 , the folding layer 225 and the normalization unit 635 also an upsampler 632 have similar operations as the upsampler 235 in 2 B performs. The modulated features are upsampled (or not) and put into the convolutional layer 225 entered as input activations. The parameters are from the convolutional layer 225 applied to the input activations to produce output activations.

The style block 640 can be restructured to decrease or decrease the strength of the normalize operation while maintaining the scale-specific effects of the styles. Simply removing the instance normalization operation improves image quality (e.g., it removes the blob artifacts). However, removing instance normalization also causes the styles' effects to be cumulative and not specific to each scale. Therefore, the controllability that scale-specific styles provide for synthesis is severely limited. An alternative that eliminates the artifacts while maintaining controllability is normalization based on the expected statistics of the incoming feature maps, but without explicit enforcement.

wobei w und w' die ursprünglichen bzw. modulierten Gewichte sind, s_i der Skalierungsfaktor ist, der der i-ten eingegebenen Merkmalskarte entspricht, und j und k die Ausgabe-Merkmalskarten bzw. einen räumlichen Footprint bzw. Fußabdruck der Faltung spezifizieren.The style block 640 in 6C has at least the modulation unit 630 , the folding layer 225 and the normalization unit 635 on. The effect of a modulation followed by a convolution is that the modulation scales each input feature map of the convolution based on the incoming style, which can alternatively be implemented by scaling the convolution weights: $$w_{ijk}^{\text{'}}=s_i\cdot w_{ijk},$$

where w and w 'are the original and modulated weights, respectively, s _{i is} the scaling factor corresponding to the i-th input feature map, and j and k specify the output feature maps and a spatial footprint of the convolution, respectively.

d. h., jede Ausgabe-Merkmalskarte wird mit der L²-Norm der entsprechenden Gewichte skaliert. Die anschließende Normalisierung zielt darauf ab, die Ausgabeaktivierungen wieder auf eine Einheitsstandardabweichung zurückzubringen. Basierend auf der Gleichung (3) wird die Wiederherstellung der Ausgaben erreicht, wenn jede Ausgabe-Merkmalskarte j mit 1/_σj skaliert („demoduliert“) wird. Alternativ kann die Demodulation auch in die Faltungsgewichte integriert werden: $$w_{ijk}^{\text{'}\text{'}}=\raisebox{1ex}{$w_{ijk}^{\text{'}}$}\!\left/ \!\raisebox{-1ex}{$\sqrt{{\displaystyle {\sum }_{i,k}{w}_{ijk}^{\text{'}}{}^2+\epsilon }}$}\right.,$$

wobei ∈ eine kleine Konstante ist, um numerische Probleme zu vermeiden.The purpose of instance normalization is now to remove the effect of s from the statistics of the output feature maps generated by each convolution. The goal of removing the effect of s from the output feature maps can be achieved more directly. Assuming the intermediate data going into the style block 640 are entered, independent and identically distributed (iid) random variables with unit standard deviation, then have those from the convolutional layer 225 generated output activations after modulation and convolution a standard deviation of $${\sigma }_j=\sqrt{{\displaystyle {\sum }_{i,k}{w}_{ijk}^{\text{'}}{}^2}},$$

that is, each output feature map is ^{scaled with the L 2} norm of the corresponding weights. The subsequent normalization aims to bring the output activations back to a unit standard deviation. Based on equation (3), the restoration of the outputs is achieved when each output feature _{map j is} scaled (“demodulated”) with 1 / σj. Alternatively, the demodulation can also be integrated into the convolution weights: $$w_{ijk}^{\text{'}\text{'}}=\raisebox{1ex}{$w_{ijk}^{\text{'}}$}\!\left/ \!\raisebox{-1ex}{$\sqrt{{\displaystyle {\sum }_{i,k}{w}_{ijk}^{\text{'}}{}^2+\epsilon }}$}\right.,$$

where ∈ is a small constant to avoid numerical problems.

In view of the practical implementation of equations (2) and (4), it is important to note that the resulting set of weights will be different for each pattern in a minibatch, which can preclude direct implementation with standard convolution primitives. In one embodiment, grouped folds can be used to temporarily reshape the weights and activations so that each fold sees a pattern with N groups instead of N patterns with one group. The grouped convolutions are efficient because the transformation operations do not actually change the content of the weight and activation tensors. In another embodiment, explicit scaling operations before and after the convolution can be used.

6D Figure 3 illustrates a block diagram of a style block 645 that can be used in implementing some embodiments of the present disclosure. The normalization unit 635 in the style block 640 is replaced by a "demodulation" operation that is applied to the with each convolutional layer 225 associated weights in the style block 645 is applied. It should be understood that these and other arrangements as described herein are presented as examples only. Other arrangements and elements (e.g., machines, interfaces, functions, arrangements, groupings of functions, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. In addition, many of the elements described herein are functional units that can be implemented as individual or distributed components, or in conjunction with other components, and in any suitable combination and location. Various functions as described here can be performed by hardware, firmware and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. In addition, those skilled in the art understand that any system that performs the operations of the style block 640 or 645 is within the scope and spirit of the embodiments of the present disclosure.

As in 6D is shown, instead of normalizing the features (e.g. the intermediate data), the weights are normalized based on the “expected statistics” of the intermediate data. Since convolution is a linear operator, the scaling of the weights is equivalent to the scaling of the features. Therefore, compared to the style block 640 the modulation unit 630 and the normalization unit 635 removed from the processing of the features and a modulation unit 650 and a demodulation unit 655 inserted into the processing path for the weights.

The weights are first provided by the modulation unit 650 modulated with the style signal to produce modulated weights. The modulation of the weights is equivalent to a scaling of the first intermediate data by the modulation unit 630 to find the modulated features in the style block 640 to create. In the style block 645 the entered characteristics are optionally carried out by the upsampler 632 upsampled and the convolutional layer 225 input to generate output features. An expected standard deviation of the output characteristics is calculated, assuming that the input characteristics are normally distributed. Finally, the modulated weights are used by the demodulation unit 655 demodulates based on the expected standard deviation to produce normalized weights. The demodulation of the modified weights is equivalent to scaling that of the convolutional layer 225 generated output characteristics. Spatial noise is optionally inserted into the output features to create modified features. A neural synthesis network that contains at least one style block 645 can then generate output data (e.g. images) according to the styles, but without the “blob” artifacts or with significantly reduced artifacts.

The weights for each folding layer 225 are within the style block 645 adjusted based on s using equations (2) and (4). To avoid the activation function in Equation (4) has to be taken into account, the activation functions can be scaled so that they maintain the expected signal variance. Compared to instance normalization, the demodulation method is weaker because it is based on statistical assumptions about the signal, rather than on the actual contents of the feature maps. Overall, the statistical analysis is carried out by the style block 645 implemented as a replacement for the data-dependent normalization.

6E provides a flow diagram of a method 660 for demodulating weights applied by a generating neural network, according to an embodiment. Each block of the method 660 as described herein includes a computing process that can be performed with any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be carried out as computer-usable instructions stored on computer storage media. The method can be provided by a stand-alone application, service or hosted service (stand-alone or in combination with another hosted service) or a plug-in for another product, to name just a few examples. In addition, the procedure 660 exemplary in relation to the style block 645 the 6D described. However, this method can additionally or alternatively be performed by any system or combination of systems including, but not limited to, the systems described herein. In addition, those skilled in the art understand that any system that incorporates the method 660 performs falls within the scope and spirit of the embodiments of the present disclosure.

In step 665 receives the style block 645 first weights for modifying first features in a first stage of a neural generator network having a plurality of stages, a last stage of the neural generator network generating output data. In one embodiment, the neural synthesis network 140 of the style-based generator system 100 restructured to the style block 645 to be implemented to generate the output data. In one embodiment, the components of the first intermediate data that are in the neural synthesis network 140 can be entered, adjusted during training and initialized with N (0.1). In another embodiment, the first intermediate data correspond to additional input data, such as e.g. B. a reference image (or are derived from it).

In step 670 modulates the modulation unit 650 the first weights with a first style control signal to generate first modulated weights. In step 675 demodulates the demodulation unit 655 the first modulated weights to produce first normalized weights. In one embodiment, the modulation unit perform 650 and the demodulation unit 655 demodulates to change the weights (instead of activations) on a per-pattern basis. In other words, the weights can be modulated for each first intermediate data entering the neural synthesis network 140 is entered. In step 680 the first normalized weights from the first stage are applied to generate modified first features. In one embodiment, the modified first features are those of the style block 645 generated output characteristics. In one embodiment, the modified first features are modified features resulting from the introduction of spatial noise into the output features. In step 685 the modified first features are processed by at least one additional stage of the neural generator network in order to generate the output data which have a content corresponding to the first features.

In one embodiment, the at least one additional level comprises a different style block 645 . In one embodiment, spatial noise and / or a bias is introduced in order to further modify the modified features, as a result of which modified intermediate data are generated which are processed by the at least one additional stage of the neural generator network. In one embodiment, the weight modulation and demodulation operations are contained in a neural synthesis network that includes one or more stages, each of which is a style block 645 and the demodulation operation is performed in the output style block 645 omitted.

6F provides output images according to one embodiment 662 and modified feature maps 664 and 668 generated by weight demodulation. Compared to the output images in 6A are missing in the output images 662 the visible artifacts or are greatly reduced. Similarly, there are no visible artifacts in the modified feature maps 664 and 668 made by style blocks 645 can be generated for different scales, or are greatly reduced. Reduction and / or removal of artifacts is achieved while maintaining full controllability of styles at different scales. In addition, the training time can be reduced by a simplified data flow for modulation and demodulation of the weights instead of the feature maps.

Smoothing regularization for a neural generator network

A smoothing regularization method can also be used in combination with or independently of the weight demodulation in order to improve the quality of the output data generated by a neural generator network. Smoothing regularization tries to ensure that small changes in the input to a neural network cause small changes in the output, in such a way that the size of the change is uniform over the space of the possible inputs as well as the possible directions of the change of the inputs remain. In other words, small changes in latent space (or the latent space which in turn changes the style signals) should result in small changes in output space (e.g., images). The aim is to achieve an equal distribution of the changes in the input space and an equal distribution of the corresponding changes in the output space. The smoothing regularization method can be used to promote a smooth transformation between a latent space and the output data, between the first intermediate data and either other intermediate data or the output, between any two intermediate data, or between any intermediate data and the output.

A metric that can be used to measure smoothness can also indicate improved quality in the output data. During training, the metric can be used to calculate the learned parameters of the style-based generator system 100 adapt. In particular, the weights for the neural synthesis network 140 adjusted by smoothing regularization to improve both the metric and the quality of the output data.

Several metrics are available for the quantitative analysis of the quality of images that have been generated using generating processes. The Frechet-Inception-Distance (FID) measures differences in the density of two distributions in the high-dimensional feature space of an InceptionV3 classifier. Precision and Recall (P&R) provide additional visibility by explicitly quantifying the percentage of generated images that are similar to training data and the percentage of training data that can be generated, respectively. While FID and P&R successfully capture many aspects of a generator, both metrics have a kind of blind spot for image quality. FID and P&R are based on classification networks that have recently been shown to focus on textures rather than shapes, and consequently these metrics do not accurately capture all aspects of image quality. A perceptual path length (PPL) metric, originally introduced as a method of estimating the quality of interpolations of latent space, correlates with the consistency and stability of shapes. PPL quantifies the smoothness of a transformation by estimating the expected path length of a random interpolation path in the latent space, measured in terms of perceptible changes in the image as the interpolation progresses. The PPL metric can be calculated for individual images (PPL per image) by taking infinitesimally short interpolation paths in the immediate vicinity of a specific image into account. In one embodiment, the neural synthesis network 140 can be trained on the basis of the PPL metric in order to favor smooth transformations and to achieve an improvement in the quality of the output data generated.

7A represents images generated by the style-based generator system, according to one embodiment 100 and have high PPL values (long interpolation paths). The six images each feature a cat, but in each image the shape of the cat is distorted and / or discontinuous, sometimes blended with other elements of the scene. In other words, the six images in 7A there is a lack of semantic consistency. The one for each of the images in 7A The calculated PPL is in the top 90th percentile of a set of random sample images generated by the style-based generator system 100 were generated.

7B FIG. 12 shows images created with the style-based generator system, according to one embodiment 100 and have low PPL (short interpolation paths) values. The six pictures in 7B also each have a cat, and each cat appears mostly as expected. None of the cats appear discontinuous or blended with other elements of the scene. In other words, the six images in 7B are generally semantically consistent. The one for each of the images in 7B The calculated PPL is in the lower 10 percentiles of a set of random sample images generated by the style-based generator system 100 were generated.

As from comparing the quality of the pictures in the 7A and 7B As can be seen, there is a correlation between the perceived image quality and the calculated PPL metric. The PPL metric is calculated by measuring the average learned perceptual image patch similarity (LPIPS) distances between generated images under small perturbations in the calculated latent space. A lower PPL (smoother generator mapping) value appears to correlate with higher overall image quality, while other metrics have no correlation with image quality.

In practice, the application of a smoothing regularization between an intermediate latent code and the output image is very advantageous for the quality of the images generated. The advantage of smoothing regularization can be achieved by comparing the distribution of the PPL values per image for the style-based generator system 100 that was trained with and without smoothing regularization.

7C shows a graph of the PPL values 700 for a set of images according to one embodiment. The images are created by changing the entered latent code or intermediate latent code by a small amount to create each new image. In other words, a small movement in the latent space should correspond to a small change in the output data. The set of images can be those in the 7A and 7B have shown images. PPL values in one range 705 of the distribution are low PPL values that correspond to high quality images that match the in 7B Include images shown. PPL values in one range 710 of the distribution are high PPL values that correspond to low quality images that match the in 7A Include images shown. A picture in the field 710 has a large and / or discontinuous change relative to the change in the input latent code that produced the image. In other words, in response to a small change in the latent code, the image changes more abruptly than smoothly from the previous image. The images are generated by the style-based generator system 100 is generated using various random starting values as input and the path lengths are calculated based on the path endpoints in W.

7D shows a distribution of the PPL values 720 for a set of images generated using smoothing regularization according to one embodiment. In one embodiment, the smoothing regularization process is performed on the style-based generator system during training 100 applied. The neural mapping network 110 receives an input latent code and generates an intermediate latent code. The neural synthesis network 140 in turn, receives the intermediate latent code and produces an output image. In particular, smoothing regularization is applied between the intermediate latent code and the output image to ensure that the transformation between the intermediate latent code and the output image is smooth. In one embodiment, smoothing regularization is applied between one of the style signals and the output data. The average FID that will be calculated for the images that reflect the distribution of the PPL values 700 and 720 are assigned are the same. However, the average PPL scores are very different and the end of the distribution within the range 710 the distribution of the PPL values 700 is missing in the distribution of the PPL values 720 .

It is not immediately obvious why a low PPL value should correlate with image quality. Because the discriminator 275 Punished broken images during training, it is perhaps the most direct path for the style-based generator system 100 to improve by effectively expanding the area of latent space that provides good images. As a result, the poor quality images can be pressed into small areas of latent space with rapid changes. While expanding and compressing different areas of latent space can improve the average output quality in the short term, the accumulated distortions affect the training dynamics and thus the final image quality. The empirical correlation between low PPL and increased output quality suggests that promoting smooth generator mapping by promoting low PPL during exercise can improve image quality.

Excessive path distortion in the style-based generator system 100 show up as poor local conditioning: every small area in the latent space W becomes when the style-based generator system is trained 100 arbitrarily pressed and stretched. A generator mapping from the latent space to the image space is considered to be well conditioned if at any point in the latent space small shifts lead to equally large changes in the image space, regardless of the direction of the disturbance. The small shifts can be applied in the latent space to produce corresponding changes of the same order of magnitude in the image space, or the small shifts can be applied in the image space to produce corresponding changes of the same order of magnitude in the latent space.

This in 2 B illustrated neural synthesis network 140 assigns a sequence of processing blocks 200 and 230 which each generate intermediate data (activations). A regularization can be applied to the neural synthesis network 140 applied to ensure that the transformation between the intermediate latent code and the output image is smooth. Alternatively, the regularization can also be between any two corresponding points in the neural synthesis network 140 be applied. For example, the relationship between the outputs (e.g., output features or modified features) of each pair of the style blocks 645 be regularized. The last style block 645 generates the output image. Alternatively, the regularization can also be applied to other neural generator network architectures. In one embodiment, the regularization, in order to adjust the computational effort of the regularization, can be carried out less frequently without the effectiveness suffering as a result.

7E shows, according to one embodiment, a conceptual illustration of paths without regularization. A linear trajectory of a style signal 725 is generated by repeatedly changing the style signal to the same extent. In response, the neural synthesis network generates 140 a curved trajectory of the output data 730 . The curved trajectory results from changes in the output data that have a different size compared to the changes in the style signal of the same size.

7F shows, according to an embodiment, a conceptual illustration of paths with regularization. The mapping network 110 is trained to map a latent code into a latent space and generate a style signal. In the 7F Figure shown is more uniform than that in 7E shown illustration. A linear trajectory of a style signal 735 is created by repeatedly changing the style signal in the same size. In response, the neural synthesis network generates 140 an almost linear trajectory of the output data 740 with changes of the same order of magnitude. A change in the style signal with a first size produces a corresponding change in the output data with a similar size.

8A Figure 3 shows a block diagram of a neural synthesis network 840 that is done using the style block 645 the end 6D for use in implementing some embodiments of the present disclosure. The neural synthesis network 840 assigns a sequence of style blocks 645 and a style block 646 on. Compared to the style blocks 645 is in the style block 646 the demodulation unit 640 omitted because it is the last style block in the neural synthesis network 840 acts. The first style block 645 receives the first intermediate data, the first style signal, and at least a portion of the weights in a set of weights. The weights are processed in accordance with the first style signal and applied to the first intermediate data to generate second intermediate data. The second intermediate data has a content encoded in the first intermediate data that is changed based on the first style signal. Spatial noise and / or a bias (not shown) can be inserted into the second intermediate data.

The second style block 645 receives the second intermediate data, the second style signal, and at least a further portion of the weights in the set of weights. The weights are processed according to the second style signal and applied to the second intermediate data to generate third intermediate data. The third intermediate data has a content encoded in the first intermediate data that is changed based on the first style signal and the second style signal. In one embodiment, the first and second style signals operate on different scales. Therefore, the style changes resulting from the first style signal are retained in the third intermediate data. In other designs, one or more additional style blocks 645 between the second style block 645 and the style block 646 to be available. In another embodiment, the second style block is 645 omitted and the style block 646 receives the second intermediate data. Spatial noise and / or a bias (not shown) can be inserted into the third intermediate data.

The style block 646 receives the third intermediate data, the third style signal, and at least another portion of the weights in the set of weights. The weights are processed according to the third style signal and applied to the third intermediate data to generate fourth intermediate data. The fourth intermediate data has a content encoded in the first intermediate data that is changed based on the first style signal, the second style signal, and the third style signal. In one embodiment, the first, second and third style signals operate on different scales. Therefore, the style modifications resulting from the first and second style signals are retained in the fourth intermediate data. Spatial noise and / or a bias (not shown) can be inserted into the fourth intermediate data in order to generate the output data.

8B Figure 3 shows a block diagram of a generator training system 800 , according to one embodiment. The generator training system 800 can be implemented by a program, a custom circuit, or a combination of a custom circuit and a program. For example, the generator training system 800 with a GPU, a CPU, or any Processor are implemented that can perform the operations described here. In addition, those skilled in the art will understand that any system capable of performing the operations of the generator training system 800 is within the scope and spirit of embodiments of the present invention.

The generator training system 800 exhibits a neural generator network 820 such as B. the style-based generator system 100 with the neural synthesis network 140 or 840 , and a training loss session 810 on. The neural generator network 820 receives input data (e.g. at least one latent code and / or noise inputs) and generates output data. Depending on the task, the output data can be an image, audio, video or other types of data (configuration setting). The neural generator network 820 may use a training data set that has exemplary output data that is used by the neural generator network 820 generated output data should be consistent. The neural generator network 820 generates output data in response to the input data, and the training loss session 810 determines whether the output data is similar to the exemplary output data included in the training data. Based on the determination, a set of parameters of the neural generator network 820 customized.

When the regularization is done, the training loss session is 810 designed so that they have two points or aspects within the neural generator network 820 identifies and applies a first modification to a first point or aspect and calculates a second modification for a second point or aspect. The second modification is consistent with the first modification and a regularization loss is calculated based on the second modification. The parameter set used by the neural generator network is then updated to reduce the loss of regularization. In one embodiment, the two points can comprise intermediate data and the output data. In one embodiment, the two points can comprise a style signal and either intermediate data or the output data.

In one embodiment, a finite difference method is used to compute the second modification by applying the first modification to the style signal or intermediate data at the first point and the first modified style signal or intermediate data through subsequent layers of the neural generator network 820 processed to produce the second modification in the intermediate data or the output data at the second point. In one embodiment, the first modification is a small random variable.

In one embodiment, a differentiation method is used in the forward mode to calculate the second modification by differentiating the intermediate data or output data at the second point with respect to a linear trajectory of the style signal or the intermediate data at the first point, the linear transformation being represented by the first modification is defined. In one embodiment, the linear trajectory is chosen at random.

Another smoothing regularization method changes the output data instead of modifying the intermediate data or a style signal and modifying the modification forward through the neural generator network 820 to propagate. In one embodiment, the differentiation method is used in reverse mode to compute the second modification by swapping the first and second points so that the first point is downstream (with respect to the processing in the neural generator network 820 ) is relative to the second point. For the differentiation in the reverse mode, an inner product (e.g. point product) between the intermediate data or output data at the first point and the first modification is calculated, and the inner product is differentiated with respect to the style signal or the intermediate data at the second point, to calculate the second modification. In one embodiment, e.g. B. a gradient vector can be calculated as a point product between the output data and a random vector which is differentiated with respect to the intermediate data or the style signal.

The regularization is performed by repeatedly calculating the second modification when the neural synthesis network 840 is trained, and then deviations in the sizes of the second modification compared to a reference value are penalized. The set of parameters (e.g. weights) is adjusted based on the deviations in order to increase the uniformity of the sizes. The first point may correspond to the first style signal, the second style signal, the third style signal, the first intermediate data, the second intermediate data or the third intermediate data. The second point may correspond to the second intermediate data, the third intermediate data, the fourth intermediate data, or the output data. When differentiation is used in reverse mode, the correspondences of the first point are swapped with those of the second point.

8C shows a flow diagram of a method 825 for smoothing regularization for use in a neural generator network according to one embodiment. Each block of the procedure 825 as described herein includes a computing process that can be performed with any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be carried out as computer-usable instructions stored on computer storage media. The method can be provided by a standalone application, service, or hosted service (standalone or in combination with another hosted service) or a plug-in for another product, to name a few. In addition, the procedure 825 exemplary in relation to the neural generator network 820 the 8B described. However, this method can additionally or alternatively be performed by any system or combination of systems including, but not limited to, the systems described herein. In addition, those skilled in the art understand that any system that incorporates the method 825 performs falls within the scope and spirit of the embodiments of the present disclosure.

In step 830 For example, output data are generated by a neural generator network based on a set of parameters, the neural generator network comprising one or more layers, each of which outputs features to a subsequent layer of the one or more layers. In one embodiment, the neural generator network comprises one or more layers, each with a first layer 120 , a second layer 130 , Processing blocks 200 or 230 and / or style blocks 600 , 640 , 645 or 646 include. In one embodiment, the neural generator network has the neural synthesis network 140 or 840 on. In one embodiment, generating the output data comprises calculating the second features before calculating the first features. In one embodiment, the first features are the output data and the second features are a style signal or intermediate data, so that the regularization is carried out using the differentiation in the reverse mode.

In step 835 first features associated with a first layer of the one or more layers and second features associated with a second layer of the one or more layers are identified. In one embodiment, the first features are one of a style signal, intermediate data, and output data. In one embodiment, the second features are one of a style signal, intermediate data, and output data.

In step 845 a first modification with respect to the first features is selected. In one embodiment, the first modification brings about a change in the first features. In one embodiment, the selection of the first modification comprises randomly selecting each component of the first modification to generate a Gaussian distribution of the first modifications.

In step 850 a second modification is calculated with respect to the second features, the second modification being consistent with the first modification. In one embodiment, calculating the second modification comprises modifying the first features (e.g. style signal or intermediate data) according to the first modification to obtain modified first features, recalculating the second features based on the modified first features to obtain modified second ones To obtain features (e.g., intermediate downstream data or the output data), and calculating the second modification as the difference between the second features and the modified second features. In one embodiment, the recalculation of the second features comprises processing the modified first features by one or more layers of the neural generator network in order to generate the modified second features.

If the backward mode differential is used to perform the regularization, the second modification can be calculated by calculating an inner product between the first features (e.g., intermediate downstream data or the output data) and the first modification and then calculating the inner product is differentiated with respect to the second modification.

In step 855 a regularization loss is calculated based on the second modification. In one embodiment, the loss of regularization is from the loss of training session 810 calculated. In one embodiment, the regularization loss comprises a magnitude of the second modification. In one embodiment, the regularization ^{loss comprises an L 2} norm function. In one embodiment, the regularization loss is calculated by comparing a variable of the second modification with a reference value. In one embodiment, the reference value is a constant. In another embodiment the reference value is calculated as the mean value of the variables over several versions of the neural generator network.

In step 860 the set of parameters is updated to reduce the loss of regularization. In one embodiment, the set of parameters are weights, with different parts of the weights being applied to the activations by the different layers in the neural generator network. In one embodiment, updating the parameter set brings the size closer to the reference value.

durch die Jacobimatrix J_w = ∂g(w)/ ∂w erfasst. Bei dem Punkt in dem latenten Zwischenraum kann es sich um die ersten Merkmale handeln, die in Schritt 840 identifiziert wurden, und er kann ein Stilsignal oder Zwischendaten umfassen, die einer Schicht des Generators zugeordnet sind. Motiviert durch den Wunsch, die erwarteten Längen von Vektoren unabhängig von der Richtung beizubehalten, kann eine Regularisierungsfunktion zur Durchführung der Differenzierung im Rückwärtsmodus formuliert werden als $${\mathbb{E}}_{w,y~\mathcal{N}\left(\mathrm{0,1}\right)}{\left({\left|{\mathbf{J}}_w^T\mathbf{y}\right|}_2-a\right)}^2,$$

wobei a ein Referenzwert ist, y Modifikationen wie Zufallsbilder mit normalverteilten Pixelintensitäten sind, und w ~ f(z) gilt, wobei die latenten Vektoren in dem latenten Raum z normalverteilt sind. Die Zufallsbilder können von dem stilbasierten Generatorsystem 100 unter Verwendung eines latenten Zufallscodes erzeugt werden, um ein Ausgangsbild bzw. initiales Bild zu erzeugen. Ein Satz von Bildern mit einem zufälligen Gaußschen Rauschen pro Pixel kann verwendet werden, um einen Satz von Modifikationen bezüglich des Ausgangsbildes zu definieren. Die Rauschbilder müssen nicht verwendet werden, um das Ausgangsbild tatsächlich zu verändern.At a single point in the latent space, w ∈ W, the generator _map's local metric scale properties become g ( _W ): $\mathcal{W.}\mapsto \mathcal{Y}$

recorded by the _{Jacobian matrix J w} = ∂g (w) / ∂w. The point in the latent space can be the first features shown in step 840 and it may include a style signal or intermediate data associated with a layer of the generator. Motivated by the desire to keep the expected lengths of vectors regardless of direction, a regularization function for performing the differentiation in the reverse mode can be formulated as $${\mathbb{E.}}_{w,y~\mathcal{N}\left(0.1\right)}{\left({\left|{\mathbf{J}}_w^T\mathbf{y}\right|}_2-a\right)}^2,$$

where a is a reference value, y are modifications such as random images with normally distributed pixel intensities, and w ~ f (z) holds true, with the latent vectors in the latent space z being normally distributed. The random images can be generated by the style-based generator system 100 can be generated using a latent random code to generate an initial image. A set of images with random Gaussian noise per pixel can be used to define a set of modifications to the output image. The noise images do not have to be used to actually change the original image.

verwendet werden, die mit Hilfe einer standardisierten Rückwärtspropagation effizient berechenbar ist. ${\mathbf{J}}_w^T\mathbf{y}$

umfasst eine zweite Modifikation, die einer Modifikation des Stilsignals oder der Zwischendaten als Ergebnis des Hinzufügens von Zufallsrauschen y zu dem Ausgangsbild g(w) entspricht. In einer Ausführungsform ist a eine Konstante, die während der Optimierung dynamisch als langlaufender exponentieller gleitender Mittelwert der Längen einer Größe der zweiten Modifikation ${\left|{\mathbf{J}}_w^T\mathbf{y}\right|}_2,$

eingestellt wird, so dass die Optimierung selbständig eine geeignete globale Skala findet. In einer Ausführungsform werden die zweiten Modifikationen, Produkte ${\mathbf{J}}_w^T\mathbf{y}$

analytisch berechnet. Gleichung (5) wird minimiert, wenn J_w orthogonal (bis zu einer globalen Skala) bei jedem w ist. Eine orthogonale Matrix bewahrt Längen und führt kein Quetschen entlang einer Dimension ein. Daher ist die orthogonale Matrix einem einheitlichen Raum zugeordnet.In order to avoid the explicit calculation of the Jacobian matrix, the identity ${\mathbf{J}}_w^T\mathbf{y}={\nabla }_w\left(G\left(\mathbf{w}\right)\cdot \mathbf{y}\right),$

can be used, which can be calculated efficiently with the help of standardized backward propagation. ${\mathbf{J}}_w^T\mathbf{y}$

comprises a second modification corresponding to a modification of the style signal or the intermediate data as a result of adding random noise y to the output image g (w). In one embodiment, a is a constant that is dynamically used during optimization as a long-running exponential moving average of the lengths of a quantity of the second modification ${\left|{\mathbf{J}}_w^T\mathbf{y}\right|}_2,$

is set so that the optimization automatically finds a suitable global scale. In one embodiment, the second modifications become products ${\mathbf{J}}_w^T\mathbf{y}$

analytically calculated. Equation (5) is minimized when J _{w is} orthogonal (up to a global scale) at every w. An orthogonal matrix preserves lengths and does not introduce squeezing along a dimension. Therefore, the orthogonal matrix is assigned to a uniform space.

Changes in the output image g (w) resulting from the modification (e.g. the noise images) are calculated as the dot product between the image and each noise image y in the set of noise images, g (w). y. Assuming that the output image is modified on the basis of each noise image, the dot product indicates how fast the output image moves in the direction of the noise image. The noise image defines changes in the image space. A direction of each change can be determined by calculating a differential gradient (∇ _w ) of the dot product with respect to the intermediate latent code w used to generate the style signals. The direction indicates the direction in which to move in the latent space in order to maximize the changes in the modified image. The length (size) of the difference gradient indicates the rate of change in the image with respect to the noise image for corresponding changes in the latent space (e.g. second modifications) which lie in the direction of the difference gradient.

As shown in equation (5), an amount of the second modification (a difference gradient length) is compared with the reference length (a), and the square of the difference is a regularization allowance for a particular choice of w and y calculated during training . The difference gradients should converge to an equal length, indicating that the transformation between the Clearing is more even. During training, variations in the difference gradient lengths can be penalized by a loss function in order to perform the smoothing regularization. If the regularization procedure for multiple choices of w and y during training of the style-based generator system 100 or the neural generator network 820 is repeated, Equation (5) computes the average of the regularization charge over the multiple choices that is minimized.

Since the calculation of the differential gradient ∇ _w (g (w) · y) is somewhat complex, a general optimization can be used that applies to all regularization methods. Typically, the main loss function (e.g. logistic loss) and regularization terms (e.g. R ₁ ) are written as a single expression and thus optimized at the same time. However, the regularization terms can be computed much less frequently than the main loss function, which significantly reduces their computational costs and overall memory consumption. For example, the R ₁ regularization can only be performed once every 16 minibatches without incurring any disadvantage. In addition, the less frequent calculation of the regularization terms can also improve training performance. In one embodiment, the reduced frequency training for a neural generator network implementing weight demodulation is performed at 61 frames / sec. compared to 37 frames / sec. carried out 40% faster without weight demodulation and regularization.

Image quality problems such as B. the in 6A artifacts shown 601 , 603 and 605 , have been identified, and restructuring the neural synthesis network to implement weight demodulation improves quality. Weight demodulation can also be applied to other neural generator networks. In addition, the regularization of the smoothing between the intermediate data output from two different layers (or levels) or between the intermediate data and the output image reduces image artifacts, e.g. B. the artifacts shown in Figure 7A. The regularization method can be used in combination with weight demodulation or separately to improve the performance of a neural generator network.

It is pointed out that the methods described here can be embodied in executable commands that are stored in a computer-readable medium for use by or in connection with a machine, a system, a device or a device, each configured for processor-based command execution. Those skilled in the art will recognize that there may be various types of computer readable media for storing data in some embodiments. As described here, a “computer-readable medium” has one or more suitable media for storing the executable instructions of a computer program, so that the machine, the system, the device or the device, which are each designed for processor-based instruction execution, carry the instructions read (or retrieve) from the computer readable medium and execute the instructions to carry out the described embodiments. Suitable storage formats have one or more of the following formats: electronic, magnetic, optical and electromagnetic. A non-exhaustive list of conventional, exemplary computer readable media includes: a portable computer disk, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), flash memory device, and optical storage devices including a portable compact disc ( CD), a portable digital video disc (DVD), and the like.

The arrangement of the components shown in the accompanying figures is for illustration purposes only and other arrangements are also possible. For example, one or more of the elements described here can be implemented in whole or in part as electronic hardware components. Other elements can be implemented in software, hardware, or a combination of software and hardware. In addition, some or all of these other elements can be combined, some can be omitted entirely, and additional components can be added while still achieving the functionality described herein. Thus, the subject matter described herein can be embodied in many different forms, and all such variations come within the scope of the claims.

To facilitate understanding of the subject matter described here, many aspects are described in the form of courses of action. Those skilled in the art will recognize that the various actions can be performed by specialized circuitry or circuitry, by program instructions executed by one or more processors, or by a combination of both. The sequence of actions described here is not intended to mean that the specific sequence described must be followed in order to carry out this sequence. All of the procedures described herein can be performed in any suitable order unless otherwise noted herein or unless the context clearly indicates otherwise.

The use of the terms “a”, “an” and “an” as well as “the”, “the” and “that” and similar references in connection with the description of the subject matter (in particular in connection with the following claims) is to be interpreted as that they include both the singular and the plural, unless otherwise stated here or the context clearly contradicts it. The use of the term “at least one” followed by a list of one or more items (e.g. “at least one of A and B”) is to be interpreted in such a way that it is an item selected from the listed items (A or B) or any combination of two or more of the listed items (A and B) is meant, unless otherwise stated here or clearly refuted by the context. Furthermore, the foregoing description is intended to be illustrative and not restrictive, as the intended scope of protection is defined by the claims listed below, along with their equivalents. The use of examples or exemplary expressions (e.g. "such as") is only used to better illustrate the subject matter and does not represent a limitation of the scope of the subject matter unless otherwise claimed. The use of the term “based on” or “dependent on” and other similar expressions that indicate a condition for producing a result, in both the claims and the written description, is not intended to exclude any other condition that could produce that result. Nothing in the specification should be construed as indicating any unclaimed element as essential to the practice of the claimed invention.

Claims (20)

A computer implemented method comprising: Receiving first weights for modifying first features in a first stage of a neural generator network having a plurality of stages, a last stage of the neural generator network generating output data; Modulating the first weights with a first style control signal to produce first modulated weights; Demodulating the first modulated weights to produce first normalized weights; Applying the first normalized weights by the first stage to generate modified first features; and Processing the modified first features by at least one additional stage of the neural generator network in order to generate the output data which have a content corresponding to the first features. Computer-implemented method according to Claim 1 wherein the first weights include first values associated with a first implementation of the neural generator network for processing a first input pattern, and second values associated with a second implementation of the neural generator network for processing a second input pattern. Computer-implemented method according to Claim 1 or 2 wherein the demodulating comprises calculating an expected standard deviation of the modified first features that is used to demodulate the first set of weights. The computer-implemented method of any preceding claim, wherein applying comprises convolving the first normalized weights with the first features to produce the modified first features. A computer-implemented method according to any one of the preceding claims, further comprising: Mapping a latent code in an input space onto a space having linear subspaces to generate an intermediate latent code, the linear subspaces each corresponding to a different attribute of the content; and Applying an affine transform to the intermediate latent code to generate the first style signal. The computer-implemented method of any preceding claim, wherein the first style control signal controls at least a first attribute of the content. The computer-implemented method of any preceding claim, further comprising: receiving second weights for modifying second features in the additional stage of the neural generator network; Modulating the second weights with a second style control signal to produce second modulated weights; Demodulating the second modulated weights to produce second normalized weights; and applying, by the additional stage, the second normalized weights to the first modified features to produce modified second features. Computer-implemented method according to Claim 7 wherein the first style control signal modifies the first features to change the content in a first scale and the second style control signal modifies the first modified features to change the content in a second scale. Computer-implemented method according to Claim 7 or 8th which further comprises inserting noise into the first modified features prior to the additional stage applying the second normalized weights. The computer-implemented method of any preceding claim further comprising: Receiving second weights for modifying second features in the last stage of the neural generator network; Modulating the second weights with a second style control signal to produce second modulated weights; Applying the second modulated weights, by the final stage, to the first modified features to produce the output data. Computer-implemented method according to one of the preceding claims, wherein the steps of receiving, modulating, demodulating, applying and processing are performed within a cloud computing environment. The computer-implemented method of any preceding claim, wherein the steps of receiving, modulating, demodulating, applying and processing are performed on a server or in a data center to generate an image, and wherein the image is provided to a device of the user is streamed. A computer-implemented method according to any preceding claim, wherein the steps of receiving, modulating, demodulating, applying and processing are performed to generate an image that is used to train, test or certify a neural network that is in a Machine, a robot or an autonomous vehicle is used. System comprehensive: a processor configured to implement a neural generator network having multiple stages, a final stage of the neural generator network generating output data and the neural generator network being configured to: receive first weights for modifying first features in a first stage of the neural generator network; modulating the first weights with a first style control signal to produce first modulated weights; demodulate the first modulated weights to produce first normalized weights; apply the first normalized weights by the first stage to generate modified first features; and to process the modified first features by at least one additional stage of the neural generator network in order to generate the output data which have a content corresponding to the first features. System according to Claim 14 wherein the first weights include first values associated with a first implementation of the neural generator network for processing a first input pattern, and second values associated with a second implementation of the neural generator network for processing a second input pattern. System according to Claim 14 or 15th wherein the demodulating comprises calculating an expected standard deviation of the modified first features that is used to demodulate the first set of weights. System according to one of the Claims 14 until 16 wherein applying comprises convolving the first normalized weights with the first features to produce the modified first features. System according to one of the Claims 14 until 17th wherein the neural generator network is further configured to: map a latent code in an input space onto a space having linear subspaces to generate an intermediate latent code, the linear subspaces each corresponding to a different attribute of the content; and to apply an affine transform to the intermediate latent code to generate the first style signal. System according to one of the Claims 14 until 18th wherein the first style control signal controls at least a first attribute of the content. Non-transitory computer readable medium that stores computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of a method according to any one of the Claims 1 until 13th perform.

Images