DE102019112595A1 - GUIDED HALLUCATION FOR MISSING PICTURE CONTENT USING A NEURONAL NETWORK - Google Patents

Abstract

A missing image content is generated with a neural network. In one embodiment, a high-resolution image and an associated high-resolution semantic label map are generated from a low-resolution image and an associated low-resolution semantic label map. The input pair of image / map (low-resolution image and associated low-resolution semantic label map) lacks details and thus content. Rather than simply improving the image / map input pair, data missing from the image / map input pair is improvised or hallucinated through a neural network, producing plausible content while maintaining space-time consistency. The missing content is hallucinated to produce a detailed enlarged section of an image. The missing content is hallucinated to produce various variations of an image, such as different seasons or weather conditions for a driving video.

Description

TECHNICAL PART

The present disclosure relates to image generation, and more particularly to generation of missing image content using a neural network.

BACKGROUND

One way of generating image content is to manually create images that have some special effect when viewed in a sequence. A sequence of images in which each image recursively appears in itself corresponds to the "Droste effect". In other words, a smaller version of the image is present in an original image, so that when a camera is zooming into the original image, the smaller version repeatedly becomes a copy of the original image. Such a sequence of images is generated manually because the limited resolution of the original image does not provide the necessary data to automatically generate an increasingly higher resolution version of the smaller image as the camera zooms in, until the higher resolution version of the smaller image replicates the image Original picture will. The manual generation of image sequences is slow and requires experienced professionals. There is a need to address these problems and / or other problems associated with the prior art.

SUMMARY

Missing image content is generated via a neural network. In one embodiment, a complete image and associated complete semantic label map are generated from an incomplete image and an associated incomplete semantic label map. The incomplete image / semantic label card pair has missing data compared to the full image / semantic label card pair. In one embodiment, the input image / map pair is a low resolution, low resolution image and a low resolution, low resolution, semantic label map, with details missing and therefore missing data or content. Rather than simply enhancing the incomplete image / map pair (e.g., by upsampling), the missing data is improvised or hallucinated through a neural network, producing plausible content while maintaining spatial-temporal consistency. In one embodiment, the missing data is hallucinated to produce a detailed, enlarged section of an image. In one embodiment, the missing data is hallucinated to produce various variations of an image, such as different seasons or weather conditions for a driving video.

A method, a computer readable medium, and a system are disclosed to use a model of a neural network to obtain a first image lacking a portion of the image data and a first semantic map corresponding to the first image and lacking a portion of the semantic data , to recieve. The model of the neural network processes the first image and the first semantic map to produce a second image of hallucinated image data representing a complete version of the first image.

list of figures

• 1A FIG. 4 illustrates a conceptual representation of a low resolution image and a low resolution semantic map and corresponding hallucinated high resolution image and a high resolution semantic map according to one embodiment. FIG.
• 1B FIG. 12 illustrates a block diagram of a system of a neural network for missing content hallucination for images and semantic label cards according to an embodiment. FIG.
• 1C FIG. 10 illustrates a flowchart of a method of hallucinating complete images according to one embodiment. FIG.
• 1D FIG. 12 illustrates a flowchart of a method of hallucinating complete images and semantic label cards according to one embodiment.
• 2A FIG. 12 is a block diagram of a training configuration for the neural network system of FIG 1B according to an embodiment.
• 2 B FIG. 3 illustrates a flowchart of a method of training the neural network system 1B according to an embodiment.
• 3 illustrates a parallel processing unit according to an embodiment.
• 4A provides a general processing cluster within the parallel processing unit of 3 according to an embodiment.
• 4B represents a storage partition unit of the parallel processing unit of 3 according to an embodiment.
• 5A turns off the streaming multiprocessor 4A according to an embodiment.
• 5B FIG. 12 is a conceptual diagram of a processing system using the PPU of FIG 3 has been implemented according to one embodiment.
• 5C FIG. 3 illustrates an example system in which the different architectures and / or functionalities of the various previous embodiments may be implemented. FIG.
• 6 FIG. 3 is a conceptual illustration of a graphics processing pipeline used by the PPU of FIG 3 has been implemented according to one embodiment.

DETAILED DESCRIPTION

Conventional imaging to produce an infinite zoom, such as the Droste effect, requires very high resolution images to continuously zoom in on an image. In reality, zooming can not produce high-resolution zoomed images by applying a high magnification ratio to a single high-resolution but limited-resolution image. Therefore, the zoom problem can be viewed as a hallucination problem with minimal semantic information from which the missing image data (e.g., resolution) can be improvised or hallucinated. A neural network may learn to generate the missing data for an incomplete input image based on the incomplete input image and semantic information corresponding to the incomplete input image. In one embodiment, a generative adversary network (GAN) is used to train the neural network to generate missing data for incomplete input images.

1A represents a conceptual representation of an incomplete picture 101 and an incomplete semantic map 103 and a corresponding complete picture 102 and a complete semantic map 104 according to one embodiment. In the incomplete image 101 and the incomplete semantic map 103 missing data or some data are hardly visible due to the limited pixel resolution. The incomplete picture 101 is a low-quality image represented by color image data for each pixel (shown as a dotted grayscale image instead of an RGB color image). In one embodiment, the incomplete image 101 a low-resolution image. The complete picture 102 is a hallucinated high-resolution image represented by color image data for each pixel (shown as a dotted grayscale image instead of an RGB color image), where the pixel resolution of the complete image 102 is greater than the pixel resolution of the incomplete image 101 ,

The incomplete semantic map 103 is a low-quality semantic map represented by color data for each pixel (represented as a grayscale image rather than an RGB color image), each color defining a label representing another object, object instance, and / or object class. In one embodiment, the incomplete semantic map is 103 a low-resolution semantic map. In one embodiment, the incomplete image includes 101 sparse data, damaged data and / or noisy data. The complete semantic map 104 is a hallucinated high-resolution semantic map represented by color data for each pixel (shown as a grayscale image instead of an RGB color image), where the pixel resolution of the complete semantic map 104 is greater than the pixel resolution of the incomplete semantic map 103 ,

To create plausible, accurate and / or realistic hallucinated image data, the hallucinated image data should retain the context (e.g., colors, shapes, etc.) that matches the incomplete input image (s). Semantic maps, even incomplete semantic maps, can help ensure that the context is consistent in creating the hallucinated image data. In addition, the hallucinated image data should coincide in time with the incomplete image data. Providing a sequence of incomplete images for the neural network model, rather than a single incomplete image, can improve temporal consistency. A GAN can be used in training a neural network model to ensure that the neural network model generates plausible content.

1B represents a block diagram of a system 100 of a neural network for hallucinating missing content for images and semantic label cards according to an embodiment. The system 100 of a neural network corresponds to a model of a neural network comprising two neural networks 115 and 105 an encoder, a residual function 110 and two neural networks 125 and 135 a decoder includes. The neural network 115 the coder processes the incomplete semantic map S _I and the neural network 105 the encoder processes the corresponding incomplete image I _i , In one embodiment, the neural network is eliminated 125 the decoder and the system 100 a neural network 100 gives complete pictures I _C without issuing complete semantic maps Sc.

The neural network 115 the coder processes the incomplete semantic map S _I according to the shift parameters (eg weights) determined during exercise. The neural network 105 The encoder processes the incomplete image Ii according to the slice parameters determined during exercise. The of the neural networks 105 and 115 The features output by the encoder are linked and from the residual function 110 processed according to the shift parameters determined during the training. The two neural networks 125 and 135 The decoder also handles those from the remainder function 110 output data according to the shift parameters determined during the training to the complete image ic and the complete semantic map sc to create. In one embodiment, the incomplete image is a low-resolution image, the incomplete semantic map is a low-resolution semantic map, the full image is a high-resolution image. The full image contains hallucinated image data that represents a complete (eg, high-resolution) version of the incomplete image. Similarly, the complete semantic map contains hallucinated semantic data that represents a complete (eg, high-resolution) version of the semantic map for the complete image.

Although the system 100 of the neural network in the context of processing units may include one or more of the units, neural networks 105 and 115 of an encoder, neural networks 125 and 135 a decoder and the residual function 110 be implemented as a program, custom circuit, or through a combination of custom circuitry and a program. For example, the neural networks 105 and or 115 an encoder can be implemented by a GPU (graphics processor unit), a CPU (central processing unit) or any processor capable of implementing layers of a neural network. In addition, the expert understands that any system that performs the functions of the system 100 of a neural network is within the scope and spirit of embodiments of the present disclosure.

Additional illustrative information will now be given on various optional architectures and functions that can be used to implement the aforementioned framework as desired by the user. It is to be expressly understood that the following is illustrative and should not be construed as limiting. One of the following features may optionally be included with or without the exclusion of other features described.

The hallucination task can be under the assumption of an incomplete picture I _i be modeled in which the system 100 of the neural network learns a function F, the I _i into a corresponding complete picture Ic can map. If the incomplete image and the semantic map have a low resolution, the corresponding full image is a high resolution image. Conventional super-resolution algorithms can not realize the generation of high-resolution images in which the difference in resolution between I _i and IC 20x or more. Unlike conventional super-resolution algorithms, the system can 100 of the neural network, however, the high resolution output image I _C hallucinate. In addition, the system can 100 of the neural network, when a low-resolution semantic card S _I (or semantic segmentation mask) for I _i In addition to the hallucination of the high-resolution output image Ic, it is also possible to hallucinate a high-resolution semantic map. In one embodiment, the low-resolution semantic map S _I be replaced by a high-resolution semantic map. If a corresponding semantic map is available, the system can 100 of the neural network learn the assignment Ic, Sc = F (I _I , S _I ), where Si and Sc is the low-resolution or high-resolution semantic map.

Considering the hallucination task as a problem of image translation, a key challenge is to learn joint distribution of the input data in different domains, namely the image domain and the domain of the semantic maps. It is assumed that an incomplete input pair ( I _i . S _I ) to a same latent code z in the common latent space 140 can be displayed. Data of a semantic card corresponds to a first domain 101 and image data of a second domain 102 , E ₁ and E ₂ are two coding functions, the incomplete images or semantic maps on latent codes in the common latent space 140 depict. The neural network 115 an encoder and the neural network 105 an encoder perform the coding functions E ₁ and E ₂ out. G ₁ and G ₂ are two decoding functions that convert the latent codes to the full images (Ic) in the first domain 101 and the complete sematic maps (Sc) in the second domain 102 depict. The neural network 125 a decoder and the neural network 135 a decoder perform the decoding function G ₁ respectively. G ₂ out. The system 100 of the neural network may be monitored (using ground truth training data) and / or not supervised (using conditional discriminators).

A common residual function R processes the latent codes in the common latent space. In one embodiment, the residual function is 110 implemented as a neural network. In one embodiment, a first portion of the neural network that is the residual function 110 first trained on incomplete images and semantic maps before a second section of the neural network is combined with the first section. The combination of the first and second sections of the neural network is then trained to complete images before entering the system 100 of the neural network is integrated. In one embodiment, the first portion is a coarse global generator network and the second portion is a fine local area network. In one embodiment, the first portion includes a convolution frontend, a set of remainder blocks, and a transposed convolutional backend. The first section can receive an input with a resolution of 1024x512 and produce output data with the same resolution. In one embodiment, the second section includes a convolution frontend, a set of remainder blocks, and a transposed convolutional backend. The second section can receive an input with a resolution of 2048 × 1024 and produce output data with the same resolution. An input to the set of residual blocks within the second section is an elemental sum of two feature maps, namely an output feature map generated by the convolution front end of the second section, and a last feature map represented by the transposed convolutional backend of the second first section of the neural network is generated. In one embodiment, the first portion is inserted within the second portion after the convolution frontend and before the set of remainder blocks. In one embodiment, an input is for the first section 2x sub-sampled and provided as input to the first section.

In addition to or instead of generating high resolution images from low resolution images, the neural network system may also be configured to hallucinate image content for portions of the incomplete input image for which no image data is available. For example, if a camera is panning, portions of the scene that are outside of a field of view associated with a first frame will not be captured in the first frame. The missing scene sections in the first frame are captured in a later frame. In one embodiment, the model of the neural network generates the image content missing in the first image frame to hallucinate a complete later image frame in a sequence. In another example, a portion of a scene may be missing because it is obscured by an object, and the portion of the scene may be hallucinated to produce a complete image.

Finally, the neural network system can hallucinate image content to produce a plurality of complete images for a single input image or for each individual image frame in a sequence of incomplete images. In summary, the system can 100 of the neural network to hallucinate an infinite number of complete images from one or more incomplete images and corresponding incomplete semantic maps. The technique of infinite images can be used to produce high-resolution zoomed-in images (over-zoom), missing content due to camera movement (over-translation), and / or infinite content generation (variations of an input image).

Recursive generation and recursive zooming are methods for "generating infinite images" for hallucinating or generating an image sequence from one or more ( I _I . S _I ) Pairs to create a sequence of complete Droste effect images. A first portion of a high resolution image is extracted to produce a low resolution image for the first portion from which, when extracted from the system 100 of the neural network is processed, a high-resolution image is generated that corresponds to the first section. A second portion of a high-resolution semantic map corresponding to the first portion of the high-resolution image is extracted to produce a low-resolution semantic map for the second portion, from which, when processed by the neural network system, produces a high-resolution image which corresponds to the second section.

In one embodiment, the recursive generation zooms into an area of a high-resolution output pair (Ic, Sc), extracts the area by cropping the area, upsampling the cropped area, and generates a blurry input pair (Fig. I _i . S _I ). The blurry Input pair ( I _i , Si) is in the system 100 of the neural network to hallucinate another high-resolution output pair. The method may be repeated to recursively generate a plurality of high-resolution output pairs. The recursive zoom can be used to automatically synthesize the Droste effect, for applications such as those used in virtual reality environments. In particular, motion picture sequences can be generated from a single image (or multiple images) to supplement existing street scene data sets.

1C FIG. 3 illustrates a flowchart of a method 120 for hallucinating high resolution images according to one embodiment. Although the method 120 is described in the context of a processing unit, the method 120 also be performed by a program, a user-defined circuit or by a combination of a user-defined circuit and a program. For example, the procedure 120 from a GPU (graphics processor unit), a CPU (central processing unit), or any processor that can implement a neural network model, such as the system 100 of the neural network. In addition, the skilled person understands that any system that performs the procedure 120 is within the scope and spirit of embodiments of the present disclosure.

At step 130 The model of the neural network receives a first image that lacks part of the image data. At step 140 For example, the model of the neural network receives a first semantic map that corresponds to the first image and lacks some of the semantic data. In one embodiment, the first image is a low-resolution image and the first semantic map is a low-resolution semantic map. The first semantic map is used by the neural network model to constrain and control the hallucination of image contents of output images to produce the missing portion of the image data that is plausible and context dependent.

At step 145 The model of the neural network processes the first image and the first semantic map to produce a second image with hallucinated image data representing a complete version of the first image. In one embodiment, the first image is a low-resolution image that is generated by cropping and upsampling a particular area of a high-resolution image. It is impractical to increase the specific area of the high resolution image that has limited resolution and may not contain enough information to be reconstructed in high resolution with conventional techniques, such as bilinear interpolation upsampling, if the zoom 15x or larger. Therefore, the missing part of the first image can not be reliably generated by conventional high-resolution techniques. In one embodiment, the model of the neural network hallucinates a high-resolution image of the corresponding area, so that the artificial zooming into the corresponding area of the high-resolution image offers a higher resolution than the first image.

In one embodiment, the first image is an incomplete image in which a portion (i.e., 20% of the right portion) of the image is blank because the camera has been panned. In one embodiment, the model of the neural network hallucinates the image content for the blank portion of the first image to produce the second image that is a complete version of the first image.

1D provides a flowchart of a process 150 for hallucinating high-resolution images and semantic label cards according to one embodiment. The steps 130 . 140 and 145 be in contact with you as before 1C described completed. At step 155 The model of the neural network processes the first image and the first semantic map to produce a second semantic map containing hallucinated semantic data corresponding to the second image. In one embodiment, the model of the neural network hallucinates a high-resolution semantic map for the second image, the second image being a high-resolution, zoomed image in the region of a high-resolution image. In one embodiment, the neural network model hallucinates the semantic content for the blank portion of the first image to generate the second semantic map.

wobei S_I und Sc die unvollständige bzw. vollständige semantische Karte sind. Um die Abbildungsfunktion zu erlernen, wird das System 100 des neuronalen Netzes unter Verwendung eines Satzes von Trainingsdaten trainiert, die durch die folgenden Parameter gegeben sind: $$\text{Trainingdatensatz}={\left\{I_I^{\left(i\right)},S_I^{\left(i\right)},I_C^{\left(i\right)},S_C^{\left(i\right)}\right\}}_{i=1:N}$$

In an incomplete picture I _i is learned by the system of the neural network a function F, the I _i into a corresponding complete picture I _C can map. The picture is: $${\text{I}}_{\text{C}}.{\text{S}}_{\text{C}}=\text{F}\left({\text{I}}_{\text{I}}.{\text{S}}_{\text{I}}\right)$$

in which S _I and Sc are the incomplete or complete semantic map. To learn the mapping function, the system becomes 100 of the neural network using a set of training data given by the following parameters: $$\text{Training data set}={\left\{I_I^{\left(i\right)}.S_I^{\left(i\right)}.I_C^{\left(i\right)}.S_C^{\left(i\right)}\right\}}_{i=1:N}$$

The learning task is therefore to connect the two associated distributions p ( I _i . S _I , Ic, Sc) and p ( I _i . S _I , F ( I _i . S _I )). The match between the two related distributions is the classic two-sample test problem in statistics, which can be solved by minimizing the maximum mean mismatch metric. In one embodiment, a variant of a conditional GAN framework is implemented to accomplish the task. The variant is based on the conditional GAN framework, where the input is the high-resolution semantic map and the output is the high-resolution image. The framework is modified so that the input is a pair of incomplete image and semantic map, while the output is a pair of full image and semantic map.

For a training data set used for the supervised training, training pairs include an incomplete image and a corresponding complete (target) image according to ground truth or complete ground truth image, respectively. In one embodiment, the incomplete images are low resolution images and the complete images are corresponding complete (target) images according to ground truth. In one embodiment, pairs of incomplete input data are generated by removing data from full image and semantic map pairs. For example, low resolution input images are generated by scaling down 256x128 pixel high resolution Cubic interpolation output images to 16x8 pixels and then scaling up the downscaled image using bicubic interpolation to produce a blurred 256x128 pixel input image. For example, low-resolution semantic maps are generated by converting a high-resolution 256 x 128 pixel semantic label map into probability values for each label class (a probability map). In one embodiment, the high-resolution semantic map is converted to a parameterized volume (e.g., a cube), where a depth of the cube corresponds to the number of labels, and each voxel of the cube is a semantic probability value of zero or one. The probability cube is scaled down to 16x8 pixels using bicubic interpolation, then the scaled down probability cubic cubic interpolation cube is scaled up to produce a 256 × 128 probability cube, and the upscaled probability cube is converted to the low resolution semantic map. The conversion of the probability cube into the semantic map means that each pixel in the low-resolution semantic map has a particular semantic label instead of having a probability value over the semantic labels.

If a subset of incomplete images in the training dataset lacks the corresponding full images, a partially supervised learning approach may be used to train the system 100 of the neural network. Therefore, the system can 100 of the neural network can be trained simultaneously with supervised and unsupervised learning.

Ubertranslation creates an image that contains areas that are not observed in the input image. Compared to Uberzoom, the over-translation mapping function F learns to expand views outside the given field of view of an input image. Therefore, the translation task can be interpreted as learning to extrapolate, ie from an incomplete picture I _i and an incomplete semantic map S _I a complete picture I _C and generate a complete semantic map Sc. In summary, the basic mapping required for the translation of image content for translation is also Ic, Sc = F (I _I , S _I ). Therefore, the system can 100 of the neural network to perform hallucinations for overzoom applications, using a different training record than it would for training the system 100 of the neural network is used for translation applications.

2A provides a block diagram of a training configuration 200 for the system 100 of the neural network of 1B according to one embodiment. In addition to the system 100 of the neural network includes the training configuration 200 a neural network 225 a semantic discriminator, a neural network 235 an image discriminator, the loss functions 215 and 205 and a parameter updating unit 220 , The combination of the neural network 135 a generator and the neural network 235 of the image discriminator forms a GAN. The combination of the neural network 125 a generator and the neural network 225 of the semantic discriminator forms a second GAN.

wobei G das neuronale Netz 125 des Generators ist, E das neuronale Netz 115 des Kodierers ist, [G(I_I,S_I)]_semantic die Ausgabe der hochaufgelösten semantischen Karte ist, I_I das unvollständige Eingabebild ist, S_I die unvollständige semantische Eingabekarte ist, K die Anzahl der semantischen Labels ist und $S_{GT}^i$

die hochaufgelöste semantische Karte gemäß Ground Truth für das semantische Label i ist.A first base loss function is defined to mean differences between the output of a complete semantic map by the neural network 125 of the generator and the semantic map according to Ground Truth. The loss function 215 receives the output of the complete semantic map of the neural network 215 of the generator and a semantic map according to ground truth and calculates a cross entropy loss value for the complete semantic map. $$L_{base-semantic}\left(G\right)=e_{\left(I_I.S.I_C\right)}-\frac{1}{K}{\displaystyle \underset{i=1}{\overset{K}{\Sigma }}{S}_{GT}^i\text{log}{\left[G\left(I_I.S_I\right)\right]}_{semantic}^i}$$

where G is the neural network 125 of the generator, E is the neural network 115 of the encoder is [G (I _I , S _I )] _{semantic is} the output of the high-resolution semantic map, I _i the incomplete input image is, S _I the incomplete semantic input card is, K is the number of semantic labels and $S_{GT}^i$

is the high-resolution semantic map according to ground truth for the semantic label i.

wobei G das neuronale Netz 135 des Generators ist, E das neuronale Netz 105 des Kodierers ist, [G(I_I, S_I)]_image das hochaufgelöste Ausgabebild ist, N die Anzahl der Schichten in einem neuronalen Netz zur Berechnung der Verlustfunktion ist.A loss function of a second basis is defined to be differences between the output of the complete image by the neural network 135 of the generator and the image according to Ground Truth. The loss function 205 receives the output of the complete image of the neural network 135 of the generator and a picture according to ground truth and calculates a perception loss value for the complete output image. $$L_{base-imaGe}\left(G\right)=e_{\left(I_I.S.I_C\right)}{{\displaystyle \underset{i=I}{\overset{N}{\Sigma }}\Vert {\mathrm{\Phi }}^{\left(i\right)}({\left[G\left(I_I.S_I\right)\right]}_{imaGe}\Vert }}_I$$

where G is the neural network 135 of the generator, E is the neural network 105 of the encoder, [G (I _I , S _I )] _{image is} the high-resolution output image, N the number of layers in a neural network to calculate the loss function is.

wobei D_semantic das neuronale Netz 235 des semantischen Diskriminators ist.The neural network 225 of the semantic discriminator is an adversary discriminator, which is the output of the complete semantic card through the neural network 125 of the generator and receives a semantic map according to Ground Truth. The neural network 225 The semantic discriminator also receives the output of the complete image through the neural network 135 of the generator and a picture according to Ground Truth and calculates a conditional enemy loss. The neural network 225 of the semantic discriminator can compute the following conditional adversary loss function to ensure that the complete semantic map is plausible. $$\begin{array}{ll}{L}_{adv-semantic}\left(G.D_{semantic}\right)\hfill & \hfill \\ \hfill & \begin{array}{l}=e_{\left(I_{GT}.S_{GT}\right)}\left[\text{log}{D}_{semantic}\left(I_{GT}.S_{GT}\right)\right]+e_{\left(I_I.S_I.I_{GT}\right)}[\text{log}(1\\ -D_{semantic}\left(I_{GT}.{\left[G(I_I.S_I\right]}_{semantic}\right))]\end{array}\hfill \end{array}$$

where D _{semantic is} the neural network 235 of the semantic discriminator.

wobei D_image das neuronale Netz 225 des Bilddiskriminators ist.The neural network 235 of the image discriminator corresponds to an opposing discriminator, which is the output of the complete image through the neural network 135 of the generator and a picture according to Ground Truth receives. The neural network 235 The image discriminator also receives the output of the complete semantic map through the neural network 125 of the generator and a semantic map according to Ground Truth and calculates a conditional enemy loss. The neural network 235 of the image discriminator can compute the following conditional enemy loss function to ensure that the complete image is plausible. $$\begin{array}{ll}{L}_{adv-imaGe}\left(G.D_{imaGe}\right)\hfill & \hfill \\ \hfill & \begin{array}{l}=e_{\left(I_{GT}.S_{GT}\right)}\left[\text{log}{D}_{imaGe}\left(S_{GT}.I_{GT}\right)\right]\\ +e_{\left(I_I.S_I.S_{GT}\right)}\left[\text{log}\left(1-D_{imaGe}\left(S_{GT}.{\left[G(I_I.S_I\right]}_{imaGe}\right)\right)\right]\end{array}\hfill \end{array}$$

where D _{image is} the neural network 225 of the image discriminator.

und $$\begin{array}{c}{L}_{FM-semantic}\left(G,D_{semantic}\right)=E_{\left(I_I,S_I,I_{GT},S_{GT}\right)}{\displaystyle {\sum }_{i=1}^T\Vert D_{semantic}^i\left(I_{GT},S_{GT}\right)-}\\ {D_{semantic}^i\left(I_{GT},{\left[G\left(I_I,S_I\right)\right]}_{semantic}\right)\Vert }_I\end{array}$$

wobei Dⁱ die Aktivierung der Schicht i aus dem neuronalen Netz 225 oder 235 des Diskriminators und T die Anzahl der Schichten in den neuronalen Netzen 225 und 235 des Diskriminators ist.Two other loss values, in particular the discriminator activation loss value or the characteristic match loss value, may pass through the neural network 225 the semantic discriminator and the neural network 235 of the image discriminator to further improve performance. $${\begin{array}{l}{L}_{FM-imaGe}\left(G.D_{imaGe}\right)\hfill \\ =+e_{\left(I_I.S_I.I_{GT}.S_{GT}\right)}{\displaystyle \underset{i=1}{\overset{T}{\Sigma }}\Vert D_{imaGe}^i\left(S_{GT}.I_{GT}\right)-D_{imaGe}^i\left(S_{GT}.{\left[G\left(I_I.S_I\right)\right]}_{imaGe}\right)\Vert }\hfill \end{array}}_I$$

and $$\begin{array}{c}{L}_{FM-semantic}\left(G.D_{semantic}\right)=e_{\left(I_I.S_I.I_{GT}.S_{GT}\right)}{\displaystyle {\Sigma }_{i=1}^T\Vert D_{semantic}^i\left(I_{GT}.S_{GT}\right)-}\\ {D_{semantic}^i\left(I_{GT}.{\left[G\left(I_I.S_I\right)\right]}_{semantic}\right)\Vert }_I\end{array}$$

in which D ⁱ the activation of layer i from the neural network 225 or 235 of the discriminator and T the number of layers in the neural networks 225 and 235 of the discriminator.

The parameter update unit 220 provides updated layer parameters (eg weights) for the neural network 115 of the encoder, the neural network 105 of the encoder, the residual function 110 , the neural network 125 of the generator and the neural network 135 of the generator based on the loss values available. In one embodiment, the parameter update unit updates 220 the parameters to minimize the losses according to the following optimization formula with six loss functions: $$\begin{array}{c}\underset{G}{{\displaystyle \text{min}}}\{L_{base-imaGe}\left(G\right)+L_{base-semantic}\left(G\right)+\underset{D_{imaGe}}{{\displaystyle \text{Max}}}(L_{adv-imaGe}\left(G.D_{imaGe}\right)+\\ L_{FM-imaGe}\left(G.D_{imaGe}\right))+\underset{D_{semantic}}{{\displaystyle \text{Max}}}(L_{adv-semantic}\left(G.D_{semantic}\right)+\\ L_{FM-semantic}\left(G.D_{semantic}\right))\}\end{array}$$

2 B provides a flowchart of a process 250 to train the system 100 of the neural network 1B according to one embodiment. Although the method 250 is described in the context of a processing unit, the method 120 also be performed by a program, a user-defined circuit or by a combination of a user-defined circuit and a program. For example, the procedure 250 by a GPU (graphics processing unit), a CPU (central processing unit), or any processor capable of implementing a training configuration for a neural network model, such as the system 100 of the neural network. In addition, the skilled person understands that any system that performs the procedure 250 is within the scope and spirit of embodiments of the present disclosure.

The steps 130 . 140 . 145 and 155 be as in connection with the 1C and 1D described completed. At step 255 become the second semantic map, the ground truth map semantic map, the second image and a ground truth image through the neural network 225 of the semantic discriminator to produce semantic loss values. In one embodiment, the semantic loss values are calculated as L _adv-semantic (G, D _semantic ) and / or L _{FM semantic} (G, D _semantic ).

At step 260 For example, the second image, the ground truth image, the second semantic map, and a ground truth truth map are passed through the neural network 235 of the image discriminator to produce image loss values. In one embodiment, the image loss values are calculated as L _adv-image (G, D _image ) and / or L _FM-image (G, D _image ).

At step 265 processes the loss function 215 the second semantic map and the ground truth map semantic map to generate additional semantic loss values. In one embodiment, the additional semantic loss values are expressed as L _{base-semantic} ( G ). At step 270 processes the loss function 205 the second image and the image according to ground truth to produce additional image loss values. In one embodiment, the additional image loss values are expressed as L _base-image ( G ). At step 275 updates the parameter update unit 220 the parameters of the system 100 of the neural network to minimize losses based on the semantic image loss values, the additional semantic loss values, the image loss values, and the additional image loss values.

After training, the system can 100 of the neural network for hallucinating image content for the applications zoom in, moving the camera and / or generating infinite content, the temporal consistency of the output image sequence providing a realistic or plausible result. For the application of zooming or enlargement, the temporal consistency can be achieved by the production of intermediate images. For example, instead of creating a high-resolution image that is 16x zoom, a 2x zoom can be generated iteratively (4 times) using the output of the previous iteration as input to the system 100 of the neural network is used for the next iteration. In another example, a user may provide multiple input images as keyframes to improve temporal consistency. For example, the keyframes for an image sequence may be downsampled, and intervening frames may be replaced by the downsampled data from the keyframes to hallucinate image data for at least one intervening image in the image sequence. In addition to creating more consistent image sequences, adopting multiple input images to create the hallucinated content provides the user with more control.

In summary, the system allows 100 the neural network generating an image sequence by hallucinating content for one or more input images whose resolution is too low (by zooming) or out of the field of view of the original camera (ie simulating a laterally moving or rotating camera). The output sequence of complete images may then be a series of images forming a short video sequence or a series of alternative versions of the input image where the content varies by hallucinations.

Hallucinating image content can also be used to create street views (looking at a vanishing point) and create video sequences that are spatially and temporally consistent. The video sequences may have a different or even infinite length of one or more low quality images and one or more corresponding low quality semantic maps. In one embodiment, a video sequence is a similar street scene to supplement a training data set needed for machine learning and computer vision tasks. Importantly, different instances of instances in a scene can be hallucinated from a single input image and a corresponding semantic map, which easily complements the training data for scene or object recognition tasks.

Parallel processing architecture

3 create a parallel processing unit (PPU) 300 according to one embodiment. In one embodiment, the PPU 300 a multithreaded processor implemented on one or more integrated circuit devices. The PPU 300 is a latency-concealing architecture designed to handle many threads in parallel. A thread (eg a thread of execution) is an instantiation of a series of statements that are required to be executed by the PPU 300 are configured. In one embodiment, the PPU 300 a graphics processing unit (GPU) configured to implement a graphics rendering pipeline for processing three-dimensional (3D) image data to produce two-dimensional (2D) image data for display on a display device such as a liquid crystal display (LCD) device , In other embodiments, the PPU 300 used to perform general purpose calculations. Although an exemplary parallel processor is provided herein for illustrative purposes, it is to be understood that this processor is for illustrative purposes only and that any processor may be used to supplement and / or replace it.

One or more PPUs 300 can be configured to accelerate thousands of high performance computing (HPC), data center, and machine learning applications. The PPU 300 can be configured to accelerate numerous systems and applications for depth learning, including autonomous vehicle platforms, deep learning, high-precision speech, image and text recognition systems, intelligent video analytics, molecular simulations, drug development, disease diagnosis, weather forecasting, big data analytics, Astronomy, molecular dynamics simulation, financial modeling, robotics, factory automation, real-time language translation, online search optimization, and personalized user recommendations and the like.

As in 3 shown includes the PPU 300 an input / output (I / O) unit 305, a front-end unit 315 , a scheduler unit 320 , a work distribution unit 325 , a network hub 330 , a crossbar (Xbar) 370 , one or more general processing clusters (GPCs) 350 and one or more storage partition units 380 , The PPU 300 can connect to one or more high-speed NVLink 310 connections to a host processor or other PPUs 300 get connected. The PPU 300 can have a connection 302 be connected to a host processor or other peripherals. The PPU 300 can also be connected to a local memory containing a number of memory devices 304 includes. In one embodiment, the local memory may include a number of Dynamic Random Access Memory (DRAM) devices. The DRAM devices may be configured as an HBM (High-Bandwidth Memory) subsystem with multiple DRAM dies stacked in each device.

The NVLink 310 connection allows systems to have one or more PPUs 300 Scale and integrate with one or more CPUs to support cache coherency between the PPUs 300 and CPUs and CPU mastering. Data and / or commands can be obtained from NVLink 310 over the network node 330 to / from other units of the PPU 300 be transmitted, such as one or more copy machines, a video encoder, a video decoder, a power management unit, etc. (not explicitly shown). The NVLink 310 will be in contact with 5B described in more detail.

The I / O unit 305 is configured to communicate (eg, commands, data, etc.) from a host processor (not shown) over the connection 302 to send and receive. The I / O unit 305 can work with the host processor directly over the connection 302 or communicate via one or more intermediate devices, such as a memory bridge. In one embodiment, the I / O unit 305 communicate with one or more other processors, such as one or more of the PPUs 300 about the connection 302 , In one embodiment, the I / O unit implements 305 a Peripheral Component Interconnect Express (PCIe) interface for communication over a PCIe bus and the connection 302 is a PCIe bus. In alternative embodiments, the I / O unit 305 implement other types of known interfaces for communicating with external devices.

The I / O unit 305 decodes packets over the connection 302 be received. In one embodiment, the packets represent commands configured to the PPU 300 to cause to perform various operations. The I / O unit 305 transmits the decoded commands to various other units of the PPU 300 as specified in the commands. For example, some commands to the frontend unit 315 be transmitted. Other commands can be sent to the network node 330 or other units of the PPU 300 such as one or more copy machines, a video encoder, a video decoder, a power management device, etc. (not explicitly shown). In other words, the I / O unit 305 is configured to communicate between the various logical units of the PPU 300 and to lead among them.

In one embodiment, a program executed by the host processor encodes an instruction stream in a buffer belonging to the PPU 300 Provides workloads for processing. A workload can include multiple statements and data to be processed by these statements. The buffer is an area in a memory owned by both the host processor and the PPU 300 is accessible (eg read / write). For example, the I / O unit 305 be configured to access the buffer in a system memory associated with the connection 302 is connected via storage requirements over the connection 302 be transmitted. In one embodiment, the host processor writes the instruction stream into the buffer and then sends a pointer to the beginning of the instruction stream to the PPU 300 , The frontend unit 315 receives pointers to one or more instruction streams. The frontend unit 315 manages one or more streams, reads commands from the streams, and forwards commands to the various units of the PPU 300 further.

The frontend unit 315 is with a scheduler unit 320 Coupled with the various GPCs 350 configured to process tasks that are defined by one or more streams. The scheduler unit 320 is configured to keep track of state information related to the various tasks performed by the scheduler unit 320 to get managed. The state can indicate which GPC 350 a task is assigned, whether the task is active or inactive, a priority level assigned to the task, and so on. The scheduler unit 320 manages the execution of a large number of tasks on one or more GPCs 350 ,

The scheduler unit 320 is with a work distribution unit 325 coupled, which is configured to perform tasks on the GPCs 350 to distribute. The work distribution unit 325 can track a number of scheduled tasks by the scheduler unit 320 were received. In one embodiment, the work distribution unit manages 325 for each of the GPCs 350 a pool of upcoming tasks and a pool of active tasks. The pool of pending tasks may include a number of slots (eg, 32 slots) that contain tasks performed by a particular GPC 350 to be edited. The active task pool may include a number of slots (eg, 4 slots) for tasks performed by the GPCs 350 be actively worked on. If a GPC 350 When a task finishes executing, this task is taken from the pool of active tasks for the GPC 350 and select one of the other tasks from the pool of upcoming tasks and run it on the GPC 350 planned. If an active task on the GPC 350 was inactive, z. For example, while waiting for the resolution of a data dependency, the active task may be from the GPC 350 Thrown and returned to the pool of upcoming tasks, while another task selected in the pool of upcoming tasks and for execution on the GPC 350 is planned.

The work distribution unit 325 communicates with one or more GPCs 350 over the XBar 370 , The XBar 370 is a connection network that many of the units of PPU 300 with other units of PPU 300 coupled. For example, the XBar 370 be configured to the work distribution unit 325 with a particular GPC 350 to pair. Although not expressly stated, one or more other units of PPU 300 also via the network node 330 with the XBar 370 get connected.

The tasks are handled by the scheduler unit 320 managed and distributed by the work distribution unit 325 to a GPC 350 cleverly. The GPC 350 is configured to edit the task and generate results. The results can be obtained from other tasks within the GPC 350 taken over the XBar 370 to another GPC 350 forwarded or in memory 304 get saved. The results can be obtained through the storage partition units 380 in the store 304 written to a memory interface for reading and writing data to / from memory 304 to implement. The results can be obtained through the NVLink 310 to another PPU 304 or CPU. In one embodiment, the PPU includes 300 a number U of storage partition units 380 equal to the number of separate and different memory devices 304 is that with the PPU 300 are coupled. A storage partition unit 380 will be in connection with 4B described in more detail.

In one embodiment, a host processor executes a driver core that implements an application programming interface (API) that allows one or more applications running on the host processor to perform operations on the PPU 300 to plan. In one embodiment, multiple computing applications simultaneously from the PPU 300 executed, and the PPU 300 provides isolation, Quality of Service (QoS) and independent address space for multiple computing applications. An application may generate instructions (eg, API calls) that cause the driver core to perform one or more tasks for execution by the PPU 300 to create. The driver core issues tasks to one or more streams from the PPU 300 are processed. Each task may include one or more groups of related threads, referred to herein as warp. In one embodiment, a warp 32 related threads that can be executed in parallel. Cooperating threads can refer to a variety of threads, including instructions for executing the task that can share data through shared memory. Threads and collaborative threads are associated with 5A described in more detail.

4A puts a GPC 350 the PPU 300 out 3 according to one embodiment. As in 4A is shown, includes every GPC 350 a set of hardware units for processing tasks. In one embodiment, each GPC includes 350 a pipeline manager 410 , a Pre-Raster Operations Unit (PROP) 415 , a raster engine 425 , a Work Distribution Crossbar (WDX) 480 , a Memory Management Unit (MMU) 490 and one or more Data Processing Clusters (DPCs) 420 , It should be noted that the GPC 350 from 4A instead of or in addition to the in 4A units shown may include other hardware units.

In one embodiment, the operation of the GPC 350 through the pipeline manager 410 controlled. The pipeline manager 410 Manages the configuration of one or more DPCs 420 for editing tasks related to the GPC 350 assigned. In one embodiment, the pipeline manager 410 at least one of the one or more DPCs 420 configure to implement at least part of a graphics rendering pipeline. For example, a DPC 420 configured to use a vertex shader program on the programmable streaming multiprocessor (SM) 440 perform. The pipeline manager 410 can also be configured to work from the work distribution unit 325 received packets to the corresponding logical units within the GPC 350 forward. For example, some packages can be attached to hard-function hardware units in PROP 415 and / or in the raster engine 425 be forwarded while other packets to the DPCs 420 for processing by the primitive engine 435 or the SM 440 can be forwarded. In one embodiment, the pipeline manager 410 at least one of the one or more DPCs 420 to implement a model of a neural network and / or a computational pipeline.

The PROP unit 415 is configured to by the raster engine 425 and the DPCs 420 generated data to a Raster Operations (ROP) unit to be used in conjunction with 4B will be described in more detail. The PROP unit 415 can also be configured to perform optimizations for color mixing, organize pixel data, perform address translations, and the like.

The raster engine 425 includes a set of fixed-function hardware units configured to perform various raster operations. In one embodiment, the raster egg includes 425 a setup engine, a Grobraster engine, a culling engine, a clipping engine, a fine grid engine, and a tile merge engine. The device engine receives transformed vertices and generates plane equations associated with the geometric primitive defined by the vertices. The plane equations are transmitted to the Grobraster engine to provide coverage information (eg x, y Mask for a tile) for the primitive. The power of the Grobraster engine is passed to the culling engine, where fragments that are assigned to the primitive and fail a z-test are sorted out and transferred to a clipping engine, where fragments are out of sight , get cut. The fragments that survive clipping and culling may be passed to the fine grid engine to generate attributes for the pixel fragments based on the plane equations generated by the device engine. The output of the raster engine 425 includes fragments, for example, of one in a DPC 420 implemented fragment shaders are to be processed.

Every DPC 420 that in GPC 350 is included, includes one M Pipe Controller (MPC) 430 , a primitive engine 435 and one or more SMs 440 , The MPC 430 controls the operation of the DPC 420 and directs those from the pipeline manager 410 received packets to the appropriate units of the DPC 420 further. For example, packets associated with a vertex may be sent to the primitive engine 435 which is configured to retrieve the vertex attributes associated with the vertex from memory 304 pick up. In contrast, packets associated with a shader program may be sent to the SM 440 be transmitted.

The SM 440 includes a programmable streaming processor configured to process tasks represented by a series of threads. Every SM 440 is multi-threaded and configured to handle a variety of threads (eg 32 Threads) from a particular group of threads concurrently. In one embodiment, the SM implements 440 a single-instruction (multiple-data) architecture in which each thread in a group of threads (eg, a warp) is configured to process another record based on the same instruction set. All threads in the set of threads execute the same statements. In another embodiment, the SM implements 440 a single-instruction, multiple thread (SIMT) architecture in which each thread in a group of threads is configured to process another record based on the same set of instructions, but individual threads in the set of threads during execution may diverge. In one embodiment, a program counter, a call stack, and an execution state are maintained for each warp, allowing for concurrency between warps and serial execution within warps when threads within the warp diverge. In another embodiment, a program counter, a call stack, and an execution state are maintained for each individual thread, allowing equivalent concurrency between all threads, within and between the warps. When running the execution state for each thread, threads that execute the same instructions can converge and execute in parallel for maximum efficiency. The SM 440 will be in connection with 5A described in more detail.

The MMU 490 provides an interface between the GPC 350 and the storage partition unit 380 , The MMU 490 can enable translation of virtual addresses into physical addresses, memory protection, and competing requests for storage. In one embodiment, the MMU 490 One or more Translation Lookaside Buffers (TLBs) are available to translate virtual addresses into physical addresses in memory 304 perform.

4B represents a storage partition unit 380 the PPU 300 from 3 according to one embodiment. As in 4B is shown includes the storage partition unit 380 a raster operations (ROP) unit 450 , a level 2 (L2) cache 460 and a memory interface 470 , The storage interface 470 is with the store 304 coupled. The storage interface 470 can 32 . 64 . 128 , 1024-bit data buses or the like for high speed data transmission. In one embodiment, the PPU includes 300 a memory interface 470 per pair of storage partition units 380 where each pair of storage partition units 380 with a corresponding storage device 304 connected is. So can the PPU 300 for example, up to Y memory devices 304 such as high-bandwidth memory stacks or double-data-rate graphics, version 5 , synchronous dynamic random access memory or other types of persistent storage.

In one embodiment, the memory interface implements 470 an HBM2 memory interface and Y is half of U. In one embodiment, the HBM2 memory stacks reside on the same physical enclosure as the PPU 300 , which results in significant power and space savings compared to traditional GDDR5 SDRAM systems. In one embodiment, each HBM2 stack includes four memory chips and Y is equal to 4, with the HBM2 stack two 128 -Bit-channels per chip for a total of 8 channels and a data bus width of 1024 bits.

In one embodiment, the memory supports 304 Single-Error Correcting Double-Error Detecting (SECDED) Error Correction Code (ECC) to protect data. ECC provides greater reliability for computing applications that are sensitive to data loss. Reliability is especially important in large clustered computer environments where PPUs are used 300 process very large data sets and / or operate applications over a longer period of time.

In one embodiment, the PPU implements 300 a multilevel storage hierarchy. In one embodiment, the storage partition unit supports 380 a unified memory to provide a single unified virtual address space for CPU and PPU 300 memory that enables data exchange between virtual storage systems. In one embodiment, the frequency of accesses to a PPU 300 Traced memory on other processors to ensure that memory pages are in the physical memory of the PPU 300 be moved if it accesses the pages more often. In one embodiment, the NVLink supports 310 Address translation services that the PPU 300 allow access directly to the page tables of a CPU, and full access to the CPU memory through the PPU 300 enable.

In one embodiment, copy machines transmit data between multiple PPUs 300 or between PPUs 300 and CPUs. The copy machines can generate page faults for addresses that are not shown in the page tables. The storage partition unit 380 can then correct the page faults and enter the addresses in the page table, whereupon the copying machine can perform the transfer. In a conventional system, the memory for multi-copy engine operations is dedicated (eg non-swappable) between multiple processors, which significantly reduces available memory. With hardware page faults, addresses can be passed to the copy machines without worrying about whether the pages are resident and the copy process is transparent.

Data from the memory 304 or any other system memory may be from the storage partition unit 380 retrieved and in L2 cache 460 stored on the chip and from the various GPCs 350 is shared. As shown, each storage partition unit includes 380 a section of the L2 cache 460 that of a corresponding storage device 304 assigned. Subordinate caches can then be in different units within the GPCs 350 be implemented. For example, each of the SMs 440 a level one ( L1 ) Cache. Of the L1 Cache is a private store that belongs to a specific SM 440 assigned. Data from the L2 cache 460 can be retrieved and in each of the LI caches for processing in the functional units of the SMs 440 get saved. Of the L2 -cache 460 is with the memory interface 470 and the XBar 370 coupled.

The ROP unit 450 performs graphic raster operations related to pixel colors, such as color compression, pixel blending, and the like. The ROP unit 450 also performs depth checks in conjunction with the raster engine 425 which receives a depth for a sample position, which is a pixel fragment from the raster engine's culling engine 425 assigned. The depth is tested against a corresponding depth in a depth buffer for a sample position associated with the fragment. If the fragment passes the sample position depth test, the ROP unit updates 450 the depth buffer and sends a result of the depth test to the raster engine 425 , It should be noted that the number of storage partition units 380 from the number of GPCs 350 can deviate and therefore every ROP unit 450 with each of the GPCs 350 can be coupled. The ROP unit 450 keeps track of those from the various GPCs 350 received packets and determines to which GPC 350 one from the ROP unit 450 Result generated by the Xbar 370 is directed. Although the ROP unit 450 in 4B within the storage partition unit 380 included is the ROP unit 450 in another embodiment, outside the storage partition unit 380 are located. For example, this may be the ROP unit 450 in the GPC 350 or another unit.

5A represents the streaming multiprocessor 440 from 4A according to one embodiment. As in 5A is shown, the SM includes 440 an instruction cache 505 , one or more scheduler units 510 , a register file 520 , one or more cores 550 , one or more special function units (SFUs) 552 , one or more load / store units (LSUs) 554 , a connection network 580 , a shared memory / L1 cache 570 ,

As described above, the work distribution unit plans 325 Tasks to run on the GPCs 350 the PPU 300 , The tasks are a specific DPC 420 within a GPC 350 and, if the task is assigned to a shader program, the task can be assigned to an SM 440 be assigned. The scheduler unit 510 receives the tasks from the work distribution unit 325 and manages command scheduling for one or more of the SMs 440 associated thread blocks. The scheduler unit 510 schedules thread blocks to run as parallel thread warps, with each thread block associated with at least one warp. In one embodiment, each warp performs 32 Threads off. The scheduler unit 510 It may manage a plurality of different thread blocks by mapping the warps to the various thread blocks and then, during each clock cycle, directing instructions from the plurality of different cooperative groups to the various functional units (eg cores 550 , SFUs 552 and LSUs 554 ) sends.

Cooperative Groups is a programming model for organizing groups of communicating threads that enables developers to express the granularity with which threads communicate, enabling the expression of richer, more efficient parallel decompositions. Cooperative launch APIs support synchronization between thread blocks to execute parallel algorithms. Traditional programming models provide a single, simple construct to synchronize cooperating threads: a barrier across all thread block threads (for example, the syncthreads () function). However, programmers often want to define groups of threads that are smaller than the granularity of thread blocks and synchronize within the defined groups to allow for more performance, design flexibility, and software reuse in the form of shared, group-wide functional interfaces.

Cooperative groups allow programmers to explicitly define groups of threads in the form of subblock (e.g., as small as a single thread) and multiblock granularities and perform common operations such as thread synchronization in a cooperative set. The programming model supports clean composition across software boundaries so that libraries and utility functions can securely synchronize in their local context without having to make assumptions about convergence. Cooperative groups of primitives enable new patterns of cooperative concurrency, including producer-consumer parallelism, opportunistic parallelism, and global synchronization across an entire network of thread blocks.

A dispatcher unit 515 is configured to send instructions to one or more of the functional units. In the embodiment, the scheduler unit includes 510 two dispatcher units 515 which allow to send two different instructions from the same warp during each clock cycle. In alternative embodiments, each scheduler unit 510 a single dispatcher unit 515 or additional dispatcher units 515 include.

Every SM 440 contains a register file 520 containing a set of registers for the functional units of the SM 440 provides. In one embodiment, the register file is 520 divided into each of the functional units, so that each functional unit a specific section of the register file 520 assigned. In another embodiment, the register file becomes 520 divided into the different warps, that of the SM 440 be executed. The register file 520 provides buffers for operands associated with the data paths of the functional units.

Every SM 440 includes L processing cores 550 , In one embodiment, the SM includes 440 a large number (eg 128 , etc.) of different processing cores 550 , Every core 550 may include a full pipeline, single precision, double precision and / or mixed precision processing unit including a floating point arithmetic logic unit and an integer arithmetic logic unit. In one embodiment, the floating-point arithmetic logic units implement the IEEE standard 754 - 2008 for floating point arithmetic. In one embodiment, the cores include 550 64 Single-precision (32-bit) floating-point cores, 64 Integer cores, 32 Double-Precision (64-bit) floating-point cores and 8 tensor cores.

Tensor cores are configured to perform matrix operations, and in one embodiment, one or more tensor cores are in the cores 550 contain. In particular, the tensor cores are configured to perform matrix arithmetic for depth learning, such as convolution operations for training and inference in neural networks. In one embodiment, each tensor core operates on a 4x4 matrix and performs a matrix multiplication and summation operation D = AB + C, where A . B . C and D 4x4 matrices are.

In one embodiment, the matrix multiplier inputs are A and B 16-bit floating-point matrices, while the summation matrices C and D 16-bit floating-point or 32-bit floating-point arrays. Tensor cores use 16-bit floating-point input data with 32-bit floating-point accumulation. The 16-bit floating-point multiplication requires 64 operations and results in a high-precision product which is then accumulated using the 32-bit floating-point addition with the other intermediates for 4x4x4x4 matrix multiplication. In practice, tensor cores are used to perform much larger two-dimensional or higher-dimensional matrix operations composed of these smaller elements. An API, such as the CUDA 9 C ++ API, provides specialized matrix loading, matrix multiplication, summation, and matrix memory operations to efficiently use tensor kernels from a CUDA C ++ program. At the CUDA level, the warp-level interface is based on 16x16 matrices, which include all 32 threads of the warp.

Every SM 440 also includes M SFUs 552 performing special functions (eg attribute evaluation, reciprocal square root and the like). In one embodiment, the SFUs 552 include a tree traversal unit configured to traverse a hierarchical tree data structure. In one embodiment, the SFUs 552 include a texture unit configured to perform texture map filtering operations. In one embodiment, the texture units are configured to render texture maps (eg, a 2D array of textures) from memory 304 to load and scan the texture maps to produce sampled texture values for use in shader programs written by the SM 440 be executed. In one embodiment, the texture maps are in shared memory / L1 cache 470 saved. The texture units implement texture operations, such as filter operations, using mip maps (eg texture maps with varying degrees of detail). In one embodiment, each SM includes 340 two texture units.

Every SM 440 also includes N LSUs 554 , the load and store operations between the shared memory / L1 cache 570 and the register file 520 to implement. Every SM 440 includes a connection network 580 containing each of the functional units with the register file 520 and the LSU 554 with the register file 520 , shared memory / L1 cache 570 combines. In one embodiment, the interconnect network is 580 a switch fabric that can be configured to associate one of the functional units with one of the registers in the register file 520 to connect and the LSUs 554 with the register file and the shared memory / L1 cache locations 570 connect to.

The shared memory / L1 cache 570 is a field of on-chip storage that supports data storage and communication between the SM 440 and the primitive engine 435 as well as between the threads in the SM 440 allows. In one embodiment, the shared memory includes / L1 cache 570 128 KB storage capacity and is located on the path from the SM 440 to the storage partition unit 380 , The common Memory / L1 cache 570 can be used for cache reading and writing. One or more of the shared memory / L1 cache 570 , L2 cache 460 and memory 304 are backup storage.

The combination of data cache and shared memory functionality in a single block of memory provides the best overall performance for both types of memory accesses. The capacity can be used as a cache for programs that do not use shared memory. For example, if the shared memory is configured to use half the capacity, texture and load / store operations can use the remaining capacity. The integration into the shared memory / L1 cache 570 allows the shared memory / L1 cache 570 to act as a high-throughput streaming line for data while providing high-bandwidth, low-latency access to frequently reused data.

In the general parallel configuration configuration, a simpler configuration compared to graphics processing can be used. In particular, the in 3 bypassed graphics processing units with a fixed function, creating a much simpler programming model. In the universal configuration of the parallel calculation, the work distribution unit 325 Thread blocks directly to the DPCs 420 to and distribute them. The threads in a block execute the same program, using a unique thread ID in the calculation to ensure that each thread produces unique results, with the SM 440 is used to run the program and perform calculations, the shared memory / L1 cache 570 to communicate between threads and the LSU 554 to the global memory through the shared memory / L1 cache 570 and the storage partition unit 380 to read and write. If the SM 440 For general parallel computation, it can also write commands to the scheduler unit 320 new work on the DPCs 420 can start.

The PPU 300 can be used in a desktop computer, a laptop computer, a tablet computer, servers, supercomputers, a smartphone (such as a wireless, portable device), a personal digital assistant (PDA), a digital camera, a vehicle, a head-mounted Display, a portable electronic device and the like may be included. In one embodiment, the PPU 300 executed on a single semiconductor substrate. In another embodiment, the PPU 300 together with one or more other devices, such as additional PPUs 300 the store 204 , a reduced instruction set CPU (RISC), a memory management unit (MMU), a digital-to-analog converter (DAC) and the like, in a system on a chip (SoC).

In one embodiment, the PPU 300 to be included on a video card containing one or more storage devices 304 includes. The graphics card can be configured to connect to a PCIe slot on a motherboard of a desktop computer. In still another embodiment, the PPU 300 an integrated graphics processing unit (iGPU) or a parallel processor included in the chipset of the motherboard.

Exemplary calculation system

Systems with multiple GPUs and CPUs are used in a variety of industries as developers provide and leverage more parallelism in applications such as artificial intelligence. Powerful GPU-accelerated systems with tens of thousands of compute nodes are being used in data centers, research facilities, and supercomputers to solve more and more problems. As the number of processing devices within the high performance systems increases, the communication and data transfer mechanisms must be scaled to support the increased bandwidth.

5B is a conceptual representation of a processing system 500 using the PPU 300 out 3 has been implemented according to one embodiment. The exemplary system 565 can be configured to do that in the 1C . 1D and / or 2B illustrated methods 120 . 150 and 250 to implement. The processing system 500 includes a CPU 530 , a switch 510 and in each case several PPUs 300 and corresponding memory 304 , The NVLink 310 provides high-speed communication links between each of the PPUs 300 , Although a certain number of NVLink 310 and interconnect 302 Connections in 5B is shown, the number of connections to each PPU 300 and the CPU 530 vary. The switch 510 connects the connection 302 with the CPU 530 , The PPUs 300 , the memory 304 and NVLinks 310 can reside on a single semiconductor platform to a parallel processing module 525 to build. In one embodiment, the switch supports 510 two or more protocols for the interface between different different connections and / or connections.

In another embodiment (not shown), the NVLink provides 310 one or more high speed communication links between each of the PPUs 300 and the CPU 530 ready, and the switch 510 connects the connection 302 with the PPUs 300 via an interface. The PPUs 300 , the memory 304 and the connection 302 can be arranged on a single semiconductor platform to a parallel processing module 525 to build. In yet another embodiment (not shown), the compound provides 302 one or more communication links between each of the PPUs 300 and the CPU 530 ready, and the switch 510 connects each of the PPUs 300 with an interface and with the NVLink 310 to one or more high-speed communication links between the PPUs 300 provide. In another embodiment (not shown), the NVLink provides 310 one or more high speed communication links between the PPUs 300 and the CPU 530 over the switch 510 ready. In yet another embodiment (not shown), the compound provides 302 one or more communication links between each of the PPUs 300 directly available. One or more of the high-speed NVLink communications links 310 can be used as a physical NVLink connection or as an on-chip or on-die connection using the same protocol as NVLink 310 be implemented.

In the present specification, a single semiconductor platform may refer to a single unitary semiconductor-based integrated circuit fabricated on a die or chip. It should be noted that the term single-semiconductor platform may also refer to multichip modules with increased connectivity that simulate on-chip operation and make significant improvements over a traditional bus implementation. Of course, the various circuits or devices may also be arranged individually or in various combinations of semiconductor platforms according to the wishes of the user. Alternatively, the parallel processing module 525 be implemented as a printed circuit board substrate and each of the PPUs 300 and / or memory 304 can be designed as a compact device. In one embodiment, the CPU is located 530 , the switch 510 and the parallel processing module 525 on a single semiconductor platform.

In one embodiment, the signal transmission rate is each NVLink 310 20 to 25 gigabits / second and every PPU 300 includes six NVLinks 310 Interfaces (as in 5B are represented, are five NVLink 310 Interfaces for each PPU 300 contain). Every NVLink 310 provides a data transfer rate of 25 gigabytes / second in each direction, delivering six links 300 Gigabyte / second. The NVLinks 310 can only be used for PPU-to-PPU communication as it is in 5B or for a combination of PPU-to-PPU and PPU-to-CPU, if the CPU 530 also one or more NVLink 310 Interfaces included.

In one embodiment, the NVLink allows 310 Direct load / memory / Atom access from the CPU 530 on the 300 Storage 304 every PPU. In one embodiment, the NVLink supports 310 Coherence operations, so that from the stores 304 read data in the cache hierarchy of the CPU 530 can be stored, which increases the cache access latency for the CPU 530 is reduced. In one embodiment, the NVLink includes 310 the support of Address Translation Services (ATS) so that the PPU 300 directly on page tables within the CPU 530 can access. One or more of the NVLinks 310 can also be configured to operate in a low-power mode.

5C represents an exemplary system 565 in which the different architectures and / or functionalities of the various previous embodiments can be implemented. The exemplary system 565 can be configured to do that in the 1C . 1D and / or 2B illustrated methods 120 . 150 and 250 to implement.

As shown, is a system 565 provided, the at least one central unit 530 which is connected to a communication bus 575 connected. The communication bus 575 can be implemented with any suitable protocol, such as PCI (Peripheral Component Interconnect), PCI Express, Accelerated Graphics Port (AGP), HyperTransport, or any other bus or point-to-point communication protocol (s). The system 565 also includes a main memory 540 , The control logic (software) and the data are stored in main memory 540 which may be implemented as random access memory (RAM).

The system 565 also includes the input devices 560 , the parallel processing system 525 and the display devices 545 For example, a conventional CRT (Cathode Ray Tube), LCD (Liquid Crystal Display), LED (Light Emitting Diode), Plasma Display or the like. User input can be from the input devices 560 received, eg keyboard, mouse, touchpad, microphone and the like. Each of the aforementioned modules and / or devices may even be located on a single semiconductor platform in order to the system 565 to build. Alternatively, the various modules can also be arranged individually or in various combinations of semiconductor platforms according to the wishes of the user.

Furthermore, the system can 565 over a network interface 535 for communication purposes with a network (eg, telecommunications network, local area network (LAN), wireless network, wide area network (WAN) such as Internet, peer-to-peer network, cable network, or the like).

The system 565 may also include a secondary memory (not shown). The secondary storage 610 includes, for example, a hard disk drive and / or a removable storage drive that is a floppy disk drive, a magnetic tape drive, a compact diskette drive, a digital versatile disk (DVD) drive, a recording device, a universal serial bus (USB) flash memory. The removable storage drive reads and / or writes in a known manner from a removable storage unit.

Computer programs or logic algorithms of computer control can be stored in main memory 540 and / or stored in secondary storage. Such computer programs make it possible for the system 565 to perform various functions when they are executed. The memory 540 , the memory and / or any other memory are possible examples of computer-readable media.

The architecture and / or functionality of the various previous figures may be implemented in the context of a general computer system, a circuit board system, a gaming console system, an application specific system, and / or another desired system. The system 565 For example, it may be in the form of a desktop computer, a laptop, a tablet computer, servers, supercomputers, a smartphone (eg, a wireless portable device), a personal digital assistant (PDA), a digital camera, a vehicle, a head mounted displays, a portable electronic device, a mobile telephone device, a television, a workstation, a game console, an embedded system and / or other type of logic.

Although various embodiments have been described above, it should be understood that they have been presented by way of example only, and not limitation. Therefore, the breadth and scope of a preferred embodiment should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Graphics processing pipeline

In one embodiment, the PPU includes 300 a graphics processor unit (GPU). The PPU 300 is configured to receive commands specifying image data shader programs. Graphics data may be defined as a series of primitives such as points, lines, triangles, quads, triangular stripes, and the like. Typically, a primitive includes data indicating a number of vertices for the primitive (eg, in a model-space coordinate system) and attributes associated with each vertex of the primitive. The PPU 300 can be configured to process the graphics primitives to generate a frame buffer (eg, pixel data for each of the pixels of the display).

An application writes model data for a scene (eg, a collection of vertices and attributes) to memory, such as system memory or memory 304 , The model data defines each of the objects that may be visible on a display. The application then makes an API call to the driver core that prompts the model data for presentation and display. The driver core reads the model data and writes commands into one or more streams to perform operations to process the model data. The commands can refer to different shader programs running on the SMs 440 the PPU 300 including one or more vertex shaders, body shaders, domain shaders, geometry shaders, and pixel shaders. For example, one or more of the SMs 440 be configured to execute a vertex shader program that processes a number of vertices defined by the model data. In one embodiment, the various SMs 440 be configured to run different shader programs simultaneously. For example, a first subset of SMs 440 be configured to execute a vertex shader program while a second subset of SMs 440 can be configured to run a pixel shader program. The first subset of SMs 440 Processes vertex data to produce processed vertex data and writes the processed vertex data into the L2 cache 460 and / or the memory 304 , After the processed vertex data has been rasterized (eg from three-dimensional data into two-dimensional Data converted to screen space) to generate fragment data, performs the second subset of SMs 440 a pixel shader to generate processed fragment data which is then mixed with other processed fragment data and stored in the frame buffer 304 to be written. The vertex shader program and the pixel shader program can execute concurrently and pipeline various data from the same scene until all the model data for the scene has been transferred to the frame buffer. Subsequently, the content of the frame buffer is transferred to a display controller for display on a display device.

6 is a conceptual representation of a graphics processing pipeline 600 that of the PPU 300 out 3 has been implemented according to one embodiment. The graphics processing pipeline 600 Figure 4 is an abstract flow diagram of the implemented processing steps for generating 2D computer images from 3D geometry data. As is known, pipelined architectures can more efficiently perform long latency operations by dividing the operation into a plurality of stages with the output of each stage coupled to the input of the next succeeding stage. Thus, the graphics processing pipeline receives 600 input data 601 going from one level to the next level of the graphics processing pipeline 600 be transmitted to output data 602 to create. In one embodiment, the graphics processing pipeline 600 represent a graphics processing pipeline defined by the OpenGL® API. Optionally, the graphics processing pipeline 600 be implemented within the functionality and architecture of the previous figures and / or any subsequent figure (s).

As in 6 includes the graphics processing pipeline 600 a pipeline architecture that includes multiple stages. The stages include, but are not limited to, a data assembly stage 610 , a vertex shading level 620 , a primitive assembly stage 630 , a geometry shading level 640 , a viewport-scale, cull, and clip (VSCC) level 650 , a screening level 660 , a fragment shading level 670 and a raster operations level 680 , In one embodiment, the input data includes 601 Commands that configure the processing units to follow the stages of the graphics processing pipeline 600 and implement geometric primitives (eg, points, lines, triangles, squares, quads, triangular stripes, or fans, etc.) to be processed by the stages. The output data 602 may include pixel data (eg, color data) that is copied to a memory in a frame buffer or other type of surface data structure.

The data assembly stage 610 receives the input data 601 which specify vertex data for high-order surfaces, primitive or the like. The data assembly stage 610 collects the vertex data in a cache or queue, for example, by receiving a command from the host processor containing a pointer to a buffer in memory and reading the vertex data from the buffer. The vertex data is then sent to the vertex shading stage for processing 620 transfer.

The vertex shading level 620 Processes vertex data by performing a series of operations (eg, a vertex shader or a program) once for each of the vertices. For example, vertices may be specified as a 4-coordinate vector (eg, <x, y, z, w>) associated with one or more vertex attributes (eg, color, texture coordinates, surface normals, etc.). The vertex shading level 620 can manipulate individual vertex attributes such as position, color, texture coordinates, and the like. In other words, the vertex shading level 620 performs operations on the vertex coordinates or other vertex attributes associated with a vertex. Such operations typically involve lighting operations (eg, changing color attributes for a vertex) and transform operations (eg, changing the coordinate space for a vertex). For example, vertices may be specified by coordinates in an object coordinate space that are transformed by multiplying the coordinates by a matrix that translates the coordinates from the object coordinate space to a world space or a normalized device coordinate space (NCD). The vertex shading level 620 generates transformed vertex data to the primitive assembly stage 630 be transmitted.

The primitive assembly stage 630 collects those from the vertex shading level 620 output vertices and groups the vertices to geometric primitives for processing by the geometry shading stage 640 , For example, the primitive assembly stage 630 be configured to use all three consecutive vertices as a geometric primitive (eg, a triangle) for transmission to the geometry shading stage 640 to group. In some embodiments, certain vertices may be reused for successive geometric primitives (eg, two consecutive triangles in a triangular strip may share two vertices). The primitive assembly stage 630 transfers geometric primitives (eg a collection of associated vertices) to the geometry shading stage 640 ,

The geometry shading level 640 processes geometric primitives by performing a series of operations (such as a geometry shader or a program) on the geometric primitive. Tessellation operations can generate one or more geometric primitives from each geometric primitive. In other words, the geometry shading level 640 may subdivide each geometric primitive into a finer mesh of two or more geometric primitives for processing by the remainder of the graphics processing pipeline 600 are provided. The geometry shading level 640 transfers geometric primitives to the viewport SCC stage 650 ,

In one embodiment, the graphics processing pipeline 600 within a streaming multiprocessor and the vertex shading stage 620 , the primitive assembly stage 630 , the geometry shading level 640 , the fragment shading level 670 and / or the associated hardware and software can perform processing operations sequentially. Once the sequential processing operations are completed in one embodiment, the viewport SCC stage 650 to use the data. In one embodiment, primitive data obtained from one or more stages of the graphics processing pipeline 600 be written to a cache (eg L1 cache, vertex cache, etc.). In this case, the viewport SCC stage 650 in one embodiment, accessing the data in the cache. In one embodiment, the viewport is SCC stage 650 and the screening level 660 implemented as a fixed function circuit.

The viewport SCC level 650 performs the scaling, culling, and clipping of geometric primitives. Each rendered surface is linked to an abstract camera position. The camera position represents a position of a viewer viewing the scene and defines a viewing volume that includes the objects of the scene. The view volume may include a viewing plane, a posterior plane and four clipping planes. Any geometric primitive that is completely outside the view volume may be discarded (eg discarded) because the geometric primitive does not contribute to the final rendered scene. Any geometric primitive that is partially within the view volume and partially outside the view volume may be trimmed (eg, converted into a new geometric primitive that is included within the view volume). In addition, geometric primitives may each be scaled based on a depth of view volume. All potentially visible geometric primitives are then sent to the grid level 660 transfer.

The grid level 660 converts the geometric 3D primitives into 2D fragments (eg displayable, etc.). The grid level 660 can be configured to use the vertices of the geometric primitives to construct a set of plane equations from which various attributes can be interpolated. The grid level 660 may also compute a coverage mask for a plurality of pixels indicating whether one or more sample positions for the pixel interrupts the geometric primitive. In one embodiment, a Z-test may also be performed to determine if the geometric primitive is obscured by other geometric primitives already rasterized. The grid level 660 generates fragment data (eg, interpolated vertex attributes associated with a particular sample location for each covered pixel) that is sent to the fragment shading stage 670 be transmitted.

The fragment shading level 670 Processes fragment data by performing a series of operations (eg, a fragment shader or a program) on each of the fragments. The fragment shading level 670 may generate pixel data (eg, color values) for the fragment, eg, by lighting operations or scanning texture maps using interpolated texture coordinates for the fragment. The fragment shading level 670 generates pixel data that is sent to the raster operations stage 680 be transmitted.

The raster operations level 680 may perform various operations on the pixel data, such as alpha testing, template testing, and mixing the pixel data with other pixel data corresponding to other fragments associated with the pixel. When the raster operations stage 680 the processing of the pixel data (eg the output data 602 ), the pixel data may be written to a rendering target such as a frame buffer, a color buffer, or the like.

It should be appreciated that, in addition to or instead of one or more of the above-described stages, one or more additional stages may be included in the graphics processing pipeline 600 can be included. Different implementations of the abstract graphics processing pipeline may implement different phases. In addition, in some embodiments (eg, geometry shading level 640 ) one or more of the above described steps are excluded from the graphics processing pipeline. Other types of graphics processing pipelines are considered within the scope of the present disclosure. In addition, each of the stages of the graphics processing pipeline 600 by a or multiple dedicated hardware units within a graphics processor such as the PPU 300 be implemented. Further stages of the graphics processing pipeline 600 can be controlled by programmable hardware units like the SM 440 the PPU 300 will be realized.

The graphics processing pipeline 600 can be implemented via an application executed by a host processor, such as a CPU. In one embodiment, a device driver may implement an application programming interface (API) that defines various functions that may be used by an application to generate graphical data for display. The device driver is a software program that contains a variety of instructions that govern the operation of the PPU 300 control. The API provides an abstraction for a programmer that allows a programmer to use special graphics hardware like the PPU 300 to generate the graphical data without the programmer having the specific instruction set for the PPU 300 must use. The application may include an API call to the device driver for the PPU 300 is forwarded. The device driver interprets the API call and performs various operations to respond to the API call. In some cases, the device driver may perform operations by executing instructions on the CPU. In other cases, the device driver may at least partially perform operations by performing operations on the PPU 300 via an input / output interface between the CPU and the PPU 300 starts. In one embodiment, the device driver is configured to control the graphics processing pipeline 600 using the hardware of the PPU 300 to implement.

Within the PPU 300 Different programs can be run to the different stages of the graphics processing pipeline 600 to realize. For example, the device driver may have a kernel on the PPU 300 start at the vertex shading level 620 on a SM 440 (or multiple SMs 440 ). The device driver (or the initial kernel used by the PPU 400 also runs other kernels on the PPU 400 start to other stages of the graphics processing pipeline 600 such as the geometry shading level 640 and the fragment shading level 670 , In addition, some of the phases of the graphics processing pipeline 600 on solid hardware like a rasterizer or inside the PPU 400 implemented data assembler can be implemented. It should be noted that the results of a kernel may be processed by one or more intervening fixed-function hardware units before being taken from a subsequent kernel on an SM 440 are processed.

Machine learning

Deep neural networks (DNNs) working on processors like the PPU 300 have been used for a variety of applications, from self-driving cars to faster drug development, from automated captioning in online image databases to intelligent real-time language translation in video chat applications. Deep learning is a technique that models the neuronal learning process of the human brain, continuously learning, becoming smarter, and delivering faster, more accurate results over time. A child is first taught by an adult to correctly identify and classify various forms in order to ultimately identify forms without coaching. Similarly, a deep learning or neural learning system must be trained in object recognition and classification so that it can more intelligently and efficiently identify basic objects, obscured objects, and so on while assigning context to the objects.

At the simplest level, neurons in the human brain look at various inputs that are received, each of these inputs is assigned importance levels, and the output is passed to other neurons to respond. An artificial neuron or perceptron is the most basic model of a neural network. In one example, a perceptron may receive one or more inputs that represent various features of an object for which the perceptron is being trained to recognize and classify, and each of these features is assigned a particular weight based on the importance of that feature based on the definition of the shape of an object.

A deep neural network (DNN) model includes multiple layers of many connected nodes (eg, perceptions, Boltzmann machines, radial basis functions, convolutional layers, etc.) that can be trained with huge amounts of input data to solve complex problems quickly and with high accuracy to solve. In one example, a first layer of the DNN model decomposes an input image of a car into different sections and looks for basic patterns such as lines and angles. The second layer assembles the lines to look for higher-level patterns such as wheels, windshields, and mirrors. The next layer identifies the vehicle type, and the last few layers create a label for the input image that identifies the model of a particular automobile brand.

Once the DNN is trained, the DNN can be used and used to identify and classify objects or patterns in a process called inference. Examples of inferences (the process by which a DNN extracts useful information from a given input) include identifying handwritten numbers on checks deposited at ATMs, identifying pictures of friends on photos, delivering film referrals to over fifty Millions of users, identifying and classifying different types of cars, pedestrians and road hazards in driverless cars or translating human speech in real time.

During training, the data flows through the DNN in a forward propagation phase until a prediction indicative of a label corresponding to the input is made. If the neural network does not correctly identify the input, errors between the correct label and the predicted label are analyzed and the weights adjusted for each feature during a reverse propagation phase until the DNN correctly identifies the input and other inputs to a training data set. The training of complex neural networks requires enormous amounts of parallel computing power, including floating-point multiplications and additions by the PPU 300 get supported. The implication is less computationally intensive than training because it is a latency-sensitive process in which a trained neural network is applied to new inputs that have not previously existed to classify images, translate, and generally provide new information derive.

Neural networks rely heavily on mathematical matrix operations, and complex multi-layered networks require massive amounts of floating point performance and bandwidth for efficiency and speed. With thousands of cores optimized for matrix math operations, and hundreds of TFLOPS in performance, the PPU is 300 a computer platform capable of delivering the performance required for deep neural net-based artificial intelligence and machine learning applications.

Claims (21)

Computer-implemented method comprising: Receiving a first image in which a portion of image data is absent through a model of a neural network; Receiving, by the model of the neural network, a first semantic map corresponding to the first image and lacking a portion of semantic data; and Processing the first image and the first semantic map through the model of the neural network to create a second image having hallucinated image data representing a complete version of the first image. Computer-implemented method according to Claim 1 further comprising processing the first image and the first semantic map by the model of the neural network to create a second semantic map having hallucinated semantic data corresponding to the second image. Computer-implemented method according to Claim 2 further comprising training the model of the neural network using a semantic discriminator configured to evaluate the second semantic map in response to a ground truth image corresponding to the full version of the first image. Computer-implemented method according to Claim 3 , where the second semantic map is a high-resolution semantic map and the first semantic map is a low-resolution semantic map. Computer-implemented method according to Claim 4 wherein the second image is a high-resolution image and the first image is a low-resolution image, and further comprising: changing the second image to produce a second low-resolution image; Changing the second semantic map to produce a second low-resolution semantic map; and processing the second low-resolution image and the second low-resolution semantic map by the neural network model to produce an additional high-resolution image that differs from the second image in a comparison. Computer-implemented method according to Claim 4 or 5 wherein the second image is a high resolution image and the first image is a low resolution image, and further comprising: extracting a first portion of the second image to produce a low resolution image for the first portion; Extracting a second portion from the second semantic map to generate a low-resolution semantic map for the second portion, the second portion corresponding to the first portion; and processing the low-resolution image for the first portion and the low-resolution semantic map for the second portion through the model of the neural network to produce a second high-resolution image. Computer-implemented method according to Claim 6 wherein the second image and the second high-resolution image are contained in a sequence of images which, when sequentially displayed on a display device, provide a zoom effect from the second image to the first portion of the high-resolution image, if it is the second high-resolution image Picture is shown. The computer implemented method of any one of the preceding claims, wherein the second image is a high resolution image and the first image is a low resolution image. A computer-implemented method according to any one of the preceding claims, further comprising training the model of the neural network using a training data set comprising pairs of low-resolution images and low-resolution semantic maps associated with ground-truth pairs of high-resolution images and high-resolution semantic maps. The computer-implemented method of claim 1, further comprising training the model of the neural network using an image discriminator configured to image the second image according to a ground truth map corresponding to a full version of the first image. evaluate. A computer-implemented method according to any one of the preceding claims, further comprising: Receiving a keyframe image; and Generating at least one additional image in a sequence of images between the second image and the keyframe image. The computer-implemented method of any preceding claim, further comprising processing the first image and the first semantic map by the model of the neural network to create a third image having additional hallucinated image data representing a complete version of the first image wherein the third image differs in comparison with the second image. A computer-implemented method according to any one of the preceding claims, wherein the hallucinated image data is outside of a field of view associated with the first image. System comprising: a model of a neural network configured to: receive a first image in which a portion of image data is missing; receive a first semantic map that corresponds to the first image and that lacks a portion of semantic data; and process the first image and the first semantic map to create a second image having hallucinated image data representing a complete version of the first image. System after Claim 14 wherein the model of the neural network is further configured to process the first image and the first semantic map to create a second semantic map having hallucinated semantic data corresponding to the second image. System after Claim 14 or 15 wherein the second image is a high resolution image and the first image is a low resolution image. System according to one of Claims 14 to 16 wherein the model of the neural network is trained by means of a training data set comprising pairs of low-resolution images and low-resolution semantic maps associated with ground-truth pairs of high-resolution images and high-resolution semantic maps. System according to one of Claims 14 to 17 wherein the model of the neural network is further configured to process the first image and the first semantic map to create a third image having additional hallucinated image data representing a complete version of the first image, the third image is different when compared with the second image. System according to one of Claims 14 to 18 wherein the hallucinated image data is outside of a field of view associated with the first image. Non-transitory computer-readable computer instructions stored on a medium which, when executed by one or more processors, cause the one or more processors to perform the steps of: Providing a first image lacking a portion of image data, a model of a neural network; Providing a first semantic map that corresponds to the first image and that lacks a portion of semantic data, the model of the neural network; and Processing the first image and the first semantic map through the model of the neural network to create a second image having hallucinated image data representing a complete version of the first image. Non-transitory computer readable computer instructions stored on a medium which, when executed by one or more processors, cause the one or more processors to perform the steps of a method according to any one of Claims 1 to 13 To run.

Images