EP4728401A1 - Progressive classifier-free guidance for image synthesis - Google Patents

Abstract

Methods and systems for generating an image are disclosed herein. The method can include receiving an input condition embedding from an encoder, the input condition embedding including a text condition and a token representing a target subject for a generated image, and generating a weakened condition embedding by replacing the token representing the target subject with an embedding representing a generic word describing the target subject. The method also includes performing at least one first denoising step based on the weakened condition embedding to receive a first output, performing at least one second denoising step based on the input condition embedding and the first output to receive a second output, and generating a generated image based on at least the second output.

Description

PROGRESSIVE CLASSIFIER-FREE GUIDANCE FOR IMAGE SYNTHESIS

PRIORITY CLAIM

[0001] The present application is based on and claims priority to United States Provisional Application 63/513,642 having a filing date of uly 14, 2023, which is incorporated by reference herein.

[0002] The present disclosure relates generally to image synthesis using machine-learned models. More particularly, the present disclosure relates to fixing token domination for personalized text-to-image synthesis by utilizing progressive classifier-free guidance.

BACKGROUND

[0003] Generative artificial intelligence, or generative Al, can be used to create novel images based on various inputs, such as text prompts, prompt images, and the like. This is performed using a machine-learned model, such as a diffusion model, that is trained to generate these images using both the inputs and random Gaussian noise. However, the synthesis process of generating these new images based on both text inputs and image inputs can be dominated by a target subject in the input image because existing personalized text-to-image synthesis networks are designed to overfit image generation to the target subject. This results in generated images that ignore important context in the input text, especially when the context information is encapsulated as a text token.

SUMMARY

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0005] One example aspect of the present disclosure is directed to a computer-implemented method for generating an image. The method can include receiving, by a processor, an input condition embedding from an encoder, the input condition embedding including a text condition and a token representing a target subject for a generated image and generating, by the processor, a weakened condition embedding by replacing the token representing the target subject with an embedding representing a generic word describing the target subject. The method can also include performing, by the processor, at least one first denoising step based on the weakened condition embedding to receive a first output, performing, by the processor, at least one second denoising step based on the input condition embedding and the first output to receive a second output, and generating, by the processor, a generated image based on at least the second output.

[0006] Another example aspect of the present disclosure is directed to a computing system that includes one or more processors and a non-transitory, computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations. The operations can include receiving an input condition embedding from an encoder, the input condition embedding including a text condition and a token representing a target subject for a generated image and generating a weakened condition embedding by replacing the token representing the target subject with an embedding representing a generic word describing the target subject. The operations can also include performing a first denoising step based on the weakened condition embedding to receive a first output, performing a second denoising step based on the input condition embedding to receive a second output, and generating the generated image based on at least the first output and the second output.

[0007] Another example aspect of the present disclosure is directed to a non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations. The operations can include receiving an input condition embedding from an encoder, the input condition embedding including a text condition and a token representing a target subject for a generated image and generating a weakened condition embedding by replacing the token representing the target subject with an embedding representing a generic word describing the target subject. The operations can also include performing a first denoising step based on the weakened condition embedding to receive a first output, performing a second denoising step based on the input condition embedding to receive a second output, and generating the generated image based on at least the first output and the second output.

[0008] Other aspects of the present disclosure are directed to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0009] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles. BRIEF DESCRIPTION OF THE DRAWINGS

[00010] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which: [00011] Figures 1 A and IB depict a block diagram of an example image synthesis model according to example embodiments of the present disclosure.

[00012] Figure 2 depicts a flow chart diagram of an example method for generating an image according to example embodiments of the present disclosure.

[00013] Figure 3 depicts example experimental data according to example embodiments of the present disclosure.

[00014] Figure 4A depicts a block diagram of an example computing system according to example embodiments of the present disclosure.

[00015] Figure 4B depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[00016] Figure 4C depicts a block diagram of an example computing device according to example embodiments of the present disclosure.

[00017] Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations.

DETAILED DESCRIPTION

Overview

[00018] Generally, the present disclosure is directed to fixing token domination for personalized text-to-image synthesis with progressive classifier-free guidance.

[00019] Personalized text-to-image synthesis aims at generating diverse image samples conditioning on text descriptions as well as sample images of the target subject. However, existing solutions are designed to overfit the target subject present in the sample input images (such as a particular person, animal, or object). Therefore, these target subjects often dominate the synthesis process, which results in accurate recreations of the target subject from the input image at the expense of ignoring other context that was input for image generation, such as text inputs that were submitted alongside the input images. For example, if the target subject is a particular dog, such as a poodle, the resulting images will recreate the dog very accurately, but will ignore other context input into the model, such as a text prompt saying “a dog swimming, drawn in the style of a famous artist.” The features of the particular dog are accurately recreated, but the context from the text prompt, such as “drawn in the style of the famous artist,” is lost during image synthesis, resulting in an image depicting the dog swimming, but lacking details and features that align with the condition “drawn in the style of the famous artist.” The fine-tuned networks of existing solutions are unable to attend to the additional content from the text prompt in the presence of much more learnable tokens, such as the detailed features of the target subject in the input image.

[00020] Aspects of the present disclosure solve this issue by utilizing progressive classifier-free guidance. Attention to the content of other conditions is encouraged by suppressing details and features of a learnable token of the target subject (e.g., the input image), which is achieved by classifier-free guidance.

[00021] Conventional classifier-free guidance utilizes null conditions. For example, a single neural network can be used to parameterize both an unconditional denoising diffusion model and a conditional model using zeroes for classifier values when predicting an output score from each model. The unconditional model and conditional model can be jointly trained by randomly setting the classifier values to the unconditional class identifier (zero). [00022] In contrast, one aspect of the present disclosure includes using a “weakened” description to guide context synthesis. Specifically, each iteration of reverse diffusion includes multiple rounds of classifier-free guidance. The given condition and the weakened condition are used in a first round of classifier-free guidance, and the output of this round is then used in a second round (and/or further round) along with the null condition to perform reverse diffusion during the training process.

[00023] Another aspect of the present disclosure includes using an encoder framework for personalized text-to-image synthesis. Because of the use of classifier-free guidance, the encoder framework does not require additional attention modules, which allows for a pretrained synthesis network to be kept intact. In some aspects of the present disclosure, only the encoder network and the cross-attention layers in the synthesis network are trainable. In other aspects, one or more other portions of the synthesis network, or the network in its entirety, can be trained. To encourage learning of subject information, a domain-specific text-image dataset can be used, where the text caption consists of a special token, such as a text-image dataset that includes a plurality of different, unique photos each with similar or identical captions, such as “an image of (special token).” During training, the “(special token)” term is replaced with an output of the encoder framework, and the resulting condition is sent to a text encoder of a synthesis network.

[00024] Aspects of the present disclosure enable image synthesis models to fix the problem of over-fitting to a target subject from an input image by utilizing classifier-free guidance. The token domination problem severely limits the diversity of image synthesis. The present disclosure includes aspects that add a simple, yet effective, method for addressing this generative diversity issue without the use of complex, heavy modules. Additionally, context consistency and subject fidelity can be balanced by the use of weakened conditions in the classifier-free guidance, which facilitates personalized and diverse images being generated in one single forward pass and requiring only a text prompt and a single reference image.

[00025] With reference now to the Figures, example embodiments of the present disclosure will be discussed in further detail.

Example Model Arrangements

[00026] Figures 1 A and IB depict block diagrams of an example image synthesis model 100 according to example embodiments of the present disclosure. The image synthesis model 100 can include a denoising diffusion model 102 and an encoder 104. [00027] The denoising diffusion model 102 iteratively denoises a Gaussian noise distribution to generate an image. The denoising diffusion model 102 can, in some embodiments, receive conditioning inputs, such as text inputs or image inputs, that apply conditions to the generated image, such as receiving a text prompt “A cup on a table” and generating an image of a cup placed on a table.

[00028] During training of the denoising diffusion model 102, a training image, such as the image input, can be provided to the denoising diffusion model 102. A noisy image is obtained by adding Gaussian noise to the training image in iterative steps. The denoising diffusion model 102 can be trained to predict the added noise given the noisy image and one or more conditions (e.g., a text a prompt or an input image), and the denoising diffusion model 102 can be optimized using a denoising objective. In some embodiments, text inputs to the denoising diffusion model 102 can include pseudo-words, whose embedding is replaced by a learnable subject embedding. For example, the pseudo-word can be represented by a term, such as S*, and are used to replace subjects in a text embedding, such as “a dog.” In this example, the text input may be “a dog swimming,” and the input with the pseudo-word replacing the subject may be “S* swimming.” In some embodiments, the pseudo-word S* can be referred to as a learnable token, and can be used to represent an input image of a specific subject to be used in generating the photo, such as a specific picture of a particular object. [00029] During actual image generation (or “inference”), the denoising diffusion model 102 starts with pure Gaussian noise, and the trained denoising diffusion model 102 can be iteratively applied to the Gaussian noise to obtain a series of intermediate outputs until a final output (the generated image) is generated.

[00030] The denoising diffusion model 102 can use classifier-free guidance to generate an image. During training, the denoised diffusion model 102 can be jointly trained by randomly replacing an input condition with a null condition (e.g., no condition). Once trained, during each iterative step in the denoising process, a weighted sum of the output generated using the input condition and the output generated using the unconditional (null) output can be computed using, for example, Equation 1.

[00031] Equation 1 : e_t = (1 + w) • e (z_t , c - w ■ e (z_t, <p).

[00032] In Equation 1, c is the input condition, <p is the null condition (unconditional), t is the iteration number, and w is a weight, where a larger w produces better quality and a smaller w produces larger diversity.

[00033] Progressive classifier-free guidance can be used to alleviate token domination in the generative process. Progressive classifier-free guidance replaces the null condition as an input in classifier-free guidance with a “weakened condition.” Progressive classifier-free guidance can include three stages.

[00034] First, a weakened condition can be constructed. A weakened condition is a less-specific condition than an input condition, but is not a null condition. In some implementations, the weak condition can comprise only natural language so that generation is not dominated by any learnable component, such as an input image. In one example, a text condition is used as an input to the denoising diffusion model 102 and includes a learnable token S*. The weakened condition c₀ can be constructed by replacing the learnable token S* with a (relatively more) generic word. In other words, in one example, the learnable token S* (e.g., an embedding representing a particular subject for the image) is replaced with an embedding of a generic term representing the particular subject. In one example, the text condition can be “a pencil sketch of S*,” where S* is the embedding representing the particular subject. The weakened condition c₀ can be generated by replacing S* with a noun accurately describing the particular subject. For example, if the particular subject is a particular woman from an input image, the weakened condition c₀ can be generated as “a pencil sketch of a woman.” [00035] The weakened condition c₀ can be used in place of the null condition in Equation 1. By replacing the null condition with the weakened condition c₀, image generation by the image synthesis model 100 can be guided toward a subject-agnostic generation. In absence of the learnable token S*, image synthesis model 100 can synthesize correct outline and structure for the image without domination by the learnable token S* erasing other context in the text condition. The replacement of the null condition with the weakened condition is shown in Equation 2.

[00036] Equation 2: e_t = (1 + w) • e (z_t , c) - w ■ e (z_t, c₀)

[00037] Additionally, unlike conventional classifier-free guidance, where the guidance weight w is generally fixed during the denoising process, a time-varying weighting scheme can be used for better performance. In particular, weights can be emphasized on the weak condition in earlier stages to allow for better structure and representation of other portions of the text condition, while attention to the learnable token S* can be used in later stages to add unique features from the learnable token S* to the structure of the generated image. This weighting scheme is shown in Equation 3.

[00038] Equation 3: w_t = {r if 0 < t < T — 1 if T < t < 1

[00039] T can be a predetermined constant. As shown in Equation 3, at earlier stages (T approximately equal to 1), image generation relies only on the weak condition for building structure and outline of the output, and then particular subject information for identity preservation is introduced at later stages of image generation.

[00040] In some embodiments, null-classifier-free guidance can be used to encourage diversity by leveraging the null condition during image generation. A constant guidance weight w can be used throughout iterations. In particular, the output of weak classifier-free guidance e_t can be used in place of conventional classifier free guidance along with the null condition, as shown in Equation 4.

[00041] Equation 4: e_t = (1 + w) ■ e_t - w ■ e (z_t,<p).

[00042] Combining the output of weak classification at one iteration with the null condition allows for more diversity in the resulting final output image.

[00043] The encoder 104 is used to obtain embeddings, or tokens, representing inputs into the denoising diffusion model 102. In some embodiments, the encoder 104 can be trained (along with cross-attention layers of the denoising diffusion model 102) using a domain-specific text-image dataset whose text caption consists of learnable tokens. Images in the domain-specific text-image dataset include captions with simple templates, such as “a photo of S*”, where S* is an entity in a domain such as “animals.” During training, the token corresponding to S* is replaced by the output of the encoder 104, and the resulting condition is then passed to the text encoder.

[00044] In some embodiments, in addition to training on the domain-specific textimage dataset, the encoder 104 can be trained using a general-domain dataset containing rich text descriptions as captions. By training with highly diverse text captions, the capability of the encoder 104 on text-understanding can be preserved. During training, samples from both the domain-specific text-image dataset and the general domain dataset can be used for regularization.

[00045] During inference by the image synthesis model 100, the encoder 104 is used to map the target subject (from an input image) into a text token, which is the learnable token S*. This learnable token then is combined with the text description (a text prompt) to form the input condition. From this input condition, a weak condition can be constructed following the process described above, such as replacing a specific subject with the learnable token S*. The image synthesis model 100 can then perform image synthesis using both the weighted input condition and the weighted weakened condition in tandem to generate an output image from Gaussian noise.

[00046] The use of encoder 104 improves synthesis results without needing auxiliary modules. Instead, an additional pass of classifier-free guidance can be used with modules from the image synthesis model 100, which reduces both development efforts for the image synthesis model 100 as well as reducing the amount of computational storage and processing necessary for image synthesis.

[00047] The denoising diffusion model 102 processes the output of the encoder 104 to generate a final output image that includes unique features of the particular subject without losing the context from the text condition (e.g., the particular subject being drawn in a specific style or manner).

Example Methods

[00048] Figure 2 depicts a flow chart diagram of an example method 200 to perform according to example embodiments of the present disclosure. Although Figure 2 depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method 200 can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the present disclosure.

[00049] At 202, a computing system receives an input condition embedding from an encoder, the input condition embedding including a text condition and a token representing a target subject for a generated image. In some embodiments, the encoder can receive a text input and an image input via user input. For example, a user can input an image of a dog and a text input (as a text prompt) that recites “a dog swimming in a pool.” The encoder can generate a text condition, such as an embedded vector, based on the text input.

[00050] In some embodiments, the encoder can also generate the token representing the target subject from the input image, such as the input image of the dog. The token (e.g., “S*”) can be learnable token as described above. Based on the generated text condition and token, the encoder can then combine the text condition and the token and create the input condition embedding that includes both an embedding of the token and the text embedding.

[00051] In some embodiments, the encoder can be trained using a combination of a domain-specific text-image dataset and a generic-domain dataset. The domain-specific textimage dataset can include text condition-input image pairs in a particular domain, such as “animals” or “people.” The generic-domain dataset can include a variety of text conditioninput image pairs that are not domain-specific. The use of both the domain-specific textimage dataset and the generic-domain dataset enables the diffusion model to be trained for specific domains while being more robust than a model trained using only domain-specific training sets due to the inclusion of the generic-domain set as a training set.

[00052] At 204, the computing system generates a weakened condition embedding by replacing the token representing the target subject with an embedding representing a generic word describing the target subject. As described above, this weakened condition can be an embedded vector that represents both the text condition and the generic word describing the target subject (e.g., using “dog”). In some embodiments, the weakened condition can include an embedded vector of only natural language phrases.

[00053] At 206, the computing system performs one or more first denoising steps based on the weakened condition embedding to receive a first output. In some embodiments, performing a first denoising step can include determining a predicted first noise based on the weakened condition embedding. This predicted first noise can be a predicted noise that occurs in the initial Gaussian noise image that would have been added during diffusion based on the embedding of the weakened condition. The denoising model can then remove the noise from the Gaussian noise image to generate the first output, which can be a partially denoised image that includes the structure of the image (e.g., a dog in a specific pose or performing a specific action), allowing the text input to have more of an effect on the initial stages of denoising to obtain the desired effects set forth in the text input without the target subject (e.g., a specific dog) dominating the denoising process. At 208, the computing system performs one or more second denoising steps based on the input condition embedding to receive a second output. In some embodiments, performing the one or second denoising steps can include determining a predicted second noise based on the input condition embedding. For example, after the first denoising step is performed to obtain a partially denoised image based on the weakened condition embedding, the second denoising step can utilize the partially denoised image as an image input and the input condition embedding to perform denoising on the partially denoised image. The input condition embedding can include an embedding representation of the learnable token that replaced the target subject (e.g., a picture of a specific dog) and the text embedding representing the text input. The use of the input condition embedding at this stage enables the specific input image to be added to the denoising process, thus placing the target subject into the iteratively denoised images (e.g., placing the specific dog in the input image in the correct pose or performing a specific action) without letting the target subject dominate denoising and lose the context provided in the text input. In some embodiments, the computing system can determine a final noise based on the first noise and the second noise. For example, during each iteration of , this final noise can be a weighted subtraction of a conditional output from a conditional diffusion model and an unconditional output from an unconditional diffusion model (such as the weighted output described in Equation 4).

[00054] In some embodiments, the weight in the weighted subtraction can be a progressive weight value defined by a weighting schedule. This weighting schedule can, in one embodiment, provide less weight to the conditional output (e.g., the output from the conditional diffusion model using the learned token embedding representing the target subject) in early iterations of diffusion to allow for the context of the text input to be considered in the diffusion, instead of the target subject dominating diffusion. As additional iterations of diffusion occur, the weighting schedule can begin to increase the weight for the conditional output, thus allowing the target subject in the input image to be more highly influential in the diffusion process. Using this weighting schedule, the computing system can allow for the structure and context of the text input to be initially considered in early diffusion iterations, and then allow the target subject to be added into the diffusion to achieve a more desirable output image (e.g., the target subject being placed in the correct context indicated by the text input).

[00055] In some embodiments, determining the final noise can also include determining a third noise based on a null condition embedding. For example, the null condition embedding can represent a partially denoised image as an input with no extra conditions (e.g., no text embeddings). The computing system can determine the final noise based on a weighted subtraction of the first noise and the third noise from the second noise.

[00056] At 210, the computing system generates the generated image based on at least the first output and the second output. For example, the generated image can include both the context indicated by the text input (such as a pencil sketch of a dog) and the target subject (a specific dog shown in an input image). without the target subject dominating the image generation process and ignoring the context of the text input.

Example Results

[00057] Figure 3 depicts various reference images (column 300), associated text conditions (rows 305, 310, 315, and 320), and resulting output images. As shown in Figure 3, based on reference images 300, subjects in text prompts can be replaced with tokens and the context of the remaining text prompt can be more accurately portrayed in the resulting output images.

[00058] In one example, row 305 illustrates a reference image of a rabbit and various input condition embeddings, such as “S* on snow” or “S*, night sky, oil painting.” These input condition embeddings can be transformed into weakened condition embeddings by replacing, for example, the “S*” term with a generic term, such as “rabbit” or “bunny.” Early diffusion iterations during image generation can use this weakened condition embedding to achieve the context indicated by the text input, such as “on snow” or “night sky, oil painting” before later iterations add in a heavier weighted input image (such as the reference image) to add in the target subject in the diffusion process.

[00059] Other examples are show in the other rows, such as an input image of a husky in row 310, an input image of a corgi in row 315, and an input image of a cat in row 320, along with various text prompts that include the learnable token “S*” for generating output images.

Example Devices and Systems

[00060] Figure 4A depicts a block diagram of an example computing system 400 that performs image synthesis according to example embodiments of the present disclosure. The system 400 includes a user computing device 402, a server computing system 430, and a training computing system 450 that are communicatively coupled over a network 480. [00061] The user computing device 402 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[00062] The user computing device 402 includes one or more processors 412 and a memory 414. The one or more processors 412 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 414 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 414 can store data 416 and instructions 418 which are executed by the processor 412 to cause the user computing device 402 to perform operations. [00063] In some implementations, the user computing device 402 can store or include one or more image synthesis models 420. For example, the image synthesis models 420 can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example image synthesis models 120 are discussed with reference to Figures 1 A and IB.

[00064] In some implementations, the one or more image synthesis models 420 can be received from the server computing system 430 over network 480, stored in the user computing device memory 414, and then used or otherwise implemented by the one or more processors 412. In some implementations, the user computing device 402 can implement multiple parallel instances of a single image synthesis model 420 (e.g., to perform parallel image synthesis across multiple instances).

[00065] More particularly, the one or more image synthesis models 420 are configured to use Gaussian noise to generate novel images and/or modified images, sometimes based on conditions, such as text prompts or image prompts.

[00066] Additionally or alternatively, one or more image synthesis models 440 can be included in or otherwise stored and implemented by the server computing system 430 that communicates with the user computing device 402 according to a client-server relationship. For example, the image synthesis models 440 can be implemented by the server computing system 440 as a portion of a web service (e.g., an image synthesis service). Thus, one or more models 420 can be stored and implemented at the user computing device 402 and/or one or more models 440 can be stored and implemented at the server computing system 430.

[00067] The user computing device 402 can also include one or more user input components 422 that receives user input. For example, the user input component 422 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[00068] The server computing system 430 includes one or more processors 432 and a memory 434. The one or more processors 432 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 434 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 434 can store data 436 and instructions 438 which are executed by the processor 432 to cause the server computing system 430 to perform operations.

[00069] In some implementations, the server computing system 430 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 430 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof. [00070] As described above, the server computing system 430 can store or otherwise include one or more image synthesis models 140. For example, the models 440 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multi-headed self-attention models (e.g., transformer models). Example models 440 are discussed with reference to Figures 1 A and IB.

[00071] The user computing device 402 and/or the server computing system 430 can train the models 420 and/or 440 via interaction with the training computing system 450 that is communicatively coupled over the network 480. The training computing system 450 can be separate from the server computing system 430 or can be a portion of the server computing system 430.

[00072] The training computing system 450 includes one or more processors 452 and a memory 454. The one or more processors 452 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 454 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 454 can store data 456 and instructions 458 which are executed by the processor 452 to cause the training computing system 450 to perform operations. In some implementations, the training computing system 450 includes or is otherwise implemented by one or more server computing devices.

[00073] The training computing system 450 can include a model trainer 460 that trains the machine-learned models 420 and/or 440 stored at the user computing device 402 and/or the server computing system 430 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and/or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations. [00074] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 460 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[00075] In particular, the model trainer 460 can train the image synthesis models 420 and/or 440 based on a set of training data 462. The training data 462 can include, for example, images that are destroyed by the model(s) 420 and 440 by adding Gaussian noise and then attempting to recreate the original image.

[00076] In some implementations, if the user has provided consent, the training examples can be provided by the user computing device 402. Thus, in such implementations, the model 420 provided to the user computing device 402 can be trained by the training computing system 450 on user-specific data received from the user computing device 402. In some instances, this process can be referred to as personalizing the model.

[00077] The model trainer 460 includes computer logic utilized to provide desired functionality. The model trainer 460 can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer 460 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 460 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media. [00078] The network 480 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network 480 can be carried via any type of wired and/or wireless connection, using a wide variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).

[00079] The machine-learned models described in this specification may be used in a variety of tasks, applications, and/or use cases.

[00080] In some implementations, the input to the machine-learned model(s) of the present disclosure can be image data. The machine-learned model(s) can process the image data to generate an output. As an example, the machine-learned model(s) can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an image segmentation output. As another example, the machine- learned model(s) can process the image data to generate an image classification output. As another example, the machine-learned model(s) can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine-learned model(s) can process the image data to generate an upscaled image data output. As another example, the machine-learned model(s) can process the image data to generate a prediction output.

[00081] In some implementations, the input to the machine-learned model(s) of the present disclosure can be text or natural language data. The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine- learned model(s) can process the natural language data to generate a language encoding output. As another example, the machine-learned model(s) can process the text or natural language data to generate a latent text embedding output. As another example, the machine- learned model(s) can process the text or natural language data to generate a translation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a textual segmentation output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is higher quality than the input text or natural language, etc.). As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output.

[00082] In some implementations, the input to the machine-learned model(s) of the present disclosure can be latent encoding data (e.g., a latent space representation of an input, etc.). The machine-learned model(s) can process the latent encoding data to generate an output. As an example, the machine-learned model(s) can process the latent encoding data to generate a recognition output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reconstruction output. As another example, the machine-learned model(s) can process the latent encoding data to generate a search output. As another example, the machine-learned model(s) can process the latent encoding data to generate a reclustering output. As another example, the machine-learned model(s) can process the latent encoding data to generate a prediction output.

[00083] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task can be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the likelihood that the one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images and, for each region, a likelihood that region depicts an object of interest. As another example, the image processing task can be image segmentation, where the image processing output defines, for each pixel in the one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories can be foreground and background. As another example, the set of categories can be object classes. As another example, the image processing task can be depth estimation, where the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task can be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, a motion of the scene depicted at the pixel between the images in the network input.

[00084] Figure 4A illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include the model trainer 160 and the training dataset 162. In such implementations, the models 120 can be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 can implement the model trainer 160 to personalize the models 120 based on user-specific data.

[00085] Figure 4B depicts a block diagram of an example computing device 10 that performs according to example embodiments of the present disclosure. The computing device 10 can be a user computing device or a server computing device.

[00086] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine- learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. [00087] As illustrated in Figure 4B, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[00088] Figure 4C depicts a block diagram of an example computing device 50 that performs according to example embodiments of the present disclosure. The computing device 50 can be a user computing device or a server computing device.

[00089] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer.

Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[00090] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in Figure 4C, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[00091] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device 50. As illustrated in Figure 4C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

Additional Disclosure

[00092] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[00093] While the present subject matter has been described in detail with respect to various specific example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method for generating an image, the method comprising: receiving, by a processor, an input condition embedding from an encoder, the input condition embedding including a text condition and a token representing a target subject for a generated image; generating, by the processor, a weakened condition embedding by replacing the token representing the target subject with an embedding representing a generic word describing the target subject; performing, by the processor, at least one first denoising step based on the weakened condition embedding to receive a first output; performing, by the processor, at least one second denoising step based on the input condition embedding and the first output to receive a second output; and generating, by the processor, a generated image based on at least the second output.

2. The method of claim 1, further comprising: receiving, by the processor, a text input and an image input; providing, by the processor, the text input and the image input to the encoder; generating, by the processor, the text condition using the encoder based on the text input; generating, by the processor, the token representing the target subject using the encoder based on the image input; and generating, by the processor, the input condition embedding by combining the text condition and the token representing the target subject.

3. The method of claim 1, wherein the encoder is trained using a combination of a domain-specific text-image dataset and a generic-domain dataset.

4. The method of claim 1, wherein the at least one first denoising step comprises determining, by the processor, a predicted first noise based on the weakened condition embedding.

5. The method of claim 4, wherein the at least one second denoising step comprises determining, by the processor, a predicted second noise based on the input condition embedding.

6. The method of claim 5, the method further comprising determining, by the processor, a final noise based on the predicted first noise and the predicted second noise.

7. The method of claim 6, wherein determining the final noise comprises a weighted subtraction of the predicted first noise from the predicted second noise.

8. The method of claim 7, wherein at least one weight in the weighted subtraction is a progressive weight value defined by a weighting schedule.

9. The method of claim 6, wherein determining the final noise comprises: determining, by the processor, a third noise based on a null condition embedding; and performing, by the processor, a weighted subtraction of the first noise and the third noise from the second noise.

10. A computing system, comprising: one or more processors; and a non-transitory, computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising receiving an input condition embedding from an encoder, the input condition embedding including a text condition and a token representing a target subject for a generated image; generating a weakened condition embedding by replacing the token representing the target subject with an embedding representing a generic word describing the target subject; performing a first denoising step based on the weakened condition embedding to receive a first output; performing a second denoising step based on the input condition embedding to receive a second output; and generating the generated image based on at least the first output and the second output.

11. The computing system of claim 10, further comprising: receiving a text input and an image input; providing text input and the image input to the encoder; generating the text condition using the encoder based on the text input; generating the token representing the target subject using the encoder based on the image input; and generating the input condition embedding by combining the text condition and the token representing the target subject.

12. The computing system of claim 10, wherein the encoder is trained using a combination of a domain-specific text-image dataset and a generic-domain dataset.

13. The computing system of claim 10, wherein the at least one first denoising step comprises determining a predicted first noise based on the weakened condition embedding.

14. The computing system of claim 13, wherein the at least one second denoising step comprises determining a predicted second noise based on the input condition embedding.

15. The computing system of claim 14, the operations further comprising determining a final noise based on the predicted first noise and the predicted second noise.

16. The computing system of claim 15 wherein determining the final noise comprises a weighted subtraction of the predicted first noise from the predicted second noise.

17. The computing system of claim 16 wherein at least one weight in the weighted subtraction is a progressive weight value defined by a weighting schedule.

18. The method of claim 15, wherein determining the final noise comprises: determining a third noise based on a null condition embedding; and performing a weighted subtraction of the first noise and the third noise from the second noise.

19. A non-transitory, computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising receiving an input condition embedding from an encoder, the input condition embedding including a text condition and a token representing a target subject for a generated image; generating a weakened condition embedding by replacing the token representing the target subject with an embedding representing a generic word describing the target subject; performing a first denoising step based on the weakened condition embedding to receive a first output; performing a second denoising step based on the input condition embedding to receive a second output; and generating the generated image based on at least the first output and the second output.

20. The non-transitory, computer-readable medium of claim 19, wherein the first denoising step comprises determining a predicted first noise based on the weakened condition embedding; the second denoising step comprises determining a predicted second noise based on the input condition embedding; and wherein the operations further comprise determining a final noise based on the predicted first noise and the predicted second noise, wherein determining the final noise comprises a weighted subtraction of the predicted first noise from the predicted second noise.