EP4728468A1 - Reference-driven generation for image filling - Google Patents

Abstract

Methods, systems, and apparatuses, including computer programs encoded on computer storage media, for processing a target image and reference images to generate a new image using a diffusion neural network that fills missing portions of the target image using knowledge from the reference images. As one example, image filling can include image inpainting, i.e., filling in missing or damaged portions of an image, image uncropping, i.e., extending the image beyond its original boundaries, or both. For example, restoring damaged images by filling in compromised portions of the image. As another example, removing unwanted objects from photos. As another example, extending backgrounds of images.

Description

Attorney Docket No.56113-0512WO1 REFERENCE-DRIVEN GENERATION FOR IMAGE FILLING CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Application No.63/580,687, filed on September 5, 2023, the contents of which are hereby incorporated by reference. BACKGROUND This specification relates to generating images using machine learning models. As one example, neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of weights. SUMMARY This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that performs generative image filling. That is, the system receives a target image and multiple reference images of the same scene, captured with potentially varying viewpoints, lighting conditions, camera apertures, image styles, and so on. Then, the system uses these images to fine-tune a diffusion neural network which generates a new image of the scene that completes the unfilled region of the target image while conditioned on the filled region of the target image. The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. This specification describes a generative image filling technique that allows for completing, e.g., uncropping or inpainting, regions of an image with ‘what should have been there’. Broadly, ‘image filling’ refers to replacing missing or compromised portions of the image by generating those regions based on the surrounding regions of the image that are not missing and not compromised. Specifically, this specification describes techniques for personalizing a generative model, e.g., a diffusion neural network, using few-shot reference images of a scene in order to fill (uncrop or inpaint) a target image with scene elements that are present in the reference images. Attorney Docket No.56113-0512WO1 Recent advances in generative image models allow for uncropping or inpainting regions of a target image, yet they necessarily generate hallucinations, i.e., content that does not accurately reflect the missing regions of the image, since they do not have knowledge about the contents of these regions. In contrast, the described techniques integrate knowledge from other images of the scene in order to avoid these hallucinations. As a result, the described techniques can be used to effectively perform any of a variety of image filling tasks. For example, the described techniques can be used for uncropping an image that was taken at the desired moment (or with the desired lighting, facial expressions, body poses, etc.) but is not well composed, in order to make the image have better aesthetic or other image quality. As another example, the described techniques can be used for inpainting images to fix non-aesthetic parts of the images while avoiding hallucinations. As another example, the described techniques can be used for inpainting images to fix corrupted, lost and/or noisy parts of images. Additionally, the described techniques do not require reference images to be aligned with the target image and can be used with reference images taken from drastically varying view angles, lighting conditions, or other large changes. The described techniques produce highly realistic images and are capable of harmonizing differences in lighting, style, focal distance, subject motion, and camera type between the reference images and the target image. This allows the described techniques to be widely applicable for performing image filling in any of a variety of circumstances. Moreover, the described techniques require only a small number of reference images, e.g., less than or equal to five, ten, or fifteen images, in order to generate high quality filled images, allowing the described techniques to be used in any of a variety of real-world use cases where large numbers of images of the same scene may not be available. Additionally, because only a small number of reference images are included for fine- tuning, the described techniques are computationally efficient methods for accurately filling in regions of an image that otherwise cannot be done accurately. The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. Attorney Docket No.56113-0512WO1 DESCRIPTION OF DRAWINGS FIG.1 shows an example of new images generated by the described techniques using a target image and reference images. FIG.2 shows an example reference-based image completion system. FIG.3 is a flow diagram of an example process to generate a new image of a scene that completes unfilled regions of a target image. FIG.4 is a flow diagram of an example process to fine-tune a diffusion neural network on reference images and a target image. FIG.5 shows an example of fine-tuning a diffusion neural network on reference images and a target image. FIG.6 is a flow diagram of an example process to generate, using the diffusion neural network and conditioned on the target image, an initial new image of the scene. FIG.7 shows an example of the performance of the described techniques. Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION FIG.1 shows an example 100 of new images generated by the described techniques using a target image and reference images. Example 100 shows that given a few reference images that roughly capture the same scene, and a target image with a missing region, the described techniques are able to complete the target image by generating new images (i.e., column labeled as ‘New Images’) with image content that is faithful to the true scenes. In contrast, standard prompt-based inpainting methods that attempt to fill in the target image hallucinate plausible but inauthentic content due to the lack of knowledge of the original scene, as illustrated in the column labeled ‘Stable Diffusion Inpainting’. More specifically, the top ‘New Image’ of the boy shows how the described techniques can faithfully recreate the boy’s outfit, e.g., shirt patterns, short stripes, and sandals, even though the target image does not contain that information. While the other standard method (i.e., the top ‘Stable Diffusion Inpainting’ image) generates content that is reasonable but not true to the scene, e.g., the boy’s outfit is not the true outfit, and the background deck color is not the true color. Attorney Docket No.56113-0512WO1 Similarly, the bottom ‘New Image’ of the dancer shows how the described techniques can faithfully recreate the patterns on the dancer’s dress even though the target image does not contain that information. While the other standard method (i.e., the bottom ‘Stable Diffusion Inpainting’ image) generates content that is reasonable but not true to the scene, e.g., the dress color and pattern is not the true dress color or pattern. FIG.2 shows an example reference-based image completion system 200. The reference-based image completion system 200 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The reference-based image completion system 200 is a system that receives a target image 204 and reference images 202 and uses a diffusion neural network 208 to generate a new image 210. To perform image filling, the system 200 receives data specifying a target image 204 of a scene. That is, the system 200 receives a target image 204 that includes data specifying the image. For example, the data specifying the image can be pixel values of the image. The target image 204 of the scene has a filled region and an unfilled region. For example, the system 200 can receive the filled region of the target image 204, i.e., an image that corresponds to the filled region, and data identifying where the unfilled region is located relative to the filled region within the target image 204, e.g., masks, bounding boxes, coordinates of interests, and so on. For example, the system 200 can perform image “uncropping” so that the unfilled region surrounds the filled region (on at least one side) within the target image 204. As another example, the system 200 can perform image “inpainting” so that the filled region surrounds the unfilled region within the target image 204. The system 200 then performs image filling on the target image 204 by generating a new image 210 that completes the unfilled region of the target image 204, i.e., that includes content in the unfilled region of the target image 204 that is predicted to complete the target image 204. More specifically, the system 200 also receives a set of reference images 202 of the scene, i.e., that are of the same scene as the target image 204 and that provide context for what content should be included in the completion of the unfilled region. Generally, the system 100 receives only a small number of reference images 202, e.g., less than 5, 10, or 15 images. Attorney Docket No.56113-0512WO1 The system 200 then fine-tunes a diffusion neural network 208 on the reference images 202 and the target image 204. That is, the system 200 fine-tunes, i.e., further trains, a pre-trained diffusion neural network 208 on the reference images 202 to improve the ability of the diffusion neural network 208 to generate images of the particular scene depicted in the target image 204 and the reference images 202. In some cases, the diffusion neural network 208 is pre-trained prior to fine-tuning, and, in other cases, the diffusion neural network 208 is also pre-trained to specifically perform image inpainting. After fine-tuning the diffusion neural network 208, the system 200 generates, using the diffusion neural network 208 and conditioned on the filled portion of the target image 202, an initial new image of the scene that completes the unfilled region of the target image 204. Thus, by fine-tuning the diffusion neural network 208 as described in this specification, the system 200 causes the diffusion neural network 208 to generate an initial new image that integrates context from the reference images 202 to generate a realistic completion of the target image 204. In some cases, the initial new image is the new image 210. In other cases, the system 200 further processes the initial image to generate a new image 210 of the scene that completes the unfilled region of the target image 204. The diffusion neural network 208 can be any appropriate diffusion neural network that is configured to receive an input that includes a representation of a conditioning image. In particular, the diffusion neural network 208 is configured to receive a denoising input that includes a representation of an input image, a representation of a conditioning image, e.g., target image 204, and, optionally, text tokens representing a text description for the target image and to process the input to generate a denoising output. In some cases, when the denoising input includes text tokens, the diffusion neural network 208 can generate the initial new image conditioned on a text prompt that includes the text tokens describing the content, style, or scene that guides the generation. For example, the denoising input to the diffusion neural network 208 can include a representation of a pre-determined text prompt, i.e., a user designed long prompt that describes the true scene in detail, and the denoising output of the diffusion neural network 208 can be conditioned on the text prompt. Conditioning the diffusion neural network 208 on the text prompt guides the generation of the initial new image to be more accurate and align with user expectations expressed in the text prompt. Attorney Docket No.56113-0512WO1 The text prompt can be, for example, a user natural language text description of the expectations of content contained in the filled regions of the initial new image. As a specific example, if the missing regions of a target image 204 are expected to contain a gold crown of a dancer present in reference images, the prompt can be “A photo of a dancer with a gold crown worn atop the dancer’s head.” In some implementations, the diffusion neural network 208 performs a diffusion process in pixel space, so that the images (“representations”) operated on and generated by the diffusion neural network 208 have values for each pixel that specify color values, e.g., RGB values or another color encoding scheme. In these examples, the denoising output is an estimate of the noise component of the input image, e.g., an estimate of the noise that has been added to a final image to arrive at the input image. Examples of such diffusion neural networks include Imagen as described in ArXiv:2205.11487. In some other implementations, the diffusion neural network performs a diffusion process in latent space, e.g., in a latent space that is lower-dimensional than the pixel space. That is, the images (“representations”) operated on by the diffusion neural network are latent images and the values for the pixels of the images are learned, latent values rather than color values. In these examples, the denoising output is an estimate of the noise that has been added to a latent representation of the final image in the latent space to arrive at the input latent representation in the latent space. Examples of such diffusion neural networks include Stable Diffusion, as described in ArXiv:2112.10752. In these implementations, during training, the trained diffusion neural network can be associated with an image encoder to encode images into the latent space and, during training and after training to generate new target images, a decoder neural network that receives an input that includes a latent representation of an image and decodes the latent representation to reconstruct the image. In some implementations, the denoising output is generated by a sub-neural network within the overall diffusion neural network, in which the sub-neural network is designed to process intermediate data derived from earlier layers of the diffusion neural network to generate the denoising output. For example, the sub-neural network may be a denoising neural network, e.g., U-net as described in ArXiv:1505.04597 implement within Stable Diffusion, as described in ArXiv:2112.10752. Attorney Docket No.56113-0512WO1 Further details of generating a new image of a scene that completes unfilled regions of a target image are described below with reference to FIG 3. FIG.3 is a flow diagram of an example process 300 to generate a new image of a scene that completes unfilled regions of a target image. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reference-based image completion system, e.g., the reference-based image completion system 200 of FIG.2, appropriately programmed in accordance with this specification, can perform the process 300. The system receives data specifying a target image of a scene, wherein the target image of the scene includes a filled region and an unfilled region (step 302). For example, the scene can be of a dancer on stage; the system can receive data specifying a target image in which the filled region corresponds to a portion of the dancer; and the unfilled region can correspond to an intricate crown that is cut out of the frame. The data of the filled regions of the image can include any appropriate data representing an image. For example, the data can include pixel values in the form of RGB values, i.e., red, green, blue values, for each pixel in a color image. As another example, the data can include greyscale intensity for each pixel. The data of the unfilled regions of the image can include any appropriate data representing filled and unfilled regions of an image. For example, the data can be a binary mask image that specifies pixel coordinates that are to be filled in the target image with 1 and pixel coordinates already filled in the target image with zero, or bounding boxes that specify rectangular coordinates specifying the location of missing regions in an image. The system receives a set of reference images of the scene (step 304). While these reference images are of the same scene as the target image they can, and often do, have different characteristics from the target image and from each other to an equivalent extent. For example, the reference images can include drastically varying viewpoints, lighting conditions, camera apertures, or image styles from the target image and from each other. As a particular example, the reference images can be of a dance performance where each image is taken from a different camera, different point of view from the audience location, different portion of the dance sequence, and different lighting during the different portion of the dance sequence. Attorney Docket No.56113-0512WO1 The system fine-tunes a diffusion neural network on the reference images and the target image (step 306). Fine-tuning a diffusion neural network involves initializing the diffusion neural network parameters to pre-trained values and further training the neural network on new data, i.e., reference images and the target image. Further details of an example of fine-tuning a diffusion neural network on reference images and a target image are described in more detail below with reference to FIG.4. The system generates, using the diffusion neural network and conditioned on the filled region of the target image, a new image of the scene that completes the unfilled region of the target image (step 308). In some cases, the new image is the initial new image generated by the diffusion neural network. To generate the initial new image of the scene, the diffusion neural network can iteratively denoise an initial representation through diffusion sampling. That is, the diffusion neural network begins with a sample of random noise that is a representation of the initial new image and updates the representation of the initial new image at each of a plurality of update iterations (otherwise referred to a time steps) using a learned diffusion process that involves using the denoising output at each iteration to gradually update the representation of the initial new image to generate an initial new image. Further details of an example process to generate, using the diffusion neural network and conditioned on the target image, an initial new image of the scene are described in more detail below with reference to FIG.6. In some cases, the system further processes the initial new image generated by the diffusion neural network to generate a new image of the scene that completes the unfilled region of the target image. For example, the system can generate the new image of the scene by blending the target image of the scene with the initial new image. As a particular example, the system can feather a binary mask (an image with pixels assigned a value of 1 or 0, with 1 denoting a pixel in the region with missing content and 0 denoting a pixel in the region with present content) defined by the data and use the binary mask to alpha composite the initial new image and the target image to generate the new image. That is, the system can first determine opacity levels for each pixel in the target image that includes using the binary mask. For example, the system can use the binary mask starting pixel values of 0 and 1 to define fully opaque pixels and fully transparent pixels Attorney Docket No.56113-0512WO1 respectively, and then, create a smooth gradient of opacity values, from fully opaque to fully transparent, for pixels as they approach regions to be filled from regions that are filled, i.e., feather the binary mask. Then, the system can combine the target image and initial new image based on the determined opacity values for pixels, i.e., alpha composite the images. For example, the system can apply the determined opacity levels to the target image and the complement of the determined opacity levels to the initial new image to generate the new ^{image. That is, the resulting pixel value for each pixel of the new image can be} '_{New Image Pixel Intensity'} ൌ ^^ ∗ 'Target Image Pixel Intensity' ^ ^1 െ ^^^ ∗ 'Initial New Image Pixel Intensity' where ^^ is the determined opacity level of the target image for the pixel. The resulting new image contains pixels that correspond to the filled regions of the target image and pixels that match the generated portions of the initial new image for the unfilled regions of the target image. In the new image, the system captures correspondences among the reference and input images. That is, corresponding contents of the reference images can be fused to the corresponding contents of the target image seamlessly because of simultaneous conditioning on the reference images (during fine-tuning) and conditioning on the target image (during completion of the unfilled region of the target image). That is, the system can prioritize the features in the reference and target images when completing the unfilled region of the target image with visually compelling contents that are faithful to the original scene. FIG.4 is a flow diagram of an example process 400 to fine-tune a diffusion neural network on reference images and a target image. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reference-based image completion system, e.g., the reference- based image completion system 200 of FIG.2, appropriately programmed in accordance with this specification, can perform the process 400. The system can repeatedly perform the process 400 using the reference images and the target image in order to fine-tune, i.e., further train, the diffusion neural network. The system selects, from the reference images and the target image, a set of one or more current images (step 402). Attorney Docket No.56113-0512WO1 For example, when the number of reference images exceeds 5, 10, or 15 images, the current images can be selected to be a random sample of 5, 10, and 15 reference images and the target image. As another example, when the number of reference images is equal to or less than 5, 10, or 15 images, the current images can be all reference images and the target image. The system then performs steps 404 through 410 for each current image. The system generates data specifying a random binary mask over a representation of the current image that assigns each pixel in the representation of the current image as either belonging to a filled region of the current image or an unfilled region of the current image (step 404). Here, ‘representation’ refers to the encoded form of the image data, e.g., pixel images where pixels encode color intensity values, or lower dimensional latent images where pixels encode learned latent values rather than color intensity values. For example, the system, for the current image x of the set of current images X, generates a random binary mask m over the current image by randomly selecting one or more regions of the representation of the current image to be assigned as belonging to the unfilled region of the representation of the current image. As a particular example, the system can generate multiple random rectangles for the current image x and take either their union or the complement of the union to get the final random mask ^^ where ^^ is a binary mask image with pixels assigned a value of 1 or 0, with 1 denoting a pixel in the region with missing content and 0 denoting a pixel in the region with present content. The system generates a representation of a masked current image resulting from applying the random binary mask over the representation of the current image (step 406). For example, the system can perform an element-wise product between the mask and the current image to generate the masked current image, e.g., ^1 െ ^^^⊙ ^^ where ⊙ denotes the element-wise product. That is, the elements of ^{^}1 െ ^^^{^} are 0, denoting region to fill, and 1, denoting existing region, so ^{^}1 െ ^^^{^}⊙ ^^ results in the masked current image. The system applies a noise component to the representation of the masked current image to generate a noisy representation (step 408). For example, the system can add different magnitudes of Gaussian noise to the current image ^^ to obtain noisy representation ^^′_௧ for an update iteration t, e.g., x′_^ ൌ _^α_௧x_^ ^ ^_^1 െ α_௧^^ where ^^_^ is the current image ^^ at an iteration index ^^ ൌ 0 without noise, the noise ^ is sampled from a multivariate Gaussian distribution with a mean of zero Attorney Docket No.56113-0512WO1 and covariance matrix Σ, e.g., ^ ∼ ^^^0,  Σ^ and Σ can be the identity covariance matrix, and α_௧ defines the noise magnitude. In this example, the system can randomly sample the iteration t from a range of possible values for t. The system processes a diffusion input that includes the representation of the masked current image and the noisy representation to generate a denoising output (step 410). For example, the system can use a parameterized diffusion neural network to generate the denoising output ^_^ (where θ denotes parameterized weights). Wherein the denoising output ^_^ is an estimate of a noise component of the noisy representation ^^′_௧, and ^_^ can take multiple inputs such as ^_^^ ^^′_௧, ^^, ^^, ^^, ^1 െ ^^^⊙ ^^^, where t is the update iteration index, x′_^ is the iteration’s noisy representation of the current image, ^^ is a text prompt, ^^ is the data specifying the random binary mask, and ^{^}1 െ ^^^{^}⊙ ^^ is the masked image. The system trains the diffusion neural network using, for each current image, the denoising output for the current image and the noise component for the current image (step 412). That is, the system’s update of the values of learnable parameters of the diffusion neural network can include sampling random update iteration values ^^, using the corresponding noise schedule value α_௧ to generate the noisy representation x′_^ as described above, and computing a loss function that includes the noise component of the current image and the denoising output for the current image, for each current image. To train the diffusion neural network the system: 1) defines trainable parameters, and 2) updates the values of the trainable parameters. As an example of defining the trainable parameters, the system can initialize a first set of parameters of the diffusion neural network to pre-trained values and initialize a second set of parameters of the diffusion neural network to initialized values and train the diffusion neural network on the denoising output of the current image, and the noise component of the current image, to update the initialized values while holding the pre- trained values fixed. As a particular example, the first set of parameters can involve at least a first weight matrix of the diffusion neural network, the second set of parameters can involve at least a second low-rank weight matrix and a third low-rank weight matrix, and wherein a product Attorney Docket No.56113-0512WO1 of the second low-rank weight matrix and the third low-rank weight matrix is added to the first weight matrix to generate a final weight matrix that the diffusion neural network uses. As an example of this first and second set of parameters, the system can inject weights of learnable residual modules, i.e., a second set of parameters, to one or more weight matrices of the pre-trained diffusion neural network, i.e., a first set of parameters, to perform low rank adaptations as described in ArXiv: 2106.09685. That is, learnable residual modules Δ ^^ are added to one or more weight matrices ^^, and Δ ^^is a composition of low-rank matrices, i.e., ^^ ^ Δ ^^ ൌ ^^ ^ ^^ ^^ where ^^ ∈ ℝ^{^ൈ^}, ^^ ∈ ℝ^{^ൈ^}, ^^ ∈ ℝ^{^ൈ^}, ^^ ^^ ^^ and only the added Δ ^^is being updated during training while the original parameters ^^ stay frozen. The system trains the learnable parameters of the diffusion neural network to perform fine-tuning using any appropriate loss function or objective function and any variety of optimization techniques, e.g., gradient descent methods using backpropagation. For example, the objective function can measure, for each current image, for each update iteration, and for each of a plurality of pixels in the representation of the current image, an error between a predicted noise value for the pixel in the denoising output for the current image and a noise value for the pixel in the noise component for the current image. When the current image is the target image, the plurality of pixels referred to above do not include pixels that correspond to the unfilled region in the target image. That is, during training, when the current image is the target image, the unfilled regions do not contribute to training. As a specific example of this noise error objective, the objective can satisfy: ℒ ൌ _{^^௫,௧,ఢ,^} ^‖ _^^ఏ ^{^} _{^^′௧ , ^^, ^^, ^^,} ^{^} _{1 െ ^^} ^{^} _{⊙ ^^} ^{^} _{െ ^^} ^‖ଶ ଶ _{where ℒ denotes loss function, ^^௫,ఢ,^denotes} taking an expectation, ^‖... ^‖ଶ ଶ denotes squared L-2 norm, ^^_ఏ is the denoising output of the noisy representation of the current image (in this example it is specifically an estimate of a noise component of the noisy representation) and ^^ is the noise component of the current image. FIG.5 shows an example 500 of fine-tuning a diffusion neural network on reference images and a target image. More specifically, the example 500 illustrates a reference-based image completion system fine-tuning the Low Rank Adaptations (LoRA) weights (i.e. learnable parameters 507) of a pretrained text-to-image diffusion inpainting diffusion model (i.e., Stable Diffusion V2) on the reference images 502 and target image 501. In example 500, the loss Attorney Docket No.56113-0512WO1 function 510 and its terms, i.e., ℒ ൌ ^^_{௫,௧,ఢ,^}‖ ^^_ఏ^ ^^′_௧, ^^, ^^, ^^, ^1 െ ^^^⊙ ^^^ െ ^^‖^ଶ ଶ, are as described earlier. The system determines each term of the loss function for example 500 as follows. The system sets the fixed language prompt “a photo of [V]” where [V] is a rare token describing the scene of the references images 502 and the target image 501. The system selects, from the reference images 502 and the target image 501, the current image 504 ^^. The system generates data specifying a random binary mask 506 ^^ over the representation of the current image 504 as randomly generated rectangles described above and generates the representation of a masked current image 508 ^{^}1 െ ^^^{^}⊙ ^^. The system also applies a noise ^^ to current image 504 to generate the noisy representation of the ^{current image as x′} _^ ^ൌ _^ ^α _௧ ^x _^ ^{^ ^} _^ ^{1 െ α} _௧ ^{^^.} With the system can compute the loss ℒ and update the descent methods using backpropagation. Then repeat the process for many different current images and update iterations ^^. FIG.6 is a flow diagram of an example process 600 to generate, using the diffusion neural network and conditioned on the target image, an initial new image of the scene. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reference-based image completion system, e.g., the reference-based image completion system 200 of FIG.2, appropriately programmed in accordance with this specification, can perform the process 600. The system initializes a representation of the initial new image (step 602). For example, initializing a representation of the initial new image can involve at least sampling at least some of the values in the representation from a noise distribution. For example, the noise distribution can be a multivariate Gaussian distribution with a mean of zero and an identity covariance matrix. The system then performs steps 604 and 606 at each of multiple update iterations to update the representation of the initial new image. The system processes a diffusion input that includes (i) the representation of the initial new image and (ii) a representation of the target image using the diffusion neural network to generate a denoising output (step 604). The denoising output can be an estimate of a noise component of the current update iteration’s representation of the initial new image. The system updates the representation using the denoising output (step 606). Attorney Docket No.56113-0512WO1 For example, when the denoising output is an estimate of a noise component of the current representation of the initial new image the system updates the representation by subtracting the noise. That is, every pixel of the current representation of the initial new image has a corresponding estimated noise, so the system updates the current representation of the initial new image by subtracting the estimated noise for each pixel of the initial new image. In some cases, the representation of the initial new image is in a latent space. In these cases, processing the representation of the initial new image after the last update iteration includes using a decoder neural network to generate the initial new image. That is, the system processes the representation after the last update iteration using the decoder neural network to generate the initial new image. Also, in these cases, generating the initial new image further includes processing at least the filled region of the target image using an encoder neural network to generate a first representation of the filled region of the filled region, such that the representation of the target image includes at least the first representation and a second, masked-out representation of the unfilled region. In some implementations, generating an initial new image of a scene can include “correspondence-based seed selection”. That is, the system generates a batch of initial new images of a scene, e.g., as descried above. Then, the system extracts a set of correspondences, i.e., matching local feature between images, using any appropriate method, e.g., detector-free local feature matching with transformers as described in DOI: 10.1109/CVPR46437.2021.00881, between the reference images and the regions of the new image that correspond to the unfilled regions of the target image. Next, the system ranks the batch of initial new images of scene according to the number of correspondences. and selects the image with the highest number of correspondences from among the batch of initial new images as the initial new image. FIG.7 shows an example 700 of the performance of the described techniques. Example 700 specifically illustrates a qualitative comparison of the described techniques and conventional baselines. In example 700 each row corresponds to a set of reference images (i.e. images underneath the column with the heading “Reference Images”), a target image with correctly filled missing region (i.e. the image underneath the column with the heading “Ground- Truth”), a new image generated by the described techniques (i.e. the image underneath the Attorney Docket No.56113-0512WO1 column with the heading “RealFill (Ours)”), a baseline generated new image as described in ArXiv: 2211.13227 (i.e. the image underneath the column with the heading “Paint-by- Example”), and a baseline generated new image as described in ArXiv: 2103.15982 (i.e. the image underneath the column with the heading “TransFill”). The transparent white masks are overlayed on the unaltered known regions of the target images. Example 700 illustrates a visual comparison between RealFill (i.e., the described techniques) and the baselines. RealFill not only generates high quality images, but also more faithfully reproduces the scene than the baseline methods. Paint-by-Example relies on the CLIP embedding of the reference images as the condition. This poses a challenge when dealing with complex scenes or attempting to restore object details, since CLIP embeddings only capture high-level semantic information. Although the geometric-based TransFill has decent numbers in terms of low-level metrics like PSNR, the outputs have much lower quality due to the lack of a good image prior, especially when the scene structure has complex depth variants beyond a planar surface, which is hard for homography transformations to approximate. This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly- embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is Attorney Docket No.56113-0512WO1 generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In Attorney Docket No.56113-0512WO1 some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers. The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input Attorney Docket No.56113-0512WO1 from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return. Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads. Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework. Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular Attorney Docket No.56113-0512WO1 embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations are correspond toed in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes correspond toed in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. What is claimed is:

Claims

Attorney Docket No.56113-0512WO1 CLAIMS 1. A method performed by one or more computers, the method comprising: receiving data specifying a target image of a scene, wherein the target image of the scene comprises a filled region and an unfilled region; receiving a set of reference images of the scene; fine-tuning a diffusion neural network on the reference images and the target image; and after fine-tuning the diffusion neural network: generating, using the diffusion neural network and conditioned on the filled region of the target image, a new image of the scene that completes the unfilled region of the target image. 2. The method of claim 1, wherein the diffusion neural network has been pre-trained prior to the fine-tuning. 3. The method of claim 2, wherein the diffusion neural network has been pre-trained to perform image inpainting. 4. The method of any preceding claim, wherein generating, using the diffusion neural network and conditioned on the target image, a new image of the scene that completes the unfilled region of the target image comprises: generating, using the diffusion neural network and conditioned on the target image, an initial new image of the scene. 5. The method of claim 4, wherein the initial new image is the new image. 6. The method of claim 4, further comprising: generating the new image of the scene by blending the target image of the scene with the initial new image.

Attorney Docket No.56113-0512WO1 7. The method of any one of claims 4-6, wherein generating the initial new image of the scene comprises: initializing a representation of the initial new image; and updating the representation of the initial new image at each of a plurality of update iterations, the updating comprising: processing a diffusion input comprising (i) the representation of the initial new image and (ii) a representation of the target image using the diffusion neural network to generate a denoising output; and updating the representation using the denoising output. 8. The method of claim 7, wherein initializing a representation of the initial new image comprises sampling at least some of the values in the representation from a noise distribution. 9. The method of claim 7 or 8, wherein the denoising output is an estimate of a noise component of the representation of the initial new image. 10. The method of any one of claims 7-9, wherein the representation of the initial new image is in a latent space and wherein generating the initial new image of the scene further comprises: processing the representation of the initial new image after the last update iteration using a decoder neural network to generate the initial new image. 11. The method of claim 10, wherein the representation of the target image is in the latent space and wherein generating the initial new image further comprises: processing at least the filled region of the target image using an encoder neural network to generate a first representation of the filled region of the filled region, wherein the representation of the target image comprises the first representation and a second, masked- out representation of the unfilled region.

Attorney Docket No.56113-0512WO1 12. The method of any preceding claim, wherein fine-tuning the diffusion neural network on the reference images and the target image comprises: selecting, from the reference images and the target image, a set of one or more current images; for each current image: generating data specifying a random binary mask over a representation of the current image that assigns each pixel in the representation of the current image as either belonging to a filled region of the current image or an unfilled region of the current image; generating a representation of a masked current image resulting from applying the random binary mask over the representation of the current image; applying a noise component to the representation of the masked current image to generate a noisy representation; and processing a diffusion input comprising the representation of the masked current image and the noisy representation to generate a denoising output; and training the diffusion neural network using, for each current image, the denoising output for the current image and the noise component for the current image. 13. The method of claim 12, wherein generating a random binary mask over the current image comprises randomly selecting one or more regions of the representation of the current image to be assigned as belonging to the unfilled region of the representation current image. 14. The method of claim 12 or claim 13, wherein training the diffusion neural network using, for each current image, the denoising output for the current image and the noise component for the current image comprises training the diffusion neural network on an objective that measures, for each current image and for each of a plurality of pixels in the representation of the current image, an error between a predicted noise value for the pixel in the denoising output for the current image and a noise value for the pixel in the noise component for the current image. 15. The method of claim 14, wherein, when the current image is the target image, the plurality of pixels do not include pixels that correspond to the unfilled region in the target image. Attorney Docket No.56113-0512WO1 16. The method of any preceding claim, wherein fine-tuning the diffusion neural network comprises: initializing a first set of parameters of the diffusion neural network to pre-trained values; initializing a second set of parameters of the diffusion neural network to initialized values; and training the diffusion neural network on the reference images and the target image to update the initialized values while holding the pre-trained values fixed. 17. The method of claim 16, wherein the first set of parameters comprises a first weight matrix of the diffusion neural network, the second set of parameters comprises a second low-rank weight matrix and a third low-rank weight matrix, and wherein a product of the second low-rank weight matrix and the third low-rank weight matrix is added to the first weight matrix to generate a final weight matrix that is used by the diffusion neural network. 18. The method of any preceding claim, wherein each diffusion input to the diffusion neural network includes a representation of a pre-determined text prompt. 19. The method of any preceding claim when dependent on claim 12, wherein the diffusion input further comprises data specifying the random binary mask. 20. The method of any preceding claim when dependent on claim 7, wherein the diffusion input further comprises data specifying the unfilled region of the target image. 21. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the method of any one of claims 1-20. 22. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-20.