EP4699097A2 - Generative artificial intelligence - Google Patents

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating images. An initial prompt is generated using a specified network location and a set of constraints that limits clauses generated by a language model. The set of constraints specifies one or more objects excluded from images generated using the clauses generated by the language model. Multiple images are generated using an image prompt that includes the clauses generated by the language model. One or more post-processing operations that evaluate one or more visual characteristics of the multiple images are performed. Each of the multiple images is ranked based on the post-processing operations. At least one output image from a set of highest ranked images is served.

Description

GENERATIVE ARTIFICIAL INTELLIGENCE

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63/503,688, filed on May 22, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.

BACKGROUND

[0002] This specification relates to data processing and generative artificial intelligence. [0003] Advances in machine learning are enabling artificial intelligence to be implemented in more applications. For example, large language models have been implemented to allow for a conversational interaction with computers using natural language rather than a restricted set of prompts. This allows for a more natural interaction with the computer.

SUMMARY

[0004] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of generating, by an artificial intelligence system, an initial prompt using a specified network location and a set of constraints that limits clauses generated by a language model, wherein the set of constraints specifies one or more objects excluded from images generated using the clauses generated by the language model; generating multiple images using an image prompt that includes the clauses generated by the language model; performing, by the artificial intelligence system, one or more post-processing operations that evaluate one or more visual characteristics of the multiple images; ranking, by the artificial intelligence system, each of the multiple images based on the post-processing operations; and serving, by the artificial intelligence system, at least one output image that is in a set of highest ranked images.

[0005] These and other embodiments can each optionally include one or more of the following features. Methods can include collecting textual content from the specified network location; summarizing the textual content collected from the network location in a text summary; generating, based on the text summary, an image summary specifying visual characteristics of objects specified for inclusion in images generated by a text-to-image generative model; and generating the image prompt that includes the image summary.

[0006] Generating multiple candidate images comprises causing the text-to-image generative model to create the multiple candidate images using the image prompt.

[0007] Generating the image prompt can include causing the language model to respond to the initial prompt with only the image prompt in a format of a specified programming language.

[0008] Generating the initial prompt can include specifying, in the set of constraints, that images generated using the clauses generated by the language model are to exclude people and text.

[0009] Performing one or more post-processing operations can include performing a plausibility analysis of each given image among the multiple images; determining, based on the plausibility analysis, that at least one given image visually depicts an object that is classified as not-plausible; and filtering the at least one given image from among the multiple images.

[0010] Serving the at least one output image can include presenting, through a user interface, a set of the multiple images to a digital component provider; receiving, through the user interface, data authorizing distribution of at least one image in the set of multiple images; and including the at least one image with a set of textual content distributed on behalf of the digital component provider.

[0011] Performing one or more post-processing operations can include for each given image among the multiple images: evaluating a relevance of the given image generated by the text-to-image generative model to the image prompt; evaluating a level of diversity of the given image to a specified set of images; ranking each given image among the multiple images comprises: scoring the given image based on the relevance and the diversity; and ranking the given image among other images among the multiple images based on the score.

[0012] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The techniques discussed herein can be used to generate new images from text descriptions that may otherwise confuse text-to-image generative models, which previously would have resulted in non-sensical images, or images that fail to include target visual characteristics. For example, text from web pages, or other online locations, that include text about objects often describe the objects in non-visual terms that are not particularly useful as input to a text-to-image generative model, and can even degrade the quality of images output by the model. To address this problem, the techniques discussed herein use the text from the web pages, or other online locations, as an input to a language model, and cause the language model to generate an image prompt that describes the visual characteristics of the model. This image prompt is then used as the input to the text-to-image generative model to create new images having the visual characteristics specified in the image prompt. By first using a language model to transform the non-visual textual content describing an object into a visual description of the object (e.g., in the form of an image prompt), the system wastes fewer computing resource generating new images that will be low-quality and potentially nonsensical. Given the high level of computing resources required to implement text-to-image generative models, reducing the number of low-quality or non-sensical images generated leads to a significant reduction in the computing resources wasted. Additionally, reducing the number of low-quality images transmitted over the network to client devices reduces the amount of wasted network bandwidth that would otherwise be used to transmit the low- quality images that are ultimately discarded by users of client device. Furthermore, reducing the number of low-quality images rendered and presented by the client devices, which are often battery-operated devices, reduces wasted battery power, thereby increasing the amount of time that the client device can operate on its available battery power.

[0013] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG. l is a block diagram of an example environment in which generative artificial intelligence can be implemented.

[0015] FIG. 2 is a block diagram illustrating interactions between an artificial intelligence system, a text generative model, a text-to-image generative model, and a client device. [0016] FIG. 3 is a flow chart of an example process for creating images with artificial intelligence.

[0017] FIG. 4 a block diagram of an example computer.

[0018] Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

[0019] This specification describes techniques for enabling artificial intelligence to generate new images of objects. Artificial intelligence (Al) is a segment of computer science that focuses on the creation of intelligent agents that can learn and act autonomously (e.g., without human intervention). Artificial intelligence systems can utilize one or more of (i) machine learning, which focuses on developing algorithms that can learn from data, (ii) natural language processing, which focuses on understanding and generating human language, and/or (iii) computer vision, which is a field that focuses on understanding and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content (e.g., images/video, text, audio, or other content) in response to input prompts.

[0020] The techniques described throughout this specification enable artificial intelligence to generate large numbers of new images using various combinations of text (or even images that can be used as seed images) that visually depict an item visually described in an input prompt. For example, an artificial intelligence system can gather information from various sources, such as various web pages or other trusted online resources, and combine this information in different ways to create prompts that visually describe an object. Generally, speaking, the system utilizes an initial prompt to a language model, such as a large language model (LLM), that outputs multiple clauses that visually describe one or more objects of interest. The system uses the clauses to create an image prompt that is then used to cause an image-to-text generative model to create multiple different images.

[0021] As discussed in more detail below, the initial prompt is constructed (e.g., created or augmented) in a manner that causes the language model to output a set of clauses that visually describe a target object (e.g., object or objects of interest), which can then be formatted as an image prompt (e.g., a textual image prompt) that causes the text-to-image generative model to create and output an image having the visual characteristics of the target object. Post-processing operations are used to evaluate the generated images to determine which of the images have higher quality than other images (e.g., given the current context), and one or more of the higher quality images are output to a computing device (e.g., user computer, mobile device, tablet device, audio device, gaming device, etc.). In some situations, the higher quality images can be suggested for inclusion in other content generated by a digital component provider, e.g., in the form of a digital component that includes one or more of the images and the other content. In some situations, the higher quality images can be automatically included in digital components, or otherwise distributed for presentation to users (e.g., with search results or recommendations).

[0022] Using the initial prompt to create the image prompt reduces wasted computing resources that would otherwise generate more low-quality images if a more general prompt were used. Similarly, the number of images generated can be reduced, thereby saving computing resources and generating an output faster, by using the initial prompt to constrain the parameters used by the text-to-image generative model to generate the images. For example, by constructing the initial prompt to specify limitations on what is to be included in the images, the language model will generate clauses visually describing objects specified in the limitations, thereby avoiding the creation of unwanted images, which reduces the time required to generate the images, the memory required to store the images, and the computing resources required to evaluate the generated images. This all contributes to a system capable of creating new images faster than if the text of the web pages, or other online sources, were directly used as the input to the text-to-image generative model.

[0023] The post-processing operations can include, for example, evaluating the images based on various criteria, and scoring each of the images based on the evaluation. For example, one post-processing operation can perform a prediction regarding the likelihood that a particular image depicts plausible features of an object depicted by the generated image. For example, the plausibility analysis can evaluate the features and classify the image as plausible or not-plausible based on the likelihood of the depiction being possible. As an example, a dog wearing a hat would generally be considered more plausible than a dog having human hands, such that depiction of the dog wearing a hat could have a higher plausibility likelihood than a depiction of a dog with human hands. Using this type of a postprocessing operation allows for looser constraints in the construction of the image prompt, which can allow the language model to generate more creative images, while still ensuring that the output images have at least a baseline level of plausibility.

[0024] The post-processing operations can also use various heuristics to evaluate different characteristics of each of the images, and the scores can be assigned based on the various heuristics. In some implementations, the scores are weighted and aggregated to create a final score, which is used to rank the images. Additionally, or alternatively, a machine learning model can be trained to score image quality, and those scores can be used to rank the images. One or more of the highest-ranking images are then selected for serving as output images.

[0025] As used throughout this document, the phrase “digital component” refers to a discrete unit of digital content or digital information (e.g., a video clip, audio clip, multimedia clip, gaming content, image, text, bullet point, artificial intelligence output, language model output, or another unit of content). A digital component can electronically be stored in a physical memory device as a single file or in a collection of files, and digital components can take the form of video files, audio files, multimedia files, image files, or text files and include advertising information, such that an advertisement is a type of digital component.

[0026] FIG. 1 is a block diagram of an example environment 100 in which generative artificial intelligence can be implemented. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects electronic document servers 104, user devices 106, digital component servers 108, and a service apparatus 110. The example environment 100 may include many different electronic document servers 104, user devices 106, and digital component servers 108.

[0027] A client device 106 is an electronic device capable of requesting and receiving online resources over the network 102. Example client devices 106 include personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices that can send and receive data over the network 102. A client device 106 typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over the network 102, but native applications (other than browsers) executed by the client device 106 can also facilitate the sending and receiving of data over the network 102.

[0028] A gaming device is a device that enables a user to engage in gaming applications, for example, in which the user has control over one or more characters, avatars, or other rendered content presented in the gaming application. A gaming device typically includes a computer processor, a memory device, and a controller interface (either physical or visually rendered) that enables user control over content rendered by the gaming application. The gaming device can store and execute the gaming application locally, or execute a gaming application that is at least partly stored and/or served by a cloud server (e.g., online gaming applications). Similarly, the gaming device can interface with a gaming server that executes the gaming application and “streams” the gaming application to the gaming device. The gaming device may be a tablet device, mobile telecommunications device, a computer, or another device that performs other functions beyond executing the gaming application.

[0029] Digital assistant devices include devices that include a microphone and a speaker. Digital assistant devices are generally capable of receiving input by way of voice, and respond with content using audible feedback, and can present other audible information. In some situations, digital assistant devices also include a visual display or are in communication with a visual display (e.g., by way of a wireless or wired connection).

Feedback or other information can also be provided visually when a visual display is present. In some situations, digital assistant devices can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices that are registered with the digital assistant device.

[0030] As illustrated, the client device 106 is presenting an electronic document 150. An electronic document is data that presents a set of content at a client device 106. Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e g., “apps” and/or gaming applications), such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents can be provided to client devices 106 by electronic document servers 104 (“Electronic Doc Servers”). [0031] For example, the electronic document servers 104 can include servers that host publisher websites. In this example, the client device 106 can initiate a request for a given publisher webpage, and the electronic server 104 that hosts the given publisher webpage can respond to the request by sending machine executable instructions that initiate presentation of the given webpage at the client device 106.

[0032] In another example, the electronic document servers 104 can include app servers from which client devices 106 can download apps. In this example, the client device 106 can download fdes required to install an app at the client device 106, and then execute the downloaded app locally (i.e., on the client device). Alternatively, or additionally, the client device 106 can initiate a request to execute the app, which is transmitted to a cloud server. In response to receiving the request, the cloud server can execute the application and stream a user interface of the application to the client device 106 so that the client device 106 does not have to execute the app itself. Rather, the client device 106 can present the user interface generated by the cloud server’s execution of the app, and communicate any user interactions with the user interface back to the cloud server for processing.

[0033] Electronic documents can include a variety of content. For example, an electronic document 150 can include native content 152 that is within the electronic document 150 itself and/or does not change over time. Electronic documents can also include dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document (e.g., electronic document 150) can maintain a data source that is used to populate portions of the electronic document. In this example, the given electronic document can include a script, such as the script 154, that causes the client device 106 to request content (e.g., a digital component) from the data source when the given electronic document is processed (e.g., rendered or executed) by a client device 106 (or a cloud server). The client device 106 (or cloud server) integrates the content (e.g., digital component) obtained from the data source into the given electronic document to create a composite electronic document including the content obtained from the data source.

[0034] In some situations, a given electronic document (e.g., electronic document 150) can include a digital component script (e.g., script 154) that references the service apparatus 110, or a particular service provided by the service apparatus 110. In these situations, the digital component script is executed by the client device 106 when the given electronic document is processed by the client device 106. Execution of the digital component script configures the client device 106 to generate a request for digital components 112 (referred to as a “component request”), which is transmitted over the network 102 to the service apparatus 110. For example, the digital component script can enable the client device 106 to generate a packetized data request including a header and payload data. The component request 112 can include event data specifying features such as a name (or network location) of a server from which the digital component is being requested, a name (or network location) of the requesting device (e.g., the client device 106), and/or information that the service apparatus 110 can use to select one or more digital components, or other content, provided in response to the request. The component request 112 is transmitted, by the client device 106, over the network 102 (e.g., a telecommunications network) to a server of the service apparatus 110.

[0035] The component request 112 can include event data specifying other event features, such as the electronic document being requested and characteristics of locations of the electronic document at which digital component can be presented. For example, event data specifying a reference (e.g., URL) to an electronic document (e.g., webpage) in which the digital component will be presented, available locations of the electronic documents that are available to present digital components, sizes of the available locations, and/or media types that are eligible for presentation in the locations can be provided to the service apparatus 110. Similarly, event data specifying keywords associated with the electronic document (“document keywords”) or entities (e.g., people, places, or things) that are referenced by the electronic document can also be included in the component request 112 (e.g., as payload data) and provided to the service apparatus 110 to facilitate identification of digital components that are eligible for presentation with the electronic document. The event data can also include a search query that was submitted from the client device 106 to obtain a search results page.

[0036] Component requests 112 can also include event data related to other information, such as information that a user of the client device has provided, geographic information indicating a state or region from which the component request was submitted, or other information that provides context for the environment in which the digital component will be displayed (e.g., a time of day of the component request, a day of the week of the component request, a type of device at which the digital component will be displayed, such as a mobile device or tablet device). Component requests 112 can be transmitted, for example, over a packetized network, and the component requests 112 themselves can be formatted as packetized data having a header and payload data. The header can specify a destination of the packet and the payload data can include any of the information discussed above.

[0037] The service apparatus 110 chooses digital components (e.g., third-party content, such as video fdes, audio files, images, text, gaming content, augmented reality content, and combinations thereof, which can all take the form of advertising content or non-advertising content) that will be presented with the given electronic document (e.g., at a location specified by the script 154) in response to receiving the component request 112 and/or using information included in the component request 112.

[0038] In some implementations, a digital component is selected in less than a second to avoid errors that could be caused by delayed selection of the digital component. For example, delays in providing digital components in response to a component request 112 can result in page load errors at the client device 106 or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document are presented at the client device 106.

[0039] Also, as the delay in providing the digital component to the client device 106 increases, it is more likely that the electronic document will no longer be presented at the client device 106 when the digital component is delivered to the client device 106, thereby negatively impacting a user's experience with the electronic document. Further, delays in providing the digital component can result in a failed delivery of the digital component, for example, if the electronic document is no longer presented at the client device 106 when the digital component is provided.

[0040] In some implementations, the service apparatus 110 is implemented in a distributed computing system that includes, for example, a server and a set of multiple computing devices 114 that are interconnected and identify and distribute digital component in response to requests 112. The set of multiple computing devices 114 operate together to identify a set of digital components that are eligible to be presented in the electronic document from among a corpus of millions of available digital components (DCi-_x). The millions of available digital components can be indexed, for example, in a digital component database 116. Each digital component index entry can reference the corresponding digital component and/or include distribution parameters (DPi-DP_x) that contribute to (e.g., trigger, condition, or limit) the distribution/transmission of the corresponding digital component. For example, the distribution parameters can contribute to (e.g., trigger) the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., either exactly or with some pre-specified level of similarity) one of the distribution parameters of the digital component.

[0041] In some implementations, the distribution parameters for a particular digital component can include distribution keywords that must be matched (e.g., by electronic documents, document keywords, or terms specified in the component request 112) in order for the digital component to be eligible for presentation. Additionally, or alternatively, the distribution parameters can include embeddings that can use various different dimensions of data, such as website details and/or consumption details (e g., page viewport, user scrolling speed, or other information about the consumption of data). The distribution parameters can also require that the component request 112 include information specifying a particular geographic region (e.g., country or state) and/or information specifying that the component request 112 originated at a particular type of client device (e.g., mobile device or tablet device) in order for the digital component to be eligible for presentation. The distribution parameters can also specify an eligibility value (e.g., ranking score, or some other specified value) that is used for evaluating the eligibility of the digital component for distribution/transmission (e.g., among other available digital components).

[0042] The identification of the eligible digital component can be segmented into multiple tasks 117a-l 17c that are then assigned among computing devices within the set of multiple computing devices 114. For example, different computing devices in the set 114 can each analyze a different portion of the digital component database 116 to identify various digital components having distribution parameters that match information included in the component request 112. In some implementations, each given computing device in the set 114 can analyze a different data dimension (or set of dimensions) and pass (e.g., transmit) results (Res 1-Res 3) 118a- 118c of the analysis back to the service apparatus 110. For example, the results 118a-l 18c provided by each of the computing devices in the set 114 may identify a subset of digital components that are eligible for distribution in response to the component request and/or a subset of the digital component that have certain distribution parameters. The identification of the subset of digital components can include, for example, comparing the event data to the distribution parameters, and identifying the subset of digital components having distribution parameters that match at least some features of the event data.

[0043] The service apparatus 110 aggregates the results 118a- 118c received from the set of multiple computing devices 114 and uses information associated with the aggregated results to select one or more digital components that will be provided in response to the request 112. For example, the service apparatus 110 can select a set of winning digital components (one or more digital components) based on the outcome of one or more content evaluation processes, as discussed below. In turn, the service apparatus 110 can generate and transmit, over the network 102, reply data 120 (e.g., digital data representing a reply) that enable the client device 106 to integrate the set of winning digital components into the given electronic document, such that the set of winning digital components (e.g., winning third- party content) and the content of the electronic document are presented together at a display of the client device 106.

[0044] In some implementations, the client device 106 executes instructions included in the reply data 120, which configures and enables the client device 106 to obtain the set of winning digital components from one or more digital component servers 108. For example, the instructions in the reply data 120 can include a network location (e g., a Uniform Resource Locator (URL)) and a script that causes the client device 106 to transmit a server request (SR) 121 to the digital component server 108 to obtain a given winning digital component from the digital component server 108. In response to the request, the digital component server 108 will identify the given winning digital component specified in the server request 121 (e.g., within a database storing multiple digital components) and transmit, to the client device 106, digital component data (DC Data) 122 that presents the given winning digital component in the electronic document at the client device 106.

[0045] When the client device 106 receives the digital component data 122, the client device will render the digital component (e.g., third-party content), and present the digital component at a location specified by, or assigned to, the script 154. For example, the script 154 can create a walled garden environment, such as a frame, that is presented within, e.g., beside, the native content 152 of the electronic document 150. In some implementations, the digital component is overlay ed over (or adjacent to) a portion of the native content 152 of the electronic document 150, and the service apparatus 110 can specify the presentation location within the electronic document 150 in the reply 120. For example, when the native content 152 includes video content, the service apparatus 110 can specify a location or object within the scene depicted in the video content over which the digital component is to be presented. [0046] The service apparatus 110 can also include an artificial intelligence system 160. Although the artificial intelligence system 160 is depicted by a separate box and described separately in this document, the entire service apparatus 110 can be considered an artificial intelligence system. The artificial intelligence system 160 is configured to autonomously generate images that can be included, for example, in digital components, either prior to a request 112 (e.g., offline) and/or in response to a request 112 (e.g., online or real-time). As described in more detail throughout this specification, the artificial intelligence (“Al”) system 160 can collect online content about a specific entity (e.g., digital component provider or another entity) and create or cause the creation of images using text prompts.

[0047] The environment 100 also includes a generative model 170. Although only one generative model 170 is depicted in FIG. 1, the generative model 170 can represent a set of multiple generative models that can each be specially configured to perform certain tasks. For example, as described in more detail below, the set of generative models represented by generative model 170 can include a large language model (“LLM”) that is configured to summarize textual content about a given object that is located at one or more online locations (e.g., web pages or other online resources).

[0048] A large language model (“LLM”) is a model that is trained to generate and understand human language. LLMs are trained on massive datasets of text and code, and they can be used for a variety of tasks. For example, LLMs can be trained to translate text from one language to another; summarize text, such as web site content, search results, news articles, or research papers; answer questions about text, such as “What is the capital of Georgia?”; create chatbots that can have conversations with humans; and generate creative text, such as poems, stories, and code. For brevity, large language models are also referred to herein as “language models.” [0049] The language model can be any appropriate language model neural network that receives an input sequence made up of text tokens selected from a vocabulary and auto- regressively generates an output sequence made up of text tokens from the vocabulary. For example, the language model can be a Transformer-based language model neural network or a recurrent neural network-based language model.

[0050] In some situations, the language model can be referred to as an auto-regressive neural network when the neural network used to implement the language model auto- regressively generates an output sequence of tokens. More specifically, the auto-regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular text token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token, and a context input that provides context for the output sequence.

[0051] For example, the current input sequence when generating a token at any given position in the output sequence can include the input sequence and the tokens at any preceding positions that precede the given position in the output sequence. As a particular example, the current input sequence can include the input sequence followed by the tokens at any preceding positions that precede the given position in the output sequence. Optionally, the input and the current output sequence can be separated by one or more predetermined tokens within the current input sequence.

[0052] More specifically, to generate a particular token at a particular position within an output sequence, the neural network of the language model can process the current input sequence to generate a score distribution (e.g., a probability distribution) that assigns a respective score, e g., a respective probability, to each token in the vocabulary of tokens. The neural network of the language model 170 can then select, as the particular token, a token from the vocabulary using the score distribution. For example, the neural network of the language model can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the distribution.

[0053] As a particular example, the language model can be an auto-regressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0054] The language model can have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Eisen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910.10683, 2019; Daniel Adiwardana, Minh- Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like opendomain chatbot. CoRR, abs/2001.09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.

[0055] Generally, however, the Transformer-based neural network includes a sequence of attention blocks, and, during the processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention block then updates each of the hidden states at least in part by applying self-attention to generate a respective output hidden state for each of the input tokens. The input hidden states for the first attention block are embeddings of the input tokens in the input sequence and the input hidden states for each subsequent attention block are the output hidden states generated by the preceding attention block.

[0056] In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate the score distribution.

[0057] Generally, because the language model is auto-regressive, the service apparatus 110 can use the same language model to generate multiple different candidate output sequences in response to the same request, e.g., by using beam search decoding from score distributions generated by the language model, using a Sample-and-Rank decoding strategy, by using different random seeds for the pseudo-random number generator that’s used in sampling for different runs through the language model or using another decoding strategy that leverages the auto-regressive nature of the language model.

[0058] In some implementations, the language model is pre-trained, i.e., trained on a language modeling task that does not require providing evidence in response to user questions, and the service apparatus 110 (e.g., using Al system 160) causes the language model to generate output sequences according to the pre-determined syntax through natural language prompts in the input sequence.

[0059] For example, the service apparatus 110 (e.g., Al system 160), or a separate training system, pre-trains the language model (e.g., the neural network) on a language modeling task, e.g., a task that requires predicting, given a current sequence of text tokens, the next token that follows the current sequence in the training data. As a particular example, the language model can be pre-trained on a maximum-likelihood objective on a large dataset of text, e.g., text that is publicly available from the Internet or another text corpus.

[0060] In some implementations, the Al system 160 can generate a prompt 172 (e g., an initial prompt) that is submitted to the language model (e.g., one of the generative models 170), and causes the language model to generate output sequences 174, also referred to as passages or simply as “output”. The Al system 160 can generate the prompt in a manner (e g., having a structure) that identifies a list of one or more online sources of information, such as a list of one or more websites or data repositories, and specifying a set of constraints the language model must use to generate the output 174 using the prompt 172.

[0061] To initiate creation of the output sequences 174, the Al system 160 submits the prompt 172 to the one or more generative models 170, which use the prompt 172 to evaluate textual information found at the one or more online sources specified in the prompt 172, and generate the output 174 that visually describes an object described by the textual information according to the constraints specified in the prompt 172. For example, assume that the Al system 160 uses the following template to create the prompt 172 (e.g., “initial prompt”):

Write an image generation prompt for a digital component for: '{Text}' by ' {Visurl}'. There should be no people or text in the photo. Answer with just the prompt as a python string.

[0062] In this example, the “Text” can be a placeholder for a set of text that is included in the digital component for which the image prompt is to be generated, and “Visurl” can be a URL linked to by the digital component, such as a web page address of a given web page. As discussed in more detail with reference to FIG. 2, the Al system 160 can populate this prompt template with the appropriate text and URL, and submit the populated prompt to the generative model 170 (e.g., an LLM in the set of LLMs represented by generative model 170), which causes the generative model 170 to process the prompt 172 and return the output 174, which can take the form of an image summary that provides a visual description of an object described by text located at the URL.

[0063] The Al system 160 can use the generated summary as part of an additional prompt 172 (e.g., an image prompt) that is sent to the generative model 170. For example, the Al system 160 can insert the generated image summary into the additional prompt 172 that is generated after receiving the output 174 (e.g., initial output) generated using the initial prompt (e.g., the prompt generated after receiving the image summary), and submit the additional prompt 172 (e.g., image prompt) to the generative model 170 (e.g., a text-to-image generative model) as a constraint for generating images for use in digital components generated and/or distributed by the Al system 160.

[0064] Submission of this additional prompt 172 to the generative model 170 causes the generative model 170 to generate an additional output 174, which includes multiple images and is communicated electronically to the Al system 160. The Al system 160 receives the images of the additional output 174, and can user the images to generate or augment one or more digital components that are available to be provided in response to the request 112. In some implementations, each different digital component can include a different combination of text and/or images generated by the generative model 170. For example, assume that the additional output 174 includes twelve different images, and that the formatting of the digital components being generated or distributed by the Al system 160 each include space for a single image. In this example, the Al system 160 could create twelve different digital components that each include the same set of text, but include a different one of the 12 different images.

[0065] Additionally, the image selected for inclusion with a given set of text for any individual request 112, could be determined by the services apparatus 110/ Al system 160 based on the context of the request (e.g., geographical origins of the request 112, time of day, keywords, etc.). In this way, the combination of and image and text provided in response to each request 112 can be determined at the time of the request. Combining the image and text at the time of the request prevents the service apparatus 110 from having to store a separate complete digital component for every possible combination of images and text, thereby reducing the amount of memory required to make the different combinations of content available. For example, only one instance of the twelve images would need to be stored, and only one instance of each set of text would need to be stored, but every combination of images and text would be available for distribution.

[0066] The additional output 174 of the images generated using the additional prompt 172 (e.g., image prompt) can be provided to a content provider (e.g., digital component provider) for approval prior to being included in a digital component. In some implementations, the images can be surfaced to the content provider as part of an account chat interface. For example, the content provider’s account user interface can include a conversational assistant feature that enables the content provider to request information by way of a natural language system. In this example, creation/presentation of images generated by the generative model 170 can be triggered, for example, by the content provider submitting a request for suggestions of images that the content provider could include in digital components being distributed by the service apparatus 110. [0067] In response to receiving the request from the content provider, the service apparatus 110 (e.g., by way of the Al system 160) can generate the initial prompt and image prompt as discussed above, and submit the image prompt to the generative model 170 to obtain images generated by the generative model 170. The service apparatus 110 can render the images and present a set of the images (e.g., one or more of the images) to the content provider through the account user interface (e.g., in the chat interface or a separate pane). The images can be presented by themselves, or the service apparatus 110 can combine the images with text of the digital components being distributed on behalf of the content provider to provide the content provider with a preview of how the digital components will appear when distributed with one or more of the images. In either case, the content provider can either authorize one or more of the images for distribution in the digital components, or decline the option to have the one or more images presented in the digital components.

[0068] Instead of, or in addition to, generating/presenting the images based on interactions by the content provider with an interactive chat interface, the service apparatus 110 can generate the images in an “offline” process, and present the images, e.g., as recommendations, when the content provider accesses their account. In some implementations, the service apparatus 110 can make the decision to generate the images, as described above and in more detail below with reference to FIG. 2, based on one or more factors related to the account of the content provider. For example, the service apparatus 110 may trigger the creation of the prompts (e.g., initial and image prompts) in response to determining that the digital components distributed on behalf of the content provider do not include or present images, or that fewer than a specified portion (e.g., percentage) of the digital components include/present images.

[0069] Once the above determination is made, the generated prompts can be used, as described throughout this document, to create images that are recommended for incorporation into the digital components distributed on behalf of the content provider. In some implementations, the recommendations (or at least the availability of the recommendations) can be presented at a welcome page of the content provider’s account. In some implementations, the recommendations can be presented in a recommendations tab or page of the content provider’s account for review by the content provider. Each of the recommendations (e.g., recommended images) can be presented with (e.g., adjacent to) an interactive user interface control that enables the content provider to authorize use of the image generated by the generative model 170, or dismiss the recommendation to use the image in digital components. In response to a given recommended image being authorized (e.g., through user interaction with the user interface control), the service apparatus 110 can store the authorized image as an image that is available for output (e.g., serving) with/in a digital component distributed on behalf of the content provider in response to a request 112 from a client device 106.

[0070] The Al system 160 can perform one or more post-processing operations that evaluate one or more characteristics of the images generated by the generative model 170. The post-processing operations can be performed on the images themselves, as received in the output 174 from the generative model 170. The post-processing operations can also be performed on a combination of the text and images that, together, constitute an updated/augmented digital component. For example, when the post-processing operations are performed on the combination of the text of the digital component and one or more images generated by the generative model 170, the evaluation of the generated images can be performed in the context of the text with which the images would appear. Of course, even in these situations, the images could still be evaluated in isolation of (e.g., independent of) the text of the digital component, which would reveal to the service apparatus 110 whether the image itself may still be a good candidate for inclusion with other text, or whether the image itself should be discarded from consideration.

[0071] In some implementations, the post-processing operations can include generating a plausibility score for each image generated by the generative model 170. The plausibility score is a value specifying a likelihood that the generated image of an object is consistent with accurate/real depictions of the object. The generation of the plausibility score is discussed in more detail with reference to FIG. 2.

[0072] The post-processing operations can also include an evaluation of the relevance of the images to the image prompt, textual content of a digital component, or textual content about the object presented at a specified network location. The post-processing operations can also include an evaluation of similarity between the generated images of the object (e.g., created using the generative model 170) to images of the object provided by the content provider. The post-processing operations can also include a determination as to whether the images generated by the generative model 170 violates one or more content policies, such as violating copyright rules, violating family safe content policies, or whether insertion of the image in a digital component will overt ap/occlude text of the digital component, an entity logo that is part of the digital component, or a link to an online resource (or triggers an action) that is embedded in the digital component. As discussed in more detail with reference to FIG. 2, the post-processing operations can be used to score, or otherwise assign a level of priority to, each of the generated images so that the Al system 160 can rank the multiple generated images relative to each other, and ultimately serve one or more of the highest ranking images for presentation to the content provider or in a digital component provided in response to a request 112 from a client device 106. Note that one or more operations of the Al system 160 and generative model 170 can be performed responsive to receipt of the request 112, or can be performed prior to receipt of the request 112.

[0073] FIG. 2 is a block diagram 200 illustrating interactions between an artificial intelligence system 160, a text generative model (“text model”) 202, a text-to-image generative model (“T2I model”) 204, and a client device 206. In some situations, the text model 202 and the T2I model 204 can both be part of the language model 170 discussed above with reference to FIG. 1. Similarly, the client device 206 can be the same or similar to the client device 106 of FIG. 1. The artificial intelligence system 160 can be part of the service apparatus 110 of FIG. 1, such that the description of the artificial intelligence system 160 can also be considered a description of the service apparatus 110.

[0074] The T2I model 206 is configured to accept, as input, a text (or voice) prompt describing visual characteristics of an object or scene, and generate, as output, an image (e.g., a still image or video) visually depicting the object or scene visually described by the text prompt. Although a single T2I model 206 is depicted in FIG. 2, the T2I model 206 can be a collection of multiple models that can each be specially trained to generate different types of images.

[0075] The text model 202 is configured to accept, as input, a text (or voice) prompt and generate a textual response, which can be output as an audio response or graphical text response. Although a single text model 202 is depicted in FIG. 2, the text model 202 can include a set of different text models that are invoked to perform different tasks for which the different text models are specially trained. For example, one text model within the set of text models may be specially trained to perform content summary tasks, while another text model may be specially trained to generate an image prompt, for example, using the summary output of the specially trained summary text model. Furthermore, the set of models can include a generalized text model that is larger is size, and capable of generating large amounts of diverse datasets, but this generalized text model may have higher latency than the specialized text models, which can make it less desirable for use in real-time operations, depending on time latency constraints required to generate content. Each text model can be implemented by way of an LLM, or another model that is configured to generate natural language text responsive to a prompt.

[0076] The artificial intelligence system 160 includes a text collection apparatus 208, a summary apparatus 210, a prompt apparatus 212, and a post processing apparatus 214. The following description refers to these different apparatuses as being implemented independently and each configured to perform a set of operations, but any of these apparatuses could be combined to perform the operations discussed below.

[0077] The artificial intelligence system 160 is in communication with a memory structure 216. The memory structure 216, can include one or more databases. As shown, the memory structure includes a collected text database 218, an image database 220, and a digital component database 222. Each of these databases 218, 220, and 222, can be implemented in a same hardware memory device, separate hardware memory devices, and/or implemented in a distributed cloud computing environment (or another data storage apparatus).

[0078] The text collection apparatus 208 is implemented using at least one computing device (e.g., one or more processors), and can include one or more language models. The text collection apparatus 208 is configured (e.g., specially programmed with executable code and/or implemented with specialized hardware) to collect information provided by online data sources, such as web pages. In some implementations, the collected information includes text collected from one or more specified online resources.

[0079] To obtain the text, the text collection apparatus 208 can crawl a specified online resource, such as a landing page to which an existing digital component is linked or another source of information about an object described by the existing digital component. For example, assume that a digital component is stored in the digital component database 222, and links to example.com. In this example, the text collection apparatus 208 can identify the link to example.com in the stored digital component, and crawl example.com to obtain/ discover text presented by example.com. The obtained text can be stored in the collected text database 218, and obtained from the collected text database 218 when an operation that uses text as input is triggered (e.g., launched or executed).

[0080] The crawling of the specified resource can be performed in an offline process (e.g., prior to when the obtained text is to be used by the Al system 160). For example, the resource crawling can be performed as part of a routine crawling performed by the Al system 160 at scheduled intervals. In some implementations, the crawling is performed in an online process (e.g., in response to a request for the Al system 160 to perform an operation that uses the text as input). For example, assume that the Al system 160 initiates an operation to generate images for a digital component that includes a link to example.com. In this example, the Al system 160 can access the collected text database 218, and search for text obtained from example.com that is stored in the collected text database 218. When the search of the collected text database 218 returns text obtained from example.com, the returned text can be used to perform the image generation. However, when the Al system 160 determines that the search of the collected text database does not return text obtained from example.com, the Al system 160 can trigger a crawl of example.com to obtain text (e.g., using the text collection apparatus 208). In this situation, the Al system 160 can use the text obtained by crawling example.com to generate images, and store the text obtained in the collected text database 218 for future use.

[0081] Additionally, or alternatively, the text collection apparatus 208 can be configured to collect text information from an account storing distribution parameters that contribute to the distribution of the digital component. For example, the text collection apparatus 208 can identify keywords, object/service descriptions, headlines, or other text contained in the account, and use this identified text as an input to the image generation processes discussed throughout this specification.

[0082] Additionally, or alternatively, the text collection apparatus 208 can be configured to issue/submit a query to a search system that responds to the query with information about a topic and/or an entity discussed in the stored digital component (or another topic/entity). In some implementations, the text collection apparatus 206 can parse, or otherwise process, the search result snippets that are returned by the search system in response to submission of the query by the text collection apparatus 206 to obtain text related to the topic/entity. The collected text can be stored, for example, in the collected text database 218.

[0083] When an entity (e.g., a company) has an online presence, e.g., website, that provides information about the entity, the query submitted by the text collection apparatus 208 can be a site-constrained query that causes the search system to only reply to the site- constrained query with information contained a specified site (e.g., the website of the company). Of course, multiple site constrained queries can be issued for multiple different sites, or multiple different sites can be specified in the site-constrained query that causes the search system to collect information related to the query from multiple different specified sites (e.g., a social networking site, web answers site, entity review site, etc.). The site constraint can be specified, for example, as a second level domain, or a specific page address depending on where the information is to be sourced from.

[0084] The text collection apparatus 208 can collect other text from other sources. In some implementations, the text collection apparatus 208 can be configured to collect conversational input submitted to the Al system 160, and use this conversational input in the image generation processes discussed herein. For example, assume that a digital component provider is interacting through a conversational chat interface with the Al system 160, and specifies a preference for a certain object or scene to be depicted in black and white rather than full color. In this example, the Al system 160 (e.g., by way of the text collection apparatus 208) can capture that text, store it in the collected text database 218, and/or use that captured text (e.g., the black and white preference) when generating images of the object.

[0085] The text collection apparatus 208 can store any, or all, of the collected text in the collected text database 218 in a manner that facilitates retrieval of the text at a later time. For example, the text collection apparatus 208 can index the collected text to the digital component, digital component provider, website, or another reference that will facilitate retrieval of the text when the Al system 160 is performing operations related to generating an image for the digital component provider.

[0086] The summary apparatus 210 is implemented using at least one computing device (e.g., a device including one or more processors), and can include one or more language models. The summary apparatus 210 is configured to summarize information about a topic or entity (e.g., person, place, thing, or concept). In some implementations, the summary apparatus 210 is configured to summarize the text collected by the text collection apparatus 208, and potentially stored in the collected text database 218. For example, the summary apparatus 210 can be configured to accept, as input, the collected text, and output a specified length (e.g., 200 words or some other number of words) summary of the contents of the collected text.

[0087] The summary can be generated using the text model 202, which can be part of the summary apparatus 210, or in data communication with the summary apparatus 210. In some implementations, the summary apparatus 210 (or the prompt apparatus 212 discussed below) can generate a summary prompt that is submitted to the text model 202 as an input prompt 224. In some implementations, the summary apparatus 210 uses a language model (e.g., text model 202) that has been specially trained to generate text summaries using content of web pages or other sources and the summary prompt. The summary prompt can specify one or more of the following:

- a set of sources that should be used to generate the summary;

- details about the set of sources the language model should consider when summarizing the content;

- factual grounding instructions specifying that the language model should provide citations to the sources used to generate the summary;

- summary constraints specifying information that should not be included in the summary (e.g., information that is not directly supported by the set of sources); and

- formatting constraints specifying how the output of the language model should be formatted (e.g., as bullet points or in paragraph form, with or without an introduction summary, total length (e.g., number of characters or separate clauses); and

- tone constraints specifying a tone of the output (e.g., creative, funny, sad, serious, or from the perspective of a specified entity, such as an artist, engineer, or story writer).

[0088] An example summary prompt can take the form of:

“Given a question and a list of sources, write a short summary that cites individual sources and summarizes all of them as comprehensively as possible. Each source is independent and might repeat or contradict content from other sources. The summary should be directly supported by the given sources and cited appropriately with a [$i] notation following a statement that is supported by $i. If a statement is based on multiple sources, all of these sources should be listed in the brackets, for example [Si, $j, $k]. The summary may start with a general statement about the answer space. The summary shouldn’t include any information that cannot be supported by the given sources.”

[0089] In this example, the notation $i can placeholders for the names of the sources. The bolded “a list of sources” can be replaced with the names of actual sources to be considered, or be a reference to locations of sources in the set of sources to be considered when creating the summary. The set of sources can be network addresses (e.g., universal resource indicators/locators - URIs/URLs) of online data sources (e.g., second level domains of websites, specific addresses of web pages, or network addressees of other data sources).

In some implementations, the set of sources can include the collected text database 218, such that the summary can be generated using the text already collected and stored by the text collection apparatus 208. The summary apparatus 210 uses the summary prompt to generate a text summary that summarizes text collected form one or more network locations of the set of sources into a passage summary. As noted above the text summary can be formatted as a set of bullet points or in paragraph form.

[0090] The text summary is generated by a text model 202, which as noted above, can be part of the summary apparatus 210, or in data communication with the summary apparatus 210. In either case, the summary apparatus 210 inputs the summary prompt into the text model 202 as an input prompt 224. The text model 202 (e.g., an LLM) processes the input prompt 224, and generates a natural language output (“NL Output”) 226 that summarizes the text (or other content) of the set of sources according to the instructions/constraints specified in the summary prompt.

[0091] An example paragraph summary of a set of sources that provide information about an automobile can take the form of:

The 2.5-liter 4-cylinder engine delivers enhanced horsepower and torque distribution to give you seamless acceleration. State-of-the-art all wheel drive technology comes with six available drive modes — eco, normal, tarmac, gravel, snow, mud — so you can drive confidently in almost any condition. Active Blind Spot Assist can help drivers avoid a collision by detecting other vehicles in the surrounding lanes, even when the human eye can't see them.

[0092] An example bullet point summary of the same set of sources can take the form of: The 2.5 -liter 4-cylinder engine delivers enhanced horsepower and torque distribution to give you seamless acceleration.

State-of-the-art all wheel drive technology comes with six available drive modes - - eco, normal, tarmac, gravel, snow, mud — so you can drive confidently in almost any condition.

Active Blind Spot Assist can help drivers avoid a collision by detecting other vehicles in the surrounding lanes, even when the human eye can't see them.

[0093] The summary can be generated in response to receipt of a request 228 from a client device 206 (e.g., in a real-time or online mode), or generated in an offline mode (e.g., independent of receipt of an instance of the request 228 from a client device 206. The request 228 can be, for example, from a digital component provider, and be requesting generation of one or more images that can be inserted into, or distributed with, digital components distributed on behalf of the digital component provider. The request 228 can be generated and submitted to the Al system 160 based on a conversational input of the digital component provider to an Al chat interface at the client device 206.

[0094] To illustrate, assume that the digital component provider has submitted a conversational (e.g., natural language) input to an Al chat interface, such as “I would like to improve the look of my digital components.” In this example, the Al system 160 can interpret this conversational input as the request 228 to generate/present images that will improve the look of the digital components distributed on behalf of the digital component provider. In turn, the Al system 160 can initiate the generation of the summary (e.g., by the summary apparatus 210) and perform other operations associated with the generation of the images.

[0095] In the real-time/online mode, the request 228 received from the client device 206 can be passed to the summary apparatus 210 in parallel with other processing being performed in response to the request 228, such as generating a conversational response to the request 228, so that the summary apparatus 210 can generate the summary while other request 228 processing operations are being performed, thereby reducing the latency associated with providing the client device 206 with the final response to the request 228. For example, while the summary apparatus 210 is generating the text summary, the Al system 160 can be generating conversational response to the digital component provider, such as, “Do you have a preferred color or shape for the object that will be presented in the digital component?”, which can be transmitted back to the client device 206 and/or presented in the Al chat interface.

[0096] In some situations, the summary apparatus 210 will complete the generation of the text summary before the digital component provider submits a conversational answer specifying a preferred color or shape of the object. As such, the summary apparatus 210 can use the text summary, as described throughout this specification, to initiate or continue generation of the image while the Al system 160 continues to interact with the digital component provider and obtain additional information (e.g., text) describing target (e.g., desired) visual characteristics of the image being created by the Al system 160. In this way, the Al system 160 can begin generating the image on the originally created summary, and update/ augment the image as more information about the image is obtained from the digital component provider. As such, the overall latency of creating the final image is reduced relative to waiting to begin generation of the image until all the information describing the target visual characteristics are received. This results in a more efficient system that provides the generated images more quickly following receipt of the target visual characteristics than other systems that wait to generate images until all of the target visual characteristics are received.

[0097] In the offline mode, the summary apparatus 210 can generate summaries for anticipated requests that can be stored in the memory structure 216 for use when the request 228 is received from the client device 206. For example, the summary apparatus 210 can identify a set of digital component providers who are most likely to submit the request 228 for generation of images to be included in, or distributed with, their digital components. In some implementations, the Al system 160 can identify set of digital component providers based on characteristics of digital component providers who have previously submitted similar requests for images. [0098] For example, assume that digital component providers who have submitted a request for image generation, referred to as requestors, have various sets of characteristics from an example set of characteristics X1-X20. In this example, the Al system 160 can map embeddings of the characteristics of each requestor in a multi-dimensional space, and also map the characteristics of other digital component providers in the same multi-dimensional space. The Al system 160 can then determine a similarity (e.g., distance) between the mappings of the requestors and the mappings of the other digital component providers. Using this similarity, the Al system can identify the set of other digital component providers who are most similar to the requestors, and classify this set of other digital component providers as most likely to submit the request 228. In turn, the Al system 160 can trigger one or more operations required to generate images. More specifically, the Al system 160 can cause the text collection apparatus 208 to collect text, cause the summary apparatus 210 to create the summary, and/or cause additional operations performed by the prompt apparatus 212, the post processing apparatus 214, and/or any other apparatus (e.g., text model 202 and/or T2I model 204) to all perform operations prior to receiving the request 228 from any of the other digital component providers.

[0099] Continuing with the discussion of the summary apparatus 210, the summary apparatus 210 can, for each digital component provider among those digital component providers most likely to submit the request (referred to as prospective requestors), identify a set of sources relevant to the digital component provider and/or their stored digital components, and perform operations similar to those discussed above to generate a set of summaries (e.g., one or more summaries) for each of the prospective requestors. This set of summaries can be stored in the memory structure 216 (e.g., with an index to the corresponding digital component provider), and when the request 228 is received from the client device 206 of one of the prospective requestors, the summary apparatus 210 (or another apparatus in the Al system 160), can query the memory structure 214 to retrieve one or more of the summaries indexed to the digital component provider who submitted the request 228 to facilitate operations performed using the summaries, as discussed in more detail below. Generating summaries in the offline mode can reduce latency associated with responding to the request 228 submitted by the client device 206 because the operations required to generate the summaries will not preclude downstream operations that rely on the summary, as discussed below.

[00100] In some implementations, the summary is provided to a prompt apparatus 212, which is implemented using at least one computing device (e.g., a device including one or more processors), and can include one or more language models. The prompt apparatus 212 is configured to generate queries that are submitted to language models, such as the text model 202 and the T2I model 204. In some implementations, the prompt apparatus 212 creates an initial prompt that is submitted to a language model (e.g., the text model 202) to obtain a visual description of an object, which is then submitted in an image prompt to another language model (e.g., the T2I model 204) to obtain images generated using the image prompt.

[00101] The initial prompt created by the prompt apparatus 212 generally includes one or more specified network locations and a set of constraints that limits clauses generated by a language model. The one or more specified network locations can be, for example, one or more of a network address (e.g., second level domain or web page address) that is included in a digital component (e.g., linked to by the digital component), or another network location (e.g., the collected text database 218) from which information related to the digital component can be obtained. The information related to the digital component can include text, as discussed above, or other information, such as images of objects related to the digital component. When the information includes images, the Al system 160 can use computer vision, for example, to analyze the images and/or generate a visual description of the images. [00102] As noted above, the information obtained from the specified network location can be used to generate a summary of the content provided at the specified network location, which can also be included in the initial prompt. The summary can summarize textual content, a visual description of the images at the specified network location, or both.

[00103] The set of constraints specifies one or more objects that should be excluded from images generated using the clauses generated by the language model. For example, the set of constraints can include one or more of a text constraint and an object constraint. The text constraint can be an instruction that prevents text from being included in the generated image, and the object constraint is an instruction that prevents a specified object from being included in the generated image. An example of an object constraint is a person constraint that prevents a person from being included in the generated image. Other object constraints can prevent other specified objects from being included in the generated image.

[00104] As discussed above, the Al system 160 can use the following template (or a similar template) to create the initial prompt:

Write an image generation prompt for a digital component for: '{Headline}' by '{Visurl}'. There should be no people or text in the photo. Answer with just the prompt as a python string.

[00105] In this example, the “Text” can be a placeholder for a set of text that is included in the digital component for which the image prompt is to be generated, or text from the summary generated by the summary apparatus 210. “Visurl” can be a URL linked to by the digital component, such as a web page address of a given web page. The prompt apparatus 212 can populate this prompt template with the appropriate text and URL, and submit the populated prompt to the text model 202 as an input prompt 224, which causes the text model 202 to process the input prompt 224 and return a set of text as the NU output 226, which can take the form of an image summary that provides a visual description of an object described by text located at the URL.

[00106] In this example, the set of constraints include a text constraint specifying that there should be no text in generated images, and an object constraint specifying that there should be no people depicted in the generated images. The set of constraints also includes a format constraint indicating that the response to the initial prompt should only include the requested image prompt, and be in the form of a python string. Strings in python are arrays of bytes representing unicode characters.

[00107] In some implementations, the set of constraints can include a text summary of a specified source of online content, which summarizes information available at the specified source of online content (e.g., as described above with reference to the summary apparatus 210). For example, the prompt apparatus 212 can insert, into the initial prompt, one or more of the summaries generated by the summary apparatus 210. In some implementations, the text summary inserted into the initial prompt operates as a contextual constraint that limits content created by the text model 202 responsive to the initial prompt that includes the summary. For example, including the summary in the initial prompt can limit the response created by the text model 202 to a visual description of the object/subject matter described by the summary.

[00108] The set of constraints can also include one or more sample initial prompts and expected responses, which are referred to as a set of sample interactions. The set of sample interactions are structured to guide/influence the structure/content of the response to the initial prompt generated by the text model 202. The sample initial prompts that are included in the set of sample interactions have a similar structure to the initial prompt to which the text model 202 is being asked to respond, and the sample expected responses are provided as positive examples the text model 202 can use to guide the creation of the actual response (e.g., NL output 226) to the initial prompt (e.g., input prompt 224). Continuing with the example above, the set of sample interactions can include:

“Sample Prompt: Write an Image generation prompt for a digital component for: 'Seasonal Specials at Send Flowers | Send Flowers Discount | Florist Flowers Delivery Today' by

'exampleflowers.example.com'. There should be no people or text in the photo. Answer with just the prompt.

Expected Response: A beautiful bouquet of flowers in a vase, with a white background. The flowers should be a mix of different colors and types, and the vase should be clean and simple.

Sample Prompt: Write an Image generation prompt for a digital component for: '75-Inch TVs At ExampleRetailer | TVs From Various Brands | Shop Online At ExampleRetailer by exampleretailer.example.com'. There should be no people or text in the photo. Answer with just the prompt.

Expected Response: A photo of a large, modem living room with a wall- mounted 75-inch TV. The TV is playing a video of a beautiful landscape. The room is decorated with comfortable furniture and plants.

[00109] By providing the set of example interactions, the text model 202 is tuned to (or otherwise weighted toward) generate a response (e.g., NL output 226) similar to the example responses when an input similar to the to the sample prompts is received. In some implementations, the text model 202 can generate embeddings of the sample prompts and expected responses in a multi-dimensional embedding space. When the initial prompt is received, the text model 202 can similarly generate an embedding of the initial prompt in the multi-dimensional space, and when the initial response is sufficiently similar (e.g., less than a specified distance from) the embeddings of the sample prompts, generate a response whose embedding is sufficiently similar (e.g., within a specified distance of) the embeddings of the sample expected responses.

[00110] The prompt apparatus 212 (or another component of the Al System 160) transmits, conveys, communicates, or otherwise submits the constructed initial prompt to the text model 202 as an input prompt 224. The text model 202 uses the initial prompt to generate a response, which is provided back to the Al system in the form of the NL output 226. The NL output 226 can be a set of clauses formatted according to formatting constraints specified in the input prompt 224. For example, if the input prompt 224 includes a formatting constraint specifying that the response should be an image prompt in the form of a python string, the text model 202 can output a visual description similar to those shown above in the form of a python string.

[00111] The prompt apparatus 212 receives the NL output 226 that contains an image summary generated by the text model 202. The image summary specifies visual characteristics of objects specified for inclusion in images generated by the T2I model 204. For example, in the expected response “A photo of a large, modern living room with a wall- mounted 75-inch TV. The TV is playing a video of a beautiful landscape. The room is decorated with comfortable furniture and plants,” the image summary specifies that the living room is large and modern, the TV is 75 inches and playing a video of a beautiful landscape, and that plants and comfortable furniture are in the room.

[00112] The prompt apparatus 212 generates an image prompt 230 that includes the image summary. In some implementations, the NL output 226 of the text model 202 generated responsive to the initial prompt can be simply designated as the image prompt 230 without modification. In these implementations, causing the text model 202 to generate the NL output 226 can be considered to constitute generating the image prompt 230. In some implementations, the prompt apparatus 212 modifies/ supplements the NL output 226 with additional information to generate the image prompt 230. For example, contextual constraints or additional formatting constraints can be added to the NL output 226 to generate the image prompt 230. The contextual constraints can specify, for example, a geographical distribution area of a digital component with which the image will be presented, a time of day/month/year during which the image will be distributed with the digital component, characteristics of an audience to whom the image will be presented, or other contextual information. Including these contextual constraints in the image prompt with the NL output 226 can cause the T2I model 204 to customize the visual characteristics of the images generated based on the context in which the images being generated will be presented.

[00113] Formatting constraints that can be included in the image prompt 230 can include one or more of sizes of images to be generated, resolution of images to be generated, shapes of images to be generated, or animation effects that can be included in the images. Other formatting constraints can also be specified, such as whether the images should be lifelike/photorealistic or cartoon-like caricatures.

[00114] The prompt apparatus 212 submits the image prompt 230 to the T2I model 204, which causes the T2I model 204 to generate multiple images using the image prompt. For example, the T2I model will process the clauses/visual summary from the NL output 226 that was included in the image prompt 230, along with any additional constraints or information included in the image prompt 230 by the prompt apparatus 212, and generate images visually depicting the description/constraints included in the image prompt 230. In some implementations, the T2I model 204 is implemented as a sequence of text-to-image diffusion models that are conditioned on embeddings of the image prompt, and generate progressively higher resolution images. An example text-to-image model is described in Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, Mohammad Norouzi, Photorealistic Text-to-image Diffusion Models with Deep Language Understanding, arXiv:2205.11487, 2022.

[00115] As discussed above, the image generation process can be initiated prior to receiving all of the target visual characteristics of the images to be generated are known. For example, when the Al system 160 is interacting with a digital component provider in realtime to define the visual characteristics of the target (e.g., desired) images, the Al system 160 can submit image prompts 230 serially with more specific visual characteristics so that general aspects of objects in the image can be created while more specific details of the objects are being collected (e.g., from the digital component provider or from an online source). For example, assume that the digital component provider distributes digital components related to pet supplies. In this example, a first image prompt could specify that the images should include a large dog drinking from a dog bowl, while the Al system 160 continues to obtain additional details about the dog and the dog bowl from the digital component provider (e.g., through an interactive chat session). Using this first image prompt, the T2I model 204 can proceed to generate an image of a large dog drinking from a dog bowl while the Al system 160 asks the digital component provider what color the dog bowl should be and whether the dog has short fur or long fur. When the Al system 160 receives the additional visual characteristics specifying the color of the dog bowl and the fur length of the dog, the Al system 160 can submit those visual characteristics to the T2I model 204, which causes the T2I model to modify the images being created to match the specified dog bowl color and fur length. This enables large portions of the images to be defined while the set of visual characteristics continue to be refined, thereby enabling the T2I model 204 to complete the generation of the images more quickly than waiting for all of the visual characteristics to be defined prior to initiating generation of the images. When the T2I model 204 completes generation of the images using the image prompt(s), the generated images 232 are provided to the Al system 160 for post-processing by the post-processing apparatus 214. [00116] The post-processing apparatus 214 of the Al system 160 is implemented using at least one computing device (e.g., a device including one or more processors), and can include one or more language models. The post-processing apparatus 214 is configured to (e.g., specially programmed with code and/or hardware) perform one or more post-processing operations on the generated images 232.

[00117] Performance of the post-processing operations includes an evaluation of one or more visual characteristics of each given image among the generated images 232. The characteristics evaluated can include, for example, one or more of the plausibility of objects depicted by the given image, a level of diversity of the given image to other images among the generated images 232, a relevance of the given image to (i) the image prompt 230, (ii) a reference set of images, (iii) the summary generated by the summary apparatus 210, or (iv) text of a digital component with which the given image will be presented. [00118] The post-processing operations can also include policy checks to ensure that the generated images 232 do not violate content policies of the service apparatus 110. The policy checks can include a copyright check to ensure that the generated images 232 do not violate copyrights, a family-safe check to ensure that the generated images 232 only include family friendly content, or an aesthetic check to ensure that the generated images were generated with a pre-specified aspect ratio. In some implementations, some, or all, of the post-processing operations can be performed on the generated images 232 before the generated images are sent to the client device 206 as output images. In some implementations, some, or all, of the post-processing operations can be performed only after the digital component provider has indicated that a given image should be considered for inclusion in digital components, which can reduce the number of images on which the postprocessing operations must be performed, thereby conserving computing resources that would have otherwise been used to evaluate all the generated images 232.

[00119] The post-processing apparatus 214 can be configured to score each given image among the generated images 232 based on the evaluations. For example, the score for a given image can be increased when the relevance of the given image is deemed more relevant to one or more of (i) the image prompt 230, (ii) a reference set of images, (iii) the summary generated by the summary apparatus 210, or (iv) text of a digital component with which the given image will be presented. On the other hand, the score for the given image can remain the same, or be lowered, when the relevance of the given image is deemed less relevant to one or more of (i) the image prompt 230, (ii) a reference set of images, (iii) the summary generated by the summary apparatus 210, or (iv) text of a digital component with which the given image will be presented. The relevance of the image to any other content can be performed, for example, using existing image-to-image or image-to-text relevance models that have been trained to evaluate semantic similarities between images and other content.

[00120] Similarly, the score for each given image can be increased when the level of diversity of the given image to a specified set of images is higher, and reduced (or remain the same) when the level of diversity of the given image is lower. The level of diversity of an image is a relative measure of visual difference between the image and one or more other images. The level of diversity of an image differs from the relevance of the image to other images because two images can be highly semantically related, and therefore relevant to each other, but have very different visual characteristics.

[00121] For illustration, assume that two images each depict a large dog drinking from a dog bowl, but one of the images depicts a white and brown labradoodle drinking from a green dog bowl that is round, and the other image depicts a grey german shepherd drinking from a silver dog bowl that is square. In this example, the images are highly semantically related because they both depict large dogs drinking from dog bowls, but they also have a high degree of diversity because the visual characteristics of each image are different by portraying different types/colors of dogs, and different shape/color dog bowls.

[00122] The post-processing apparatus 214 can be configured to filter the generated images 232 based on the determined scores. For example, the post-processing apparatus 214 can remove one or more images from consideration for inclusion the output images 234 that are provided to the client device 206 for review by the digital component provider. The threshold score used to delineate between images that are included in the output images 234 from those that are not included in the output images 234, and therefore preventing one or more of the generated images 230 from being included in the output images 234, can be specified by an administrator or designer of the Al system 160. The threshold score used to filter the generated images can be an aggregate score based on the combined scores generated based on the evaluations described above, or separate threshold scores can be used for each evaluation. For example, in some situations, a given image having a plausibility score that is less than a specified threshold plausibility score can be filtered irrespective of the other scores generated using the other evaluations. Using a score to delineate between plausible and non-plausible images removes subjectivity related to a person’s evaluation of plausible and non- plausible, thereby making it an objective evaluation. In some situations, an image that fails one of the policy checks can be removed from consideration for inclusion in the output images 234 based on that failed policy check alone.

[00123] The post-processing apparatus 214 can be configured to rank the generated images 232 based on one or more of the post-processing operations. For example, the postprocessing apparatus 214 can rank each of the generated images 232 based on any of the scores/evaluations discussed above, or a combination of the scores/evaluations discussed above. In some implementations, the post-processing apparatus can sum or average multiple different scores to obtain an aggregate score for a given image. In some implementations, the scores can be weighted based on a relative importance of each evaluation to obtain the aggregate score (e.g., weighted average). Using the aggregate scores, the post-processing apparatus 214 can rank the images, which can be used to determine a presentation order in which the images are arranged for presentation at the client device. In some implementations, the creation and serving of the output images 234 are performed after receiving the request 228. In some implementations, the output images 234 can be generated in an offline process (e.g., prior to receipt of the request 228), and stored in an image database 220 until receipt of the request 228. At that time, one or more of the output images 234 can be retrieved from the image database 220, and served to the client device 206. [00124] When the output images 234 are presented at the client device, instructions provided to the client device 206 with the output images 234 can cause the client device 206 to present one or more of the output images 234 in a user interface with an interactive user interface control that facilitates selection and/or approval for images to be used in digital components distributed on behalf of the digital component provider. For example, in response to interaction with the interactive user interface control corresponding to a given image, a message can be sent to the Al system 160 instructing the Al system (e.g., or another component of the service apparatus 110) to include the given image in digital components distributed on behalf of the digital component provider. In some implementations, the image can be embedded in, or appended to, one or more digital components and presented when the digital component is provided to user devices, as described with reference to FIG. 1.

[00125] FIG. 3 is a flow chart of an example process 300 for creating images with artificial intelligence. Operations of the process 300 can be performed, for example, by the service apparatus 110 of FIG. 1 (e.g., including the Al system 160 and/or language model 170), or another data processing apparatus. The operations of the process 300 can also be implemented as instructions stored on a computer readable medium, which can be non- transitory. Execution of the instructions, by one or more data processing apparatus, cause the one or more data processing apparatus to perform operations of the process 300.

[00126] Textual content is collected from a specified network location (302). As discussed above, the textual content can be collected in various ways. For example, the textual content can be collected by crawling the specified network locations, extracting textual content from one or more digital components distributed on behalf of a digital component provider for whom the images are being generated, from search results snippets, using a site-constrained query that requires the textual content to be collected from one or more network locations specified in a site-constraint. In a specific example, the textual content can be collected from a web page (e.g., item detail page) to which a digital component of the digital component provider is linked.

[00127] The textual content can be collected in an offline process (e.g., independent of and/or prior to when a request for recommended images is received from a client device), or in an online process performed when processing the request received from the client device. Additional details of the textual content collection are provided above with reference to the text collection apparatus 8.

[00128] The collected passages are summarized into a text summary (304). As discussed above, the summarization of the textual content collected can be performed by a language model that is trained to summarize multiple passages of text, such as a large language model. In some implementations, the textual content to be summarized is collected and submitted to a large language model, as described above.

[00129] The summary prompt submitted to the large language model can specify one or more of the following:

- a set of sources that should be used to generate the summary;

- details about the set of sources the language model should consider when summarizing the content;

- factual grounding instructions specifying that the language model should provide citations to the sources used to generate the summary;

- summary constraints specifying information that should not be included in the summary (e.g., information that is not directly supported by the set of sources); and

- formatting constraints specifying how the output of the language model should be formatted (e.g., as bullet points or in paragraph form, with or without an introduction summary, total length (e.g., number of characters or separate clauses); and

- tone constraints specifying a tone of the output (e.g., creative, funny, sad, serious, or from the perspective of a specified entity, such as an artist, engineer, or story writer). [00130] An example summary prompt, and additional details regarding the generation of the summary are provided above with reference to the summary apparatus 210.

[00131] An initial prompt is generated/constructed (306). The initial prompt can be generated, for example, using the specified network location and a set of constraints that limits clauses generated by a language model. The initial prompt differs from the summary prompt, and as described above, the set of constraints can optionally include the summary generated using the summary prompt as a summary constraint. For example, generation of the initial prompt can include inserting at least a portion of the summary into the initial prompt as a contextual constraint that limits/guides content created by the language model to subject matter described the contextual constraint.

[00132] Generation of the initial prompt can also include inserting, into the initial prompt, a set of constraints specifying one or more objects excluded from images generated using clauses generated by the language model using the initial prompt. For example, the set of constraints can specify that images generated using the clauses generated by the language model are to exclude people and text. This type of constraint causes the language model processing the initial prompt to generate clauses that, when included in the image prompt that is submitted to the text-to-image model, causes the text-to-image model to generate images that do not include depictions of people or text.

[00133] In some implementations, the creation of the initial prompt can include providing a reference image, set of reference images, or specifying a network location at which one or more reference images can be accessed. Providing access to these reference images in the initial prompt can enable the language model to evaluate the images, and provide a visual description of the image. The reference images could similarly be included in the summary prompt provided to the summary apparatus and/or in the image prompt that is provided to the text-to-image model so that these models can use the reference images as a guide in constructing their respective outputs.

[00134] An image prompt is generated (308). In some implementations, the image prompt is generated, in part, by including clauses generated by the language model using the initial prompt. For example, when the initial prompt is submitted to the language, the initial prompt causes the language model to generate an image summary. The image summary specifies visual characteristics of objects specified for inclusion in images generated by a text-to- image generative model. As described above, the initial prompt can include at least a portion of the summary generated, for example, by the summary apparatus to help facilitate the generation of the image summary.

[00135] Some, or all, of the image summary can be included in the image prompt that is generated for submission to the text-to-image generative model. For example, at least some of the clauses included in the image summary can be included in the image prompt as part of the image prompt generation process. In some implementations, the image summary is the only information included in the image prompt, and the image summary is formatted for delivery to a text-to-image generative model. For example, generating the image prompt can include causing/instructing the language model to respond to the initial prompt with only the image prompt, and formatted according to a specified programming language. For example, as discussed above, the initial prompt can include a format constraint specifying that the response to the initial prompt is to be formatted in a python string. In some implementations, additional information (e.g., constraints) can be inserted into the image prompt with the clauses from the image summary.

[00136] Multiple images are generated/constructed using the image prompt that includes clauses generated by the language model (310). As discussed above, the clauses generated by the language model can be in the form of an image summary or the image prompt that is submitted to the text-to-image generative model. In some implementations, the image generation is performed by causing the text-image generative model to create the multiple candidate images using the image prompt. For example, as discussed above, the image prompt can describe the visual characteristics of the images that are to be generated, and the text-to-image model can evaluate the visual characteristics and generate an image having the specified visual characteristics.

[00137] One or more post-processing operations are performed (312). In some implementations, the one or more post processing operations include operations that evaluate one or more visual characteristics of each given image among the multiple different images. [00138] Performance of the one or more the post-processing operations can include performing a plausibility analysis on each given image among the multiple images. As discussed above, the plausibility analysis can evaluate the features of an image and classify the image as plausible or not-plausible based on the likelihood of the depiction of an object being possible. As an example, a dog wearing a hat would generally be considered more plausible than a dog having human hands, such that depiction of the dog wearing a hat could have a higher plausibility likelihood than a depiction of a dog with human hands. Therefore, an image depicting a dog wearing a hat could be assigned a higher plausibility score than an image depicting a dog having human hands.

[00139] The post-processing operations can include determining, based on the plausibility analysis, that at least one given image among the multiple images visually depicts an object classified as not-plausible, and filtering (e g., removing) the at least one image from among the multiple images based on its classification of not-plausible. The classification of the given image as not-plausible can be performed, for example, based on the plausibility score. For example, images having a plausibility score less than a specified threshold plausibility can classified as not-plausible, while images having a plausibility score higher than the specified threshold plausibility can be classified as plausible.

[00140] The plausibility scores of images can be generated, for example, by inputting the images into a machine learning model (and/or language model) that is specially trained to evaluate image plausibility. For example, the machine learning model can be trained using a set of training data that include labeled images (e.g., plausible vs. not-plausible). The machine learning model can be adjusted to reduce the error of the model, for example, by adjusting weights of features of the images used to evaluate the plausibility of images. By training a machine learning model to perform the plausibility analysis and generate a plausibility score, this removes the subjectivity from the plausibility analysis, and provides a consistent evaluation across different images.

[00141] Performing the one or more post-processing operations can include evaluating each given image among the multiple images based on one or more of the relevance of the given, the level of diversity of the given image, and one or more policy checks. As discussed above, the policy checks are used to ensure that the generated images 232 do not violate content policies of the service apparatus (or another apparatus or service distributing the images). The policy checks can include a copyright check to ensure that the generated images do not violate copyrights, a family-safe check to ensure that the generated images only include family friendly content, or an aesthetic check to ensure that the generated images were generated with a pre-specified aspect ratio. [00142] In some implementations, the family-safe check can be automated using computer vision. For example, object identification models can be used to ensure that generated images do not include a specified set of objects, such as cigarettes, drug paraphernalia, or weapons. Other models can also be used to detect other potentially offensive content, such as overly revealing clothing, which can be detected using a skin detection mode that is trained to identify skin in images, and evaluate how much skin is being exposed. The evaluation of the relevance and diversity of images is discussed in detail above with reference to FIG. 2.

[00143] The multiple images are ranked (314). In some implementations, each given image among the multiple images is ranked based on scores assigned to the given image. For example, as discussed above, each given image can be scored based on any of the postprocessing operations, and the ranking of images can be performed based on the scores assigned to the images. In some situations, each given image is assigned an aggregate score that considers the score of that given image generated by each post-processing operations. To illustrate, assume that each given image is assigned a relevance score, a plausibility score, and one or more policy check scores. In this example, the final score used to rank the images can be an average, sum, or weighted average of the individual scores assigned to each given image.

[00144] At least one output image is served based on the rankings (316). In some implementations, a set of highest ranked images are served as output images. For example, a set of N images having the highest ranks (e.g., based on their respective aggregate scores), can be identified as output images, and served.

[00145] In some implementations, serving the at least one output image is performed by presenting a set of the multiple images to a digital component provider. For example, as discussed above, the set of the multiple images having a highest ranking can be presented to the digital component provider through a user interface (e.g., account interface). The user interface can be, for example, an Al chat interface through which the digital component provider is interacting with the Al system 160 of FIG. 1. In this example, the Al system 160 can interpret a chat submission by the digital component provider as a request to receive generated images that can be distributed with digital components in the account of the digital component provider. In response, the Al system 160 can initiate the operations discussed above to generate/retrieve images that are then transmitted for presentation in the Al chat interface.

[00146] In another example, the set of highest-ranking images can be presented in a separate tab or welcome dialog displayed with the digital component provider accesses their account (e.g., logs in). In this example, the digital component provider can access/review the images, which can be identified as recommendations, by navigating to the separate tab or interacting with the welcome dialog.

[00147] In either scenario, the images presented to the digital component provider can be accompanied by a corresponding user interface control (e.g., an electronic button), that enables the digital component provider to authorize distribution of one or more of the images. For example, when the digital component provider interacts with the corresponding user interface control for one of the images, the Al system 160 receives data authorizing distribution of that corresponding image, such that the image is now eligible to be included with a set of textual content (e.g., a digital component) distributed on behalf of the digital component provider. After receiving the authorization, the Al system 160 (or service apparatus 100), can include the authorized image in a digital component with the textual content and distribute the digital component with various electronic documents that are served to users.

[00148] FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input/output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In one implementation, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi -threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430. [00149] The memory 420 stores information within the system 400. In one implementation, the memory 420 is a computer-readable medium. In one implementation, the memory 420 is a volatile memory unit. In another implementation, the memory 420 is a non-volatile memory unit. [00150] The storage device 430 is capable of providing mass storage for the system 400. In one implementation, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device. [00151] The input/output device 440 provides input/output operations for the system 400. In one implementation, the input/output device 440 can include one or more of a network interface devices, e.g., an Ethernet card, a serial communication device, e.g., and RS-232 port, and/or a wireless interface device, e.g., and 802.11 card. In another implementation, the input/output device can include driver devices configured to receive input data and send output data to other devices, e.g., keyboard, printer, display, and other peripheral devices 460. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[00152] Although an example processing system has been described in FIG. 4, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[00153] An electronic document (which for brevity will simply be referred to as a document) does not necessarily correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[00154] For situations in which the systems discussed here collect and/or use personal information about users, the users may be provided with an opportunity to enable/disable or control programs or features that may collect and/or use personal information (e.g., information about a user’s social network, social actions or activities, a user’s preferences, or a user’s current location). In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information associated with the user is removed. For example, a user’s identity may be anonymized so that the no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined.

[00155] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[00156] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[00157] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[00158] This document refers to a service apparatus. As used herein, a service apparatus is one or more data processing apparatus that perform operations to facilitate the distribution of content over a network. The service apparatus is depicted as a single block in block diagrams. However, while the service apparatus could be a single device or single set of devices, this disclosure contemplates that the service apparatus could also be a group of devices, or even multiple different systems that communicate in order to provide various content to client devices. For example, the service apparatus could encompass one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, a gaming service, or any other service.

[00159] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[00160] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). [00161] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[00162] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser. [00163] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[00164] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[00165] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. [00166] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[00167] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

CLAIMS What is claimed is:

1. A method, comprising: generating, by an artificial intelligence system, an initial prompt using a specified network location and a set of constraints that limits clauses generated by a language model, wherein the set of constraints specifies one or more objects excluded from images generated using the clauses generated by the language model; generating multiple images using an image prompt that includes the clauses generated by the language model; performing, by the artificial intelligence system, one or more post-processing operations that evaluate one or more visual characteristics of the multiple images; ranking, by the artificial intelligence system, each of the multiple images based on the post-processing operations; and serving, by the artificial intelligence system, at least one output image that is in a set of highest ranked images.

2. The method of claim 1, further comprising: collecting textual content from the specified network location; summarizing the textual content collected from the network location in a text summary; generating, based on the text summary, an image summary specifying visual characteristics of objects specified for inclusion in images generated by a text-to-image generative model; and generating the image prompt that includes the image summary.

3. The method of claim 2, wherein generating multiple candidate images comprises causing the text-to-image generative model to create the multiple candidate images using the image prompt.

4. The method of claim 3, wherein generating the image prompt comprises causing the language model to respond to the initial prompt with only the image prompt in a format of a specified programming language.

5. The method of claim 4, wherein generating the initial prompt comprises specifying, in the set of constraints, that images generated using the clauses generated by the language model are to exclude people and text.

6. The method of claim 1, wherein performing one or more post-processing operations comprises performing a plausibility analysis of each given image among the multiple images; determining, based on the plausibility analysis, that at least one given image visually depicts an object that is classified as not-plausible; and filtering the at least one given image from among the multiple images.

7. The method of claim 6, wherein serving the at least one output image comprises: presenting, through a user interface, a set of the multiple images to a digital component provider; receiving, through the user interface, data authorizing distribution of at least one image in the set of multiple images; and including the at least one image with a set of textual content distributed on behalf of the digital component provider.

8. The method of claim 1, wherein: performing one or more post-processing operations comprises: for each given image among the multiple images: evaluating a relevance of the given image generated by the text-to- image generative model to the image prompt; evaluating a level of diversity of the given image to a specified set of images; ranking each given image among the multiple images comprises: scoring the given image based on the relevance and the diversity; and ranking the given image among other images among the multiple images based on the score.

9. One or more non-transitory computer readable medium storing instructions, that when executed by an artificial intelligence system, causes the artificial intelligence system to perform operations comprising: generating an initial prompt using a specified network location and a set of constraints that limits clauses generated by a language model, wherein the set of constraints specifies one or more objects excluded from images generated using the clauses generated by the language model; generating multiple images using an image prompt that includes the clauses generated by the language model; performing one or more post-processing operations that evaluate one or more visual characteristics of the multiple images; ranking each of the multiple images based on the post-processing operations; and serving at least one output image that is in a set of highest ranked images.

10. The one or more non-transitory computer readable medium of claim 9, wherein the instructions cause the artificial intelligence system to perform operations further comprising: collecting textual content from the specified network location; summarizing the textual content collected from the network location in a text summary; generating, based on the text summary, an image summary specifying visual characteristics of objects specified for inclusion in images generated by a text-to-image generative model; and generating the image prompt that includes the image summary.

11 . The one or more non-transitory computer readable medium of claim 10, wherein generating multiple candidate images comprises causing the text-to-image generative model to create the multiple candidate images using the image prompt.

12. The one or more non-transitory computer readable medium of claim 11, wherein generating the image prompt comprises causing the language model to respond to the initial prompt with only the image prompt in a format of a specified programming language.

13. The one or more non-transitory computer readable medium of claim 12, wherein generating the initial prompt comprises specifying, in the set of constraints, that images generated using the clauses generated by the language model are to exclude people and text.

14. The one or more non-transitory computer readable medium of claim 9, wherein performing one or more post-processing operations comprises performing a plausibility analysis of each given image among the multiple images; determining, based on the plausibility analysis, that at least one given image visually depicts an object that is classified as not-plausible; and filtering the at least one given image from among the multiple images.

15. The one or more non-transitory computer readable medium of claim 14, wherein serving the at least one output image comprises: presenting, through a user interface, a set of the multiple images to a digital component provider; receiving, through the user interface, data authorizing distribution of at least one image in the set of multiple images; and including the at least one image with a set of textual content distributed on behalf of the digital component provider.

16. The one or more non-transitory computer readable medium of claim 9, wherein: performing one or more post-processing operations comprises: for each given image among the multiple images: evaluating a relevance of the given image generated by the text-to- image generative model to the image prompt; evaluating a level of diversity of the given image to a specified set of images; ranking each given image among the multiple images comprises: scoring the given image based on the relevance and the diversity; and ranking the given image among other images among the multiple images based on the score.

17. A system, comprising: one or more memory devices; and one or more computing devices configured to execute code including a set of instructions that, upon execution, cause the one or more computing devices to perform operations comprising: generating an initial prompt using a specified network location and a set of constraints that limits clauses generated by a language model, wherein the set of constraints specifies one or more objects excluded from images generated using the clauses generated by the language model; generating multiple images using an image prompt that includes the clauses generated by the language model; performing one or more post-processing operations that evaluate one or more visual characteristics of the multiple images; ranking each of the multiple images based on the post-processing operations; and serving at least one output image that is in a set of highest ranked images.

18. The system of claim 17, wherein the instructions cause the one or more computing devices to perform operations further comprising: collecting textual content from the specified network location; summarizing the textual content collected from the network location in a text summary; generating, based on the text summary, an image summary specifying visual characteristics of objects specified for inclusion in images generated by a text-to-image generative model; and generating the image prompt that includes the image summary.

19. The system of claim 18, wherein generating multiple candidate images comprises causing the text-to-image generative model to create the multiple candidate images using the image prompt.

20. The system of claim 19, wherein generating the image prompt comprises causing the language model to respond to the initial prompt with only the image prompt in a format of a specified programming language.

21. The system of claim 20, wherein generating the initial prompt comprises specifying, in the set of constraints, that images generated using the clauses generated by the language model are to exclude people and text.

22. The system of claim 17, wherein performing one or more post-processing operations comprises performing a plausibility analysis of each given image among the multiple images; determining, based on the plausibility analysis, that at least one given image visually depicts an object that is classified as not-plausible; and filtering the at least one given image from among the multiple images.

23. The system of claim 22, wherein serving the at least one output image comprises: presenting, through a user interface, a set of the multiple images to a digital component provider; receiving, through the user interface, data authorizing distribution of at least one image in the set of multiple images; and including the at least one image with a set of textual content distributed on behalf of the digital component provider.

24. The system of claim 17, wherein: performing one or more post-processing operations comprises: for each given image among the multiple images: evaluating a relevance of the given image generated by the text-to- image generative model to the image prompt; evaluating a level of diversity of the given image to a specified set of images; ranking each given image among the multiple images comprises: scoring the given image based on the relevance and the diversity; and ranking the given image among other images among the multiple images based on the score.