WO2026015136A1

WO2026015136A1 - Updating output sequences generated by a neural network based on new documents

Info

Publication number: WO2026015136A1
Application number: PCT/US2024/037289
Authority: WO
Inventors: Florian Nils HARTMANN; Matthew Sharifi
Original assignee: Google LLC
Current assignee: Google LLC
Priority date: 2024-07-10
Filing date: 2024-07-10
Publication date: 2026-01-15
Anticipated expiration: 2027-01-10

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating updated output sequences using a language model neural network. One of the methods includes maintaining a data store that stores a plurality of prompt vectors, wherein the plurality of prompt vectors correspond to historic prompts that have been received by a language model neural network; obtaining a new document; a query vector for the new document; performing a search in the data store for one or more most similar prompt vectors to the query vector according to a similarity measure; an input sequence based on (i) the one or more historic prompts that correspond to the one or more most similar prompt vectors and (ii) the new document; and processing, using the language model neural network, the input sequence to generate an updated output sequence.

Description

UPDATING OUTPUT SEQUENCES GENERATED BY A NEURAL NETWORK

BASED ON NEW DOCUMENTS

BACKGROUND

[0001] This specification relates to processing inputs using neural networks to generate output sequences.

[0002] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., another hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.

SUMMARY

[0003] This specification describes a neural network system implemented as computer programs on one or more computers in one or more locations that generates output sequences using a language model neural network and then, as new documents become available, updates the output sequences based on the new documents using the same language model neural network to generate updated output sequences.

[0004] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0005] The described techniques allow for the output sequences generated by using a language model neural network to be incrementally and continuously updated to maintain or even improve their quality compared with when they are initially generated by making use of vector matching techniques. As a particular example, when a news article about a specific news event is published on the Internet, an updated output sequence about the specific news event can be quickly generated based on the content of the news article to ensure the high quality of the output sequence, e.g., to ensure that it is accurate and up-to-date, within minutes or even seconds of its publication, without any user involvement.

[0006] In particular, to determine which output sequence that has already been generated by using the language model neural network is amenable to updating because of a new document, a neural network augmentation system determines a respective similarity score between a query vector for the new document and the respective prompt vectors for historic prompts based on which the output sequences have been generated. For an output sequence that has been generated based on a historic prompt received in the past, the neural network augmentation system determines, based on the respective similarity score, whether to trigger the neural network system to use the language model neural network to generate an updated output sequence based on the new document.

[0007] In this way, because the updated output sequence can incorporate up-to-date information from the new document that was not available at the time the historic prompt was received, the quality, e.g., timeliness, accuracy, or completeness, of the updated output sequence can be improved relative to the output sequence that was initially generated in response to the historic prompt. The updated output sequence can be presented for display, e.g., either in addition to or in place of the initially generated output sequence. This in turn, improves the performance of the language model neural network on many text processing tasks, including question answering and sentence completion tasks.

[0008] Also, the described techniques allow for updating the output sequences in a computationally efficient matter. By selectively updating only a small number of output sequences that are relevant to a new document, the neural network system preserves computing resources, such as processor cycles and memory resources, that would otherwise be consumed by some existing systems that use the new document to update all previously generated output sequences, e.g., by making repeated inference calls to the language model neural network.

|0009| The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[00010] FIG. 1 is a diagram of an example neural network system and an example neural network augmentation system.

[00011] FIG. 2 is a flow diagram of an example process for generating an updated output sequence.

[00012] FIG. 3 is a flow diagram of sub-steps of one of the steps of the process of FIG. 2. [00013] FIG. 4 is a flow diagram of an example process for presenting an updated output sequence.

[00014] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[00015] FIG. 1 is a diagram of an example neural network system 100 and an example neural network augmentation system 120. The neural network system 100 and the neural network augmentation system 120 are examples of systems implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[00016] The neural network system 100 is a system that generates output sequences 1 16, either in response to received requests, or alternatively in response to updates to an external database 160.

[00017] The neural network system 100 can be an output sequence generation system that receives a prompt 112 as part of or in association with the request and generates an output sequence 116 that is a response to the prompt 112. The prompt 112 generally provides context for the output sequence 116.

[00018] For example, the prompt 112 can be an input submitted to the neural network system 100, e.g.. by a user through a client device, a question submitted to the neural network system 100, or a different request that requires a response in the form of an output sequence 116 from the neural network system 100.

[00019] In some cases, the neural network system 100 receives the prompt 112 as text from the client device. In some cases, the neural network system 100 receives the prompt 112 as part of a multi-model input from the client device. In general, a multi-modal input is a combination of two or more different types of data, e.g., two or more of text data, audio data, image data, or graph data. As one example the multi-modal input may include a combination of i) text data representing text in a natural language and ii) pixels of an image or of video or audio data representing values of an audio waveform.

[00020] In some other cases, the neural network system 100 receives a natural language speech input from the user and converts the speech into the prompt 112 by applying a speech recognition engine to the speech. The prompt 112 may be received in the form of a sound (speech) signal, captured by a microphone of the client device, which is converted by a speech recognition engine, i.e., a speech-to-text converter to form the prompt 112.

Alternatively, the prompt 112 may be entered by typing using a data input device of the client device.

[00021] As an example, the neural network system 100 can be a text generation system that generates text sequences, i.e., each output sequence 116 generated by the neural network system 100 is a sequence of text tokens from a vocabulary of text tokens that includes, e.g.. one or more of characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in natural language text. For example, the neural network system 100 can generate text sequences in response to received requests and provide the text sequences for presentation to users, e.g., on a display of the client device of the user.

[00022] As a particular example, the neural network system 100 can be part of a dialog system and the prompt 112 can include audio or text from the most recent conversational turn submitted by a user of the dialog system during the dialog while the output sequence 116 is the next turn in the conversation, e.g., either text or audio that is a response to the most recent conversational turn. Optionally, the prompt 112 can also include one or more historical conversational turns that occurred earlier in the conversation.

[00023] As another particular example, the neural network system 100 can be part of a machine translation system and the prompt 112 can include text in a source language while the output sequence 116 is text in a target language that is a translation of the source text into the target language.

[00024] As another particular example, the neural network system 100 can be part of a computer code generation system and the prompt 112 can be a text description of a desired piece of code or a snippet of computer code in a programming language and the output sequence 116 can be computer code, e.g., a snippet of code that is described by the prompt 112 or a snippet of code that follows the prompt 112 in a computer program.

|00025| As another particular example, the neural network system 100 can be part of a search system that facilitates searching of resources on the Internet. A resource can be any data that can be provided over the Internet. A resource can be identified by a resource address that is associated with the resource. Resources include web pages, word processing documents, portable document format (PDF) documents, images, video, and news feed sources, to name a few.

[00026] In this particular example, the search system can receive search queries submitted by client devices and, in response, identify resources that are relevant to the search uery in the form of search results and return the search results to the user devices in search results pages. A search result page can include search result data generated by the search system that identifies a resource responsive to a search query, and includes a link to the resource. The search result page can additionally include an output sequence 116 that is generated by the neural network system 100 based on a prompt 112 derived from the search query.

[00027] To generate the output sequences 116. the neural network system 100 includes a language model neural network 110. The language model neural network 1 10 can be any appropriate language model neural network that receives an input sequence 114 made up of text tokens selected from a vocabulary of text tokens and generates an output sequence 116 made up of text tokens from the vocabulary.

[00028] The vocabulary of text tokens can include any of a variety of tokens that represent text symbols or other symbols. For example, the vocabulary of text tokens can include one or more of characters, sub-words, words, punctuation marks, numbers, or other symbols that appear in a corpus of text in a natural language and/or a computer programming language.

[00029] For example, the language model neural network 110 can be a Transformer-based language model neural network or a recurrent neural network-based language model. As a particular example, the language model neural network 110 can be an auto-regressive Transformer-based neural network that has, e.g., an encoder-only Transformer architecture, an encoder-decoder Transformer architecture, or a decoder-only Transformer architecture. [00030] Examples of such architectures include those described in Colin Raffel, et al..

Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910.10683. 2019; Daniel Adiwardana. et al. Towards a human-like open-domain chatbot. CoRR, abs/2001.09977. 2020; Tom B Brown, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020; Aakanksha Chowdhery, et al. PaLM: Scaling Language Modeling with Pathways, arXiv preprint arXiv: 2204.02311; and Rohan Anil, et al. Palm 2 technical report. arXiv preprint arXiv: 2305.10403. 2023.

|00031| In some cases, the input sequence 114 includes the prompt 112 that is received by the neural network system 100 as part of or in association with the request for an output sequence 116. That is, the neural network system 100 receives a prompt 112 from a user of the system, generates an input sequence 114 that includes the prompt 112, and then provides the input sequence 114 to the language model neural network 110 for processing to generate the output sequence 116.

[00032] In some other cases, as will be explained further below, the input sequence 114 includes update data that is generated by the neural network augmentation system 120 in response to an update to a database 160. That is, the neural network system 100 receives update data from the neural network augmentation system 120. generates an input sequence 114 that includes the update data, and then provides the input sequence 1 14 to the language model neural network 110 for processing to generate the output sequence 116.

[00033] The neural network augmentation system 120 interfaces the neural network system 100 and the database 160. For example, the neural network augmentation system 120 can be in communication with the neural network system 100 and the database 160 over a network, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, but is separate from the neural network system 100. the database 160, or both.

[00034] The neural network augmentation system 120 includes a vector generation engine 130, a database management engine 140, and a search engine 150.

[00035] The database 160, which can include any appropriate corpus of documents, represents a knowledge base external to the language model neural network 110. While only one database 160 illustrated, in practice the database 160 may include multiple storage components (e.g., multiple drives or multiple databases) that may also span multiple computing devices (e.g., multiple server computers) that operate together to provide the functionalities of the database 160.

[00036] The corpus of documents may, for example, be web pages and other documents, e.g., word processing documents, portable document format (PDF) documents, feed sources, and source code files, accessible through the Internet that include data in one or more modalities, e.g., textual modality, image modality, video modality, or audio modality. Alternatively, the corpus of documents may. for example, be part of a proprietary database, e.g., of a scientific publisher or other organization or part of a proprietary code repository. In practice these documents may be stored across different storage components included in the database 160 that may span different computing devices.

|00037| The vector generation engine 130 is configured to generate a prompt vector for each historic prompt 112 that has been received by the neural network system 100 in the past. The prompt vectors for the historic prompts 112 are stored in a data store 125. Thus, each prompt vector stored in the data store 125 corresponds to a historic prompt 112 that has been received by the neural network system 100 in the past.

[00038] Moreover, copies of the historic prompts 112 can be stored in the data store 125. Further, copies of the historic output sequences 116 that have each been generated by the language model neural network 110 based on a corresponding one of the historic prompts 112 can be stored in the data store 125. The data store 125 can have any appropriate data structure that, for a historic prompt 112. relates the corresponding historic output sequence 116, the corresponding prompt vector, and the historic prompt 112.

[00039] In some implementations, each prompt vector can be an embedding space representation of the corresponding historic prompt 112, and the vector generation engine 130 can generate the prompt vector for each historic prompt 112 by processing the historic prompt 112 using an embedding neural network that includes one or more neural network layers of any appropriate type, or some other machine learning model.

[00040] Such an embedding neural network can be trained on unlabeled training data based on optimizing a self-supervised or unsupervised loss function to generate vectors in an embedding space that have a fixed dimensionality. In some implementations, the embedding neural network can be trained as part of another neural network (that e.g. have a larger architecture) on tasks that involve generating embedding space representations, e.g., text classification or semantic analysis tasks.

[00041] The database maintenance engine 140 is configured to manage the corpus of documents included in the database 160. For example, the database maintenance engine 140 can obtain new documents 162 and add the new documents 162 to the database 160, and optionally, delete existing documents from the database 160.

[00042] The new document 162 can be obtained in many different ways. For example, the database maintenance engine 140 can be configured as a web-crawling engine that crawls web pages that can be found on the Internet. When the web-crawling engine receives a crawled web page, the web-crawling engine adds the crawled web page to the database 160 as a new document 162.

[00043] As another example, the database maintenance engine 140 can receive the new⁷ document as an upload from a user of the system over a data communication network, e.g., using an application programming interface (API) or another data interface made available by the neural network augmentation system 120. When the database maintenance engine 140 receives an uploaded document, the database maintenance engine 140 adds the uploaded document to the database 160 as a new⁷ document 162.

[00044] As a new document 162 is being obtained by the database maintenance engine 140, the vector generation engine 130 can generate a query vector for the new document 162. In some implementations, the query vector can be an embedding space representation of the corresponding new document 162, and the vector generation engine 130 can generate the query vector for the new document 162 by processing the new document 162 using an embedding neural network.

[00045] The vector generation engine 130 may, but need not, use the same embedding neural network or the same machine learning model to generate the query vectors for the new documents 162 and the prompt vectors for the historic prompts 112. In implementations where the same embedding neural network is used, the embedding space of the query vectors for the new- documents 162 may be a shared embedding space with the prompt vectors for the historic prompts 112. That is, the query vectors and the prompt vectors may have a same predetermined dimensionality, i.e., they may include a same number of numeric values, e.g., floating point values or other values.

[00046] For the query vector, the search engine 150 is configured to perform a A-nearest neighbor (ANN) search computation to search through the plurality⁷ of prompt vectors that are stored in the data store 125 for k prompt vectors that are most similar to the query⁷ vector according to some similarity measure.

[00047] For some similarity⁷ measures, e.g.. Euclidean distance or Hamming distance or other distance measures, the most similar prompt vectors are those that are closest to the query⁷ vector (have the smallest similarity⁷ measure with the query vector).

[00048] For some other similarity measures, e.g., inner product similarity or cosine similarity, the most similar prompt vectors are those that have the largest similarity measure with the query⁷ vector.

[00049] ? can generally be any positive integer, i.e., any integer greater than or equal to one, but is generally much smaller than the total number N of prompt vectors in the data store 125. For example, if there are a million prompt vectors, the value of k can be less than one hundred and, in some cases, can be less than fifty, less than ten, or less than five.

[00050] In some cases, the ANN search computation can be an exact ANN search computation. That is. the search engine 150 can compute a respective similarity score between the query vector and each prompt vector according to the similarity measure, and then select the A most similar prompt vectors to the query vector based on the respective similarity scores.

[00051] For example, when the similarity measure is inner product similarity⁷ or cosine similarity, prompt vectors that have the highest similarity scores, or prompt vectors that have similarity scores greater than a given value can be selected.

[00052] Alternatively, as another example, when the similarity⁷ measure is Euclidean distance or Hamming distance or other distance measures, prompt vectors that have the lowest similarity scores, or prompt vectors that have similarity scores lower than a given value can be selected.

[00053] In some other cases, the ANN search computation can be approximate ANN search computation. For example, the search engine 150 can apply⁷ can any known search space pruning, quantization, or other approximation techniques to enable faster and more computationally efficient computation of the respective similarity scores between the query⁷ vector and the prompt vectors. A few example quantization techniques that can be used by the search engine 150 are described in more detail in US patent US10719509 entitled “Hierarchical quantization for fast inner product search"’ and in US patent US 10255323 entitled “Quantization-based fast inner product search”.

[00054] Generally, for a given prompt vector that is generated based on a historic prompt 112, the more similar the given prompt vector is to the query vector for the new document 162, the greater the relevance between the given historic prompt 112 and the new document 162, and therefore, the higher the likelihood that a corresponding historic output sequence 116 which has been generated by the language model neural network 110 based on the given historic prompt 1 12 in the past is amenable to updating because of the new document 162.

[00055] Being “amenable” to updating means that, if not updated, the corresponding historic output sequence will more likely experience a drop in its quality because of the new document than other historic output sequences that have been generated by the language model neural network 110 in the past.

[00056] For each of some or all of the selected k most similar prompt vectors, the neural network augmentation system 120 triggers the neural network system 100 to use the language model neural network 110 to update (or modify) a corresponding historic output sequence 116, i.e., to generate a corresponding updated output sequence, based on processing the new document 162 (which was not available at the time the corresponding historic output sequence 116 was initially generated).

[00057] By selecting k most similar prompt vectors in response to obtaining the new document, and then triggering the neural network system 100 to generate one or more updated output sequences that each reflect an update (or modification) to a corresponding historic output sequence that has been generated in the past, the neural network augmentation system 120 operates in tandem with the neural network system 100 to continuously, e.g., every time a new document is obtained, and automatically, e.g., without any user involvement, update the output sequences to maintain or even improve their quality.

[00058] When an output sequence 116 is initially generated, the output sequence 116 may have a satisfactory quality. As time elapses, how ever, the quality⁷ of the output sequence 116 may decrease in many scenarios. For example, the content of the output sequence 116 may become stale, and may further become inaccurate and/or incomplete over time.

[00059] Suppose, in one example scenario, the neural network system 100 generates an output sequence 116 that is about a specific new s event by using the language model neural network 110. The output sequence 116 is presented for display to the user.

[00060] As time elapses, there are more new documents (e.g., in the form of news articles) published from various news sources that provide additional details or further updates about the specific news event. Gradually, the quality of the output sequence 116 is no longer satisfactory, because it lacks the additional details or the further updates.

[00061] Suppose, in another example scenario, the neural network system 100 generates an output sequence 116 is that is about a specific academic field (e.g., a specific mathematics field, a specific literature field, or a specific science field). The output sequence 116 is presented for display to the user.

[00062] As time elapses, there are more new documents (e.g., in the form of academic articles) published from various academic facilities that describe latest research advancements in the specific academic field. Gradually, the quality of the output sequence 116 is no longer satisfactory , because it lacks the mention of the latest research advancements.

[00063] In these scenarios and many other scenarios, by continuously and automatically updating the output sequences 116 to incorporate up-to-date information from new documents that is previously unavailable, the neural network augmentation system 120 improves user experience with the neural network system 100 which generates the output sequences 116 for presentation to users.

[00064] In particular, by making use of vector matching techniques, every time a new document 162 is obtained, the neural network augmentation system 120 can identify output sequences 116 that are amenable to updating because of the new document 162 and present the updated output sequences for display to the users in a proactive manner with minimal user involvement, e.g., without a user submitting another request for output sequences.

[00065] From another point of view, the neural network augmentation system 120 improves the performance of the language model neural netw ork 110 that is implemented within the neural network system 100 on many text processing tasks, including question answering and sentence completion tasks because it can now make efficient use of up-to-date information to generate timely, accurate, and complete output sequences, and can advantageously do this quickly, immediately after the new information becomes available.

[00066] As another example, the input to the text generation task can be an input other than text, e.g.. an image, and the output sequence can be text that describes the input.

[00067] FIG. 2 is a flow diagram of an example process 200 for generating an updated output sequence. For convenience, the process 200 will be described as being performed by one or more systems of one or more computers located in one or more locations. For example, a neural netw ork augmentation system, e.g.. the neural network augmentation system 120 of FIG. 1. appropriately programmed in accordance with this specification, can perform the process 200. [00068] The neural network augmentation system can be in communication with a neural network system and a database. The neural network system includes a language model neural network. The language model neural network can be any appropriate language model neural network that receives an input sequence made up of text tokens selected from a vocabulary of text tokens and generates an output sequence made up of text tokens from the vocabulary. The database represents to a knowledge base external to the language model neural network. [00069] The neural network augmentation system maintains a data store. The data store stores a plurality of prompt vectors. Each prompt vector corresponds to a historic prompt that has been received by the neural network system in the past. The data store can have any appropriate data structure that, for a historic prompt, relates the corresponding historic output sequence that has been generated by the language model neural network based on the historic prompt, the corresponding prompt vector, and the historic prompt.

[00070] The neural network augmentation system obtains a new document for addition to the database (step 202). The new document can be obtained in many different ways. For example, the neural network augmentation system can obtain the new document by crawling, e.g., the new document may be a new web page that is found by crawling on the world-wi deweb. As another example, the neural network augmentation system can receive the new document as an upload from a user of the system over a data communication network, e.g., using an application programming interface (API) made available by the neural network augmentation system.

[00071] The neural network augmentation system generates a query vector for the new document (step 204). In some implementations, the neural network augmentation system can generate the query vector for the new document by processing the new document using an embedding neural network that includes one or more neural network layers of any appropriate type, or some other machine learning model.

[00072] The neural network augmentation system performs a search in the database for k most similar prompt vectors to the query vector according to a similarity measure (step 206). K can generally be any positive integer, i.e., any integer greater than or equal to one. The similarity measure can be any appropriate similarity measure. Examples of similarity measures include inner product similarity, cosine similarity. Euclidean distance, and Hamming distance, to name just a few.

[00073] For example, as part of the search, the neural network augmentation system can compute a respective similarity score according to the similarity measure between (i) the query vector that has been generated for the new document and (ii) each of the plurality of prompt vectors stored in the data store, and then select k prompt vectors from the plurality of prompt vectors as the k most similar prompt vectors based on the respective similarity scores. [00074] In response to obtaining the new document, the neural network augmentation system generates an input sequence based on the new document (step 208). An example of how the input sequence can be generated is described below with reference to FIG. 3, which is a flow diagram of sub-steps 302-306 of step 208 of the process 200 of FIG. 2.

[00075] In particular, such an input sequence is a database-augmented input sequence that is automatically generated by the system in response to obtaining the new document that is being added to the database. This is contrast to a conventional input sequence which includes a prompt that is usually submitted by a user of the system.

[00076] The neural network augmentation system determines, for each of the historic prompts that correspond to the k most similar prompt vectors, a difference according to a difference measure between (i) the corresponding historic output sequence that has been generated by the language model neural network based on the historic prompt and (ii) the new document (step 302). In practice any difference measure, e.g., a text space difference measure or an embedding space difference measure, can be used.

[00077] As a particular example, the neural network augmentation system can map the historic output sequence and the new document into respective representations that reside in a shared embedding space, and then compute a distance (e.g., a Euclidean distance, a Hamming distance, or some other distance) between the respective embedding space representations of the historic output sequence and the new document. The computed distance can then be used as the difference between the historic output sequence and the new document.

[00078] The neural network augmentation system determines, for each of the historic prompts that correspond to the k most similar prompt vectors, whether the difference satisfies an update distance threshold (step 304).

[00079] For any historic prompt that corresponds to one of the k most similar prompt vectors, in response to determining that the difference satisfies the update distance threshold, the neural network augmentation system generates an input sequence (step 306). In the particular example above, the difference for a historic prompt may satisfy the update distance threshold when the computed distance is greater than a threshold distance in the shared embedding space.

[00080] Alternatively, for any historic prompt that corresponds to one of the k most similar prompt vectors, in response to determining that the difference does not satisfy the update distance threshold, the neural network augmentation system refrains from generating an input sequence. In the particular example above, the difference for a historic prompt may not satisfy the update distance threshold when the computed distance is lower than a threshold distance in the shared embedding space.

[00081] In some implementations, the neural network augmentation system generates a respective input sequence for each historic prompt that corresponds to one of the k most similar prompt vectors and that has a distance that satisfies the update distance threshold. [00082] In these implementations, for each such historic prompt, the respective input sequence can include the new document, data derived (e.g., extracted) from the document, or both. Optionally, the respective input sequence can also include the historic prompt itself, data derived from the historic prompt, or both. Further optionally, the respective input sequence can also include the historic output sequence that has been generated by the language model neural network based on the historic prompt, data derived from the historic output sequence, or both.

[00083] In some other implementations, the neural network augmentation system generates a common input sequence for multiple historic prompts that each corresponds to one of the k most similar prompt vectors and that has a distance that satisfies the update distance threshold.

[00084] In these implementations, for the multiple historic prompts, the common input sequence can include the new document, data derived (e.g., extracted) from the document, or both. Optionally, the input sequence can also include the multiple historic prompts, data derived from the multiple historic prompts, or both. Further optionally, the input sequence can also include the multiple historic output sequences that have been generated by the language model neural network based on the multiple historic prompts, data derived from the multiple historic output sequences, or both.

[00085] The neural network augmentation system provides the input sequence to the neural network system, and triggers the neural network system to process, using the language model neural network, the input sequence to generate an updated output sequence (step 210). The updated output sequence represents an update or a modification to a corresponding historic output sequence. In implementations where a respective input sequence has been generated for each historic prompt, the language model neural network can be used to generate a respective updated output sequence based on each input sequence.

[00086] In some cases, the updated output sequence can be a continuation of the corresponding historic output sequence. For example, the updated output sequence can include an additional sequence of text tokens selected from the vocabulary of text tokens that follows an original sequence of text tokens included in the corresponding historic output sequence.

[00087] In some cases, the updated output sequence can be at least partially identical to the corresponding historic output sequence. For example, the updated output sequence and the corresponding historic output sequence can include at least some identical text tokens, e.g., at either the same positions or different positions within the output sequences.

[00088] In some cases, the updated output sequence can incorporate information from the new document or include information that references the new document. For example, the updated output sequence can include a summary of the new document. As another example, the updated output sequence can include a link to the new document and possibly a textual description of the link.

[00089] As a particular example, the updated output sequence can have the following format: “In response to your previous prompt [HISTORIC PROMPT], I generated [HISTORIC OUTPUT SEQUENCE], I would like to update it to [UPDATED OUTPUT SEQUENCE], This update is based on [NEW DOCUMENT]”, where “[HISTORIC PROMPT]” is the historic prompt previously submitted by a user. “[HISTORIC OUTPUT SEQUENCE]” is the historic output sequence previously generated by the language model neural network based on the historic prompt, “[UPDATED OUTPUT SEQUENCE]” is the updated output sequence generated by the language model neural network based on the input sequence includes the new document, and “[NEW DOCUMENT]” represents information that references the new document, e g., a link to the new document and a textual description of the link.

[00090] In some cases, when the new document includes data other than text, the updated output sequence can also include data other than text. That is. the updated output sequence can be a multi-modal output sequence that includes, e.g., both textual data and image data, or both textual data and audio data, where the image data or the audio data is obtained from the new document.

[00091] The neural network system can then provide the updated output sequence as an output, e.g., by providing the updated output sequence as speech that is played back to a user or as text to be presented on a display of a client device to the user. Notably, some implementations of the neural network system can proactively generate the updated output sequence and present the updated output sequence for display to users without the users asking for it. Additionally or alternatively, after generating the updated output sequence, the neural network system can provide the updated output sequence to one or more external systems for storage or further processing.

[00092] A particular example of how the updated output sequence can be used is discussed in FIG. 4. In this example, the neural network system selectively identifies a subset of the updated output sequences for presentation on the client devices to preserve network bandwidth and to reduce power consumption of the client devices.

[00093] FIG. 4 is a flow diagram of an example process 400 for presenting an updated output sequence. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a neural network system, e.g., the neural network system 100 of FIG. 1, appropriately programmed, can perform the process 400.

[00094] The neural network system determines, for each of the historic prompts that correspond to the k most similar prompt vectors, a difference according to a difference measure between (i) the corresponding historic output sequence that has been generated by the language model neural network based on the historic prompt and (ii) the updated output sequence (step 402). In practice any difference measure, e.g.. a text space difference measure or an embedding space difference measure, can be used. Moreover, the difference measure may or may not be the same as the difference measure used in process 200 discussed above. [00095] The neural network system determines, for each of the historic prompts that correspond to the k most similar prompt vectors, whether the difference satisfies a presentation distance threshold (step 404).

[00096] For any historic prompt that corresponds to one of the k most similar prompt vectors, in response to determining that the difference satisfies the presentation distance threshold, the neural network system provides the updated output sequence for presentation one or more users on one or more client devices (step 406). The one or more users can be the user who submitted the historic prompt. More generally, the one or more users can include any user who submitted one of the historic prompts that correspond to the k most similar prompt vectors.

[00097] Alternatively, for any historic prompt that corresponds to one of the k most similar prompt vectors, in response to determining that the difference does not satisfy the presentation distance threshold, the neural network system refrains from providing the updated output sequence for presentation on any client device. For example, the neural network system can refrain from providing the updated output sequence for presentation if the updated output sequence and the corresponding historic output sequence are too similar to each other according to the difference measure.

[00098] This specification uses the term ‘'configured’’ in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[00099] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[000100] The term '‘data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. [000101] A computer program, which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[000102] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all. and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[000103] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. Tn some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[000104] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry', e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. [000105] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory' or a random access memory' or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memorydevices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to. or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. [000106] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory', media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM. EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[000107] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[000108] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, e.g., inference, workloads.

[000109] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a J AX framework.

[000110] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[000111] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g.. a result of the user interaction, can be received at the server from the device.

[000112] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[000113] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[000114] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[000115] What is claimed is:

Claims

1. A method performed by one or more computers, the method comprising: maintaining a data store that stores a plurality of prompt vectors, wherein the plurality of prompt vectors correspond to historic prompts that have been received by a language model neural network; obtaining a new document; generating a query⁷ vector for the new document; performing a search in the data store for one or more most similar prompt vectors to the query vector according to a similarity measure; generating an input sequence based on (i) the one or more historic prompts that correspond to the one or more most similar prompt vectors and (ii) the new document; and processing, using the language model neural network, the input sequence to generate an updated output sequence.

2. The method of claim 1 , wherein maintaining the data store that stores the plurality of prompt vectors comprises: receiving a historic prompt from a user; processing, using the language model neural network, at least the historic prompt to generate a historic output sequence; generating a prompt vector based on the historic prompt; and storing, in the data store, the prompt vector in association with the historic output sequence.

3. The method of any one of claims 1-2, wherein an input sequence comprises, for each the one or more historic prompts that correspond to the one or more most similar prompt vectors, a historic output sequence that has been generated by the language model neural network.

4. The method of claim 3, wherein the updated output sequence is a continuation of the historic output sequence or is partially identical to the historic output sequence.

5. The method of any one of claims 2-4, wherein generating the input sequence comprises, for each of the one or more historic prompts that correspond to the one or more most similar prompt vectors: determining a difference between (i) the historic output sequence that has been generated by the language model neural network based on the historic prompt and (ii) the new document: determining whether the difference satisfies an update distance threshold; and in response to determining that the difference satisfies the update distance threshold, generating the input sequence.

6. The method of any one of claims 1-5, further comprising, for each the one or more historic prompts that correspond to the one or more most similar prompt vectors: determining a difference between (i) the historic output sequence that has been generated by the language model neural network based on the historic prompt and (ii) the updated output sequence; determining whether the difference satisfies a presentation distance threshold ; and in response to determining that the difference satisfies the presentation distance threshold, providing the updated output sequence for presentation to one or more users.

7. The method of claim 6, wherein providing the updated output sequence for presentation to the one or more users comprises: for each of the one or more historic prompts that correspond to the one or more most similar prompt vectors, providing the updated output sequence for presentation to a user who submitted the historic prompt.

8. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-7.

9. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-7.