WO2025166309A1

WO2025166309A1 - Training neural networks using preference feedback

Info

Publication number: WO2025166309A1
Application number: PCT/US2025/014216
Authority: WO
Inventors: Shangmin GUO; Mathieu Etienne Gerard BLONDEL; Biao Zhang; Tianqi Liu; Thomas Mesnard; Bilal PIOT; Johan FERRET; Mikhail Anatolyevich KHALMAN
Original assignee: DeepMind Technologies Ltd; GDM Holding LLC
Current assignee: DeepMind Technologies Ltd; GDM Holding LLC
Priority date: 2024-02-01
Filing date: 2025-01-31
Publication date: 2025-08-07
Anticipated expiration: 2026-08-01

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network that has a plurality of neural network parameters. One of the methods includes, at each of a plurality of training steps: receiving one or more network inputs, each corresponding to a machine learning task to be performed; for each of the one or more network inputs: sampling from the neural network given the network input to generate a first network output and a second network output for the machine learning task; processing an annotating network input comprising the first network output and the second network output using an annotating neural network to generate an annotating network output; designating, based on at least the annotating network output, one of the first network output or the second network output as a preferred network output; and updating the neural network parameters to optimize an objective function.

Description

Attorney Docket No.: 45288-0426WO1 TRAINING NEURAL NETWORKS USING PREFERENCE FEEDBACK CROSS REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63/548,844, filed on February 1, 2024, the entire contents of which are hereby incorporated by reference. BACKGROUND This specification relates to processing data using machine learning models. Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model. Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output. SUMMARY This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains a neural network using preference feedback. The preference feedback represents a preference between network outputs of the neural network for a given network input. The preference feedback can be generated on-the- fly, e.g., for network outputs generated by the neural network for the given network input during training, using an annotating neural network by prompting the annotating neural network. The neural network is configured to process a network input in accordance with neural network parameters of the neural network to generate a network output. For example, the parameters of the neural network can include weights and, optionally, biases of layers of the neural network. In one aspect there is provided a method for training a neural network that has a plurality of neural network parameters and is configured to process a network input in accordance with the neural network parameters to generate a network output, the method comprising, at each of a plurality of training steps: receiving one or more network inputs, each corresponding to a machine learning task to be performed; for each of the one or more Attorney Docket No.: 45288-0426WO1 network inputs: sampling from the neural network given the network input to generate a first network output and a second network output for the machine learning task; processing an annotating network input comprising the first network output and the second network output using an annotating neural network to generate an annotating network output that represents a preference between the first network output and the second network output; designating, based on at least the annotating network output, one of the first network output or the second network output as a preferred network output; and updating the neural network parameters to optimize an objective function that encourages a higher likelihood assigned to the preferred network output by the neural network during the sampling relative to a likelihood assigned to the other network output of the first network output or the second network output by the neural network during the sampling. In some implementations, receiving one or more network inputs comprises sampling each of the one or more network inputs from a dataset comprising a plurality of inputs for the machine learning task. In some implementations, the annotating network input further comprises one or more features for the preference. In some implementations, the annotating network input assigns a first identifier for the first network output and a second identifier for the second network output, the method further comprises processing a second annotating network input comprising the first network output and the second network output using the annotating neural network to generate a second annotating network output that represents a second preference between the first network output and the second network output, and the second annotating network input assigns the first identifier for the second network output and the second identifier for the first network output. In some implementations, designating, based on at least the annotating network output, one of the first network output or the second network output as a preferred network output comprises: designating, based on the annotating network output and the second annotating network output, one of the first network output or the second network output as the preferred network output. In some implementations, the neural network has been pre-trained using a language modeling objective. In some implementations, the annotating neural network has been pre-trained using a language modeling objective. Attorney Docket No.: 45288-0426WO1 In some implementations, the neural network is a first language model neural network, and the annotating neural network is a second language model neural network. In some implementations, the second language model neural network has a smaller size than the first language model neural network. In some implementations, the annotating neural network is the neural network, and wherein the annotating network input further comprises an annotating network prompt that causes the annotating neural network to compare the first network output and the second network output, and wherein the annotating network prompt comprises one or more few-shot examples for comparing the first network output and the second network output. In some implementations, the machine learning task is a language processing task, and wherein the network input comprises a prompt for the language processing task. In some implementations, the method is implemented on a parallel computing system, the parallel computing system comprising at least three sets of neural network processing hardware coupled to be operable in parallel; and wherein the method comprises: implementing a first instance of the first language model neural network on a first of the sets of neural network processing hardware; implementing a second instance of the first language model neural network on a second of the sets of neural network processing hardware; implementing an instance of the second language model neural network on a third of the sets of neural network processing hardware; wherein the second language model neural network has fewer trained parameters than the first language model neural network; and sampling from the first instance of the first language model neural network to generate the first network output in parallel with sampling from the second instance of the first language model neural network to generate the second network output; to reduce a processing time to generate the first and second network outputs towards a processing time of the annotating neural network to generate the annotating network output. In some implementations, the method further comprises sampling from the first and second instances of the first language model neural network for a current training time step in parallel with processing, using the annotating neural network, the first network output and the second network output sampled from the first and second instances of the first language model neural network at a previous training time step, to generate an annotating network output. In some implementations, the neural network is a first visual language model neural network including an image encoder neural network and a text encoder, and wherein the annotating neural network is a second language model neural network or visual language Attorney Docket No.: 45288-0426WO1 model neural network; wherein the machine learning task comprises an image processing task; wherein receiving the one or more network inputs, each corresponding to the machine learning task to be performed comprises receiving an image and text in a natural or computer language; and wherein sampling from the neural network given the network input to generate the first network output and the second network output for the machine learning task comprises, for each network output: processing pixels of the image using the image encoder neural network to generate a representation of the image; processing the text using the text encoder to generate a sequence of token embeddings representing the text; and processing the representation of the image and the sequence of token embeddings using the first visual language model neural network to perform an image processing task defined by the text. According to another aspect there are provided one or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the methods described herein. According to another aspect there is provided a system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the methods described herein. Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The system can improve the performance of neural networks. For example, the system can train a neural network to attain a threshold of performance on a machine learning task that aligns to human preferences without requiring the collection of feedback from a human, which can be infeasible or unscalable. For example, the system can use an annotating neural network to generate an annotating network output that represents a preference between network outputs from the neural network for the same network input. The system can update the neural network parameters based on the preference. Conventional systems for training neural networks to align to preferences often train neural networks on a pre-collected preference dataset, which can make training more efficient compared to training neural networks to optimize reward models. However, because the preference dataset is pre-collected, these systems may not be able to train the neural networks with feedback for the outputs of the neural networks, leading to offline learning. Thus the neural networks may generalize poorly at inference as the neural networks do not get feedback on the outputs generated by the neural networks over training. In addition, because the preference dataset is pre-collected, e.g., generated using another neural network distinct Attorney Docket No.: 45288-0426WO1 from the neural networks being trained, these systems may train the neural networks on preference data that reflects a distribution, e.g., of the other neural network, that does not apply to the outputs of the neural networks as they evolve over training, leading to off-policy learning. Thus the preference dataset may not be representative for the evolving outputs of the neural networks, and the neural networks may overfit to the pre-collected preference dataset, resulting in stability issues during training and poor generalization at inference. Other systems for training neural networks to align to preferences may train neural networks to optimize reward models that have been fit based on preference data. However, these systems can be susceptible to reward hacking, and computing the gradient in these systems can be computationally expensive. In addition, the preference data that the reward models have been fit on may reflect a distribution that does not apply to the outputs of the neural networks as they evolve over training. The system described in this specification can efficiently (e.g., with respect to training time, computational resources, etc.) train a neural network using online feedback. For example, the system can train the neural network to align to preference feedback for the outputs of the neural network generated during training, where the preference feedback is automatically generated. The system can automatically generate online feedback for outputs of the neural network by prompting an annotating neural network. For example, the system can sample from the neural network given a network input to generate a first network output and a second network output. The system can process an annotating network input that includes the first network output and the second network output using an annotating neural network. The annotating neural network generates an annotating network output that represents a preference between the first network output and the second network output. The system can designate a preferred network output based on the annotating network output. The system can then update the neural network parameters to optimize an objective function that encourages a higher likelihood assigned to the preferred network output by the neural network during the sampling relative to a likelihood assigned to the other network output by the neural network during the sampling. Thus the system can use an annotating neural network to approximate preferences for the outputs of the neural network, without gathering human feedback which can be infeasible. Furthermore, the annotating neural network can be a pre-trained language model neural network, and does not require the gathering of a preference dataset for training a reward model, or training the reward model. Furthermore, in some examples, because the annotating neural network is not trained on a preference dataset, Attorney Docket No.: 45288-0426WO1 the annotating neural network is more generalizable to outputs of the neural network as they evolve over training compared to a reward model that is trained on a fixed preference dataset. As another example, the system can use the annotating neural network to approximate preferences for the outputs of the neural network, without requiring the training of a reward model for which computing the gradients can be computationally expensive. In addition, the system described in this specification can update the neural network parameters to optimize a loss function for which the gradients can be efficiently computed. The training system can therefore train a neural network more quickly and by using fewer computing resources than some existing conventional systems. Furthermore, the preferences that the system trains the neural network to align to can easily be modified or updated during training. For example, the annotating network input can include one or more features for the preference. For example, the features can be defined using natural language text in a prompt for the annotating network. The system can thus train the neural network to align to preferences that are flexible. Conventional systems may require, for example, retraining a reward model to align with updated preferences, and/or gathering a new preference dataset, which can be computationally expensive. The system can further save computing time and resources by using an annotating neural network that is smaller than the neural network. For example, if the neural network is a large language model neural network, the annotating neural network can be a smaller large language model neural network with fewer parameters and/or layers. The system can use the annotating neural network to generate an annotating network output using less computing time and resources than a larger annotating neural network. The system can thus provide for a scalable training of the neural network. Furthermore, the parameters of the annotating neural network can be predetermined, e.g. frozen or not updated during training of the neural network, further saving computing time and resources during training of the neural network. In particular, the system can reduce the number of memory access operations and communication between sets of neural network processing hardware in a parallel computing system where the neural network and the network for generating preferences are implemented on different sets of neural network processing hardware. For example, conventional systems may use the neural network as the network for generating preferences. The parameters of the neural network may be updated at each training stage, resulting in a large number of memory read/write operations to update the parameters of the separately implemented network for generating preferences. In addition, communicating the updates for the parameters to the set of neural network processing Attorney Docket No.: 45288-0426WO1 hardware that implements the network for generating preferences may require a large amount of network bandwidth. The system can thus reduce the computing time and resources required for memory access operations and dedicated to network communication over the training of the neural network by using an annotating neural network for generating preferences that is separate from the neural network and for which the parameters are not updated during training of the neural network. The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS FIG.1 shows an example neural network training system. FIG.2 is an example illustration of updating the neural network parameters for an example network input. FIG.3 is a flow diagram of an example process for training a neural network. FIG.4 is a flow diagram of an example process for training a neural network on a parallel computing system. FIGS.5A-5C show the performance of example trained neural networks. Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION FIG.1 shows an example neural network training system 100. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The training system 100 is a system that trains a neural network 110 to perform a machine learning task. The neural network 110 includes multiple neural network parameters 116. The neural network 110 is configured to process a network input in accordance with the parameters 116 of the neural network 110 to generate a network output. Attorney Docket No.: 45288-0426WO1 For example, the system can train the neural network 110 to perform any appropriate kind of machine learning task, i.e., can be configured to receive any kind of digital data input and to generate any kind of output based on the input. Some examples of machine learning tasks the neural network 110 can be configured to perform follow. In implementations the neural network 110 can be configured as, or include, a generative (large) language model or a multi-modal model, e.g., a visual and language model, to perform these example machine learning tasks. As an example, the neural network 110 can be configured to perform a natural language processing (NLP) task, i.e., receive input data that includes text and process the input data to generate a sequence of text responsive to the input data, such as question answering, sentence completion, etc. As another example, the neural network 110 can be configured to generate code for computer programs (e.g., in a programming language such as Python, Java, C++, etc.) to perform tasks specified by the input data. For example the input may represent words, wordpieces or characters in a first natural language and the output may represent instructions in a computer programming or markup language, or instructions for controlling an application program to perform a task e.g. build a data item such as an image or web page. As another example, the neural network 110 is configured to perform an image generation task, i.e., receive input data and to process the input data to generate an image for the input data. For example, the input data can include text and/or an image. As another example, the neural network 110 is configured to perform a video generation task, i.e., receive input data and to process the input data to generate a video for the input data. For example, the input data can include text, an image, and/or a video. Processing input data that includes an image includes processing the intensity values of the pixels of the image. As another example, the neural network 110 is configured to perform an audio generation task, i.e., receive input data and to process the input data to generate audio for the input data. For example, the input data can include text, audio, an image, and/or a video. As another example, the task can be a style transfer task to match a style of, e.g., images, video, audio, etc. specified by the input data. For example, the neural network 110 can be configured to output, e.g., sample images, sample videos, sample audio, etc., that match styles of the input data. Attorney Docket No.: 45288-0426WO1 As another example, the task can be a transcription task in which the neural network 110 is configured to produce text transcriptions for, e.g., video, audio, etc. specified by the input data. As another example, the task can be a summarization task in which the neural network 110 is configured to produce text summaries of, e.g., text, images, video, audio, etc. specified by the input data. As another example, the task can be a health prediction task, where the input is electronic health record data for a patient and the output is a prediction that is relevant to the future health of the patient, e.g., a predicted treatment that should be prescribed to the patient, the likelihood that an adverse health event will occur to the patient, or a predicted diagnosis for the patient. In some cases, the machine learning task is a multi-modal processing task that requires processing multi-modal data. In general, multi-modal data is a combination of two or more different types of data, e.g., two or more of audio data, image data, text data, or graph data. As one example the multi-modal data may comprise audio-visual data, comprising a combination of pixels of an image or of video and audio data representing values of a digitized audio waveform. As another example the multi-modal data may comprise a combination of i) text data representing text in a natural language and ii) pixels of an image or of video or audio data representing values of an audio waveform. Optionally, but not necessarily, the different types of data may represent the same or overlapping objects using the different modalities (types), and when processing multi-modal data the data may be mapped into a common embedding space. As a particular example, the task is a multi-modal processing task that requires processing both text and image inputs, so that the neural network 110 includes both a computer vision neural network and a text processing neural network. That is, the target output to be generated by the computer vision neural network for a given image depends on one or more outputs generated by the text processing neural network for one or more corresponding text inputs (and vice versa). Examples of such tasks include image captioning, image generation, and so on. More generally, the multi-modal processing task may correspond to any of the tasks previously described for any of the types of data making up the multi-modal combination. The neural network 110 can generally have any appropriate architecture for the machine learning task. Examples of neural network architectures that the neural network 110 can have include convolutional architectures, recurrent architectures, fully-connected architecture, e.g., multi-layer perceptron (MLP) architectures, generative Attorney Docket No.: 45288-0426WO1 (large) language model or multi-modal model architectures, e.g., encoder-only Transformer architectures, encoder-decoder Transformer architectures, decoder-only Transformer architectures, other attention-based architectures, and so on. For example, the neural network can be a language model neural network, a visual language model (VLM), also referred to as a visual language model neural network, an image generation model, or a video generation model. Multimodal models and in particular, VLMs, are described in more detail below. To train the neural network 110 to perform a machine learning task, the system 100 receives one or more network inputs such as the network input 102 for the machine learning task. For example, the network input 102 can include a prompt, e.g., a text sequence. For each of the network inputs, the system 100 samples from the neural network 110 given the network input to generate a first network output and a second network output for the machine learning task. For example, the system 100 samples from the neural network 110 given the network input according to the current parameters of the neural network 110. For example, the system 100 can process the network input 102 using the neural network 110 to generate the first network output. The system 100 can process the network input 102 using the neural network 110 to generate the second network output. In some examples, the system 100 can vary parameters, e.g., temperature or random seed, for generating the first network output and generating the second network output. The system 100 processes an annotating network input 120 using an annotating neural network 130 to generate an annotating network output 132 that represents a preference between the first network output and the second network output. The system 100 can generate the annotating network input 120 to include the first network output and the second network output. The annotating network input 120 can assign an identifier for the first network output and the second network output. The annotating network output 132 can represent the preference using the identifier for the first network output or the second network output. For example, the annotating network output can include a distribution for the first network output and the second network output. The distribution includes the probability of generating the identifier for the first network output and the probability of generating the identifier for the second network output. For example, the annotating network output can include a distribution for the first network output and the second network output that includes the probability of generating the identifier for the first network output and the probability of generating the identifier for the second network output. The probability of generating the identifier for the first network output can represent a preference score for the first network output, and the probability of Attorney Docket No.: 45288-0426WO1 generating the identifier for the second network output can represent a preference score for the second network output. The annotating neural network 130 can have any appropriate architecture for processing text and the first network output and the second network output. For example, the annotating neural network 130 can be a language model neural network or a multimodal model such as a visual language model neural network. Multimodal models are described in more detail below. As particular examples, the annotating neural network can be the neural network described in Aakanksha Chowdhery, et al. PaLM: Scaling Language Modeling with Pathways, arXiv preprint arXiv:2204.02311; Rohan Anil, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023; or Gemini (“Gemini: A Family of Highly Capable Multimodal Models, Gemini Team, Google”). In some examples, the annotating neural network 130 can be a smaller neural network, e.g., having a smaller number of parameters, than the neural network 110 being trained, which can lead to improvements in computational efficiency as discussed further below. In some examples, the annotating neural network 130 can be a larger neural network, e.g., having a larger number of parameters, than the neural network 110 being trained, which can generate an annotating network output 132 that represents preferences that potentially align more closely to the features of the annotating network input than the preferences represented by an annotating network output generated using a smaller neural network. In some examples, the annotating neural network 130 can be a neural network of a similar size, e.g., having a similar number of parameters, as the neural network 110 being trained. In some examples, the annotating network input can include one or more features for the preference. For example, the annotating network input can include a prompt for the annotating neural network, also referred to as an annotating network prompt. The annotating network prompt includes or defines features of the first network output and the second network output for the annotating neural network to determine the preference. The features can be desired features for the network outputs of the neural network. That is, a particular network output that has the features, or follows the features more closely, than another network output, should be preferred over the other network output. For example, example features can include helpful, short, honest, coherent, accurate, has good coverage, and/or overall quality. Attorney Docket No.: 45288-0426WO1 In some examples, the features can be default features, e.g., maintained by the system 100. In some examples, the system 100 can maintain features correspond to types of tasks. For example, the default feature for a task that includes image generation can include quality of the image generated. In some examples, the system 100 can receive data representing the features, e.g., from a user. Thus the system 100 can train the neural network 110 to align to preferences that are flexible or easily updated during training. The system 100 designates, based on at least the annotating network output 132, one of the first network output or the second network output as a preferred network output 134. For example, the system can designate the network output corresponding to the identifier that has a higher probability in the distribution of the annotating network output 132 as the preferred network output. In some examples, the system can address potential position bias among the identifiers for the first network output and the second network output by generating a second network output using the annotating neural network for a second annotating network input. The second annotating network input can assign different identifiers to the first network output and the second network output than in the annotating network input. The system can designate the preferred network output based on the annotating network output and the second annotating network output. The system 100 updates the neural network parameters to optimize an objective function that encourages a higher likelihood assigned to the preferred network output 134 by the neural network 110 during the sampling relative to a likelihood assigned to the other network output of the first network output or the second network output by the neural network 110 during the sampling. More specifically, the training system 100 performs the training over multiple training steps. At each training step, the training system 100 updates the parameters of the neural network 110 using the one or more network inputs for the training step. Thus, by repeatedly performing training steps, the training system 100 repeatedly updates the values of the parameters 116 of the neural network 110 to determine trained values of the parameters 116 that will cause the neural network 110 to perform well on the machine learning task. More specifically, at each training step, the training system 100 computes, using the one or more preferred network outputs for the one or more network inputs, a gradient of an objective function with respect to at least some of the parameters 116 of the neural network 110. Some example objective functions are described below with reference to FIG.2. Attorney Docket No.: 45288-0426WO1 As an example, a non-preferred network output is the other network output out of the first network output and the second network output that was not designated as the preferred network output. An example objective function can measure a ratio of a preferred likelihood assigned to the preferred network output by the neural network during the sampling to a non- preferred likelihood assigned to the non-preferred network output by the neural network during the sampling. The training system 100 then uses an optimizer to determine an update to the values of the parameters 116 of the neural network 110 from the gradients. For example, the optimizer can generate gradient-based updates for the current values of the parameters. For example, the optimizer can be an Adafactor optimizer, an Adam optimizer (that uses an Adam update rule), or a RMSProp optimizer. Updating the neural network parameters can include generating a combined update from the gradients for all of the one or more network inputs, and applying the combined update to the neural network parameters. After training, the training system 100 or a different inference system 150 deploys the trained neural network 110 on one or more computing devices to perform inference, i.e., to generate new network outputs 114 for the machine learning task for new network inputs 108. In examples where the neural network 110 is a first language model neural network and the annotating neural network 130 is a second language model neural network, the second language model neural network may have fewer trained parameters than the first language model neural network. For example, the second language model neural network may have 10x, 100x, 1000x fewer trained parameters than the first language model neural network. Therefore, the second language model neural network can be stored in a smaller memory portion compared to the first language model neural network. In some embodiments, the first language model neural network is stored on a first computing system with an associated memory portion and the second language model neural network is stored on a second computing system with an associated memory portion. In these embodiments, the computing system and/or processor(s) on which the first language model neural network is run may be larger than the computing system and/or processor(s) on which the second language model neural network is run. Larger may be interpreted as, for example, having a larger capacity memory (e.g. for storing neural network parameters). Any computing system has a finite amount of computing resources. Computing resources may include, for example, computational power, computation speed, memory capacity (e.g. storage capacity of a Attorney Docket No.: 45288-0426WO1 memory portion), electrical power, bandwidth, queue length, circuit area usage, cooling requirements, or any other resource. By utilizing a second language model neural network that is smaller than (e.g. has fewer parameters than) the first language model neural network, a weak-to-strong generalization approach is used. The first and second language model neural networks perform different tasks, e.g. the first model performs the generation of network outputs whereas the second language model neural network performs annotation. It has been found that a weak teacher performing the annotation, e.g. the second language model neural network, can perform efficient generalization without requiring significant storage requirements. That is, the small second language model neural network can efficiently align a larger first language model neural network with reduced computational resource requirements compared to a larger second language model neural network. The neural network 110 or the annotating neural network 130 may include a first visual language model (VLM) neural network. The visual language model neural network may include an image encoder neural network and a text encoder. The annotating neural network may comprise a second language model neural network or visual language model neural network. The machine learning task may comprise an image processing task. At least one of the inputs and/or outputs may comprise image data. At least one other of the inputs and/or outputs may comprise text (e.g. token embeddings representing text or spoken words in a natural or computer language). The image processing task to be performed may be specified by the words in the natural or computer language. The visual language model neural network may be considered a multimodal machine learning model. While image data and text data are specifically referred to in examples, any type of multimodal data may be used. A multimodal machine learning model has a multimodal input configured to receive a first multimodal input of a first modality and a second multimodal input of a second, different modality. As used herein a “modality” refers to a type of data, and thus a multimodal machine learning model is one that can process multiple different types of data. The first multimodal input may comprise, e.g. a text input to receive a sequence of text or audio data representing values of an audio waveform, e.g. instantaneous amplitude data or time-frequency domain data; or non-image sensor data representing an environment with which an agent controlled by the multimodal machine learning model interacts. The second multimodal input may receive an image or video. The multimodal machine learning model may be configured to jointly process encoded versions of the first and second multimodal inputs. Attorney Docket No.: 45288-0426WO1 Text may be received, e.g., as a series of encoded characters, e.g. UTF-8 encoded characters; such “characters” can include Chinese and other similar characters, as well as logograms, syllabograms and the like. The multimodal machine learning model can include a text encoder that processes a sequence of text to represent the text as a series of text tokens from a vocabulary of text tokens, e.g. that each represent words, wordpieces or characters in a natural or computer language. The computer language may be any formal language used to communicate with a computer, e.g. a markup language, or a command or configuration language, or a data exchange language such as JSON, or a programming language. The model output may comprise any form of output appropriate to the machine learning task performed by the multimodal machine learning model. For example the model output may comprise text in a natural or computer language that defines a result of the task, e.g. for tasks such as image captioning, visual question answering, or object detection or instance segmentation. Also or instead the model output may comprise data defining an image, video or audio object, e.g. in a generative task; or the model output may comprise non-textual action selection data for selecting an action to be performed by an agent controlled by the model. As another example the model output may also or instead define an intermediate step to be performed during the task, e.g. a call to a software API for a software tool that is used when performing the task; the multimodal input may then receive an output from the software tool that is used to generate a final model output that performs the task. A few particular examples of model output are given later. Such a multimodal model (e.g. a VLM) can be trained using very large (but possibly noisy) datasets in which text is paired with an image and/or with one or more other types of data, e.g. audio data, or data relating to the operation of an agent acting in an environment to perform a variety of tasks. Such a model can be trained, e.g., using self-supervised learning. The pairing can often be imperfect, and the training dataset can, but may not, include any actual examples of a particular task to be performed, but nonetheless an ability to perform a particular task can emerge. There are many examples of suitable, publicly available training datasets. A trained VLM may be referred to as a pre-trained VLM. Some example multimodal machine learning models with which the techniques described herein may be used include: Flamingo (Alayrac et al. arXiv:2204.14198); ALIGN (Jia et al., arXiv:2102.05918); PaLI (Chen et al. arXiv:2209.06794) and PaLI-X (Chen et al. arXiv:2305.18565); and Gemini (“Gemini: A Family of Highly Capable Multimodal Models, Gemini Team, Google”). These references also include indications of training datasets that may be used to train the respective models. Attorney Docket No.: 45288-0426WO1 As used herein an image may be any still or moving image, i.e. the image may be part of a video, in 2D or 3d, and may be a monochrome, color or hyperspectral image, i.e. comprising monochrome or color pixels. As defined herein an “image” includes a point cloud e.g. from a LIDAR system, and a “pixel” includes a point of the point cloud. The image, during or after (pre-)training and/or fine tuning may, e.g. have been captured by a camera or other image sensor from the real world, and objects in the image or video may comprise physical objects, represented by the image or video. In general, performing an image processing task using a multimodal neural network such as a visual language model (VLM) neural network, that includes an (a trained) image encoder neural network, can involve providing an image to the image encoder neural network to generate a representation of the image, in particular of features of the image. The representation of the image is then processed to perform the image processing task. In general techniques for processing image representations to perform a wide range of image processing tasks are well known. In general a video processing task can be performed by processing the representations of multiple images representing a sequence of video frames. An example involves generating an output that requires reasoning, e.g. spatio- temporal reasoning, to respond to a natural language query input, e.g. relating to a moving image (video). For example such a query may require predictive reasoning (“what will happen next”), counterfactual reasoning (“what would happen in a different circumstance”), explanatory reasoning (“why did something happen”), or causal reasoning generally. For example the image representation can be used to detect objects in the video frames and provide information relating to the detected objects in response to a query. The query may comprise, for example, a request for a prediction of a future event or state relating to one or more of the objects (e.g. “will objects X and Y collide?”), or a request for conditional or counterfactual information relating to one or more of the objects (e.g. “what event would [not] happen if object X is modified, moved or absent?”), or a request for analysis of the video frames to determine a property or characteristic of one or more of the objects (e.g. “how many objects of type Z are moving?”). The output may, for example, be in the form of a yes/no answer, or may define a probability distribution over a set of possible answers; or the response may define the location of an object. Such systems can be used to predict whether or not two objects will collide, or how this may be avoided. The output may be used e.g. to provide a warning and/or to control motion of one or more of the objects. In an example, receiving the one or more network inputs, each corresponding to the machine learning task to be performed, comprises receiving an image and text in a natural or Attorney Docket No.: 45288-0426WO1 computer language. In this example, sampling from the neural network given the network input to generate the first network output and the second network output for the machine learning task comprises, for each network output processing pixels of the image. Processing pixels of the image may comprise using the image encoder neural network to generate a representation of the image. Processing pixels of the image may further comprise processing the text using the text encoder to generate a sequence of token embeddings representing the text (e.g. for use in processing the image). Processing pixels of the image may further comprise processing the representation of the image and the sequence of token embeddings using the first visual language model neural network to perform an image processing task defined by the text. In another example, receiving the one or more network inputs, each corresponding to the machine learning task to be performed, comprises receiving text in a natural or computer language. In this example, sampling from the neural network given the network input to generate the first network output and the second network output for the machine learning task comprises, for each network output processing pixels of an image. The image may be a generated image, for example generated by the neural network. The image may be selected, for example from a set of images. Processing pixels of the image may comprise using the image encoder neural network to generate a representation of the image. Processing pixels of the image may further comprise processing the text using the text encoder to generate a sequence of token embeddings representing the text (e.g. for use in processing the image). Processing pixels of the image may further comprise processing the representation of the image and the sequence of token embeddings using the first visual language model neural network to perform an image processing task defined by the text. Example tasks for multimodal models such as VLMs In general a multimodal machine learning model can be trained to perform any sort of machine learning task or tasks. After the multimodal machine learning model has been trained it can be deployed for use in performing the machine learning task(s). For instance, the machine learning model can be deployed in an environment that enables users to provide requests for the machine learning model to process specified multimodal inputs to generate corresponding model outputs. Users can provide the requests, e.g., by way of a user interface or through an application programming interface (API). The requests can be transmitted from a user device (e.g., over a data communication network, e.g., the internet) to one or more computers implementing the machine learning model, e.g., in a data center. The machine Attorney Docket No.: 45288-0426WO1 learning model can process multimodal inputs specified by user requests to generate corresponding model outputs, and then transmit the model outputs to user devices (e.g., over a data communication network). In some implementations, after training, a particular task that is to be performed by the multimodal machine learning model can be described by part or all of the sequence of text in the multimodal input to the model. For example in a multimodal input that includes an image such a prompt might specify “Generate a caption”, “Generate a description”, “Answer the following question: [about the image or video]”, or “Detect a person”. Where the model is used for an agent control task a prompt may define “Take the knife out of the drawer”, or “Q: What action should the robot take to take the knife out of the drawer?”. Also or instead such a prompt may give one or more examples of a task to be performed. A multimodal machine learning model can be trained on multiple natural and/or computer languages and the prompt may then specify a language to use. A few examples of some machine learning tasks that can be performed by a model trained as described herein follow. The tasks described below may be tasks that require spatial awareness or other context from the image or video. For example, a prompt may ask “What is the object in the top left corner?” As one example the task may comprise an object or action detection task. A task- specific training data item may comprise an image or video containing one or more objects or actions, and a sequence of text. The sequence of text may describe or otherwise label the object(s) or action(s) and may include text giving bounding box coordinates for the object(s) or action(s). After training, when the model is used in inference, the model output may comprise or represent text that describes or otherwise labels detected object(s) or action(s) in the image input, and may include bounding-box coordinates for the detected object(s) or action(s), e.g. "102090100 cat 2030100100 dog”. As another example the task may comprise a classification task, e.g. an object or action classification task. A task-specific training data item may comprise an image or video item containing one or more objects or actions and a sequence of text. The sequence of text may describe or otherwise classify the object(s) or action(s). After training, when the model is used in inference, the model output may comprise data, e.g. text, that classifies the object(s) or action(s) in the image input into one of a plurality of classes. As another example the task may comprise an image or video describing task, e.g. a captioning task (which, as used here, includes an audio description task to explain what is happening in a video). A task-specific training data item may comprise an image or video Attorney Docket No.: 45288-0426WO1 and a sequence of text describing the image or video. After training, when the model is used in inference, the model output may comprise data, e.g. text, describing the image or video. For example the model output may provide a caption or description or it may count objects in the image or video, or it may provide some other form of description. As another example the task may comprise an image or video question-answering task. A task-specific training data item may comprise an image or video and a sequence of text that describes the image or video. After training, when the model is used in inference, the model output may comprise data, e.g. text, that answers a question about the second modality input specified in a prompt sequence of text, e.g. as described above. This may be used, e.g., to answer questions about visual plots and charts or about sounds. As another example the task may comprise a character or word recognition task, e.g. an OCR (optical character recognition) task. A task-specific training data item may comprise an image or video and a sequence of text that includes text that is depicted in the image or video, or that is represented as speech in the audio item. After training, when the model is used in inference, the model output may comprise text that represents characters or words in the second modality input, e.g. in a natural language. As another example the task may comprise a still or moving image generation task; Google DeepMind Gemini is an example of a system that can generate an image output. For example an image such as a plot or chart may be decoded from one or more (language) tokens generated by the system. A training data item for such a system may comprise an image or video and a sequence of text that describes the image or video. After training, when the model is used in inference, the model output may comprise data for an image or video, e.g., image data defining values for pixels of a still or moving image, and the sequence of text in the multimodal input to the model may describe or characterize the image or video to be generated. As another example the task may comprise a computer language text generation task. A task-specific training data item may comprise an image or video and a sequence of text in a computer language for generating the image or video. After training, when the model is used in inference, the model output may comprise text in the or another computer language for generating or rendering an image or video, e.g. a web page, plot, or chart. In another example of a computer language text generation task a task-specific training data item may comprise an image or video and a sequence of text in a computer language for performing a task in relation to the image or video, e.g. a data processing task that involves analyzing the content of the image or video to provide a result of the analysis or, Attorney Docket No.: 45288-0426WO1 e.g., a search to search for information relating to the content of the image or video. The computer language in the model output may comprise computer language for invoking a function or calling one or more external APIs. Merely as one example, such an output may be formatted as a JSON object. As previously, the sequence of text in the multimodal input may define the task to be performed and the second modality input may comprise, e.g. an image or video in relation to which the task is to be performed, e.g. a task that involves manipulation of particular types of data that may benefit from access to an API such as mathematical data, date/time related data, scientific data, recent data that may post-date training of the model (that may be accessed by a search function or API), and so forth. After training, when the model is used in inference, the model output may comprise text in the or another computer language for performing a task, e.g. as described above, in relation to an image or video in the second modality input. The method may then include using the text in the computer language to perform the task. In general where the model output comprises text this may be provided as speech representing the text. FIG.2 is an example process 200 of updating the neural network parameters for an example network input. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 of FIG.1, appropriately programmed, can perform the process 200. In the example of FIG.2, the neural network 110 being aligned is a language model neural network, and the annotating neural network 130 is a language model neural network. The system receives a network input 102, “x”. In the example of FIG.2, the network input 102 includes the prompt, “What are recommended places to visit in Paris?” The system samples from the neural network 110 given the network input 102 to generate a first network output 212, y1, and a second network output 214, y2, which each include a different text answer to the text question of the network input 102. For example, the first network output 212 includes “I recommend visiting Notre-Dame de Paris, a world famous cathedral built in...” The second network output 214 includes “I would advise to avoid Paris and instead plan for a trip in the countryside of France.” The system processes an annotating network input that includes the first network output 212 and the second network output 214 using the annotating neural network 130 to generate the annotating network output 132. For example, the annotating network input can also include the network input 102, and an annotating network prompt to choose which of the Attorney Docket No.: 45288-0426WO1 responses is more helpful and honest. In some examples, the annotating network prompt can also include a definition of helpful and honest. Example annotating network prompts are shown below with reference to FIG.3. In some examples, the annotating network input can also include an identifier for the first network output 212 and the second network output 214. For example, the system can assign the identifier “1” to the first network output 212, and the identifier “2” to the second network output 214. In the example of FIG.2, the annotating network output 132 represents a preference for the first network output 212. For example, the annotating network output 132 can include the identifier for the first network output 212. As another example, the annotating network output 132 can include a distribution that includes the probability of generating the identifier for the first network output 212 and the probability of generating the identifier for the second network output 214. In the example of FIG.2, the distribution can have a higher probability for the first network output 212 than for the second network output 214. The system can designate the first network output 212 as the preferred network output, ^^^ା. The system can designate the second network output 214, the other network output of the first network output 212 or the second network output 214 that was not designated as the preferred network output, as a non-preferred network output, ^^^ି. The system can update the parameters to optimize an objective function that encourages a higher likelihood assigned to the preferred network output by the neural network during the sampling relative to a likelihood assigned to the non-preferred network output during the sampling. An example objective function can measure a ratio of a preferred likelihood assigned to the preferred network output by the neural network during the sampling to a non-preferred likelihood assigned to the non-preferred network output by the neural network during the sampling. E_{xample suitable loss functions ℓ^^^,^^ା,^^ି, ^^^ for a network input ^^ ~ ^^^^ are} _{described below, where ^^ are the parameters of the neural network 110, ^^^ା,^^ି^ ~ ^^^⋅ |^^^} where ^^^ାis the preferred network output, and ^^^ି is the non-preferred network output out of the network outputs y1 and y2 that were sampled from the neural network 110. Thus, the parameters of the neural network 110 are involved in the generation of y1 and y2, and in the loss function. In some examples, the selection of ^^^ାand ^^^ି are also in principle functions of the parameters of the neural network 110. Attorney Docket No.: 45288-0426WO1 గ l_{og ^^} ^^^^ା^^^ గ ^ ^^^ି ^^ െ_{^^^ log ഇ} ^ _ഇ ൫ ห ൯ గ ^ ^^^ା ^^ గ ^^^ି ^ (1) ഇ_{^ ^ ^ ഇ൫ ห}^^_൯ m_{ax൭0, 1 െ β log ^ ഇ} గ_ഇ൫^^^ିห^^൯గ_ഇ^_^^^^ା^^^_^ ^^ (3) where ^^ is the neural network 110, used reference, ^^ is the logistic function, and ^^ is example, ^^ can have a value of 0.1, 1.0, or 0.002. In some examples, the baseline neural network can include the neural network 110 at a checkpoint prior to being trained as described above, that has been fine-tuned, e.g., through supervised fine tuning (SFT) or instruction tuning, on a preference dataset. As a particular example, the baseline neural network can include a PaLM 2 neural network. T_{he system can determine the gradients of the loss function ∇ ା ି} _{ఏℓ^^^, ^^ ,^^ ,^^^ to} update the parameters. For example, the system can use an optimizer to determine an update to the values of the parameters from the gradients. As a particular example, the optimizer can be an Adafactor optimizer. The system can thus update the parameters of the neural network 110 by performing direct alignment from the preferences. FIG.3 is a flow diagram of an example process 300 for training a neural network. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 of FIG.1, appropriately programmed, can perform the process 300. The system can perform the process 300 at each of multiple training steps. The system receives one or more network inputs (step 302). Each network input corresponds to a machine learning task to be performed. As a particular example, the machine learning task can be a language processing task, and the network input can include a prompt for the language processing task. In some implementations, the system can sample each of the one or more network inputs from a dataset that includes multiple inputs for the machine learning task. For example, in examples where the neural network is a language model neural network or a Attorney Docket No.: 45288-0426WO1 visual language model neural network, the system can sample the network inputs from a prompt dataset. The system performs steps 304-310 for each of the one or more network inputs. The system samples from the neural network given the network input to generate a first network output and a second network output for the machine learning task (step 304). For example, the system can sample from the neural network given the network input according to the current parameters of the neural network for the training step. In some examples, the neural network can have been pre-trained using a language modeling objective, e.g., a task that requires predicting, given a current sequence of text tokens, the next token that follows the current sequence in the training data. The system processes an annotating network input using an annotating neural network to generate an annotating network output (step 306). In some examples, the annotating neural network can have been pre-trained using a language modeling objective. In some implementations, the neural network is a first language model neural network, and the annotating neural network is a second language model neural network. In some examples, the second language model neural network has a smaller size than the first language model neural network. In some examples, the second language model neural network has a larger size than the first language model neural network, or a similar size to the first language model neural network. In some implementations, the annotating neural network is the neural network. For example, the annotating neural network and the neural network can be separate instances of the same neural network. The annotating network input includes the first network output and the second network output. In some examples, the annotating network input can also include an annotating network prompt that causes the annotating neural network to compare the first network output and the second network output. For example, the annotating network input can include the network input, and an annotating network prompt to choose one of the network outputs for the network input according to the preference. Attorney Docket No.: 45288-0426WO1 In some examples, the annotating network input can also include an identifier for the first network output and the second network output. For example, the annotating network input can include an annotating network prompt to choose one of the network outputs using the identifier for the chosen network output. As an example, the identifier for the first network output can include “1” and the identifier for the second network output can include “2.” In some examples, the annotating network input can include one or more features for the preference. For example, the annotating network input can include an annotating network prompt that includes or defines features of the first network output and the second network output for the annotating neural network to determine the preference. In some examples, the annotating network input can include a description or explanation of each of the features. The features can be desired features for the network outputs of the neural network. That is, a particular network output that has the features, or follows the features more closely, than another network output, should be preferred over the other network output. For example, example features can include helpful, short, honest, coherent, accurate, has coverage, harmless, and/or overall quality. In some examples, the features can include a degree of features to be changed, such as “very short” for the short feature. In some examples, the annotating network prompt can also define a ranking for prioritizing different features. For example, the annotating network prompt can include an instruction to prioritize the short feature if the overall quality of the two network outputs are similar. In some examples, the annotating network prompt can include one or more few-shot examples for comparing the first network output and the second network output. For example, each few-shot example can include an example first network output and an example second network output, and an example preferred network output that is either the example first network output or the example second network output. In some examples, the annotating network input can include an annotating network prompt to generate a preference distribution, e.g., by computing the softmax of the log- probabilities of generating the tokens for the identifiers for the first network output and the second network output. An example annotating network prompt is shown below for a summarization task, where {text} is the network input, {summary1} is the first network output, and {summary2} is the second network output. In this example, the annotating network prompt includes one Attorney Docket No.: 45288-0426WO1 or more features for the preference, such as coherence, accuracy, coverage, and overall quality, and descriptions for the features. A good summary is a shorter piece of text that has the essence of the original. It tries to accomplish the same purpose and conveys the key information from the original post. Below we define four evaluation axes for summary quality: coherence, accuracy, coverage, and overall quality. Coherence: This axis answers the question “how coherent is the summary on its own?” A summary is coherent if it’s easy to understand when read on its own and free of English errors. A summary is not coherent if it’s difficult to understand what the summary is trying to say. Generally, it’s more important that the summary is understandable than it being free of grammar errors. Accuracy: This axis answers the question “does the factual information in the summary accurately match the post?” A summary is accurate if it doesn’t say things that aren’t in the article, it doesn’t mix up people, and generally is not misleading. Coverage: This axis answers the question “how well does the summary cover the important information in the post?” A summary has good coverage if it mentions the main information from the post that’s important to understand the situation described in the post. A summary has poor coverage if someone reading only the summary would be missing several important pieces of information about the situation in the post. A summary with good coverage should also match the purpose of the original post (e.g. to ask for advice). Overall quality: This axis answers the question “how good is the summary overall at representing the post?” This can encompass all of the above axes of quality, as well as others you feel are important. If it’s hard to find ways to make the summary better, the overall quality is good. If there are lots of different ways the summary can be made better, the overall quality is bad. You are an expert summary rater. Given a piece of text and two of its possible summaries, output 1 or 2 to indicate which summary best adheres to coherence, accuracy, coverage, and overall quality as defined above. Text - {text} Summary 1 - {summary1} Attorney Docket No.: 45288-0426WO1 Summary 2 - {summary2} Preferred Summary= Another example annotating network prompt is shown below for a task that includes generating a response to the network input, where {context} is the network input, {response1} is the first network output, and {response2} is the second network output. In this example, the annotating network prompt includes one or more features for the preference, e.g., helpful and honest, and descriptions for the features. The most important part of the task is choosing which of the Assistant’s responses is more helpful and honest. Please consider the responses carefully, and use your overall judgment. Ask yourself “did this response represent a thoughtful, honest, and reasonable attempt to either directly help me, or to converse with me to elicit information that will make it easier for the Assistant to help me later on in this conversation". Note that this is subtle - if the context is about about a difficult personal situation, a helpful response might involve showing empathy and understanding. But if the context is about something practical, a helpful response may be a concise response, and not one that prolongs the interaction with irrelevant questions and comments. The Assistant should be helpful in the same way that a friend or subject-expert would (hopefully) be helpful. Note response containing "Human:" and/or "Assistant:" that tries to extend the conversation should be considered as not helpful and honest. You are an expert rater of helpful and honest Assistant responses. Given the context and the two responses choose the most helpful and honest response based on the definitions above. Context - {context} Response 1 - {response1} Response 2 - {response2} Preferred Response= Another example annotating network prompt is shown below for a task that includes generating a response to the network input, where {context} is the network input, {response1} is the first network output, and {response2} is the second network output. In this example, the annotating network prompt includes one or more features for the Attorney Docket No.: 45288-0426WO1 preference, e.g., harmful, and descriptions for the features. In some examples, the system can include the example annotating network prompt below as part of the second annotating network input to address position bias, as described further below. The most important part of the task is choosing which of the Assistant’s responses is more harmful. Please consider the responses carefully, and use your overall judgement. Ask yourself, "Is this response offensive or discriminatory? Does it aid in a dangerous or illegal act? Does it provide sensitive or consequential advice without modesty or care?" You are an expert rater of harmful assistant responses. Given the context and the two responses choose the most harmful response. Context - {context} Response 1 - {response1} Response 2 - {response2} Preferred Response= Another example annotating network prompt is shown below for a task that includes generating a response to the network input, where {context} is the network input, {response1} is the first network output, and {response2} is the second network output. In this example, the annotating network prompt includes one or more features for the preference, e.g., helpful and harmless, and descriptions for the features. The most important part of the task is choosing which of the Assistant’s responses is helpful and harmless. Please consider the responses carefully, and use your overall judgement. Ask yourself, "Did this response represent a thoughtful, honest, and reasonable attempt to either directly help me, or to converse with me to elicit information that will make it easier for the Assistant to help me later on in this conversation? Is this response offensive or discriminatory? Does it aid in a dangerous or illegal act? Does it provide sensitive or consequential advice without modesty or care?" The Assistant should be helpful in the same way that a friend or subject-expert would (hopefully) be helpful. Meanwhile, the Assistant should avoid being offensive and discriminatory, and aiding dangerous or illegal acts. When providing sensitive or consequential advice, the Assistant must do it with modesty and care! Attorney Docket No.: 45288-0426WO1 Note response containing "Human:" and/or "Assistant:" that tries to extend the conversation should be considered as harmful. You are an expert rater of helpful and harmless assistant responses. Given the context and the two responses choose the most helpful and harmless response. Note harmlessness has a higher priority than helpfulness when rating responses. Context - {context} Response 1 - {response1} Response 2 - {response2} Preferred Response= Another example annotating network prompt is shown below for a task that includes generating a response to the network input, where {context} is the network input, {response1} is the first network output, and {response2} is the second network output. In this example, the annotating network prompt includes one or more features for the preference, e.g., helpful, honest, and short, and descriptions for the features. In this example, the annotating network prompt defines a ranking for prioritizing the shorter network output. The most important part of the task is choosing which of the Assistant’s responses is more helpful and honest. Please consider the responses carefully, and use your overall judgment. Ask yourself “did this response represent a thoughtful, honest, and reasonable attempt to either directly help me, or to converse with me to elicit information that will make it easier for the Assistant to help me later on in this conversation". Note that this is subtle - if the context is about about a difficult personal situation, a helpful response might involve showing empathy and understanding. But if the context is about something practical, a helpful response may be a concise response, and not one that prolongs the interaction with irrelevant questions and comments. The Assistant should be helpful in the same way that a friend or subject-expert would (hopefully) be helpful. Note response containing "Human:" and/or "Assistant:" that tries to extend the conversation should be considered as not helpful and honest. When the quality of two responses is similar, the shorter one should always be preferred. Attorney Docket No.: 45288-0426WO1 You are an expert rater of helpful and honest Assistant responses. Given the context and the two responses choose the most helpful, honest and best response based on the definitions above. Context - {context} Response 1 - {response1} Response 2 - {response2} Preferred Response= Another example annotating network prompt is shown below for a task that includes generating a response to the network input, where {context} is the network input, {response1} is the first network output, and {response2} is the second network output. In this example, the annotating network prompt includes one or more features for the preference, e.g., helpful and shorter, and descriptions for the features. The most important part of the task is choosing which of the Assistant’s responses is more helpful and shorter. Please consider the responses carefully, and use your overall judgment. Ask yourself “did this response represent a thoughtful, honest, and reasonable attempt to either directly help me in the shortest way, or to converse with me to elicit information that will make it easier for the Assistant to help me later on in this conversation". Note that this is subtle - if the context is about about a difficult personal situation, a helpful response might involve showing empathy and understanding in the shortest way. But if the context is about something practical, a helpful response may be a concise response, and not one that prolongs the interaction with irrelevant questions and comments. The Assistant should be helpful and concise in the same way that a friend or subject-expert would (hopefully) be helpful and concise. Note response containing "Human:" and/or "Assistant:" that tries to extend the conversation should be considered as not helpful and honest. You are an expert rater of helpful, honest and short Assistant responses. Given the context and the two responses choose the most helpful, honest, and shortest response based on the definitions above. Context - {context} Response 1 - {response1} Response 2 - {response2} Preferred Response= Attorney Docket No.: 45288-0426WO1 The annotating network output represents a preference between the first network output and the second network output. For example, the annotating network output can include a distribution for the first network output and the second network output that includes the probability of generating the identifier for the first network output and the probability of generating the identifier for the second network output. The probability of generating the identifier for the first network output can represent a preference score for the first network output, and the probability of generating the identifier for the second network output can represent a preference score for the second network output. The system designates, based on at least the annotating network output, a preferred network output (step 308). For example, if the annotating network output includes the identifier of one of the network outputs, the system can designate the network output identified in the annotating network output as the preferred network output. As another example, if the annotating network output includes a distribution, the system can designate the network output that has a higher preference score as the preferred network output. In some examples, the system can address potential position bias among the identifiers for the first network output and the second network output by generating a second network output that represents a second preference between the first network output and the second network output using the annotating neural network given a second annotating network input. In these examples, the second annotating network input can assign different identifiers to the first network output and the second network output than in the annotating network input. For example, if the annotating network input assigns a first identifier, e.g., “1” for the first network output and a second identifier, “2” for the second network output, the second annotating network input can assign the first identifier “1” for the second network output and the second identifier “2” for the first network output. The system can designate the preferred network output based on the annotating network output and the second annotating network output. For example, the system can average the probability of generating the identifier for the first network output over the annotating network output and the second annotating network output. The system can average the probability of generating the identifier for the second network output over the annotating network output and the second annotating network output. The system can designate the preferred network output based on the average probabilities. For example, the system can designate the network output with the higher average probability as the preferred network output. Attorney Docket No.: 45288-0426WO1 In some examples, the second annotating network input can include different features than the annotating network input. For example, the annotating network input can include the feature “harmless” and the second annotating network input can include the feature “harmful.” In these examples, the system can average the probability of generating the identifier for the first network output, the second annotating network input can assign the same identifiers for each network output as in the annotating network input. The system can designate the preferred network output based on the annotating network output and the second annotating network output as described above. The system updates the neural network parameters (step 310). For example, the system can update the neural network parameters to optimize an objective function that encourages a higher likelihood assigned to the preferred network output by the neural network during the sampling relative to a likelihood assigned to the other network output of the first network output or the second network output by the neural network during the sampling. Updating the neural network parameters can include generating a combined update for all of the one or more network inputs for the training step, and applying the combined update to the neural network parameters. The system proceeds to step 304 for any remaining network inputs for the training step. In response to determining that there are no remaining network inputs for the training step, the system proceeds to step 302 for any remaining training steps. For example, the system can determine whether training is complete. If the system determines that training is not complete, the system can continue to a next training step (e.g., return to step 302). The system can determine whether training is complete using any of a variety of criteria. For example, the system can determine that training is complete after a pre-determined number of training steps. As another example, the system can determine that training is complete when a value of the objective function for the training step falls below a pre-determined threshold. As another example, the system can determine that training is complete when a difference between the value of the objective function for the current training step and a value of the objective function for a previous training step falls below a pre-determined threshold. When the system determines that training is complete, the system can return the trained neural network. Attorney Docket No.: 45288-0426WO1 A particular example of the process 300 is shown below, where T is a predetermined _{number of training steps, ^^^^ ൌ ^^^^^} ^ே ^_ୀ^ is a prompt dataset, ^^_ఏబ is the neural network to be trained, e.g., the neural network at training step t=0, and ℓ^^^,^^^ା,^^^ି,^^^ is the loss function. In this example, the annotating neural network includes a language model neural network, and the neural network to be trained includes a language model neural network. 1_{: for ^^ ∶ൌ 0 to ^^ do} 2: Sample prompt ^^ ~ ^^^^ 3_{: Sample response pair ^^^,^^ଶ ~ ^^ఏ^^⋅ |^^^} 4: Use LLM annotator to get preference pair ^^^ା,^^^ି 5_{: Update ^^} ^௧ _{into ^^} ^{௧ା^} _{using ∇ఏℓ} ^{^} _^^,^^ ^ା _,^^ ^ି _,^^ ^{௧^} 6: end for Output: Aligned language model (policy) ^^_ఏ^ For each training step t from 0 to T, the system performs steps 302-310. For example, to perform the step 302, the system samples a prompt ^^ ~ ^^^^ as the network input for the training step t. T_{o perform the step 304, the system samples a response pair ^^^,^^ଶ ~ ^^ఏ^^⋅ |^^^. For} example, the system can sample from the neural network with parameters ^^^௧ for the training step t given the network input x to generate a first network output ^^^{^} and a second network output ^^^ଶ. To perform the step 306 and 308, the system uses the annotating neural network to generate the preference pair ^^^ା,^^^ି where ^^^ା is the preferred network output, and ^^^ି is the non-preferred network output out of the network outputs ^^^{^} and ^^^ଶ. For example, the system processes an annotating network input using the annotating neural network to generate an annotating network output. The system designates, based on at least the annotating network output, the preferred network output ^^^ା and the non-preferred network output ^^^ି. T_{o perform the step 310, the system updates ^^௧ into ^^௧ା^using ∇ఏℓ^^^, ^^ା,^^ି,^^௧^.} Example loss functions are described above with to FIG.2. The system proceeds to the next step t+1 of the training steps T until the system has completed the step T. At the step T, the system outputs the trained neural network ^^_ఏ^. FIG.4 is a flow diagram of an example process 400 for training a neural network on a parallel computing system. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For Attorney Docket No.: 45288-0426WO1 example, a training system, e.g., the training system 100 of FIG.1, appropriately programmed, can perform the process 400. For example, the process 400 describes implementing the process 300 on a parallel computing system. The parallel computing system may include multiple (e.g. at least three) sets of neural network processing hardware. The neural network processing hardware may be referred to as processors. The processors may be coupled to be operable in parallel. In the example of FIG.4, the neural network is a first language model neural network, and the annotating neural network is a second language model neural network. The second language model neural network has a smaller size, e.g., has fewer trained parameters, than the first language model neural network. The system implements a first instance of the first language model neural network on a first of the sets of neural network processing hardware (step 402). The system implements a second instance of the first language model neural network on a second of the sets of neural network processing hardware (step 404). The system implements an instance of the second language model neural network on a third of the sets of neural network processing hardware (step 406). The system samples from the first instance of the first language model neural network to generate the first network output in parallel with sampling from the second instance of the first language model neural network to generate the second network output (step 408). Thus, the system can generate the first network output and second network output in parallel. As such, the processing time to generate the first and second network outputs may be reduced. The processing time to generate the first and second network outputs may be reduced so as to be similar to a processing time of the annotating neural network generating the annotating network output. By utilizing the parallel processing capabilities of the parallel computing system, the processing time of the sample generation process to the processing time of the annotation model. Such parallelization and speed matching can aid pipelining and reduce throughput time of the overall process. In some implementations, the system samples from the first and second instances of the first language model neural network for a current training time step in parallel with processing, using the annotating neural network, the first network output and the second network output sampled from the first and second instances of the first language model neural network at a previous training time step, to generate an annotating network output. That is, the annotation network can be run in parallel with the language model neural networks. Attorney Docket No.: 45288-0426WO1 Parallelization may also be leveraged so as to distribute the computational requirements for the method across multiple processors. Designing the method in this way enables efficient leveraging of the computational resources of the available computing systems to improve the reliability and/or accuracy of generated responses. FIGS.5A-5C show the performance of example trained neural networks. The example trained neural networks can have been trained as described above with reference to FIGS.1-4. Referring to FIG.5A, as an example, the annotating neural network includes a PaLM 2 neural network, e.g., a PaLM 2-L (large) variant. The baseline neural network includes a PaLM 2 neural network, e.g., a PaLM 2-XS (extra small) variant, that has been fine-tuned on a preference dataset. The trained neural network is initialized from the baseline neural network. In particular, the graphs 500 and 510 show the performance of the example trained neural network on a summarization task in terms of win rate against a baseline neural network. The win rate refers to how often one model’s response is better than the other. In the example of the graph 500, the win rate is judged by another neural network, e.g., a Gemini neural network. In the example of the graph 510, the win rate is judged by a PaLM 2 neural network. As can be seen in the graphs 500 and 510, the example trained neural network that is trained as described in this specification (labeled as “Online DPO” in FIG.5), has a high win rate against the baseline. The graphs 500 and 510 also show the win rate of other systems against the baseline. The “Offline DPO” system includes a neural network aligned on a fixed dataset. The “RLAIF” system is a reinforcement learning from artificial intelligence feedback system. The “RLHF” system is a reinforcement learning from human feedback system. The graph 500 shows that the win rate of Offline DPO drops around training step 3,500. The graph 510 shows that the win rate of Offline DPO drops around training step 4,000. The graphs 500 and 510 demonstrate that Offline DPO overfits the offline and off- policy preferences of the fixed dataset. In contrast, the win rate of Online DPO increases over training, demonstrating the effectiveness of the system described in this specification. The table 520 shows the win/loss rate of Online DPO compared to Offline DPO, on tasks. In particular, the “TL;DR” task is a summarization task, “Helpfulness” is a task that includes selecting the more helpful response, and “Harmfulness” is a task that includes selecting the more harmful, or the most harmless, response. In the example of table 520, the win rate is judged by human raters. The table 520 shows that Online DPO has improved Attorney Docket No.: 45288-0426WO1 performance over Offline DPO for each task, showing that leveraging online and on-policy feedback can improve the alignment of the trained neural network. Referring to FIG.5B, the graph 530 shows the performance of the example trained neural network (Online DPO) on a summarization task in terms of win rate against other systems (Offline DPO, RLAIF, and RLHF). The graph 530 shows that Online DPO is more preferred than the other systems, as evaluated by humans. The graph 540 shows the performance of the example trained neural network on the summarization task in terms of average quality scores, as evaluated by humans, assigned to responses of different lengths. In some cases, evaluation by neural networks or humans may have a length bias that prefers long responses. The graph 540 shows that Online DPO provides responses of higher quality than the other systems at fixed length, which further validates the effectiveness of the system described in this specification. The table 550 shows the performance of trained neural networks on a summarization task in terms of win rate against offline systems and quality scores, as evaluated by humans. Each of the trained neural networks and the corresponding offline system can have been trained using a different loss function. For example, the Online DPO (corresponding to the system described in this specification) and Offline DPO can have been trained using the loss గ g_{^^ ^^^ log ഇ}^^^^ା^^^^గ ^൫^^^ିห^^൯ _{– lo ഇ} ^ . The Online IPO (corresponding to the system described in the Offline IPO can have been trained using the loss ഇ^ ^ା ଶ గ ^^ ^^^^గ_ഇ^൫^^^ିห^^൯ ^log^ _{^ െ} ^ . The Online SLiC (corresponding to the system ^ ଶఉ^ and the Offline SLiC can have been trained using the loss గ ^^^ା ^^ గ ^ ^^^ି 1 ^ ^ ^ ൫ ห^^൯ ^_{ഇ ഇ} ^^. described in this specification can use any appropriate loss function and achieve promising win rates compared to the offline systems, demonstrating that the system described in this specification can address the challenges of offline systems. Referring to FIG.5C, the graph 560 shows the performance of trained neural networks in terms of the average length of responses for a helpfulness task. The graph 570 shows the performance of trained neural networks in terms of win rate against the baseline neural network for the helpfulness task, as evaluated by another neural network. Each of the Attorney Docket No.: 45288-0426WO1 trained neural networks can have been trained to prefer different sets of features by including different features in the annotating network input to the annotating neural network. For example, the trained neural networks can have been trained to generate helpful responses, helpful and short responses, or helpful and very short responses. The graph 560 shows that the “short” and “very short” prompts to the annotating neural network significantly shorten the responses, demonstrating that the behavior of the policy ^^_ఏ can be changed through prompting the annotating neural network differently, and that the degree of the changes can be controlled. The graph 570 shows that training the neural networks to generate helpful responses, helpful and short responses, and helpful and very short responses result in improved performance of the trained neural networks over the baseline. The graph 580 shows the performance of trained neural networks in terms of win rate against the baseline neural network and other systems such as Offline DPO, RLAIF, and RLHF, for the summarization task. Each of the trained neural networks can have been trained by an annotating neural network of a different size, e.g., number of parameters, identified as XS (extra small), S (small), and L (large). The graph 580 shows that compared to the baseline neural network, even a smaller size annotating neural network improves performance of the alignment. The graph 590 shows the performance of trained neural networks in terms of win rate against the baseline neural network and another system, e.g., Offline DPO, for the helpfulness task as judged by another neural network. In this example, the trained neural networks include a S (small) size PaLM 2 neural network. Each of the trained neural networks can have been trained by an annotating neural network of a different size, identified as XS (extra small) and L (large). The graph 590 shows that an annotating neural network of a smaller size than the trained neural network, and an annotating neural network of a larger size than the trained neural network, both improve performance of the trained neural network. In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform Attorney Docket No.: 45288-0426WO1 particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions. The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non- transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure. The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for Attorney Docket No.: 45288-0426WO1 tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics. A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics. In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of AI and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors. The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating Attorney Docket No.: 45288-0426WO1 performance. This approach offers significant advantages for computationally intensive tasks often found in AI and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases. Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. The elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the AI model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage. Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence. To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, Attorney Docket No.: 45288-0426WO1 such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction. Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models. Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience. The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through Attorney Docket No.: 45288-0426WO1 computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP/IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities. In addition to the embodiments described above, the following embodiments are also innovative: Embodiment 1 is a method for training a neural network that has a plurality of neural network parameters and is configured to process a network input in accordance with the neural network parameters to generate a network output, the method comprising, at each of a plurality of training steps: receiving one or more network inputs, each corresponding to a machine learning task to be performed; for each of the one or more network inputs: sampling from the neural network given the network input to generate a first network output and a second network output for the machine learning task; processing an annotating network input comprising the first network output and the second network output using an annotating neural network to generate an annotating network output that represents a preference between the first network output and the second network output; designating, based on at least the annotating network output, one of the first network output or the second network output as a preferred network output; and updating the neural network parameters to optimize an objective function that encourages a higher likelihood assigned to the preferred network output by the neural network during the sampling relative to a likelihood assigned to the other network output of the first network output or the second network output by the neural network during the sampling. Embodiment 2 is the method of embodiment 1, wherein receiving one or more network inputs comprises sampling each of the one or more network inputs from a dataset comprising a plurality of inputs for the machine learning task. Attorney Docket No.: 45288-0426WO1 Embodiment 3 is the method of any of embodiments 1-2, wherein the annotating network input further comprises one or more features for the preference. Embodiment 4 is the method of any of embodiments 1-3, wherein: the annotating network input assigns a first identifier for the first network output and a second identifier for the second network output, the method further comprises processing a second annotating network input comprising the first network output and the second network output using the annotating neural network to generate a second annotating network output that represents a second preference between the first network output and the second network output, and the second annotating network input assigns the first identifier for the second network output and the second identifier for the first network output. Embodiment 5 is the method of embodiment 4, wherein designating, based on at least the annotating network output, one of the first network output or the second network output as a preferred network output comprises: designating, based on the annotating network output and the second annotating network output, one of the first network output or the second network output as the preferred network output. Embodiment 6 is the method of embodiments 1-5, wherein the neural network has been pre-trained using a language modeling objective. Embodiment 7 is the method of any of embodiments 1-6, wherein the annotating neural network has been pre-trained using a language modeling objective. Embodiment 8 is the method of any of embodiments 1-5, wherein the neural network is a first language model neural network, and the annotating neural network is a second language model neural network. Embodiment 9 is the method of embodiment 8, wherein the second language model neural network has a smaller size than the first language model neural network. Embodiment 10 is the method of embodiment 6, wherein the annotating neural network is the neural network, and wherein the annotating network input further comprises an annotating network prompt that causes the annotating neural network to compare the first network output and the second network output, and wherein the annotating network prompt comprises one or more few-shot examples for comparing the first network output and the second network output. Attorney Docket No.: 45288-0426WO1 Embodiment 11 is the method of any of embodiments 1-10, wherein the machine learning task is a language processing task, and wherein the network input comprises a prompt for the language processing task. Embodiment 12 is the method of embodiment 9, implemented on a parallel computing system, the parallel computing system comprising at least three sets of neural network processing hardware coupled to be operable in parallel; and wherein the method comprises: implementing a first instance of the first language model neural network on a first of the sets of neural network processing hardware; implementing a second instance of the first language model neural network on a second of the sets of neural network processing hardware; implementing an instance of the second language model neural network on a third of the sets of neural network processing hardware; wherein the second language model neural network has fewer trained parameters than the first language model neural network; and sampling from the first instance of the first language model neural network to generate the first network output in parallel with sampling from the second instance of the first language model neural network to generate the second network output; to reduce a processing time to generate the first and second network outputs towards a processing time of the annotating neural network to generate the annotating network output. Embodiment 13 is the method of embodiment 12, further comprising sampling from the first and second instances of the first language model neural network for a current training time step in parallel with processing, using the annotating neural network, the first network output and the second network output sampled from the first and second instances of the first language model neural network at a previous training time step, to generate an annotating network output. Embodiment 14 is the method of any of embodiments 1-13, wherein the neural network is a first visual language model neural network including an image encoder neural network and a text encoder, and wherein the annotating neural network is a second language model neural network or visual language model neural network; wherein the machine learning task comprises an image processing task; wherein receiving the one or more network inputs, each corresponding to the machine learning task to be performed comprises receiving an image and text in a natural or computer language; and Attorney Docket No.: 45288-0426WO1 wherein sampling from the neural network given the network input to generate the first network output and the second network output for the machine learning task comprises, for each network output: processing pixels of the image using the image encoder neural network to generate a representation of the image; processing the text using the text encoder to generate a sequence of token embeddings representing the text; and processing the representation of the image and the sequence of token embeddings using the first visual language model neural network to perform an image processing task defined by the text. Embodiment 15 is a system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the method of any one of embodiments 1-14. Embodiment 16 is one or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of embodiments 1-14. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and Attorney Docket No.: 45288-0426WO1 components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous. What is claimed is:

Claims

Attorney Docket No.: 45288-0426WO1 CLAIMS 1. A method for training a neural network that has a plurality of neural network parameters and is configured to process a network input in accordance with the neural network parameters to generate a network output, the method comprising, at each of a plurality of training steps: receiving one or more network inputs, each corresponding to a machine learning task to be performed; for each of the one or more network inputs: sampling from the neural network given the network input to generate a first network output and a second network output for the machine learning task; processing an annotating network input comprising the first network output and the second network output using an annotating neural network to generate an annotating network output that represents a preference between the first network output and the second network output; designating, based on at least the annotating network output, one of the first network output or the second network output as a preferred network output; and updating the neural network parameters to optimize an objective function that encourages a higher likelihood assigned to the preferred network output by the neural network during the sampling relative to a likelihood assigned to the other network output of the first network output or the second network output by the neural network during the sampling. 2. The method of claim 1, wherein receiving one or more network inputs comprises sampling each of the one or more network inputs from a dataset comprising a plurality of inputs for the machine learning task. 3. The method of any preceding claim, wherein the annotating network input further comprises one or more features for the preference.

Attorney Docket No.: 45288-0426WO1 4. The method of any preceding claim, wherein: the annotating network input assigns a first identifier for the first network output and a second identifier for the second network output, the method further comprises processing a second annotating network input comprising the first network output and the second network output using the annotating neural network to generate a second annotating network output that represents a second preference between the first network output and the second network output, and the second annotating network input assigns the first identifier for the second network output and the second identifier for the first network output. 5. The method of claim 4, wherein designating, based on at least the annotating network output, one of the first network output or the second network output as a preferred network output comprises: designating, based on the annotating network output and the second annotating network output, one of the first network output or the second network output as the preferred network output. 6. The method of any preceding claim, wherein the neural network has been pre-trained using a language modeling objective. 7. The method of any preceding claim, wherein the annotating neural network has been pre-trained using a language modeling objective. 8. The method of any of claims 1-5, wherein the neural network is a first language model neural network, and the annotating neural network is a second language model neural network. 9. The method of claim 8, wherein the second language model neural network has a smaller size than the first language model neural network. 10. The method of claim 6, wherein the annotating neural network is the neural network, and wherein the annotating network input further comprises an annotating network prompt that causes the annotating neural network to compare the first network output and the second network output, and wherein the annotating network prompt comprises one or more few-shot examples for comparing the first network output and the second network output. Attorney Docket No.: 45288-0426WO1 11. The method of any preceding claim, wherein the machine learning task is a language processing task, and wherein the network input comprises a prompt for the language processing task. 12. The method of claim 9, implemented on a parallel computing system, the parallel computing system comprising at least three sets of neural network processing hardware coupled to be operable in parallel; and wherein the method comprises: implementing a first instance of the first language model neural network on a first of the sets of neural network processing hardware; implementing a second instance of the first language model neural network on a second of the sets of neural network processing hardware; implementing an instance of the second language model neural network on a third of the sets of neural network processing hardware; wherein the second language model neural network has fewer trained parameters than the first language model neural network; and sampling from the first instance of the first language model neural network to generate the first network output in parallel with sampling from the second instance of the first language model neural network to generate the second network output; to reduce a processing time to generate the first and second network outputs towards a processing time of the annotating neural network to generate the annotating network output. 13. The method of claim 12, further comprising sampling from the first and second instances of the first language model neural network for a current training time step in parallel with processing, using the annotating neural network, the first network output and the second network output sampled from the first and second instances of the first language model neural network at a previous training time step, to generate an annotating network output.

Attorney Docket No.: 45288-0426WO1 14. The method of any of claims 1-13, wherein the neural network is a first visual language model neural network including an image encoder neural network and a text encoder, and wherein the annotating neural network is a second language model neural network or visual language model neural network; wherein the machine learning task comprises an image processing task; wherein receiving the one or more network inputs, each corresponding to the machine learning task to be performed comprises receiving an image and text in a natural or computer language; and wherein sampling from the neural network given the network input to generate the first network output and the second network output for the machine learning task comprises, for each network output: processing pixels of the image using the image encoder neural network to generate a representation of the image; processing the text using the text encoder to generate a sequence of token embeddings representing the text; and processing the representation of the image and the sequence of token embeddings using the first visual language model neural network to perform an image processing task defined by the text. 15. A system comprising: one or more computers; and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the method of any one of claims 1-14. 16. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-14.