EP4627480A1 - Controlling agents using language-based success detectors - Google Patents

Controlling agents using language-based success detectors

Info

Publication number
EP4627480A1
EP4627480A1 EP24700225.6A EP24700225A EP4627480A1 EP 4627480 A1 EP4627480 A1 EP 4627480A1 EP 24700225 A EP24700225 A EP 24700225A EP 4627480 A1 EP4627480 A1 EP 4627480A1
Authority
EP
European Patent Office
Prior art keywords
task
neural network
language model
observation
environment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP24700225.6A
Other languages
German (de)
French (fr)
Inventor
Yu Qing DU
Ksenia KONYUSHKOVA
Serkan CABI
Joao Ferdinando GOMES DE FREITAS
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
GDM Holding LLC
Original Assignee
GDM Holding LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by GDM Holding LLC filed Critical GDM Holding LLC
Publication of EP4627480A1 publication Critical patent/EP4627480A1/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • This specification relates to controlling agents using neural network based multimodal language models.
  • Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input.
  • Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.
  • Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input.
  • Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer.
  • Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.
  • This specification describes systems and methods, implemented as computer programs on one or more computers in one or more locations, that can be used for controlling an agent to act in an environment to perform a task, and for training or evaluating an action selection system used by such an agent, and for obtaining training data for training such an action selection system.
  • the action selection system may, but need not be, based on a neural network.
  • the task can be one of a plurality of different tasks that the agent can be trained, or instructed, to perform.
  • an agent configured to select actions to perform one or more tasks in an environment, the agent comprising a system as described herein.
  • the agent includes one or more observation capture subsystems to capture observations of the environment, e.g. still or moving images of a real-world environment.
  • a digital assistant device comprising a system as described herein.
  • the digital assistant device can include a user interface to enable a user to request assistance and to output information.
  • Some implementations of the described systems provide a robust and generalizable approach to determining whether or not a task has been achieved. For example, whilst it is possible to train a success detector for a particular task such a model often has difficulties when the environmental conditions change from those observed in the training data, or if the task changes. This can arise, for example, from natural visual variations in the environment e.g. due to lighting changes, or from changes camera position, or from differences between particular objects, or from distractions in the environment.
  • the described techniques are robust to such changes, and also to variations in language used to describe a task (e.g. “lift a rubber duck” or “lift a toy duck object”), and can handle ambiguities, e.g.
  • Implementations of the described systems can be used to detect task success based on object state or agent behavior, and across a wide range of tasks and environmental conditions.
  • the robustness and generalizability of the described techniques facilitates the training, evaluation, and use, of agent action selection systems that have generalized action selection policies.
  • FIG. 8 is an example multi-modal language model neural network.
  • the agent 600 includes one or more observation capture subsystems 610 to capture observations of the environment, in particular for the action selection system 110 and for the language model neural network 120.
  • the environment 104 may be a real-world environment and the observation capture subsystem(s) 610 can, e.g., capture a still or moving image of the real -world environment 104.
  • the agent can use the language model interface 630 to interface with the language model neural network 120.
  • the language model neural network 120 can process an input token string, in particular the previously described input token string 122 representing a question that asks whether the task has been achieved, and a second captured observation of the environment, to thereby obtain the language model output 124 comprising the answer to the question.
  • the second captured observation of the environment may be, but need not be, the same as the first captured observation of the environment.
  • the input token string 122 representing the question may be generated by the agent 600, e.g. using the language model interface 630, or it may be generated by the language model neural network 120. That is, in some implementations the language model neural network 120 may receive a text input from the agent 600 and generate a text output for the agent 600.
  • the agent 102 comprises a human user of a digital assistant device such as a smart speaker, smart display, or other device.
  • the information defining the task can be obtained from the digital assistant device, and the digital assistant device can be used to instruct the user based on the answer.
  • this may comprise receiving, at the digital assistant device, a request from the user for assistance and determining, in response to the request, a series of tasks for the user to perform, e.g. steps or sub-tasks of an overall task.
  • the digital assistant device can be used to output to the user an indication of the task, e.g.
  • step or sub-task to be performed. This may be done using natural language, e.g. on a display and/or using a speech synthesis subsystem of the digital assistant device. Visual, e.g. video, and/or audio observations of the user performing the task may be captured, e.g. using the digital assistant device. A system as described above may then be used to determine whether the user has successfully achieved the task or a task step, or sub-task, from the answer. If there are further tasks to be completed the digital assistant device may then, in response, progress to the next task (if any) of the series of tasks, e.g. by outputting an indication of the next task to be performed. In this way the user may be led step-by-step through a series of tasks to perform an overall task.
  • FIG. 7 shows an example of a digital assistant device 700, including a system as described above that may be implemented as computer programs on one or more computers in one or more locations.
  • the digital assistant device 700 can include a user interface 710 to enable a user to request assistance and to output information.
  • this is a natural language user interface and may comprise a keyboard, voice input-output subsystem, and/or a display.
  • the user interface 710 can receive a request for assistance with a task, e.g. an overall task as described above, and can output instructions to the user to perform actions 114 to complete each of a series of sub-tasks of the overall task.
  • the digital assistant device 700 can include an observation capture subsystem 750, and a language model interface 740 for the language model neural network 120, which may also be implemented locally or remotely.
  • the observation capture subsystem 750 may be similar to the observation capture subsystem 610 described above, and can capture visual and/or audio observations of the human user performing a task, and provide corresponding observation data to the language model interface 740.
  • the input token string may additionally include text tokens representing words and/or characters that are intended to guide the language model output.
  • the natural or computer language question may include prompt language or tags, e.g. to prepend “Q:” or “Question:” or to append “?”, “A:” or “Answer:_”.
  • the prompt language or tags may also be converted into text tokens of the input token string.
  • successful performance of the task may be defined by a goal observation, such as a goal image or goal sound, or by a goal trajectory defining a target movement.
  • a goal image may be used show the environment, e.g. an object in the environment, in a particular state, or it may show a desired trajectory; or a goal video may illustrate a goal behavior or trajectory.
  • a goal sound may indicate when the environment, e.g. an object in the environment, has achieved a particular state, for example the click sound of a male Ethernet connector when it is fully inserted into its socket.
  • obtaining the information defining the task can comprise obtaining goal observation data defining a goal state or states of the environment for successful performance of the task.
  • the goal observation may be referenced in the input token string, e.g. by including a tag that indicates to the language model neural network that the goal observation is part of the natural or computer language question.
  • a tag may be included, for example, at the start of the natural or computer language question.
  • an “ ⁇ image>” tag may be used to reference a goal image, followed by a question such as “Did the agent successfully achieve this goal state?”.
  • a token for the tag may be included in the token vocabulary.
  • the input token string including the tag referencing the goal observation may be provided as a prompt to the multimodal language model neural network before the language model neural network processes the observation data from the environment. That is the multimodal language model neural network may process the input token string, and the goal observation data from the environment, and then the observation data from the environment, prompting the multimodal language model neural network 120 in a way that is broadly analogous to prompting a large language model.
  • the language model neural network is configured to jointly model the input token string 122 and the observation data to determine the language model output 124.
  • the multi-modal language model neural network 120 may comprise a pre-trained visual language model (VLM), of which there are many examples.
  • VLM visual language model
  • FIG. 8 shows details of one way in which the multi-modal language model neural network 120 can be implemented.
  • the compressed representation can comprise a set of observation tokens 812; this may comprise a fixed number of tokens irrespective of the observation data, e.g. a fixed number of tokens per image or video clip.
  • the observation tokens 812 can be determined by processing the observation data using the observation encoder 814 to generate encoded observation data comprising a set of observation features, e.g. visual features. For example an image or each image of a video clip may be encoded in this way.
  • the set of observation tokens 812 can then be determined from the encoded observation data by processing a set of (learned) latent vectors 818 using one or more observation encoder cross-attention layers configured to cross-attend to the observation features, to generate the set of observation tokens, e.g.
  • the set of latent vectors comprises a set of predetermined latent vectors.
  • This approach can be used for example to generate a (fixed size) set of visual tokens from an image or video clip.
  • the language model neural network 800 can include a token embedding subsystem that applies an embedding function to each token of the input token string to transform it to a respective token embedding, i.e. a vector of numeric values, prior to processing the input token string using the stack of processing layers.
  • a token embedding subsystem that applies an embedding function to each token of the input token string to transform it to a respective token embedding, i.e. a vector of numeric values, prior to processing the input token string using the stack of processing layers.
  • Such an embedding function may be fixed or learned.
  • a positional encoding can be applied, e.g. added or concatenated, to the token embeddings to indicate a position of each respective token in the input token string 122.
  • a pre-trained model e.g. a VLM
  • a VLM may be used for the language model neural network 120 of FIG. 1, because such pre-trained models can be capable of performing a question answering task, e.g. a visual question answering task, of the type used herein for task success detection. Nonetheless in some implementations such a pretrained model can be “fine-tuned”, i.e. further trained, using additional training data items. These additional training data items may be specific to task success detection. Such an additional training data item can include an observation of a task, a natural or computer language question asking whether the task was achieved, and an answer appropriate to the observation.
  • the multimodal language model neural network 120 can be prompted with one or more examples of task success detection, e.g. comprising an image, question text, and example answer text.
  • the (additional) training data items can be generated by manually annotating observations of tasks to indicate whether the task was competed. For observations that extend in time, such as video or audio clips, the manual annotation can also indicate, if the task was competed, when success occurred. For example, for a video or audio clip observation this can be done by splitting the clip into non-overlapping subsequences, before and after the point of success. Two (additional) training data items can then be generated, each including a question as to whether the task was successfully completed, and each including an appropriate ground truth answer, e.g. “yes” if the subsequence ends in one or more success frames, and “no” otherwise. With multiple human raters success or failure can be determined by a majority vote; and the point of success can be determined as the median of the first annotated success frame across the raters.
  • These (additional) training data items can include observations of environments or tasks that are similar to, or the same as, those for which the system will later be used, although this is not essential.
  • a task description “ ⁇ task ⁇ ”
  • the question can be generated using a template such as “Did the robot/agent/person successfully ⁇ task ⁇ ”.
  • a trained VLM such as Flamingo (arXiv:2204.14198) can be used to convert the narration into a question, e.g. to convert a narration such as “The person is scooping the ice cream” into “Did the person successfully scoop the ice cream?”.
  • a training dataset of (additional) training data items was generated by using human operators to provide 101,789 demonstrations of 6 tasks using a 6DoF control device, a Panda robot arm (Franka Emika GmbH).
  • Each episode was then annotated by humans with rewards for each task, each episode having 6 reward annotations, one for each task, labelling with a positive reward all frames with a success state, i.e. if the task is solved, and labelling with zero reward otherwise (if a task is accidentally undone within an episode, at that point the reward annotation reverted to zero).
  • the reward annotations and corresponding episode frames were then converted into training data items.
  • FIG. 9 taken from this training dataset, shows an example of generating two (additional) training data items 910, 912 from comprising an annotated behavior trajectory video 900 for a task, with a point of success 902.
  • Each training data item 910, 912 includes a natural language question, a part of the video indicating respectively failure or success of the task, and a corresponding answer to natural language question, respectively “no” or “yes”. Where the answer is “yes” the part of the video in the training data item contains the success frame(s) or follows the success frame(s).
  • FIG. 10 is a flow diagram of an example process for training a multimodal language model neural network, such as the language model neural network 800 of FIG. 8.
  • the process of FIG. 10 can be performed by a system of one or more computers located in one or more locations, e.g. by the computer system 100 of FIG. 1.
  • model e.g. VLM
  • VLM multimodal language model neural network
  • the model is trained, e.g. fine-tuned, for task success detection this can be done in a way that is appropriate to training the model generally.
  • the multimodal language model neural network may comprise a model, such as a transformer-based model, that is configured to process a sequence of tokens comprising tokens from the input token string and observation tokens.
  • the observation tokens may comprise image tokens, e.g. from encoding an image or patches of an image, or multimodal tokens.
  • a model can process, and be trained on, a mixed sequence of such tokens.
  • a model can be natively trained on a combination, e.g. sequence, of text tokens and observation tokens (such as image tokens) provided as an input to the model.
  • such training can involve backpropagating gradients of a loss function, e.g. as described below.
  • such a model may also include a crossattention mechanism as described below.
  • training the multimodal language model neural network 800 starts from a pre-trained and frozen text-only language model. However it is not essential to start from a pre-trained and frozen text-only language model.
  • the above described token processing layers 830 may be obtained from the pre-trained and frozen text-only language model, e.g. from an autoregressive model incorporating such layers.
  • the language model can be a so-called large language model, e.g. having greater than IO 10 , 10 11 or 10 12 trainable (trained) parameters. There are many such models; merely as one example the “Chinchilla” model may be used (Hoffmann et al., “Training Compute-Optimal Large Language Models”, arXiv:2203.15556).
  • Such a text-only language model can be trained on the very large quantities of unlabeled text available e.g. in books and on the Internet.
  • the multimodal language model neural network 800 can then be trained whilst keeping the neural network parameter values, e.g. weights, of the token processing layers 830 frozen, i.e. constant, during the training.
  • this example training process may comprise obtaining a trained generative natural or computer language neural network (step 1000).
  • the generative natural or computer language neural network is configured to process a string of input tokens representing words in a natural or computer language, to generate output tokens for the natural or computer language, and comprises a stack of trained token processing layers.
  • the process can then form a multimodal language model neural network (step 1002).
  • the multimodal language model neural network is configured to process i) a sequence of text tokens representing words in a natural or computer language, and ii) observation data comprising an observation of the environment, to generate a language model output comprising a string of one or more output tokens, e.g. one at a time.
  • the process can further comprise obtaining a set of training data items (step 1004).
  • Each training data item can comprise an observation of an environment, and natural or computer language text relating to the observation.
  • the process can train the multimodal language model neural network, using the training data items, by adjusting parameters of the cross-attention layers. In some implementations, but not necessarily, this is done whilst keeping parameters of the trained token processing layers frozen, i.e. constant (step 1006).
  • the multimodal language model neural network can be trained to predict the text tokens representing the natural or computer language text in the training data item, whilst conditioned on observation data derived from the observation in the training data item.
  • Training the multimodal language model neural network can involve backpropagating gradients (with respect to the parameters of the cross-attention layers) of a loss function, through the trained token processing layers 830.
  • the loss function can be one that encourages the multimodal language model neural network to generate a language model output 124, in particular an output token string representing the natural or computer language text, that matches the ground truth language corresponding to an observation, in a training data item. This can involve processing, using the multimodal language model neural network, observation data derived from the observation in the training data item and an input token string representing the natural or computer language text for the observation.
  • the output token string may be generated one token at a time.
  • the loss function may be based on p(y x) where x denotes the observation data (in particular, the above described compressed representation), L is the number of tokens in the input token string, y t denotes the Z-th token of the input token string, y ⁇ t denotes the I — 1 preceding characters of the input token string, and where p (y t
  • the loss function may comprise a negative log-likelihood term — y ⁇ i> x ⁇ i), e.g. averaged over a minibatch of training data items.
  • training the multimodal language model neural network also involves training, i.e. adjusting parameters of, the above described observation encoder neural network 814.
  • a pre-trained and frozen observation encoder neural network may be used, e.g. one trained based on contrastive learning that does not require labelled data.
  • a gating parameter is used for the cross-attention layers, this may be gradually increased during training to gradually increase the influence of the cross-attention layers.
  • the multimodal language model neural network may be trained partially or wholly in simulation, i.e. using simulated training data representing a simulation of a real-world environment, before the multimodal language model neural network is used to process observations from the real-world environment that was simulated.
  • the simulation need only be approximate as one advantage of the described techniques is that they are able to generalize to novel tasks, environments, behaviors, and language.
  • FIGS. 12A-C illustrate evaluations of success detection performance when varying camera viewpoint, and when including one or more distractor objects.
  • FIG 12A illustrates a video clip from a front camera, as used in the training data in this particular implementation (a baseline);
  • FIG. 12B illustrates a video clip from a rear camera (absent from the training data);
  • FIG. 12C illustrates a video clip from a test including distractor objects (pegs).
  • the baseline success detection performance for an “insert” task of the type illustrated was in the range 90-95%, falling by around 5% with the change to a rear camera viewpoint, and exhibiting little change when distractor objects were present.
  • the robustness of the described techniques to variations of this type in the task, agent and environment e.g. visual changes such as camera view angles, lighting conditions, and background changes) facilitates effective training, evaluation, and use of the action selection system 110 as previously described. Implementations of the system can thus provide an action selection system 110 that has an improved capability of acting in a real -world environment
  • implementations of the described techniques are also able to detect successful performance of a task by a human, and hence can be used to guide a human performing a task.
  • a system can be trained, e.g., using the publicly available Ego4D dataset of egocentric human-in-the-wild videos.
  • the videos show people executing common tasks (e.g., washing dishes, cleaning cars, gardening), and the Ego4D Forecasting + Hands & Objects (FHO) dataset has corresponding narrations that describe the actions of the camera wearer in the videos.
  • Other annotations include so-called critical state changes, i.e.
  • FIG. 13 illustrates generating (additional) training data items for human task performance success detection from the Ego4D dataset, for a task of rolling dough.
  • the agent 102 is a mechanical agent
  • the environment 104 is a real -world environment
  • the observations, in particular the observations processed by the action selection system 110 and the observation from which the observation data is derived for the language model neural network 120 are from one or more sensors sensing the real -world environment.
  • the actions 114 are for controlling the mechanical agent acting in the real-world environment to perform the task, and can be used to control the agent 102 in the real -world environment.
  • the agent 102 is a simulation of a mechanical agent
  • the environment 104 is a simulation of a particular real -world environment
  • the observations relate to the particular real-world environment
  • the actions relate to actions to be performed by the mechanical agent acting in the particular real -world environment to perform the task.
  • the process can then use the mechanical agent in the particular real-world environment to perform the task. That is, training or evaluating of the action selection system may be partly or wholly performed in simulation before using the action selection system in the real world.
  • the agent 102 can be a mechanical agent interacting with the real -world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task.
  • the agent may be a robot or vehicle interacting with the environment to accomplish a specific task, e.g., to locate an object of interest in the environment, or to manipulate or change the state of a specified object, or to move a specified object to a specified location in the environment, or to navigate to a specified destination in the environment.
  • the task may be characterized by an ongoing behavior, e.g. “vacuum the house”.
  • the information defining the task e.g. the object, state, location, destination, or behavior, may be specified by a natural or computer language or other instruction, e.g. from a user interface of the system, as previously described.
  • the observations processed by the action selection system 110, and/or by the language model neural network 120 can be derived from one or more sensors of the environment 104, for example from a camera or other image sensor or from a microphone.
  • the sensor(s) may be mounted on the agent and/or located separately from the agent in the environment.
  • the observations may comprise various types of sensor data as well as or instead of that previously described, e.g. object position data, data from a distance or position sensor, data from an actuator, or sensed electronic signals such as motor current or a temperature signal.
  • the observations may also include data characterizing the current state of the mechanical agent or robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot or of one or more parts of the agent.
  • the observation at any given time step may include (observation) data from a previous time step that may be beneficial in characterizing the environment.
  • the actions 114 may comprise control signals to control the robot or other mechanical agent; a control system can be used to generate control signals for controlling the mechanical agent.
  • the control signal may comprise, e.g., torques for the joints of the robot or torques to a control surface or other control elements e.g. steering control elements of a vehicle, or higher-level control commands. More generally the control signals can include, for example, position, velocity, or force, torque, or acceleration data for one or more joints or other parts of a robot or other mechanical agent.
  • the control signals may also or instead include electronic control data such as motor control data, or signals to control navigation, e.g., steering, movement, braking, and/or acceleration of the agent.
  • This specification uses the term “configured” in connection with systems and computer program components.
  • a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions.
  • one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.
  • Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
  • Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus.
  • the computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
  • the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
  • data processing apparatus refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
  • the apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • the apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • a computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • a program may, but need not, correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code.
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
  • the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations.
  • the index database can include multiple collections of data, each of which may be organized and accessed differently.
  • engine is used broadly to refer to a softwarebased system, subsystem, or process that is programmed to perform one or more specific functions.
  • an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.
  • the processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output.
  • the processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
  • Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit.
  • a central processing unit will receive instructions and data from a read only memory or a random access memory or both.
  • the elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.
  • the central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • a computer need not have such devices.
  • a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
  • PDA personal digital assistant
  • GPS Global Positioning System
  • USB universal serial bus
  • Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
  • semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
  • magnetic disks e.g., internal hard disks or removable disks
  • magneto optical disks e.g., CD ROM and DVD-ROM disks.
  • embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • keyboard and a pointing device e.g., a mouse or a trackball
  • Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
  • Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.
  • Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework.
  • Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components.
  • the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
  • LAN local area network
  • WAN wide area network
  • the computing system can include clients and servers.
  • a client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
  • a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client.
  • Data generated at the user device e.g., a result of the user interaction, can be received at the server from the device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

Systems and methods that can be used for controlling an agent to act in an environment to perform a task, for training or evaluating an action selection system used by such an agent, and for obtaining training data for training such an action selection system. The task can be one of a plurality of different tasks that the agent can be trained, or instructed, to perform. The systems and methods use a multi-modal language model that jointly processes language and data of another modality, such as visual data or sound data. The multi-modal language model is used as a "success detector", to determine whether or not a task performed by the agent has been achieved.

Description

CONTROLLING AGENTS USING LANGUAGE-BASED SUCCESS DETECTORS
BACKGROUND
[0001] This specification relates to controlling agents using neural network based multimodal language models.
[0002] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.
[0003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.
SUMMARY
[0004] This specification describes systems and methods, implemented as computer programs on one or more computers in one or more locations, that can be used for controlling an agent to act in an environment to perform a task, and for training or evaluating an action selection system used by such an agent, and for obtaining training data for training such an action selection system. The action selection system may, but need not be, based on a neural network. The task can be one of a plurality of different tasks that the agent can be trained, or instructed, to perform.
[0005] The systems and methods use a multi-modal language model, that is a model that jointly processes natural or computer language and data of another modality, such as visual data or sound data. The multi-modal language model is used as a “success detector”, i.e. it is used to determine whether or not a task performed by the agent has been achieved, i.e. accomplished successfully.
[0006] In one aspect there is described a method, and a corresponding system, implemented by one or more computers, for using a language model neural network to train or evaluate an action selection system for controlling an agent acting in an environment to perform a task.
[0007] In another aspect there is described a method, and a corresponding system, implemented by one or more computers for using a language model neural network to select training data for training an action selection system for controlling an agent acting in an environment to perform a particular task.
[0008] In another aspect there is described a method, and a corresponding system, implemented by one or more computers, for instructing, e.g. controlling, an agent acting in an environment to perform a task using a language model neural network.
[0009] In another aspect there is described an agent configured to select actions to perform one or more tasks in an environment, the agent comprising a system as described herein. The agent includes one or more observation capture subsystems to capture observations of the environment, e.g. still or moving images of a real-world environment.
[0010] In a further aspect there is provided a digital assistant device comprising a system as described herein. The digital assistant device can include a user interface to enable a user to request assistance and to output information.
[0011] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.
[0012] Some implementations of the described systems provide a robust and generalizable approach to determining whether or not a task has been achieved. For example, whilst it is possible to train a success detector for a particular task such a model often has difficulties when the environmental conditions change from those observed in the training data, or if the task changes. This can arise, for example, from natural visual variations in the environment e.g. due to lighting changes, or from changes camera position, or from differences between particular objects, or from distractions in the environment. The described techniques are robust to such changes, and also to variations in language used to describe a task (e.g. “lift a rubber duck” or “lift a toy duck object”), and can handle ambiguities, e.g. where success is not precisely defined (e.g. “move around quickly”). Implementations of the described systems can be used to detect task success based on object state or agent behavior, and across a wide range of tasks and environmental conditions. The robustness and generalizability of the described techniques facilitates the training, evaluation, and use, of agent action selection systems that have generalized action selection policies.
[0013] In implementations of the described system, using a pre-trained (multi-modal) language model neural network facilitates generalization by the system, e.g. facilitating recognizing the same underlying tasks specified with different language, or under environmental changes such as naturalistic visual changes. Implementations of the system can therefore facilitate the operation of an agent when performing a task in a real-world environment, potentially in an unstructured, open, or evolving setting, e.g. with previously unseen objects or under new conditions.
[0014] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] FIG. l is a first example computer system, using a language model neural network to train or evaluate an action selection system for controlling an agent to perform a task.
[0016] FIG. 2 is a flow diagram of a first example process, for using a language model neural network to train or evaluate an action selection system for controlling an agent to perform a task.
[0017] FIG. 3 is a flow diagram of a second example process, for using a language model neural network to train or evaluate an action selection system for controlling an agent to perform a task.
[0018] FIGS. 4A and 4B are second and third example computer systems, for instructing an agent to perform a task.
[0019] FIG. 5 is a flow diagram of a third example process, for instructing an agent to perform a task.
[0020] FIG. 6 is an example of an agent configured to select actions to perform one or more tasks in an environment.
[0021] FIG. 7 is an example of a digital assistant device for instructing a human to perform a task.
[0022] FIG. 8 is an example multi-modal language model neural network.
[0023] FIG. 9 illustrates generating example training data items.
[0024] FIG. 10 is a flow diagram of an example process for training a multimodal language model neural network.
[0025] FIG. 11 illustrates use of a multimodal language model neural network for task success detection.
[0026] FIGS. 12A-C illustrate evaluation of success detection performance.
[0027] FIG. 13 illustrates generating training data items for human task performance success detection.
[0028] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION
[0029] FIG. 1 shows a first example computer system 100, that uses a language model neural network 120 to train or evaluate an action selection system 110 for controlling an agent 102 acting in an environment 104 to perform a task. The computer system 100 is an example of a system, implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below are implemented.
[0030] The action selection system 110 is configured to process an observation 106 characterizing a current state of the environment 104, to generate a policy output 112 for selecting an action 114 to be implemented by the agent 102 to perform the task. The observation 106, which may be termed an action-selection observation, may be processed at each time step of a succession of time steps to generate the policy output 112 at each time step. The observation 106 may comprise an observation of a physical, real -world environment of the agent, such as an image of the environment captured by a camera. The action may comprise an action performed in the physical, real-world environment by the agent. The agent in the physical, real-world environment may be, but need not be, a mechanical agent such as a robot.
[0031] There are many ways in which the policy output 112 can be used to select actions. For example the policy output may define a probability distribution, e.g. a Gaussian distribution, over a set of actions that can be performed by the agent. An action may then be selected using the probability distribution, e.g. by sampling from the probability distribution to select the action stochastically, or by selecting an action with the highest probability. The policy output 112 may parameterize such a probability distribution, or it may define the probability distribution as a set of scores according to which an action can be selected, e.g. a score for each action of a set of possible actions. As another example the policy output can (directly) define an action to be implemented by the agent to perform the task, e.g. by identifying a position, speed or torque for a mechanical action. In general an action may be continuous or discrete; optionally a continuous action may be discretized. An action may comprise multiple individual or primitive actions to be performed at a time step e.g. a mixture of continuous and discrete actions. In some implementations the policy output 112 may comprise multiple outputs, e.g. from multiple heads on a neural network, for selecting multiple actions at a particular time step. [0032] In general any type of action selection system may be used for the action selection system 110. In some implementations, but not essentially, the action selection system 110 comprises an action selection neural network. The action selection system 110 may use the action selection neural network to process the observation 106, in accordance with learnable parameters, e.g. weights, of the action selection neural network, to generate the policy output 112. Such an action selection neural network can have any appropriate architecture, e.g. an attention-based neural network architecture such as a transformer architecture, a convolutional architecture, a fully-connected architecture, or any other appropriate neural network architecture. The action selection system 110 may be pre-trained, or it may be trained using the language model neural network 120, e.g. as described later.
[0033] Optionally, e.g. in a multi-task implementation, the action selection system 110 can also be configured to process a task input 132 that provides information defining a task to be performed by the action selection system 110, e.g. by describing the task or a goal of the task. The task input 132 can be provided as a sequence of text, e.g. in a natural or computer language. The sequence of text can be encoded using a tokenizer, such as SentencePiece, to represent the text as a series of text tokens from a vocabulary of text tokens, e.g. that each represent words, wordpieces or characters in the natural or computer language. The action selection system 110, e.g. the action selection neural network, can process the observation conditioned on an encoded representation of the sequence of text to generate the policy output 112. For example the observation can also be encoded into a sequence of observation tokens, e.g. using an observation encoder, and the text tokens and observations tokens can be processed by the action selection system 110, e.g. implemented as a sequence processing action selection neural network, to generate the policy output 112. Some examples of action selection systems of this type are described in: PaLM-E (Driess et al. arXiv:2303.03378); RT- 1 (Brohan et al. arXiv:2212.06817); and RT-2 (Brohan et al. arXiv:2307.15818).
[0034] The language model neural network 120 comprises a multimodal model and is configured to process i) a sequence of text tokens representing text, e.g. words in a natural or computer language, and ii) observation data comprising an observation of the environment, which may be termed a language model observation, to generate a language model output 124. The language model observation may, but need not be, one of the action-selection observations processed by the action selection system 110. As described later, in implementations the language model neural network 120 is used to process an input token string 122 that represents a question that asks whether or not the task, or a stage of the task, has been achieved. [0035] The multi-modal language model neural network 120 is multi-modal because it processes multiple different types, or modality, of input data, i.e. text and observation data. In some implementations the multi-modal language model neural network 120 can have a multimodal input for the input token string 122 and a multimodal input for the observation data. In some implementations the multi-modal language model neural network 120 processes these internally by representing the observation data as tokens that are processed together with tokens of the input token string 122. Also or instead the multi-modal language model neural network 120 may use cross-attention to combine the observation data and data from the input token string 122. In some implementations the observation data comprises visual data representing a still or moving image observation. Also or instead the observation data may comprise audio data representing values of an audio waveform, e.g. instantaneous amplitude data or time-frequency domain data, or data representing other types of observations, e.g. proprioceptive observations of a robot agent 102.
[0036] In some implementations the (multi-modal) language model neural network 120 comprises a pre-trained language model neural network. That is, in implementations the language model neural network 120 is not trained whilst controlling the agent or whilst using the language model neural network 120 to train or evaluate the action selection system 110; instead it is trained beforehand. In some implementations language model neural network 120 may be trained, e.g. fine-tuned, as described later.
[0037] The multi-modal language model neural network 120 can have any appropriate architecture, e.g. an attention-based neural network architecture (such as a transformer architecture), a convolutional architecture, a fully-connected architecture, or any other appropriate neural network architecture.
[0038] In some implementations the multi-modal language generation neural network 120 comprises a so-called sequence-to-sequence model, that receives an input string of natural or computer language text tokens, and the observation data (which may also be encoded as tokens, i.e. observation tokens), and generates an output string comprising one or more natural or computer language text tokens. The output string can be generated autoregressively, a token at a time.
[0039] One particular example implementation of the multi-modal language generation neural network 120 is described later, but there are many different types of models that can be employed. For example, where the observation data comprises visual data the multi-modal language model neural network 120 may comprise a visual language model (VLM) such as Flamingo (Alayrac et al. arXiv:2204.14198), PaLI (Chen et al. arXiv: 2209.06794), or PaLI- X (Chen et al. arXiv:2305.18565). Some implementations of the multi-modal language model neural network 120 can be described as “large” multimodal models, e.g. they can have more than IO10 learnable (trained) parameters, but applications of the described techniques are not restricted to such models.
[0040] In general, as used herein, a language may be a natural language or a computer language. For example the multi-modal language model neural network 120 may be configured, e.g. trained, to process formal machine readable language such as code in a computer language such as a computer programming language or other formal language. When the language is a computer language a “word” may be an element of that language, such as a formal command.
[0041] In implementations the input token string 122 represents a question, in a natural or computer language, that asks whether or not the task, or a stage of the task, has been achieved. That is, the input token string 122 may comprise text tokens, selected from a token vocabulary, that represents words of a natural or computer language in which the question is posed. The question can be an explicit question asking whether the task was achieved, such as “Did the robot successfully place the medium-sized gear on the shaft?” or “Did the robot successfully fetch the bananas?”. The question can be an implicit question, e.g. stating a goal and asking whether the statement is true or false, such as “The medium-sized gear is on the shaft?” or “The robot brought the bananas - true or false”.
[0042] In implementations the language model output 124 comprises (or defines) an answer to the natural or computer language question, and the answer defines whether or not the task has been achieved. That is, in implementations the language model output 124 provides a binary classification output for classifying a result of the task into one of two categories, success of the task or failure of the task. Thus implementations of the multimodal language model neural network 120 perform “success detection”.
[0043] In implementations the multimodal language model neural network 120 comprises a language generation neural network. The language model output 124 can be configured to generate an output token string comprising one or more output tokens, i.e. text tokens selected from the token vocabulary, that defines a natural or computer language answer to the question. For example the answer can comprise a single word or symbol in the natural or computer language that defines success or failure of the task, in particular a binary answer such as “yes” or “no”, or “Y” or “N”.
[0044] In some implementations the language model neural network processes the input token string 122 and the observation data from the environment to generate a score distribution, e.g., a probability distribution, that assigns a respective score (probability), to each text token in the token vocabulary. The language model neural network can then select a text token for the output token string from the token vocabulary using the score distribution. For example a highest-scoring text token may be selected, or a text token may be sampled from the distribution, e.g., using nucleus or other sampling.
[0045] The language model neural network need not explicitly generate a natural or computer language output. For example it may process the input token string to generate a language model output 124 comprising a vector or scalar output, rather than generating an output token string. In some implementations, where the language model output defines a distribution of text token scores over the token vocabulary, e.g. comprising a score for each text token of the vocabulary, the token scores for one, two, or more, respective tokens of the token vocabulary can be processed to determine the answer, which may be, e.g., a scalar value. For example, the answer may be determined from the score for a token that indicates success; or from a difference between, or ratio of, scores for two tokens indicating respectively success and failure of the task.
[0046] Some implementations of the computer system 100 include a user interface 130 for a human user, although this is not essential as the described techniques can be used without human input. The user interface 130 can provide an input mechanism for obtaining information from a user defining a task to be performed by the action selection system 110. The information defining the task may comprise, e.g., a natural or computer language input from the user. Also or instead (in a multitask implementation) information defining the task to be performed may be obtained in some other way, e.g. automatically, or by the user selecting a task from a plurality of available tasks, e.g. via the user interface.
[0047] The information defining the task can be used to generate a natural or computer language question that asks whether or not the task, or a stage of the task, has been achieved. Where the input from the user is already in the form of a suitable natural or computer language question or statement, e.g. a goal definition, this may involve using the input from the user with little modification or with no modification. In implementations, e.g. multitask implementations, information defining the task may also be provided to the action selection system 110.
[0048] Merely as an example, the user interface 130 may comprise, e.g., a mobile device, a keyboard (and optionally display), and/or a speech-based input mechanism, e.g. to input audio data characterizing a speech waveform of speech representing the input from the user in the natural or computer language and to convert the audio data into tokens representing the speech in the natural or computer language, i.e. representing a transcription of the spoken input. Also or instead the user interface 130 may include a camera, e.g. to enable the user to demonstrate a goal state or trajectory, and/or to observe the user performing the task.
[0049] In some implementations the computer system 100 includes a training engine 140, to train the action selection system 110, as described later. In some implementations, e.g. for training the action selection system 110 using imitation learning, the training engine 140 has access to training data items (not shown in FIG. 1).
[0050] FIG. 2 is a flow diagram of a first example process, for using a language model neural network to train or evaluate an action selection system for controlling an agent acting in an environment to perform a task. The process of FIG. 2 can be performed by a system of one or more computers located in one or more locations, e.g. by the computer system 100 of FIG. 1.
[0051] The process involves obtaining information defining a task (step 200). The information defining the task can define the task as a discrete or continuing behavior such as “Being me the banana that is in the pantry”, or “Move around the room quickly”, or as a desired goal state that defines when the task has been achieved, or as a desired movement trajectory. The information defining the task may comprise text in a natural language or in a computer language, e.g. code, and/or information in some other form such as a task identifier or an image defining a desired goal state for the task, e.g. a particular item placed on a table, or an image defining a desired trajectory for the task, e.g. a sketch or example of a desired trajectory of a robot end effector.
[0052] Optionally the process can also involve generating, from this, a natural or computer language question that asks whether or not the task has been achieved, i.e. accomplished (step 202). As used herein “question” is to be understood broadly and, as previously described, the question can be explicit or implicit. The information defining the task may already implicitly define a question, and generating the question may comprise using the information defining the task without modification. Optionally a question mark, or “Question:” may be added to a natural or computer language statement. In some other implementations a question may be generated from a natural language definition of the task by processing the natural language definition of the task according to a template, or by using a language model (which may be the multi-modal language generation neural network 120) with a suitable prompt that asks the language model to convert the task definition to a natural or computer language question that asks whether or not the task has been achieved. [0053] The process can then generate the input token string 122 representing text derived from the information defining the task, e.g. representing the natural or computer language question for the language model neural network (step 204).
[0054] The process can also obtain observation data from an observation that characterizes a current state of the environment 104 (step 206). The observation data represents the agent acting in the environment to perform the task whilst the agent uses the action selection system to attempt to perform the task, e.g. at one or more action selection time steps. In implementations the observation data is derived from observation 106. For example, the observation data may represent an observation of a physical, real-world environment of the agent, such as an image of the environment 106 captured by a camera.
[0055] The input token string 122, and the observation data from an observation of the environment, are processed using the language model neural network 120, to generate the language model output 124 (step 208). In implementations the language model output 124 comprises an answer, in particular to the natural or computer language question. The answer defines whether or not the task has been achieved, i.e. accomplished. The answer may comprise text, e.g. one or more words in a natural or computer language, or a score or figure- of-merit that characterizes an outcome of the task e.g. that represents a degree to which, or how well, the task was achieved, or a likelihood that the task was accomplished successfully. Such a score can then be processed to obtain a binary classification of whether or not the task was achieved.
[0056] The process can then involve using the action selection system 110 based on the answer (step 210). As one example the answer can be used to train the action selection system 110 to perform the task. As another example the answer can be used to evaluate whether or not to use the action selection system 110 for performing a task, e.g. the same task, or another similar task to that it was observed performing.
[0057] Implementations of the process can then use the action selection system 110 to control the agent 102 to perform the task, after the training or dependent on a result of the evaluation (step 212).
[0058] As one example, multiple (trained) action selection systems 110 can each be evaluated using a process as described above and the results of the evaluations used to select a particular action selection system for performing the task, i.e. based on the answer to the natural or computer language question that asks whether or not the task has been achieved. For example an action selection system that has an answer indicating “yes” or having a score above a threshold, can be chosen. [0059] As another example, the above-described process can also involve training the action selection system 110, e.g. using a reinforcement learning technique based on a reward determined from the answer, or using an imitation learning technique such as behavioral cloning. Any reinforcement learning technique or imitation learning technique may be used. For example the training may be performed online using an online reinforcement learning technique such as Maximum a-posteriori Policy Optimization. Also or instead the training can be performed offline using previously stored data e.g. using an offline reinforcement learning technique such as critic regularized regression (Wang et al. 2020). Also or instead the training can be performed offline using an imitation learning technique such as behavioral cloning, inverse reinforcement learning, or Generative Adversarial Imitation Learning (GAIL. arXiv: 1606.03476, Ho et al.). Training using imitation learning can involve filtering the previously stored data to remove examples of performing a task that were judged unsuccessful by the multimodal language model neural network 120. That is, a corpus of training data can be obtained comprising training data items each of which includes one or more examples of an agent (robot or human) performing one or more tasks; such an agent may be referred to as a demonstrating agent The training data can then be filtered to retain only training data items that include one or more successful examples of the demonstrating agent performing one or more of the tasks, and then the action selection system 110 can be trained, e.g. using imitation learning, on the filtered training data.
[0060] In general training the action selection system 110 involves iteratively adjusting learnable parameter values of the action selection system 110, e.g. learnable parameter values such as weights of an action selection neural network of the action selection system 110. This can involve iteratively backpropagating gradients of a reinforcement learning or imitation learning objective function through the action selection system 110 to update the learnable parameters. Such training can use any appropriate gradient descent optimization algorithm, e.g. Adam or another optimization algorithm. Any appropriate reinforcement learning or imitation learning objective function may be used, e.g., one dependent on a squared Bellman error (for reinforcement learning), or one that depends on a difference between a distribution of actions defined by the policy output 112 and a distribution of actions defined by the actions of one or more demonstrating agents in a set of training data items (for imitation learning).
[0061] FIG. 3 is a flow diagram of a second example process, for using a language model neural network to select training data for training an action selection system for controlling an agent acting in an environment to perform a particular task. The process of FIG. 3 can be performed by a system of one or more computers located in one or more locations, e.g. by the computer system 100 of FIG. 1.
[0062] The process involves obtaining a set of training data items (step 300). Each training data item comprises an observation, e.g. a visual observation, of an entity acting in an environment whilst attempting to perform a task. The entity can be a human or another agent, e.g. an agent controlled by another computer system; the set of training data items may include both these. In general but not essentially, as the system can generalize, the set of training data items will include at least some data items in which the environment is similar to that in which the trained agent will act, and at least some data items in which the task matches one particular task for which the action selection system is to be trained. In implementations the observation of the entity comprises a still or moving image of the entity. The observation, e.g. the visual observation, can be a simulated observation and the language model neural network can be trained partly or wholly using simulated data, e.g. prior to using the system, in particular the language model neural network 120, to process real -world data.
[0063] The process can also obtain information defining a particular task (step 302), optionally generating from that a natural or computer language question that asks whether or not the particular task has been achieved (step 304). In some implementations e.g. where different training data items comprise observations of different tasks, the information defining a particular task, and a corresponding question, may be obtained for each training data item. [0064] For each of the training data items the process generates the input token string 122 representing text derived from the information defining the task, e.g. the natural or computer language question for the language model neural network (step 306). The input token string 122, and observation data from the observation in the training data item, is processed using the language model neural network, to generate the language model output 124 (step 308). The observation in the training data item can be, e.g., a still or moving image, such as a video clip. The language model output 124 comprises an answer e.g. to the natural or computer language question that defines whether or not the particular task has been achieved in the training data item, as described above. The process identifies, for the training data item, whether or not the particular task has been achieved in the training data item (step 310). This provides identified training data items, i.e. training data items that have an identification that indicates whether or not the particular task is achieved in the training data item. The training data items can relate to multiple different tasks, in which case the identification can indicate whether or not the task represented by the particular observation in the training data item is achieved. [0065] The process can train the action selection system 110 to perform the particular task using the identified training data items (step 312), e.g. by selectively using the identified training data items. For example, each of the training data items can be labelled using the answer, e.g. according to whether or not the particular task was achieved, and the action selection system 100 can be trained using the labelled training data items. As another example, the training data items can be filtered to retain a subset of only those training data items in which the particular task was achieved, or was achieved with a score greater than a threshold. Then the subset of training data items can be used to train the action selection system 110. In some implementations the action selection system is trained, based on the labelled or filtered training data items, using imitation learning, e.g. behavioral cloning. In some implementations the action selection system is trained whilst being used to perform a task, e.g. by reinforcement learning.
[0066] The process can also involve using the trained action selection system to control the agent acting in the environment to perform the particular task (step 314).
[0067] FIG. 4A shows a second example computer system 400 for instructing, e.g. controlling, the agent 102 acting in the environment 104 to perform a task, using the language model neural network 120. The computer system 400 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.
[0068] In the arrangement of FIG. 4 A the action selection system 110 receives and uses the language model output 124 from the language model neural network 120, e.g. as described with reference to FIG. 5.
[0069] FIG. 5 is a flow diagram of a third example process, for instructing, e.g. controlling, an agent acting in an environment to perform a task. The process of FIG. 5 can be performed by a system of one or more computers located in one or more locations, e.g. by the computer system 400 of FIG. 4 A.
[0070] The process involves obtaining information defining the task (step 500), and optionally generating from that a natural or computer language question that asks whether or not the task has been achieved (step 502). The input token string 122 representing text derived from the information defining the task, e.g. the natural or computer language question, is provided to the language model neural network 120 (step 504).
[0071] The process also obtains observation data from the environment 104 (step 506). The observation data represents, i.e. comprises observations of, actions taken by the agent 102 in the environment 104 to perform the task. The input token string 122 and the observation data from the environment are processed using the language model neural network 120 to generate the language model output 124 (step 508). The language model output 124 comprises an answer to the natural or computer language question that defines whether or not the task has been achieved.
[0072] The process involves instructing the agent using the answer (step 510), e.g. by controlling the action selection system 110 using the answer.
[0073] As one example, the process can involve controlling the agent to stop taking actions to perform the task when the answer indicates that the task has been achieved, e.g. for early termination of a task. Also or instead the agent can be controlled to continue taking actions to perform the task when the answer indicates that the task has been not achieved. Controlling the agent can involve controlling the action selection system 110.
[0074] As another example the agent 102, more particularly the action selection system 110, may be configured to perform a series of tasks. The process can then involve controlling the agent to stop taking actions to perform one task when the answer indicates that task has been achieved, and to start taking actions to perform a next task in the series. Each task in the series of tasks may be therefore be a step or sub-task of a larger, overall task.
[0075] In some implementations instructing the agent using the answer comprises controlling actions taken by the agent performing the task using the answer. For example, controlling the agent, e.g. a mechanical agent, can involve controlling the action selection system 110. Instructing the agent may comprise providing control signals to the agent to control the actions taken by the agent.
[0076] FIG. 4B shows a third example computer system 450 for instructing an agent to perform a task, in which the agent 102 is a human. The computer system 450 is another example of a system implemented as computer programs on one or more computers in one or more locations in which the process of FIG. 5 can be implemented.
[0077] In FIG. 4B instructing the agent 102 using the answer comprises instructing the human using the answer. A digital assistant device 700, also sometimes referred to as a virtual assistant, observes the human agent 102, and based on the observations provides instructions that enable the human to perform the task. The digital assistant device 700 provides observations, or observation data derived from the observations, to the multimodal language model neural network 120, together with a question 702, as previously described, and receives an answer 704 from the multimodal language model neural network 120, that is used to instruct the human 102. [0078] Instructing the human agent 102 using the answer can therefore comprise instructing a human user of the digital assistant device 700. The digital assistant device 700 can be a smart speaker, or a smart display, or a mobile device, or in general any computing device that is suitable for communicating with a human user to implement the process of FIG. 5. The digital assistant device 700 captures observations of actions taken by the human 102 in the environment 104 to perform the task, e.g. using a camera. As well as, or instead of, instructing the agent using the answer, the answer can be used to control the digital assistant device, e.g. to stop receiving and/or processing observations, for privacy or to reduce power use.
[0079] In general, in the systems and methods described above an observation of the environment may comprise any type of observation including, but not limited to, a visual observation, an audio observation, and a proprioceptive observation. As examples, in some implementations the observation may comprise a captured still image or a captured moving image (a video clip), or an audio clip, and the observation data may comprise visual data, e.g. comprising an image or sequence of images, or audio data. The observation data may therefore comprise, e.g., pixel values of pixels of a still or moving image, audio signal values in the time or frequency domain, or (for a proprioceptive observation) sensor signal values.
[0080] In some implementations the observation may be captured by a camera or a LIDAR sensor, i.e. “image” and “visual data” includes a LIDAR point cloud; or by a microphone or other audio transducer; or by a position, force, torque, acceleration or other sensor. In implementations the image or audio or proprioceptive information is captured from the real world, e.g. by a camera or microphone or sensor. Observations used by the action selection system can come from sensors that are the same as, or different to, observations from sensors used by the language model neural network, and they may be captured at the same or different times.
[0081] The answer (to the natural or computer language question) that defines whether or not the task has been achieved, may be used in various ways. For example it may be used directly or indirectly for training or evaluating the action selection system, identifying a training data item, or controlling the agent.
[0082] In some implementations a system or method as described herein can involve obtaining observation data, e.g. a video or audio clip, from the environment at a sequence of stages of performance of the task, e.g. at the start and end of the task, or at multiple stages throughout the task, e.g. as defined by a sliding time window. Then, for each of the stages, the method can involve processing the input token string (which may represent a question that asks whether the stage has been achieved), and the observation data from the environment for the stage, using the language model neural network, to generate the language model output comprising the answer for the stage. The method can determine whether or not the (overall) task has been achieved from the language model output for two or more of the stages. The question may ask whether the (overall) task has been completed or whether one of the stages of the task has been completed. The method can then, e.g., control the agent, or train or evaluate the action selection system, or select a training data item, in response to the determining of whether or not the task, or a stage of the task, has been achieved.
[0083] As one example, determining whether or not the task has been achieved may comprise determining a difference between the answers for two or more of the stages, one towards the start of the task and one towards the end of the task, to determine whether the task has been achieved, e.g. to reduce the computational load of processing each observation. This can be appropriate where success in a task is determined by a state of the environment (which can be defined by a state of one or more objects in the environment).
[0084] As another example, determining whether or not the task has been achieved may comprise determining whether or not the answer indicates that the task has been achieved for any of the stages. This can be appropriate where success in a task is determined by an ongoing behavior of the agent, e.g. moving around and cleaning a room. Where success detection is specified by an ongoing behavior it can be useful if the observation data includes moving image, i.e. video, data.
[0085] FIG. 6 shows an example of an agent 600 configured to select actions to perform one or more tasks in an environment. In FIG. 6 the agent 102 as described above includes a system, implemented as computer programs on one or more computers in one or more locations, in which a process as described above can be implemented.
[0086] The agent 600 includes one or more observation capture subsystems 610 to capture observations of the environment, in particular for the action selection system 110 and for the language model neural network 120. The environment 104 may be a real-world environment and the observation capture subsystem(s) 610 can, e.g., capture a still or moving image of the real -world environment 104.
[0087] The agent also has a natural or computer language interface 620 to receive a natural or computer language description of a task to be performed 622, i.e. comprising information defining the task.
[0088] The agent 600 includes the action selection system 110 and can include, or have a language model interface 630 to, the language model neural network 120 (which may be local to the agent or remote from the agent, e.g. on a remote server). The action selection system 110 is configured to process a first captured observation of the environment to generate the action selection policy output 112 for selecting the action 114 at the time step to control the agent to perform the task. The agent 600 may include an agent control system (not shown) to interface with and control the agent in accordance with the selected actions to perform the task. For example where the agent 600 comprises a robot the agent 600 may include a robot control system.
[0089] The agent can use the language model interface 630 to interface with the language model neural network 120. The language model neural network 120 can process an input token string, in particular the previously described input token string 122 representing a question that asks whether the task has been achieved, and a second captured observation of the environment, to thereby obtain the language model output 124 comprising the answer to the question. The second captured observation of the environment may be, but need not be, the same as the first captured observation of the environment. The input token string 122 representing the question may be generated by the agent 600, e.g. using the language model interface 630, or it may be generated by the language model neural network 120. That is, in some implementations the language model neural network 120 may receive a text input from the agent 600 and generate a text output for the agent 600.
[0090] The action selection system 110 can be configured to be controlled using the answer from the language model neural network 120, e.g. obtained via the language model interface 630 (if present).
[0091] As previously described, in some implementations the agent 102 comprises a human user of a digital assistant device such as a smart speaker, smart display, or other device. Then the information defining the task can be obtained from the digital assistant device, and the digital assistant device can be used to instruct the user based on the answer. For example, this may comprise receiving, at the digital assistant device, a request from the user for assistance and determining, in response to the request, a series of tasks for the user to perform, e.g. steps or sub-tasks of an overall task. Then for one or more tasks of the series of tasks, in implementations for each task, e.g. until a final task of the series, the digital assistant device can be used to output to the user an indication of the task, e.g. step or sub-task, to be performed. This may be done using natural language, e.g. on a display and/or using a speech synthesis subsystem of the digital assistant device. Visual, e.g. video, and/or audio observations of the user performing the task may be captured, e.g. using the digital assistant device. A system as described above may then be used to determine whether the user has successfully achieved the task or a task step, or sub-task, from the answer. If there are further tasks to be completed the digital assistant device may then, in response, progress to the next task (if any) of the series of tasks, e.g. by outputting an indication of the next task to be performed. In this way the user may be led step-by-step through a series of tasks to perform an overall task.
[0092] FIG. 7 shows an example of a digital assistant device 700, including a system as described above that may be implemented as computer programs on one or more computers in one or more locations.
[0093] The digital assistant device 700 can include a user interface 710 to enable a user to request assistance and to output information. In implementations this is a natural language user interface and may comprise a keyboard, voice input-output subsystem, and/or a display. The user interface 710 can receive a request for assistance with a task, e.g. an overall task as described above, and can output instructions to the user to perform actions 114 to complete each of a series of sub-tasks of the overall task.
[0094] The digital assistant device can include a local or remote assistance subsystem 730 configured to determine, in response to the request, a series of tasks for the user to perform. In implementations this may comprise a local or remote generative (large) language model, in particular configured for dialog, e.g. a conversation agent such as LaMDA (Thoppilan et al. arXiv: 2201.08239). The language model may be suitably prompted to generate an output that identifies the series of tasks (e.g. “How would you. . . ?”). The generative (large) language model may be the language model neural network 120, or it may be another model.
[0095] The digital assistant device 700 can include an observation capture subsystem 750, and a language model interface 740 for the language model neural network 120, which may also be implemented locally or remotely. The observation capture subsystem 750 may be similar to the observation capture subsystem 610 described above, and can capture visual and/or audio observations of the human user performing a task, and provide corresponding observation data to the language model interface 740.
[0096] The digital assistant device 700 can include an assistance control subsystem 720 configured to assist the user. The assistance control subsystem 720 can receive from the assistance subsystem 730, serially or in parallel, an indication of the series of tasks that the user should perform. The assistance control subsystem can be configured to implement a process as described above for one or more tasks e.g. of a series of tasks, e.g. until a final task of the series. More particularly the assistance control subsystem 720 can output to the user an indication of a next task to be performed, e.g. via the user interface 710. [0097] The assistance control subsystem 720 can also capture, using the observation capture subsystem 750, visual and/or audio observations of the user performing the task. It can use the language model interface 740 to provide a question to the language model neural network 120 (based on the task and the observations) that asks whether the task has been achieved, and can determine from the answer whether the user has successfully achieved the task. In response to determining that the user has successfully achieved the task the digital assistant device 700, in particular the assistance control subsystem 720, can progress to a next task of the series of tasks and/or control the digital assistant device 700, e.g. to stop capturing observations.
[0098] As an illustrative example, a user may be interacting with the digital assistant device 700 and ask for help in performing an overall task with multiple steps, e.g. cooking a pasta dish. While the user performs the task, the digital assistant device receives audio and/or video inputs representative of the user's progress on the task, e.g. images or video or sound clips of the user cooking. The captured audio and/or video is provided to the language model neural network 120 together with a question that asks whether the user has completed a particular step, e.g. 'Has the user finished chopping the peppers?', to determine whether the user has successfully completed the step. If the answer confirms that the user has successfully completed the step then the digital assistant device 700 progresses to telling the user to perform the next step or, if at the end of the task, or if the overall task is a single-step task, then the digital assistant device may indicate this to the user. The digital assistant device may then stop receiving or processing audio and/or video inputs to ensure privacy and/or reduce power use.
[0099] The various systems and processes described above generally involve providing an input token string representing the natural or computer language question to the language model neural network 120. The input token string can be obtained using a tokenization system as previously described to process the natural or computer language question; many such tokenization systems are known. In general the input token string may comprise a sequence of text tokens representing words in a natural or computer language; the tokens may be selected from a token vocabulary. The sequence of text tokens specifies to the language model neural network what is required from the language model output, i.e. the text tokens represent the natural or computer language question that asks whether or not the particular task has been achieved. Typically a text token defines a word or wordpiece, e.g. a word segment or morpheme, but it may also define a letter, or multiple words. The tokens may include tokens representing punctuation, characters, numbers, or symbols, particularly in the context of a computer language.
[0100] In some implementations the input token string may additionally include text tokens representing words and/or characters that are intended to guide the language model output. For example the natural or computer language question may include prompt language or tags, e.g. to prepend “Q:” or “Question:” or to append “?”, “A:” or “Answer:_”. The prompt language or tags may also be converted into text tokens of the input token string.
[0101] In some implementations successful performance of the task may be defined by a goal observation, such as a goal image or goal sound, or by a goal trajectory defining a target movement. For example, a goal image may be used show the environment, e.g. an object in the environment, in a particular state, or it may show a desired trajectory; or a goal video may illustrate a goal behavior or trajectory. Similarly a goal sound may indicate when the environment, e.g. an object in the environment, has achieved a particular state, for example the click sound of a male Ethernet connector when it is fully inserted into its socket. In such implementations obtaining the information defining the task can comprise obtaining goal observation data defining a goal state or states of the environment for successful performance of the task.
[0102] Optionally the goal observation may be referenced in the input token string, e.g. by including a tag that indicates to the language model neural network that the goal observation is part of the natural or computer language question. Such a tag may be included, for example, at the start of the natural or computer language question. As an example, an “<image>” tag may be used to reference a goal image, followed by a question such as “Did the agent successfully achieve this goal state?”. A token for the tag may be included in the token vocabulary.
[0103] In such cases the input token string including the tag referencing the goal observation may be provided as a prompt to the multimodal language model neural network before the language model neural network processes the observation data from the environment. That is the multimodal language model neural network may process the input token string, and the goal observation data from the environment, and then the observation data from the environment, prompting the multimodal language model neural network 120 in a way that is broadly analogous to prompting a large language model.
[0104] In some implementations the language model neural network is configured to jointly model the input token string 122 and the observation data to determine the language model output 124. [0105] As previously described, the multi-modal language model neural network 120 may comprise a pre-trained visual language model (VLM), of which there are many examples. Merely by way of illustration, FIG. 8 shows details of one way in which the multi-modal language model neural network 120 can be implemented.
[0106] More particularly FIG. 8 shows an example multi-modal language model neural network 800 that may be implemented as computer programs on one or more computers in one or more locations. The multi-modal language model neural network 800 is suitable for use as multi-modal language model neural network 120.
[0107] In FIG. 8 the input token string 122 and the observation data from the environment are processed using the language model neural network 800 to generate the language model output 124. The observation 106, more particularly the observation data, is processed using an observation encoder neural network subsystem 810 to determine a compressed representation of the observation data, e.g. a set of observation tokens 812. The compressed representation, and data derived from the input token string 122, may be processed by applying a cross-attention mechanism between the compressed representation and the data derived from the input token string, to generate the language model output 124.
[0108] The cross-attention mechanism may be implemented by a cross-attention neural network layer 820. There are many possible cross-attention mechanisms. One such a crossattention mechanism is a query-key-value (QKV) attention operation in which keys and values are obtained from the observation data, using text token-derived queries that attend to the observation data. For example, a set of key-value vector pairs can be obtained from the compressed representation and one or more query vectors can be derived from the input token string 122. The output of the cross-attention mechanism may be determined by determining a weighted sum of the values, weighted by a similarity function of the query to each respective key.
[0109] In some implementations the language model neural network comprises a stack of processing layers including a plurality of token processing neural network layers 830 and a plurality of cross-attention neural network layers 820. Each cross-attention neural network layer 820 can be arranged to receive, e.g. can be conditioned on, the compressed representation and can implement a cross-attention mechanism as described above. The token processing neural network layers 830 can be interleaved with the cross-attention layers. For example a layer output of one of the cross-attention neural network layers 820 can provide a layer input for one of the token processing neural network layers 830. Similarly a layer output of one of the token processing neural network layers 830 can provide a layer input for one of the cross-attention neural network layers 820; and so forth.
[0110] In some implementations some or all of the cross-attention neural network layers 820 are gated cross-attention layers. In a gated cross-attention layer the output of the layer is combined with the input to the layer in a proportion that depends on a gating parameter, e.g. a larger gating parameter indicating a larger proportion of the layer output. This is so that during training the gating parameter may be gradually increased, e.g. as the token processing neural network layers and the cross-attention neural network layers learn to cooperate with one another.
[0111] Processing the compressed representation of the observation data and the data derived from the input token string 122 may comprise processing the data derived from the input token string using the stack of processing neural network layers 820, 830, using each of the cross-attention layers 820 to attend to the compressed representation.
[0112] In some implementations a token processing neural network layer 830 can comprise a self-attention neural network layer, and is optionally followed by a feedforward neural network layer. A self-attention neural network layer has an attention layer input and is configured to apply an attention mechanism over the attention layer input to generate an attention layer output. There are many different attention mechanisms that may be used, e.g. a QKV mechanism as described above. In a self-attention mechanism the query, key, and value vectors are all derived from the (same) attention layer input. The self-attention neural network layer, and/or the feedforward neural network layer, may have a residual connection. [0113] In some implementations the observation encoder neural network subsystem 810 comprises an observation encoder 814, e.g. a neural network such as a ResNet, and one or more observation encoder cross-attention neural network layers 816.
[0114] The compressed representation can comprise a set of observation tokens 812; this may comprise a fixed number of tokens irrespective of the observation data, e.g. a fixed number of tokens per image or video clip. The observation tokens 812 can be determined by processing the observation data using the observation encoder 814 to generate encoded observation data comprising a set of observation features, e.g. visual features. For example an image or each image of a video clip may be encoded in this way. The set of observation tokens 812 can then be determined from the encoded observation data by processing a set of (learned) latent vectors 818 using one or more observation encoder cross-attention layers configured to cross-attend to the observation features, to generate the set of observation tokens, e.g. as described in Jaegle et al., “Perceiver IO: A General Architecture for Structured Inputs and Outputs”, arXiv 2107.14795. That is, after training the set of latent vectors comprises a set of predetermined latent vectors. This approach can be used for example to generate a (fixed size) set of visual tokens from an image or video clip.
[0115] The language model neural network 800 can include a token embedding subsystem that applies an embedding function to each token of the input token string to transform it to a respective token embedding, i.e. a vector of numeric values, prior to processing the input token string using the stack of processing layers. Such an embedding function may be fixed or learned. In implementations a positional encoding can be applied, e.g. added or concatenated, to the token embeddings to indicate a position of each respective token in the input token string 122.
[0116] As previously described a pre-trained model, e.g. a VLM, may be used for the language model neural network 120 of FIG. 1, because such pre-trained models can be capable of performing a question answering task, e.g. a visual question answering task, of the type used herein for task success detection. Nonetheless in some implementations such a pretrained model can be “fine-tuned”, i.e. further trained, using additional training data items. These additional training data items may be specific to task success detection. Such an additional training data item can include an observation of a task, a natural or computer language question asking whether the task was achieved, and an answer appropriate to the observation.
[0117] Also or instead, should further adaption of a pre-trained multimodal language model neural network be beneficial, the multimodal language model neural network 120 can be prompted with one or more examples of task success detection, e.g. comprising an image, question text, and example answer text.
[0118] In implementations neither fine tuning, nor prompting, of the multimodal language model neural network 120 is essential.
[0119] Where the multimodal language model neural network 120, or the language model neural network 800, is trained from scratch a two-stage training process can be used. In a first, pre-training stage the multimodal language model neural network can be trained on a large but noisy corpus of data. In this corpus the training data items may comprise images or other observations and associated text, but need not be specific to a success detection task. Such training data items may be obtained, e.g., from the web; an example of this is the ALIGN dataset (Jia et al., arXiv:2102.05918). In this context, “large” can mean having greater than one billion training data items, e.g. image-text pairs. In a second, fine tuning stage the multimodal language model neural network can be trained on a smaller corpus of additional training data items, relatively fewer in number than the training data items used in the first stage, but specific to success detection.
[0120] Where fine tuning is used the (additional) training data items can be generated by manually annotating observations of tasks to indicate whether the task was competed. For observations that extend in time, such as video or audio clips, the manual annotation can also indicate, if the task was competed, when success occurred. For example, for a video or audio clip observation this can be done by splitting the clip into non-overlapping subsequences, before and after the point of success. Two (additional) training data items can then be generated, each including a question as to whether the task was successfully completed, and each including an appropriate ground truth answer, e.g. “yes” if the subsequence ends in one or more success frames, and “no” otherwise. With multiple human raters success or failure can be determined by a majority vote; and the point of success can be determined as the median of the first annotated success frame across the raters.
[0121] These (additional) training data items can include observations of environments or tasks that are similar to, or the same as, those for which the system will later be used, although this is not essential. When a task description, “{task}”, is available the question can be generated using a template such as “Did the robot/agent/person successfully {task}”. When no task description is available but there is a narration corresponding to the actions, e.g. as in the Ego4D dataset (Grauman et al’, arXiv:2110.07058), a trained VLM such as Flamingo (arXiv:2204.14198) can be used to convert the narration into a question, e.g. to convert a narration such as “The person is scooping the ice cream” into “Did the person successfully scoop the ice cream?”.
[0122] In one merely illustrative example a training dataset of (additional) training data items was generated by using human operators to provide 101,789 demonstrations of 6 tasks using a 6DoF control device, a Panda robot arm (Franka Emika GmbH). Each episode was then annotated by humans with rewards for each task, each episode having 6 reward annotations, one for each task, labelling with a positive reward all frames with a success state, i.e. if the task is solved, and labelling with zero reward otherwise (if a task is accidentally undone within an episode, at that point the reward annotation reverted to zero). The reward annotations and corresponding episode frames were then converted into training data items. Ground truth answers were obtained from the human annotations (labelling a video clip with a “successful” answer if it contained a single transition from zero to positive reward or only had positive rewards throughout, otherwise labelled the clip as unsuccessful). The multimodal language model neural network 120 was then trained to detect success for the multiple different tasks. [0123] FIG. 9, taken from this training dataset, shows an example of generating two (additional) training data items 910, 912 from comprising an annotated behavior trajectory video 900 for a task, with a point of success 902. Each training data item 910, 912, includes a natural language question, a part of the video indicating respectively failure or success of the task, and a corresponding answer to natural language question, respectively “no” or “yes”. Where the answer is “yes” the part of the video in the training data item contains the success frame(s) or follows the success frame(s).
[0124] Using the example language model neural network 800 of FIG. 8, there is now described one example training process that can be used for either pre-training or fine tuning. Other language model neural networks can be trained in a manner appropriate to their architecture, which may be similar to or different from that of the example of FIG. 8.
[0125] FIG. 10 is a flow diagram of an example process for training a multimodal language model neural network, such as the language model neural network 800 of FIG. 8. The process of FIG. 10 can be performed by a system of one or more computers located in one or more locations, e.g. by the computer system 100 of FIG. 1.
[0126] As previously described, there are many different types of model, e.g. VLM, that can used for the multimodal language model neural network. Where the model is trained, e.g. fine-tuned, for task success detection this can be done in a way that is appropriate to training the model generally.
[0127] As one example, the multimodal language model neural network may comprise a model, such as a transformer-based model, that is configured to process a sequence of tokens comprising tokens from the input token string and observation tokens. The observation tokens may comprise image tokens, e.g. from encoding an image or patches of an image, or multimodal tokens. Such a model can process, and be trained on, a mixed sequence of such tokens. For example such a model can be natively trained on a combination, e.g. sequence, of text tokens and observation tokens (such as image tokens) provided as an input to the model. In general such training can involve backpropagating gradients of a loss function, e.g. as described below. Optionally, but not necessarily, such a model may also include a crossattention mechanism as described below.
[0128] In the example of FIG. 10 training the multimodal language model neural network 800 starts from a pre-trained and frozen text-only language model. However it is not essential to start from a pre-trained and frozen text-only language model.
[0129] Continuing this particular example, in some implementations the above described token processing layers 830 may be obtained from the pre-trained and frozen text-only language model, e.g. from an autoregressive model incorporating such layers. The language model can be a so-called large language model, e.g. having greater than IO10, 1011 or 1012 trainable (trained) parameters. There are many such models; merely as one example the “Chinchilla” model may be used (Hoffmann et al., “Training Compute-Optimal Large Language Models”, arXiv:2203.15556). Such a text-only language model can be trained on the very large quantities of unlabeled text available e.g. in books and on the Internet. The multimodal language model neural network 800 can then be trained whilst keeping the neural network parameter values, e.g. weights, of the token processing layers 830 frozen, i.e. constant, during the training.
[0130] In more detail, this example training process may comprise obtaining a trained generative natural or computer language neural network (step 1000). The generative natural or computer language neural network is configured to process a string of input tokens representing words in a natural or computer language, to generate output tokens for the natural or computer language, and comprises a stack of trained token processing layers.
[0131] The process can then form a multimodal language model neural network (step 1002). The multimodal language model neural network is configured to process i) a sequence of text tokens representing words in a natural or computer language, and ii) observation data comprising an observation of the environment, to generate a language model output comprising a string of one or more output tokens, e.g. one at a time.
[0132] The multimodal language model neural network can be as described above, i.e. similar to the multimodal language model neural network 800. For example the multimodal language model neural network may comprise a stack of processing layers, the stack of processing layers including a plurality of the trained token processing layers 830 and a plurality of crossattention layers 820, each cross-attention layer being arranged to receive a compressed representation of the observation data, the trained token processing layers being interleaved with the cross-attention layers.
[0133] The process can further comprise obtaining a set of training data items (step 1004). Each training data item can comprise an observation of an environment, and natural or computer language text relating to the observation.
[0134] The process can train the multimodal language model neural network, using the training data items, by adjusting parameters of the cross-attention layers. In some implementations, but not necessarily, this is done whilst keeping parameters of the trained token processing layers frozen, i.e. constant (step 1006). For example, the multimodal language model neural network can be trained to predict the text tokens representing the natural or computer language text in the training data item, whilst conditioned on observation data derived from the observation in the training data item.
[0135] Training the multimodal language model neural network can involve backpropagating gradients (with respect to the parameters of the cross-attention layers) of a loss function, through the trained token processing layers 830. The loss function can be one that encourages the multimodal language model neural network to generate a language model output 124, in particular an output token string representing the natural or computer language text, that matches the ground truth language corresponding to an observation, in a training data item. This can involve processing, using the multimodal language model neural network, observation data derived from the observation in the training data item and an input token string representing the natural or computer language text for the observation.
[0136] The output token string may be generated one token at a time. For example the loss function may be based on p(y x) where x denotes the observation data (in particular, the above described compressed representation), L is the number of tokens in the input token string, yt denotes the Z-th token of the input token string, y<t denotes the I — 1 preceding characters of the input token string, and where p (yt | y<t, x) denotes the probability of the multimodal language model neural network generating an output token that corresponds to yt by processing y<t and x. For example the loss function may comprise a negative log-likelihood term — y<i> x<i), e.g. averaged over a minibatch of training data items.
[0137] In some implementations training the multimodal language model neural network also involves training, i.e. adjusting parameters of, the above described observation encoder neural network 814. Optionally, however, a pre-trained and frozen observation encoder neural network may be used, e.g. one trained based on contrastive learning that does not require labelled data. Where a gating parameter is used for the cross-attention layers, this may be gradually increased during training to gradually increase the influence of the cross-attention layers.
[0138] For the pre-training the training data items may comprise, e.g., still or moving images, e.g. video clips, or audio clips, and accompanying natural or computer language describing the content of the image or audio clip, e.g. narration. It is not necessary for the natural or computer language in the training data items to be in a question and answer format. For the fine tuning, where necessary some (additional) training data items can be generated using manual or automatic annotation, e.g. collecting visual data from a camera in an environment, or collecting data from robots for proprioceptive observations.
[0139] In some implementations the multimodal language model neural network may be trained partially or wholly in simulation, i.e. using simulated training data representing a simulation of a real-world environment, before the multimodal language model neural network is used to process observations from the real-world environment that was simulated. In such cases the simulation need only be approximate as one advantage of the described techniques is that they are able to generalize to novel tasks, environments, behaviors, and language.
[0140] Some examples of the robustness of the described techniques are now described. A task such as “Bring me the banana that is in the pantry”, can also be expressed as “Bring some fruits or vegetables from the pantry” or “Bring the yellow colored object near me”. Nonetheless the language model-based, in particular VLM-based, approach helps the system to understand that these can relate to different descriptions of the same task. One example implementation of the system exhibited close to human performance (88%) when testing for success detection on previously unseen task examples (held out examples from the training dataset), and on previously unseen behaviors (the behaviors of new, out of distribution, agents). The described techniques were also able to detect success on new tasks, not seen during training, e.g. “arrange 4 pointy objects in a square shape in the bedroom” (where only “arc” and “triangle” and not “square” are mentioned in the training dataset), or “hit the candle using the pillow which is left of airplane in the living room” (where “hit” is not mentioned in the training dataset). It is surmised that this ability to generalize is facilitated by pre-training of the multimodal language model neural network on a large, e.g. web-scale, dataset.
[0141] FIG. 11 illustrates use of the multimodal language model neural network 120 for task success detection, in particular for determining whether or not a Panda robot arm task was successfully performed (“Q: Did the robot successfully insert a medium gear?”).
[0142] FIGS. 12A-C illustrate evaluations of success detection performance when varying camera viewpoint, and when including one or more distractor objects. FIG 12A illustrates a video clip from a front camera, as used in the training data in this particular implementation (a baseline); FIG. 12B illustrates a video clip from a rear camera (absent from the training data); and FIG. 12C illustrates a video clip from a test including distractor objects (pegs). The baseline success detection performance for an “insert” task of the type illustrated was in the range 90-95%, falling by around 5% with the change to a rear camera viewpoint, and exhibiting little change when distractor objects were present. [0143] The robustness of the described techniques to variations of this type in the task, agent and environment (e.g. visual changes such as camera view angles, lighting conditions, and background changes) facilitates effective training, evaluation, and use of the action selection system 110 as previously described. Implementations of the system can thus provide an action selection system 110 that has an improved capability of acting in a real -world environment.
[0144] As previously described, implementations of the described techniques are also able to detect successful performance of a task by a human, and hence can be used to guide a human performing a task. Such a system can be trained, e.g., using the publicly available Ego4D dataset of egocentric human-in-the-wild videos. The videos show people executing common tasks (e.g., washing dishes, cleaning cars, gardening), and the Ego4D Forecasting + Hands & Objects (FHO) dataset has corresponding narrations that describe the actions of the camera wearer in the videos. Other annotations include so-called critical state changes, i.e. “how the camera wearer changes the state of an object by using or manipulating it”, and “critical frames” PRE, Point of No Return (PNR), and POST (which indicate when the state change has occurred), as well as action verbs, object nouns, and state change types. The PNR frame annotates the start of a state change, the PRE frame indicates a point before the state change, and the POST frame indicates a point after the state change is completed. A PNR frame can be treated as a point at which “success” occurs; to generate a negative example frames prior to the PRE frame can be used. Questions can be generated by rephrasing the narrations into questions, e.g. using a VLM such as Flamingo (ibid). FIG. 13 illustrates generating (additional) training data items for human task performance success detection from the Ego4D dataset, for a task of rolling dough.
[0145] In some implementations of the above described systems and processes the agent 102 is a mechanical agent, the environment 104 is a real -world environment, and the observations, in particular the observations processed by the action selection system 110 and the observation from which the observation data is derived for the language model neural network 120, are from one or more sensors sensing the real -world environment. The actions 114 are for controlling the mechanical agent acting in the real-world environment to perform the task, and can be used to control the agent 102 in the real -world environment.
[0146] In some implementations the agent 102 is a simulation of a mechanical agent, the environment 104 is a simulation of a particular real -world environment, the observations relate to the particular real-world environment, and the actions relate to actions to be performed by the mechanical agent acting in the particular real -world environment to perform the task. After training or evaluating the action selection system 110 as described above, using the simulation of the mechanical agent in the simulation of the particular real-world environment, the process can then use the mechanical agent in the particular real-world environment to perform the task. That is, training or evaluating of the action selection system may be partly or wholly performed in simulation before using the action selection system in the real world.
[0147] More particularly, where the environment 104 is a real -world environment, the agent 102 can be a mechanical agent interacting with the real -world environment, such as a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the agent may be a robot or vehicle interacting with the environment to accomplish a specific task, e.g., to locate an object of interest in the environment, or to manipulate or change the state of a specified object, or to move a specified object to a specified location in the environment, or to navigate to a specified destination in the environment. Also or instead the task may be characterized by an ongoing behavior, e.g. “vacuum the house”. The information defining the task, e.g. the object, state, location, destination, or behavior, may be specified by a natural or computer language or other instruction, e.g. from a user interface of the system, as previously described.
[0148] The observations processed by the action selection system 110, and/or by the language model neural network 120, can be derived from one or more sensors of the environment 104, for example from a camera or other image sensor or from a microphone. The sensor(s) may be mounted on the agent and/or located separately from the agent in the environment. The observations may comprise various types of sensor data as well as or instead of that previously described, e.g. object position data, data from a distance or position sensor, data from an actuator, or sensed electronic signals such as motor current or a temperature signal. The observations may also include data characterizing the current state of the mechanical agent or robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot or of one or more parts of the agent. Optionally in any of the described implementations the observation at any given time step may include (observation) data from a previous time step that may be beneficial in characterizing the environment.
[0149] The actions 114 may comprise control signals to control the robot or other mechanical agent; a control system can be used to generate control signals for controlling the mechanical agent. The control signal may comprise, e.g., torques for the joints of the robot or torques to a control surface or other control elements e.g. steering control elements of a vehicle, or higher-level control commands. More generally the control signals can include, for example, position, velocity, or force, torque, or acceleration data for one or more joints or other parts of a robot or other mechanical agent. The control signals may also or instead include electronic control data such as motor control data, or signals to control navigation, e.g., steering, movement, braking, and/or acceleration of the agent.
[0150] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
[0151] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0152] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
[0153] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.
[0154] Similarly, in this specification the term “engine” is used broadly to refer to a softwarebased system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.
[0155] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
[0156] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. [0157] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
[0158] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.
[0159] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.
[0160] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework. [0161] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
[0162] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.
[0163] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
[0164] Similarly, while operations are correspond toed in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. [0165] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes correspond toed in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method of using a language model neural network to train or evaluate an action selection system for controlling an agent acting in an environment to perform a task, wherein the action selection system is configured to process an observation characterizing a current state of the environment, to generate a policy output for selecting an action to be implemented by the agent to perform the task, and wherein the language model neural network comprises a multimodal model and is configured to process i) a sequence of text tokens representing words in a natural or computer language, and ii) observation data comprising an observation of the environment, to generate a language model output; the method comprising: obtaining information defining the task; generating, from the information defining the task, a natural or computer language question that asks whether or not the task, or a stage of the task, has been achieved; generating an input token string representing the natural or computer language question for the language model neural network; obtaining observation data from the environment whilst the agent uses the action selection system to attempt to perform the task, the observation data representing the agent acting in the environment to perform the task; and processing the input token string, and the observation data from the environment, using the language model neural network, to generate the language model output, wherein the language model output comprises an answer to the natural or computer language question that defines whether or not the task, or the stage of the task, has been achieved; and training the action selection system to perform the task using the answer, or evaluating whether to use the action selection system to perform the task using the answer.
2. The method of claim 1, further comprising using the action selection system to control the agent to perform the task after the training or dependent on a result of the evaluating.
3. The method of claim 1 or 2, wherein training the action selection system to perform the task using the answer comprises: training the action selection system using a reinforcement learning technique based on a reward determined from the answer.
4. A computer-implemented method of using a language model neural network to train an action selection system for controlling an agent acting in an environment to perform a particular task, wherein the action selection system is configured to process an observation characterizing a current state of the environment, to generate a policy output for selecting an action to be implemented by the agent to perform the particular task, and wherein the language model neural network comprises a multimodal model and is configured to process i) a sequence of text tokens representing words in a natural or computer language, and ii) observation data comprising an observation of the environment, to generate a language model output; the method comprising: obtaining a set of training data items, each training data item comprising an observation of an entity acting in an environment whilst attempting to perform a task; obtaining information defining the particular task; generating, from the information defining the particular task, a natural or computer language question that asks whether or not the particular task, or a stage of the particular task, has been achieved; and for each of the training data items: generating an input token string representing the natural or computer language question for the language model neural network; processing the input token string and observation data from the observation in the training data item, using the language model neural network, to generate the language model output, wherein the language model output comprises an answer to the natural or computer language question that defines whether or not the particular task, or the stage of the particular task, has been achieved in the training data item; and identifying, for the training data item, whether or not the particular task has been achieved in the training data item; and training the action selection system to perform the particular task using the identified training data items.
5. A computer-implemented method of instructing an agent acting in an environment to perform a task using a language model neural network, wherein the language model neural network comprises a multimodal model and is configured to process i) a sequence of text tokens representing words in a natural or computer language, and ii) observation data comprising an observation of the environment, to generate a language model output, the method comprising: obtaining information defining the task; generating, from the information defining the task, a natural or computer language question that asks whether or not the task, or a stage of the task, has been achieved; generating an input token string representing the natural or computer language question to the language model neural network; obtaining observation data from the environment, representing the agent acting in the environment to perform the task; processing the input token string, and the observation data from the environment, using the language model neural network, to generate the language model output, wherein the language model output comprises an answer to the natural or computer language question that defines whether or not the task, or the stage of the task, has been achieved; and instructing the agent using the answer.
6. The method of claim 5, wherein instructing the agent using the answer comprises controlling actions taken by the agent performing the task using the answer.
7. The method of any one of claims 1-6, wherein the language model neural network is configured to jointly model the input token string and the observation data to determine the language model output.
8. The method of any one of claims 1-7, wherein processing the input token string and the observation data from the environment using the language model neural network comprises: processing the observation data using an observation encoder neural network subsystem to determine a compressed representation of the observation data; and processing the compressed representation and data derived from the input token string by applying a cross-attention mechanism between the compressed representation and the data derived from the input token string, to generate the language model output.
9. The method of claim 8, wherein the language model neural network comprises a stack of processing layers, the stack of processing layers including a plurality of token processing layers and a plurality of cross-attention layers, each cross-attention layer being arranged to receive the compressed representation, the token processing layers being interleaved with the cross-attention layers; and wherein processing the compressed representation and the data derived from the input token string comprises: processing the data derived from the input token string using the stack of processing layers, wherein processing the data derived from the input token string using the stack of processing layers further comprises using each of the cross-attention layers to attend to the compressed representation.
10. The method of claim 8 or 9, wherein the compressed representation comprises a set of observation tokens, and wherein processing the observation data using the observation encoder neural network subsystem to determine the compressed representation of the observation data further comprises: processing the observation data using an observation encoder to generate encoded observation data comprising a set of observation features; and determining the set of observation tokens from the encoded observation data by processing a set of latent vectors using one or more observation encoder cross-attention layers configured to cross-attend to the observation features, to generate the set of observation tokens.
11. The method of any one of claims 1-10, wherein the language model neural network is a language generation neural network, wherein the input token string comprises text tokens selected from a token vocabulary that represents words of a natural or computer language of the natural or computer language question, and wherein the language model output is configured to generate an output token string comprising one or more text tokens selected from the token vocabulary, that defines a natural or computer language answer to the natural or computer language question.
12. The method of any one of claims 1-10, wherein the language model neural network is a language generation neural network, wherein the input token string comprises text tokens selected from a token vocabulary that represents words of a natural or computer language of the natural or computer language question, and wherein the language model output defines a distribution of token scores over the token vocabulary, the method further comprising processing the token scores for two respective tokens of the token vocabulary to determine the answer.
13. The method of claim 11 or 12, wherein the language generation neural network is an autoregressive neural network that is configured to process tokens of the input token string to sequentially generate tokens of the output string.
14. The method of claim 11, or claim 13 when dependent upon claim 11, wherein the natural or computer language answer comprises a single word in the natural or computer language that defines success or failure of the task.
15. The method of any one of claims 1-14, wherein obtaining observation data from the environment comprises: obtaining observation data from the environment at a sequence of stages of performance of the task and, for each of the stages: processing the input token string, and the observation data from the environment for the stage, using the language model neural network, to generate the language model output comprising the answer for the stage; and determining whether or not the task, or the stage of the task, has been achieved from the language model output for two or more of the stages.
16. The method of claim 15, wherein determining whether or not the task, or the stage of the task, has been achieved comprises determining a difference between the answers for two or more of the stages.
17. The method of claim 15, wherein determining whether or not the task, or the stage of the task, has been achieved comprises determining whether or not the answer indicates that the task has been achieved for any of the stages.
18. The method of any one of claims 1-17, wherein obtaining the information defining the task comprises obtaining goal observation data defining a goal state of the environment for successful performance of the task, or the stage of the task; and further comprising: generating a prompt for the language model neural network from a combination of the natural or computer language question and the goal observation data; and using the language model neural network to process the prompt comprising the natural or computer language question and the goal observation data, and then to process the observation data from the environment.
19. The method of any one of claims 1-18, wherein the observation data comprises visual data comprising an image or sequence of images.
20. The method of any one of claims 1-19, wherein whether or not the task, or the stage of the task, has been achieved is defined by whether or not the agent displays an ongoing behavior.
21. A computer-implemented method of training a multimodal language model neural network for use in the method of any one of claims 1-20, wherein the method comprises: obtaining a trained generative natural or computer language neural network, wherein the generative natural or computer language neural network configured to process a string of input tokens representing words in a natural or computer language to generate output tokens in the natural or computer language, and comprises a stack of trained token processing layers; forming the multimodal language model neural network, wherein the multimodal language model neural network is configured to process i) a sequence of text tokens representing words in a natural or computer language, and ii) observation data comprising an observation of the environment, to generate a language model output comprising a string of one or more output tokens, the multimodal language model neural network comprising a stack of processing layers, the stack of processing layers including a plurality of the trained token processing layers and a plurality of cross-attention layers, each cross-attention layer being arranged to receive a compressed representation of the observation data, the trained token processing layers being interleaved with the cross-attention layers; obtaining a set of training data items, each training data item comprising an observation of an environment and natural or computer language text relating to the observation; and training the multimodal language model neural network using the training data items by adjusting parameters of the cross-attention layers whilst keeping parameters of the trained token processing layers frozen.
22. The method of any one of claims 1-21, wherein the agent is a mechanical agent, the environment is a real-world environment, the observations are from one or more sensors sensing the real-world environment, and the actions are for controlling the mechanical agent acting in the real-world environment to perform the task.
23. The method of any one of claims 1-21 when dependent on any one of claims 1-4, wherein the agent is a simulation of a mechanical agent, the environment is a simulation of a particular real-world environment, the observations relate to the particular real-world environment, and the actions relate to actions to be performed by the mechanical agent acting in the particular real-world environment to perform the task, the method further comprising, after training or evaluating the action selection system using the simulation of the mechanical agent in the simulation of the particular real-world environment, using the mechanical agent in the particular real-world environment to perform the task.
24. The method of any one of claims 5-21 when dependent on claim 5, wherein the agent comprises a user of a digital assistant, the method comprising: obtaining the information defining the task from the digital assistant; and using the digital assistant to instruct the user based on the answer.
25. The method of claim 24, further comprising receiving, at the digital assistant, a request from the user for assistance; determining, in response to the request, a series of tasks for the user to perform; and for one or more tasks of the series of tasks: outputting, from the digital assistant to the user, an indication of the task to be performed; capturing, using the digital assistant, visual or audio observations of the user performing the task; determining from the answer whether the user has successfully achieved the task and, in response, progressing to a next task of the series of tasks.
26. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-25.
27. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-25.
28. An agent configured to select actions to perform one or more tasks in an environment, the agent comprising the system of claim 27, and further comprising: one or more observation capture subsystems to capture observations of the environment; a natural or computer language interface to receive a natural or computer language description of a task to be performed, wherein the natural or computer language description of the task to be performed comprises the information defining the task; the action selection system, wherein the action selection system is configured to process a first captured observation of the environment to generate an action selection policy output for selecting an action at the time step to control the agent to perform the task; and an interface for the language model neural network, wherein the language model neural network is configured to process the input token string and a second captured observation of the environment to generate the language model output comprising the answer; and wherein the action selection system is configured to be controlled using the answer from the language model neural network.
29. A digital assistant device including the system of claim 27, and further comprising: a user interface to enable a user to request assistance and to output information; an assistance subsystem configured to determine, in response to the request, a series of tasks for the user to perform; an observation capture subsystem to capture visual or audio observations of the user performing a task; an interface for the language model neural network; and an assistance control subsystem configured to assist the user, wherein the assistance control subsystem is configured to, for one or more tasks of the series of tasks: output, from the digital assistant to the user, an indication of the task to be performed; capture, using the observation capture subsystem, visual or audio observations of the user performing the task; and determine from the answer whether the user has successfully achieved the task and, in response, progress to a next task of the series of tasks.
EP24700225.6A 2023-01-05 2024-01-05 Controlling agents using language-based success detectors Pending EP4627480A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202363437344P 2023-01-05 2023-01-05
PCT/EP2024/050243 WO2024146961A1 (en) 2023-01-05 2024-01-05 Controlling agents using language-based success detectors

Publications (1)

Publication Number Publication Date
EP4627480A1 true EP4627480A1 (en) 2025-10-08

Family

ID=89619094

Family Applications (1)

Application Number Title Priority Date Filing Date
EP24700225.6A Pending EP4627480A1 (en) 2023-01-05 2024-01-05 Controlling agents using language-based success detectors

Country Status (3)

Country Link
EP (1) EP4627480A1 (en)
CN (1) CN120712567A (en)
WO (1) WO2024146961A1 (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12437238B1 (en) * 2024-03-20 2025-10-07 Anthropic, Pbc Generation of agentic trajectories for training artificial intelligence agents to automate multimodal interface task workflows
CN119150863B (en) * 2024-11-05 2025-03-18 中国计量大学 A dynamic reasoning method and system under a large language model
CN119377679B (en) * 2024-12-27 2025-04-01 鹏城实验室 Model training method, device, storage medium and computer equipment

Also Published As

Publication number Publication date
CN120712567A (en) 2025-09-26
WO2024146961A1 (en) 2024-07-11

Similar Documents

Publication Publication Date Title
JP7674599B2 (en) Controlling interactive agents using multimodal input
US11663441B2 (en) Action selection neural network training using imitation learning in latent space
CN112119409B (en) Neural network with relational memory
EP3788554B1 (en) Imitation learning using a generative predecessor neural network
WO2024146961A1 (en) Controlling agents using language-based success detectors
US12505346B2 (en) Autoregressively generating sequences of data elements defining actions to be performed by an agent
US20250209340A1 (en) Intra-agent speech to facilitate task learning
JP2026502834A (en) Real-World Robot Control Using Transformer Neural Networks
JP2025540687A (en) Training Multimodal Interactive Agents with Reward Models
US20250363381A1 (en) Multi-turn reinforcement learning for generative machine learning models
CN121179443B (en) Autonomous planning method based on large language model
US20250245502A1 (en) Training neural networks using weight norm regularizations
WO2025265056A1 (en) Weight averaged rewarded policy training for machine learning models
WO2025189013A1 (en) Training neural networks through reinforcement learning using multi-objective reward neural networks
US20240412063A1 (en) Demonstration-driven reinforcement learning
US20260127431A1 (en) Autoregressively generating sequences of data elements defining actions to be performed by an agent
US20250363354A1 (en) Composing machine learning models to perform new tasks
CN121328720A (en) Use compressed representations to adapt generative models to new contextual data.
WO2026050538A1 (en) Neural networks with per layer embedding injection

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20250704

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)