EP4639415A1 - Training reinforcement learning agents to perform multiple tasks across diverse domains - Google Patents

Training reinforcement learning agents to perform multiple tasks across diverse domains

Info

Publication number
EP4639415A1
EP4639415A1 EP24700412.0A EP24700412A EP4639415A1 EP 4639415 A1 EP4639415 A1 EP 4639415A1 EP 24700412 A EP24700412 A EP 24700412A EP 4639415 A1 EP4639415 A1 EP 4639415A1
Authority
EP
European Patent Office
Prior art keywords
neural network
latent representation
environment
state
generate
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP24700412.0A
Other languages
German (de)
French (fr)
Inventor
Danijar HAFNER
Jurgis Pasukonis
Timothy Paul Lillicrap
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
GDM Holding LLC
Original Assignee
GDM Holding LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by GDM Holding LLC filed Critical GDM Holding LLC
Publication of EP4639415A1 publication Critical patent/EP4639415A1/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/008—Artificial life, i.e. computing arrangements simulating life based on physical entities controlled by simulated intelligence so as to replicate intelligent life forms, e.g. based on robots replicating pets or humans in their appearance or behaviour
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G06N3/0455—Auto-encoder networks; Encoder-decoder networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/092—Reinforcement learning

Definitions

  • Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input.
  • Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.
  • This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains a policy neural network that can be used to control an agent to interact with an environment by selecting actions to be performed by the agent.
  • the system implements a general reinforcement learning framework that can effectively train the policy neural network to select actions that enable the agent to accomplish a large number of tasks across a wide range of domains.
  • the tasks can include, for example, navigating to a specified location in the environment, identifying a specific object in the environment, manipulating the specific object in a specified way, and so on.
  • the tasks are long-horizon tasks that require the agent to select and perform thousands or millions of actions in a sequence in DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application order to complete a task.
  • the tasks are tasks performed by the agent in a complex environment, e.g., a complex workspace that includes a large number of obstacle and/or target objects, a large number of target locations, or both.
  • a domain of each task generally defines the properties and/or characteristics of various aspects of the task, e.g., the environment within which the agent is deployed, the space of possible actions that can be performed by the agent, the data that characterizes the environment (also referred to in this specification as an “observation”) received by the agent, and so on.
  • the domains of these tasks can differ from each other in one or more of these aspects, e.g., in the action space (e.g., continuous versus discrete action space), observation space (e.g., higher-dimensional visual inputs versus lower-dimensional textual inputs, or 2D versus 3D camera images), data budgets, reward frequencies, or reward scales.
  • this specification describes a reinforcement learning method performed by one or more computers for training a policy neural network to control an agent acting in an environment based on rewards received as a result of the actions preformed.
  • the policy neural network can be trained using a reinforcement learning objective function that depends on the rewards received, and the rewards, or values dependent on the rewards, can be scaled or “squashed” using a symmetric logarithmic transformation as described herein.
  • the one innovative aspect of the subject matter described in this specification can be embodied in a method of training a policy neural network having a plurality of policy network parameters used to select actions to be performed by an agent in an environment, the method comprising: receiving a latent representation that is in a latent space and that characterizes a current state of the environment; generating an imagination trajectory of latent representations that starts with the received latent representation by performing the following at each of a plurality of time steps: processing a current latent representation using the policy neural network and in accordance with current values of the policy network parameters to generate an action selection output; selecting, using the action selection output, an action that would be performed by the agent when the environment is in a state characterized by the current latent representation; and processing the current latent representation and the selected action to generate an updated latent representation that characterizes a state that the environment would transition into if the agent performed the selected action when the environment is in a state characterized by the current latent representation; for each latent representation in the imagination DeepMind
  • the symmetric logarithmic transformation may be in the form of: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ or ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , where x is the target state value.
  • the encoded transformed target state value may be represented by a vector of a predetermined length having a predetermined number of non-zero entries.
  • Generating the predicted state value for each latent representation in the imagination trajectory may comprise: processing the latent representation to assign a respective probability to each predetermined interval in a sequence of predetermined intervals between a possible range of state values; generating an output of the value neural network by computing a weighted average of the state values represented by the sequence of predetermined intervals weighted by the respective probabilities; and applying a symmetric exponential transformation to the output of the value neural network to generate the predicted state value for the latent representation in the imagination trajectory.
  • the symmetric exponential transformation may be in the form of: ⁇ where x is the output of the value neural network.
  • Determining the update to the current values of the policy network parameters by optimizing the actor objective function may comprise: generating, by using a scaling factor and from the target state values, a downscaled target state value for each latent representation in the imagination trajectory; and determining the update to the current values of the policy network parameters by optimizing the actor objective function that includes the first term dependent on the downscaled target state values.
  • Generating the downscaled target state value may comprise: dividing the target state value by a maximum of a given integer value and the scaling factor, wherein the scaling factor is computed using respective percentiles of one or more target state values for the latent representations in the imagination trajectory.
  • the actor objective function may include a second term dependent on a scaled entropy of the action selection outputs generated by the policy neural network, wherein the scaled entropy may be scaled by a predetermined entropy scaling factor.
  • Processing the current latent representation and the selected action to generate the updated latent representation may comprise: processing the current latent representation and the selected action using a sequence model neural network in accordance with current values of sequence model network parameters to update a hidden state of the sequence model neural network to generate an updated hidden state that characterizes the current state of the environment; and processing the updated hidden state using a dynamics predictor neural network in accordance with current values of dynamics predictor network parameters to generate the updated latent representation.
  • Receiving the latent representation that characterizes the current state of the environment may comprise: processing an observation that characterizes the current state of the environment using an encoder neural network in accordance with current values of encoder network parameters to generate the latent representation.
  • the method may further comprise: processing the latent representation and the updated hidden state using a decoder neural network in accordance with current values of decoder network parameters to generate a reconstruction of the observation; applying the symmetric logarithmic transformation to the observation to generate a transformed observation; and determining, based on computing a gradient of a prediction objective function that includes a first term measuring a difference between the reconstruction of the observation and the transformed observation and with respect to the decoder network parameters, an update to the current values of the decoder network parameters.
  • determining the predicted reward may comprise: processing the latent representation and an updated hidden state of the sequence model neural network using a reward neural network in accordance with current values of reward network parameters to generate the predicted reward.
  • the method may further comprise: applying the symmetric logarithmic transformation to a reward received by the agent when the environment is at the state characterized by the latent representation to generate a transformed reward; and determining, based on computing a gradient of the prediction objective function that includes a second term measuring a difference between the predicted reward and the transformed reward and with respect to the reward network parameters, an update to the current values of the reward network parameters.
  • the method may further comprise: determining, based on computing a gradient of the dynamics objective function with respect to the sequence model neural network parameters and the dynamics predictor neural network parameters, an update to the current values of the sequence model neural network parameters and the dynamics predictor neural network parameters, wherein the dynamics objective function is dependent on a Kullback–Leibler divergence between (i) a latent representation generated by the encoder neural network and (ii) a latent representation generated by the dynamics predictor neural network.
  • the encoder neural network and the decoder neural network may each be configured as a respective convolutional neural network or multi-layer perceptron (MLP); the dynamics predictor neural network and the reward neural network may each be configured as a multi-layer perceptron (MLP); and the sequence model neural network may be configured as a recurrent neural network.
  • the convolutional neural network, the multi-layer perceptron (MLP), or the recurrent neural network may comprise: one or more layer normalization layers and one or more sigmoid linear unit (SiLU) activation layers.
  • the method may further comprise initializing the current values of the reward network parameters and the value network parameters to zeros.
  • the method may further comprise using the policy neural network to select actions to be performed by the agent in the environment by processing an input derived from data characterizing the current state of the environment at a time step, using the policy neural DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application network, to generate an action selection output used to determine an action to be performed by the agent at the time step.
  • this specification describes a method performed by one or more computers for training an encoder-decoder neural network.
  • the encoder-decoder neural network can comprise an encoder neural network configured to receive an encoder network input and to generate an encoder network output, and a decoder neural network to receive a decoder network input comprising the encoder network output and to generate a decoder network output.
  • the encoder-decoder neural network is configured to receive an input data item (at the encoder network input), process the input data item to generate (an encoder network output comprising) a latent representation of the input data item, and to process the latent representation of the input data item (using the decoder neural network) to generate a reconstruction of the input data item (at the decoder network output).
  • An update to current values of the encoder-decoder network parameters is determined based on evaluating a loss function that measures a difference between the reconstruction of a training data item and a transformed training data item that has been transformed using a symmetric logarithmic transformation.
  • the encoder-decoder neural network is configured to receive an input data item, process the input data item to generate a latent representation of the input data item, and to process the latent representation of the input data item to generate a reconstruction of the input data item, and wherein the method comprises: receiving a training data item; processing the training data item using the encoder-decoder neural network to determine a latent representation of the training data item and to generate a reconstruction of the training data item in accordance with current values of encoder-decoder network parameters; applying a symmetric logarithmic transformation to the training data item to generate
  • the symmetric logarithmic transformation may be in the form of: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ or ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , where x is the training data item.
  • the loss function may be a mean squared error loss function.
  • Generating the reconstruction of the input data item may comprise: processing the training data item to determine the latent representation of the training data item; processing the latent representation of the training data item to generate an output of the encoder-decoder neural network; and applying a symmetric exponential transformation to the output of the encoder-decoder neural network to generate the reconstruction of the training data item.
  • the symmetric exponential transformation may be in the form of: ⁇ ⁇ ⁇ or ⁇ ⁇ ⁇ , where x is the output of the encoder-decoder neural network.
  • the input data item may comprise an image.
  • FIG. 1 A system of one or more computers can be configured to perform particular operations or actions by virtue of software, firmware, hardware, or any combination thereof installed on the system that in operation may cause the system to perform the actions.
  • One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.
  • RL reinforcement learning
  • the described reinforcement learning framework can use a fixed set of hyperparameters that are readily applicable across these diverse domains and is thus easy to setup whenever learning a new task is needed.
  • training DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application an instance of the policy neural network for every new task may thus consume less wall clock time and fewer computational resources, e.g., memory and processing power, than conventional RL systems because hyperparameter tuning by iteratively running multiple training trials is no longer needed.
  • the described reinforcement learning framework is also scalable, meaning that it is capable of training an arbitrarily large policy neural network, where increased model size (e.g., in terms of model parameters) corresponds to both higher task performance and improved data efficiency.
  • the described framework alleviates common issues including divergence, stagnation, and non-stationarity that might otherwise occur to hinder the RL training when some of the existing loss functions and normalization techniques are used, and instead ensures stabilized training of the neural networks across the diverse domains by unifying gradient signals in different orders of magnitude.
  • FIG. 1 shows an example training system.
  • FIG. 2A is an example illustration of training a world model neural network.
  • FIG. 2B is an example illustration of training a policy neural network and a value neural network.
  • FIG. 3 illustrates an example data flow for selecting actions to be performed by an agent interacting with an environment.
  • FIG. 4 is a flow diagram of an example process for training a policy neural network and a value neural network.
  • FIG. 5 is a flow diagram of an example process for generating an imagination trajectory.
  • FIG. 6 is a flow diagram of an example process for training a world model neural network.
  • FIG. 1 shows an example training system.
  • FIG. 2A is an example illustration of training a world model neural network.
  • FIG. 2B is an example illustration of training a policy neural network and a value neural network.
  • FIG. 3 illustrates an example data flow for selecting actions to be performed by an agent interacting with an environment.
  • FIG. 4 is a flow diagram of an example process for training a policy neural network and a
  • FIG. 7 is a flow diagram of an example process for training an encoder-decoder neural network.
  • DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0049]
  • FIG. 8 shows a quantitative example of the performance gains that can be achieved by using the neural networks described in this specification.
  • Like reference numbers and designations in the various drawings indicate like elements.
  • DETAILED DESCRIPTION [0051] This specification describes a training system implemented as computer programs on one or more computers in one or more locations for training a policy neural network.
  • the policy neural network can be used to control a reinforcement learning agent interacting with an environment by, at each of multiple time steps, processing an input derived from data characterizing the current state of the environment at the time step (i.e., an “observation”) to generate an output used to determine an action to be performed by the agent.
  • the training system implements a general reinforcement learning (RL) framework to train the policy neural network by generating multiple imagination trajectories and then training the policy neural network on the multiple imagination trajectories. By doing so, the training system trains the policy neural network so that the policy neural network can be used to control the agent to perform any of multiple tasks.
  • RL general reinforcement learning
  • Each imagination trajectory includes data that characterizes a sequence of predicted state transitions of the environment caused by the actions that would be performed by the agent when performing a task.
  • each imagination trajectory includes multiple latent representations in a latent space that characterize different states of the environment spanning multiple time steps during the imagination trajectory.
  • the multiple latent representations include a latent representation that characterizes an initial state of the environment, a latent representation that characterizes a terminal state of the environment (e.g., the goal state of the environment that is achieved upon the completion of the task), and one or more latent representations that respectively characterize the intermediate states of the environment between the initial state and the terminal state.
  • a “latent representation” refers to a representation of a state of the environment that is more data efficient, e.g., more compact, than an observation of the same state of the environment.
  • the latent representation includes an ordered collection of numerical values, e.g., a tensor (e.g., a vector or matrix) of numerical values, that generally has a pre-determined dimensionality that is lower than that of the observation.
  • a tensor e.g., a vector or matrix
  • the space of possible tensors having the pre-determined dimensionality is referred to as the “latent space.”
  • FIG. 1 shows an example training system 100.
  • the training system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. [0056]
  • the training system 100 includes a policy neural network 120, a value neural network 130, and a world model neural network 140.
  • the policy neural network 120 is a neural network ⁇ ⁇ configured to receive, at a given time step ⁇ during a trajectory, a policy network input that includes a latent representation ⁇ ⁇ that characterizes a current state of the environment at the given time step ⁇ and to process the a policy network input in accordance with a set of parameters ⁇ (referred to in this specification as “policy network parameters”) to generate an action selection output that can be used to select an action ⁇ ⁇ to be performed by the agent at the given time step ⁇ .
  • the latent representation ⁇ ⁇ is later also referred to as an updated latent representation; obtaining such an updated latent representation ⁇ ⁇ for the given time step ⁇ will be described further below.
  • the action selection output can include a respective numerical probability value for each action in a set of possible actions that can be performed by the agent. If being used to select the action, the action to be performed by the agent could be selected, e.g., by sampling an action in accordance with the probability values for the actions, or by selecting the action with the highest probability value.
  • the action selection output can directly define the action to be performed by the agent, e.g., by defining the values of torques that should be applied to the joints of a robotic agent.
  • Such an action can be an action that is predicted to yield the highest long-term time-discounted reward if performed by the agent in response to the observation.
  • the action selection output can include a respective Q value for each action in the set of possible actions that can be performed by the agent. If being used to select the action, the Q values could be processed (e.g., using a softmax function) to generate a respective probability value for each possible action, which can be used to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application select the action to be performed by the agent (as described earlier). The action with the highest Q value could also be selected as the action to be performed by the agent.
  • the Q value for an action is an estimate of a “return” that would result from the agent performing the action in response to a current observation and thereafter selecting future actions performed by the agent in accordance with current values of the policy network parameters.
  • a return refers to a cumulative measure of “rewards” received by the agent, for example, a time-discounted sum of rewards.
  • the agent can receive a respective reward at each time step, where the reward is specified by a scalar numerical value and characterizes, e.g., a progress of the agent towards completing an assigned task.
  • the value neural network 130 is a neural network ⁇ configured to receive, at a given time step ⁇ during the trajectory, a value network input that includes the updated latent representation ⁇ ⁇ that characterizes the current state of the environment at the given time step ⁇ and to process the value network input in accordance with a set of parameters ⁇ (referred to in this specification as “value network parameters”) to generate a predicted state value, i.e., a predicted value of the current state of the environment.
  • the predicted state value represents a time-adjusted sum of predicted rewards that would be received by the agent by selecting actions using action selection outputs of the policy neural network during the trajectory starting from the current state of the environment that corresponds to the updated latent representation ⁇ ⁇ .
  • the world model neural network 140 captures what the training system 100 knows about the environment (the “world”) so that world model neural network 140 can make predictions about the trajectory in a latent space based on what it knows about the past.
  • the world model neural network 140 is configured to simulate state transitions of the environment that would be caused by actions performed by the agent that could be selected using the action selection outputs of the policy neural network 120 at the multiple time points during the trajectory.
  • the world model neural network 140 can include a sequence model neural network 150, an encoder-decoder neural network which, in turn, includes an encoder neural network 160 and a decoder neural network 165, a dynamics predictor neural network 170, a reward neural network 180, and a continuation predictor neural network 190.
  • the world model neural network 140 includes all of the neural network 150, 160, 165, 170, 180, and 190.
  • the world model neural DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application network 140 includes only some of these neural networks, e.g., the decoder neural network 165 and the continuation predictor neural network 190 may be optional components of the world model neural network 140.
  • the sequence model neural network 150 is a neural network ⁇ configured to receive, at a given time step ⁇ during the trajectory, a sequence model network input that includes a current latent representation ⁇ ⁇ that characterizes a previous state of the environment and data specifying a previously selected action ⁇ ⁇ in accordance with a set of parameters (referred to in this specification as “sequence model network parameters”) to update a hidden state ⁇ ⁇ of the sequence model neural network ⁇ , e.g., to modify the current state ⁇ ⁇ that has been generated by processing a further previous latent representation ⁇ ⁇ and the data specifying a further previously selected action ⁇ ⁇ by processing the previous latent representation and the previously selected action ⁇ ⁇ .
  • the subscript ⁇ is used to denote parameters of the world model neural network 140, and parameters of any of the neural networks that this includes.
  • “parameters” refers to trainable parameters of a neural network, such as weights, as distinct from hyperparameters.
  • the hidden state of the sequence model neural network 150 after processing a world model network input will be referred to in this specification as an updated hidden state ⁇ ⁇ that characterizes a current state of the environment.
  • the current state is a state that the environment would transition into if the agent performed the previously selected action ⁇ ⁇ when the environment was in the previous state characterized by the current latent representation ⁇ ⁇ .
  • the updated hidden state ⁇ ⁇ of the sequence model neural network 150 can be a pre-determined initial hidden state.
  • a “hidden state” refers to a representation of a state of the environment that is more data efficient, e.g., more compact, than an observation of the same state of the environment.
  • the hidden state includes an ordered collection of numerical values, e.g., a tensor (e.g., a vector or matrix) of numerical values, that generally has a pre-determined dimensionality that is lower than that of the observation.
  • the encoder neural network 160 included in the encoder-decoder neural network is a neural network ⁇ ⁇ configured to receive, at the given time step ⁇ , an encoder network input that includes the updated hidden state ⁇ ⁇ that characterizes the current state of the environment and a current observation ⁇ ⁇ of the current state of the environment, and to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application process the encoder network input in accordance with a set of parameters ⁇ (referred to in this specification as “encoder network parameters”) to generate the updated latent representation ⁇ ⁇ that characterizes the current state of the environment.
  • encoder network parameters referred to in this specification as “encoder network parameters”
  • each observation of a state of an environment typically includes high- dimensional sensor data, e.g., image or video data, that characterizes the state of the environment
  • the latent representation that characterizes the same state of the environment can be represented as an ordered collection of numerical values, e.g., a vector or matrix of numerical values, that generally has a lower dimensionality than the observation.
  • each observation can be an RGB image that has 64 ⁇ 64 ⁇ 3 pixels
  • each latent representation can be a 32-dimensional array that includes 32 latent variables.
  • the decoder neural network 165 included in the encoder-decoder neural network is a neural network ⁇ ⁇ configured to receive, at the given time step ⁇ , a decoder network input that includes the updated latent representation ⁇ ⁇ that characterizes the current state of the environment and, in some implementations, the updated hidden state ⁇ ⁇ that characterizes the current state of the environment, and to process the decoder network input in accordance with a set of parameters ⁇ (referred to in this specification as “decoder network parameters”) to generate a reconstructed current observation ⁇ ⁇ ⁇ of the current state of the environment.
  • the reconstructed current observation ⁇ ⁇ is a reconstruction of the current observation ⁇ ⁇ that is received by the encoder neural network 160.
  • the dynamics predictor neural network 170 is a neural network ⁇ configured to receive, at the given time step ⁇ , a dynamics predictor network input that includes the updated hidden state ⁇ ⁇ that characterizes the current state of the environment, and to process the dynamics predictor network input in accordance with a set of parameters ⁇ (referred to in this specification as “dynamics predictor network parameters”) to generate an estimated updated latent representation ⁇ ⁇ that characterizes the current state of the environment.
  • the estimated updated latent representation ⁇ ⁇ is an estimate of the updated latent representation ⁇ ⁇ that characterizes the current state of the environment and that can be generated by the encoder neural network 160.
  • the estimated updated latent representation ⁇ ⁇ can have the same dimensionality as the updated latent representation ⁇ ⁇ .
  • the dynamics predictor neural network 170 is configured to generate the estimated updated latent representation ⁇ ⁇ from limited information without relying on the current observation ⁇ ⁇ of DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application the current state of the environment, i.e. in implementations the dynamics predictor network input does not include the observation ⁇ ⁇ .
  • the reward neural network 180 is a neural network ⁇ configured to receive, at the given time step ⁇ , a reward network input that includes the updated latent representation ⁇ ⁇ that characterizes the current state of the environment and, in some implementations, the updated hidden state ⁇ ⁇ that characterizes the current state of the environment, and to process reward network input in accordance with a set of parameters ⁇ (referred to in this specification as “reward network parameters”) to generate a predicted reward ⁇ ⁇ ⁇ .
  • the predicted reward ⁇ ⁇ ⁇ is a prediction of a reward ⁇ ⁇ ⁇ that would be received by the agent when the environment is in the current state.
  • a neural network denoted ⁇ can refer to any neural network that makes a prediction, e.g.
  • the continuation predictor neural network 190 is a neural network ⁇ configured to receive, at the given time step ⁇ , a continuation predictor network input that includes the updated latent representation ⁇ ⁇ that characterizes the current state of the environment and, in some implementations, the updated hidden state ⁇ ⁇ that characterizes the current state of the environment, and to process the continuation predictor network input in accordance with a set of parameters ⁇ (referred to in this specification as “continuation predictor network parameters”) to generate a predicted trajectory continuation flag ⁇ ⁇ that indicates whether the current state is a terminal state in the trajectory, that is, whether another updated latent representation should be generated in order to characterize a future state of the environment.
  • the predicted trajectory continuation flag ⁇ ⁇ ⁇ includes a binary value, where 1 define that the current state is a terminal state in the trajectory (such that no more updated latent representations should be generated), 0 defines the current state is not a terminal state in the trajectory (such that another updated latent representation should be generated), or vice versa.
  • a trajectory might terminate upon completion of a task by the agent, or after a threshold number of time steps, or the like.
  • Application continuation predictor neural network 190 can each be implemented with any appropriate neural network architecture that enables them to perform their described functions.
  • the policy neural network 120 and the value neural network 130 can include any appropriate number of layers (e.g., 5 layers, 10 layers, or 25 layers) of any appropriate type (e.g., fully connected layers, convolutional layers, recurrent layers, attention layers, etc.) and connected in any appropriate configuration (e.g., as a linear sequence of layers, with or without residual connections).
  • the policy neural network 120 can be a large- scale neural network, e.g., a neural network that includes a large number of layers (e.g., 50 layers, 100 layers, or more) and, correspondingly, has a large number of model parameters.
  • a larger number of model parameters corresponds to both higher task performance and improved data efficiency.
  • a larger scale policy neural network that has 18 million parameters will generally have a higher performance, e.g., in terms of the returns received by an agent controlled using the neural network, than a smaller scale policy neural network that has 8 million parameters after the same number of training iterations. That is, the number of neural layers of the neural networks, and the number of trainable parameters of the system, can be scaled according to a complexity of a task to be learned.
  • the value neural network 130 is configured to generate the predicted state values in a discrete regression approach.
  • the value neural network 130 first generates, e.g., using a softmax layer of the value neural network, a probability distribution ⁇ ⁇ ( ⁇ ⁇ ⁇ ⁇ ⁇ over a sequence ⁇ of ⁇ predetermined intervals a possible range of state value, where ⁇ can be any positive integer (e.g., 63, 127, 255, and so on). That is, the probability distribution assigns a respective probability score to each predetermined interval ⁇ ⁇ in the sequence ⁇ of ⁇ predetermined intervals.
  • the value neural network 130 then generates the predicted state value by computing an average of the state values represented by the sequence of predetermined intervals. The average is computed as a weighted average that is weighted by the respective probability scores.
  • the sequence model neural network 150 is configured as a recurrent neural network that includes one or more recurrent layers, e.g., long short-term memory (LSTM) layers or gated recurrent unit (GRU) layers.
  • LSTM long short-term memory
  • GRU gated recurrent unit
  • the hidden state can, for example, be the hidden state of the last recurrent DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application layer in a stack of multiple recurrent layers included in the sequence model neural network 150.
  • LSTM long short-term memory
  • GRU gated recurrent unit
  • the encoder neural network 160 and the decoder neural network 165 can each be configured as a respective convolutional neural network (CNN) that includes one or more convolutional layers. In some other implementations, they are each configured as a respective multi-layer perceptron (MLP) that includes one or more fully connected layers. In some implementations one or more CNNs can be included to process one or more visual observations and one or more MLPs can be included to process one or more lower dimensional observations.
  • CNN convolutional neural network
  • MLP multi-layer perceptron
  • the dynamics predictor neural network 170, the reward neural network 180, and the continuation predictor neural network 190 are each configured as a respective multi-layer perceptron (MLP) that includes one or more fully connected layers.
  • MLP multi-layer perceptron
  • the sequence model neural network 150, the encoder neural network 160, the decoder neural network 165, the dynamics predictor neural network 170, the reward neural network 180, and the continuation predictor neural network 190 can each include one or more normalization layers, e.g. one or more layer normalization layers and one or more activation layers, e.g. one or more sigmoid linear unit (SiLU) activation layers.
  • SiLU sigmoid linear unit
  • the outputs mentioned above can be generated in any of a variety of ways.
  • the updated hidden state ⁇ ⁇ can be generated deterministically by, e.g., a last recurrent layer of, the sequence model neural network: [0085]
  • each of one or more of the updated latent representation ⁇ ⁇ , the estimated updated latent representation ⁇ ⁇ , the reconstructed current observation ⁇ ⁇ , the predicted reward ⁇ ⁇ ⁇ , or the trajectory continuation flag ⁇ ⁇ can be generated stochastically.
  • the training system 100 includes or has access to a replay buffer 104 that stores historical trajectories that can be generated as a consequence of the interaction of the agent (or another agent) with the environment (or with another instance of the environment) to attempt to perform one or more tasks.
  • the replay buffer 104 can be any available memory, e.g., one or more physical or logical storage devices.
  • each historical trajectory can include a tuple at each of multiple time points between ⁇ ⁇ ⁇ to ⁇ ⁇ ⁇ during the historical trajectory.
  • the tuple includes (i) a current observation ⁇ ⁇ characterizing a current state of the environment, (ii) a current action ⁇ ⁇ performed by the agent, (iii) a current reward ⁇ ⁇ received by the agent, and (iv) a ground truth continuation flag ⁇ ⁇ that indicates whether the current state of the environment is a terminal state of the environment (e.g., the goal state of the environment that is achieved upon the completion of a task).
  • a training engine 110 of the training system 100 repeatedly selects historical trajectories from the replay buffer 140, and trains the world model neural network 140 based on the selected historical trajectories using suitable world model learning techniques as described below.
  • the training engine 110 uses the (trained) world model neural network 140 to train the policy neural network 120 and the value neural network 130. This is done by using the (trained) world model neural network 140 to generate multiple imagination trajectories, and then jointly training the policy neural network 120 and the value neural network 130 based on the multiple imagination trajectories using suitable reinforcement learning techniques.
  • Training the policy neural network 120 and the value neural network 130 based on the multiple imagination trajectories means that the policy neural network 120 and the value neural network 130 are each configured to either receive as inputs the latent representations during the training (e.g., rather than directly processing the actual observations, e.g., sensor data, e.g., image or video data, that characterize the environment states).
  • Implementations of the training system 100 include a symlog transformation engine 112 that applies a symmetric logarithmic transformation to an input ⁇ .
  • a symlog transformation engine 112 that applies a symmetric logarithmic transformation to an input ⁇ .
  • Application symmetric logarithmic transformation can be characterized as a logarithmic transformation, i.e.
  • the logarithm can have any base.
  • the symmetric logarithmic transformation can be in the form: ⁇ ⁇ ⁇ ⁇ where x is the input or argument.
  • the symmetric logarithmic transformation can use a log to base 10 (or another base) and can, e.g.
  • FIG. 2A is an example illustration of training a world model neural network.
  • the world model neural network can correspond to the world model neural network 140 of FIG.
  • the training engine 110 performs the operations illustrated in FIG. 2A on each of multiple historical trajectories selected from the replay buffer 104 to train these neural networks using suitable world model learning techniques to enable these neural networks to more accurately simulate the effects on the environment of the actions performed by the agent when interacting with the environment.
  • the sequence model neural network receives a sequence model network input that includes a current latent representation ⁇ ⁇ that characterizes a previous state of the environment and data specifying a previously selected action ⁇ ⁇ , and processes the sequence model network input to generate an updated DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application hidden state ⁇ ⁇ that characterizes a current state of the environment.
  • the current state of the environment is a state that the environment would transition into if the agent performed the previously selected action when the environment was in the previous state characterized by the current latent representation.
  • the sequence model neural network generates an updated hidden state ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ that characterizes the current state of the environment.
  • the current state of the environment at time step ⁇ ⁇ ⁇ is a state that the environment would transition into if the agent performed the previously selected action when the environment was in the previous state characterized by the current latent representation ⁇ ⁇ .
  • the encoder neural network (“enc”) receives an encoder network input that includes an updated hidden state ⁇ ⁇ that characterizes a current state of the environment and a current observation ⁇ ⁇ of the current state of the environment, and processes the encoder network input to generate an updated latent representation ⁇ ⁇ ⁇ that characterizes the current state of the environment.
  • the encoder neural network (“enc”) generates an updated latent representation ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ that characterizes the current state of the environment.
  • the current state of the environment at time step ⁇ ⁇ ⁇ is a state that the environment would transition into if the agent performed the previously selected action when the environment is in the previous state characterized by the current latent representation ⁇ ⁇ .
  • the decoder neural network (“dec”) receives a decoder network input that includes the updated latent representation ⁇ ⁇ that characterizes the current state of the environment and, in some implementations, the updated hidden state ⁇ ⁇ that characterizes the current state of the environment, and processes the decoder network input to generate a reconstructed current observation ⁇ ⁇ ⁇ of the current state of the environment.
  • the reconstructed current observation ⁇ ⁇ is a reconstruction of the current observation ⁇ ⁇ that is received by the encoder neural network 160.
  • the decoder neural network (“dec”) generates a reconstructed current observation ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ .
  • Application observation ⁇ ⁇ is a reconstruction of the current observation ⁇ ⁇ that is received by the encoder neural network (“enc”).
  • the dynamics predictor neural network receives a dynamics predictor network input that includes an updated hidden state ⁇ ⁇ that characterizes the current state of the environment, and processes the dynamics predictor network input to generate an estimated updated latent representation that characterizes the current state of the environment.
  • the estimated updated latent representation ⁇ ⁇ is an estimate of the updated latent representation ⁇ ⁇ that characterizes the current state of the environment and that can be generated by the encoder neural network 160.
  • the dynamics predictor neural network generates an estimated updated latent representation ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ that characterizes the current state of the environment.
  • the estimated updated latent representation ⁇ ⁇ is an estimate of the updated latent representation ⁇ ⁇ that characterizes the current state of the environment at time step ⁇ ⁇ ⁇ and that can be generated by the encoder neural network 160.
  • the reward neural network receives a reward network input that includes the updated latent representation ⁇ ⁇ that characterizes the current state of the environment and, in some implementations, the updated hidden state ⁇ ⁇ that characterizes the current state of the environment, and processes the decoder network input to generate a predicted reward ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ .
  • the predicted reward ⁇ ⁇ ⁇ is a prediction of a reward ⁇ ⁇ ⁇ that would be received by the agent when the environment is in the current state.
  • the reward neural network 180 At time step ⁇ ⁇ ⁇ ⁇ , the reward neural network 180 generates a predicted reward ⁇ ⁇ ⁇ that is a prediction of a reward ⁇ ⁇ ⁇ that would be received by the agent when the environment is in the current state at time step ⁇ ⁇ ⁇ .
  • the continuation predictor neural network receives a continuation predictor network input that includes the updated latent representation ⁇ ⁇ that characterizes the current state of the environment and, in some implementations, the updated hidden state ⁇ ⁇ that characterizes the current state of the environment, and processes the continuation predictor network input to generate a predicted trajectory continuation flag ⁇ ⁇ ⁇ that indicates whether the current state is a terminal state in the trajectory.
  • the continuation predictor neural network generates a predicted trajectory continuation flag ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ that indicates that the current state is not a terminal state in the historical trajectory.
  • the training system proceeds to use the sequence model neural network to generate an updated hidden state ⁇ ⁇ that characterizes a future state of the environment.
  • the future state of the environment is a state that the environment would transition into if the agent performed the selected action ⁇ ⁇ when the environment is in the current state characterized by the current latent representation ⁇ ⁇ .
  • the training engine 110 updates the values of the parameters ⁇ of the dynamics predictor neural network, the reward neural network, and the continuation predictor neural network included in the world model neural network based on optimizing a prediction objective function ⁇ ⁇ ⁇ .
  • the prediction objective function can include a first term that measures, for each of multiple time steps ⁇ during the given trajectory, a difference, e.g. a squared difference, between the (i) reconstructed current observation ⁇ ⁇ and (ii) a transformed current observation that is generated by the symlog transformation engine 112 included in the training engine 110 from the current observation ⁇ ⁇ ⁇ included in the historical trajectory.
  • the symlog transformation engine 112 applies a symmetric logarithmic transformation to the current observation ⁇ ⁇ .
  • the symmetric logarithmic transformation can be in the form of: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ or ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ where ⁇ ⁇ is the current observation.
  • the symmetric logarithmic transformation can be applied to any type of observation including, e.g., a low-dimensional inputs. Where applied to a vector or matrix it can be applied to each element of the vector or matrix.
  • the first term trains the decoder neural network to generate reconstructed current observations that are proximate to the current observations included in the historical trajectories, and, by virtue of backpropagation, trains the encoder neural network to generate semantically meaningful updated latent representations and the sequence model neural network to generate semantically meaningful updated hidden states.
  • DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0109]
  • the prediction objective function ⁇ ⁇ ⁇ can include a second term that measures, for each of multiple time steps ⁇ during the given trajectory, a difference, e.g.
  • the second term trains the reward neural network to generate more accurate predicted rewards, i.e., to generate predicted rewards that are proximate to the rewards included in the historical trajectory, and, by virtue of backpropagation, trains the encoder neural network to generate semantically meaningful updated latent representations and the sequence model neural network to generate semantically meaningful updated hidden states.
  • the prediction objective function ⁇ ⁇ ⁇ can include a third term that measures, for each of multiple time steps ⁇ during the given trajectory, a difference between the predicted trajectory continuation flag ⁇ ⁇ ⁇ and the ground truth trajectory continuation flag.
  • the third term trains the continuation predictor neural network to generate more accurate predicted trajectory continuation, i.e., to generate predicted trajectory continuation flags that are proximate to the trajectory continuation flags included in the historical trajectory, and, by virtue of backpropagation, trains the encoder neural network to generate semantically meaningful updated latent representations and the sequence model neural network to generate semantically meaningful updated hidden states.
  • the prediction objective function can be expressed as: where in practice the terms are differences that can each be computed as a cross entropy loss (a binary classification loss for the continuation term).
  • the training engine 110 also updates the values of the parameters ⁇ of the dynamics predictor neural network included in the world model neural network 140 based on optimizing a dynamics objective function ⁇ ⁇ ⁇ .
  • the dynamics objective function ⁇ ⁇ ⁇ can include a term that is dependent on a difference that measures, for each of multiple time steps ⁇ during the given trajectory, a difference between (i) an updated latent representation ⁇ ⁇ generated by the encoder neural network and (ii) an estimated updated latent representation ⁇ ⁇ generated by the dynamics predictor neural network.
  • the term trains the dynamics predictor neural network to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application generate estimated updated latent representations that are proximate to the updated latent representations generated by the encoder neural network.
  • the difference can be determined as a Kullback–Leibler (KL) divergence
  • the dynamics objective function ⁇ ⁇ ⁇ can be: where sg represents a stop gradient operator applied to the parameters of the encoder neural network (i.e. during training gradients are not backpropagated into the encoder neural network), and where ⁇ refers to taking the maximum value of the arguments inside the bracket. Taking the maximum effectively disables the effect of this loss once it is reduced below a minimum level (1 nat), allowing the training to focus on the other losses, e.g. the prediction loss.
  • the training engine 110 updates the values of the parameters ⁇ of the encoder neural network included in the world model neural network based on optimizing a representation objective function ⁇ ⁇ ⁇ .
  • the representation objective function ⁇ ⁇ ⁇ ) can include a term that is dependent on a difference that measures, for each of multiple time steps ⁇ during the given trajectory, a difference between (i) an updated latent representation ⁇ ⁇ generated by the encoder neural network and (ii) an estimated updated latent representation ⁇ ⁇ generated by the dynamics predictor neural network.
  • the term trains the encoder neural network to generate updated latent representations that are proximate to the estimated updated latent representations generated by the dynamics predictor neural network
  • the difference can be determined as a Kullback–Leibler (KL) divergence
  • the representation objective function ⁇ ⁇ ⁇ can be: where sg represents a stop gradient operator applied to the parameters of the dynamics predictor neural network.
  • the training engine 110 end-to-end trains the world model neural network, i.e. the complete world model neural network 140 is trained by optimizing a combined objective function that combines the prediction objective function ⁇ ⁇ ⁇ , the dynamics objective function ⁇ ⁇ ⁇ , and the representation objective function ⁇ ⁇ ⁇ mentioned above.
  • the combined objective function can be: DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application
  • ⁇ ⁇ , ⁇ ⁇ ⁇ and ⁇ ⁇ are respective weights placed on the prediction objective function ⁇ ⁇ ⁇ , the dynamics objective function ⁇ ⁇ ⁇ , and the representation objective function ⁇ ⁇ ⁇ .
  • ⁇ ⁇ , ⁇ ⁇ ⁇ and ⁇ ⁇ are hyperparameters of the training system 100 (some illustrative hyperparameter values are given later).
  • FIG. 2B is an example illustration of training a policy neural network and a value neural network.
  • the policy neural network and the value neural network can correspond to the policy neural network 120 and the value neural network 130, respectively, of FIG. 1.
  • the training engine 110 trains the policy neural network 120 and the value neural network 130 on the imagination trajectories generated using the world model neural network 140.
  • the training engine 110 performs the operations illustrated in FIG. 2B on each of multiple imagination trajectories by using the world model neural network together with the policy neural network and the value neural network using suitable reinforcement learning techniques to enable the policy neural network to generate action selection outputs that can be used to select actions that cause the agent to effectively perform one or more tasks.
  • each imagination trajectory can include a tuple at each of multiple time points between ⁇ ⁇ ⁇ to ⁇ ⁇ ⁇ during the imagination trajectory.
  • the tuple includes (i) an updated latent representation ⁇ ⁇ characterizing a current state of the environment, (ii) an updated hidden state ⁇ ⁇ characterizing the current state of the environment, (iii) a current action ⁇ ⁇ selected by an action selection output, (iii) a predicted current reward ⁇ ⁇ , and (iv) a predicted continuation flag ⁇ ⁇ that indicates whether the current state of the environment is a terminal state of the environment (e.g., the goal state of the environment that is achieved upon the completion of a task).
  • an imagination trajectory includes an updated latent representation ⁇ ⁇ at each of multiple time steps between ⁇ ⁇ ⁇ to ⁇ ⁇ ⁇ during the imagination trajectory.
  • the updated latent representation ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ is generated by the encoder neural network based on a pre-determined initial hidden state of the sequence model neural network and on the current observation ⁇ ⁇ included in a historical trajectory which characterizes the initial state of the environment.
  • the updated latent representation is the estimated updated latent representation generated by dynamics predictor neural network based on the updated hidden state ⁇ ⁇ of the sequence model neural network.
  • the imagination trajectories thus differ from the historical trajectories stored in the replay buffer 104 in that each imagination trajectory includes at least one updated latent representation that is not generated based on the observation, e.g., sensor data, e.g., image or video data, of the environment state. Rather, the imagination trajectory includes updated latent representations that are generated based on the hidden states of the sequence model neural network.
  • the training engine 110 trains the policy neural network and the value neural network using an ⁇ actor ⁇ critic technique, where the policy neural network corresponds to an actor neural network, and the value neural network corresponds to a critic neural network.
  • the policy neural network receives receive a policy network input that includes an updated latent representation ⁇ ⁇ that characterizes a current state of the environment at the time step ⁇ and processes the policy network input to generate an action selection output ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ that can be used to select an action ⁇ ⁇ to be performed by the agent at the time step ⁇ , where ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ , and can represent an imagined model state.
  • the policy neural network At time step ⁇ ⁇ ⁇ ⁇ , the policy neural network generates an action selection output ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ that can be used to select an action ⁇ ⁇ at the time step ⁇ ⁇ ⁇ [0127]
  • the value neural network receives a value network input that includes the updated latent representation ⁇ ⁇ that characterizes the current state of the environment at the time step ⁇ and processes the value network input to generate a predicted state value ⁇ ⁇ ( ⁇ ⁇ ).
  • the value neural network At time step ⁇ ⁇ ⁇ , the value neural network generates a predicted value ⁇ ⁇ ( ⁇ ⁇ ⁇ (referred to as “ ⁇ ⁇ ” in FIG.
  • the training engine 110 can use the symlog transformation engine 112 to apply a symmetric exponential transformation to the output of the value neural network to generate the predicted state values.
  • symmetric exponential transformation is the inverse of the previously described symmetric logarithmic transformation.
  • the symmetric exponential transformation, ⁇ ⁇ ⁇ ⁇ can be in the form of: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ or ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ where x is the output of the value neural network.
  • the predicted state value for each of multiple time steps ⁇ during the given imagination trajectory can be computed as: [0130]
  • the training engine 110 determines a target state value for each time step ⁇ , and updates the values of the parameters ⁇ of the policy neural network based on optimizing an actor objective function.
  • the actor objective function includes a first term dependent on the target state values.
  • the actor objective function trains the policy neural network to generate action selection outputs that can facilitate the selection of more effective actions that, in turn, can result in higher predicted state values to be computed by the value neural network and higher predicted rewards to be computed by the reward neural network.
  • the target state value for each time step ⁇ can be determined based on predicted rewards generated by the reward neural network, the predicted state values generated by using the value neural network, or both the predicted rewards ⁇ ⁇ ⁇ and the predicted state values
  • the target state value can be determined by computing the bootstrapped ⁇ -returns that integrate both the predicted rewards and the predicted state values: , , where ⁇ a discount factor, ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ is a trace decay parameter for which larger values give more credit to more distant (in time) rewards, and ⁇ ⁇ is an imagined model state after a trajectory of T time steps.
  • ⁇ and ⁇ are hyperparameters of the training system 100. In some other approaches, n-step returns can be used.
  • the first term included in the actor objective function is dependent on downscaled target state values to better accommodate for sparse rewards.
  • the training engine 110 scales down larger target state values without scaling up smaller target state values. This is done by dividing each target state value by a maximum of (i) a given integer value (e.g., one, two, ten, and so on), and (ii) a scaling factor ⁇ . DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0133]
  • the scaling factor ⁇ can be computed using respective quantiles, e.g.
  • the scaling factor ⁇ can be computed as an exponentially decaying average of the range from their 5th to their 95th batch percentile: .
  • the actor objective function includes a second, entropy regularization term that is dependent on an entropy ⁇ of the action selection outputs ⁇ ⁇ ⁇ generated by the policy neural network.
  • the entropy ⁇ can be a scaled entropy that is scaled by a predetermined entropy scaling factor ⁇ .
  • the actor objective function can be: where sg represents a stop gradient operator applied to the parameters of the world model neural network.
  • the maximum value operator implements the scaling only if the returns exceed a minimum threshold, e.g.1, which facilitates using a single entropy scale (defined by a parameter ⁇ ) across dense and sparse rewards.
  • the training engine 110 updates the values of the parameters of the value neural network based on optimizing a critic objective function.
  • the critic objective function includes a term that measures a difference between (i) the predicted state values ⁇ ⁇ ( ⁇ ⁇ ⁇ and (ii) encoded transformed target state values that are generated by the symlog transformation engine 112 from the target state values.
  • the critic objective function trains the value neural network to generate more accurate predicted state values, i.e., to generate predicted state values that more accurately represent the time- adjusted sums of predicted rewards that would be received by the agent during the given imagination trajectory.
  • the training engine 110 uses the symlog transformation engine 112 to apply, for each time step in the imagination trajectory, a symmetric logarithmic transformation to the target state value to generate a transformed target state value for the time step in the imagination trajectory, and encodes the transformed target state value using a suitable encoding technique to generate an encoded transformed target state value for the time step in the imagination trajectory.
  • the symmetric logarithmic transformation can for example be in the form of: ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ or ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ where x is the target state value.
  • the encoded transformed target state value can be represented by a vector of a predetermined length that has a predetermined number of non-zero entries.
  • the training engine can use a two-hot encoding technique.
  • the encoded transformed target state value is represented by a vector of length ⁇ ⁇ ⁇ where all elements are zero expect for the two entries closest to the encoded continuous number, at positions ⁇ and ⁇ ⁇ ⁇ .
  • the critic objective function can be: , where sg represents a stop gradient operator applied to the parameters of the world model neural network, and the difference between the predicted state values ⁇ ⁇ ( ⁇ ⁇ ⁇ and the encoded transformed target state values is computed as a (categorical) cross entropy loss.
  • the training system 100 maintains, e.g., in one or more physical or logical storage devices accessible by the training system, predetermined values for a set of hyperparameters 118.
  • Hyperparameters are parameters of a neural network that are set before the neural network is trained. Hyperparameters might have an impact on the quality of the training of neural networks, and setting optimal values for the hyperparameters may thus be an important part of the training.
  • these predetermined values can be adjusted, e.g., through an automated search process or a manual hyperparameter fine tuning, prior to the commencement of the training.
  • these predetermined DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application values can be fixed, and the training system 100 repeatedly applies the set of hyperparameters having the same predetermined values across different tasks on which the neural networks will be trained.
  • the predetermined values for the set of hyperparameters 118 maintained by the training system 100 reduces the need for hyperparameter fine tuning, which can be time consuming and computationally expensive, and enables it to more efficiently train the neural networks across a range of different task domains.
  • Examples of the set of hyperparameters and their predetermined values are listed below in Table 1.
  • “number of latents” refers to the dimensionality of each latent representation (it is a 32-dimensional array that specifies a probability distribution of each of a set of 32 possible latent variables)
  • horizontal refers to the number of time steps included in each imagination trajectory
  • EMA is the acronym for exponential moving average.
  • the training engine 110 can determine trained values of the parameters 114 of the neural networks included in the training system 100, i.e., the parameters of the policy neural network 120, the value neural network 130, and the world model neural network 140 which, in turn, can include the sequence model neural network 150, the encoder neural network 160, the decoder neural network 165, the dynamics predictor neural network 170, the reward neural network 180, and the continuation predictor neural network 190.
  • the neural networks included in the training system 100 can be deployed and used to control an agent to perform one or more tasks by interacting with an environment.
  • the sequence model encoder neural network 150 and the encoder neural network 160 can operate in tandem to generate an updated DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application latent representation for each new observation.
  • the sequence model encoder neural network 150 can update a hidden state of the sequence model neural network to generate an updated hidden state, ⁇ ⁇ , that characterizes the current state of the environment.
  • the encoder neural network 160 can process an observation ⁇ ⁇ that characterizes the current state of the environment to generate the latent representation ⁇ ⁇ .
  • the policy neural network 120 can then process the latent representation ⁇ ⁇ for the observation ⁇ ⁇ to generate an action selection output that is used to select an action to be performed by the agent in response to the new observation.
  • the training system 100 can provide, e.g., by a wired or wireless connection, the trained values of some or all of the network parameters 114 to an external system that can cause the agent to perform different actions, e.g., by directly transmitting control signals to the agent or by transmitting data identifying the actions to a control system for the agent.
  • FIG. 3 illustrates an example data flow for using a policy neural network 320, a sequence model neural network 350, and an encoder neural network 360 to select actions 308 to be performed by an agent 302 interacting with an environment 304 at each of multiple time steps.
  • the policy neural network 320, the sequence model neural network 350, and the encoder neural network 360 can correspond to the policy neural network 120, the sequence model encoder neural network 150, and the encoder neural network 160, respectively, of FIG. 1.
  • the trained values of the parameters 114 can be determined (and outputted) by the training system 100 of FIG. 1.
  • the sequence model neural network 350 processes a sequence model network input that includes the current observation 306 in accordance with the trained values of the parameters of the sequence model neural network to generate an updated hidden state 352 that characterizes the current state of the environment 304.
  • the encoder neural network 360 processes an encoder network input that includes the updated hidden state 352 and the current observation 306 in accordance with the trained values of the parameters of the encoder neural network to generate an updated latent representation 362 that characterizes the current state of the environment 304.
  • the policy neural network 320 processes a policy network input that includes the updated latent representation 362 in accordance with the trained values of the parameters of the policy neural network to generate an action selection output 322 that can be used to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application select an action 308 to be performed by the agent 302 in response to the current observation 306. Examples of the action selection output and how it can be used to select the action are described above with reference to FIG. 1.
  • each received observation has a one-to-one correspondence with an updated hidden state or an updated latent representation. Note that this is different from training through latent imagination where each observation typically has a one-to-many correspondence with the updated hidden stated or the updated latent representations, i.e., during training a sequence of multiple updated hidden stated or updated latent representations are generated from a single observation.
  • the environment is a real-world environment
  • the agent is a mechanical (or electro-mechanical) agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment
  • the actions are actions taken by the mechanical agent in the real-world environment to perform the task.
  • the agent may be a robot interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment.
  • the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator.
  • the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot.
  • the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent.
  • the observations may be defined in 1, 2 or 3 dimensions, and may be absolute and/or relative observations.
  • the observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and/or image or video data for example captured by a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.
  • the actions may be control signals to control the robot or other mechanical agent, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements e.g. steering control elements of the vehicle, or higher-level control commands.
  • the control signals can include for example, position, velocity, or force/torque/acceleration data for one or more joints of a robot or parts of another mechanical agent.
  • the control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment.
  • electronic control data such as motor control data
  • the control signals may define actions to control navigation e.g. steering, and movement e.g., braking and/or acceleration of the vehicle.
  • the environment is a simulation of the above-described real-world environment, and the agent is implemented as one or more computers interacting with the simulated environment.
  • the simulated environment may be a simulation of a robot or vehicle and the reinforcement learning system may be trained on the simulation and then, once trained, used in the real-world.
  • the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product.
  • a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material e.g. to remove pollutants, to generate a cleaned or recycled product.
  • the manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e.g. robots, for processing solid or other materials.
  • the manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g. via pipes or mechanical conveyance.
  • manufacture of a product also includes manufacture of a food product by a kitchen robot.
  • the agent may comprise an electronic agent configured to control a manufacturing unit, or a machine such as a robot, that operates to manufacture the product. That is, the agent may comprise a control system configured to control the manufacture of the chemical, biological, or mechanical product.
  • the control system may be configured to control one or more of the manufacturing units or machines or to control DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application movement of an intermediate version or component of the product between the manufacturing units or machines.
  • a task performed by the agent may comprise a task to manufacture the product or an intermediate version or component thereof.
  • a task performed by the agent may comprise a task to control, e.g. minimize, use of a resource such as a task to control electrical power consumption, or water consumption, or the consumption of any material or consumable used in the manufacturing process.
  • the actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product, or an intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment e.g. between the manufacturing units or machines.
  • the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below.
  • the reinforcement learning situation includes a reward calculation unit for generating a reward (e.g. in the form of a number), typically from the observation.
  • the rewards may relate to a metric of performance of the task.
  • the metric may comprise a metric of a quantity of the product that is manufactured, a quality of the product, a speed of production of the product, or to a physical cost of performing the manufacturing task, e.g. a metric of a quantity of energy, materials, or other resources, used to perform the task.
  • the metric may comprise any metric of usage of the resource.
  • the reward may indicate whether the object has been correctly manipulated according to a predefined criterion.
  • observations of a state of the environment may comprise any electronic signals representing the functioning of electronic and/or mechanical items of equipment.
  • a representation of the state of the environment may be derived from DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT
  • sensors sensing a state of the manufacturing environment e.g. sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines.
  • sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow/movement rate or flow/movement path of one or more materials; physical or chemical conditions e.g.
  • the agent is a machine such as a robot
  • the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g. data characterizing the current state of the machine or robot or of an item held or processed by the machine or robot.
  • the observations may also include, for example, sensed electronic signals such as motor current or a temperature signal, or image or video data for example from a camera or a LIDAR sensor (e.g. mounted on the machine). Sensors such as these may be part of or located separately from the agent in the environment.
  • the environment is the real-world environment of a service facility comprising a plurality of items of electronic equipment, such as a server farm or data center, for example a telecommunications data center, or a computer data center for storing or processing data, or any service facility.
  • the service facility may also include ancillary control equipment that controls an operating environment of the items of equipment, for example environmental control equipment such as temperature control e.g. cooling equipment, or air flow control or air conditioning equipment.
  • the task may comprise a task to control, e.g. minimize, use of a resource, such as a task to control electrical power consumption, or water consumption.
  • the agent may comprise an electronic agent configured to control operation of the items of equipment, or to control operation of the ancillary, e.g. environmental, control equipment.
  • the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below. These may include actions to control, or to impose operating conditions on, the items of equipment or the ancillary control equipment, e.g.
  • the observations of a state of the environment may comprise any electronic signals representing the functioning of the facility or of equipment in the facility.
  • a representation of the state of the environment may be derived from observations made by any sensors sensing a state of a physical environment of the facility or observations made by any sensors sensing a state of one or more of items of equipment or one or more items of ancillary control equipment.
  • the rewards may relate to a metric of performance of a task relating to the efficient operation of the facility. For example in the case of a task to control, e.g. minimize, use of a resource, such as a task to control use of electrical power or water, the metric may comprise any metric of use of the resource.
  • the environment is the real-world environment of a power generation facility e.g. a renewable power generation facility such as a solar farm or wind farm.
  • the task may comprise a control task to control power generated by the facility, e.g. to control the delivery of electrical power to a power distribution grid, e.g. to meet demand or to reduce the risk of a mismatch between elements of the grid, or to maximize power generated by the facility.
  • the agent may comprise an electronic agent configured to control the generation of electrical power by the facility or the coupling of generated electrical power into the grid.
  • the actions may comprise actions to control an electrical or mechanical configuration of an electrical power generator such as the electrical or mechanical configuration of one or more renewable power generating elements e.g. to control a configuration of a wind turbine or of a solar panel or panels or mirror, or the electrical or mechanical configuration of a rotating electrical power generation machine.
  • Mechanical control actions may, for example, comprise actions that control the conversion of an energy input to an electrical energy output, e.g. an efficiency of the conversion or a degree of coupling of the energy input to the electrical energy output.
  • Electrical control actions may, for example, comprise actions that control one or more of a voltage, current, frequency or phase of electrical power generated.
  • the rewards may relate to a metric of performance of a task relating to power distribution.
  • the metric may relate to a measure of power transferred, or to a measure of an electrical mismatch between the power generation facility and the grid such as a voltage, current, frequency or phase mismatch, or to a measure of electrical power or energy loss in the power generation facility.
  • the metric may relate to a measure of electrical power or energy transferred to the grid, or to a measure of electrical power or energy loss in the power generation facility.
  • observations of a state of the environment may comprise any electronic signals representing the electrical or mechanical functioning of power generation equipment in the power generation facility.
  • a representation of the state of the environment may be derived from observations made by any sensors sensing a physical or electrical state of equipment in the power generation facility that is generating electrical power, or the physical environment of such equipment, or a condition of ancillary equipment supporting power generation equipment.
  • Such observations may thus include observations of wind levels or solar irradiance, or of local time, date, or season.
  • sensors may include sensors configured to sense electrical conditions of the equipment such as current, voltage, power or energy; temperature or cooling of the physical environment; fluid flow; or a physical configuration of the equipment; and observations of an electrical condition of the grid e.g. from local or remote sensors.
  • Observations of a state of the environment may also comprise one or more predictions regarding future conditions of operation of the power generation equipment such as predictions of future wind levels or solar irradiance or predictions of a future electrical condition of the grid.
  • the environment may be a chemical synthesis or protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical.
  • the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals/intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical.
  • the agent may be a mechanical agent that indirectly performs or controls the protein folding actions, or chemical synthesis steps, e.g. by controlling synthesis steps selected by the system automatically without human interaction.
  • the observations may comprise direct or DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application indirect observations of a state of the protein or chemical/ intermediates/precursors and/or may be derived from simulation.
  • the system may be used to automatically synthesize a protein with a particular function such as having a binding site shape, e.g. a ligand that binds with sufficient affinity for a biological effect that it can be used as a drug.
  • a ligand that binds with sufficient affinity for a biological effect that it can be used as a drug.
  • it may be an agonist or antagonist of a receptor or enzyme; or it may be an antibody configured to bind to an antibody target such as a virus coat protein, or a protein expressed on a cancer cell, e.g. to act as an agonist for a particular receptor or to prevent binding of another ligand and hence prevent activation of a relevant biological pathway.
  • the environment may be a drug design environment such that each state is a respective state of a potential pharmaceutically active compound pharmaceutically active compound and the agent is a computer system for determining elements of the pharmaceutically active compound and/or a synthetic pathway for the pharmaceutically active compound.
  • the drug/synthesis may be designed based on a reward derived from a target for the pharmaceutically active compound, for example in simulation.
  • the agent may be a mechanical agent that performs or controls synthesis of the pharmaceutically active compound.
  • the environment is a real-world environment and the agent manages distribution of tasks across computing resources e.g. on a mobile device and/or in a data center.
  • the actions may include assigning tasks to particular computing resources.
  • the observations may include observations of computing resources such as compute and/or memory capacity, or Internet-accessible resources; and the actions may include assigning tasks to particular computing resources.
  • the reward(s) may be configured to maximize or minimize one or more of: utilization of computing resources, electrical power, bandwidth, and computation speed.
  • the actions may include presenting advertisements, the observations may include advertisement impressions or a click-through count or rate, and the reward may characterize previous selections of items or content taken by one or more users.
  • the observations may include textual or spoken instructions provided to the agent by a third-party (e.g., an operator of the agent).
  • the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide textual or spoken instructions to the agent (e.g., to navigate to a particular location).
  • DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0175]
  • the environment may be an electrical, mechanical or electro- mechanical design environment, e.g. an environment in which the design of an electrical, mechanical or electro-mechanical entity is simulated.
  • the simulated environment may be a simulation of a real-world environment in which the entity is intended to work.
  • the task may be to design the entity.
  • the observations may comprise observations that characterize the entity, i.e.
  • the actions may comprise actions that modify the entity e.g. that modify one or more of the observations.
  • the rewards may comprise one or more metrics of performance of the design of the entity. For example rewards may relate to one or more physical characteristics of the entity such as weight or strength or to one or more electrical characteristics of the entity such as a measure of efficiency at performing a particular function for which the entity is designed.
  • the design process may include outputting the design for manufacture, e.g. in the form of computer executable instructions for manufacturing the entity.
  • the process may include making the entity according to the design.
  • a design of an entity may be optimized, e.g.
  • the environment may be a simulated environment.
  • the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions.
  • the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation.
  • the actions may be control inputs to control the simulated user or simulated vehicle.
  • the agent may be implemented as one or more computers interacting with the simulated environment.
  • the simulated environment may be a simulation of a particular real-world environment and agent.
  • the system may be used to select actions in the simulated environment during training or evaluation of the system and, after training, or evaluation, or both, are complete, the action selection policy may be deployed for controlling a real-world agent in the particular real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear on and damage to the real- DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application world environment or real-world agent and can allow the control neural network to be trained and evaluated on situations that occur rarely or are difficult or unsafe to re-create in the real-world environment.
  • the system may be partly trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and afterwards deployed to control the real mechanical agent in the particular real-world environment.
  • the observations of the simulated environment relate to the real-world environment
  • the selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment.
  • the agent may not include a human being (e.g. it is a robot).
  • the agent comprises a human user of a digital assistant such as a smart speaker, smart display, or other device. Then the information defining the task can be obtained from the digital assistant, and the digital assistant can be used to instruct the user based on the task.
  • the reinforcement learning system may output to the human user, via the digital assistant, instructions for actions for the user to perform at each of a plurality of time steps.
  • the instructions may for example be generated in the form of natural language (transmitted as sound and/or text on a screen) based on actions chosen by the reinforcement learning system.
  • the reinforcement learning system chooses the actions such that they contribute to performing a task.
  • a monitoring system e.g. a video camera system
  • the reinforcement learning system can determine whether the task has been completed.
  • the experience tuples may record the action which the user actually performed based on the instruction, rather than the one which the reinforcement learning system instructed the user to perform.
  • the reward value of each experience tuple may be generated, for example, by comparing the action the user took with a corpus of data showing a human expert performing the task, e.g. using techniques known from imitation learning. Note that if the user performs actions incorrectly (i.e. performs a different action from the one the reinforcement learning system instructs the user to perform) this adds one more source of noise to sources of noise which may already exist in the environment.
  • the reinforcement learning system may identify actions which the user performs incorrectly with more than a certain DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application probability. If so, when the reinforcement learning system instructs the user to perform such an identified action, the reinforcement learning system may warn the user to be careful. Alternatively or additionally, the reinforcement learning system may learn not to instruct the user to perform the identified actions, i.e. ones which the user is likely to perform incorrectly. [0180] More generally, the digital assistant instructing the user may comprise receiving, at the digital assistant, a request from the user for assistance and determining, in response to the request, a series of tasks for the user to perform, e.g. steps or sub-tasks of an overall task.
  • the digital assistant can be used to output to the user an indication of the task, e.g. step or sub-task, to be performed.
  • This may be done using natural language, e.g. on a display and/or using a speech synthesis subsystem of the digital assistant.
  • Visual, e.g. video, and/or audio observations of the user performing the task may be captured, e.g. using the digital assistant.
  • a system as described above may then be used to determine whether the user has successfully achieved the task e.g. step or sub-task, i.e. from the answer as previously described.
  • a digital assistant device including a system as described above.
  • the digital assistant can also include a user interface to enable a user to request assistance and to output information.
  • this is a natural language user interface and may comprise a keyboard, voice input-output subsystem, and/or a display.
  • the digital assistant can further include an assistance subsystem configured to determine, in response to the request, a series of tasks for the user to perform. In implementations this may comprise a generative (large) language model, in particular for dialog, e.g. a conversation agent such as Sparrow or Chinchilla.
  • the digital assistant can have an observation capture subsystem to capture visual and/or audio observations of the user performing a task; and an interface for the above-described language model neural network (which may be implemented locally or remotely).
  • the digital assistant can also have an assistance control subsystem configured to assist the user.
  • the assistance control DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application subsystem can be configured to perform the steps described above, for one or more tasks e.g. of a series of tasks, e.g. until a final task of the series. More particularly the assistance control subsystem and output to the user an indication of the task to be performed, capture, using the observation capture subsystem, visual or audio observations of the user performing the task, determine from the above-described answer whether the user has successfully achieved the task. In response the digital assistant can progress to a next task of the series of tasks and/or control the digital assistant, e.g. to stop capturing observations. [0182] In some implementations, the environment may not include a human being or animal.
  • the agent may be an autonomous vehicle in an environment which is a location (e.g. a geographical location) where there are human beings (e.g. pedestrians or drivers/passengers of other vehicles) and/or animals, and the autonomous vehicle itself may optionally contain human beings.
  • the environment may also be at least one room (e.g. in a habitation) containing one or more people.
  • the human being or animal may be an element of the environment which is involved in the task, e.g. modified by the task (indeed, the environment may substantially consist of the human being or animal).
  • the environment may be a medical or veterinary environment containing at least one human or animal subject, and the task may relate to performing a medical (e.g. surgical) procedure on the subject.
  • the environment may comprise a human user who interacts with an agent which is in the form of an item of user equipment, e.g. a digital assistant.
  • the item of user equipment provides a user interface between the user and a computer system (the same computer system(s) which implement the reinforcement learning system, or a different computer system).
  • the user interface may allow the user to enter data into and/or receive data from the computer system, and the agent is controlled by the action selection policy to perform an information transfer task in relation to the user, such as providing information about a topic to the user and/or allowing the user to specify a component of a task which the computer system is to perform.
  • the information transfer task may be to teach the user a skill, such as how to speak a language or how to navigate around a geographical location; or the task may be to allow the user to define a three-dimensional shape to the computer system, e.g. so that the computer system can control an additive manufacturing (3D printing) system to produce an object having the shape.
  • Actions may comprise outputting information to the user (e.g.
  • an action may comprise setting a problem for a user to perform relating to the skill (e.g. asking the user to choose between multiple options for correct usage of the language, or asking the user to speak a passage of the language out loud), and/or receiving input from the user (e.g. registering selection of one of the options, or using a microphone to record the spoken passage of the language).
  • Rewards may be generated based upon a measure of how well the task is performed. For example, this may be done by measuring how well the user learns the topic, e.g.
  • the action may comprise presenting a (visual, haptic or audio) user interface to the user which permits the user to specify an element of the component of the task, and receiving user input using the user interface.
  • the rewards may be generated based on a measure of how well and/or easily the user can specify the component of the task for the computer system to perform, e.g. how fully or well the three-dimensional object is specified.
  • FIG. 4 is a flow diagram of an example process 400 for training a policy neural network and a value neural network. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations.
  • a training system e.g., the training system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400.
  • the system can repeatedly perform iterations of the process 400 on different imagination trajectories generated by using a world model neural network to update the parameters of the policy neural network and the value neural network.
  • DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0186]
  • the policy neural network is configured to process a policy network input that includes an updated latent representation in accordance with current values of the policy network parameters to generate an action selection output.
  • the value neural network is configured to process a value network input that includes an updated latent representation in accordance with current values of the value network parameters to generate a predicted state value.
  • the system can continue performing iterations of the process 400 until termination criteria for the training of the neural networks have been satisfied, e.g., until the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of the process 400 have been performed.
  • the system receives a latent representation that is in a latent space and that characterizes a current state of the environment (step 402).
  • the latent representation can be generated by the world model neural network based on a current observation (e.g., an image or a video frame) of the current state of the environment.
  • the world model neural network can include a sequence model neural network, an encoder neural network, a decoder neural network.
  • the latent representation can be generated by the encoder neural network based on a pre-determined initial hidden state of the sequence model neural network and on the current observation.
  • the system generates an imagination trajectory that includes an updated latent representation at each of multiple time steps during the imagination trajectory (step 404).
  • the imagination trajectory starts with the received latent representation. That is, at the first time step, the updated latent representation is the received latent representation.
  • generating each imagination trajectory can include repeatedly generating an action selection output by using the policy neural network, selecting an action that would be performed by the agent by using the action selection output, and then using the world model neural network to generate an updated latent representation based at least on the selected action.
  • the system determines a predicted reward (step 406) and determines a predicted state value (step 408).
  • the predicted state value represents a predicted value of the environment being in the state characterized by the updated latent representation in terms of a cumulative measure of predicted rewards to be received by the agent.
  • the predicted state value can DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application specify a long-term time-adjusted, e.g., time-discounted, sum of predicted rewards that would be received by the agent by selecting actions using action selection outputs starting from the state characterized by the updated latent representation.
  • the system can determine a predicted reward by processing the updated latent representation and an updated hidden state of the sequence model neural network using a reward neural network in accordance with current values of the reward network parameters to generate the predicted reward, which can involve processing the current latent representation and the selected action using a sequence model neural network to generate the updated hidden state.
  • the system can determine a predicted state value by processing the updated latent representation using the value neural network to generate the predicted state value.
  • the system determines a corresponding target state value for each updated latent representation in the imagination trajectory based on the predicted rewards and the predicted state values for the updated latent representations in the trajectory (step 410).
  • the target state value can be determined by computing the bootstrapped ⁇ - returns that integrate both the predicted rewards and the predicted state values. An example way of computing the bootstrapped ⁇ -returns is described above with reference to FIG. 2B.
  • the system determines an update to the current values of the policy network parameters by optimizing an actor objective function (step 412).
  • the actor objective function includes a first term that is dependent on the target state values. In some implementations, the actor objective function includes a second term that is dependent on an entropy of the action selection outputs generated by the policy neural network.
  • An example of the actor objective function is described above with reference to FIG. 2B.
  • the system can determine the update by computing, e.g., through backpropagation, respective gradients of the actor objective function with respect to the policy network parameters, and then applying an update rule, e.g., an Adam update rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule, to the respective gradients.
  • an update rule e.g., an Adam update rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule
  • the system applies a symmetric logarithmic transformation to each target state value to generate a transformed target state value for each updated latent representation in the imagination trajectory (step 414).
  • DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0196]
  • the system encodes each transformed target state value to generate an encoded transformed target state value for each updated latent representation imagination in the trajectory (step 416).
  • the system can use a two-hot encoding technique to generate the encoded transformed target state values that can each be represented by a vector of a predetermined length having a predetermined number of non-zero entries.
  • the system determines an update to the current values of the value network parameters by optimizing a critic objective function (step 418).
  • the critic objective function includes a term that measures a difference between the predicted state values and the encoded transformed target state values.
  • An example of the critic objective function is described above with reference to FIG. 2B.
  • the system can determine the update by computing, e.g., through backpropagation, respective gradients of the critic objective function with respect to the value network parameters, and then applying an update rule, e.g., an Adam update rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule, to the respective gradients.
  • FIG. 5 is a flow diagram of an example process 500 for generating an imagination trajectory of latent representations. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations.
  • a training system e.g., the training system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500.
  • the system can repeatedly perform the process 500 to generate an imagination trajectory that includes an updated latent representation at each of multiple time steps during the imagination trajectory.
  • the imagination trajectory starts with the latent representation received at step 402 of process 400. That is, at the first time step, the updated latent representation is the received latent representation.
  • the system processes a current latent representation using the policy neural network and in accordance with current values of the policy network parameters to generate an action selection output (step 502). For the first time step, the current latent representation is the received latent representation.
  • the current latent representation is the updated latent representation that has been generated in the preceding time step.
  • the system selects, using the action selection output, an action that would be performed by the agent when the environment is in a state characterized by the current DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application latent representation (step 504). Selecting the action by using the action selection output is similarly described above with reference to FIG. 1.
  • the system generates an updated latent representation based on the current latent representation and the selected action (step 506).
  • the updated latent representation characterizes a state that the environment would transition into if the agent performed the selected action when the environment is in a state characterized by the current latent representation.
  • the system processes the current latent representation and the selected action using the sequence model neural network and in accordance with the current values of sequence model network parameters to generate an updated hidden state.
  • the system then processes the updated hidden state using the dynamics predictor neural network to generate the updated latent representation.
  • the system trains the world model neural network to ensure the quality of the imagination trajectories.
  • the system trains the encoder neural network and the dynamics predictor neural network to encourage the generation of high-quality latent representations, e.g., latent representations that can capture salient (e.g., task-specific) information contained in observation of the environment, or latent representations that are predictive of subsequent states of the environment.
  • the system also trains the sequence model neural network to more accurately simulate latent dynamics of the environment, i.e., state transitions of the environment that would be caused by the agent performing the proposed actions, trains the reward neural network to more accurately estimate rewards that would be received by the agent from the environment in response to performing the selected actions, and trains the continuation predictor neural network to more accurately predict whether a current state is a terminal state of the environment.
  • FIG. 6 is a flow diagram of an example process 600 for training a world model neural network.
  • the process 600 will be described as being performed by a system of one or more computers located in one or more locations.
  • a training system e.g., the training system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 600.
  • the system can sample historical trajectories from a replay buffer and train the world model neural network on the sampled historical trajectories.
  • the replay buffer can be any available memory that stores historical trajectories generated as a consequence of DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application the interaction of the agent (or another agent) with the environment (or with another instance of the environment) to attempt to perform one or more tasks.
  • each historical trajectory can include a tuple at each of multiple time points between during the historical trajectory.
  • the tuple includes (i) a current observation characterizing a current state of the environment, (ii) a current action performed by the agent, (iii) a current reward received by the agent, and (iv) a ground truth continuation flag that indicates whether the current state of the environment is a terminal state of the environment (e.g., the goal state of the environment that is achieved upon the completion of a task).
  • the system can repeatedly perform iterations of the process 600 on different historical trajectories sampled from the replay buffer to update the parameters of the world model neural network.
  • the system can continue performing iterations of the process 600 until termination criteria for the training of the world model neural network have been satisfied, e.g., until the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of the process 600 have been performed. [0210] In general, the system can perform the following steps 602-612 at each of multiple time steps during the historical trajectory.
  • the system processes (i) a current latent representation that is generated at a preceding time step and that characterizes a previous state of the environment and (ii) data specifying a previously selected action selected at the preceding time step using the sequence model neural network in accordance with current values of sequence model network parameters to generate an updated hidden state that characterizes the current state of the environment (step 602).
  • the system processes the updated hidden state and the current observation ⁇ of the current state of the environment using the encoder neural network in accordance with current values of encoder network parameters to generate an updated latent representation that characterizes the current state of the environment (step 604).
  • the system processes the updated hidden state using the dynamics predictor neural network in accordance with current values of dynamics predictor network parameters to generate an estimated updated latent representation (step 606).
  • the estimated updated latent representation is an estimate of the updated latent representation that characterizes the current state of the environment and that can be generated by the encoder neural network. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0214]
  • the system processes the updated latent representation and, optionally, the updated hidden state using the decoder neural network in accordance with current values of decoder network parameters to generate a reconstructed current observation of the current state of the environment (step 608).
  • the system processes the updated latent representation and, optionally, the updated hidden state using the reward neural network in accordance with current values of reward network parameters to generate a predicted reward (step 610).
  • the predicted reward is a prediction of the reward that would be received by the agent when the environment is in the current state.
  • the system processes the updated latent representation and, optionally, the updated hidden state using the continuation predictor neural network in accordance with current values of decoder network parameters to generate a predicted continuation predictor flag that indicates whether the current state is a terminal state in the trajectory (step 612).
  • the system determines an update to the current values of the parameters of the world model neural network based on optimizing a combined objective function (step 614).
  • the combined objective function measures a difference between the outputs of the neural networks and the ground truth information included in the historical trajectory.
  • An example of the combined objective function is described above with reference to FIG. 2A.
  • determining an update to the current values of the parameters of the world model neural network can involve determining, based on computing a gradient of a dynamics objective function with respect to the sequence model neural network parameters and the dynamics predictor neural network parameters, an update to the current values of the sequence model neural network parameters and the dynamics predictor neural network parameters.
  • the dynamics objective function is dependent on a Kullback– Leibler divergence between a latent representation generated by the encoder neural network and (ii) a latent representation generated by the dynamics predictor neural network.
  • sequence model neural network parameters are parameters of a sequence model neural network that is configured to process the current latent representation and the selected action to generate an updated hidden state of the sequence model neural network.
  • dynamics predictor neural network parameters are parameters of a dynamics predictor neural network that is configured to process the updated hidden state to generate the updated latent representation.
  • the encoder neural network is DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application configured to process an observation to generate the latent representation that characterizes the current state of the environment.
  • the system can determine the update by computing, e.g., through backpropagation, respective gradients of the combined objective function with respect to the appropriate network parameters, and then applying an update rule, e.g., an Adam update rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule, to the respective gradients.
  • an update rule e.g., an Adam update rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule
  • the system can incorporate any number of techniques to improve the speed, the effectiveness, or both of the training process. For example, the system can refrain from using hyperparameter annealing, weight decay, or dropout.
  • the system can initialize the values of the parameters of some of the neural network mentioned above, e.g., the reward network parameters and the value network parameters, to zeros.
  • the techniques for training an encoder-decoder neural network can be applied in any of a wider variety of tasks, e.g., tasks that may not involve controlling an agent.
  • the encoder-decoder neural network can be configured to receive an input data item, process the input data item to generate a latent representation of the input data item, and to process the latent representation of the input data item to generate a reconstruction of the input data item.
  • the encoder-decoder neural network can operate on any of a variety of different types of data items, e.g., images, videos, audio signals, and so on.
  • the encoder-decoder neural network can operate on high-dimensional (and therefore large) data items.
  • the encoder-decoder neural network can be used to compress and reconstruct high-resolution images, e.g., medical images and geographical images.
  • the input data item may comprise an image of a real- world entity captured by an image sensor such as a camera or LIDAR sensor or captured a medical image sensor such as an X-ray or MRI machine.
  • FIG. 7 is a flow diagram of an example process 700 for training an encoder- decoder neural network.
  • the process 700 will be described as being performed by a system of one or more computers located in one or more locations.
  • a training system e.g., the training system 100 of FIG.
  • the system receives a training data item (step 702).
  • the system processes the training data item using the encoder-decoder neural network to determine a latent representation of the training data item and to generate a reconstruction of the training data item in accordance with current values of the parameters of the encoder-decoder neural network (step 704).
  • the latent representation is a lower-dimension, i.e., more compact, version of the training data item.
  • the encoder- decoder neural network includes an encoder neural network and a decoder neural network.
  • the system processes the training data item using the encoder neural network to determine a latent representation of the training data item.
  • the system processes the latent representation of the training data item using the decoder neural network to generate an output of the encoder-decoder neural network.
  • the system then applies a symmetric exponential transformation to the output of the encoder-decoder neural network to generate the reconstruction of the training data item.
  • the symmetric exponential transformation can be in the form of: ⁇ ⁇ ⁇ or ⁇ ⁇ ⁇ , where x is the output of the encoder-decoder neural network.
  • the system applies a symmetric logarithmic transformation to the training data item to generate a transformed training data item (step 706).
  • the symmetric logarithmic transformation is in the form of: ⁇ ⁇ ⁇ ⁇ or ⁇ ⁇ ⁇ ⁇ , where x is the training data item.
  • the system determines an update to the current values of the encoder-decoder network parameters based on evaluating a loss function that measures a difference between the reconstruction of the training data item and the transformed training data item (step 708).
  • the loss function can be a mean squared error loss function.
  • the system can determine the update by computing, e.g., through backpropagation, respective gradients of the combined objective function with respect to the parameters of the encoder-decoder neural network, and then applying an update rule, e.g., an Adam update DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule, to the respective gradients.
  • an update rule e.g., an Adam update DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule, to the respective gradients.
  • the tasks include proprioceptive sensor-based control tasks and visual control tasks (described in arXiv:1801.00690), Atari tasks (described in arXiv:1903.00374 and arXiv:1207.4708), Behavior Suite tasks, or bsuite tasks for short (described in arXiv:1908.03568), and Crafter tasks (described in arXiv:2109.06780).
  • the vertical axes in FIG.8 show the returns received by an agent when controlled using each of multiple neural network systems, where higher returns indicate better performance.
  • the DreamerV3 system corresponds to a system that includes the neural networks described in this specification.
  • the other systems are existing reinforcement learning systems for agent control, e.g., the MPO system is described in arXiv:1806.06920, the DDPG system is described in arXiv:1509.02971, and D4PG system is described in arXiv:1804.08617. It will be appreciated that DreamerV3 outperforms the existing systems on all of these tasks.
  • the DreamerV3 system can achieve this performance gain across these tasks after having been trained using the same values for the set of hyperparameters as listed above in Table. 1.
  • This specification uses the term “configured” in connection with systems and computer program components.
  • a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions.
  • one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.
  • Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
  • Embodiments of the subject matter described in this specification can be DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus.
  • the computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
  • the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
  • data processing apparatus refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
  • the apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • the apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • a computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • a program may, but need not, correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code.
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
  • database is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations.
  • the DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application index database can include multiple collections of data, each of which may be organized and accessed differently.
  • the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions.
  • an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.
  • the processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output.
  • the processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
  • Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both.
  • the essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.
  • the central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • a computer need not have such devices.
  • a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
  • PDA personal digital assistant
  • GPS Global Positioning System
  • USB universal serial bus
  • Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
  • a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • a keyboard and a pointing device e.g., a mouse or a trackball
  • a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser.
  • a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.
  • Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.
  • Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a JAX framework.
  • Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components.
  • the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network.
  • Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
  • the computing system can include clients and servers.
  • a client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
  • a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client.
  • Data generated at the user device e.g., a result of the user interaction, can be received at the server from the device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Robotics (AREA)
  • Image Analysis (AREA)

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a policy neural network used to select an action to be performed by an agent interacting with an environment. In one aspect, a method includes: receiving a latent representation that characterizes a current state of the environment; generating an imagination trajectory of latent representations; for each latent representation in the imagination trajectory: determining a predicted reward; and generating a predicted state value; determining a target state value for each latent representation; determining an update to the current values of the policy network parameters; applying a symmetric logarithmic transformation to each target state value; encoding each transformed target state value to generate an encoded transformed target state value; and determining an update to the current values of the value network parameters by optimizing a critic objective function.

Description

DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application TRAINING REINFORCEMENT LEARNING AGENTS TO PERFORM MULTIPLE TASKS ACROSS DIVERSE DOMAINS CROSS-REFERENCE TO RELATED APPLICATION [0001] This application claims priority to U.S. Provisional Application No. 63/437,931, filed on January 9, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND [0002] This specification relates to reinforcement learning. [0003] In a reinforcement learning system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment. [0004] Some reinforcement learning systems select the action to be performed by the agent in response to receiving a given observation in accordance with an output of a neural network. [0005] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY [0006] This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains a policy neural network that can be used to control an agent to interact with an environment by selecting actions to be performed by the agent. The system implements a general reinforcement learning framework that can effectively train the policy neural network to select actions that enable the agent to accomplish a large number of tasks across a wide range of domains. [0007] The tasks can include, for example, navigating to a specified location in the environment, identifying a specific object in the environment, manipulating the specific object in a specified way, and so on. In some cases, the tasks are long-horizon tasks that require the agent to select and perform thousands or millions of actions in a sequence in DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application order to complete a task. In some cases, the tasks are tasks performed by the agent in a complex environment, e.g., a complex workspace that includes a large number of obstacle and/or target objects, a large number of target locations, or both. [0008] A domain of each task generally defines the properties and/or characteristics of various aspects of the task, e.g., the environment within which the agent is deployed, the space of possible actions that can be performed by the agent, the data that characterizes the environment (also referred to in this specification as an “observation”) received by the agent, and so on. The domains of these tasks can differ from each other in one or more of these aspects, e.g., in the action space (e.g., continuous versus discrete action space), observation space (e.g., higher-dimensional visual inputs versus lower-dimensional textual inputs, or 2D versus 3D camera images), data budgets, reward frequencies, or reward scales. [0009] In one aspect this specification describes a reinforcement learning method performed by one or more computers for training a policy neural network to control an agent acting in an environment based on rewards received as a result of the actions preformed. The policy neural network can be trained using a reinforcement learning objective function that depends on the rewards received, and the rewards, or values dependent on the rewards, can be scaled or “squashed” using a symmetric logarithmic transformation as described herein. [0010] In general, the one innovative aspect of the subject matter described in this specification can be embodied in a method of training a policy neural network having a plurality of policy network parameters used to select actions to be performed by an agent in an environment, the method comprising: receiving a latent representation that is in a latent space and that characterizes a current state of the environment; generating an imagination trajectory of latent representations that starts with the received latent representation by performing the following at each of a plurality of time steps: processing a current latent representation using the policy neural network and in accordance with current values of the policy network parameters to generate an action selection output; selecting, using the action selection output, an action that would be performed by the agent when the environment is in a state characterized by the current latent representation; and processing the current latent representation and the selected action to generate an updated latent representation that characterizes a state that the environment would transition into if the agent performed the selected action when the environment is in a state characterized by the current latent representation; for each latent representation in the imagination DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application trajectory: determining a predicted reward; and processing the latent representation using a value neural network having a plurality of value network parameters and in accordance with current values of the value network parameters to generate a predicted state value that represents a time-adjusted sum of predicted rewards that would be received by the agent by selecting actions using action selection outputs starting from the state characterized by the latent representation; determining a target state value for each latent representation in the imagination trajectory from the predicted rewards and the predicted state values for the latent representations in the imagination trajectory; determining an update to the current values of the policy network parameters by optimizing an actor objective function that includes a first term dependent on the target state values; applying a symmetric logarithmic transformation to each target state value to generate a transformed target state value for each latent representation in the imagination trajectory; encoding each transformed target state value to generate an encoded transformed target state value for each latent representation in the imagination trajectory; and determining an update to the current values of the value network parameters by optimizing a critic objective function that measures a difference between the predicted state values and the encoded transformed target state values. [0011] The symmetric logarithmic transformation may be in the form of: ^^^^ ^ ^ ^ ^^ ^ȁ ^ ȁ ^ ^ ^ or ^^^^ ^ ^ ^ ^^^ ^ȁ ^ ȁ ^ ^ ^ , where x is the target state value. [0012] The encoded transformed target state value may be represented by a vector of a predetermined length having a predetermined number of non-zero entries. [0013] Generating the predicted state value for each latent representation in the imagination trajectory may comprise: processing the latent representation to assign a respective probability to each predetermined interval in a sequence of predetermined intervals between a possible range of state values; generating an output of the value neural network by computing a weighted average of the state values represented by the sequence of predetermined intervals weighted by the respective probabilities; and applying a symmetric exponential transformation to the output of the value neural network to generate the predicted state value for the latent representation in the imagination trajectory. [0014] The symmetric exponential transformation may be in the form of: ^^^^^^^^^^^^ȁ^ȁ^ where x is the output of the value neural network. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0015] Determining the update to the current values of the policy network parameters by optimizing the actor objective function may comprise: generating, by using a scaling factor and from the target state values, a downscaled target state value for each latent representation in the imagination trajectory; and determining the update to the current values of the policy network parameters by optimizing the actor objective function that includes the first term dependent on the downscaled target state values. [0016] Generating the downscaled target state value may comprise: dividing the target state value by a maximum of a given integer value and the scaling factor, wherein the scaling factor is computed using respective percentiles of one or more target state values for the latent representations in the imagination trajectory. [0017] The actor objective function may include a second term dependent on a scaled entropy of the action selection outputs generated by the policy neural network, wherein the scaled entropy may be scaled by a predetermined entropy scaling factor. [0018] Processing the current latent representation and the selected action to generate the updated latent representation may comprise: processing the current latent representation and the selected action using a sequence model neural network in accordance with current values of sequence model network parameters to update a hidden state of the sequence model neural network to generate an updated hidden state that characterizes the current state of the environment; and processing the updated hidden state using a dynamics predictor neural network in accordance with current values of dynamics predictor network parameters to generate the updated latent representation. [0019] Receiving the latent representation that characterizes the current state of the environment may comprise: processing an observation that characterizes the current state of the environment using an encoder neural network in accordance with current values of encoder network parameters to generate the latent representation. [0020] The method may further comprise: processing the latent representation and the updated hidden state using a decoder neural network in accordance with current values of decoder network parameters to generate a reconstruction of the observation; applying the symmetric logarithmic transformation to the observation to generate a transformed observation; and determining, based on computing a gradient of a prediction objective function that includes a first term measuring a difference between the reconstruction of the observation and the transformed observation and with respect to the decoder network parameters, an update to the current values of the decoder network parameters. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0021] For each latent representation in the imagination trajectory, determining the predicted reward may comprise: processing the latent representation and an updated hidden state of the sequence model neural network using a reward neural network in accordance with current values of reward network parameters to generate the predicted reward. [0022] The method may further comprise: applying the symmetric logarithmic transformation to a reward received by the agent when the environment is at the state characterized by the latent representation to generate a transformed reward; and determining, based on computing a gradient of the prediction objective function that includes a second term measuring a difference between the predicted reward and the transformed reward and with respect to the reward network parameters, an update to the current values of the reward network parameters. [0023] The method may further comprise: determining, based on computing a gradient of the dynamics objective function with respect to the sequence model neural network parameters and the dynamics predictor neural network parameters, an update to the current values of the sequence model neural network parameters and the dynamics predictor neural network parameters, wherein the dynamics objective function is dependent on a Kullback–Leibler divergence between (i) a latent representation generated by the encoder neural network and (ii) a latent representation generated by the dynamics predictor neural network. [0024] The encoder neural network and the decoder neural network may each be configured as a respective convolutional neural network or multi-layer perceptron (MLP); the dynamics predictor neural network and the reward neural network may each be configured as a multi-layer perceptron (MLP); and the sequence model neural network may be configured as a recurrent neural network. [0025] The convolutional neural network, the multi-layer perceptron (MLP), or the recurrent neural network may comprise: one or more layer normalization layers and one or more sigmoid linear unit (SiLU) activation layers. [0026] The method may further comprise initializing the current values of the reward network parameters and the value network parameters to zeros. [0027] The method may further comprise using the policy neural network to select actions to be performed by the agent in the environment by processing an input derived from data characterizing the current state of the environment at a time step, using the policy neural DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application network, to generate an action selection output used to determine an action to be performed by the agent at the time step. [0028] In another aspect this specification describes a method performed by one or more computers for training an encoder-decoder neural network. The encoder-decoder neural network can comprise an encoder neural network configured to receive an encoder network input and to generate an encoder network output, and a decoder neural network to receive a decoder network input comprising the encoder network output and to generate a decoder network output. The encoder-decoder neural network is configured to receive an input data item (at the encoder network input), process the input data item to generate (an encoder network output comprising) a latent representation of the input data item, and to process the latent representation of the input data item (using the decoder neural network) to generate a reconstruction of the input data item (at the decoder network output). An update to current values of the encoder-decoder network parameters is determined based on evaluating a loss function that measures a difference between the reconstruction of a training data item and a transformed training data item that has been transformed using a symmetric logarithmic transformation. [0029] In general, the other innovative aspect of the subject matter described in this specification can be embodied in a method performed by one or more computers for training an encoder-decoder neural network, wherein the encoder-decoder neural network is configured to receive an input data item, process the input data item to generate a latent representation of the input data item, and to process the latent representation of the input data item to generate a reconstruction of the input data item, and wherein the method comprises: receiving a training data item; processing the training data item using the encoder-decoder neural network to determine a latent representation of the training data item and to generate a reconstruction of the training data item in accordance with current values of encoder-decoder network parameters; applying a symmetric logarithmic transformation to the training data item to generate a transformed training data item; and determining an update to the current values of the encoder-decoder network parameters based on evaluating a loss function that measures a difference between the reconstruction of the training data item and the transformed training data item. [0030] The symmetric logarithmic transformation may be in the form of: ^^^^ ^ ^ ^ ^^ ^ȁ ^ ȁ ^ ^ ^ or ^^^^ ^ ^ ^ ^^^ ^ȁ ^ ȁ ^ ^ ^ , where x is the training data item. [0031] The loss function may be a mean squared error loss function. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0032] Generating the reconstruction of the input data item may comprise: processing the training data item to determine the latent representation of the training data item; processing the latent representation of the training data item to generate an output of the encoder-decoder neural network; and applying a symmetric exponential transformation to the output of the encoder-decoder neural network to generate the reconstruction of the training data item. [0033] The symmetric exponential transformation may be in the form of: ^^^^^^^^^^^^ȁ^ȁ^ െ ^^ or ^^^^^^^൫^^ȁ௫ȁି^൯, where x is the output of the encoder-decoder neural network. [0034] The input data item may comprise an image. [0035] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. A system of one or more computers can be configured to perform particular operations or actions by virtue of software, firmware, hardware, or any combination thereof installed on the system that in operation may cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions. [0036] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. [0037] Many complex tasks, e.g., robotic tasks, involve reward scales which can vary across the domain of application, between particular tasks, and across stages of learning, sometimes by large factors. This can be detrimental to stability and can result in slow or failed learning. The described techniques can implement a general reinforcement learning (RL) framework that trains a policy neural network together with one or more additional neural networks based on latent representations of an environment to achieve robust performance in any of multiple tasks across a wide range of domains, even including long- horizon tasks (that require the agent to select and perform thousands or millions of actions in a sequence) and tasks performed in sparse-reward or hard-to-explore environments. [0038] The described reinforcement learning framework can use a fixed set of hyperparameters that are readily applicable across these diverse domains and is thus easy to setup whenever learning a new task is needed. Under the described framework, training DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application an instance of the policy neural network for every new task may thus consume less wall clock time and fewer computational resources, e.g., memory and processing power, than conventional RL systems because hyperparameter tuning by iteratively running multiple training trials is no longer needed. The described reinforcement learning framework is also scalable, meaning that it is capable of training an arbitrarily large policy neural network, where increased model size (e.g., in terms of model parameters) corresponds to both higher task performance and improved data efficiency. [0039] In particular, by leveraging a symmetric logarithmic transformation which compresses the magnitudes of both large positive and negative output values of a neural network while preserving the input sign, the described framework alleviates common issues including divergence, stagnation, and non-stationarity that might otherwise occur to hinder the RL training when some of the existing loss functions and normalization techniques are used, and instead ensures stabilized training of the neural networks across the diverse domains by unifying gradient signals in different orders of magnitude. [0040] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS [0041] FIG. 1 shows an example training system. [0042] FIG. 2A is an example illustration of training a world model neural network. [0043] FIG. 2B is an example illustration of training a policy neural network and a value neural network. [0044] FIG. 3 illustrates an example data flow for selecting actions to be performed by an agent interacting with an environment. [0045] FIG. 4 is a flow diagram of an example process for training a policy neural network and a value neural network. [0046] FIG. 5 is a flow diagram of an example process for generating an imagination trajectory. [0047] FIG. 6 is a flow diagram of an example process for training a world model neural network. [0048] FIG. 7 is a flow diagram of an example process for training an encoder-decoder neural network. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0049] FIG. 8 shows a quantitative example of the performance gains that can be achieved by using the neural networks described in this specification. [0050] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION [0051] This specification describes a training system implemented as computer programs on one or more computers in one or more locations for training a policy neural network. During or after training, the policy neural network can be used to control a reinforcement learning agent interacting with an environment by, at each of multiple time steps, processing an input derived from data characterizing the current state of the environment at the time step (i.e., an “observation”) to generate an output used to determine an action to be performed by the agent. [0052] In particular, the training system implements a general reinforcement learning (RL) framework to train the policy neural network by generating multiple imagination trajectories and then training the policy neural network on the multiple imagination trajectories. By doing so, the training system trains the policy neural network so that the policy neural network can be used to control the agent to perform any of multiple tasks. [0053] Each imagination trajectory includes data that characterizes a sequence of predicted state transitions of the environment caused by the actions that would be performed by the agent when performing a task. To that end, each imagination trajectory includes multiple latent representations in a latent space that characterize different states of the environment spanning multiple time steps during the imagination trajectory. The multiple latent representations include a latent representation that characterizes an initial state of the environment, a latent representation that characterizes a terminal state of the environment (e.g., the goal state of the environment that is achieved upon the completion of the task), and one or more latent representations that respectively characterize the intermediate states of the environment between the initial state and the terminal state. [0054] In this specification, a “latent representation” refers to a representation of a state of the environment that is more data efficient, e.g., more compact, than an observation of the same state of the environment. The latent representation includes an ordered collection of numerical values, e.g., a tensor (e.g., a vector or matrix) of numerical values, that generally has a pre-determined dimensionality that is lower than that of the observation. In DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application this specification, the space of possible tensors having the pre-determined dimensionality is referred to as the “latent space.” [0055] FIG. 1 shows an example training system 100. The training system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. [0056] The training system 100 includes a policy neural network 120, a value neural network 130, and a world model neural network 140. The policy neural network 120 is a neural network ^ఏ configured to receive, at a given time step ^ during a trajectory, a policy network input that includes a latent representation ^௧ that characterizes a current state of the environment at the given time step ^ and to process the a policy network input in accordance with a set of parameters ^ (referred to in this specification as “policy network parameters”) to generate an action selection output that can be used to select an action ^௧ to be performed by the agent at the given time step ^. The latent representation ^௧ is later also referred to as an updated latent representation; obtaining such an updated latent representation ^௧ for the given time step ^ will be described further below. [0057] A few examples of the action selection output and how it can be used to select the action are described next. In one example, the action selection output can include a respective numerical probability value for each action in a set of possible actions that can be performed by the agent. If being used to select the action, the action to be performed by the agent could be selected, e.g., by sampling an action in accordance with the probability values for the actions, or by selecting the action with the highest probability value. [0058] In another example, the action selection output can directly define the action to be performed by the agent, e.g., by defining the values of torques that should be applied to the joints of a robotic agent. Such an action can be an action that is predicted to yield the highest long-term time-discounted reward if performed by the agent in response to the observation. [0059] In another example, the action selection output can include a respective Q value for each action in the set of possible actions that can be performed by the agent. If being used to select the action, the Q values could be processed (e.g., using a softmax function) to generate a respective probability value for each possible action, which can be used to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application select the action to be performed by the agent (as described earlier). The action with the highest Q value could also be selected as the action to be performed by the agent. [0060] In this example, the Q value for an action is an estimate of a “return” that would result from the agent performing the action in response to a current observation and thereafter selecting future actions performed by the agent in accordance with current values of the policy network parameters. [0061] A return refers to a cumulative measure of “rewards” received by the agent, for example, a time-discounted sum of rewards. The agent can receive a respective reward at each time step, where the reward is specified by a scalar numerical value and characterizes, e.g., a progress of the agent towards completing an assigned task. [0062] The value neural network 130 is a neural network ^ configured to receive, at a given time step ^ during the trajectory, a value network input that includes the updated latent representation ^௧ that characterizes the current state of the environment at the given time step ^ and to process the value network input in accordance with a set of parameters ^ (referred to in this specification as “value network parameters”) to generate a predicted state value, i.e., a predicted value of the current state of the environment. In some implementations, the predicted state value represents a time-adjusted sum of predicted rewards that would be received by the agent by selecting actions using action selection outputs of the policy neural network during the trajectory starting from the current state of the environment that corresponds to the updated latent representation ^௧. [0063] The world model neural network 140 captures what the training system 100 knows about the environment (the “world”) so that world model neural network 140 can make predictions about the trajectory in a latent space based on what it knows about the past. At a high level, the world model neural network 140 is configured to simulate state transitions of the environment that would be caused by actions performed by the agent that could be selected using the action selection outputs of the policy neural network 120 at the multiple time points during the trajectory. [0064] The world model neural network 140 can include a sequence model neural network 150, an encoder-decoder neural network which, in turn, includes an encoder neural network 160 and a decoder neural network 165, a dynamics predictor neural network 170, a reward neural network 180, and a continuation predictor neural network 190. In some implementations, the world model neural network 140 includes all of the neural network 150, 160, 165, 170, 180, and 190. In other implementations, the world model neural DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application network 140 includes only some of these neural networks, e.g., the decoder neural network 165 and the continuation predictor neural network 190 may be optional components of the world model neural network 140. [0065] The sequence model neural network 150 is a neural network ^ configured to receive, at a given time step ^ during the trajectory, a sequence model network input that includes a current latent representation ^௧ି^ that characterizes a previous state of the environment and data specifying a previously selected action ^௧ି^ in accordance with a set of parameters (referred to in this specification as “sequence model network parameters”) to update a hidden state ^௧ି^ of the sequence model neural network ^, e.g., to modify the current state ^௧ି^ that has been generated by processing a further previous latent representation ^௧ିଶ and the data specifying a further previously selected action ^௧ିଶ by processing the previous latent representation and the previously selected action ^௧ି^. Generally herein the subscript ^ is used to denote parameters of the world model neural network 140, and parameters of any of the neural networks that this includes. As used herein “parameters” refers to trainable parameters of a neural network, such as weights, as distinct from hyperparameters. [0066] At the given time step ^, the hidden state of the sequence model neural network 150 after processing a world model network input will be referred to in this specification as an updated hidden state ^௧ that characterizes a current state of the environment. The current state is a state that the environment would transition into if the agent performed the previously selected action ^௧ି^ when the environment was in the previous state characterized by the current latent representation ^௧ି^. For the initial state of the environment, the updated hidden state ^௧ of the sequence model neural network 150 can be a pre-determined initial hidden state. [0067] Like the latent representation, a “hidden state” refers to a representation of a state of the environment that is more data efficient, e.g., more compact, than an observation of the same state of the environment. The hidden state includes an ordered collection of numerical values, e.g., a tensor (e.g., a vector or matrix) of numerical values, that generally has a pre-determined dimensionality that is lower than that of the observation. [0068] The encoder neural network 160 included in the encoder-decoder neural network is a neural network ^థ configured to receive, at the given time step ^, an encoder network input that includes the updated hidden state ^௧ that characterizes the current state of the environment and a current observation^^௧ of the current state of the environment, and to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application process the encoder network input in accordance with a set of parameters ^ (referred to in this specification as “encoder network parameters”) to generate the updated latent representation ^௧ that characterizes the current state of the environment. [0069] While an observation of a state of an environment typically includes high- dimensional sensor data, e.g., image or video data, that characterizes the state of the environment, the latent representation that characterizes the same state of the environment can be represented as an ordered collection of numerical values, e.g., a vector or matrix of numerical values, that generally has a lower dimensionality than the observation. For example, each observation can be an RGB image that has 64 × 64 ×3 pixels, whereas each latent representation can be a 32-dimensional array that includes 32 latent variables. [0070] The decoder neural network 165 included in the encoder-decoder neural network is a neural network ^థ configured to receive, at the given time step ^, a decoder network input that includes the updated latent representation ^௧ that characterizes the current state of the environment and, in some implementations, the updated hidden state ^௧ that characterizes the current state of the environment, and to process the decoder network input in accordance with a set of parameters ^ (referred to in this specification as “decoder network parameters”) to generate a reconstructed current observation ^^௧^of the current state of the environment. The reconstructed current observation ^^௧ is a reconstruction of the current observation ^௧ that is received by the encoder neural network 160. [0071] The dynamics predictor neural network 170 is a neural network ^ configured to receive, at the given time step ^, a dynamics predictor network input that includes the updated hidden state ^௧ that characterizes the current state of the environment, and to process the dynamics predictor network input in accordance with a set of parameters ^ (referred to in this specification as “dynamics predictor network parameters”) to generate an estimated updated latent representation ^Ƹ௧ that characterizes the current state of the environment. The estimated updated latent representation ^Ƹ௧ is an estimate of the updated latent representation ^௧ that characterizes the current state of the environment and that can be generated by the encoder neural network 160. [0072] The estimated updated latent representation ^Ƹ௧ can have the same dimensionality as the updated latent representation ^௧. Unlike the encoder neural network, the dynamics predictor neural network 170 is configured to generate the estimated updated latent representation ^Ƹ௧ from limited information without relying on the current observation^^௧ of DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application the current state of the environment, i.e. in implementations the dynamics predictor network input does not include the observation ^௧. [0073] The reward neural network 180 is a neural network ^ configured to receive, at the given time step ^, a reward network input that includes the updated latent representation ^௧ that characterizes the current state of the environment and, in some implementations, the updated hidden state ^௧ that characterizes the current state of the environment, and to process reward network input in accordance with a set of parameters ^ (referred to in this specification as “reward network parameters”) to generate a predicted reward ^௧Ƹ. The predicted reward ^௧Ƹ is a prediction of a reward ^௧^that would be received by the agent when the environment is in the current state. As used herein a neural network denoted ^ can refer to any neural network that makes a prediction, e.g. of a reward, dynamics, continuation, or observation (see below). [0074] The continuation predictor neural network 190 is a neural network ^ configured to receive, at the given time step ^, a continuation predictor network input that includes the updated latent representation ^௧ that characterizes the current state of the environment and, in some implementations, the updated hidden state ^௧ that characterizes the current state of the environment, and to process the continuation predictor network input in accordance with a set of parameters ^ (referred to in this specification as “continuation predictor network parameters”) to generate a predicted trajectory continuation flag ^௧ that indicates whether the current state is a terminal state in the trajectory, that is, whether another updated latent representation should be generated in order to characterize a future state of the environment. [0075] In some implementations, the predicted trajectory continuation flag Ƹ௧^ includes a binary value, where 1 define that the current state is a terminal state in the trajectory (such that no more updated latent representations should be generated), 0 defines the current state is not a terminal state in the trajectory (such that another updated latent representation should be generated), or vice versa. For example, a trajectory might terminate upon completion of a task by the agent, or after a threshold number of time steps, or the like. [0076] The policy neural network 120, the value neural network 130, the sequence model neural network 150, the encoder neural network 160, the decoder neural network 165, the dynamics predictor neural network 170, the reward neural network 180, and the DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application continuation predictor neural network 190 can each be implemented with any appropriate neural network architecture that enables them to perform their described functions. [0077] In some implementations, the policy neural network 120 and the value neural network 130 can include any appropriate number of layers (e.g., 5 layers, 10 layers, or 25 layers) of any appropriate type (e.g., fully connected layers, convolutional layers, recurrent layers, attention layers, etc.) and connected in any appropriate configuration (e.g., as a linear sequence of layers, with or without residual connections). [0078] In some of these implementations, the policy neural network 120 can be a large- scale neural network, e.g., a neural network that includes a large number of layers (e.g., 50 layers, 100 layers, or more) and, correspondingly, has a large number of model parameters. Generally, a larger number of model parameters corresponds to both higher task performance and improved data efficiency. For example, a larger scale policy neural network that has 18 million parameters will generally have a higher performance, e.g., in terms of the returns received by an agent controlled using the neural network, than a smaller scale policy neural network that has 8 million parameters after the same number of training iterations. That is, the number of neural layers of the neural networks, and the number of trainable parameters of the system, can be scaled according to a complexity of a task to be learned. [0079] In some implementations, the value neural network 130 is configured to generate the predicted state values in a discrete regression approach. Specifically, the value neural network 130 first generates, e.g., using a softmax layer of the value neural network, a probability distribution ^ట(^^ȁ^௧^ over a sequence ^ of ^ predetermined intervals a possible range of state value, where ^ can be any positive integer (e.g., 63, 127, 255, and so on). That is, the probability distribution assigns a respective probability score to each predetermined interval ^^ in the sequence ^ of ^ predetermined intervals. The value neural network 130 then generates the predicted state value by computing an average of the state values represented by the sequence of predetermined intervals. The average is computed as a weighted average that is weighted by the respective probability scores. [0080] In some implementations, the sequence model neural network 150 is configured as a recurrent neural network that includes one or more recurrent layers, e.g., long short-term memory (LSTM) layers or gated recurrent unit (GRU) layers. When there is more than one recurrent layer, the hidden state can, for example, be the hidden state of the last recurrent DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application layer in a stack of multiple recurrent layers included in the sequence model neural network 150. [0081] Merely as an example, in some implementations, e.g. where an observation includes an image, the encoder neural network 160 and the decoder neural network 165 can each be configured as a respective convolutional neural network (CNN) that includes one or more convolutional layers. In some other implementations, they are each configured as a respective multi-layer perceptron (MLP) that includes one or more fully connected layers. In some implementations one or more CNNs can be included to process one or more visual observations and one or more MLPs can be included to process one or more lower dimensional observations. [0082] In some implementations, similarly to the encoder neural network 160 and the decoder neural network 165, the dynamics predictor neural network 170, the reward neural network 180, and the continuation predictor neural network 190 are each configured as a respective multi-layer perceptron (MLP) that includes one or more fully connected layers. [0083] In some implementations, the sequence model neural network 150, the encoder neural network 160, the decoder neural network 165, the dynamics predictor neural network 170, the reward neural network 180, and the continuation predictor neural network 190 can each include one or more normalization layers, e.g. one or more layer normalization layers and one or more activation layers, e.g. one or more sigmoid linear unit (SiLU) activation layers. [0084] Depending on their architectures, the outputs mentioned above can be generated in any of a variety of ways. For example, the updated hidden state ^௧ can be generated deterministically by, e.g., a last recurrent layer of, the sequence model neural network: [0085] As another example, each of one or more of the updated latent representation ^௧, the estimated updated latent representation ^Ƹ௧, the reconstructed current observation ^^௧, the predicted reward ^௧Ƹ, or the trajectory continuation flag ^௧ can be generated stochastically. For example, in some implementations an output of the encoder neural network parameterizes a distribution from which the updated latent representation ^௧ can be sampled: ^௧̱^థ^^௧ȁ^௧ǡ ^௧^; an output of the decoder neural network parameterizes a distribution from which the reconstructed current observation ^^௧ can be sampled: ^௧^; an output of the dynamics predictor neural network parameterizes a distribution from which the estimated updated latent representation ^Ƹ௧ can be sampled: DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application ^Ƹ௧̱^థ^^Ƹ௧^ȁ^௧^; an output of the reward neural network parameterizes a distribution from which the predicted reward ^௧Ƹ can be sampled: and an output of the continuation predictor neural network parameterizes a distribution from which the predicted trajectory continuation flag Ƹ௧^ can be sampled: ௧ Ƹ^̱^థ^ ௧ Ƹ^ȁ^௧ǡ ^௧^. [0086] The training system 100 includes or has access to a replay buffer 104 that stores historical trajectories that can be generated as a consequence of the interaction of the agent (or another agent) with the environment (or with another instance of the environment) to attempt to perform one or more tasks. The replay buffer 104 can be any available memory, e.g., one or more physical or logical storage devices. [0087] In some implementations, each historical trajectory can include a tuple at each of multiple time points between ^ ൌ ^ to ^ ൌ ^ during the historical trajectory. At a given time point ^, the tuple includes (i) a current observation ^௧ characterizing a current state of the environment, (ii) a current action ^௧ performed by the agent, (iii) a current reward ^௧ received by the agent, and (iv) a ground truth continuation flag ^௧ that indicates whether the current state of the environment is a terminal state of the environment (e.g., the goal state of the environment that is achieved upon the completion of a task). [0088] A training engine 110 of the training system 100 repeatedly selects historical trajectories from the replay buffer 140, and trains the world model neural network 140 based on the selected historical trajectories using suitable world model learning techniques as described below. After training the world model neural network 140, the training engine 110 uses the (trained) world model neural network 140 to train the policy neural network 120 and the value neural network 130. This is done by using the (trained) world model neural network 140 to generate multiple imagination trajectories, and then jointly training the policy neural network 120 and the value neural network 130 based on the multiple imagination trajectories using suitable reinforcement learning techniques. [0089] Training the policy neural network 120 and the value neural network 130 based on the multiple imagination trajectories means that the policy neural network 120 and the value neural network 130 are each configured to either receive as inputs the latent representations during the training (e.g., rather than directly processing the actual observations, e.g., sensor data, e.g., image or video data, that characterize the environment states). [0090] Implementations of the training system 100 include a symlog transformation engine 112 that applies a symmetric logarithmic transformation to an input ^. In general a DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application symmetric logarithmic transformation can be characterized as a logarithmic transformation, i.e. a transformation that applies logarithmic function to its input or argument, ^, to generate a transformed output, and that is symmetrical about the origin (a zero value of the input and logarithmically transformed input) whilst preserving the sign of the input or argument, ^. The logarithm can have any base. As one example, the symmetric logarithmic transformation can be in the form: ^^^^^^^ ^^^ȁ^ȁ ^ ^^ where x is the input or argument. As one example, the symmetric logarithmic transformation can use a log to base 10 (or another base) and can, e.g. be in the form: ^^^^ ^ ^ ^ ^^^ ^ȁ ^ ȁ ^ ^ ^ [0091] The symmetric logarithmic transformation compresses the magnitudes of both large positive and negative values while preserving the input sign. By leveraging such a symmetric logarithmic transformation, the training engine 110 alleviates common issues including divergence, stagnation, and non-stationarity that might otherwise occur to hinder the training when some of the existing loss functions and normalization techniques are used, and instead ensures stabilized training of the neural networks across the diverse domains by unifying gradient signals in different orders of magnitude. [0092] FIG. 2A is an example illustration of training a world model neural network. For example, the world model neural network can correspond to the world model neural network 140 of FIG. 1, which can include a sequence model neural network 150, an encoder-decoder neural network which, in turn, includes an encoder neural network 160 and a decoder neural network 165, a dynamics predictor neural network 170, a reward neural network 180, and a continuation predictor neural network 190. [0093] The training engine 110 performs the operations illustrated in FIG. 2A on each of multiple historical trajectories selected from the replay buffer 104 to train these neural networks using suitable world model learning techniques to enable these neural networks to more accurately simulate the effects on the environment of the actions performed by the agent when interacting with the environment. [0094] As illustrated, at each of multiple time steps ^ during a given historical trajectory selected from the replay buffer 104, the sequence model neural network receives a sequence model network input that includes a current latent representation ^௧ି^ that characterizes a previous state of the environment and data specifying a previously selected action ^௧ି^, and processes the sequence model network input to generate an updated DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application hidden state ^௧ that characterizes a current state of the environment. The current state of the environment is a state that the environment would transition into if the agent performed the previously selected action when the environment was in the previous state characterized by the current latent representation. [0095] For example, at time step ^ ൌ ʹ, the sequence model neural network generates an updated hidden state ^ଶ ൌ ^^ǡ ^^^ that characterizes the current state of the environment. The current state of the environment at time step ^ ൌ ʹ is a state that the environment would transition into if the agent performed the previously selected action when the environment was in the previous state characterized by the current latent representation ^^. [0096] At each of multiple time steps ^ during the given historical trajectory selected from the replay buffer 104, the encoder neural network (“enc”) receives an encoder network input that includes an updated hidden state ^௧ that characterizes a current state of the environment and a current observation^^௧ of the current state of the environment, and processes the encoder network input to generate an updated latent representation ^௧^ that characterizes the current state of the environment. [0097] For example, at time step ^ ൌ ʹ, the encoder neural network (“enc”) generates an updated latent representation ^ଶ̱^థ ^^ଶ ȁ^ଶǡ ^ଶ^ that characterizes the current state of the environment. The current state of the environment at time step ^ ൌ ʹ is a state that the environment would transition into if the agent performed the previously selected action when the environment is in the previous state characterized by the current latent representation ^^. [0098] At each of multiple time steps ^ during the given historical trajectory selected from the replay buffer 104, the decoder neural network (“dec”) receives a decoder network input that includes the updated latent representation ^௧ that characterizes the current state of the environment and, in some implementations, the updated hidden state ^௧ that characterizes the current state of the environment, and processes the decoder network input to generate a reconstructed current observation ^^௧^of the current state of the environment. The reconstructed current observation ^^௧ is a reconstruction of the current observation ^௧ that is received by the encoder neural network 160. [0099] For example, at time step ^ ൌ ʹ, the decoder neural network (“dec”) generates a reconstructed current observation ^^ଶ̱^థ^^^ଶȁ^ଶǡ ^ଶ^. The reconstructed current DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application observation ^^ଶ is a reconstruction of the current observation ^ଶ that is received by the encoder neural network (“enc”). [0100] At each of multiple time steps ^ during the given historical trajectory selected from the replay buffer 104, the dynamics predictor neural network receives a dynamics predictor network input that includes an updated hidden state ^௧ that characterizes the current state of the environment, and processes the dynamics predictor network input to generate an estimated updated latent representation that characterizes the current state of the environment. The estimated updated latent representation ^Ƹ௧ is an estimate of the updated latent representation ^௧ that characterizes the current state of the environment and that can be generated by the encoder neural network 160. [0101] For example, at time step ^ ൌ ʹ, the dynamics predictor neural network generates an estimated updated latent representation ^Ƹଶ̱^థ^^Ƹଶȁ^ଶ^ that characterizes the current state of the environment. The estimated updated latent representation ^Ƹଶ is an estimate of the updated latent representation ^ଶ that characterizes the current state of the environment at time step ^ ൌ ʹ and that can be generated by the encoder neural network 160. [0102] At each of multiple time steps ^ during the given historical trajectory selected from the replay buffer 104, the reward neural network receives a reward network input that includes the updated latent representation ^௧ that characterizes the current state of the environment and, in some implementations, the updated hidden state ^௧ that characterizes the current state of the environment, and processes the decoder network input to generate a predicted reward ^௧Ƹ̱^థ ^^௧Ƹȁ^௧ǡ ^௧^. The predicted reward ^௧Ƹ is a prediction of a reward ^௧^that would be received by the agent when the environment is in the current state. For example, at time step ^ ൌ ʹ, the reward neural network 180 generates a predicted reward ^ଶ^ that is a prediction of a reward ^ଶ^that would be received by the agent when the environment is in the current state at time step ^ ൌ ʹ. [0103] At each of multiple time steps ^ during the given historical trajectory selected from the replay buffer 104, the continuation predictor neural network receives a continuation predictor network input that includes the updated latent representation ^௧ that characterizes the current state of the environment and, in some implementations, the updated hidden state ^௧ that characterizes the current state of the environment, and processes the continuation predictor network input to generate a predicted trajectory continuation flag ^௧^ that indicates whether the current state is a terminal state in the trajectory. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0104] For example, at time step ^ ൌ ʹ, the continuation predictor neural network generates a predicted trajectory continuation flag ଶ Ƹ^̱^థ^ ଶ Ƹ^ȁ^ଶǡ ^ଶ^ that indicates that the current state is not a terminal state in the historical trajectory. Thus, for example, the training system proceeds to use the sequence model neural network to generate an updated hidden state ^ଷ that characterizes a future state of the environment. The future state of the environment is a state that the environment would transition into if the agent performed the selected action ^ଶ when the environment is in the current state characterized by the current latent representation ^ଶ. [0105] For the given historical trajectory selected from the replay buffer 104, the training engine 110 updates the values of the parameters ^ of the dynamics predictor neural network, the reward neural network, and the continuation predictor neural network included in the world model neural network based on optimizing a prediction objective function ^^^^ௗ^^^. [0106] The prediction objective function can include a first term that measures, for each of multiple time steps ^^during the given trajectory, a difference, e.g. a squared difference, between the (i) reconstructed current observation ^^௧ and (ii) a transformed current observation that is generated by the symlog transformation engine 112 included in the training engine 110 from the current observation ^௧^included in the historical trajectory. [0107] In particular, to generate the transformed current observation, the symlog transformation engine 112 applies a symmetric logarithmic transformation to the current observation ^௧. For example, the symmetric logarithmic transformation can be in the form of: ^^^^^^^௧^ ^^^ȁ^௧ȁ ^ ^^ or ^^^^^^௧^ ^^^^ȁ^௧ȁ ^ ^^ where ^௧ is the current observation. The symmetric logarithmic transformation can be applied to any type of observation including, e.g., a low-dimensional inputs. Where applied to a vector or matrix it can be applied to each element of the vector or matrix. [0108] The first term trains the decoder neural network to generate reconstructed current observations that are proximate to the current observations included in the historical trajectories, and, by virtue of backpropagation, trains the encoder neural network to generate semantically meaningful updated latent representations and the sequence model neural network to generate semantically meaningful updated hidden states. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0109] The prediction objective function ^^^^ௗ^^^ can include a second term that measures, for each of multiple time steps ^^during the given trajectory, a difference, e.g. a squared difference, between the predicted reward ^௧Ƹ and a transformed reward that is generated by the symlog transformation engine 112 included in the training engine 110 from the reward ^௧ included in the historical trajectory. The second term trains the reward neural network to generate more accurate predicted rewards, i.e., to generate predicted rewards that are proximate to the rewards included in the historical trajectory, and, by virtue of backpropagation, trains the encoder neural network to generate semantically meaningful updated latent representations and the sequence model neural network to generate semantically meaningful updated hidden states. [0110] The prediction objective function ^^^^ௗ^^^ can include a third term that measures, for each of multiple time steps ^ during the given trajectory, a difference between the predicted trajectory continuation flag ^௧^and the ground truth trajectory continuation flag. The third term trains the continuation predictor neural network to generate more accurate predicted trajectory continuation, i.e., to generate predicted trajectory continuation flags that are proximate to the trajectory continuation flags included in the historical trajectory, and, by virtue of backpropagation, trains the encoder neural network to generate semantically meaningful updated latent representations and the sequence model neural network to generate semantically meaningful updated hidden states. [0111] For example, the prediction objective function can be expressed as: where in practice the terms are differences that can each be computed as a cross entropy loss (a binary classification loss for the continuation term). [0112] For the given trajectory selected from the replay buffer 104, the training engine 110 also updates the values of the parameters ^ of the dynamics predictor neural network included in the world model neural network 140 based on optimizing a dynamics objective function ^ௗ௬^^^^. [0113] The dynamics objective function ^ௗ௬^^^^ can include a term that is dependent on a difference that measures, for each of multiple time steps ^ during the given trajectory, a difference between (i) an updated latent representation ^௧ generated by the encoder neural network and (ii) an estimated updated latent representation ^Ƹ௧ generated by the dynamics predictor neural network. The term trains the dynamics predictor neural network to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application generate estimated updated latent representations that are proximate to the updated latent representations generated by the encoder neural network. [0114] For example, the difference can be determined as a Kullback–Leibler (KL) divergence, and the dynamics objective function ^ௗ௬^^^^ can be: where sg represents a stop gradient operator applied to the parameters of the encoder neural network (i.e. during training gradients are not backpropagated into the encoder neural network), and where ^^^^ή^ refers to taking the maximum value of the arguments inside the bracket. Taking the maximum effectively disables the effect of this loss once it is reduced below a minimum level (1 nat), allowing the training to focus on the other losses, e.g. the prediction loss. [0115] For the given trajectory selected from the replay buffer 104, the training engine 110 updates the values of the parameters ^ of the encoder neural network included in the world model neural network based on optimizing a representation objective function ^^^^^^^. [0116] Like the dynamics objective function ^ௗ௬^^^^, the representation objective function ^^^^^^) can include a term that is dependent on a difference that measures, for each of multiple time steps ^ during the given trajectory, a difference between (i) an updated latent representation ^௧ generated by the encoder neural network and (ii) an estimated updated latent representation ^Ƹ௧ generated by the dynamics predictor neural network. The term trains the encoder neural network to generate updated latent representations that are proximate to the estimated updated latent representations generated by the dynamics predictor neural network [0117] For example, the difference can be determined as a Kullback–Leibler (KL) divergence, and the representation objective function ^^^^^^^ can be: where sg represents a stop gradient operator applied to the parameters of the dynamics predictor neural network. [0118] In some implementations, the training engine 110 end-to-end trains the world model neural network, i.e. the complete world model neural network 140 is trained by optimizing a combined objective function that combines the prediction objective function ^^^^ௗ^^^, the dynamics objective function ^ௗ௬^^^^, and the representation objective function ^^^^^^^ mentioned above. For example, the combined objective function can be: DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0119] In this example, ^^^^ௗ, ^ௗ௬^ǡ and ^^^^ are respective weights placed on the prediction objective function ^^^^ௗ^^^, the dynamics objective function ^ௗ௬^^^^, and the representation objective function ^^^^^^^. In implementations ^^^^ௗ, ^ௗ௬^ǡ and ^^^^ are hyperparameters of the training system 100 (some illustrative hyperparameter values are given later). [0120] FIG. 2B is an example illustration of training a policy neural network and a value neural network. For example, the policy neural network and the value neural network can correspond to the policy neural network 120 and the value neural network 130, respectively, of FIG. 1. After training the world model neural network 140, the training engine 110 trains the policy neural network 120 and the value neural network 130 on the imagination trajectories generated using the world model neural network 140. [0121] The training engine 110 performs the operations illustrated in FIG. 2B on each of multiple imagination trajectories by using the world model neural network together with the policy neural network and the value neural network using suitable reinforcement learning techniques to enable the policy neural network to generate action selection outputs that can be used to select actions that cause the agent to effectively perform one or more tasks. [0122] In some implementations, each imagination trajectory can include a tuple at each of multiple time points between ^ ൌ ^ to ^ ൌ ^ during the imagination trajectory. At a given time point ^, the tuple includes (i) an updated latent representation ^௧ characterizing a current state of the environment, (ii) an updated hidden state ^௧ characterizing the current state of the environment, (iii) a current action ^௧ selected by an action selection output, (iii) a predicted current reward ^௧, and (iv) a predicted continuation flag ^௧ that indicates whether the current state of the environment is a terminal state of the environment (e.g., the goal state of the environment that is achieved upon the completion of a task). [0123] As illustrated, an imagination trajectory includes an updated latent representation ^௧ at each of multiple time steps between ^ ൌ ^ to ^ ൌ ^ during the imagination trajectory. For the first time step ^ ൌ ^, the updated latent representation ^^̱^థ^^^ȁ^^ǡ ^^^ is generated by the encoder neural network based on a pre-determined initial hidden state of the sequence model neural network and on the current observation^^^ included in a historical trajectory which characterizes the initial state of the environment. For any DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application subsequent time step, the updated latent representation is the estimated updated latent representation generated by dynamics predictor neural network based on the updated hidden state ^௧ of the sequence model neural network. [0124] The imagination trajectories thus differ from the historical trajectories stored in the replay buffer 104 in that each imagination trajectory includes at least one updated latent representation that is not generated based on the observation, e.g., sensor data, e.g., image or video data, of the environment state. Rather, the imagination trajectory includes updated latent representations that are generated based on the hidden states of the sequence model neural network. [0125] At a high level, the training engine 110 trains the policy neural network and the value neural network using an^actor^critic technique, where the policy neural network corresponds to an actor neural network, and the value neural network corresponds to a critic neural network. [0126] At each of multiple time steps ^ during a given imagination trajectory, the policy neural network receives receive a policy network input that includes an updated latent representation ^௧ that characterizes a current state of the environment at the time step ^ and processes the policy network input to generate an action selection output ^ఏ^^௧ȁ ^௧^ that can be used to select an action ^௧ to be performed by the agent at the time step ^, where ^௧ ൌ ^^௧ǡ ^௧ ^, and can represent an imagined model state. For example, at time step ^ ൌ ʹ, the policy neural network generates an action selection output ^ఏ ^^ଶ ȁ^ଶ^ that can be used to select an action ^ଶ at the time step ^ ൌ ʹǤ [0127] At each of multiple time steps ^ during a given imagination trajectory, the value neural network receives a value network input that includes the updated latent representation ^௧ that characterizes the current state of the environment at the time step ^ and processes the value network input to generate a predicted state value ^ట(^௧). For example, at time step ^ ൌ ʹ, the value neural network generates a predicted value ^ట(^௧^ (referred to as “^ଶ” in FIG. 2B) of the current state of the environment at the time step ^ ൌ ʹǤ [0128] In implementations where the value neural network generates as output a probability distribution ^ట(^^ȁ^௧^ over a sequence ^ of ^ predetermined intervals a possible range of state value, the training engine 110 can use the symlog transformation engine 112 to apply a symmetric exponential transformation to the output of the value neural network to generate the predicted state values. In general the DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application symmetric exponential transformation is the inverse of the previously described symmetric logarithmic transformation. [0129] For example, the symmetric exponential transformation, ^^^^^^^^^, can be in the form of: ^^^^ ^ ^ ^^ ^^^ ^ȁ ^ ȁ^ െ ^ ^ or ^^^^ ^ ^ ^ ൫^^ ȁ௫ȁି^ ൯ where x is the output of the value neural network. The predicted state value for each of multiple time steps ^ during the given imagination trajectory can be computed as: [0130] For the given imagination trajectory, the training engine 110 determines a target state value for each time step ^, and updates the values of the parameters ^ of the policy neural network based on optimizing an actor objective function. The actor objective function includes a first term dependent on the target state values. The actor objective function trains the policy neural network to generate action selection outputs that can facilitate the selection of more effective actions that, in turn, can result in higher predicted state values to be computed by the value neural network and higher predicted rewards to be computed by the reward neural network. [0131] The target state value for each time step ^ can be determined based on predicted rewards generated by the reward neural network, the predicted state values generated by using the value neural network, or both the predicted rewards ^௧Ƹ and the predicted state values For example, the target state value can be determined by computing the bootstrapped ^-returns that integrate both the predicted rewards and the predicted state values: , , where ^ a discount factor, ^ ^ ^ ^ ^ is a trace decay parameter for which larger values give more credit to more distant (in time) rewards, and ^் is an imagined model state after a trajectory of T time steps. In implementations ^ and ^ are hyperparameters of the training system 100. In some other approaches, n-step returns can be used. [0132] In some implementations, the first term included in the actor objective function is dependent on downscaled target state values to better accommodate for sparse rewards. In particular, the training engine 110 scales down larger target state values without scaling up smaller target state values. This is done by dividing each target state value by a maximum of (i) a given integer value (e.g., one, two, ten, and so on), and (ii) a scaling factor ^. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0133] For example, the scaling factor ^ can be computed using respective quantiles, e.g. percentiles of one or more target state values for the latent representations in the trajectory. As a particular example, the scaling factor ^ can be computed as an exponentially decaying average of the range from their 5th to their 95th batch percentile: . Such an approach can accelerate exploration under sparse rewards without sacrificing final performance under dense rewards. Whilst using a fixed entropy scale. [0134] In some implementations, the actor objective function includes a second, entropy regularization term that is dependent on an entropy ^ of the action selection outputs ^௧^ generated by the policy neural network. The entropy ^ can be a scaled entropy that is scaled by a predetermined entropy scaling factor ^. [0135] For example, the actor objective function can be: where sg represents a stop gradient operator applied to the parameters of the world model neural network. Here the maximum value operator implements the scaling only if the returns exceed a minimum threshold, e.g.1, which facilitates using a single entropy scale (defined by a parameter ^) across dense and sparse rewards. [0136] For the given imagination trajectory, the training engine 110 updates the values of the parameters of the value neural network based on optimizing a critic objective function. The critic objective function includes a term that measures a difference between (i) the predicted state values ^ట(^௧^ and (ii) encoded transformed target state values that are generated by the symlog transformation engine 112 from the target state values. The critic objective function trains the value neural network to generate more accurate predicted state values, i.e., to generate predicted state values that more accurately represent the time- adjusted sums of predicted rewards that would be received by the agent during the given imagination trajectory. [0137] In particular, to generate the encoded transformed target state values, the training engine 110 uses the symlog transformation engine 112 to apply, for each time step in the imagination trajectory, a symmetric logarithmic transformation to the target state value to generate a transformed target state value for the time step in the imagination trajectory, and encodes the transformed target state value using a suitable encoding technique to generate an encoded transformed target state value for the time step in the imagination trajectory. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0138] As explained above, the symmetric logarithmic transformation can for example be in the form of: ^^^^ ^ ^ ^ ^^ ^ȁ ^ ȁ ^ ^ ^ or ^^^^ ^ ^ ^ ^^^ ^ȁ ^ ȁ ^ ^ ^ where x is the target state value. [0139] In implementations the encoded transformed target state value can be represented by a vector of a predetermined length that has a predetermined number of non-zero entries. For example, the training engine can use a two-hot encoding technique. The two-hot encoding technique allows the value neural network to maintain a fixed distribution over potential returns to avoid a widespread return distribution that could otherwise slow down the training. In this example, the encoded transformed target state value is represented by a vector of length ȁ^ȁ where all elements are zero expect for the two entries closest to the encoded continuous number, at positions ^ and ^ ^ ^. These two entries sum up to 1, with more weight given to the entry that is closer to the encoded number: [0140] For example, the critic objective function can be: , , where sg represents a stop gradient operator applied to the parameters of the world model neural network, and the difference between the predicted state values ^ట(^௧^ and the encoded transformed target state values is computed as a (categorical) cross entropy loss. [0141] Referring back to FIG. 1, to assist with the training of the policy neural network 120, the value neural network 130, and the world model neural network 140, the training system 100 maintains, e.g., in one or more physical or logical storage devices accessible by the training system, predetermined values for a set of hyperparameters 118. Hyperparameters are parameters of a neural network that are set before the neural network is trained. Hyperparameters might have an impact on the quality of the training of neural networks, and setting optimal values for the hyperparameters may thus be an important part of the training. [0142] In some implementations, these predetermined values can be adjusted, e.g., through an automated search process or a manual hyperparameter fine tuning, prior to the commencement of the training. In other implementations, however, these predetermined DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application values can be fixed, and the training system 100 repeatedly applies the set of hyperparameters having the same predetermined values across different tasks on which the neural networks will be trained. In those other implementations, the predetermined values for the set of hyperparameters 118 maintained by the training system 100 reduces the need for hyperparameter fine tuning, which can be time consuming and computationally expensive, and enables it to more efficiently train the neural networks across a range of different task domains. [0143] Examples of the set of hyperparameters and their predetermined values are listed below in Table 1. In Table, “number of latents” refers to the dimensionality of each latent representation (it is a 32-dimensional array that specifies a probability distribution of each of a set of 32 possible latent variables), “horizon” refers to the number of time steps included in each imagination trajectory, and EMA is the acronym for exponential moving average.
DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application Table 1. [0144] By performing the operations described in FIGS. 2A-B, the training engine 110 can determine trained values of the parameters 114 of the neural networks included in the training system 100, i.e., the parameters of the policy neural network 120, the value neural network 130, and the world model neural network 140 which, in turn, can include the sequence model neural network 150, the encoder neural network 160, the decoder neural network 165, the dynamics predictor neural network 170, the reward neural network 180, and the continuation predictor neural network 190. [0145] Once trained, some or all of the neural networks included in the training system 100 can be deployed and used to control an agent to perform one or more tasks by interacting with an environment. For example, the sequence model encoder neural network 150 and the encoder neural network 160 can operate in tandem to generate an updated DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application latent representation for each new observation. For example the sequence model encoder neural network 150 can update a hidden state of the sequence model neural network to generate an updated hidden state, ^௧, that characterizes the current state of the environment. The encoder neural network 160 can process an observation ^௧ that characterizes the current state of the environment to generate the latent representation ^௧. The policy neural network 120 can then process the latent representation ^௧ for the observation ^௧ to generate an action selection output that is used to select an action to be performed by the agent in response to the new observation. [0146] In some implementations, the training system 100 can provide, e.g., by a wired or wireless connection, the trained values of some or all of the network parameters 114 to an external system that can cause the agent to perform different actions, e.g., by directly transmitting control signals to the agent or by transmitting data identifying the actions to a control system for the agent. [0147] FIG. 3 illustrates an example data flow for using a policy neural network 320, a sequence model neural network 350, and an encoder neural network 360 to select actions 308 to be performed by an agent 302 interacting with an environment 304 at each of multiple time steps. [0148] The policy neural network 320, the sequence model neural network 350, and the encoder neural network 360 can correspond to the policy neural network 120, the sequence model encoder neural network 150, and the encoder neural network 160, respectively, of FIG. 1. The trained values of the parameters 114 can be determined (and outputted) by the training system 100 of FIG. 1. [0149] At each time step, the sequence model neural network 350 processes a sequence model network input that includes the current observation 306 in accordance with the trained values of the parameters of the sequence model neural network to generate an updated hidden state 352 that characterizes the current state of the environment 304. [0150] The encoder neural network 360 processes an encoder network input that includes the updated hidden state 352 and the current observation 306 in accordance with the trained values of the parameters of the encoder neural network to generate an updated latent representation 362 that characterizes the current state of the environment 304. [0151] The policy neural network 320 processes a policy network input that includes the updated latent representation 362 in accordance with the trained values of the parameters of the policy neural network to generate an action selection output 322 that can be used to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application select an action 308 to be performed by the agent 302 in response to the current observation 306. Examples of the action selection output and how it can be used to select the action are described above with reference to FIG. 1. [0152] In particular, during inference and as illustrated in the example FIG. 3, each received observation has a one-to-one correspondence with an updated hidden state or an updated latent representation. Note that this is different from training through latent imagination where each observation typically has a one-to-many correspondence with the updated hidden stated or the updated latent representations, i.e., during training a sequence of multiple updated hidden stated or updated latent representations are generated from a single observation. [0153] In some implementations, the environment is a real-world environment, the agent is a mechanical (or electro-mechanical) agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the agent may be a robot interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment. [0154] In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator. For example in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in 1, 2 or 3 dimensions, and may be absolute and/or relative observations. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and/or image or video data for example captured by a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0155] In these implementations, the actions may be control signals to control the robot or other mechanical agent, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements e.g. steering control elements of the vehicle, or higher-level control commands. The control signals can include for example, position, velocity, or force/torque/acceleration data for one or more joints of a robot or parts of another mechanical agent. The control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi- autonomous land or air or sea vehicle the control signals may define actions to control navigation e.g. steering, and movement e.g., braking and/or acceleration of the vehicle. [0156] In some implementations the environment is a simulation of the above-described real-world environment, and the agent is implemented as one or more computers interacting with the simulated environment. For example the simulated environment may be a simulation of a robot or vehicle and the reinforcement learning system may be trained on the simulation and then, once trained, used in the real-world. [0157] In some implementations the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product. As used herein a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material e.g. to remove pollutants, to generate a cleaned or recycled product. The manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e.g. robots, for processing solid or other materials. The manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g. via pipes or mechanical conveyance. As used herein manufacture of a product also includes manufacture of a food product by a kitchen robot. [0158] The agent may comprise an electronic agent configured to control a manufacturing unit, or a machine such as a robot, that operates to manufacture the product. That is, the agent may comprise a control system configured to control the manufacture of the chemical, biological, or mechanical product. For example the control system may be configured to control one or more of the manufacturing units or machines or to control DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application movement of an intermediate version or component of the product between the manufacturing units or machines. [0159] As one example, a task performed by the agent may comprise a task to manufacture the product or an intermediate version or component thereof. As another example, a task performed by the agent may comprise a task to control, e.g. minimize, use of a resource such as a task to control electrical power consumption, or water consumption, or the consumption of any material or consumable used in the manufacturing process. [0160] The actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product, or an intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment e.g. between the manufacturing units or machines. In general the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below. These may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. The actions may include actions imposing operating conditions on a manufacturing unit or machine, or actions that result in changes to settings to adjust, control, or switch on or off the operation of a manufacturing unit or machine. [0161] In some implementations, the reinforcement learning situation includes a reward calculation unit for generating a reward (e.g. in the form of a number), typically from the observation. The rewards may relate to a metric of performance of the task. For example in the case of a task that is to manufacture a product the metric may comprise a metric of a quantity of the product that is manufactured, a quality of the product, a speed of production of the product, or to a physical cost of performing the manufacturing task, e.g. a metric of a quantity of energy, materials, or other resources, used to perform the task. In the case of a task that is to control use a resource the metric may comprise any metric of usage of the resource. In the case of a task which is to control an electromechanical agent such as a robot to perform a manipulation of an object, the reward may indicate whether the object has been correctly manipulated according to a predefined criterion. [0162] In general, observations of a state of the environment may comprise any electronic signals representing the functioning of electronic and/or mechanical items of equipment. For example a representation of the state of the environment may be derived from DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application observations made by sensors sensing a state of the manufacturing environment, e.g. sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines. As some examples such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow/movement rate or flow/movement path of one or more materials; physical or chemical conditions e.g. a physical state, shape or configuration or a chemical state such as pH; configurations of the units or machines such as the mechanical configuration of a unit or machine, or valve configurations; image or video sensors to capture image or video observations of the manufacturing units or of the machines or movement; or any other appropriate type of sensor. In the case that the agent is a machine such as a robot the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g. data characterizing the current state of the machine or robot or of an item held or processed by the machine or robot. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal, or image or video data for example from a camera or a LIDAR sensor (e.g. mounted on the machine). Sensors such as these may be part of or located separately from the agent in the environment. [0163] In some implementations the environment is the real-world environment of a service facility comprising a plurality of items of electronic equipment, such as a server farm or data center, for example a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include ancillary control equipment that controls an operating environment of the items of equipment, for example environmental control equipment such as temperature control e.g. cooling equipment, or air flow control or air conditioning equipment. The task may comprise a task to control, e.g. minimize, use of a resource, such as a task to control electrical power consumption, or water consumption. The agent may comprise an electronic agent configured to control operation of the items of equipment, or to control operation of the ancillary, e.g. environmental, control equipment. [0164] In general the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below. These may include actions to control, or to impose operating conditions on, the items of equipment or the ancillary control equipment, e.g. actions that result in DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application changes to settings to adjust, control, or switch on or off the operation of an item of equipment or an item of ancillary control equipment. [0165] In general, the observations of a state of the environment may comprise any electronic signals representing the functioning of the facility or of equipment in the facility. For example a representation of the state of the environment may be derived from observations made by any sensors sensing a state of a physical environment of the facility or observations made by any sensors sensing a state of one or more of items of equipment or one or more items of ancillary control equipment. These include sensors configured to sense electrical conditions such as current, voltage, power or energy; a temperature of the facility; fluid flow, temperature or pressure within the facility or within a cooling system of the facility; or a physical facility configuration such as whether or not a vent is open. [0166] The rewards may relate to a metric of performance of a task relating to the efficient operation of the facility. For example in the case of a task to control, e.g. minimize, use of a resource, such as a task to control use of electrical power or water, the metric may comprise any metric of use of the resource. [0167] In some implementations the environment is the real-world environment of a power generation facility e.g. a renewable power generation facility such as a solar farm or wind farm. The task may comprise a control task to control power generated by the facility, e.g. to control the delivery of electrical power to a power distribution grid, e.g. to meet demand or to reduce the risk of a mismatch between elements of the grid, or to maximize power generated by the facility. The agent may comprise an electronic agent configured to control the generation of electrical power by the facility or the coupling of generated electrical power into the grid. The actions may comprise actions to control an electrical or mechanical configuration of an electrical power generator such as the electrical or mechanical configuration of one or more renewable power generating elements e.g. to control a configuration of a wind turbine or of a solar panel or panels or mirror, or the electrical or mechanical configuration of a rotating electrical power generation machine. Mechanical control actions may, for example, comprise actions that control the conversion of an energy input to an electrical energy output, e.g. an efficiency of the conversion or a degree of coupling of the energy input to the electrical energy output. Electrical control actions may, for example, comprise actions that control one or more of a voltage, current, frequency or phase of electrical power generated. [0168] The rewards may relate to a metric of performance of a task relating to power distribution. For example in the case of a task to control the delivery of electrical power to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application the power distribution grid the metric may relate to a measure of power transferred, or to a measure of an electrical mismatch between the power generation facility and the grid such as a voltage, current, frequency or phase mismatch, or to a measure of electrical power or energy loss in the power generation facility. In the case of a task to maximize the delivery of electrical power to the power distribution grid the metric may relate to a measure of electrical power or energy transferred to the grid, or to a measure of electrical power or energy loss in the power generation facility. [0169] In general observations of a state of the environment may comprise any electronic signals representing the electrical or mechanical functioning of power generation equipment in the power generation facility. For example a representation of the state of the environment may be derived from observations made by any sensors sensing a physical or electrical state of equipment in the power generation facility that is generating electrical power, or the physical environment of such equipment, or a condition of ancillary equipment supporting power generation equipment. Such observations may thus include observations of wind levels or solar irradiance, or of local time, date, or season. Such sensors may include sensors configured to sense electrical conditions of the equipment such as current, voltage, power or energy; temperature or cooling of the physical environment; fluid flow; or a physical configuration of the equipment; and observations of an electrical condition of the grid e.g. from local or remote sensors. Observations of a state of the environment may also comprise one or more predictions regarding future conditions of operation of the power generation equipment such as predictions of future wind levels or solar irradiance or predictions of a future electrical condition of the grid. [0170] As another example, the environment may be a chemical synthesis or protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals/intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical. As another example, the agent may be a mechanical agent that indirectly performs or controls the protein folding actions, or chemical synthesis steps, e.g. by controlling synthesis steps selected by the system automatically without human interaction. The observations may comprise direct or DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application indirect observations of a state of the protein or chemical/ intermediates/precursors and/or may be derived from simulation. Thus the system may be used to automatically synthesize a protein with a particular function such as having a binding site shape, e.g. a ligand that binds with sufficient affinity for a biological effect that it can be used as a drug. For example e.g. it may be an agonist or antagonist of a receptor or enzyme; or it may be an antibody configured to bind to an antibody target such as a virus coat protein, or a protein expressed on a cancer cell, e.g. to act as an agonist for a particular receptor or to prevent binding of another ligand and hence prevent activation of a relevant biological pathway. [0171] In a similar way the environment may be a drug design environment such that each state is a respective state of a potential pharmaceutically active compound pharmaceutically active compound and the agent is a computer system for determining elements of the pharmaceutically active compound and/or a synthetic pathway for the pharmaceutically active compound. The drug/synthesis may be designed based on a reward derived from a target for the pharmaceutically active compound, for example in simulation. As another example, the agent may be a mechanical agent that performs or controls synthesis of the pharmaceutically active compound. [0172] In some further applications, the environment is a real-world environment and the agent manages distribution of tasks across computing resources e.g. on a mobile device and/or in a data center. In these implementations, the actions may include assigning tasks to particular computing resources. In these applications, the observations may include observations of computing resources such as compute and/or memory capacity, or Internet-accessible resources; and the actions may include assigning tasks to particular computing resources. The reward(s) may be configured to maximize or minimize one or more of: utilization of computing resources, electrical power, bandwidth, and computation speed. [0173] As further example, the actions may include presenting advertisements, the observations may include advertisement impressions or a click-through count or rate, and the reward may characterize previous selections of items or content taken by one or more users. [0174] In some cases, the observations may include textual or spoken instructions provided to the agent by a third-party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide textual or spoken instructions to the agent (e.g., to navigate to a particular location). DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0175] As another example the environment may be an electrical, mechanical or electro- mechanical design environment, e.g. an environment in which the design of an electrical, mechanical or electro-mechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to work. The task may be to design the entity. The observations may comprise observations that characterize the entity, i.e. observations of a mechanical shape or of an electrical, mechanical, or electro-mechanical configuration of the entity, or observations of parameters or properties of the entity. The actions may comprise actions that modify the entity e.g. that modify one or more of the observations. The rewards may comprise one or more metrics of performance of the design of the entity. For example rewards may relate to one or more physical characteristics of the entity such as weight or strength or to one or more electrical characteristics of the entity such as a measure of efficiency at performing a particular function for which the entity is designed. The design process may include outputting the design for manufacture, e.g. in the form of computer executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus a design of an entity may be optimized, e.g. by reinforcement learning, and then the optimized design output for manufacturing the entity, e.g. as computer executable instructions; an entity with the optimized design may then be manufactured. [0176] As previously described the environment may be a simulated environment. Generally in the case of a simulated environment the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions. For example the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle. Generally the agent may be implemented as one or more computers interacting with the simulated environment. [0177] The simulated environment may be a simulation of a particular real-world environment and agent. For example, the system may be used to select actions in the simulated environment during training or evaluation of the system and, after training, or evaluation, or both, are complete, the action selection policy may be deployed for controlling a real-world agent in the particular real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear on and damage to the real- DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application world environment or real-world agent and can allow the control neural network to be trained and evaluated on situations that occur rarely or are difficult or unsafe to re-create in the real-world environment. For example the system may be partly trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and afterwards deployed to control the real mechanical agent in the particular real-world environment. Thus in such cases the observations of the simulated environment relate to the real-world environment, and the selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment. [0178] In some implementations, as described above, the agent may not include a human being (e.g. it is a robot). Conversely, in some implementations the agent comprises a human user of a digital assistant such as a smart speaker, smart display, or other device. Then the information defining the task can be obtained from the digital assistant, and the digital assistant can be used to instruct the user based on the task. [0179] For example, the reinforcement learning system may output to the human user, via the digital assistant, instructions for actions for the user to perform at each of a plurality of time steps. The instructions may for example be generated in the form of natural language (transmitted as sound and/or text on a screen) based on actions chosen by the reinforcement learning system. The reinforcement learning system chooses the actions such that they contribute to performing a task. A monitoring system (e.g. a video camera system) may be provided for monitoring the action (if any) which the user actually performs at each time step, in case (e.g. due to human error) it is different from the action which the reinforcement learning system instructed the user to perform. Using the monitoring system the reinforcement learning system can determine whether the task has been completed. During an on-policy training phase and/or another phase in which the history database is being generated, the experience tuples may record the action which the user actually performed based on the instruction, rather than the one which the reinforcement learning system instructed the user to perform. The reward value of each experience tuple may be generated, for example, by comparing the action the user took with a corpus of data showing a human expert performing the task, e.g. using techniques known from imitation learning. Note that if the user performs actions incorrectly (i.e. performs a different action from the one the reinforcement learning system instructs the user to perform) this adds one more source of noise to sources of noise which may already exist in the environment. During the training process the reinforcement learning system may identify actions which the user performs incorrectly with more than a certain DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application probability. If so, when the reinforcement learning system instructs the user to perform such an identified action, the reinforcement learning system may warn the user to be careful. Alternatively or additionally, the reinforcement learning system may learn not to instruct the user to perform the identified actions, i.e. ones which the user is likely to perform incorrectly. [0180] More generally, the digital assistant instructing the user may comprise receiving, at the digital assistant, a request from the user for assistance and determining, in response to the request, a series of tasks for the user to perform, e.g. steps or sub-tasks of an overall task. Then for one or more tasks of the series of tasks, e.g. for each task, e.g. until a final task of the series the digital assistant can be used to output to the user an indication of the task, e.g. step or sub-task, to be performed. This may be done using natural language, e.g. on a display and/or using a speech synthesis subsystem of the digital assistant. Visual, e.g. video, and/or audio observations of the user performing the task may be captured, e.g. using the digital assistant. A system as described above may then be used to determine whether the user has successfully achieved the task e.g. step or sub-task, i.e. from the answer as previously described. If there are further tasks to be completed the digital assistant may then, in response, progress to the next task (if any) of the series of tasks, e.g. by outputting an indication of the next task to be performed. In this way the user may be led step-by-step through a series of tasks to perform an overall task. During the training of the neural network, training rewards may be generated e.g. from video data representing examples of the overall task (if corpuses of such data are available) or from a simulation of the overall task. [0181] In a further aspect there is provided a digital assistant device including a system as described above. The digital assistant can also include a user interface to enable a user to request assistance and to output information. In implementations this is a natural language user interface and may comprise a keyboard, voice input-output subsystem, and/or a display. The digital assistant can further include an assistance subsystem configured to determine, in response to the request, a series of tasks for the user to perform. In implementations this may comprise a generative (large) language model, in particular for dialog, e.g. a conversation agent such as Sparrow or Chinchilla. The digital assistant can have an observation capture subsystem to capture visual and/or audio observations of the user performing a task; and an interface for the above-described language model neural network (which may be implemented locally or remotely). The digital assistant can also have an assistance control subsystem configured to assist the user. The assistance control DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application subsystem can be configured to perform the steps described above, for one or more tasks e.g. of a series of tasks, e.g. until a final task of the series. More particularly the assistance control subsystem and output to the user an indication of the task to be performed, capture, using the observation capture subsystem, visual or audio observations of the user performing the task, determine from the above-described answer whether the user has successfully achieved the task. In response the digital assistant can progress to a next task of the series of tasks and/or control the digital assistant, e.g. to stop capturing observations. [0182] In some implementations, the environment may not include a human being or animal. In other implementations, however, it may comprise a human being or animal. For example, the agent may be an autonomous vehicle in an environment which is a location (e.g. a geographical location) where there are human beings (e.g. pedestrians or drivers/passengers of other vehicles) and/or animals, and the autonomous vehicle itself may optionally contain human beings. The environment may also be at least one room (e.g. in a habitation) containing one or more people. The human being or animal may be an element of the environment which is involved in the task, e.g. modified by the task (indeed, the environment may substantially consist of the human being or animal). For example the environment may be a medical or veterinary environment containing at least one human or animal subject, and the task may relate to performing a medical (e.g. surgical) procedure on the subject. In a further implementation, the environment may comprise a human user who interacts with an agent which is in the form of an item of user equipment, e.g. a digital assistant. The item of user equipment provides a user interface between the user and a computer system (the same computer system(s) which implement the reinforcement learning system, or a different computer system). The user interface may allow the user to enter data into and/or receive data from the computer system, and the agent is controlled by the action selection policy to perform an information transfer task in relation to the user, such as providing information about a topic to the user and/or allowing the user to specify a component of a task which the computer system is to perform. For example, the information transfer task may be to teach the user a skill, such as how to speak a language or how to navigate around a geographical location; or the task may be to allow the user to define a three-dimensional shape to the computer system, e.g. so that the computer system can control an additive manufacturing (3D printing) system to produce an object having the shape. Actions may comprise outputting information to the user (e.g. in a certain format, at a certain rate, etc.) and/or configuring the interface to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application receive input from the user. For example, an action may comprise setting a problem for a user to perform relating to the skill (e.g. asking the user to choose between multiple options for correct usage of the language, or asking the user to speak a passage of the language out loud), and/or receiving input from the user (e.g. registering selection of one of the options, or using a microphone to record the spoken passage of the language). Rewards may be generated based upon a measure of how well the task is performed. For example, this may be done by measuring how well the user learns the topic, e.g. performs instances of the skill (e.g. as measured by an automatic skill evaluation unit of the computer system). In this way, a personalized teaching system may be provided, tailored to the aptitudes and current knowledge of the user. In another example, when the information transfer task is to specify a component of a task which the computer system is to perform, the action may comprise presenting a (visual, haptic or audio) user interface to the user which permits the user to specify an element of the component of the task, and receiving user input using the user interface. The rewards may be generated based on a measure of how well and/or easily the user can specify the component of the task for the computer system to perform, e.g. how fully or well the three-dimensional object is specified. This may be determined automatically, or a reward may be specified by the user, e.g. a subjective measure of the user experience. In this way, a personalized system may be provided for the user to control the computer system, again tailored to the aptitudes and current knowledge of the user. [0183] Optionally, in any of the above implementations, the observation at any given time step may include data from a previous time step that may be beneficial in characterizing the environment, e.g., the action performed at the previous time step, the reward received at the previous time step, or both. [0184] FIG. 4 is a flow diagram of an example process 400 for training a policy neural network and a value neural network. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 400. [0185] The system can repeatedly perform iterations of the process 400 on different imagination trajectories generated by using a world model neural network to update the parameters of the policy neural network and the value neural network. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0186] The policy neural network is configured to process a policy network input that includes an updated latent representation in accordance with current values of the policy network parameters to generate an action selection output. The value neural network is configured to process a value network input that includes an updated latent representation in accordance with current values of the value network parameters to generate a predicted state value. [0187] The system can continue performing iterations of the process 400 until termination criteria for the training of the neural networks have been satisfied, e.g., until the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of the process 400 have been performed. [0188] The system receives a latent representation that is in a latent space and that characterizes a current state of the environment (step 402). The latent representation can be generated by the world model neural network based on a current observation (e.g., an image or a video frame) of the current state of the environment. The world model neural network can include a sequence model neural network, an encoder neural network, a decoder neural network. a dynamics predictor neural network, a reward neural network, and a continuation predictor neural network. When the current state is an initial state of the environment, the latent representation can be generated by the encoder neural network based on a pre-determined initial hidden state of the sequence model neural network and on the current observation. [0189] The system generates an imagination trajectory that includes an updated latent representation at each of multiple time steps during the imagination trajectory (step 404). The imagination trajectory starts with the received latent representation. That is, at the first time step, the updated latent representation is the received latent representation. In brief, and as will be described in more detail below with reference to FIG. 5, generating each imagination trajectory can include repeatedly generating an action selection output by using the policy neural network, selecting an action that would be performed by the agent by using the action selection output, and then using the world model neural network to generate an updated latent representation based at least on the selected action. [0190] For each updated latent representation in the imagination trajectory, the system determines a predicted reward (step 406) and determines a predicted state value (step 408). The predicted state value represents a predicted value of the environment being in the state characterized by the updated latent representation in terms of a cumulative measure of predicted rewards to be received by the agent. For example, the predicted state value can DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application specify a long-term time-adjusted, e.g., time-discounted, sum of predicted rewards that would be received by the agent by selecting actions using action selection outputs starting from the state characterized by the updated latent representation. [0191] In more detail, for each updated latent representation in the imagination trajectory, the system can determine a predicted reward by processing the updated latent representation and an updated hidden state of the sequence model neural network using a reward neural network in accordance with current values of the reward network parameters to generate the predicted reward, which can involve processing the current latent representation and the selected action using a sequence model neural network to generate the updated hidden state. Likewise, the system can determine a predicted state value by processing the updated latent representation using the value neural network to generate the predicted state value. [0192] The system determines a corresponding target state value for each updated latent representation in the imagination trajectory based on the predicted rewards and the predicted state values for the updated latent representations in the trajectory (step 410). For example, the target state value can be determined by computing the bootstrapped ^- returns that integrate both the predicted rewards and the predicted state values. An example way of computing the bootstrapped ^-returns is described above with reference to FIG. 2B. [0193] The system determines an update to the current values of the policy network parameters by optimizing an actor objective function (step 412). In some implementations, the actor objective function includes a first term that is dependent on the target state values. In some implementations, the actor objective function includes a second term that is dependent on an entropy of the action selection outputs generated by the policy neural network. An example of the actor objective function is described above with reference to FIG. 2B. [0194] The system can determine the update by computing, e.g., through backpropagation, respective gradients of the actor objective function with respect to the policy network parameters, and then applying an update rule, e.g., an Adam update rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule, to the respective gradients. [0195] The system applies a symmetric logarithmic transformation to each target state value to generate a transformed target state value for each updated latent representation in the imagination trajectory (step 414). DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0196] The system encodes each transformed target state value to generate an encoded transformed target state value for each updated latent representation imagination in the trajectory (step 416). For example, the system can use a two-hot encoding technique to generate the encoded transformed target state values that can each be represented by a vector of a predetermined length having a predetermined number of non-zero entries. [0197] The system determines an update to the current values of the value network parameters by optimizing a critic objective function (step 418). The critic objective function includes a term that measures a difference between the predicted state values and the encoded transformed target state values. An example of the critic objective function is described above with reference to FIG. 2B. [0198] The system can determine the update by computing, e.g., through backpropagation, respective gradients of the critic objective function with respect to the value network parameters, and then applying an update rule, e.g., an Adam update rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule, to the respective gradients. [0199] FIG. 5 is a flow diagram of an example process 500 for generating an imagination trajectory of latent representations. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 500. [0200] The system can repeatedly perform the process 500 to generate an imagination trajectory that includes an updated latent representation at each of multiple time steps during the imagination trajectory. The imagination trajectory starts with the latent representation received at step 402 of process 400. That is, at the first time step, the updated latent representation is the received latent representation. [0201] The system processes a current latent representation using the policy neural network and in accordance with current values of the policy network parameters to generate an action selection output (step 502). For the first time step, the current latent representation is the received latent representation. For any subsequent time step, the current latent representation is the updated latent representation that has been generated in the preceding time step. [0202] The system selects, using the action selection output, an action that would be performed by the agent when the environment is in a state characterized by the current DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application latent representation (step 504). Selecting the action by using the action selection output is similarly described above with reference to FIG. 1. [0203] The system generates an updated latent representation based on the current latent representation and the selected action (step 506). The updated latent representation characterizes a state that the environment would transition into if the agent performed the selected action when the environment is in a state characterized by the current latent representation. [0204] To do this, the system processes the current latent representation and the selected action using the sequence model neural network and in accordance with the current values of sequence model network parameters to generate an updated hidden state. The system then processes the updated hidden state using the dynamics predictor neural network to generate the updated latent representation. [0205] Prior to the training of the policy neural network and value neural network by performing processes 400 and 500, the system trains the world model neural network to ensure the quality of the imagination trajectories. Specifically, the system trains the encoder neural network and the dynamics predictor neural network to encourage the generation of high-quality latent representations, e.g., latent representations that can capture salient (e.g., task-specific) information contained in observation of the environment, or latent representations that are predictive of subsequent states of the environment. The system also trains the sequence model neural network to more accurately simulate latent dynamics of the environment, i.e., state transitions of the environment that would be caused by the agent performing the proposed actions, trains the reward neural network to more accurately estimate rewards that would be received by the agent from the environment in response to performing the selected actions, and trains the continuation predictor neural network to more accurately predict whether a current state is a terminal state of the environment. [0206] FIG. 6 is a flow diagram of an example process 600 for training a world model neural network. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 of FIG. 1, appropriately programmed in accordance with this specification, can perform the process 600. [0207] The system can sample historical trajectories from a replay buffer and train the world model neural network on the sampled historical trajectories. The replay buffer can be any available memory that stores historical trajectories generated as a consequence of DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application the interaction of the agent (or another agent) with the environment (or with another instance of the environment) to attempt to perform one or more tasks. [0208] In some implementations, each historical trajectory can include a tuple at each of multiple time points between during the historical trajectory. At a given time point, the tuple includes (i) a current observation characterizing a current state of the environment, (ii) a current action performed by the agent, (iii) a current reward received by the agent, and (iv) a ground truth continuation flag that indicates whether the current state of the environment is a terminal state of the environment (e.g., the goal state of the environment that is achieved upon the completion of a task). [0209] The system can repeatedly perform iterations of the process 600 on different historical trajectories sampled from the replay buffer to update the parameters of the world model neural network. The system can continue performing iterations of the process 600 until termination criteria for the training of the world model neural network have been satisfied, e.g., until the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of the process 600 have been performed. [0210] In general, the system can perform the following steps 602-612 at each of multiple time steps during the historical trajectory. [0211] The system processes (i) a current latent representation that is generated at a preceding time step and that characterizes a previous state of the environment and (ii) data specifying a previously selected action selected at the preceding time step using the sequence model neural network in accordance with current values of sequence model network parameters to generate an updated hidden state that characterizes the current state of the environment (step 602). [0212] The system processes the updated hidden state and the current observation^of the current state of the environment using the encoder neural network in accordance with current values of encoder network parameters to generate an updated latent representation that characterizes the current state of the environment (step 604). [0213] The system processes the updated hidden state using the dynamics predictor neural network in accordance with current values of dynamics predictor network parameters to generate an estimated updated latent representation (step 606). The estimated updated latent representation is an estimate of the updated latent representation that characterizes the current state of the environment and that can be generated by the encoder neural network. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0214] The system processes the updated latent representation and, optionally, the updated hidden state using the decoder neural network in accordance with current values of decoder network parameters to generate a reconstructed current observation of the current state of the environment (step 608). [0215] The system processes the updated latent representation and, optionally, the updated hidden state using the reward neural network in accordance with current values of reward network parameters to generate a predicted reward (step 610). The predicted reward is a prediction of the reward that would be received by the agent when the environment is in the current state. [0216] The system processes the updated latent representation and, optionally, the updated hidden state using the continuation predictor neural network in accordance with current values of decoder network parameters to generate a predicted continuation predictor flag that indicates whether the current state is a terminal state in the trajectory (step 612). [0217] The system determines an update to the current values of the parameters of the world model neural network based on optimizing a combined objective function (step 614). Generally, the combined objective function measures a difference between the outputs of the neural networks and the ground truth information included in the historical trajectory. An example of the combined objective function is described above with reference to FIG. 2A. [0218] More particularly, determining an update to the current values of the parameters of the world model neural network can involve determining, based on computing a gradient of a dynamics objective function with respect to the sequence model neural network parameters and the dynamics predictor neural network parameters, an update to the current values of the sequence model neural network parameters and the dynamics predictor neural network parameters. The dynamics objective function is dependent on a Kullback– Leibler divergence between a latent representation generated by the encoder neural network and (ii) a latent representation generated by the dynamics predictor neural network. Here the sequence model neural network parameters are parameters of a sequence model neural network that is configured to process the current latent representation and the selected action to generate an updated hidden state of the sequence model neural network. The dynamics predictor neural network parameters are parameters of a dynamics predictor neural network that is configured to process the updated hidden state to generate the updated latent representation. The encoder neural network is DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application configured to process an observation to generate the latent representation that characterizes the current state of the environment. [0219] The system can determine the update by computing, e.g., through backpropagation, respective gradients of the combined objective function with respect to the appropriate network parameters, and then applying an update rule, e.g., an Adam update rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule, to the respective gradients. [0220] During the training, the system can incorporate any number of techniques to improve the speed, the effectiveness, or both of the training process. For example, the system can refrain from using hyperparameter annealing, weight decay, or dropout. As another example, the system can initialize the values of the parameters of some of the neural network mentioned above, e.g., the reward network parameters and the value network parameters, to zeros. This can accelerate early learning by alleviating the problem of the delay in the onset of training caused by large magnitude predicted rewards, which might otherwise occur if the parameters were randomly initialized at the start of training. [0221] It will be appreciated that the techniques for training an encoder-decoder neural network can be applied in any of a wider variety of tasks, e.g., tasks that may not involve controlling an agent. As a general example, the encoder-decoder neural network can be configured to receive an input data item, process the input data item to generate a latent representation of the input data item, and to process the latent representation of the input data item to generate a reconstruction of the input data item. [0222] The encoder-decoder neural network can operate on any of a variety of different types of data items, e.g., images, videos, audio signals, and so on. As a particular example, the encoder-decoder neural network can operate on high-dimensional (and therefore large) data items. For example, the encoder-decoder neural network can be used to compress and reconstruct high-resolution images, e.g., medical images and geographical images. In more detail, the input data item may comprise an image of a real- world entity captured by an image sensor such as a camera or LIDAR sensor or captured a medical image sensor such as an X-ray or MRI machine. The latent representation of the input data item constitutes a compressed version of the input data item that may be, e.g. stored and/or transmitted over a communications network. The compressed version of the input data item may be retrieved from storage and/or received over the communications network and the input data item reconstructed. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0223] FIG. 7 is a flow diagram of an example process 700 for training an encoder- decoder neural network. For convenience, the process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 of FIG. 1 or another training system that includes a symlog transformation engine, appropriately programmed in accordance with this specification, can perform the process 700. [0224] The system receives a training data item (step 702). [0225] The system processes the training data item using the encoder-decoder neural network to determine a latent representation of the training data item and to generate a reconstruction of the training data item in accordance with current values of the parameters of the encoder-decoder neural network (step 704). The latent representation is a lower-dimension, i.e., more compact, version of the training data item. The encoder- decoder neural network includes an encoder neural network and a decoder neural network. [0226] More specifically, the system processes the training data item using the encoder neural network to determine a latent representation of the training data item. The system processes the latent representation of the training data item using the decoder neural network to generate an output of the encoder-decoder neural network. The system then applies a symmetric exponential transformation to the output of the encoder-decoder neural network to generate the reconstruction of the training data item. For example, the symmetric exponential transformation can be in the form of: ^^^^^^^^^^^^ȁ^ȁ^ െ ^^ or ^^^^^^^൫^^ȁ௫ȁି^൯, where x is the output of the encoder-decoder neural network. [0227] The system applies a symmetric logarithmic transformation to the training data item to generate a transformed training data item (step 706). For example, the symmetric logarithmic transformation is in the form of: ^^^^^^^ ^^^ȁ^ȁ ^ ^^ or ^^^^^^^ ^^^^ȁ^ȁ ^ ^^, where x is the training data item. [0228] The system determines an update to the current values of the encoder-decoder network parameters based on evaluating a loss function that measures a difference between the reconstruction of the training data item and the transformed training data item (step 708). For example, the loss function can be a mean squared error loss function. The system can determine the update by computing, e.g., through backpropagation, respective gradients of the combined objective function with respect to the parameters of the encoder-decoder neural network, and then applying an update rule, e.g., an Adam update DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application rule, an AdamW update rule, an Adagrad update rule, or an RMSProp update rule, to the respective gradients. [0229] FIG. 8 shows a quantitative example of the performance gains that can be achieved by using the neural networks described in this specification across a large number of tasks that span 7 different domains that include continuous and discrete actions, visual and low- dimensional inputs, dense and sparse rewards, different reward scales, 2D and 3D worlds, and procedural generation. [0230] The tasks include proprioceptive sensor-based control tasks and visual control tasks (described in arXiv:1801.00690), Atari tasks (described in arXiv:1903.00374 and arXiv:1207.4708), Behavior Suite tasks, or bsuite tasks for short (described in arXiv:1908.03568), and Crafter tasks (described in arXiv:2109.06780). [0231] The vertical axes in FIG.8 show the returns received by an agent when controlled using each of multiple neural network systems, where higher returns indicate better performance. In FIG. 8, the DreamerV3 system corresponds to a system that includes the neural networks described in this specification. The other systems are existing reinforcement learning systems for agent control, e.g., the MPO system is described in arXiv:1806.06920, the DDPG system is described in arXiv:1509.02971, and D4PG system is described in arXiv:1804.08617. It will be appreciated that DreamerV3 outperforms the existing systems on all of these tasks. Notably, the DreamerV3 system can achieve this performance gain across these tasks after having been trained using the same values for the set of hyperparameters as listed above in Table. 1. [0232] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. [0233] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. [0234] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. [0235] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. [0236] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application index database can include multiple collections of data, each of which may be organized and accessed differently. [0237] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers. [0238] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. [0239] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. [0240] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application [0241] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return. [0242] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads. [0243] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a JAX framework. [0244] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. [0245] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device. [0246] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. [0247] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. [0248] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application CLAIMS 1. A method of training a policy neural network having a plurality of policy network parameters used to select actions to be performed by an agent in an environment, the method comprising: receiving a latent representation that is in a latent space and that characterizes a current state of the environment; generating an imagination trajectory of latent representations that starts with the received latent representation by performing the following at each of a plurality of time steps: processing a current latent representation using the policy neural network and in accordance with current values of the policy network parameters to generate an action selection output; selecting, using the action selection output, an action that would be performed by the agent when the environment is in a state characterized by the current latent representation; and processing the current latent representation and the selected action to generate an updated latent representation that characterizes a state that the environment would transition into if the agent performed the selected action when the environment is in a state characterized by the current latent representation; for each latent representation in the imagination trajectory: determining a predicted reward; and processing the latent representation using a value neural network having a plurality of value network parameters and in accordance with current values of the value network parameters to generate a predicted state value that represents a time-adjusted sum of predicted rewards that would be received by the agent by selecting actions using action selection outputs starting from the state characterized by the latent representation; determining a target state value for each latent representation in the imagination trajectory from the predicted rewards and the predicted state values for the latent representations in the imagination trajectory; determining an update to the current values of the policy network parameters by optimizing an actor objective function that includes a first term dependent on the target state values; applying a symmetric logarithmic transformation to each target state value to DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application generate a transformed target state value for each latent representation in the imagination trajectory; encoding each transformed target state value to generate an encoded transformed target state value for each latent representation in the imagination trajectory; and determining an update to the current values of the value network parameters by optimizing a critic objective function that measures a difference between the predicted state values and the encoded transformed target state values. 2. The method of claim 1, wherein the symmetric logarithmic transformation is in the form of: ^^^^ ^ ^ ^ ^^ ^ȁ ^ ȁ ^ ^ ^ or ^^^^ ^ ^ ^ ^^^ ^ȁ ^ ȁ ^ ^ ^ , where x is the target state value. 3. The method of any one of claims 1-2, wherein the encoded transformed target state value is represented by a vector of a predetermined length having a predetermined number of non-zero entries. 4. The method of any one of claims 1-3, wherein generating the predicted state value for each latent representation in the imagination trajectory comprises: processing the latent representation to assign a respective probability to each predetermined interval in a sequence of predetermined intervals between a possible range of state values; generating an output of the value neural network by computing a weighted average of the state values represented by the sequence of predetermined intervals weighted by the respective probabilities; and applying a symmetric exponential transformation to the output of the value neural network to generate the predicted state value for the latent representation in the imagination trajectory. 5. The method of claim 4, wherein the symmetric exponential transformation is in the form of: ^^^^^^^^^^^^ȁ^ȁ^ െ ^^ or ^^^^^^^൫^^ȁ௫ȁି^൯, where x is the output of the value neural network. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application 6. The method of any one of claims 1-5, wherein determining the update to the current values of the policy network parameters by optimizing the actor objective function comprises: generating, by using a scaling factor and from the target state values, a downscaled target state value for each latent representation in the imagination trajectory; and determining the update to the current values of the policy network parameters by optimizing the actor objective function that includes the first term dependent on the downscaled target state values. 7. The method of claim 6, wherein generating the downscaled target state value comprises: dividing the target state value by a maximum of a given integer value and the scaling factor, wherein the scaling factor is computed using respective percentiles of one or more target state values for the latent representations in the imagination trajectory. 8. The method of any one of claims 1-7, wherein the actor objective function includes a second term dependent on a scaled entropy of the action selection outputs generated by the policy neural network, wherein the scaled entropy is scaled by a predetermined entropy scaling factor. 9. The method of any one of claims 1-8, wherein processing the current latent representation and the selected action to generate the updated latent representation comprises: processing the current latent representation and the selected action using a sequence model neural network in accordance with current values of sequence model network parameters to update a hidden state of the sequence model neural network to generate an updated hidden state that characterizes the current state of the environment; and processing the updated hidden state using a dynamics predictor neural network in accordance with current values of dynamics predictor network parameters to generate the updated latent representation. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application 10. The method of claim 9, wherein receiving the latent representation that characterizes the current state of the environment comprises: processing an observation that characterizes the current state of the environment using an encoder neural network in accordance with current values of encoder network parameters to generate the latent representation. 11. The method of claim 10, further comprising: processing the latent representation and the updated hidden state using a decoder neural network in accordance with current values of decoder network parameters to generate a reconstruction of the observation; applying the symmetric logarithmic transformation to the observation to generate a transformed observation; and determining, based on computing a gradient of a prediction objective function that includes a first term measuring a difference between the reconstruction of the observation and the transformed observation and with respect to the decoder network parameters, an update to the current values of the decoder network parameters. 12. The method of any one of claims 1-11, wherein for each latent representation in the imagination trajectory, determining the predicted reward comprises: processing the latent representation and an updated hidden state of the sequence model neural network using a reward neural network in accordance with current values of reward network parameters to generate the predicted reward. 13. The method of claim 12, further comprising: applying the symmetric logarithmic transformation to a reward received by the agent when the environment is at the state characterized by the latent representation to generate a transformed reward; and determining, based on computing a gradient of the prediction objective function that includes a second term measuring a difference between the predicted reward and the transformed reward and with respect to the reward network parameters, an update to the current values of the reward network parameters. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application 14. The method of any one of claims 1-13, further comprising: determining, based on computing a gradient of the dynamics objective function with respect to the sequence model neural network parameters and the dynamics predictor neural network parameters, an update to the current values of the sequence model neural network parameters and the dynamics predictor neural network parameters, wherein the dynamics objective function is dependent on a Kullback–Leibler divergence between (i) a latent representation generated by the encoder neural network and (ii) a latent representation generated by the dynamics predictor neural network. 15. The method of any one of claims 1-14, wherein: the encoder neural network and the decoder neural network are each configured as a respective convolutional neural network or multi-layer perceptron (MLP); the dynamics predictor neural network and the reward neural network are each configured as a multi-layer perceptron (MLP); and the sequence model neural network is configured as a recurrent neural network. 16. The method of claim 15, wherein the convolutional neural network, the multi-layer perceptron (MLP), or the recurrent neural network comprise: one or more layer normalization layers and one or more sigmoid linear unit (SiLU) activation layers. 17. The method of any one of claims 15-16, further comprising initializing the current values of the reward network parameters and the value network parameters to zeros. 18. The method of any one of claims 1-17, further comprising using the policy neural network to select actions to be performed by the agent in the environment by processing an input derived from data characterizing the current state of the environment at a time step, using the policy neural network, to generate an action selection output used to determine an action to be performed by the agent at the time step. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application 19. A method performed by one or more computers for training an encoder-decoder neural network, wherein the encoder-decoder neural network is configured to receive an input data item, process the input data item to generate a latent representation of the input data item, and to process the latent representation of the input data item to generate a reconstruction of the input data item, and wherein the method comprises: receiving a training data item; processing the training data item using the encoder-decoder neural network to determine a latent representation of the training data item and to generate a reconstruction of the training data item in accordance with current values of encoder- decoder network parameters; applying a symmetric logarithmic transformation to the training data item to generate a transformed training data item; and determining an update to the current values of the encoder-decoder network parameters based on evaluating a loss function that measures a difference between the reconstruction of the training data item and the transformed training data item. 20. The method of claim 18, wherein the symmetric logarithmic transformation is in the form of: ^^^^ ^ ^ ^ ^^ ^ȁ ^ ȁ ^ ^ ^ or ^^^^ ^ ^ ^ ^^^ ^ȁ ^ ȁ ^ ^ ^ , where x is the training data item. 21. The method of any one of claims 18-19, wherein the loss function is a mean squared error loss function. 22. The method of any one of claims 18-20, wherein generating the reconstruction of the input data item comprises: processing the training data item to determine the latent representation of the training data item; processing the latent representation of the training data item to generate an output of the encoder-decoder neural network; and applying a symmetric exponential transformation to the output of the encoder- decoder neural network to generate the reconstruction of the training data item. DeepMind Technologies Limited F&R Ref.45288-0301WO1 PCT Application 23. The method of claim 21, wherein the symmetric exponential transformation is in the form of: ^^^^ ^ ^ ^^ ^^^ ^ȁ ^ ȁ^ െ ^ ^ or ^^^^ ^ ^ ^ ൫^^ ȁ௫ȁି^ ൯, where x is the output of the encoder-decoder neural network. 24. The method of any one of claims 18-22, wherein the input data item comprises an image. 25. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of any preceding claim. 26. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any preceding claim.
EP24700412.0A 2023-01-09 2024-01-09 Training reinforcement learning agents to perform multiple tasks across diverse domains Pending EP4639415A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202363437931P 2023-01-09 2023-01-09
PCT/EP2024/050380 WO2024149747A1 (en) 2023-01-09 2024-01-09 Training reinforcement learning agents to perform multiple tasks across diverse domains

Publications (1)

Publication Number Publication Date
EP4639415A1 true EP4639415A1 (en) 2025-10-29

Family

ID=89619791

Family Applications (1)

Application Number Title Priority Date Filing Date
EP24700412.0A Pending EP4639415A1 (en) 2023-01-09 2024-01-09 Training reinforcement learning agents to perform multiple tasks across diverse domains

Country Status (5)

Country Link
EP (1) EP4639415A1 (en)
JP (1) JP2026504213A (en)
KR (1) KR20250133389A (en)
CN (1) CN120836033A (en)
WO (1) WO2024149747A1 (en)

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20250377668A1 (en) * 2024-06-05 2025-12-11 The Boeing Company Decentralized multi-agent actor-critic reinforcement learning model for controlling autonomous vehicles in multi-vehicle environments
CN119337960A (en) * 2024-09-27 2025-01-21 清华大学 Offline reinforcement learning method and device based on time reversal symmetry
CN119090492B (en) * 2024-11-11 2025-05-13 杭州市北京航空航天大学国际创新研究院(北京航空航天大学国际创新学院) Cluster system intelligent operation and maintenance system based on reinforcement learning
CN119126988B (en) * 2024-11-14 2025-03-18 北京航空航天大学 Human-computer interaction likelihood evaluation method, device and medium based on deep RNN
CN119539022A (en) * 2024-11-18 2025-02-28 言生行(北京)科技有限公司 A reward-based action model optimization method
CN119322459B (en) * 2024-12-19 2025-03-21 衡阳山泰化工有限公司 A control system and method for continuous production line of polymerization reaction
CN119740630B (en) * 2024-12-23 2025-10-31 北京航空航天大学 Multi-agent reinforcement learning strategy optimization method based on intrinsic reward network
CN119476305B (en) * 2025-01-13 2025-03-18 贵阳康养职业大学 A method and system for processing semantics of consultation dialogues for smart health care
CN120105000B (en) * 2025-02-11 2025-11-14 武汉大学 A method and equipment for forecasting heavy precipitation events using prior diffusion networks based on multi-source meteorological data.
CN120029697B (en) * 2025-03-05 2026-03-24 中国人民解放军92941部队 A real-time performance indicator evaluation system, method, and procedure product
CN120355146B (en) * 2025-04-03 2025-11-18 山东华越智能科技有限公司 A Dynamic Allocation Method for Terminal Operation Resources Based on Multi-Objective Optimization Algorithm
CN120072278B (en) * 2025-04-28 2025-08-08 江西展群科技有限公司 A data processing method and system based on cataract detection
CN120111734B (en) * 2025-05-09 2025-10-03 煤炭科学技术研究院有限公司 A mining lamp LED brightness adjustment method and system based on dynamic fuzzy PID control
CN120326636B (en) * 2025-06-17 2025-10-10 中国科学院自动化研究所 Humanoid robot welding posture adjustment method and system
CN120568501B (en) * 2025-07-31 2025-10-10 联通沃音乐文化有限公司 5G network resource allocation method, system and equipment
CN121677123A (en) * 2026-02-09 2026-03-17 广东三雄极光照明股份有限公司 A method and system for coordinated control of air conditioning terminals based on convolutional neural networks and channel attention mechanism

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020062911A1 (en) * 2018-09-26 2020-04-02 Huawei Technologies Co., Ltd. Actor ensemble for continuous control
US12533800B2 (en) * 2019-11-27 2026-01-27 Google Llc Training reinforcement learning agents to learn farsighted behaviors by predicting in latent space

Also Published As

Publication number Publication date
JP2026504213A (en) 2026-02-03
KR20250133389A (en) 2025-09-05
CN120836033A (en) 2025-10-24
WO2024149747A1 (en) 2024-07-18

Similar Documents

Publication Publication Date Title
WO2024149747A1 (en) Training reinforcement learning agents to perform multiple tasks across diverse domains
US12572803B2 (en) Multi-agent reinforcement learning with matchmaking policies
JP2023511630A (en) Planning for Agent Control Using Learned Hidden States
GB2629890A (en) Controlling agents using sub-goals generated by language model neural networks
EP3864581A1 (en) Controlling agents over long time scales using temporal value transport
US20230083486A1 (en) Learning environment representations for agent control using predictions of bootstrapped latents
WO2019170905A1 (en) Training an unsupervised memory-based prediction system to learn compressed representations of an environment
US20250209338A1 (en) Exploration by bootstrapped prediction
WO2022167657A2 (en) Attention neural networks with short-term memory units
US20250093828A1 (en) Training a high-level controller to generate natural language commands for controlling an agent
EP4384953A1 (en) Retrieval augmented reinforcement learning
US20240232642A1 (en) Reinforcement learning using epistemic value estimation
US20250124297A1 (en) Controlling reinforcement learning agents using geometric policy composition
US20250200379A1 (en) Hierarchical reinforcement learning at scale
US20240403652A1 (en) Hierarchical latent mixture policies for agent control
US20240104379A1 (en) Agent control through in-context reinforcement learning
US20240256884A1 (en) Generating environment models using in-context adaptation and exploration
US20240386281A1 (en) Controlling agents by transferring successor features to new tasks
US20260087311A1 (en) Controlling agents using ambiguity-sensitive neural networks and risk-sensitive neural networks
US20230093451A1 (en) State-dependent action space quantization
US20240220795A1 (en) Planning using a jumpy trajectory decoder neural network
WO2024153739A1 (en) Controlling agents using proto-goal pruning
WO2024153797A1 (en) Sequence processing using attention and sub-sampling

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20250722

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)