EP4655718A1 - Learning reinforcement learning policies with local planning - Google Patents

Learning reinforcement learning policies with local planning

Info

Publication number
EP4655718A1
EP4655718A1 EP24702704.8A EP24702704A EP4655718A1 EP 4655718 A1 EP4655718 A1 EP 4655718A1 EP 24702704 A EP24702704 A EP 24702704A EP 4655718 A1 EP4655718 A1 EP 4655718A1
Authority
EP
European Patent Office
Prior art keywords
environment
history
agent
observation
neural network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP24702704.8A
Other languages
German (de)
French (fr)
Inventor
Dong YIN
Sridhar THIAGARAJAN
Nevena LAZIC
Botao Hao
Csaba SZEPESVARI
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
GDM Holding LLC
Original Assignee
GDM Holding LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by GDM Holding LLC filed Critical GDM Holding LLC
Publication of EP4655718A1 publication Critical patent/EP4655718A1/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/092—Reinforcement learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]

Definitions

  • Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input.
  • Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.
  • This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains a policy neural network that can be used to control an agent to interact with an environment by selecting actions to be performed by the agent.
  • the subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.
  • An environment that has a very large state space can be quickly and effectively explored.
  • the techniques described in this specification can distinguish between portions of the state DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application space that have already been sufficiently explored and the portions that are underexplored or that are yet to be explored.
  • Using the described techniques can allow the system to effectively control the agent to return to certain states of the environment, for example, distant, hard to reach states for exploration, and can avoid spending resources repeatedly exploring states that are easily reachable, for example, states near the starting point.
  • Performing more efficient exploration of the environment during training leads to a training process that takes fewer training iterations, using fewer computational resources, e.g., fewer processor cycles, fewer energy resources, and so on.
  • the value and in particular, the diversity, of the data stored in the history buffer can be improved.
  • applying the data from the history buffer as training data to train policy neural networks used in selecting actions to be performed by agents can increase the speed of training of those neural networks.
  • the amount of computing resources necessary for the training of the neural networks can therefore be reduced.
  • the increased speed of training of neural networks can be especially significant for complex neural networks that are harder to train or for training neural networks to select actions to be performed by agents performing complex reinforcement learning tasks.
  • FIG.1 shows an example reinforcement learning system.
  • FIG.2 is a flow diagram of an example process for training a policy neural network.
  • FIG.3 is a flow diagram of sub-steps of one of the steps of the process of FIG.2.
  • FIG.4 is a flow diagram of sub-steps of another one of the steps of the process of FIG.2.
  • Like reference numbers and designations in the various drawings indicate like elements.
  • This specification describes a reinforcement learning system implemented as computer programs on one or more computers in one or more locations that trains a policy neural network that is used to select actions to be performed by an agent interacting with an environment. Once the reinforcement learning system selects an action to be performed by the agent, the reinforcement learning system can cause the agent to perform the selected action. Performance of the selected actions by the agent generally causes the environment to transition into successive new states, where the state of the environment at one time step may depend on the state of the environment at the previous time step and the action performed by the agent at the previous time step. Data characterizing a state of the environment, e.g.
  • the environment is a real-world environment
  • the agent is a mechanical (or electro-mechanical) agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment
  • the actions are actions taken by the mechanical agent in the real-world environment to perform the task.
  • the agent may be a robot interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment.
  • the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator.
  • the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot.
  • the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent.
  • the observations may be defined in 1, 2 or 3 dimensions, and may be absolute and/or relative observations.
  • the observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and/or image or video data for example captured by a camera or a LIDAR sensor, DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.
  • the actions may be control signals to control the robot or other mechanical agent, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements e.g. steering control elements of the vehicle, or higher-level control commands.
  • the control signals can include for example, position, velocity, or force/torque/acceleration data for one or more joints of a robot or parts of another mechanical agent.
  • the control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment.
  • control signals may define actions to control navigation e.g. steering, and movement e.g., braking and/or acceleration of the vehicle.
  • Control signals can be directly generated for controlling controllable elements of the agent.
  • the system can transmit data specifying the selected action to a control system of the agent, which controls the agent to perform the action.
  • the environment is a simulation of the above-described real-world environment, and the agent is implemented as one or more computers interacting with the simulated environment.
  • the simulated environment may be a simulation of a robot or vehicle and the reinforcement learning system may be trained on the simulation and then, once trained, used in the real-world.
  • the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product.
  • a manufacturing environment may be referred to as a manufacturing plant.
  • a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material e.g. to remove pollutants, to generate a cleaned or recycled product.
  • the manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e.g. robots, for processing solid or other materials.
  • the manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g. via pipes or mechanical conveyance.
  • the agent may comprise an electronic agent configured to control a manufacturing unit, or a machine such as a robot, that operates to manufacture the product. That is, the agent may comprise a control system configured to control the manufacture of the chemical, biological, or mechanical product. For example the control system may be configured to control one or more of the manufacturing units or machines or to control movement of an intermediate version or component of the product between the manufacturing units or machines.
  • a task performed by the agent may comprise a task to manufacture the product or an intermediate version or component thereof.
  • a task performed by the agent may comprise a task to control, e.g. minimize, use of a resource such as a task to control electrical power consumption, or water consumption, or the consumption of any material or consumable used in the manufacturing process.
  • the actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product, or an intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment e.g. between the manufacturing units or machines.
  • the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below.
  • the reinforcement learning situation includes a reward calculation unit for generating a reward (e.g. in the form of a number), typically from the observation.
  • the rewards may relate to a metric of performance of the task.
  • the metric may comprise a metric of a quantity of the product that is manufactured, a quality of the product, a speed of production of the product, or to a physical cost of performing the manufacturing task, e.g. a metric of a quantity of energy, materials, or other resources, used to perform the task.
  • the metric may comprise any metric of usage of the resource.
  • observations of a state of the environment may comprise any electronic signals representing the functioning of electronic and/or mechanical items of equipment.
  • a representation of the state of the environment may be derived from observations made by sensors sensing a state of the manufacturing environment, e.g. sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines.
  • sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow/movement rate or flow/movement path of one or more materials; physical or chemical conditions e.g. a physical state, shape or configuration or a chemical state such as pH; configurations of the units or machines such as the mechanical configuration of a unit or machine, or valve configurations; image or video sensors to capture image or video observations of the manufacturing units or of the machines or movement; or any other appropriate type of sensor.
  • the agent is a machine such as a robot
  • the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g.
  • the environment is the real-world environment of a service facility comprising a plurality of items of electronic equipment, such as a server farm or data center, for example a telecommunications data center, or a computer data center for storing or processing data, or any service facility.
  • a service facility comprising a plurality of items of electronic equipment, such as a server farm or data center, for example a telecommunications data center, or a computer data center for storing or processing data, or any service facility.
  • the service facility may also include ancillary control equipment that controls an operating environment of the items of equipment, for example environmental control equipment such as temperature control e.g. cooling equipment, heating equipment, a humidifier, air flow control equipment, air conditioning equipment, or any other hardware that modifies a property of air in the real- world environment.
  • the task may comprise a task to control, e.g. minimize, use of a resource, such as a task to control electrical power consumption, or water consumption.
  • the agent may comprise an electronic agent configured to control operation of the items of equipment, or to control operation of the ancillary, e.g. environmental, control equipment.
  • the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below. These may include actions to control, or to impose operating conditions on, the items of equipment or the ancillary control equipment, e.g. actions that result in changes to settings to adjust, control, or switch on or off the operation of an item of equipment or an item of ancillary control equipment.
  • the observations of a state of the environment may comprise any electronic signals representing the functioning of the facility or of equipment in the facility.
  • a representation of the state of the environment may be derived from observations made by any sensors sensing a state of a physical environment of the facility or observations made by any sensors sensing a state of one or more of items of equipment or one or more items of ancillary control equipment. These include sensors configured to sense electrical conditions such as current, voltage, power or energy; a temperature of the facility; fluid flow, temperature or pressure within the facility or within a cooling system of the facility; or a physical facility configuration such as whether or not a vent is open.
  • the rewards may relate to a metric of performance of a task relating to the efficient operation of the facility. For example in the case of a task to control, e.g.
  • the metric may comprise any metric of use of the resource.
  • the environment is the real-world environment of a power generation facility e.g. a renewable power generation facility such as a solar farm or wind farm.
  • the task may comprise a control task to control power generated by the facility, e.g. to control the delivery of electrical power to a power distribution grid, e.g. to meet demand or to reduce the risk of a mismatch between elements of the grid, or to maximize power generated by the facility.
  • the agent may comprise an electronic agent configured to control the generation of electrical power by the facility or the coupling of generated electrical power into the grid.
  • the actions may comprise actions to control an electrical or mechanical configuration of an electrical power generator such as the electrical or mechanical configuration of one or more renewable power generating elements e.g. to control a configuration of a wind turbine or of a solar panel or panels or mirror, or the electrical or mechanical configuration of a rotating electrical power DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application generation machine.
  • Mechanical control actions may, for example, comprise actions that control the conversion of an energy input to an electrical energy output, e.g. an efficiency of the conversion or a degree of coupling of the energy input to the electrical energy output.
  • Electrical control actions may, for example, comprise actions that control one or more of a voltage, current, frequency or phase of electrical power generated.
  • the rewards may relate to a metric of performance of a task relating to power distribution.
  • the metric may relate to a measure of power transferred, or to a measure of an electrical mismatch between the power generation facility and the grid such as a voltage, current, frequency or phase mismatch, or to a measure of electrical power or energy loss in the power generation facility.
  • the metric may relate to a measure of electrical power or energy transferred to the grid, or to a measure of electrical power or energy loss in the power generation facility.
  • observations of a state of the environment may comprise any electronic signals representing the electrical or mechanical functioning of power generation equipment in the power generation facility.
  • a representation of the state of the environment may be derived from observations made by any sensors sensing a physical or electrical state of equipment in the power generation facility that is generating electrical power, or the physical environment of such equipment, or a condition of ancillary equipment supporting power generation equipment.
  • Such observations may thus include observations of wind levels or solar irradiance, or of local time, date, or season.
  • sensors may include sensors configured to sense electrical conditions of the equipment such as current, voltage, power or energy; temperature or cooling of the physical environment; fluid flow; or a physical configuration of the equipment; and observations of an electrical condition of the grid e.g. from local or remote sensors.
  • Observations of a state of the environment may also comprise one or more predictions regarding future conditions of operation of the power generation equipment such as predictions of future wind levels or solar irradiance or predictions of a future electrical condition of the grid.
  • the environment may be a chemical synthesis or protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical.
  • the DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals/intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical.
  • the agent may be a mechanical agent that indirectly performs or controls the protein folding actions, or chemical synthesis steps, e.g. by controlling synthesis steps selected by the system automatically without human interaction.
  • the observations may comprise direct or indirect observations of a state of the protein or chemical/ intermediates/precursors and/or may be derived from simulation.
  • the system may be used to automatically synthesize a protein with a particular function such as having a binding site shape, e.g. a ligand that binds with sufficient affinity for a biological effect that it can be used as a drug.
  • a ligand that binds with sufficient affinity for a biological effect that it can be used as a drug.
  • it may be an agonist or antagonist of a receptor or enzyme; or it may be an antibody configured to bind to an antibody target such as a virus coat protein, or a protein expressed on a cancer cell, e.g. to act as an agonist for a particular receptor or to prevent binding of another ligand and hence prevent activation of a relevant biological pathway.
  • the environment may be a drug design environment such that each state is a respective state of a potential pharmaceutically active compound pharmaceutically active compound and the agent is a computer system for determining elements of the pharmaceutically active compound and/or a synthetic pathway for the pharmaceutically active compound.
  • the drug/synthesis may be designed based on a reward derived from a target for the pharmaceutically active compound, for example in simulation.
  • the agent may be a mechanical agent that performs or controls synthesis of the pharmaceutically active compound.
  • the environment is a real-world environment and the agent manages distribution of tasks across computing resources e.g. on a mobile device and/or in a data center.
  • the actions may include assigning tasks to particular computing resources.
  • the observations may include observations of computing resources such as compute and/or memory capacity, or Internet-accessible resources; and the actions may include assigning tasks to particular computing resources.
  • the reward(s) may be configured to maximize or minimize one or more of: utilization of computing resources, electrical power, bandwidth, and computation speed.
  • the agent may be a software agent configured to manage distribution of tasks across computing resources. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0037]
  • the agent may be a software agent i.e. a computer program, configured to perform a task.
  • the environment may be a circuit or an integrated circuit design or routing environment and the agent may be configured to perform a design or routing task for routing interconnection lines of a circuit or of an integrated circuit e.g. an ASIC.
  • the reward(s) may then be dependent on one or more routing metrics such as interconnect length, resistance, capacitance, impedance, loss, speed or propagation delay; and/or physical line parameters such as width, thickness or geometry, and design rules.
  • the reward(s) may also or instead include one or more reward(s) relating to a global property of the routed circuitry e.g. component density, operating speed, power consumption, material usage, a cooling requirement, level of electromagnetic emissions, and so forth.
  • the observations may be e.g.
  • the actions may comprise component placing actions e.g. to define a component position or orientation and/or interconnect routing actions e.g. interconnect selection and/or placement actions.
  • the task may be, e.g., to optimize circuit operation to reduce electrical losses, local or external interference, or heat generation, or to increase operating speed, or to minimize or optimize usage of available circuit area.
  • the system may be configured to cause the circuit or integrated circuit to be made to the design, or with interconnection lines routed as determined by the system.
  • the software agent manages the processing, e.g. by one or more real-world servers, of a queue of continuously arriving jobs.
  • the observations may comprise observations of the times of departures of successive jobs, or the time intervals between the departures of successive jobs, or the time a server takes to process each job, e.g. the start and end of a range of times, or the arrival times, or time intervals between the arrivals, of successive jobs, or data characterizing the type of job(s).
  • the actions may comprise actions that allocate particular jobs to particular computing resources; the reward(s) may be configured to minimize an overall queueing or processing time or the queueing or processing time for one or more individual jobs, or in general to optimize any metric based on the observations.
  • the environment may comprise a real-world computer system or network
  • the observations may comprise any observations characterizing operation of the computer system or network
  • the actions performed by the software agent may comprise actions to control the operation e.g. to limit or correct abnormal or undesired operation e.g. because of the presence of a virus or other security breach
  • Application reward(s) may comprise any metric(s) characterizing desired operation of the computer system or network.
  • the environment is a data packet communications network environment, and the agent is part of a router to route packets of data over the communications network.
  • the actions may comprise data packet routing actions and the observations may comprise e.g.
  • routing metrics such as a metric of routing path length, bandwidth, load, hop count, path cost, delay, maximum transmission unit (MTU), and reliability.
  • the reward(s) may be defined in relation to one or more of the routing metrics i.e. configured to maximize one or more of the routing metrics.
  • Some of the above applications generally relate to the agent managing computing resources.
  • Computing resources may include, for example computation speed, memory capacity, electrical power, bandwidth, queue length, circuit area usage, cooling requirements, or any other of the resources mentioned above. It should be understood that a general-purpose computer or distributed system has a finite amount of any of these resources.
  • the agent may be configured to manage computational resources in a real-world computer system or related hardware e.g. circuit or integrated circuit.
  • the actions may include presenting advertisements, the observations may include advertisement impressions or a click-through count or rate, and the reward may characterize previous selections of items or content taken by one or more users.
  • the observations may include textual or spoken instructions provided to the agent by a third-party (e.g., an operator of the agent).
  • the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide textual or spoken instructions to the agent (e.g., to navigate to a particular location).
  • the environment is an Internet or mobile communications environment and the agent is a software agent which manages a personalized recommendation for a user.
  • the observations may comprise previous actions taken by the user, e.g.
  • the actions may include actions recommending items such as content items to a user.
  • the reward(s) may be configured to maximize one or more of: an estimated likelihood that the user will respond favorably to being recommended the (content) item, a suitability unsuitability of one or more DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application recommended items, a cost of the recommended item(s), and a number of recommendations received by the user, optionally within a time span.
  • the environment may be an electrical, mechanical or electro- mechanical design environment, e.g. an environment in which the design of an electrical, mechanical or electro-mechanical entity is simulated.
  • the simulated environment may be a simulation of a real-world environment in which the entity is intended to work.
  • the task may be to design the entity.
  • the observations may comprise observations that characterize the entity, i.e. observations of a mechanical shape or of an electrical, mechanical, or electro-mechanical configuration of the entity, or observations of parameters or properties of the entity.
  • the actions may comprise actions that modify the entity e.g. that modify one or more of the observations.
  • the rewards may comprise one or more metrics of performance of the design of the entity. For example rewards may relate to one or more physical characteristics of the entity such as weight or strength or to one or more electrical characteristics of the entity such as a measure of efficiency at performing a particular function for which the entity is designed.
  • the design process may include outputting the design for manufacture, e.g.
  • the process may include making the entity according to the design.
  • a design of an entity may be optimized, e.g. by reinforcement learning, and then the optimized design output for manufacturing the entity, e.g. as computer executable instructions; an entity with the optimized design may then be manufactured.
  • the environment may be a simulated environment.
  • the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions.
  • the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation
  • the agent may be a simulated vehicle navigating through the motion simulation.
  • the actions may be control inputs to control the simulated user or simulated vehicle.
  • the agent may be implemented as one or more computers interacting with the simulated environment.
  • the simulated environment may be a simulation of a particular real-world environment and agent.
  • the system may be used to select actions in the simulated environment during training or evaluation of the system and, after training, or evaluation, or both, are complete, the action selection policy may be deployed for DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application controlling a real-world agent in the particular real-world environment that was the subject of the simulation.
  • This can avoid unnecessary wear and tear on and damage to the real- world environment or real-world agent and can allow the control neural network to be trained and evaluated on situations that occur rarely or are difficult or unsafe to re-create in the real-world environment.
  • the system may be partly trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and afterwards deployed to control the real mechanical agent in the particular real-world environment.
  • the observations of the simulated environment relate to the real-world environment
  • the selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment.
  • the agent may not include a human being (e.g. it is a robot).
  • the agent comprises a human user of a digital assistant such as a smart speaker, smart display, or other device. Then the information defining the task can be obtained from the digital assistant, and the digital assistant can be used to instruct the user based on the task.
  • the reinforcement learning system may output to the human user, via the digital assistant, instructions for actions for the user to perform at each of a plurality of time steps.
  • the instructions may for example be generated in the form of natural language (transmitted as sound and/or text on a screen) based on actions chosen by the reinforcement learning system.
  • the reinforcement learning system chooses the actions such that they contribute to performing a task.
  • a monitoring system e.g. a video camera system
  • the reinforcement learning system can determine whether the task has been completed.
  • the experience tuples may record the action which the user actually performed based on the instruction, rather than the one which the reinforcement learning system instructed the user to perform.
  • the reward value of each experience tuple may be generated, for example, by comparing the action the user took with a corpus of data showing a human expert performing the task, e.g. using techniques known from imitation learning. Note that if the user performs actions incorrectly (i.e. performs a different action from the one the reinforcement learning system instructs the user to perform) this adds one more source of noise to sources of noise which may already DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application exist in the environment.
  • the reinforcement learning system may identify actions which the user performs incorrectly with more than a certain probability. If so, when the reinforcement learning system instructs the user to perform such an identified action, the reinforcement learning system may warn the user to be careful. Alternatively or additionally, the reinforcement learning system may learn not to instruct the user to perform the identified actions, i.e. ones which the user is likely to perform incorrectly.
  • the digital assistant instructing the user may comprise receiving, at the digital assistant, a request from the user for assistance and determining, in response to the request, a series of tasks for the user to perform, e.g. steps or sub-tasks of an overall task. Then for one or more tasks of the series of tasks, e.g. for each task, e.g.
  • the digital assistant can be used to output to the user an indication of the task, e.g. step or sub-task, to be performed. That is, the digital assistant can provide an instruction or indication to the user as an output. This may be done using natural language, e.g. on a display and/or using a speech synthesis subsystem of the digital assistant. Visual, e.g. video, and/or audio observations of the user performing the task may be captured, e.g. using the digital assistant. A system as described above may then be used to determine whether the user has successfully achieved the task e.g. step or sub-task, i.e. from the answer as previously described.
  • the digital assistant may then, in response, progress to the next task (if any) of the series of tasks, e.g. by outputting an indication of the next task to be performed. In this way the user may be led step-by-step through a series of tasks to perform an overall task.
  • training rewards may be generated e.g. from video data representing examples of the overall task (if corpuses of such data are available) or from a simulation of the overall task.
  • a user may be interacting with a digital assistant and ask for help performing an overall task consisting of multiple steps, e.g. cooking a pasta dish.
  • the digital assistant receives audio and/or video inputs representative of the user's progress on the task, e.g. images or video or sound clips of the user cooking.
  • the digital assistant uses a system as described above, in particular by providing it with the captured audio and/or video and a question that asks whether the user has completed a particular step, e.g. 'Has the user finished chopping the peppers?', to determine whether the user has successfully completed the step.
  • a digital assistant device including a system as described above.
  • the digital assistant can also include a user interface to enable a user to request assistance and to output information.
  • this is a natural language user interface and may comprise a keyboard, voice input-output subsystem, and/or a display.
  • the digital assistant can further include an assistance subsystem configured to determine, in response to the request, a series of tasks for the user to perform. In implementations this may comprise a generative (large) language model, in particular for dialog, e.g. a conversation agent such as Sparrow or Chinchilla.
  • the digital assistant can have an observation capture subsystem to capture visual and/or audio observations of the user performing a task; and an interface for the above-described language model neural network (which may be implemented locally or remotely).
  • the digital assistant can also have an assistance control subsystem configured to assist the user.
  • the assistance control subsystem can be configured to perform the steps described above, for one or more tasks e.g. of a series of tasks, e.g. until a final task of the series. More particularly the assistance control subsystem and output to the user an indication of the task to be performed, capture, using the observation capture subsystem, visual or audio observations of the user performing the task, determine from the above-described answer whether the user has successfully achieved the task. In response the digital assistant can progress to a next task of the series of tasks and/or control the digital assistant, e.g. to stop capturing observations.
  • the environment may not include a human being or animal. In other implementations, however, it may comprise a human being or animal.
  • the agent may be an autonomous vehicle in an environment which is a location (e.g. a geographical location) where there are human beings (e.g. pedestrians or drivers/passengers of other vehicles) and/or animals, and the autonomous vehicle itself may optionally contain human beings.
  • the environment may also be at least one room (e.g. in a habitation) containing one or more people.
  • the human being or animal may be an element of the environment which is involved in the task, e.g. modified by the task (indeed, the environment may substantially consist of the human being or animal).
  • the environment may be a medical or veterinary environment containing at least DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application one human or animal subject, and the task may relate to performing a medical (e.g. surgical) procedure on the subject.
  • the environment may comprise a human user who interacts with an agent which is in the form of an item of user equipment, e.g. a digital assistant.
  • the item of user equipment provides a user interface between the user and a computer system (the same computer system(s) which implement the reinforcement learning system, or a different computer system).
  • the user interface may allow the user to enter data into and/or receive data from the computer system, and the agent is controlled by the action selection policy to perform an information transfer task in relation to the user, such as providing information about a topic to the user and/or allowing the user to specify a component of a task which the computer system is to perform.
  • the information transfer task may be to teach the user a skill, such as how to speak a language or how to navigate around a geographical location; or the task may be to allow the user to define a three-dimensional shape to the computer system, e.g. so that the computer system can control an additive manufacturing (3D printing) system to produce an object having the shape.
  • Actions may comprise outputting information to the user (e.g.
  • an action may comprise setting a problem for a user to perform relating to the skill (e.g. asking the user to choose between multiple options for correct usage of the language, or asking the user to speak a passage of the language out loud), and/or receiving input from the user (e.g. registering selection of one of the options, or using a microphone to record the spoken passage of the language).
  • Rewards may be generated based upon a measure of how well the task is performed. For example, this may be done by measuring how well the user learns the topic, e.g. performs instances of the skill (e.g. as measured by an automatic skill evaluation unit of the computer system).
  • the action may comprise presenting a (visual, haptic or audio) user interface to the user which permits the user to specify an element of the component of the task, and receiving user input using the user interface.
  • the rewards may be generated based on a measure of how well and/or easily the user can specify the component of the task for the computer system to perform, e.g. how fully or well the three-dimensional object is specified. This may be determined automatically, or a reward may be specified by the user, e.g. a subjective measure of the user experience.
  • FIG.1 shows an example reinforcement learning system 100.
  • the reinforcement learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.
  • the reinforcement learning system 100 trains a policy neural network 150 that has a plurality of parameters by repeatedly updating the values of the plurality of parameters of the policy neural network 150 based on the interactions of an agent 104 with the environment 106.
  • the parameters can include the weights and, optionally, the biases of the layers of the neural network 150.
  • the agent 104 interacts with the environment 106 by performing actions 102 selected by using the policy neural network 150 during training. Performance of the selected actions 102 by the agent 102 generally causes the environment 106 to transition into successive new states.
  • the policy neural network 150 is a neural network that is configured to receive a policy network input that includes data that characterizes a state of the environment 106 (i.e., an “observation 108”) and to process the policy network input in accordance with the current values of the plurality of parameters of the policy neural network 150 to generate an action selection output that can be used to determine an action 102 to be performed by the agent 104 in response to the observation 108.
  • the action selection output can be a probability distribution over the set of possible actions that can be performed by agent in response to the observation.
  • the action selection output can be a Q-value that is an estimate of the long-term time-discounted reward that would be received if the agent performs a particular action in response to the observation (in this example the policy network input can also include data that identifies the particular action).
  • the action selection output can identify a particular action that is predicted to yield the highest long-term time- discounted reward if performed by the agent in response to the observation. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0060]
  • the policy neural network 150 can have any appropriate architecture that allows the policy neural network 150 to process a policy network input that includes an observation 108 to generate an action selection output.
  • the policy neural network 150 can be a convolutional neural network.
  • the observations 108 include only relatively lower-dimensional inputs, e.g., sensor readings that characterize the current state of the robot, the policy neural network 150 can be a multi-layer perceptron.
  • the policy neural network 150 can include a convolutional encoder that encodes the high-dimensional data, a fully-connected encoder that encodes the lower-dimensional data, and a policy subnetwork that operates on a combination, e.g., a concatenation, of the encoded data to generate the action selection output.
  • the environment 106 can be a simulated environment implemented as one or more computer programs
  • the agent 104 can be a simulated agent implemented as one or more computer programs.
  • the environment 106 is a computer simulation of any of the real-world environments mentioned above
  • the agent 104 is a computer simulation any of the real-world agents mentioned above that can interact with the simulated environment through computer simulation.
  • the actions 102 can generally include simulated versions of one or more of the previously described actions or types of actions.
  • the environment 106 can be a motion simulation environment, e.g., a driving simulation or a flight simulation
  • the agent 104 can be a simulated vehicle navigating through the motion simulation.
  • the environment 106 can be a computer simulation of a real-world environment and the agent 104 can be a simulated robot interacting with the computer simulation.
  • the actions 102 can be control inputs to control the simulated vehicle or simulated robot.
  • the environment 106 can be a real-world environment
  • the agent 104 can be a real-world agent that can interact with the real- world environment.
  • the environment 106 can be reset to a certain state, for example, by physically placing the agent 104 at a particular location within the environment 106, instructing a machine or another system to modify aspects of the environment 106 to return to a particular state, and so on.
  • the reinforcement learning system 100 maintains a history buffer 120 that stores environment interaction data generated as a result of the agent 104 interacting with the environment 106 during training.
  • the history buffer 120 stores a plurality of history observations 108 received by the agent 104 during past interaction of the agent 104 with the environment 106.
  • the past interaction of the agent 104 with the environment 106 includes one or more past interactions of the agent 104 with the environment 106.
  • Each history observation 108 is associated with one of the past interactions.
  • Each history observation 108 characterizes a respective previous state of the environment 106.
  • each history observation 108 comprises data characterizing a respective previous state of the environment 106.
  • the data characterizing the respective previous state may include, for example, images, object position data, sensor data and/or any other data capable of characterizing a state of an environment.
  • the history buffer 120 additionally stores data identifying a plurality of history actions 102.
  • the history buffer 120 can store each of the plurality of history observations 108 in association with data defining a corresponding history action 102, e.g., in the format of history observation-action pairs.
  • the corresponding history action 102 is an action that was performed by the agent 104 when the environment 106 was in the state characterized by the history observation 108.
  • the history buffer 120 can be any suitable data structure implemented in any available memory, e.g., one or more physical or logical storage devices, that is accessible by the reinforcement learning system 100.
  • a first-in first-out (FIFO) queue is an example of a suitable data structure.
  • the history observations 108 are stored in the history buffer 120 according to the time at which they are generated. At certain time points during the training, the most obsolete history observations 108 (and, when stored, data identifying the corresponding history actions 102) can be deleted to ensure that the history buffer 120 does not consume greater than a threshold amount of memory resource.
  • the reinforcement learning system 100 trains the policy neural network 150 over a plurality of training iterations. At each of at least some of the plurality of training iterations of the policy neural network 150, the reinforcement learning system 100 executes a local planning process followed by a data collection process. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0068] In the local planning process, the reinforcement learning system 100 samples a batch of history observations from the history buffer 120 and uses an uncertainty metric engine 130 to compute an uncertainty metric for each history observation included in the batch.
  • FILO first-in last-out
  • LIFO last-in first-out
  • the reinforcement learning system 100 identifies a selected history observation 140 from the batch of history observations based on the uncertainty metrics. For example, the reinforcement learning system 100 can identify, as the selected history observation 140, a history observation that has a highest uncertainty metric amongst the batch of history observations. As another example, the reinforcement learning system 100 can identify, as the selected history observation 140, a history observation that has an uncertainty metric greater than a given threshold uncertainly metric. [0069] As will be explained in more detail below, the uncertainty metric engine 130 can compute an uncertainty metric for a given history observation that is obtained from the history buffer 120 in many different ways.
  • the uncertainty metric for a particular observation may be generally considered an inverse measure of (e.g. a quantification of) how well trained the policy neural network 150 is with respect to that particular observation.
  • the reinforcement learning system 100 controls the agent 104 to return to an initial state of the environment characterized by the selected history observation 140.
  • the initial state to which the agent 104 returns to may be referred to as a selected initial state or a training iteration initial state. That is, the initial state may not necessarily be associated with a state which occurred chronologically before other states. Rather, the initial state denotes an intended state for the agent 104 to return to at the current training iteration.
  • the reinforcement learning system 100 After the reinforcement learning system 100 has controlled the agent 104 to return to the initial state of the environment characterized by the selected history observation 140, it controls the agent to perform one or more actions 102 selected using the policy neural network 150. This is in contrast to some conventional reinforcement learning systems that may control the agent 104 to return to a fixed initial state or a randomly selected initial state at each training iteration.
  • the reinforcement learning system 100 controls the agent 104 for an episode of a task.
  • An “episode” of a task is a sequence of interactions during which the agent 104 attempts to perform a single instance of the task starting from some starting state of the environment 106.
  • each task episode begins with the environment 106 being in the initial state characterized by the selected history observation 140, and ends when the agent 104 has successfully completed the task or when some termination criterion is satisfied, e.g., the environment 106 enters a state that has been designated as a terminal state or the agent 104 performs a threshold number of actions 102 without successfully completing the task.
  • the reinforcement learning system 100 selects the one or more actions 102 in accordance with the values of the plurality of parameters of the policy neural network 150.
  • the agent 104 is subsequently controlled in accordance with a current action selection policy that is defined by the current values of the plurality of parameters of the policy neural network 150 as of the training iteration to generate new environment interaction data to be stored in the history buffer 120.
  • the new environment interaction data to be stored in the history buffer 120 includes the observations 108 received by the agent 104 during the data collection process and, optionally, data defining the actions 102 that are selected by the policy neural network 150 to be performed by the agent 104 during the data collection process.
  • the reinforcement learning system 100 determines one or more updates to the current values of the plurality of parameters of the policy neural network 150 by training the policy neural network 150 on the environment interaction data obtained from the history buffer 120 by using an appropriate reinforcement learning technique, e.g., an online reinforcement learning technique.
  • the trained policy neural network 150 can be used to control the interactions of a real-world agent with a real-world environment, e.g., to control the agent that was being simulated in the simulated environment. Training the policy neural network 150 based on interactions of an agent with a simulated environment (i.e., instead of a real-world environment) can avoid wear- and-tear on the agent and can reduce the likelihood that, by performing poorly chosen actions, the agent can damage itself or aspects of its environment.
  • the reinforcement learning system 100 can provide, e.g., by a wired or wireless connection, data specifying the trained policy neural network 150, e.g., data specifying the trained values of the plurality of parameters, and, optionally, the architecture of the policy neural network 150, to an agent control system that can cause the agent to perform different actions.
  • the reinforcement learning system 100 can directly function as an agent control system that can cause the agent to perform different actions.
  • FIG.2 is a flow diagram of an example process 200 for training a policy neural network.
  • the process 200 will be described as being performed by a system of one or more computers located in one or more locations.
  • a reinforcement learning system e.g., the reinforcement learning system 100 of FIG.1, appropriately programmed, can perform the process 200.
  • the policy neural network is a neural network that includes a plurality of parameters and that is used to select actions to be performed by an agent interacting with an environment to perform a task. Performance of the selected actions by the agent generally causes the environment to transition into successive new states.
  • the system To train the policy neural network, the system maintains a history buffer.
  • the history buffer stores a plurality of history observations received by the agent during past interaction of the agent with the environment during training. Each history observation characterizes a respective previous state of the environment that has been visited by the agent during the training.
  • the history buffer stores each of the plurality of history observations in association with data defining a corresponding history action that was performed by the agent when the environment was in the state characterized by the history observation.
  • the history buffer stores a plurality of history observation-action pairs that each include a history observation and data defining a history action.
  • the system can repeatedly perform iterations of the process 200 to update the values of the plurality of parameters of the policy neural network.
  • the system can continue performing iterations of the process 200 until termination criteria for the training of the policy neural network have been satisfied, e.g., until the values of the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of the process 200 have been performed.
  • the system determines an initial observation of the environment (step 202).
  • the system chooses between executing a local planning process and a random process to determine the initial observation.
  • the system can alternate between the local planning process and the random process to perform step 202 at different iterations of the process 200. That is, the system can execute the local planning process to determine the initial observation at the first, third, fifth, and so on iterations of the process 200, while the system can execute the random process to determine the initial observation at the second, fourth, sixth, and so on iterations of the process 200.
  • the system can sample a number with some measure of randomness from a predetermined range and then choose the local planning process if the sampled number is greater than a given threshold within the predetermined range. In other implementations, the system always executes the local planning process to determine the initial observation of the environment. [0082] In the random process, the system samples an observation from a given distribution of history observations and uses the sampled observation as the initial observation.
  • the given distribution of history observations can generally be determined from the plurality of history buffers stored in the history buffer, e.g., as a weighted distribution of history observations where each distinct history observation is weighted by a total number of the history observation stored in the history buffer.
  • executing the local planning process to determine the initial observation of the environment will depend on the uncertainty metrics for the history observations stored in the history buffer, as will be explained in more detail with reference to FIG.3, which shows sub-steps 302-306 corresponding to executing the local planning process to perform step 202.
  • the system samples a batch of history observations or, in some implementations, a batch of history observation-action pairs that each include a history observation and data DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application defining a history action, from the history buffer (step 302).
  • the system can generally obtain different history observations (or different history observation-action pairs) at different iterations by performing the sampling with some measure of randomness, although the total number of the history observations (or the total number of the history observation-action pairs) obtained by the system at different iterations can be the same. [0085]
  • the system computes an uncertainty metric for each history observation in the sampled batch of history observations or, in some implementations, an uncertainty metric for each history observation-action pair in the sampled batch of history observation-action pairs (step 304).
  • the system can compute the uncertainty metric in any of a variety of different ways.
  • the system can use the policy neural network to compute an uncertainty metric for each observation-action pair included in the batch of history observation-action pairs.
  • the system can process, for each history observation-action pair, a policy network input that includes the history observation and data defining the history action in accordance with the current parameters values of the policy neural network to generate an action selection output that includes a Q-value for the history observation-action pair.
  • the system can then compute a standard deviation of the respective Q values for the history observation-action pairs included in the sampled batch, and compute the uncertainty metric from the standard deviation.
  • the system maintains multiple instances of the policy neural network, where each different instance of the policy neural network is used to process a different history observation- action pair included in the sampled batch to generate a Q-value for the history observation-action pair.
  • the uncertainty metric for each history observation-action pair can thus be computed as: , where M is the number of the instances of the policy neural network maintained by the system. ⁇ ⁇ , ⁇ ; ⁇ ⁇ ⁇ is the Q-value for an observation-action pair ⁇ ⁇ , ⁇ that includes a history observation s and a history action ⁇ .
  • the parameters of the ⁇ -th instance of the policy neural network are a summation of the DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application values of a trainable component ⁇ ⁇ and the values of a fixed randomized prior policy neural network ⁇ ⁇ ⁇ .
  • the system can similarly compute an uncertainty metric for each history observation included in the sampled batch of history observations.
  • the system can do this by randomly selecting an action from a set of possible actions that can be performed by the agent in response to the history observation, and then processing the history observation and data defining the randomly selected action using policy neural network to generate an action selection output. That is, the history action ⁇ in the equation above can be replace by a randomly selected action ⁇ ⁇ in implementations where the history actions are not stored in the history buffer in association with the history observations.
  • the system can use a covariance-based method to compute the uncertainty metric for each history observation-action pair included in the batch of history observation-action pairs.
  • the system can process the history observation-action pair using a representation neural network to generate a feature vector, and then determine a covariance matrix of the feature vectors that have been generated for the sampled batch of history observation-action pairs. The system can then compute the uncertainty metric from the covariance matrix.
  • the representation neural network can be any appropriate neural network.
  • the representation neural network can be a pre-trained neural network that has been pre-trained on a representation learning task.
  • the representation neural network can be a randomly initialized neural network.
  • the representation neural network can be a combination of both a pre-trained neural network and a randomly initialized neural network.
  • the covariance matrix can be computed as: where ⁇ ⁇ , ⁇ represents the feature vector generated by using the representation neural network for an observation-action pair ⁇ ⁇ , ⁇ that includes a history observation s and a history action ⁇ , and ⁇ is a regularization parameter.
  • the uncertainty metric for each history observation can then be computed as: [0092]
  • the system can similarly compute an uncertainty metric for each history observation included in the sampled batch of history observations. Specifically, for each history observation included in the sampled batch of history observations, the system can do this by randomly selecting an action from a set of possible actions that can be performed by the agent in response to the history observation, and then processing the history observation and data defining the randomly selected action using representation neural network to generate a feature vector. That is, the history action ⁇ in the equations above can be replace by a randomly selected action ⁇ ⁇ in implementations where the history actions are not stored in the history buffer in association with the history observations.
  • the system can compute the uncertainty metric by determining an estimation of a total count of each distinct history observation-action pair in the history buffer, and then computing the uncertainty metric from the estimations of the total counts of distinct history observations.
  • the uncertainty metric for each history observation-action pair included in sampled batch of history observation- action pairs can be computed as: where ⁇ ⁇ , ⁇ is the estimated total count of the observation-action pair ⁇ ⁇ , ⁇ , and ⁇ is a regularization parameter.
  • the estimated total count ⁇ ⁇ , ⁇ can be determined, e.g., by a predetermined function ⁇ : ⁇ ⁇ ⁇ ⁇ that maps each observation ⁇ ⁇ ⁇ to a corresponding discrete element in a discrete observation space ⁇ ⁇ where
  • the system can similarly compute an uncertainty metric for each history observation included in the sampled batch of history observations. That is, the history action ⁇ in the equations above can be replace by a randomly selected action ⁇ ⁇ in implementations where the history actions are not stored in the history buffer in association with the history observations.
  • the system can compute the uncertainty metric through random network distillation (RND).
  • RTD random network distillation
  • the system can process the history observation DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application using a first neural network and a second neural network to generate a first feature vector and a second feature vector, respectively, and then compute the uncertainty metric from a difference between the first feature vector and the second feature vector.
  • the first neural network can have trained parameter values while the second neural network can have randomly initialized parameter values.
  • the uncertainty metric for each history observation can be computed as: , where ⁇ ⁇ ⁇ represents the first neural network and ⁇ is the second neural network, and the difference is computed as error.
  • the system determines, based on the uncertainty metrics, a selected history observation from the batch of history observations or the batch of history observation- action pairs (step 306).
  • the selected history observation is then used as the initial observation of the environment for the iteration.
  • the system can identify, as the selected history observation, a history observation that has a highest uncertainty metric amongst the sampled batch of history observations or the sampled batch of history observation-action pairs.
  • the system can identify, as the selected history observation, a history observation that has an uncertainty metric greater than a given threshold uncertainly metric. In either example, because different batches of history observations or different batches of history observation-action pairs are sampled, the system will generally determine different selected history observations at different iterations.
  • the system executes a data collection process that begins from (i.e. starts with) the initial observation of the environment (step 204).
  • Step 204 is explained in more detail with reference to FIG. 4, which shows sub-steps 402-406 corresponding to step 204.
  • the system can repeatedly perform multiple iterations of sub-steps 402- 406 to execute the data collection process.
  • the system can control the agent for an episode of the task that begins with the environment being in an initial state characterized by the initial observation.
  • the system selects, at each of multiple (environment) time steps during the performance of the episode of the task, an action to be performed by the agent in response to an observation that characterizes the state of the environment at the time step (step 402).
  • the observation can be the DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application initial observation.
  • the selected action can be a history action that is included in a particular observation-action pair that includes the initial observation.
  • the system can randomly select the action from a set of possible actions that can be performed by the agent in response to the initial observation, or can select the action by using the policy neural network.
  • the system can process a policy network input that includes the initial observation in accordance with current values of the plurality of parameters of the policy neural network to generate an action selection output that can be used to select the action.
  • the observation can be an observation that characterizes a state that the environment transitioned into as a result of the system performing an action at a preceding time step that precedes the time step.
  • the action can be selected by using the policy neural network.
  • the system can process a policy network input that includes the observation characterizing the state of the environment at the time step in accordance with the current values of the plurality of parameters of the policy neural network to generate an action selection output that can be used to select the action.
  • the system stores, in the history buffer, data identifying the selected action in association with the observation of the environment at the time step.
  • the system controls the agent to perform the selected action to cause the environment to transition into a new state of the environment at a subsequent time step (step 404).
  • the system stores, in the history buffer, a new observation that is received in response to the agent performing the selected action and that characterizes the new state of the environment at the subsequent time step (step 406).
  • the system trains, using an online reinforcement learning technique, the policy neural network based on the environment interaction data obtained from the history buffer to update the current values of the plurality of parameters of the policy neural network (step 206).
  • the online reinforcement learning technique can be a deep Q network (DQN) technique.
  • DQN technique is described in V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
  • the online reinforcement learning technique can be a double deep Q network (DDQN) technique.
  • DDQN technique is described in H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
  • the online reinforcement learning technique can be a bootstrapped DQN technique. Bootstrapped DQN technique is described in I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems, 29, 2016.
  • the online reinforcement learning technique can be a distributional DDQN technique. Distributional DDQN technique is described in M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449–458. PMLR, 2017. [0108] As yet another example, the online reinforcement learning technique can be an approximate policy iteration technique. Approximate policy iteration technique is described in D. P. Bertsekas. Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications, 9(3):310–335, 2011. [0109] An example algorithm for training the policy neural network is shown below.
  • the system chooses between executing a local planning process and a random process to determine the initial observation ⁇ ⁇ based on sampling a number from a predetermined range of ⁇ 0,1 ⁇ .
  • the system chooses the local planning process if the sampled number is greater than a given threshold ⁇ ⁇ .
  • ⁇ ⁇ can be an adjustable parameter of the system.
  • ⁇ ⁇ can depend on the history buffer, the size of the batch of history observations or history observation-action pairs to be sampled from the history buffer at each training DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application iteration, or both.
  • Agent.UPDATE() refers to training the policy neural network using an online reinforcement learning technique to update the current parameters values of the policy neural network.
  • An example algorithm for the data collection process (“DATACOLLECTION”) is shown below.
  • the system controls the agent to return to an initial state characterized by the initial observation ⁇ ⁇ : “ENV.RESET( ⁇ ⁇ )”, and then to perform an episode of the task that begins with the environment being in initial state.
  • ⁇ ⁇ Agent.ACT( ⁇ ) refers to selecting, in accordance with the values of the plurality of parameters of the policy neural network, an action ⁇ to be performed by the agent in response to an observation ⁇ .
  • the observation ⁇ is stored into the history buffer ⁇ : “ ⁇ ⁇ ⁇ ⁇ ”. Performance of the ⁇ causes the environment to transition (or “step”) into a new state that has a new observation: “ ⁇ ⁇ ENV.STEP( ⁇ )”.
  • This specification uses the term “configured” in connection with systems and computer program components.
  • a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions.
  • one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.
  • Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
  • Embodiments of the subject matter described in this specification can be DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus.
  • the computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.
  • the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
  • data processing apparatus refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
  • the apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • the apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • code that creates an execution environment for computer programs e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • a system, artificial neural network, or trained artificial neural network as described herein can be implemented in hardware using electronic circuitry, e.g. in a physical box.
  • computer code as described herein can be code to emulate such hardware or code for a hardware description language.
  • a computer program which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • a program may, but need not, correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code.
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.
  • the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations.
  • the index database can include multiple collections of data, each of which may be organized and accessed differently.
  • the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations.
  • one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.
  • the processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output.
  • the processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.
  • Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit.
  • a central processing unit will receive instructions and data from a read only memory or a random access memory or both.
  • the elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.
  • the central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • a computer need not have such devices.
  • a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
  • PDA personal digital assistant
  • GPS Global Positioning System
  • USB universal serial bus
  • Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
  • semiconductor memory devices e.g., EPROM, EEPROM, and flash DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application memory devices
  • magnetic disks e.g., internal hard disks or removable disks
  • magneto optical disks e.g., CD ROM and DVD-ROM disks.
  • embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • keyboard and a pointing device e.g., a mouse or a trackball
  • Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
  • a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser.
  • a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.
  • Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.
  • Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a JAX framework.
  • a machine learning framework e.g., a TensorFlow framework or a JAX framework.
  • Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components.
  • a back end component e.g., as a data server
  • a middleware component e.g., an application server
  • a front end component e.g., a client computer having a graphical
  • the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network.
  • Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.
  • the computing system can include clients and servers.
  • a client and server are generally remote from each other and typically interact through a communication network.
  • the relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
  • a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client.
  • Data generated at the user device e.g., a result of the user interaction, can be received at the server from the device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a policy neural network used to select actions performed by an agent interacting with an environment by performing actions that cause the environment to transition states. In one aspect, one of the methods include: maintaining a history buffer; executing a local planning process using the history buffer to determine an initial observation of the environment; executing a data collection process that begins from the determined initial observation of the environment; and training, using an online reinforcement learning technique, the policy neural network on the history buffer to update the current parameters values of the policy neural network.

Description

DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application LEARNING REINFORCEMENT LEARNING POLICIES WITH LOCAL PLANNING CROSS-REFERENCE TO RELATED APPLICATION [0001] This application claims priority to U.S. Provisional Application No.63/441,429, filed on January 26, 2023. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application. BACKGROUND [0002] This specification relates to reinforcement learning. [0003] In a reinforcement learning system, an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment. [0004] Some reinforcement learning systems select the action to be performed by the agent in response to receiving a given observation in accordance with an output of a neural network. [0005] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters. SUMMARY [0006] This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains a policy neural network that can be used to control an agent to interact with an environment by selecting actions to be performed by the agent. [0007] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages. [0008] An environment that has a very large state space can be quickly and effectively explored. By storing history observations received by the agent through past interaction with the environment in association of their uncertainty metrics in a history buffer, the techniques described in this specification can distinguish between portions of the state DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application space that have already been sufficiently explored and the portions that are underexplored or that are yet to be explored. Using the described techniques can allow the system to effectively control the agent to return to certain states of the environment, for example, distant, hard to reach states for exploration, and can avoid spending resources repeatedly exploring states that are easily reachable, for example, states near the starting point. Performing more efficient exploration of the environment during training leads to a training process that takes fewer training iterations, using fewer computational resources, e.g., fewer processor cycles, fewer energy resources, and so on. [0009] By making use of the described techniques, the value and in particular, the diversity, of the data stored in the history buffer can be improved. As such, applying the data from the history buffer as training data to train policy neural networks used in selecting actions to be performed by agents can increase the speed of training of those neural networks. The amount of computing resources necessary for the training of the neural networks can therefore be reduced. The increased speed of training of neural networks can be especially significant for complex neural networks that are harder to train or for training neural networks to select actions to be performed by agents performing complex reinforcement learning tasks. [0010] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS [0011] FIG.1 shows an example reinforcement learning system. [0012] FIG.2 is a flow diagram of an example process for training a policy neural network. [0013] FIG.3 is a flow diagram of sub-steps of one of the steps of the process of FIG.2. [0014] FIG.4 is a flow diagram of sub-steps of another one of the steps of the process of FIG.2. [0015] Like reference numbers and designations in the various drawings indicate like elements. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application DETAILED DESCRIPTION [0016] This specification describes a reinforcement learning system implemented as computer programs on one or more computers in one or more locations that trains a policy neural network that is used to select actions to be performed by an agent interacting with an environment. Once the reinforcement learning system selects an action to be performed by the agent, the reinforcement learning system can cause the agent to perform the selected action. Performance of the selected actions by the agent generally causes the environment to transition into successive new states, where the state of the environment at one time step may depend on the state of the environment at the previous time step and the action performed by the agent at the previous time step. Data characterizing a state of the environment, e.g. an image of the environment, is referred to herein as an observation. [0017] In some implementations, the environment is a real-world environment, the agent is a mechanical (or electro-mechanical) agent interacting with the real-world environment, e.g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the agent may be a robot interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment. [0018] In these implementations, the observations may include, e.g., one or more of: images, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator. For example in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity, force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in 1, 2 or 3 dimensions, and may be absolute and/or relative observations. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and/or image or video data for example captured by a camera or a LIDAR sensor, DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment. [0019] In these implementations, the actions may be control signals to control the robot or other mechanical agent, e.g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements e.g. steering control elements of the vehicle, or higher-level control commands. The control signals can include for example, position, velocity, or force/torque/acceleration data for one or more joints of a robot or parts of another mechanical agent. The control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi- autonomous land or air or sea vehicle the control signals may define actions to control navigation e.g. steering, and movement e.g., braking and/or acceleration of the vehicle. Control signals can be directly generated for controlling controllable elements of the agent. As another example, the system can transmit data specifying the selected action to a control system of the agent, which controls the agent to perform the action. [0020] In some implementations the environment is a simulation of the above-described real-world environment, and the agent is implemented as one or more computers interacting with the simulated environment. For example the simulated environment may be a simulation of a robot or vehicle and the reinforcement learning system may be trained on the simulation and then, once trained, used in the real-world. [0021] In some implementations the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product. A manufacturing environment may be referred to as a manufacturing plant. As used herein a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material e.g. to remove pollutants, to generate a cleaned or recycled product. The manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e.g. robots, for processing solid or other materials. The manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g. via pipes or mechanical conveyance. As used herein manufacture of a product also includes manufacture of a food product by a kitchen robot. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0022] The agent may comprise an electronic agent configured to control a manufacturing unit, or a machine such as a robot, that operates to manufacture the product. That is, the agent may comprise a control system configured to control the manufacture of the chemical, biological, or mechanical product. For example the control system may be configured to control one or more of the manufacturing units or machines or to control movement of an intermediate version or component of the product between the manufacturing units or machines. [0023] As one example, a task performed by the agent may comprise a task to manufacture the product or an intermediate version or component thereof. As another example, a task performed by the agent may comprise a task to control, e.g. minimize, use of a resource such as a task to control electrical power consumption, or water consumption, or the consumption of any material or consumable used in the manufacturing process. [0024] The actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product, or an intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment e.g. between the manufacturing units or machines. In general the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below. These may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. The actions may include actions imposing operating conditions on a manufacturing unit or machine, or actions that result in changes to settings to adjust, control, or switch on or off the operation of a manufacturing unit or machine. [0025] In some implementations, the reinforcement learning situation includes a reward calculation unit for generating a reward (e.g. in the form of a number), typically from the observation. The rewards may relate to a metric of performance of the task. For example in the case of a task that is to manufacture a product the metric may comprise a metric of a quantity of the product that is manufactured, a quality of the product, a speed of production of the product, or to a physical cost of performing the manufacturing task, e.g. a metric of a quantity of energy, materials, or other resources, used to perform the task. In the case of a task that is to control use of a resource, the metric may comprise any metric of usage of the resource. In the case of a task which is to control an electromechanical DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application agent such as a robot to perform a manipulation of an object, the reward may indicate whether the object has been correctly manipulated according to a predefined criterion. [0026] In general, observations of a state of the environment may comprise any electronic signals representing the functioning of electronic and/or mechanical items of equipment. For example a representation of the state of the environment may be derived from observations made by sensors sensing a state of the manufacturing environment, e.g. sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines. As some examples such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow/movement rate or flow/movement path of one or more materials; physical or chemical conditions e.g. a physical state, shape or configuration or a chemical state such as pH; configurations of the units or machines such as the mechanical configuration of a unit or machine, or valve configurations; image or video sensors to capture image or video observations of the manufacturing units or of the machines or movement; or any other appropriate type of sensor. In the case that the agent is a machine such as a robot the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of one or more parts of the machine, e.g. data characterizing the current state of the machine or robot or of an item held or processed by the machine or robot. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal, or image or video data for example from a camera or a LIDAR sensor (e.g. mounted on the machine). Sensors such as these may be part of or located separately from the agent in the environment. [0027] In some implementations the environment is the real-world environment of a service facility comprising a plurality of items of electronic equipment, such as a server farm or data center, for example a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include ancillary control equipment that controls an operating environment of the items of equipment, for example environmental control equipment such as temperature control e.g. cooling equipment, heating equipment, a humidifier, air flow control equipment, air conditioning equipment, or any other hardware that modifies a property of air in the real- world environment. The task may comprise a task to control, e.g. minimize, use of a resource, such as a task to control electrical power consumption, or water consumption. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application The agent may comprise an electronic agent configured to control operation of the items of equipment, or to control operation of the ancillary, e.g. environmental, control equipment. [0028] In general the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below. These may include actions to control, or to impose operating conditions on, the items of equipment or the ancillary control equipment, e.g. actions that result in changes to settings to adjust, control, or switch on or off the operation of an item of equipment or an item of ancillary control equipment. [0029] In general, the observations of a state of the environment may comprise any electronic signals representing the functioning of the facility or of equipment in the facility. For example a representation of the state of the environment may be derived from observations made by any sensors sensing a state of a physical environment of the facility or observations made by any sensors sensing a state of one or more of items of equipment or one or more items of ancillary control equipment. These include sensors configured to sense electrical conditions such as current, voltage, power or energy; a temperature of the facility; fluid flow, temperature or pressure within the facility or within a cooling system of the facility; or a physical facility configuration such as whether or not a vent is open. [0030] The rewards may relate to a metric of performance of a task relating to the efficient operation of the facility. For example in the case of a task to control, e.g. minimize, use of a resource, such as a task to control use of electrical power or water, the metric may comprise any metric of use of the resource. [0031] In some implementations the environment is the real-world environment of a power generation facility e.g. a renewable power generation facility such as a solar farm or wind farm. The task may comprise a control task to control power generated by the facility, e.g. to control the delivery of electrical power to a power distribution grid, e.g. to meet demand or to reduce the risk of a mismatch between elements of the grid, or to maximize power generated by the facility. The agent may comprise an electronic agent configured to control the generation of electrical power by the facility or the coupling of generated electrical power into the grid. The actions may comprise actions to control an electrical or mechanical configuration of an electrical power generator such as the electrical or mechanical configuration of one or more renewable power generating elements e.g. to control a configuration of a wind turbine or of a solar panel or panels or mirror, or the electrical or mechanical configuration of a rotating electrical power DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application generation machine. Mechanical control actions may, for example, comprise actions that control the conversion of an energy input to an electrical energy output, e.g. an efficiency of the conversion or a degree of coupling of the energy input to the electrical energy output. Electrical control actions may, for example, comprise actions that control one or more of a voltage, current, frequency or phase of electrical power generated. [0032] The rewards may relate to a metric of performance of a task relating to power distribution. For example in the case of a task to control the delivery of electrical power to the power distribution grid the metric may relate to a measure of power transferred, or to a measure of an electrical mismatch between the power generation facility and the grid such as a voltage, current, frequency or phase mismatch, or to a measure of electrical power or energy loss in the power generation facility. In the case of a task to maximize the delivery of electrical power to the power distribution grid the metric may relate to a measure of electrical power or energy transferred to the grid, or to a measure of electrical power or energy loss in the power generation facility. [0033] In general observations of a state of the environment may comprise any electronic signals representing the electrical or mechanical functioning of power generation equipment in the power generation facility. For example a representation of the state of the environment may be derived from observations made by any sensors sensing a physical or electrical state of equipment in the power generation facility that is generating electrical power, or the physical environment of such equipment, or a condition of ancillary equipment supporting power generation equipment. Such observations may thus include observations of wind levels or solar irradiance, or of local time, date, or season. Such sensors may include sensors configured to sense electrical conditions of the equipment such as current, voltage, power or energy; temperature or cooling of the physical environment; fluid flow; or a physical configuration of the equipment; and observations of an electrical condition of the grid e.g. from local or remote sensors. Observations of a state of the environment may also comprise one or more predictions regarding future conditions of operation of the power generation equipment such as predictions of future wind levels or solar irradiance or predictions of a future electrical condition of the grid. [0034] As another example, the environment may be a chemical synthesis or protein folding environment such that each state is a respective state of a protein chain or of one or more intermediates or precursor chemicals and the agent is a computer system for determining how to fold the protein chain or synthesize the chemical. In this example, the DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application actions are possible folding actions for folding the protein chain or actions for assembling precursor chemicals/intermediates and the result to be achieved may include, e.g., folding the protein so that the protein is stable and so that it achieves a particular biological function or providing a valid synthetic route for the chemical. As another example, the agent may be a mechanical agent that indirectly performs or controls the protein folding actions, or chemical synthesis steps, e.g. by controlling synthesis steps selected by the system automatically without human interaction. The observations may comprise direct or indirect observations of a state of the protein or chemical/ intermediates/precursors and/or may be derived from simulation. Thus the system may be used to automatically synthesize a protein with a particular function such as having a binding site shape, e.g. a ligand that binds with sufficient affinity for a biological effect that it can be used as a drug. For example e.g. it may be an agonist or antagonist of a receptor or enzyme; or it may be an antibody configured to bind to an antibody target such as a virus coat protein, or a protein expressed on a cancer cell, e.g. to act as an agonist for a particular receptor or to prevent binding of another ligand and hence prevent activation of a relevant biological pathway. [0035] In a similar way the environment may be a drug design environment such that each state is a respective state of a potential pharmaceutically active compound pharmaceutically active compound and the agent is a computer system for determining elements of the pharmaceutically active compound and/or a synthetic pathway for the pharmaceutically active compound. The drug/synthesis may be designed based on a reward derived from a target for the pharmaceutically active compound, for example in simulation. As another example, the agent may be a mechanical agent that performs or controls synthesis of the pharmaceutically active compound. [0036] In some further applications, the environment is a real-world environment and the agent manages distribution of tasks across computing resources e.g. on a mobile device and/or in a data center. In these implementations, the actions may include assigning tasks to particular computing resources. In these applications, the observations may include observations of computing resources such as compute and/or memory capacity, or Internet-accessible resources; and the actions may include assigning tasks to particular computing resources. The reward(s) may be configured to maximize or minimize one or more of: utilization of computing resources, electrical power, bandwidth, and computation speed. In these applications, the agent may be a software agent configured to manage distribution of tasks across computing resources. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0037] In some applications the agent may be a software agent i.e. a computer program, configured to perform a task. For example the environment may be a circuit or an integrated circuit design or routing environment and the agent may be configured to perform a design or routing task for routing interconnection lines of a circuit or of an integrated circuit e.g. an ASIC. The reward(s) may then be dependent on one or more routing metrics such as interconnect length, resistance, capacitance, impedance, loss, speed or propagation delay; and/or physical line parameters such as width, thickness or geometry, and design rules. The reward(s) may also or instead include one or more reward(s) relating to a global property of the routed circuitry e.g. component density, operating speed, power consumption, material usage, a cooling requirement, level of electromagnetic emissions, and so forth. The observations may be e.g. observations of component positions and interconnections; the actions may comprise component placing actions e.g. to define a component position or orientation and/or interconnect routing actions e.g. interconnect selection and/or placement actions. The task may be, e.g., to optimize circuit operation to reduce electrical losses, local or external interference, or heat generation, or to increase operating speed, or to minimize or optimize usage of available circuit area. The system may be configured to cause the circuit or integrated circuit to be made to the design, or with interconnection lines routed as determined by the system. [0038] In another example the software agent manages the processing, e.g. by one or more real-world servers, of a queue of continuously arriving jobs. The observations may comprise observations of the times of departures of successive jobs, or the time intervals between the departures of successive jobs, or the time a server takes to process each job, e.g. the start and end of a range of times, or the arrival times, or time intervals between the arrivals, of successive jobs, or data characterizing the type of job(s). The actions may comprise actions that allocate particular jobs to particular computing resources; the reward(s) may be configured to minimize an overall queueing or processing time or the queueing or processing time for one or more individual jobs, or in general to optimize any metric based on the observations. [0039] As another example the environment may comprise a real-world computer system or network, the observations may comprise any observations characterizing operation of the computer system or network, the actions performed by the software agent may comprise actions to control the operation e.g. to limit or correct abnormal or undesired operation e.g. because of the presence of a virus or other security breach, and the DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application reward(s) may comprise any metric(s) characterizing desired operation of the computer system or network. [0040] In some applications the environment is a data packet communications network environment, and the agent is part of a router to route packets of data over the communications network. The actions may comprise data packet routing actions and the observations may comprise e.g. observations of a routing table which includes routing metrics such as a metric of routing path length, bandwidth, load, hop count, path cost, delay, maximum transmission unit (MTU), and reliability. The reward(s) may be defined in relation to one or more of the routing metrics i.e. configured to maximize one or more of the routing metrics. [0041] Some of the above applications generally relate to the agent managing computing resources. Computing resources may include, for example computation speed, memory capacity, electrical power, bandwidth, queue length, circuit area usage, cooling requirements, or any other of the resources mentioned above. It should be understood that a general-purpose computer or distributed system has a finite amount of any of these resources. In these applications, the agent may be configured to manage computational resources in a real-world computer system or related hardware e.g. circuit or integrated circuit. [0042] As further example, the actions may include presenting advertisements, the observations may include advertisement impressions or a click-through count or rate, and the reward may characterize previous selections of items or content taken by one or more users. [0043] In some cases, the observations may include textual or spoken instructions provided to the agent by a third-party (e.g., an operator of the agent). For example, the agent may be an autonomous vehicle, and a user of the autonomous vehicle may provide textual or spoken instructions to the agent (e.g., to navigate to a particular location). [0044] In some other applications the environment is an Internet or mobile communications environment and the agent is a software agent which manages a personalized recommendation for a user. The observations may comprise previous actions taken by the user, e.g. features characterizing these; the actions may include actions recommending items such as content items to a user. The reward(s) may be configured to maximize one or more of: an estimated likelihood that the user will respond favorably to being recommended the (content) item, a suitability unsuitability of one or more DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application recommended items, a cost of the recommended item(s), and a number of recommendations received by the user, optionally within a time span. [0045] As another example the environment may be an electrical, mechanical or electro- mechanical design environment, e.g. an environment in which the design of an electrical, mechanical or electro-mechanical entity is simulated. The simulated environment may be a simulation of a real-world environment in which the entity is intended to work. The task may be to design the entity. The observations may comprise observations that characterize the entity, i.e. observations of a mechanical shape or of an electrical, mechanical, or electro-mechanical configuration of the entity, or observations of parameters or properties of the entity. The actions may comprise actions that modify the entity e.g. that modify one or more of the observations. The rewards may comprise one or more metrics of performance of the design of the entity. For example rewards may relate to one or more physical characteristics of the entity such as weight or strength or to one or more electrical characteristics of the entity such as a measure of efficiency at performing a particular function for which the entity is designed. The design process may include outputting the design for manufacture, e.g. in the form of computer executable instructions for manufacturing the entity. The process may include making the entity according to the design. Thus a design of an entity may be optimized, e.g. by reinforcement learning, and then the optimized design output for manufacturing the entity, e.g. as computer executable instructions; an entity with the optimized design may then be manufactured. [0046] As previously described the environment may be a simulated environment. Generally in the case of a simulated environment the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions. For example the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle. Generally the agent may be implemented as one or more computers interacting with the simulated environment. [0047] The simulated environment may be a simulation of a particular real-world environment and agent. For example, the system may be used to select actions in the simulated environment during training or evaluation of the system and, after training, or evaluation, or both, are complete, the action selection policy may be deployed for DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application controlling a real-world agent in the particular real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear on and damage to the real- world environment or real-world agent and can allow the control neural network to be trained and evaluated on situations that occur rarely or are difficult or unsafe to re-create in the real-world environment. For example the system may be partly trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and afterwards deployed to control the real mechanical agent in the particular real-world environment. Thus in such cases the observations of the simulated environment relate to the real-world environment, and the selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment. [0048] In some implementations, as described above, the agent may not include a human being (e.g. it is a robot). Conversely, in some implementations the agent comprises a human user of a digital assistant such as a smart speaker, smart display, or other device. Then the information defining the task can be obtained from the digital assistant, and the digital assistant can be used to instruct the user based on the task. [0049] For example, the reinforcement learning system may output to the human user, via the digital assistant, instructions for actions for the user to perform at each of a plurality of time steps. The instructions may for example be generated in the form of natural language (transmitted as sound and/or text on a screen) based on actions chosen by the reinforcement learning system. The reinforcement learning system chooses the actions such that they contribute to performing a task. A monitoring system (e.g. a video camera system) may be provided for monitoring the action (if any) which the user actually performs at each time step, in case (e.g. due to human error) it is different from the action which the reinforcement learning system instructed the user to perform. Using the monitoring system the reinforcement learning system can determine whether the task has been completed. During an on-policy training phase and/or another phase in which the history buffer is being generated, the experience tuples may record the action which the user actually performed based on the instruction, rather than the one which the reinforcement learning system instructed the user to perform. The reward value of each experience tuple may be generated, for example, by comparing the action the user took with a corpus of data showing a human expert performing the task, e.g. using techniques known from imitation learning. Note that if the user performs actions incorrectly (i.e. performs a different action from the one the reinforcement learning system instructs the user to perform) this adds one more source of noise to sources of noise which may already DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application exist in the environment. During the training process the reinforcement learning system may identify actions which the user performs incorrectly with more than a certain probability. If so, when the reinforcement learning system instructs the user to perform such an identified action, the reinforcement learning system may warn the user to be careful. Alternatively or additionally, the reinforcement learning system may learn not to instruct the user to perform the identified actions, i.e. ones which the user is likely to perform incorrectly. [0050] More generally, the digital assistant instructing the user may comprise receiving, at the digital assistant, a request from the user for assistance and determining, in response to the request, a series of tasks for the user to perform, e.g. steps or sub-tasks of an overall task. Then for one or more tasks of the series of tasks, e.g. for each task, e.g. until a final task of the series the digital assistant can be used to output to the user an indication of the task, e.g. step or sub-task, to be performed. That is, the digital assistant can provide an instruction or indication to the user as an output. This may be done using natural language, e.g. on a display and/or using a speech synthesis subsystem of the digital assistant. Visual, e.g. video, and/or audio observations of the user performing the task may be captured, e.g. using the digital assistant. A system as described above may then be used to determine whether the user has successfully achieved the task e.g. step or sub-task, i.e. from the answer as previously described. If there are further tasks to be completed the digital assistant may then, in response, progress to the next task (if any) of the series of tasks, e.g. by outputting an indication of the next task to be performed. In this way the user may be led step-by-step through a series of tasks to perform an overall task. During the training of the neural network, training rewards may be generated e.g. from video data representing examples of the overall task (if corpuses of such data are available) or from a simulation of the overall task. [0051] As an illustrative example a user may be interacting with a digital assistant and ask for help performing an overall task consisting of multiple steps, e.g. cooking a pasta dish. While the user performs the task, the digital assistant receives audio and/or video inputs representative of the user's progress on the task, e.g. images or video or sound clips of the user cooking. The digital assistant uses a system as described above, in particular by providing it with the captured audio and/or video and a question that asks whether the user has completed a particular step, e.g. 'Has the user finished chopping the peppers?', to determine whether the user has successfully completed the step. If the answer confirms that the use has successfully completed the step then the digital assistant progresses to DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application telling the user to perform the next step or, if at the end of the task, or if the overall task is a single-step task, then the digital assistant may indicate this to the user. The digital assistant may then stop receiving or processing audio and/or video inputs to ensure privacy and/or reduce power use. [0052] In a further aspect there is provided a digital assistant device including a system as described above. The digital assistant can also include a user interface to enable a user to request assistance and to output information. In implementations this is a natural language user interface and may comprise a keyboard, voice input-output subsystem, and/or a display. The digital assistant can further include an assistance subsystem configured to determine, in response to the request, a series of tasks for the user to perform. In implementations this may comprise a generative (large) language model, in particular for dialog, e.g. a conversation agent such as Sparrow or Chinchilla. The digital assistant can have an observation capture subsystem to capture visual and/or audio observations of the user performing a task; and an interface for the above-described language model neural network (which may be implemented locally or remotely). The digital assistant can also have an assistance control subsystem configured to assist the user. The assistance control subsystem can be configured to perform the steps described above, for one or more tasks e.g. of a series of tasks, e.g. until a final task of the series. More particularly the assistance control subsystem and output to the user an indication of the task to be performed, capture, using the observation capture subsystem, visual or audio observations of the user performing the task, determine from the above-described answer whether the user has successfully achieved the task. In response the digital assistant can progress to a next task of the series of tasks and/or control the digital assistant, e.g. to stop capturing observations. [0053] In some implementations, the environment may not include a human being or animal. In other implementations, however, it may comprise a human being or animal. For example, the agent may be an autonomous vehicle in an environment which is a location (e.g. a geographical location) where there are human beings (e.g. pedestrians or drivers/passengers of other vehicles) and/or animals, and the autonomous vehicle itself may optionally contain human beings. The environment may also be at least one room (e.g. in a habitation) containing one or more people. The human being or animal may be an element of the environment which is involved in the task, e.g. modified by the task (indeed, the environment may substantially consist of the human being or animal). For example the environment may be a medical or veterinary environment containing at least DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application one human or animal subject, and the task may relate to performing a medical (e.g. surgical) procedure on the subject. In a further implementation, the environment may comprise a human user who interacts with an agent which is in the form of an item of user equipment, e.g. a digital assistant. The item of user equipment provides a user interface between the user and a computer system (the same computer system(s) which implement the reinforcement learning system, or a different computer system). The user interface may allow the user to enter data into and/or receive data from the computer system, and the agent is controlled by the action selection policy to perform an information transfer task in relation to the user, such as providing information about a topic to the user and/or allowing the user to specify a component of a task which the computer system is to perform. For example, the information transfer task may be to teach the user a skill, such as how to speak a language or how to navigate around a geographical location; or the task may be to allow the user to define a three-dimensional shape to the computer system, e.g. so that the computer system can control an additive manufacturing (3D printing) system to produce an object having the shape. Actions may comprise outputting information to the user (e.g. in a certain format, at a certain rate, etc.) and/or configuring the interface to receive input from the user. For example, an action may comprise setting a problem for a user to perform relating to the skill (e.g. asking the user to choose between multiple options for correct usage of the language, or asking the user to speak a passage of the language out loud), and/or receiving input from the user (e.g. registering selection of one of the options, or using a microphone to record the spoken passage of the language). Rewards may be generated based upon a measure of how well the task is performed. For example, this may be done by measuring how well the user learns the topic, e.g. performs instances of the skill (e.g. as measured by an automatic skill evaluation unit of the computer system). In this way, a personalized teaching system may be provided, tailored to the aptitudes and current knowledge of the user. In another example, when the information transfer task is to specify a component of a task which the computer system is to perform, the action may comprise presenting a (visual, haptic or audio) user interface to the user which permits the user to specify an element of the component of the task, and receiving user input using the user interface. The rewards may be generated based on a measure of how well and/or easily the user can specify the component of the task for the computer system to perform, e.g. how fully or well the three-dimensional object is specified. This may be determined automatically, or a reward may be specified by the user, e.g. a subjective measure of the user experience. In this way, a personalized system DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application may be provided for the user to control the computer system, again tailored to the aptitudes and current knowledge of the user. [0054] Optionally, in any of the above implementations, the observation at any given time step may include data from a previous time step that may be beneficial in characterizing the environment, e.g., the action performed at the previous time step, the reward received at the previous time step, or both. [0055] FIG.1 shows an example reinforcement learning system 100. The reinforcement learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. [0056] The reinforcement learning system 100 trains a policy neural network 150 that has a plurality of parameters by repeatedly updating the values of the plurality of parameters of the policy neural network 150 based on the interactions of an agent 104 with the environment 106. For example, the parameters can include the weights and, optionally, the biases of the layers of the neural network 150. [0057] The agent 104 interacts with the environment 106 by performing actions 102 selected by using the policy neural network 150 during training. Performance of the selected actions 102 by the agent 102 generally causes the environment 106 to transition into successive new states. [0058] The policy neural network 150 is a neural network that is configured to receive a policy network input that includes data that characterizes a state of the environment 106 (i.e., an “observation 108”) and to process the policy network input in accordance with the current values of the plurality of parameters of the policy neural network 150 to generate an action selection output that can be used to determine an action 102 to be performed by the agent 104 in response to the observation 108. [0059] For example, the action selection output can be a probability distribution over the set of possible actions that can be performed by agent in response to the observation. As another example, the action selection output can be a Q-value that is an estimate of the long-term time-discounted reward that would be received if the agent performs a particular action in response to the observation (in this example the policy network input can also include data that identifies the particular action). As another example, the action selection output can identify a particular action that is predicted to yield the highest long-term time- discounted reward if performed by the agent in response to the observation. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0060] The policy neural network 150 can have any appropriate architecture that allows the policy neural network 150 to process a policy network input that includes an observation 108 to generate an action selection output. As a particular example, when the observations 108 include high-dimensional sensor data, e.g., images or laser data, the policy neural network 150 can be a convolutional neural network. As another example, when the observations 108 include only relatively lower-dimensional inputs, e.g., sensor readings that characterize the current state of the robot, the policy neural network 150 can be a multi-layer perceptron. As yet another example, when the observations 108 include both high-dimensional sensor data and lower-dimensional inputs, the policy neural network 150 can include a convolutional encoder that encodes the high-dimensional data, a fully-connected encoder that encodes the lower-dimensional data, and a policy subnetwork that operates on a combination, e.g., a concatenation, of the encoded data to generate the action selection output. [0061] In some implementations, in FIG.1, the environment 106 can be a simulated environment implemented as one or more computer programs, and the agent 104 can be a simulated agent implemented as one or more computer programs. For example, the environment 106 is a computer simulation of any of the real-world environments mentioned above, and the agent 104 is a computer simulation any of the real-world agents mentioned above that can interact with the simulated environment through computer simulation. The actions 102 can generally include simulated versions of one or more of the previously described actions or types of actions. [0062] For example, the environment 106 can be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent 104 can be a simulated vehicle navigating through the motion simulation. As another example, the environment 106 can be a computer simulation of a real-world environment and the agent 104 can be a simulated robot interacting with the computer simulation. In these examples, the actions 102 can be control inputs to control the simulated vehicle or simulated robot. [0063] In other implementations, in FIG.1, the environment 106 can be a real-world environment, and the agent 104 can be a real-world agent that can interact with the real- world environment. In those implementations, the environment 106 can be reset to a certain state, for example, by physically placing the agent 104 at a particular location within the environment 106, instructing a machine or another system to modify aspects of the environment 106 to return to a particular state, and so on. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0064] To assist in the training of the policy neural network 150, the reinforcement learning system 100 maintains a history buffer 120 that stores environment interaction data generated as a result of the agent 104 interacting with the environment 106 during training. In particular, the history buffer 120 stores a plurality of history observations 108 received by the agent 104 during past interaction of the agent 104 with the environment 106. The past interaction of the agent 104 with the environment 106 includes one or more past interactions of the agent 104 with the environment 106. Each history observation 108 is associated with one of the past interactions. Each history observation 108 characterizes a respective previous state of the environment 106. That is, each history observation 108 comprises data characterizing a respective previous state of the environment 106. The data characterizing the respective previous state may include, for example, images, object position data, sensor data and/or any other data capable of characterizing a state of an environment. [0065] Optionally, the history buffer 120 additionally stores data identifying a plurality of history actions 102. For example, the history buffer 120 can store each of the plurality of history observations 108 in association with data defining a corresponding history action 102, e.g., in the format of history observation-action pairs. For each history observation 108, the corresponding history action 102 is an action that was performed by the agent 104 when the environment 106 was in the state characterized by the history observation 108. [0066] The history buffer 120 can be any suitable data structure implemented in any available memory, e.g., one or more physical or logical storage devices, that is accessible by the reinforcement learning system 100. A first-in first-out (FIFO) queue is an example of a suitable data structure. When implemented as a FIFO queue, the history observations 108 are stored in the history buffer 120 according to the time at which they are generated. At certain time points during the training, the most obsolete history observations 108 (and, when stored, data identifying the corresponding history actions 102) can be deleted to ensure that the history buffer 120 does not consume greater than a threshold amount of memory resource. Other examples of suitable data structure include a first-in last-out (FILO) queue, a last-in first-out (LIFO) queue, and so on. [0067] In particular, the reinforcement learning system 100 trains the policy neural network 150 over a plurality of training iterations. At each of at least some of the plurality of training iterations of the policy neural network 150, the reinforcement learning system 100 executes a local planning process followed by a data collection process. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0068] In the local planning process, the reinforcement learning system 100 samples a batch of history observations from the history buffer 120 and uses an uncertainty metric engine 130 to compute an uncertainty metric for each history observation included in the batch. The reinforcement learning system 100 then identifies a selected history observation 140 from the batch of history observations based on the uncertainty metrics. For example, the reinforcement learning system 100 can identify, as the selected history observation 140, a history observation that has a highest uncertainty metric amongst the batch of history observations. As another example, the reinforcement learning system 100 can identify, as the selected history observation 140, a history observation that has an uncertainty metric greater than a given threshold uncertainly metric. [0069] As will be explained in more detail below, the uncertainty metric engine 130 can compute an uncertainty metric for a given history observation that is obtained from the history buffer 120 in many different ways. Generally, however, the higher the uncertainty metric for a given history observation, the more likely that a state of the environment 108 characterized by the given history observation corresponds to an underexplored or unexplored state of the environment, i.e., corresponds to a state that has not been frequently visited by the agent (e.g., a number of times that the state visited by the agent is below a visit threshold number) as of the training iteration. The uncertainty metric for a particular observation may be generally considered an inverse measure of (e.g. a quantification of) how well trained the policy neural network 150 is with respect to that particular observation. [0070] In the data collection process, the reinforcement learning system 100 controls the agent 104 to return to an initial state of the environment characterized by the selected history observation 140. The initial state to which the agent 104 returns to may be referred to as a selected initial state or a training iteration initial state. That is, the initial state may not necessarily be associated with a state which occurred chronologically before other states. Rather, the initial state denotes an intended state for the agent 104 to return to at the current training iteration. After the reinforcement learning system 100 has controlled the agent 104 to return to the initial state of the environment characterized by the selected history observation 140, it controls the agent to perform one or more actions 102 selected using the policy neural network 150. This is in contrast to some conventional reinforcement learning systems that may control the agent 104 to return to a fixed initial state or a randomly selected initial state at each training iteration. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0071] In some implementations, the reinforcement learning system 100 controls the agent 104 for an episode of a task. An “episode” of a task is a sequence of interactions during which the agent 104 attempts to perform a single instance of the task starting from some starting state of the environment 106. In other words, each task episode begins with the environment 106 being in the initial state characterized by the selected history observation 140, and ends when the agent 104 has successfully completed the task or when some termination criterion is satisfied, e.g., the environment 106 enters a state that has been designated as a terminal state or the agent 104 performs a threshold number of actions 102 without successfully completing the task. [0072] In particular, the reinforcement learning system 100 selects the one or more actions 102 in accordance with the values of the plurality of parameters of the policy neural network 150. In this way, at each of the at least some of the plurality of training iteration of the policy neural network 150, after returning to the initial state characterized by the selected history observation 140, the agent 104 is subsequently controlled in accordance with a current action selection policy that is defined by the current values of the plurality of parameters of the policy neural network 150 as of the training iteration to generate new environment interaction data to be stored in the history buffer 120. As mentioned above, the new environment interaction data to be stored in the history buffer 120 includes the observations 108 received by the agent 104 during the data collection process and, optionally, data defining the actions 102 that are selected by the policy neural network 150 to be performed by the agent 104 during the data collection process. [0073] At the end of each training iteration, the reinforcement learning system 100 determines one or more updates to the current values of the plurality of parameters of the policy neural network 150 by training the policy neural network 150 on the environment interaction data obtained from the history buffer 120 by using an appropriate reinforcement learning technique, e.g., an online reinforcement learning technique. [0074] After the policy neural network 150 is trained, the trained policy neural network 150 can be used to control the interactions of a real-world agent with a real-world environment, e.g., to control the agent that was being simulated in the simulated environment. Training the policy neural network 150 based on interactions of an agent with a simulated environment (i.e., instead of a real-world environment) can avoid wear- and-tear on the agent and can reduce the likelihood that, by performing poorly chosen actions, the agent can damage itself or aspects of its environment. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0075] In some implementations, the reinforcement learning system 100 can provide, e.g., by a wired or wireless connection, data specifying the trained policy neural network 150, e.g., data specifying the trained values of the plurality of parameters, and, optionally, the architecture of the policy neural network 150, to an agent control system that can cause the agent to perform different actions. In other implementations, the reinforcement learning system 100 can directly function as an agent control system that can cause the agent to perform different actions. [0076] The agent control system controls the agent interacting with an environment by selecting actions to be performed by the agent and then causing the agent to perform the selected actions, such as by transmitting control data to the agent which instructs the agent to perform the action. Performance of the selected actions by the agent generally causes the environment to transition into successive new states. By repeatedly causing the agent to act in the environment, the agent control system can control the agent to complete a specified task. [0077] FIG.2 is a flow diagram of an example process 200 for training a policy neural network. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a reinforcement learning system, e.g., the reinforcement learning system 100 of FIG.1, appropriately programmed, can perform the process 200. [0078] The policy neural network is a neural network that includes a plurality of parameters and that is used to select actions to be performed by an agent interacting with an environment to perform a task. Performance of the selected actions by the agent generally causes the environment to transition into successive new states. [0079] To train the policy neural network, the system maintains a history buffer. The history buffer stores a plurality of history observations received by the agent during past interaction of the agent with the environment during training. Each history observation characterizes a respective previous state of the environment that has been visited by the agent during the training. Optionally, the history buffer stores each of the plurality of history observations in association with data defining a corresponding history action that was performed by the agent when the environment was in the state characterized by the history observation. That is, in some implementations, the history buffer stores a plurality of history observation-action pairs that each include a history observation and data defining a history action. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0080] The system can repeatedly perform iterations of the process 200 to update the values of the plurality of parameters of the policy neural network. The system can continue performing iterations of the process 200 until termination criteria for the training of the policy neural network have been satisfied, e.g., until the values of the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of the process 200 have been performed. Because the initial observation is selected at each training iteration based on the uncertainty metrics, training is more efficient and so the wall clock time is reduced compared to conventional methods. [0081] The system determines an initial observation of the environment (step 202). In some implementations, the system chooses between executing a local planning process and a random process to determine the initial observation. For example, the system can alternate between the local planning process and the random process to perform step 202 at different iterations of the process 200. That is, the system can execute the local planning process to determine the initial observation at the first, third, fifth, and so on iterations of the process 200, while the system can execute the random process to determine the initial observation at the second, fourth, sixth, and so on iterations of the process 200. As another example, at each iteration of the process 200, the system can sample a number with some measure of randomness from a predetermined range and then choose the local planning process if the sampled number is greater than a given threshold within the predetermined range. In other implementations, the system always executes the local planning process to determine the initial observation of the environment. [0082] In the random process, the system samples an observation from a given distribution of history observations and uses the sampled observation as the initial observation. The given distribution of history observations can generally be determined from the plurality of history buffers stored in the history buffer, e.g., as a weighted distribution of history observations where each distinct history observation is weighted by a total number of the history observation stored in the history buffer. [0083] In contrast, executing the local planning process to determine the initial observation of the environment will depend on the uncertainty metrics for the history observations stored in the history buffer, as will be explained in more detail with reference to FIG.3, which shows sub-steps 302-306 corresponding to executing the local planning process to perform step 202. [0084] The system samples a batch of history observations or, in some implementations, a batch of history observation-action pairs that each include a history observation and data DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application defining a history action, from the history buffer (step 302). The system can generally obtain different history observations (or different history observation-action pairs) at different iterations by performing the sampling with some measure of randomness, although the total number of the history observations (or the total number of the history observation-action pairs) obtained by the system at different iterations can be the same. [0085] The system computes an uncertainty metric for each history observation in the sampled batch of history observations or, in some implementations, an uncertainty metric for each history observation-action pair in the sampled batch of history observation-action pairs (step 304). The system can compute the uncertainty metric in any of a variety of different ways. [0086] For example, when the policy neural network is configured to action selection outputs in the form of Q-values, the system can use the policy neural network to compute an uncertainty metric for each observation-action pair included in the batch of history observation-action pairs. Specifically, the system can process, for each history observation-action pair, a policy network input that includes the history observation and data defining the history action in accordance with the current parameters values of the policy neural network to generate an action selection output that includes a Q-value for the history observation-action pair. The system can then compute a standard deviation of the respective Q values for the history observation-action pairs included in the sampled batch, and compute the uncertainty metric from the standard deviation. In some implementations, the system maintains multiple instances of the policy neural network, where each different instance of the policy neural network is used to process a different history observation- action pair included in the sampled batch to generate a Q-value for the history observation-action pair. [0087] In this example, the uncertainty metric for each history observation-action pair can thus be computed as: , where M is the number of the instances of the policy neural network maintained by the system. ^^^ ^^, ^^; ^ ^ ^^^ is the Q-value for an observation-action pair ^ ^^, ^^^ that includes a history observation s and a history action ^^. ^ ^ ^^ represents the parameter values of the ^^- th instance of the policy neural network, where ^ ^ ^ ^ ^ := ^^ ^^ + ^^^ . Specifically, the parameters of the ^^-th instance of the policy neural network are a summation of the DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application values of a trainable component ^^^ and the values of a fixed randomized prior policy neural network ^^^ ^ . [0088] Moreover, in this example, the system can similarly compute an uncertainty metric for each history observation included in the sampled batch of history observations. Specifically, for each history observation included in the sampled batch of history observations, the system can do this by randomly selecting an action from a set of possible actions that can be performed by the agent in response to the history observation, and then processing the history observation and data defining the randomly selected action using policy neural network to generate an action selection output. That is, the history action ^^ in the equation above can be replace by a randomly selected action ^^~ ^^ in implementations where the history actions are not stored in the history buffer in association with the history observations. [0089] As another example, the system can use a covariance-based method to compute the uncertainty metric for each history observation-action pair included in the batch of history observation-action pairs. Specifically, for each observation-action pair included in the sampled batch of history observation-action pairs, the system can process the history observation-action pair using a representation neural network to generate a feature vector, and then determine a covariance matrix of the feature vectors that have been generated for the sampled batch of history observation-action pairs. The system can then compute the uncertainty metric from the covariance matrix. [0090] The representation neural network can be any appropriate neural network. For example, the representation neural network can be a pre-trained neural network that has been pre-trained on a representation learning task. As another example, the representation neural network can be a randomly initialized neural network. As yet another example, the representation neural network can be a combination of both a pre-trained neural network and a randomly initialized neural network. [0091] In this example, the covariance matrix can be computed as: where ^^^ ^^, ^^^ represents the feature vector generated by using the representation neural network for an observation-action pair ^ ^^, ^^^ that includes a history observation s and a history action ^^, and ^^ is a regularization parameter. As a particular example, ^^^ ^^, ^^^ ൌ ⊗ ^^^, where ^^^ ^^^ represents random Fourier features of the history observation ^^ DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application extracted by the pre-trained neural network, and ^^^ is an | ^^|-dimensional vector defining the history action ^^. The uncertainty metric for each history observation can then be computed as: [0092] Like the previous example, in this example, the system can similarly compute an uncertainty metric for each history observation included in the sampled batch of history observations. Specifically, for each history observation included in the sampled batch of history observations, the system can do this by randomly selecting an action from a set of possible actions that can be performed by the agent in response to the history observation, and then processing the history observation and data defining the randomly selected action using representation neural network to generate a feature vector. That is, the history action ^^ in the equations above can be replace by a randomly selected action ^^~ ^^ in implementations where the history actions are not stored in the history buffer in association with the history observations. [0093] As another example, the system can compute the uncertainty metric by determining an estimation of a total count of each distinct history observation-action pair in the history buffer, and then computing the uncertainty metric from the estimations of the total counts of distinct history observations. In this example, the uncertainty metric for each history observation-action pair included in sampled batch of history observation- action pairs can be computed as: where ^^^ ^^, ^^^ is the estimated total count of the observation-action pair ^ ^^, ^^^, and ^^ is a regularization parameter. The estimated total count ^^^ ^^, ^^^ can be determined, e.g., by a predetermined function ^^: ^^ → ^^̅ that maps each observation ^^ ∈ ^^ to a corresponding discrete element in a discrete observation space ^^ ̅ where | ^^ ̅ | ^ ∞. [0094] Like the previous two examples, in this example, the system can similarly compute an uncertainty metric for each history observation included in the sampled batch of history observations. That is, the history action ^^ in the equations above can be replace by a randomly selected action ^^~ ^^ in implementations where the history actions are not stored in the history buffer in association with the history observations. [0095] As yet another example, the system can compute the uncertainty metric through random network distillation (RND). Specifically, for each history observation included in the sampled batch of history observations, the system can process the history observation DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application using a first neural network and a second neural network to generate a first feature vector and a second feature vector, respectively, and then compute the uncertainty metric from a difference between the first feature vector and the second feature vector. For example, the first neural network can have trained parameter values while the second neural network can have randomly initialized parameter values. [0096] In this example, the uncertainty metric for each history observation can be computed as: , where ^ ^ ^ represents the first neural network and ^^ is the second neural network, and the difference is computed as error. [0097] The system determines, based on the uncertainty metrics, a selected history observation from the batch of history observations or the batch of history observation- action pairs (step 306). The selected history observation is then used as the initial observation of the environment for the iteration. For example, the system can identify, as the selected history observation, a history observation that has a highest uncertainty metric amongst the sampled batch of history observations or the sampled batch of history observation-action pairs. As another example, the system can identify, as the selected history observation, a history observation that has an uncertainty metric greater than a given threshold uncertainly metric. In either example, because different batches of history observations or different batches of history observation-action pairs are sampled, the system will generally determine different selected history observations at different iterations. [0098] The system executes a data collection process that begins from (i.e. starts with) the initial observation of the environment (step 204). Step 204 is explained in more detail with reference to FIG. 4, which shows sub-steps 402-406 corresponding to step 204. [0099] In general, the system can repeatedly perform multiple iterations of sub-steps 402- 406 to execute the data collection process. By repeatedly performing sub-steps 402-406, the system can control the agent for an episode of the task that begins with the environment being in an initial state characterized by the initial observation. [0100] The system selects, at each of multiple (environment) time steps during the performance of the episode of the task, an action to be performed by the agent in response to an observation that characterizes the state of the environment at the time step (step 402). When the time step is the first time step during the task episode, the observation can be the DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application initial observation. In implementations where the history buffer stores a plurality of history observation-action pairs, the selected action can be a history action that is included in a particular observation-action pair that includes the initial observation. Alternatively, in implementations where the history actions are not stored in the history buffer in association with the history observations, the system can randomly select the action from a set of possible actions that can be performed by the agent in response to the initial observation, or can select the action by using the policy neural network. Specifically, the system can process a policy network input that includes the initial observation in accordance with current values of the plurality of parameters of the policy neural network to generate an action selection output that can be used to select the action. [0101] When the time step is a subsequent time step during the task episode, the observation can be an observation that characterizes a state that the environment transitioned into as a result of the system performing an action at a preceding time step that precedes the time step. The action can be selected by using the policy neural network. Specifically, the system can process a policy network input that includes the observation characterizing the state of the environment at the time step in accordance with the current values of the plurality of parameters of the policy neural network to generate an action selection output that can be used to select the action. Optionally, the system stores, in the history buffer, data identifying the selected action in association with the observation of the environment at the time step. [0102] The system controls the agent to perform the selected action to cause the environment to transition into a new state of the environment at a subsequent time step (step 404). [0103] The system stores, in the history buffer, a new observation that is received in response to the agent performing the selected action and that characterizes the new state of the environment at the subsequent time step (step 406). [0104] The system trains, using an online reinforcement learning technique, the policy neural network based on the environment interaction data obtained from the history buffer to update the current values of the plurality of parameters of the policy neural network (step 206). For example, the online reinforcement learning technique can be a deep Q network (DQN) technique. DQN technique is described in V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0105] As another example, the online reinforcement learning technique can be a double deep Q network (DDQN) technique. DDQN technique is described in H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double Q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016. [0106] As another example, the online reinforcement learning technique can be a bootstrapped DQN technique. Bootstrapped DQN technique is described in I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN. Advances in Neural Information Processing Systems, 29, 2016. [0107] As another example, the online reinforcement learning technique can be a distributional DDQN technique. Distributional DDQN technique is described in M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pages 449–458. PMLR, 2017. [0108] As yet another example, the online reinforcement learning technique can be an approximate policy iteration technique. Approximate policy iteration technique is described in D. P. Bertsekas. Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications, 9(3):310–335, 2011. [0109] An example algorithm for training the policy neural network is shown below. [0110] In the example algorithm shown above, at each training iteration, the system chooses between executing a local planning process and a random process to determine the initial observation ^^^ based on sampling a number from a predetermined range of ^0,1^. The system chooses the local planning process if the sampled number is greater than a given threshold ^^^^^௧. ^^^^^௧ can be an adjustable parameter of the system. For example, ^^^^^௧ can depend on the history buffer, the size of the batch of history observations or history observation-action pairs to be sampled from the history buffer at each training DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application iteration, or both. “Agent.UPDATE()” refers to training the policy neural network using an online reinforcement learning technique to update the current parameters values of the policy neural network. [0111] An example algorithm for the data collection process (“DATACOLLECTION”) is shown below. [0112] In the example algorithm shown above, the system controls the agent to return to an initial state characterized by the initial observation ^^^: “ENV.RESET( ^^^)”, and then to perform an episode of the task that begins with the environment being in initial state. “ ^^ ←Agent.ACT( ^^)” refers to selecting, in accordance with the values of the plurality of parameters of the policy neural network, an action ^^ to be performed by the agent in response to an observation ^^. The observation ^^ is stored into the history buffer ^^: “ ^^⋃^ ^^^”. Performance of the ^^ causes the environment to transition (or “step”) into a new state that has a new observation: “ ^^ ←ENV.STEP( ^^)”. [0113] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. [0114] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. [0115] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. Thus a system, artificial neural network, or trained artificial neural network as described herein, can be implemented in hardware using electronic circuitry, e.g. in a physical box. Similarly computer code as described herein can be code to emulate such hardware or code for a hardware description language. [0116] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application [0117] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently. [0118] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers. [0119] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers. [0120] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few. [0121] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. [0122] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return. [0123] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads. [0124] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a JAX framework. [0125] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet. [0126] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device. [0127] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. [0128] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. [0129] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application CLAIMS 1. A computer-implemented method for training a policy neural network used to select actions performed by an agent interacting with an environment by performing actions that cause the environment to transition states, the method comprising: maintaining a history buffer that stores information about a plurality of history observations received by the agent during past interaction of the agent with the environment, wherein each history observation characterizes a respective previous state of the environment; executing a local planning process using the history buffer to determine an initial observation of the environment, wherein executing the local planning process comprises: sampling a batch of history observations from the history buffer; computing an uncertainty metric for each history observation in the batch; and determining, based on the uncertainty metrics, a particular history observation from the batch of history observations as the initial observation of the environment; executing a data collection process that begins from the determined initial observation of the environment, wherein executing the data collection process comprises: selecting an action to be performed by the agent using the policy neural network in accordance with current parameters values of the policy neural network; controlling the agent to perform the selected action to cause the environment to transition into a new state of the environment; and storing, in the history buffer, a new observation that is received in response to the agent performing the action and that characterizes the new state of the environment; and training, using an online reinforcement learning technique, the policy neural network on the history buffer to update the current parameters values of the policy neural network. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application 2. The method of claim 1, wherein executing the local planning process further comprises randomly selecting an action from a set of possible actions that can be performed by the agent in response to the particular history observation, and wherein executing the data collection process comprises: controlling the agent to perform the randomly selected action in response to the initial observation of the environment. 3. The method of claim 1, wherein computing the uncertainty metric for each sampled history observation comprises: computing an uncertainty metric for each observation-action pair that includes (i) the sampled history observation and (ii) a respective one of the set of possible actions that can be performed by the agent in response to the sampled history observation. 4. The method of claim 3, wherein executing the data collection process comprises: controlling the agent to perform a particular action in response to the initial observation of the environment, wherein the particular action is included in a particular observation-action pair that includes the particular history observation. 5. The method of any one of claims 1-4, further comprising: sampling another initial observation from a given distribution; and executing another instance of the data collection process that begins from the sampled initial observation of the environment. 6. The method of any one of claims 1-5, wherein the policy neural network is configured to process a policy network input that includes one of the sampled batch of history observations in accordance with the current parameters values of the policy neural network to generate a Q-value for the sampled history observation. 7. The method of claim 6, wherein computing the uncertainty metric for each sampled history observation comprises: computing a standard deviation of respective Q-values for the sampled history observation that have been determined using multiple instances of the policy neural network; and computing the uncertainty metric from the standard deviation. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application 8. The method of any one of claims 1-5, wherein computing the uncertainty metric for each sampled history observation comprises: processing the sampled history observation using a representation neural network to generate a feature vector; determining a covariance matrix of the feature vectors that have been generated for the sampled batch of history observations; and computing the uncertainty metric from the covariance matrix. 9. The method of any one of claims 1-5, wherein computing the uncertainty metric for each sampled history observation comprises: determining an estimation of a total count of each distinct history observation in the history buffer; and computing the uncertainty metric from the estimations of the total counts of distinct history observations. 10. The method of any one of claims 1-5, wherein computing the uncertainty metric for each sampled history observation comprises: processing the sampled history observation using a first neural network having trained parameter values to generate a first feature vector; processing the sampled history observation using a second neural network having randomly initialized parameter values to generate a second feature vector; and computing the uncertainty metric from a difference between the first feature vector and the second feature vector. 11. The method of any one of claims 1-10, wherein determining the particular history observation from the batch of history observations comprises: selecting a history observation that has a highest uncertainty metric from amongst the batch of history observations. 12. The method of any one of claims 1-11, wherein the history buffer is a first-in, first- out (FIFO) queue. 13. The method of any one of claims 1-12, wherein the online reinforcement learning technique comprises a DQN technique, a DDQN technique, a bootstrapped DQN technique, a distributional DDQN technique, or a policy iteration technique. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application 14. The method of any one of claims 1-13, wherein the agent is a mechanical agent and the environment is a real-world environment. 15. The method of claim 14, wherein the agent is a robot. 16. The method of any one of claims 1-13, wherein the environment is a real-world environment of a service facility comprising a plurality of items of electronic equipment and the agent is an electronic agent configured to control operation of the service facility. 17. The method of any one of claims 1-13, wherein the environment is a real-world manufacturing environment for manufacturing a product and the agent comprises an electronic agent configured to control a manufacturing unit or a machine that operates to manufacture the product. 18. The method of any one of claims 1-13, wherein the environment is a simulation of a real-world environment and wherein the method further comprises: after training the policy neural network, controlling a real-world agent in the real- world environment using the policy neural network. 19. The method of any one of claims 1-13, wherein the environment is a simulation of a real-world environment and wherein the method further comprises: after training the policy neural network, providing data specifying the policy neural network for use in controlling a real-world agent in the real-world environment. 20. The method of any one of claims 1-13, wherein the agent is a digital assistant and wherein actions performed by the agent include outputs that are provided by the digital assistant to a user. 21. The method of claim 20, wherein the outputs include one or more of: text displayed to a user in a user interface of the digital assistant; an image displayed to the user in the user interface of the digital assistant; or speech output through one or more speakers of the digital assistant. 22. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the operations of the respective method of any one of claims 1-21. DeepMind Technologies Limited F&R Ref.: 45288-0315WO1 PCT Application 23. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1-21.
EP24702704.8A 2023-01-26 2024-01-26 Learning reinforcement learning policies with local planning Pending EP4655718A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202363441429P 2023-01-26 2023-01-26
PCT/EP2024/051924 WO2024156877A1 (en) 2023-01-26 2024-01-26 Learning reinforcement learning policies with local planning

Publications (1)

Publication Number Publication Date
EP4655718A1 true EP4655718A1 (en) 2025-12-03

Family

ID=89771808

Family Applications (1)

Application Number Title Priority Date Filing Date
EP24702704.8A Pending EP4655718A1 (en) 2023-01-26 2024-01-26 Learning reinforcement learning policies with local planning

Country Status (2)

Country Link
EP (1) EP4655718A1 (en)
WO (1) WO2024156877A1 (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119449373A (en) * 2024-10-14 2025-02-14 北京理工大学 An automated penetration testing method based on reinforcement learning

Also Published As

Publication number Publication date
WO2024156877A1 (en) 2024-08-02

Similar Documents

Publication Publication Date Title
EP4639415A1 (en) Training reinforcement learning agents to perform multiple tasks across diverse domains
US20240265263A1 (en) Methods and systems for constrained reinforcement learning
JP7354460B2 (en) Learning environment representation for agent control using bootstrapped latency predictions
US20250348748A1 (en) System and method for reinforcement learning based on prior trajectories
JP7820626B2 (en) Multi-objective reinforcement learning using weighted policy projection
WO2021058583A1 (en) Training action selection neural networks using q-learning combined with look ahead search
US20250209338A1 (en) Exploration by bootstrapped prediction
KR20230153481A (en) Reinforcement learning using ensembles of discriminator models
US20240232642A1 (en) Reinforcement learning using epistemic value estimation
WO2024236081A1 (en) Imitation learning using shaped rewards
US20250124297A1 (en) Controlling reinforcement learning agents using geometric policy composition
EP4655718A1 (en) Learning reinforcement learning policies with local planning
US20240256882A1 (en) Reinforcement learning by directly learning an advantage function
US20250068919A1 (en) Reinforcement learning using hindsight to model unpredictable aspects of the future
CN118871917A (en) Reinforcement learning using meta-policies to explore environments
US20230325635A1 (en) Controlling agents using relative variational intrinsic control
US20240256883A1 (en) Reinforcement learning using quantile credit assignment
US20240256884A1 (en) Generating environment models using in-context adaptation and exploration
US20240126945A1 (en) Generating a model of a target environment based on interactions of an agent with source environments
US20240386281A1 (en) Controlling agents by transferring successor features to new tasks
CN121241348A (en) Active offline policy selection using policy representation
WO2025190472A1 (en) Controlling an agent using pre-committed sequences of actions
EP4526809A1 (en) Model-free reinforcement learning with regularized nash dynamics
WO2024068789A1 (en) Learning tasks using skill sequencing for temporally-extended exploration

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20250826

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)