US11132609B2 - Multi-task neural network systems with task-specific policies and a shared policy - Google Patents

Multi-task neural network systems with task-specific policies and a shared policy Download PDF

Info

Publication number
US11132609B2
US11132609B2 US16/689,020 US201916689020A US11132609B2 US 11132609 B2 US11132609 B2 US 11132609B2 US 201916689020 A US201916689020 A US 201916689020A US 11132609 B2 US11132609 B2 US 11132609B2
Authority
US
United States
Prior art keywords
task
policy
training
shared
policies
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US16/689,020
Other languages
English (en)
Other versions
US20200090048A1 (en
Inventor
Razvan Pascanu
Raia Thais Hadsell
Victor Constant Bapst
Wojciech Czarnecki
James Kirkpatrick
Yee Whye Teh
Nicolas Manfred Otto Heess
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
DeepMind Technologies Ltd
Original Assignee
DeepMind Technologies Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by DeepMind Technologies Ltd filed Critical DeepMind Technologies Ltd
Priority to US16/689,020 priority Critical patent/US11132609B2/en
Assigned to DEEPMIND TECHNOLOGIES LIMITED reassignment DEEPMIND TECHNOLOGIES LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KIRKPATRICK, JAMES, CZARNECKI, Wojciech, BAPST, Victor Constant, HADSELL, Raia Thais, HEESS, NICOLAS MANFRED OTTO, PASCANU, Razvan, TEH, Yee Whye
Publication of US20200090048A1 publication Critical patent/US20200090048A1/en
Priority to US17/486,842 priority patent/US11983634B2/en
Application granted granted Critical
Publication of US11132609B2 publication Critical patent/US11132609B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/004—Artificial life, i.e. computing arrangements simulating life
    • G06N3/006—Artificial life, i.e. computing arrangements simulating life based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/10—Interfaces, programming languages or software development kits, e.g. for simulating neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G06N5/043—Distributed expert systems; Blackboards

Definitions

  • This specification relates to methods of training neural networks to perform multiple tasks, and to adaptive computer systems, such as neural network systems, for performing multiple tasks.
  • Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input.
  • Some neural networks are deep neural networks that include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer.
  • Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.
  • an agent interacts with an environment by performing actions that are selected by the reinforcement learning system in response to receiving observations that characterize the current state of the environment.
  • Some reinforcement learning systems select the action to be performed by the agent in response to receiving a given observation in accordance with an output of a neural network.
  • This specification describes an adaptive system, such as a neural network system, implemented as computer programs on one or more computers in one or more locations for performing multiple different tasks, and methods for training a system to perform such tasks.
  • the tasks are related; respective parts of the system may then learn a shared behavior and a task-specific behavior. Knowledge may be shared between these parts of the system.
  • Workers may be associated with corresponding tasks or groups of tasks.
  • the worker(s) associated with each task or group of tasks are arranged to perform that task, or group of tasks.
  • the system may employ and/or generate, for each task, at least one respective “task policy” which indicates how the worker(s) associated with that task should perform the task. If there is one worker per policy, this may equivalently be regarded as a “worker policy”. If there are multiple workers associated with a given one of the tasks, they may share the task policy for that task (i.e. the task policy is used as a worker policy for each of those multiple workers), or alternatively each of the multiple workers may have a respective worker policy which is a task policy for that task.
  • a shared or common part of the system may learn a multitask policy.
  • the shared part may be, but need not be, a neural network.
  • the task policies may stay close to the common, multitask policy according to a measure of distance between the policies; for example the multitask policy may define a centroid of the task policies.
  • the task policies may be, but need not be, implemented using respective neural networks. They may, but need not, operate substantially independently of one another.
  • the system may be part of a reinforcement learning system.
  • the tasks may comprise the control of respective agents (“workers”) acting on or within one or more environments (e.g. the same environment or respective environments).
  • the system may output data (“action data”) for selecting actions for the agents to perform.
  • action data For example the system may input data to observe or otherwise receive data relating to states of the environment(s), and based on an observed state may determine an action to perform.
  • Each environment may be real or simulated.
  • the data for selecting the actions to perform may be generated using the one or more task policies.
  • Each task policy may specify an action to be performed for an observed state, or may specify a distribution from which the action to be performed for the observed state may be obtained as a sample.
  • the system may be part of a reinforcement learning system, it need not be.
  • the techniques described can be applied to tasks such as image or video recognition or classification, for example using a feedforward network with convolution.
  • the system can be configured to receive any kind of digital data input and to generate any kind of score, classification, or regression output based on the input.
  • the outputs generated by the neural network for a given image may be scores for each of a set of object categories, with each score representing an estimated likelihood that the image contains an image of an object belonging to the category.
  • the output generated by the neural network for a given Internet resource, document, or portion of a document may be a score for each of a set of topics, with each score representing an estimated likelihood that the Internet resource, document, or document portion is about the topic.
  • the output generated by the neural network may be a score that represents an estimated likelihood that the particular advertisement will be clicked on.
  • the output generated by the neural network may be a score for each of a set of content items, with each score representing an estimated likelihood that the user will respond favorably to being recommended the content item.
  • the output generated by the neural network may be a score for each of a set of pieces of text in another language, with each score representing an estimated likelihood that the piece of text in the other language is a proper translation of the input text into the other language.
  • the output generated by the neural network may be a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript for the utterance.
  • the environment may be a simulated environment such as video game and the worker may be a simulated user playing the video game.
  • the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the worker is a simulated vehicle navigating through the motion simulation.
  • the actions may be control inputs to control the simulated user or simulated vehicle.
  • the environment is a real-world environment and the worker is a mechanical worker interacting with the real-world environment.
  • the worker may be a robot interacting with the environment to accomplish a specific task.
  • the worker may be an autonomous or semi-autonomous vehicle navigating through the environment.
  • the actions may be control inputs to control the robot or the autonomous vehicle.
  • one innovative aspect of the subject matter described in this specification can be embodied in methods of training a multitask computer system, such as a multitask neural network system, the system including a set of trainable workers in combination with a shared module, such as a shared policy network.
  • the said trainable workers and the shared module may be trained in a coordinated manner on a plurality of different tasks such that each worker learns to perform a corresponding associated task according to a respective task policy, and the shared module learns a multitask policy representing common behavior for the tasks.
  • the system may learn via a supervised or unsupervised loss, for example with reinforcement learning.
  • the shared part of the system i.e. the shared module, e.g. the shared policy network
  • the shared part of the system i.e. the shared module, e.g. the shared policy network
  • the propagated errors may be determined by optimizing an objective function.
  • the objective function may comprise a first entropy term measuring a difference between a distribution of the task policy and a distribution of the multitask policy. This term may comprise a Kullback-Leibler divergence, and may be discounted by a discount factor.
  • the objective function may also comprise a second entropy term dependent upon the distribution of the task policy, to encourage exploration. The second entropy term may also be discounted by the same or a different discount factor.
  • the task policy distribution and the multitask policy distribution may comprise state-action distributions.
  • the objective function may include a discounted reward term dependent upon an expected reward from taking an action in a state.
  • Such a reinforcement learning system may be based upon, for example, a Deep Q-Network (DQN), a Double-DQN, an Advantage Actor Critic (A3C) network, or other architectures.
  • Learning may be synchronous or asynchronous; for example the Hogwild!Algorithm may be employed for parallel stochastic gradient descent (or ascent) learning.
  • the coordinated training may involve alternating between training one or more of the trainable workers (i.e. the respective task policy of the associated tasks) and training the shared policy network.
  • the system may alternate between learning the task (worker) policies and distilling the multitask policy, for example learning in an iterative manner.
  • a set of worker neural networks and the shared policy network may be trained jointly, i.e. optimized together for both worker (task) and multitask policies, for example by using a combined loss function as described above.
  • a separate network may be used for each worker (task) policy and the shared policy network only for the multitask policy, or worker networks may define soft advantage over the multitask policy, where the soft advantage may be parameterized by a controllable temperature value.
  • the coordinated training may comprise back propagation including a gradient term which aims to match probabilities from the task policies to probabilities from the multitask policy to distil the multitask policy from the task policies.
  • a multitask neural network system processes input data relating to one of a plurality of learned tasks to be performed and provides output data to control performance of the task.
  • Such a system may comprise an input to receive the input data and a set of workers, coupled to the input, each configured to perform a respective task by outputting action data according to a respective task policy.
  • a shared policy network may be coupled to one or both of the input and the set of workers, and may define a learned multitask policy representing common behavior for the tasks.
  • the multitask neural network system may be configured to combine the output of an adaptive system for each task with the learned multitask policy to define a task policy for the task, to enable at least one worker to perform the corresponding learned task.
  • the workers may comprise neural networks; they may share some, all, or no weights with the shared policy network.
  • the system may have a multicolumn architecture in which worker neural network modules define at least one column of neural network layers, and in which the shared policy network defines a second column of neural network layers.
  • the workers and shared policy network may together define, for example, a reinforcement learning system or a feedforward neural network, for example with convolution.
  • the subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.
  • the methods can be used to more efficiently train a neural network.
  • the networks can more effectively benefit from transfer learning in which training on one task can be used to bootstrap training on another, related task. This may reduce the total amount of training data required per task.
  • the methods can be used to distil common behavior from workers operating on multiple different tasks. Moreover such systems can learn faster and produce improved final performance on a task after training. They may be more robust to hyper-parameter settings. They may also have fewer parameters overall for a given performance on each task.
  • the techniques are generally applicable to many different neural network architectures, including architectures for supervised and unsupervised learning, architectures for reinforcement learning, and architectures for data classification.
  • FIG. 1 shows schematically a multitask adaptive computer system such as a multitask neural network system.
  • FIG. 2 shows the multitask adaptive computer system of FIG. 1 used to perform reinforcement learning.
  • FIG. 3 shows a method performed by the multitask adaptive computer system.
  • FIG. 4 shows a method for training the multitask adaptive computer system.
  • FIG. 5 shows a portion of a first multitask neural network system.
  • FIG. 6 shows a portion of a second multitask neural network system.
  • This specification discloses a multitask adaptive computer system, such as a multitask neural network system.
  • the system is to be implemented by one or more computers configured to perform particular operations or actions.
  • Each computer has installed on it software, firmware, hardware, or a combination of them that in operation cause the computer to perform the operations or actions.
  • For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.
  • FIG. 1 provides a high level illustration of the multitask adaptive computer system. For the sake of example only it is illustrated as being for performing four tasks, but the system is readily generalized to any plural number of tasks.
  • the neural network system may comprise four workers 11 , 12 , 13 , 14 which are each for performing a respective one of the tasks.
  • the number of workers may be higher than the number of tasks, such that one of more of the tasks may be associated with a respective sub-set of the workers which includes a plurality of the workers, and those task(s) are performed by the associated set of workers.
  • the subsets of the workers may be non-overlapping.
  • Each worker generates an output which is a respective function of its inputs.
  • the function depends upon the task which the worker for performing, and is referred to as a respective task policy (alternatively it can be referred to as a worker policy).
  • the respective task policies for the four workers are respectively denoted by ⁇ 1 , ⁇ 2 , ⁇ 3 and ⁇ 4 .
  • Each policy may not be deterministic. In other words, it may be a sample from a probability distribution defined by the respective policy, which specifies respective probabilities for all possible outputs of the worker.
  • the multitask adaptive computer system comprises a shared module 15 , which stores a “multitask policy”, denoted ⁇ 0 , which is also a function which generates at least one output from its inputs.
  • the function may optionally be stochastic; that is, it may be a sample from a probability distribution defined by the multitask policy.
  • the shared module 15 may be implemented as a neural network referred to here as a “shared policy network”.
  • the task policies ⁇ 1 , ⁇ 2 , ⁇ 3 and ⁇ 4 , and the multitask policy ⁇ 0 are generated together in a coordinated training procedure described below.
  • each worker learns to perform a respective one of the tasks according to the respective task policy, and the shared policy network learns the multitask policy ⁇ 0 .
  • the multitask policy ⁇ 0 is adapted to “distil” the common behaviors or representations from the task-specific task policies ⁇ 1 , ⁇ 2 , ⁇ 3 and ⁇ 4 .
  • the multitask policy ⁇ 0 is used to “regularize” the task policies ⁇ 1 , ⁇ 2 , ⁇ 3 and ⁇ 4 , such that the task policies do not overly diverge from the multitask policy.
  • the regularization may be based on a Kullback-Leibler (KL) divergence. In this way, knowledge gained in one task is distilled into the shared policy, then transferred to other tasks.
  • KL Kullback-Leibler
  • the tasks performed by the respective workers are generating actions to be performed by respective agents which act on, or within, one or more environments.
  • each agent may act on, or within, the same environment, and this is assumed below.
  • there may be multiple environments, and a corresponding one or more of the agents may act on, or within, each environment; the environments typically resemble each other according to some criterion.
  • the computer system receives data (“state data”) characterizing the current state of the environment and causes each agent to perform a respective action a which an element of an action space denoted by A, i.e., a discrete action space or continuous action space, in response to the received data.
  • A an element of an action space denoted by A
  • A i.e., a discrete action space or continuous action space
  • Data characterizing a state of the environment will be referred to in this specification as an observation.
  • the state data may be denoted as s t , and indicates the state of the (corresponding) environment at time t.
  • S The set of all possible realizations of the state data is denoted by S. Thus s t ⁇ S.
  • FIG. 2 An example of the situation is illustrated in FIG. 2 , where, for simplicity a single computer 2 is illustrated.
  • the computer 2 implements the multitask adaptive computer system illustrated in FIG. 1 .
  • the four tasks are to control four respective agents 21 , 22 , 23 , 24 which act on or within an environment 20 .
  • FIG. 3 illustrates a method 300 performed by the multitask computer system of FIG. 1 .
  • the computer 1 receives state data s t from the environment 20 (step 1).
  • the integer index i is used to label the tasks (thus, i takes one of the values 1 , 2 , 3 or 4 ). Since in this example the number of tasks equals the number of workers, the index i can also be considered as labelling the workers.
  • step 302 the four workers of the multitask computer system of FIG. 1 generate respective action data.
  • the action data specifies, for each of the agents 21 , 22 , 23 , 24 a respective action a t which is a respective element of A.
  • step 303 the computer 2 transmits the action data to the respective agents 21 , 22 , 23 , 24 .
  • step 304 once the agents 21 , 22 , 23 , 24 have performed the respective actions at on the environment 20 , the multitask computer system receives reward data specifying a reward ⁇ R i (a t , s t ) ⁇ for each of the tasks. These may be transmitted from the environment 20 to the computer 2 , or calculated by a separate module (not shown).
  • step 305 the policies for each of the agents and the multitask policy are updated.
  • step 306 it is determined whether a termination criterion is satisfied. If so, the method terminates (step 307 ). If not, the method returns to step 301 , to await the next state data. Index t is updated to t+1.
  • FIG. 2 can be varied in several respects.
  • the computer 2 may be replaced by multiple computers which act in concert to implement the multitask adaptive computer system.
  • the environment is a simulated environment (e.g., simulated by the computer 2 ) and each agent is implemented as one or more computers interacting with the simulated environment.
  • the simulated environment may be a video game and the agent may be a simulated user playing the video game.
  • the environment is a real-world environment and the agent is a mechanical agent interacting with the real-world environment.
  • the agent may be a robot interacting with the environment to accomplish a specific task or an autonomous or semi-autonomous vehicle navigating through the environment.
  • the observation can be data captured by one or more sensors of the agent as it interacts with the environment, e.g., a camera, a LIDAR sensor, a temperature sensor, and so forth.
  • the system has an advantage that each agent such as a robot, or autonomous or semi-autonomous vehicle can improve its interaction with a simulated or real-world environment.
  • ⁇ the discount factor
  • the method can be easily generalized to other scenarios like undiscounted finite horizon.
  • a given action is denoted by a, where a ⁇ A.
  • s, a) and reward functions R i (a,s) are different for each task i.
  • ⁇ i be a task-specific stochastic task policy for the respective i-th task.
  • the task policy can equivalently be considered as a worker policy.
  • the dynamics and policies give rise to joint distributions over state and action trajectories starting from some initial state, and we will also denote these distributions by ⁇ i .
  • the neural network system also stores a “multitask policy” ⁇ 0 .
  • the multitask policy ⁇ 0 will capture agent behavior that is common across the tasks.
  • the task policies and the multitask policy are generated together in a coordinated training process by optimizing an objective function which comprises a term indicative of expected returns and one or more regularization terms which provide policy regularization.
  • a first regularization term ensures that each task policy ⁇ i is regularized towards the multitask policy, and may be defined using discounted KL divergences
  • the objective function to be maximized is:
  • s t ) ] ⁇ ⁇ i ⁇ E ⁇ i [ ⁇ t ⁇ 0 ⁇ ⁇ t ⁇ R i ⁇ ( a t , s t ) + ⁇ ⁇ t ⁇ ⁇ ⁇ log ⁇ ⁇ ⁇ 0 ⁇ ( a t
  • s t ) term can be thought of as a reward shaping term which encourages actions which have high probability under the multitask policy, while the entropy term ⁇ log ⁇ i (a t
  • the same regularization costs c KL and c Ent for all tasks. However, it is straightforward to generalize this to task-specific costs; this can be important if tasks differ substantially in their reward scales and amounts of exploration needed, although it does introduce additional hyper-parameters.
  • This training process 400 is illustrated in FIG. 4 .
  • a step 401 an initial estimate of the multitask policy ⁇ 0 is set (e.g. all actions equally likely).
  • step 401 ⁇ i is modified with ⁇ 0 fixed.
  • ⁇ 0 fixed (1) decomposes into separate maximization problems for each task, and is an entropy regularized expected return with a redefined (regularized) reward
  • V i ⁇ ( s t ) 1 ⁇ ⁇ log ⁇ ⁇ a t ⁇ ⁇ 0 ⁇ ⁇ ( a t
  • Q i ⁇ ( a t , s t ) R i ⁇ ( a t , s t ) + ⁇ ⁇ ⁇ s t + 1 ⁇ p i ⁇ ( s t + 1
  • the Bellman updates are softened in the sense that the usual max operator over actions for the state values V i , is replaced by a soft-max at inverse temperature, which hardens into a max operator as: ⁇ .
  • the optimal policy ⁇ i is then a Boltzmann policy at inverse temperature ⁇ : ⁇ i ( a t
  • s t ) ⁇ 0 ⁇ (a t
  • s t ) e ⁇ Q i (a t ,s t ) ⁇ / ⁇ V i (s t ) ⁇ 0 ⁇ (a t
  • s t ) e ⁇ A i (a t ,s t ) (4) where A i (a,s) Q i (a,s) ⁇ V i (s t ) is a softened advantage function.
  • the softened state values V i (s) act as the log normalizers in the above.
  • the multitask policy ⁇ 0 can be interpreted as a policy prior, a perspective known in the literature on RL as probabilistic inference. However, unlike in past works, it is raised to a power of a ⁇ 1. This softens the effect of the prior ⁇ 0 on ⁇ i , and is the result of the additional entropy regularization beyond the KL divergence.
  • step 403 a modification is made to the multitask policy ⁇ 0 given the each of the modified task policies ⁇ i .
  • the neural network system will learn ⁇ 0 instead of it being pre-set as in some earlier multitask learning techniques (typically as a uniform distribution over actions).
  • the only terms in (1) depending on ⁇ 0 are:
  • a maximum likelihood (ML) estimator can be derived from state-action visitation frequencies under roll-outs in each task, with the optimal ML solution given by the mixture of state-conditional action distributions.
  • step 404 it is determined whether a termination criterion has been met. If not, the method 400 returns to step 402 . If so, the method terminates in step 405 .
  • the second approach to minimization of the objective function is based on the use of a policy gradient and a better parameterization.
  • the algorithm in the first approach alternates between maximization of the multitask policy ⁇ 0 and the task policies ⁇ i , and is pronounced of the EM algorithm for learning latent variable models, with ⁇ 0 playing the role of parameters, while ⁇ i plays the role of the posterior distributions for the latent variables.
  • ⁇ 0 and ⁇ i are implemented using, say, deep neural networks, such an alternating maximization procedure can be slower than optimizing (1) with respect to task policies and multitask policies jointly by stochastic gradient ascent.
  • the gradient update for ⁇ i is simply given by a policy gradient with an entropic regularization, and can be carried out within a framework such as advantage actor-critic.
  • One simple parameterization of policies would be to use a respective neural network (“task network”, or equivalently “worker network” if there is one worker per task) with parameters ⁇ i to implement each task policy ⁇ i , such that the function h ⁇ i (a
  • Another neural network (the “shared policy network”) is used to implement the multitask policy ⁇ 0 , where the function performed by the policy network h 0 is equated with ⁇ 0 .
  • This parameterization is illustrated schematically in FIG. 5 , where the shared policy network is denoted 50 , and one of the task networks is denoted by 5 i .
  • the policy network 50 implements the policy ⁇ 0 .
  • Each task network 5 i operates to implement a respective policy T ⁇ i .
  • the only interaction between the networks 50 and 5 i is during the training process, not when the multitask neural network system is in use performing the tasks.
  • the networks 50 and 5 i interact via the KL regularization.
  • the reward term of the objective function causes the task network 50 i to diverge from the policy network 50 to increase the expected reward.
  • the entropy term of the objective function produces a further tendency for the networks 50 and 5 i to diverge.
  • a neural network system can parameterize the multitask policy using a policy network with parameters:
  • s t ) exp ⁇ ( h ⁇ 0 ⁇ ( a t
  • s t ) f ⁇ i ⁇ ( a t
  • Equations (6) and (7) use “hat” notation to denote parameterized approximators of the corresponding quantities.
  • the policy for task i then becomes parameterized as,
  • s t ) ⁇ ⁇ ⁇ 0 ⁇ ( a t
  • s t ) ) ) ⁇ exp ⁇ ( ⁇ ⁇ ⁇ h ⁇ 0 ⁇ ( a t
  • the neural network system may use (8) to parameterize a policy which is optimized by policy gradients.
  • the situation is illustrated schematically in FIG. 6 , where the policy network is denoted 60 and the i-th task network by 6 i .
  • the shared policy network 60 implements the policy ⁇ 0 .
  • the policy network 60 and each task network 6 i operate together to produce a respective policy T ⁇ i .
  • This can be seen as a two-column architecture for the policy, with one column being the multitask policy, and the other being the adjustment required to specialize to task i.
  • the reward term of the objective function causes the task network 60 i to diverge from the policy network 60 to increase the expected reward.
  • the entropy term of the objective function produces a further tendency for the networks 60 and 60 i to diverge.
  • the first term is the same as for the policy gradient of ⁇ i .
  • the second term tries to match the probabilities under the task policy ⁇ circumflex over ( ⁇ ) ⁇ i and under the multitask policy ⁇ circumflex over ( ⁇ ) ⁇ 0 .
  • the second term would not be present if the system parameterized ⁇ i using the same architecture ⁇ circumflex over ( ⁇ ) ⁇ i , but did not use a KL regularization for the policy.
  • the presence of the KL regularization causes the multitask policy to learn to be the centroid of all task policies, in the sense that the second term would be zero if
  • the multitask neural network system has a purely entropy-regularized objective function which does not couple and transfer across tasks.
  • the multitask neural network system has a purely KL regularized objective function, which does couple and transfer across tasks, but might prematurely stop exploration if the multitask and task policies become similar and greedy.
  • the multitask neural network system has an objective function which includes both terms.
  • Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
  • Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus.
  • the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.
  • the computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal.
  • data processing apparatus encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers.
  • the apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • the apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
  • a computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.
  • a computer program may, but need not, correspond to a file in a file system.
  • a program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code.
  • a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
  • an “engine,” or “software engine,” refers to a software implemented input/output system that provides an output that is different from the input.
  • An engine can be an encoded block of functionality, such as a library, a platform, a software development kit (“SDK”), or an object.
  • SDK software development kit
  • Each engine can be implemented on any appropriate type of computing device, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs, smart phones, or other stationary or portable devices, that includes one or more processors and computer readable media. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices.
  • the processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output.
  • the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • special purpose logic circuitry e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
  • the processes and logic flows can be performed by and apparatus can also be implemented as a graphics processing unit (GPU).
  • GPU graphics processing unit
  • Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit.
  • a central processing unit will receive instructions and data from a read only memory or a random access memory or both.
  • the essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.
  • a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks.
  • a computer need not have such devices.
  • a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
  • PDA personal digital assistant
  • GPS Global Positioning System
  • USB universal serial bus
  • Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.
  • semiconductor memory devices e.g., EPROM, EEPROM, and flash memory devices
  • magnetic disks e.g., internal hard disks or removable disks
  • magneto optical disks e.g., CD ROM and DVD-ROM disks.
  • the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
  • a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
  • a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
  • keyboard and a pointing device e.g., a mouse or a trackball
  • Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
  • a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a
  • Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components.
  • the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.
  • LAN local area network
  • WAN wide area network
  • the computing system can include clients and servers.
  • a client and server are generally remote from each other and typically interact through a communication network.
  • the relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • User Interface Of Digital Computer (AREA)
  • Machine Translation (AREA)
  • Image Analysis (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Information Transfer Between Computers (AREA)
US16/689,020 2017-05-19 2019-11-19 Multi-task neural network systems with task-specific policies and a shared policy Active US11132609B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US16/689,020 US11132609B2 (en) 2017-05-19 2019-11-19 Multi-task neural network systems with task-specific policies and a shared policy
US17/486,842 US11983634B2 (en) 2017-05-19 2021-09-27 Multi-task neural network systems with task-specific policies and a shared policy

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201762508991P 2017-05-19 2017-05-19
PCT/EP2018/063275 WO2018211138A1 (en) 2017-05-19 2018-05-22 Multitask neural network systems
US16/689,020 US11132609B2 (en) 2017-05-19 2019-11-19 Multi-task neural network systems with task-specific policies and a shared policy

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2018/063275 Continuation WO2018211138A1 (en) 2017-05-19 2018-05-22 Multitask neural network systems

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/486,842 Continuation US11983634B2 (en) 2017-05-19 2021-09-27 Multi-task neural network systems with task-specific policies and a shared policy

Publications (2)

Publication Number Publication Date
US20200090048A1 US20200090048A1 (en) 2020-03-19
US11132609B2 true US11132609B2 (en) 2021-09-28

Family

ID=62217991

Family Applications (2)

Application Number Title Priority Date Filing Date
US16/689,020 Active US11132609B2 (en) 2017-05-19 2019-11-19 Multi-task neural network systems with task-specific policies and a shared policy
US17/486,842 Active US11983634B2 (en) 2017-05-19 2021-09-27 Multi-task neural network systems with task-specific policies and a shared policy

Family Applications After (1)

Application Number Title Priority Date Filing Date
US17/486,842 Active US11983634B2 (en) 2017-05-19 2021-09-27 Multi-task neural network systems with task-specific policies and a shared policy

Country Status (5)

Country Link
US (2) US11132609B2 (ja)
EP (1) EP3602412A1 (ja)
JP (1) JP6854921B2 (ja)
CN (1) CN110892418A (ja)
WO (1) WO2018211138A1 (ja)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220083869A1 (en) * 2017-05-19 2022-03-17 Deepmind Technologies Limited Multi-task neural network systems with task-specific policies and a shared policy

Families Citing this family (35)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11250328B2 (en) 2016-10-26 2022-02-15 Cognizant Technology Solutions U.S. Corporation Cooperative evolution of deep neural network structures
US11507844B2 (en) 2017-03-07 2022-11-22 Cognizant Technology Solutions U.S. Corporation Asynchronous evaluation strategy for evolution of deep neural networks
US11562287B2 (en) 2017-10-27 2023-01-24 Salesforce.Com, Inc. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning
US10573295B2 (en) * 2017-10-27 2020-02-25 Salesforce.Com, Inc. End-to-end speech recognition with policy learning
US11182677B2 (en) 2017-12-13 2021-11-23 Cognizant Technology Solutions U.S. Corporation Evolving recurrent networks using genetic programming
WO2019118290A1 (en) * 2017-12-13 2019-06-20 Sentient Technologies (Barbados) Limited Evolutionary architectures for evolution of deep neural networks
US11527308B2 (en) 2018-02-06 2022-12-13 Cognizant Technology Solutions U.S. Corporation Enhanced optimization with composite objectives and novelty-diversity selection
CA3103470A1 (en) 2018-06-12 2019-12-19 Intergraph Corporation Artificial intelligence applications for computer-aided dispatch systems
US11481639B2 (en) 2019-02-26 2022-10-25 Cognizant Technology Solutions U.S. Corporation Enhanced optimization with composite objectives and novelty pulsation
CA3129731A1 (en) 2019-03-13 2020-09-17 Elliot Meyerson System and method for implementing modular universal reparameterization for deep multi-task learning across diverse domains
WO2020198520A1 (en) 2019-03-27 2020-10-01 Cognizant Technology Solutions U.S. Corporation Process and system including an optimization engine with evolutionary surrogate-assisted prescriptions
US20200327379A1 (en) * 2019-04-09 2020-10-15 GE Precision Healthcare LLC Fastestimator healthcare ai framework
US11699062B2 (en) * 2019-09-06 2023-07-11 Honda Motor Co., Ltd. System and method for implementing reward based strategies for promoting exploration
US11403668B2 (en) 2019-10-30 2022-08-02 Target Brands, Inc. Multitask transfer learning for optimization of targeted promotional programs
WO2020143847A2 (en) * 2020-04-02 2020-07-16 Alipay (Hangzhou) Information Technology Co., Ltd. Determining action selection policies of an execution device
CN111309893A (zh) * 2020-05-15 2020-06-19 支付宝(杭州)信息技术有限公司 基于源问题生成相似问题的方法和装置
US11775841B2 (en) 2020-06-15 2023-10-03 Cognizant Technology Solutions U.S. Corporation Process and system including explainable prescriptions through surrogate-assisted evolution
CN111708355B (zh) * 2020-06-19 2023-04-18 中国人民解放军国防科技大学 基于强化学习的多无人机动作决策方法和装置
CN111783606B (zh) * 2020-06-24 2024-02-20 北京百度网讯科技有限公司 一种人脸识别网络的训练方法、装置、设备及存储介质
CN111680934B (zh) * 2020-06-30 2023-04-07 西安电子科技大学 基于群体熵和q学习的无人机任务分配方法
CN112001585B (zh) * 2020-07-14 2023-09-22 北京百度网讯科技有限公司 多智能体决策方法、装置、电子设备及存储介质
WO2022025568A1 (ko) * 2020-07-27 2022-02-03 옴니어스 주식회사 멀티 태스크 러닝을 이용하여 상품의 속성을 인식하는 방법, 시스템 및 비일시성의 컴퓨터 판독 가능 기록 매체
CN114528469A (zh) * 2020-11-23 2022-05-24 中兴通讯股份有限公司 推荐方法、装置、电子设备、存储介质
CN112527383B (zh) * 2020-12-15 2024-05-31 北京百度网讯科技有限公司 用于生成多任务模型的方法、装置、设备、介质和程序
CN112784958B (zh) * 2020-12-31 2023-05-23 中电海康集团有限公司 一种基于持续学习方法的家庭服务型机器人
CN112685318A (zh) * 2021-01-07 2021-04-20 广州三星通信技术研究有限公司 产生测试脚本的方法和系统
CN112766493B (zh) * 2021-01-19 2023-04-07 北京市商汤科技开发有限公司 多任务神经网络的训练方法、装置、电子设备及存储介质
CN113240796B (zh) * 2021-05-14 2023-12-12 Oppo广东移动通信有限公司 视觉任务处理方法及装置、计算机可读介质和电子设备
CN113556287B (zh) * 2021-06-15 2022-10-14 南京理工大学 一种基于多智能体强化学习的软件定义网络路由方法
CN113485203B (zh) * 2021-08-03 2022-05-31 上海德衡数据科技有限公司 一种智能控制网络资源共享的方法及系统
CN114968520B (zh) * 2022-05-19 2023-11-24 北京百度网讯科技有限公司 任务搜索方法及装置、服务器和存储介质
CN114936783B (zh) * 2022-06-02 2023-01-17 暨南大学 一种基于mmddpg算法的rgv小车调度方法及系统
WO2024056891A1 (en) * 2022-09-15 2024-03-21 Deepmind Technologies Limited Data-efficient reinforcement learning with adaptive return computation schemes
CN115545188B (zh) * 2022-10-24 2024-06-14 上海人工智能创新中心 基于不确定性估计的多任务离线数据共享方法及系统
CN115496208B (zh) * 2022-11-15 2023-04-18 清华大学 协同模式多样化导向的无监督多智能体强化学习方法

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180165602A1 (en) * 2016-12-14 2018-06-14 Microsoft Technology Licensing, Llc Scalability of reinforcement learning by separation of concerns

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4803212B2 (ja) * 2008-05-28 2011-10-26 ソニー株式会社 データ処理装置、データ処理方法、及びプログラム
JP5733166B2 (ja) * 2011-11-14 2015-06-10 富士通株式会社 パラメータ設定装置、コンピュータプログラム及びパラメータ設定方法
CN110892418A (zh) * 2017-05-19 2020-03-17 渊慧科技有限公司 多任务神经网络系统
US20230376961A1 (en) * 2022-05-19 2023-11-23 Oracle Financial Services Software Limited Reinforcement learning agent simulation to measure monitoring system strength

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180165602A1 (en) * 2016-12-14 2018-06-14 Microsoft Technology Licensing, Llc Scalability of reinforcement learning by separation of concerns

Non-Patent Citations (42)

* Cited by examiner, † Cited by third party
Title
Anonymous, "DisTral: Distilling and Transferring Common Behavior for Multitask Deep RL," 31st Conference on Neural Information Processing Systems, 2017, 11 pages.
Beattie et al., "DeepMind Lab," arXiv:161203801v2, Dec. 13, 2016, 11 pages.
Bellemare et al., "The Arcade Learninig Environment: An Evaluation Platform for General Agents," Journal of Artificial Intelligence Research 47 (2013) 253-279, Jun. 2013, 27 pages.
Bengio; Yoshua, "Deep Learning of Representations for Unsupervised and Transfer Learning," JMLR: Workshop and Conference Proceedings 27:17-37, 2012, 21 pages.
Boyd et al., "Distributed optimization and statistical learning via the alternating direction method of multipliers," Found. Trends Mach. Learn., 3(1), Jan. 2011, 13 pages.
Bucil{hacek over (a)} al., "Model compression," In Proc. of the Int'l Conference on Knowledge Discovery and Data Mining (KDD), 2006, 7 pages.
Caruana; Rich, "Multitask learning," Machine Learning, 28(1):41-75, Jul. 1997, 35 pages.
Dempster et al., "Maximum likelihood from incomplete data via the em algorithm," Journal of the royal statistical society, Series B (methodological), pp. 1-38, 1977, 39 pages.
Devin et al, "Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer," arXiv:1609.07088v1, Sep. 22, 2016, 8 pages (Year: 2016). *
Devine et al., "Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer," arXiv:1609.07088v1, Sep. 22, 2016, 8 pages.
Duan et al, "A multi-agent reinforcement learning approach to robot soccer", Jun. 2011, Springer Science+Business Media, all pages (Year: 2011). *
Fox et al., "Principled option learning in markov decision processes," In European Workshop on Reinforcement Learning (EWRL), 2016, 7 pages.
Fox, et al., "Taming the noise in reinforcement learning via soft updates," In Uncertainty in Artificial Intelligence (UAI), 2016, 11 pages.
Gelman et al., "Bayesian data analysis," vol. 2. Chapman & Hall/CRC Boca Raton, FL, USA, 2014, 4 pages.
Hinton et al., "Distilling the knowledge in a neural network," NIPS Deep Learning Workshop, 2014, 9 pages.
Jaderberg et al., "Reinforcement learning with unsupervised auxiliary tasks," Int'l Conference on Learning Representations (ICLR), 2016, 17 pages.
Kappen et al., "Optimal control as a graphical model inference problem," Machine learning, 87(2):159-182, 2012, 2 pages.
Lample et al., "Playing FPS games with deep reinforcement learning," Association for the Advancement of Artificial Intelligence (AAAI), 2017, 7 pages.
Levine et al, "Guided Policy Search", Proceedings of the 30th International Conference on Machine Learning, 2013, 9 pages, (Year: 2013). *
Lin et al., "Collaborative Deep Reinforcement Learning," arXiv:1702.05796v1, Woodstock '97, El Paso, Texas, Jul. 1997, 9 pages.
Mirowski et al., "Learning to navigate in complex environments," Int'l Conference on Learning Representations (ICLR), 2016, 16 pages.
Mnih et al., "Asynchronous methods for deep reinforcement learning," In Int'l Conference on Machine Learning (ICML), 2016, 19 pages.
Mnih et al., "Human-level control through deep reinforcement learning," Nature, 518(7540):529-533, Feb. 2015, 13 pages.
Nachum et al., Bridging the gap between value and policy based reinforcement learning. arXiv:1702.08892, 2017, 21 pages.
Parisotto et al., "Actor-mimic: Deep multitask and transfer reinforcement learning," In Int'l Conference on Learning Representations (ICLR), 2016, 16 pages.
Pascanu et al., "Revisiting natural gradient for deep networks," Int'l Conference on Learning Representations (ICLR), 2014, 18 pages.
PCT International Preliminary Report on Patentability in International Appln. No. PCT/EP2018/063275, dated Nov. 19, 2019, 13 pages.
PCT International Search Report and Written Opinion in International Appln. No. PCT/EP2018/063275, dated Aug. 28, 2018, 19 pages.
Rawlik et al., "On stochastic optimal control and reinforcement learning by approximate inference," In Robotics: Science and Systems (RSS), 2012, 5 pages.
Rusu et al., "Policy Distillation," arXiv:1511.06295v2, London, United Kingdom, Jan. 7, 2016, 13 pages.
Rusu et al., "Policy distillation," In Int'l Conference on Learning Representations (ICLR), 2016, 13 pages.
Schaul et al., "Prioritized experience replay," CoRR, abs/1511.05952, 2015, 21 pages.
Schulman et al., "Equivalence between policy gradients and soft Q-Learning," arXiv:1704.06440, 2017, 15 pages.
Schulman et al., "Trust region policy optimization," In Int'l Conference on Machine Learning (ICML), 2015, 9 pages.
Sutton et al., "Policy gradient methods for reinforcement learning with function approximation," In Adv. in Neural Information Processing Systems (NIPS), vol. 99, pp. 1057-1063, 1999, 7 pages.
Taylor et al., "An introduction to inter-task transfer for reinforcement learning," AI Magazine, 32(1):15-34, 2011, 15 pages.
Teh et al., "Distral: Robust Multitask Reinforment Learning," arXiv:1707.04175v1, Jul. 13, 2017, 13 pages.
Toussaint et al., "Probabilistic inference for solving (PO)MDPs," Technical Report EDI-INF-RR-0934, University of Edinburgh, School of Informatics, 2006, 24 pages.
Van Hasselt et al., "Deep reinforcement learning with double Q-learning," Association for the Advancement of Artificial Intelligence (AAAI), 2016, 7 pages.
Yin et al., "Knowledge Transfer for Deep Reinforcement," Proceedings of the Thirty-First AAA1 Conference on Artificial Intelligence, Nanyang Technological University, Singapore, 2017, 7 pages.
Yosinski, et al., "How transferable are features in deep neural networks?" In Adv. in Neural Information Processing Systems (NIPS), 2014, 9 pages.
Zhang,et al., "Deep learning with elastic averaging SGD," In Adv. in Neural Information Processing Systems (NIPS), 2015, 9 pages.

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220083869A1 (en) * 2017-05-19 2022-03-17 Deepmind Technologies Limited Multi-task neural network systems with task-specific policies and a shared policy
US11983634B2 (en) * 2017-05-19 2024-05-14 Deepmind Technologies Limited Multi-task neural network systems with task-specific policies and a shared policy

Also Published As

Publication number Publication date
US20220083869A1 (en) 2022-03-17
EP3602412A1 (en) 2020-02-05
JP2020521205A (ja) 2020-07-16
US11983634B2 (en) 2024-05-14
JP6854921B2 (ja) 2021-04-07
WO2018211138A8 (en) 2019-12-26
CN110892418A (zh) 2020-03-17
US20200090048A1 (en) 2020-03-19
WO2018211138A1 (en) 2018-11-22

Similar Documents

Publication Publication Date Title
US11132609B2 (en) Multi-task neural network systems with task-specific policies and a shared policy
US10733502B2 (en) Training policy neural networks using path consistency learning
EP3696737B1 (en) Training action selection neural networks
US11868894B2 (en) Distributed training using actor-critic reinforcement learning with off-policy correction factors
US11443162B2 (en) Resource constrained neural network architecture search
US10860920B2 (en) Distributional reinforcement learning
US11941527B2 (en) Population based training of neural networks
EP3459021B1 (en) Training neural networks using synthetic gradients
US11922281B2 (en) Training machine learning models using teacher annealing
US20180189950A1 (en) Generating structured output predictions using neural networks
CN111602148A (zh) 正则化神经网络架构搜索
US11544498B2 (en) Training neural networks using consistency measures
US20220036186A1 (en) Accelerated deep reinforcement learning of agent control policies
US20200104686A1 (en) Decreasing neural network inference times using softmax approximation
US12020164B2 (en) Neural networks for scalable continual learning in domains with sequentially learned tasks
US20210117786A1 (en) Neural networks for scalable continual learning in domains with sequentially learned tasks
US20230079338A1 (en) Gated linear contextual bandits

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: DEEPMIND TECHNOLOGIES LIMITED, UNITED KINGDOM

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PASCANU, RAZVAN;HADSELL, RAIA THAIS;BAPST, VICTOR CONSTANT;AND OTHERS;SIGNING DATES FROM 20180611 TO 20180925;REEL/FRAME:051662/0796

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: FINAL REJECTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE