JP2021154481A - Controller and method for controlling operation of robot executing tasks - Google Patents

Abstract

To provide a method for controlling operation of a robot executing tasks which method simplifies learning of the robot, thereby, enabling the robot to execute a new task more efficiently.SOLUTION: A controller comprises: an input interface configured to receive a current state of a robot for each control step; and a memory. The memory is configured to store a robot skill library, and the skills are task-agnostics. The memory is configured to store learned functions for a task. The controller comprises a processor. The processor is configured to execute the learned function on the current state so as to select a skill from the skill library for each control step in a sequence of control steps that reach a termination condition, as well as, to execute the selected skill for the current state so as to select an action from the selected skill in order to transit a state of the robot from the current state to a next state.SELECTED DRAWING: Figure 1

Description

The present invention relates to robot control, and more specifically to learning and controlling a robot to perform a particular task.

Robot operation is at the core of achieving robot promising. The definition of a robot itself requires that the robot have actuators that can be used to make a difference to the world. The potential of autonomous operations applications is enormous. That is, robots capable of manipulating their own environment can be deployed in hospitals, elderly care and childcare, factories, outer space, restaurants, service industries, and homes. These wide variety of deployment scenarios, as well as widespread and unsystematic environmental changes in highly specialized scenarios such as food preparation, suggest that there is a need for rapid learning of robots. There is. Many methods focus on the question of how robots should learn to manipulate the world around them.

For example, some methods train robots to learn individual operational skills from human demonstrations in order to perform new tasks. Some solutions use carefully designed demonstrations that are adapted to a particular robot. However, these methods are very labor intensive, time consuming, and / or resource intensive. Some other methods attempt to use hierarchical learning, curriculum learning, and optional frameworks. These methods learn more complex tasks from a previously learned set of less complex tasks. This complicates the goals of the new task rather than performing the new task.

Some other methods use reinforcement learning (RL) to train a robot to learn how to perform a new task in an effective manner. However, while many actions are virtually borderless when it comes to learning new tasks for robotic manipulation, RL expects to choose from a finite set of actions.

Therefore, there is a need to provide a way to simplify the learning of the robot, thereby allowing the robot to perform new tasks more efficiently.

An object of some embodiments is to provide a system and a method for learning and controlling a robot to perform a task. Additional or alternative, the purpose of some embodiments is to modify the learning process of the robot. In some embodiments, a robot learned to learn a new skill to perform a new task learns to reuse an existing task agnostic skill to "mimic" the new task. Based on the recognition that it can be modified to. Such a formulation allows embodiments to replace the learning objectives of new skills with selective objectives. Moreover, such a formulation allows the embodiment to distance itself from certain items of the new task so as to "mimic" only the effects or actions of the new task. Additional or alternative, such a formulation allows some embodiments to adaptively select skills in real time to account for possible obstacles and the nature of new tasks. .. Additional or alternative, such a formulation allows some embodiments to use reinforcement learning (RL) to learn how to perform tasks in an effective manner. It becomes possible to train a robot.

For example, an object of some embodiments is to train a robot to perform a task from multiple demonstrations of task execution where states and actions are observed but skills are not observed. In fact, some embodiments are based on the recognition that skills are unknown or undefined, while state / action pairs are often observable from demonstrations. In addition, state / action pairs are more important than skills. The reason is that the state / action pair actually performs the task in the robot's spatial domain, while the skill connects the state / action pair sequence in the time domain.

As such, some embodiments train the function to minimize the probability of difference between the selected action for a state and the corresponding action from multiple demonstrations of the task. The embodiment trains a function to select a skill to perform an action. Function learning can be thought of as robot learning. The selected skill can be task agnostic as long as the skill returns the required action for the state entered. In fact, some embodiments happen to return the correct action for one state value for a new task that defines a particular action for a particular state, while returning the wrong action for a different state value. Is based on the recognition that exists. However, for that different state value, there may be another skill that can return the correct action.

Therefore, some embodiments train the function to reuse existing skills and replace learning objectives with selective objectives. Such choices can be made deterministically or adaptively to account for possible obstacles and the nature of the task. The side effect of the embodiment is that even if the selected skill is a multi-step skill configured to be executed for multiple control steps, it is necessary to execute a trained function for each control step in order to execute the skill. Can include being. However, the purpose of choice makes it possible to simplify the reward function for reinforcement learning.

Specifically, reinforcement learning (RL) is an area of machine learning about how software agents should take action in the environment to maximize some concept of cumulative reward. The RL agent interacts with its environment in discrete time steps. At each time t, the agent is typically receive observations o _t containing reward r _t. Then, RL controller, to increase the compensation, select an action a _t the set of available actions, the action a _t is then sent to the environment. However, while many actions are virtually borderless when it comes to learning new tasks for robotic manipulation, RL expects to choose from a finite set of actions. As such, selection-based RLs in some embodiments replace borderless selection of actions with bordered selection of skills from a finite library of skills. In fact, such a selection-based RL simplifies the learning of task-specific functions using a library of task agnostic skills. For example, in one embodiment, a deep reinforcement learner (DRL) determines values for each skill and its actions.

Different embodiments use different methods to train the parameterized functions that form the RL controller. For example, in some embodiments, the parameterized function is a deep deterministic policy gradient method, an advantage-actor critic method, or a proximity-actor critic method (proximal policy optimization method). It is learned using one of the proximal policy optimization method, the deep Q-network method, or the Monte Carlo policy gradient method.

Accordingly, one embodiment discloses a controller for controlling the movement of a robot to perform a task. The controller includes an input interface that receives the current state of the robot for each control step and a memory, which is configured to (1) store a library of robot skills, and each skill holds the current state. A stochastic function of the robot's state that returns a distribution of actions in response to submission, the skill is task insane, and the memory is configured to (2) store the learned functions for the task. The learned function is a probability function of the robot's state that returns the distribution of skills in response to submitting the state of the robot. The controller also includes a processor, which performs a trained function on the current state for each control step in the sequence of control steps that reaches the termination condition, so that it selects the skill with the highest probability on the skill distribution. Performs the selected skill for the current state to select the action with the highest probability on the distribution of actions in order to perform and transition the robot's state from the current state to the next state. It is configured as follows. The controller also includes an output interface that is configured to instruct the robot to perform selected actions to perform the task.

Another embodiment discloses a method for controlling the movement of a robot to perform a task. The method uses a processor that is bound to memory, which stores (1) a library of robot skills, and each skill returns a distribution of actions in response to submitting its current state. It is a stochastic function of the robot's state, the skill is task insane, the memory stores (2) the learned function for the task, and the learned function responds to submitting the robot's state. , A stochastic function of the state of the robot that returns the distribution of skills, the processor is combined with the stored instructions and accepts the current state of the robot for each control step when executed by the processor to perform a method step. That, for each control step in the sequence of control steps that reach the end condition, execute a trained function for the current state to select the skill with the highest probability on the skill distribution, and the state of the robot. To perform the selected skill for the current state and to perform a task to select the action with the highest probability on the distribution of actions in order to transition from the current state to the next state. Includes outputting commands to the robot to perform the selected action.

Definitions In describing embodiments of the invention, the following definitions are applicable throughout (including the above).

"Robot" refers to a machine whose underlying purpose is to automatically perform one or more tasks.

A "task" refers to a task to be performed by a robot, which task is defined by a sequence of skills in the time domain. Alternatively, a task refers to a sequence of state / action pairs to be performed by a robot in the time domain.

"Skill" refers to the probability function of a robot's state that returns an action or distribution over the action in response to submitting the current state.

"State" refers to the value of the robot and its environment at a particular time t. These values define the robot's settings at a particular time t. Examples of robot settings include, for example, motor settings, joint settings, gripper settings, tool settings, robot position settings. Examples of values in that environment are, for example, the position of one or more objects, the position of one or more obstacles, the distance to a target position, and an image of the environment.

“Action” refers to a vector of values that shifts the robot from the current state to the next state at a particular time t. The vector of such values determines the next setting of the robot at a particular time t.

“Control step” refers to taking action on a state at a particular time t.

"Computer" refers to any device capable of accepting structured inputs, processing the structured inputs according to defined rules, and producing the results of the processing as output. Examples of computers are, for example, general purpose computers, supercomputers, mainframes, superminicomputers, minicomputers, workstations, microcomputers, servers, interactive televisions, computers and interactive televisions. Includes hybrid combinations and application-specific hardware that emulates computers and / or software. A computer may have a single processor, or may have multiple processors that can and / or cannot operate in parallel. Further, the computer refers to the two or more computers connected via a network for transmitting and receiving information between the two or more computers. Examples of such computers include decentralized computer systems for processing information through network-linked computers.

A "central processing unit (CPU)" or "processor" refers to a computer or computer component that reads and executes software instructions.

"Memory" or "computer-readable medium" refers to any storage accessible by a computer for storing data. Examples include magnetic hard disks, floppy (registered trademark) disks, optical disks such as CD-ROMs or DVDs, magnetic tapes, memory chips, and to be used for sending and receiving e-mails or accessing networks. Includes a carrier used to carry computer-readable electronic data and computer memory such as, for example, random access memory (RAM).

"Software" refers to the prescribed rules for operating a computer. Examples of software include code segments, instructions, computer programs, and programmed logic. The software of the intelligent system may be self-learning.

A "module" or "unit" refers to a basic component of a computer that performs a task or part of a task. A "module" or "unit" can be implemented by either software or hardware.

"Controller" and / or "control system" refers to a device or set of devices for managing, instructing, directing or coordinating the behavior of another device or system. The controller can be realized by hardware, a processor whose operation is composed of software, and a combination thereof. The controller can be an embedded system.

The embodiments disclosed herein are further described with reference to the accompanying drawings. The drawings shown are not necessarily scaled and instead are generally emphasized in explaining the principles of the embodiments disclosed herein.

FIG. 6 illustrates a schematic overview of the principles used by some embodiments to learn and control the movements of a robot to perform tasks. It is a figure which shows the block diagram of the robot control system for learning and controlling the operation of a robot to execute a task. It is a figure which shows the exemplary skill stored in the skill library module of a robot control system. It is a figure which shows the task demonstration for the robot which uses a remote control device according to some embodiments. It is a figure which shows the task demonstration for the robot by manual operation according to some embodiments. It is a figure which shows the exemplary task demonstration stored in the task demonstration module according to some embodiments. It is a figure which shows the example architecture diagram of the learning module which trains a function for selecting a skill according to some embodiments. FIG. 5 illustrates an exemplary architecture for training a function using selection-based reinforcement learning, according to some embodiments. It is a figure which shows the example architecture diagram of the execution module which executes a task according to some embodiments. It is a figure which shows the flowchart which shows the operation performed by the execution module according to some embodiments. It is a figure which shows the exemplary learning phase and execution phase for performing a task according to some embodiments.

In the following description, for explanatory purposes, a number of specific details are provided to provide a complete understanding of the present disclosure. However, it will be apparent to those skilled in the art that this disclosure may be carried out without these specific details. In other cases, devices and methods are shown only in the form of block diagrams to avoid obscuring this disclosure.

As used herein and in the claims, terms such as "for example," "as an example," and "like," and verbs such as "provide," "have," and "include." And their other verb forms, when used in connection with a list of one or more components or other items, should each be construed as open-ended, which list is another additive. Means that no component or item should be considered to be excluded. The term "based on" means at least partially based. In addition, it should be understood that the expressions and terms used herein are for explanatory purposes and should not be considered limiting. None of the headings used in this description are for convenience only and have no legal or limiting effect.

FIG. 1 shows a schematic overview of the principles used by some embodiments to learn and control the movements of the robot to perform tasks, according to some embodiments. A robot is a machine whose fundamental purpose is to automatically perform one or more tasks. Tasks are tasks that are difficult or repetitive for humans to perform. For example, tasks correspond to industrial work, maintenance of household chores, and serving in a restaurant. In some embodiments, the immature robot 102 refers to an unlearned robot or a learning robot. In order for a robot (eg, an unskilled robot 102) to perform a task, the robot needs to be widely learned about the task. However, the task is complex and involves several steps to be performed. Therefore, in order to assist the learning of the robot, the task of the robot can be divided into subtasks (hereinafter referred to as skills). The task can be to assemble some parts into some complex object. The skills required for assembly are moving the robot's arm to the left, opening the gripper, lowering the gripper, lifting parts, raising the arm, moving the arm to the right to a specific position. It can be caused, the arm lowered, a part inserted into another part, and so on.

Some embodiments are based on the recognition that a robot that has learned about a task can be manipulated to perform a new task. For example, a robot that has been trained to maintain household chores, such as washing dishes and cleaning the floor, operates to wash dishes, clean the floor, and dispose of debris. Can be done. However, in order to operate the robot, the robot also requires rapid and extensive learning for new tasks. Therefore, the purpose of some embodiments is to train the robot to learn new tasks. As shown in FIG. 1, the untrained robot 102 is trained by the robot control system 104, which forms a trained robot that executes a new task. In some embodiments, the robot control system 104 includes a learning module 104a that trains an unskilled robot 102 to learn individual operational skills from a human demonstration. For example, the learning module 104a uses a carefully designed demonstration adapted to the specifics of the robot. However, this approach requires additional work to design the demonstration, which is a long and time consuming process. In some other embodiments, the learning module 104a uses reinforcement learning (RL) to train the robot to learn new tasks in an effective manner. In reinforcement learning (RL), RL agents interact with their environment in discrete time steps. At each time t, the agent typically receive observations o _t containing reward r _t. Then, RL controller, to increase the compensation, select an action a _t the set of available actions, then the action a _t is sent to the environment. However, while many actions are virtually borderless when it comes to learning new tasks for robotic manipulation, RL expects to choose from a finite set of actions. In addition, the RL requires the need to design appropriate reward functions for learning new tasks. It is difficult to design these reward functions.

Therefore, the purpose of some embodiments is to provide an improved learning method for performing new tasks. Therefore, the robot control system 104 including the learning module 104a for learning purposes is modified by the learning module 108a for selection purposes. The learning module 108a trains the unlearned robot 102 so as to reuse the existing task agnostic skill to convert the unlearned robot into the learned robot 106. In addition, the learning module 108a allows the embodiment to be distanced from certain items of the new task so as to "mimic" only the effects or actions of the new task. Additional or alternative, the learning module 108a allows some embodiments to adaptively select skills in real time to account for possible obstacles and the nature of new tasks. Moreover, in some embodiments, the selection objective makes it possible to simplify the reward function for reinforcement learning. In some embodiments, the learning module 108a uses the RL to train the immature robot 102 to perform new tasks in an efficient manner. Specifically, the learning module 108a uses the RL to train the immature robot 102 to select a skill from a finite library of existing task agnostic skills. As such, selection-based RLs in some embodiments replace borderless selection of actions with bordered selection of skills from a finite library of skills.

FIG. 2 shows a block diagram of a robot control system 200 for learning and controlling robot movements to perform tasks, according to some embodiments. Hereinafter, "robot control system" and "controller" can be used interchangeably and mean the same thing. The controller 200 includes an input interface and an output interface for connecting the controller 200 to other systems and devices. In some embodiments, the input interface includes a human machine interface (HMI) 202 and an imaging interface 220. The HMI 202 may connect the controller 200 to an input device such as a keyboard 230 and / or a pointing device / medium 232. The pointing device 232 may include, for example, a mouse, trackball, touchpad, joystick, pointing stick, stylus, or touch screen. The imaging interface 220 may be adapted to connect the controller 200 to the imaging device 226. The imaging device 226 may include an RGBD camera, a depth camera, an RGB camera, a grayscale camera, a computer, a scanner, a mobile device, a webcam, or any combination thereof. According to some embodiments, the input interface may be configured to receive / accept the current state of the robot 236 for each control step of the robot 236. The input interface further includes a network interface controller (NIC) 212 adapted to connect the controller 200 to the network 234 via the bus 210. The controller 200 receives the current state of the robot 236 for each control step via the network 234, either wirelessly or by wire.

According to some embodiments, the input interface may be further configured to receive a set of task executions. A set of executions is a demonstration of a task for a robot to perform a task. Hereinafter, "task execution set" and "task demonstration" can be used interchangeably and mean the same thing. Some embodiments are based on the recognition that the imaging interface 220 may receive a task demonstration for the robot 236 from the imaging device 226. In some embodiments, the imaging device 226 may provide task demonstrations in image and / or video formats. Additional or alternative, the controller 200 may include an image processing module 218. The image processing module 218 is configured to process images and videos to provide a task demonstration as input to the learning module 206d or to store the task demonstration in the task demonstration module 206b. In some embodiments, the image processing 218 may enhance the task demonstration input to the learning module 206d or enhance the task demonstration stored in the module 206b. Additional or alternative, the controller 200 includes a task processing module 216 that may be configured to receive task demonstrations directly from robot 236 or via network 234. In some embodiments, the task processing module 216 processes the task demonstration to provide the task demonstration as input to the learning module 206d or to store the task demonstration in the task demonstration module 206b. It can be further configured. The task processing module 216 acquires a task demonstration in some format and converts the task demonstration into a sequence of state / action pairs. For example, tasks may be received as robot commands, and task processing module 216 may translate those commands into state / action pairs. The task processing module 216 may be used in enhancing the task demonstration from the elements provided by the image processing 218.

The controller 200 further includes a processor 204 and a memory 208 that stores instructions that can be executed by the processor 204. Processor 204 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. Memory 208 may include random access memory (RAM), read only memory (ROM), flash memory, or any other suitable memory system. Processor 204 is connected to one or more input and output devices via bus 210. The stored instructions provide a way to learn and control the behavior of a robot (eg, robot 236) to perform a task.

In some embodiments, memory 208 may be further expanded to include storage 206. In some embodiments, the storage 206 includes a learned skill library module 206a, a task demonstration module 206b, a robot controller 206c, a learning module 206d and an execution module 206e. The skill library module 206a may be configured to store a library of skills (ie, a plurality of skills). A skill is a probability function of the state of a robot (eg, robot 236) that returns an action or distribution over the action in response to submitting the current state of the robot. Some embodiments are based on the recognition that a skill in module 206a is a subsequence or subtask of a task. In addition, each skill can be a sequence of state / action pairs or state / action pairs in the time domain. Therefore, the skill in module 206a can be a single-step skill or a multi-step skill. For example, a skill corresponding to a state / action pair can be coined as a single-step skill, and a skill corresponding to a state / action pair sequence can be created as a multi-step skill. Further, the skills stored in the module 206a will be described in detail with reference to FIG.

Some embodiments are based on the recognition that the task demonstration module 206b can be configured to store a set of task executions (ie, task demonstrations for robot 236). In some embodiments, the task demonstration is observed or recorded by the input device as a sequence of state / action pairs, while the skill is unknown or undefined. Therefore, the task demonstration is stored in module 206b as a sequence of state / action pairs. In addition, state / action pairs are more important than skills. The reason is that the state / action pair actually performs the task in the robot's spatial domain, while the skill connects a sequence of state / action pairs in the time domain. Task demonstrations for robots will be described in detail with reference to FIGS. 4A-4C.

Robot controller 206c may be configured to store instructions to execute one or more modules in storage 206 when executed by processor 204. Some embodiments are based on the recognition that the robot controller 206c manages the learning and execution phases of the robot.

The learning module 206d may be configured to store instructions that, when executed by the processor 204, cause the processor 204 to learn a function for selecting a skill. Some embodiments are received for a state from an instruction to receive a set of task executions, each defining a sequence of state / action pairs, and for each state from a sequence of state / action pairs. An instruction to determine the distribution of actions that fits the action, and an instruction to determine the skill with the highest probability of returning the action sampled on the distribution of actions determined for that state for a given state. Is based on the recognition that the learning module 206d stores. An object of some embodiments is to train a function to select a skill that penalizes switching between skills. Therefore, some embodiments use reinforcement learning (RL) to train the function. For example, the RL includes a reward function that includes a penalty for switching between skills and a penalty based on the difference between the action selected by the RL and the action of the demonstrated task. Further, a learning module 206d (ie, a learning phase) for training a robot to perform a task will be described in detail with reference to FIGS. 5-6.

Execution module 206e may be configured to store learned functions for selecting skills. A trained function is a single argument function. In some embodiments, the learned function is a probability function of the robot's state that returns a distribution of skills in response to submitting the robot's state. Further, the execution module 206e may be configured to store an instruction to select an action from the selected skill. Further, the execution module 206e is configured to store an instruction to execute skill selection for each control step, and the skill corresponds to a multi-step skill. Further, the execution module 206e (ie, the execution phase) for executing the task will be described in detail with reference to FIGS. 7-9.

The output interface can be configured to instruct the robot to perform selected actions in order to perform a task. The output interface may include a display interface 214 and an application interface 222. The display interface 214 may connect the controller 200 to the display device 224. The display device 224 may include, for example, a computer monitor, camera, television, projector, or mobile device. The application interface 222 may connect the controller 200 to the application device 228. In some embodiments, the application device may be embodied within the controller 200. Therefore, the application device 228 operates based on the selected action to perform the task. For example, application device 228 provides a control signal for a robot to perform a selected action to perform a task.

FIG. 3 shows exemplary skills stored in skill library module 206a, according to some embodiments. Some embodiments are based on the recognition that the skills stored in the skill library module 206a are basic skills such as move options, rotation options, stop options, pick options, and placement options. Therefore, the skill library module 206a has a left movement 302, a right movement 304, an XYZ movement 306, an up movement 308, a down movement 310, a stop 312, a target arrangement 314, and a target pick. Includes skills such as 316 and tool rotation 318. In some embodiments, the skill stored in the skill library module 206a may consist of two or more basic skills that lead to complex skills. For example, complex skills may include basic skills such as movement 306 in the XYZ direction and placement of objects 316. Some embodiments are based on the recognition that there are multiple libraries of skills within module 206a to perform one task or multiple tasks. For example, if the task requests skills from multiple libraries, module 206a stores multiple libraries of skills. In some other embodiments, it may be necessary to learn new skills in order to perform some task demonstrations. Therefore, the new learned skill can be stored in a library of existing skills, or a new library of skills can be created to store the new learned skill. The decision to learn a new skill depends entirely on the decision of a human expert. An object of some embodiments is to maintain all required skills in skill library module 206a in order to perform a task.

Further, each skill is stored in the module 206a as a probability function. Thus, the skill in module 206a is a probability function of the robot's state that returns an action or distribution over the action in response to submitting the current state. Therefore, it is mathematically defined as follows.

Therefore, the skill in module 206a is

It can be expressed mathematically as such.

FIG. 4A shows a task demonstration for a robot using a remote control device, according to some embodiments. As shown in FIG. 4A, the remote control device 402 can be used to demonstrate a task for robot 236. Some embodiments are based on the recognition that an operator 404 (eg, a technician and a user) operates a remote control device 402, and then the remote control device 402 operates a robot 236 to demonstrate a particular task. ing. In some embodiments, the remote control device 402 sends robot configuration settings (ie, robot settings) to the robot 236 to demonstrate a particular task. For example, the remote control device 402 sends control commands such as movement in the XYZ directions, speed control commands, joint position commands, and the like to demonstrate a particular task. For example, as shown in FIG. 4A, a particular task can be defined by two skills: a skill that picks a target and a skill that moves the target upwards. The remote control device 402 sends robot configuration settings for both skills to demonstrate a particular task. In some other embodiments, the demonstration of a particular task is done in a control step. For example, the remote control device 402 transmits robot configuration settings for each control step. As such, the particular task demonstrated for Robot 236 can be recorded in state / action pairs by the input device. The input device may provide the recorded data to the controller 200.

FIG. 4B shows a task demonstration for a manually operated robot according to some embodiments. As shown in FIG. 4B, the robot 236 can be manually manipulated to demonstrate a particular task. Some embodiments are based on the recognition that the operator 404 manually controls the robot 236 settings, such as the position of the motor, joints, grippers, tools, robot 236, to demonstrate a particular task. In some embodiments, the manually demonstrated task does not have to define skills and skill configurations for the demonstrated task. As such, task demonstrations are recorded by the input device in state / action pairs in the time domain. The input device may provide the recorded data to the controller 200.

In some other embodiments, the task demonstration can be done by simulation. In some other embodiments, the task demonstration can be the execution of wrist points in 3D space reached during the task demonstration for the movement of the end of the robot, such as a tool. In some other embodiments, task demonstrations can be achieved by a combination of manual operation and remote control device operation. In some other embodiments, the imaging device 226 may record the task demonstration and the image processing 218 may process the recorded task demonstration so as to enhance the task demonstration by visual elements. Moreover, task demonstrations for robots cannot be limited to the exemplary description described above. Instead, all kinds of demonstrations within the scope of the invention should be considered.

FIG. 4C shows an exemplary task demonstration stored in the task demonstration module 206b, according to some embodiments. An object of some embodiments is to store a set of task executions (ie, task demonstrations for a robot) received from an input device. Some embodiments are based on the recognition that the skill is unknown or undefined, while the task demonstration is received in a state / action pair. In addition, state / action pairs are more important than skills. The reason is that state / action pairs actually perform tasks in the robot's spatial domain, while skills connect a sequence of state / action pairs in the time domain. Therefore, the received task demonstration can be stored in the task demonstration module 206b as a sequence of state / action pairs. In some embodiments, the sequence of state / action pairs is defined in the time domain. For example, a task demonstration is represented as a sequence of state / action pairs from start time to end time.

State S _t at a particular time t contains a value that if there is to define the current configuration of the robot is enhanced by elements from the environment. In some embodiments, robot settings include, but are not limited to, settings for motors, joints, grippers, tools, positions, images, or any combination thereof. The value represents, for example, a joint value for a particular robot configuration. Examples of values in that environment are the position of one or more objects, the position of one or more obstacles, the distance to a target position, and an image of the environment. Action A _t at a particular time t includes a vector of values to determine the different settings of the robot. Some embodiments are based on the recognition that the action causes the robot to move from one state to the next. This action can be discrete or continuous in nature. The action can be a single value or a vector of values, and so on.

In FIG. 4C, block 406 may represent a task demonstration stored in task demonstration module 206b. Further, the task demonstration module 206b stores not only the task demonstration 406 but also some task demonstrations. For example, the task demonstration module 206b stores a finite number of task demonstrations. Block 460 may represent the last or final task demonstration stored for the robot. Task Demonstration 406 begins at time _{t 0,} and stores the status and actions 406-0 at time _{t 0} (e.g., _{S t0, A t0} is a pair of state / action). In some embodiments, state _{(i.e., S t0)} action to be performed on _{(i.e., A t0)} is defined as a control step for time _{t 0.} The task demonstration module 206b keeps storing the task demonstration 406 until the end time of the task demonstration 406. In the end time _{t N,} the task demonstration module 206b, the state and action 406-N (e.g., _S tN is a pair of state / _{action, A tN)} stores. Therefore, task demonstration 406 contains N + 1 state / action pairs for N + 1 time instances. At time _{t N,} the final action _{A tN} is demonstrated for the task demonstration 406, the robot is in a state _{S tN + 1.} Therefore, as task demonstration 406, states 406-N + 1 are further stored. According to some embodiments, the stored task demonstration 406 is in orbit.

Can be expressed as.

In addition, the task demonstration module 206b stores all finite number of task demonstrations, as described with reference to task demonstration 406. According to some embodiments, the task demonstration in the task demonstration module 206b can have the same or different sequence of state / action pairs. For example, state / action pairs 460-0 to 460-O may be the same as or different from state / action pairs 406-0'to 406-N. In addition, the first state and action pair for demonstration may be the same or different. Further, the task demonstrations in the task demonstration module 206b may have the same length or may have different lengths. For example, task demonstration 460 may include O + 1 state / action pairs for O + 1 time instances, and task demonstration 406 may include N + 1 state / action pairs for N + 1 time instances. Therefore, the task demonstration module 206b stores M finite number of task demonstrations, and M ≧ 1. According to some embodiments, the M finite demonstration tasks are a set of orbits D, i.e.

Can be expressed as.

FIG. 5 shows an exemplary architectural diagram of a learning module 206d for training a function for selecting a skill, according to some embodiments. The learning module 206d may be configured to train a function for selecting a skill from the skill library module 206a based on a task demonstration in the module 206b. Learning a function can be thought of as training a robot to select a skill. In some embodiments, the skill in the skill library module 206a is task agnostic to perform the task. In some embodiments, the task demonstration in the task demonstration module 206b, which defines a particular action for a particular state, has instructions as to which skill should be selected from the skill library module 206a to perform the task. It is based on the recognition that it does not exist. Therefore, the purpose of some embodiments is to train a function for selecting a skill to perform a task.

In some embodiments, the learning module 206d is a set of task executions from at least one of a task processing module 216, an image processing module 218, or a task demonstration module 206b (ie, a task demonstration for a robot). Is based on the recognition that it is configured to receive. The task demonstration received is a sequence of state / action pairs, as illustrated in FIG. 4C. The demonstration received is a demonstration of the task that needs to be performed by the robot at runtime. In some embodiments, all task demonstrations (ie, all of the M finite task demonstrations for a particular task) may be received to train the function. In some embodiments, the received M finite task demonstration is a set of orbits D, i.e.

The formula is expressed as

Is equivalent to.

Some embodiments are based on the recognition that the learning module 206d is further configured for each state in the set of trajectories D to determine the distribution of actions that fit the actions received for that state. For explanatory purposes, consider _{the state St0.} At time _{t 0,} the state _{S t0} receives the _{A t0} action of the M finite from M finite task demonstration in the set D. In some embodiments, the M finite _At0 actions from the M finite task demonstrations can be the same or different. Therefore, the learning module 206d is the state _{S t0,} determines the distribution over the M finite _{A t0} action. Further, the learning module 206d, as described in connection with state S _t0, repeating the determining a distribution of actions for each state in the set of trajectories D. Therefore, the learning module 206d determines the distribution of actions for all states in set D. Further, for each state in the set of trajectories D, a learning module 206d for determining the distribution of actions that fits the actions received for that state is described in detail with reference to FIG.

In some embodiments, the learning module 206d is configured to sample actions on the distribution of actions determined for the corresponding states for each state in the set of orbits D. Some embodiments are based on the recognition that the sampled actions correspond to the actions with the highest probability in the distribution of actions. In fact, an object of some embodiments is to determine the skill for module 206a for a state in the set of orbit D, based on the actions sampled for that state. Therefore, for a state, it is necessary to train a function to determine the skill from module 206a that corresponds to the highest probability of returning an action sampled on the distribution of actions determined for that state.

In some embodiments, the learning module 206d is configured to train a function for determining skills from module 206a. In addition, function learning is described in detail below.

For clarity of explanation, consider a single demonstration of the task, namely M = 1. This problem is formulated as follows.

Available information:
The sequence of actions is represented as _{A = At0: tN.}

The sequence of states is represented as _{S = St0: tN + 1.}

The discrete set of skills in module 206a is represented as H = {H ₁ , H ₂ , ..., H _j }.

Information to be derived:
Determining the sequence of skills selected to maximize the likelihood is _{expressed as L θ} (H, S, A). L _θ (H, S, A) indicates that the discrete skills selected from the set of skills H in the sequence make the observations S, A most likely, i.e., have the highest probability. .. In some embodiments, the discrete skills selected in the sequence are a subset of the set of skills H or equal to the set of skills H. Therefore, maximizing the sequence of skill H can be achieved from maximum likelihood optimization. Furthermore, using Bayesian probability theory and excluding terms that do not play a role in optimization, maximizing _Lθ (H, S, A) can be done.

Is equivalent to.

Since _{H is unknown in L θ} (H, S, A),

Ignore as follows.

^{Π h} (A | S) in equation (1) is a previously learned skill from skill library module 206a. _{F θ} (S) in equation (1) is a function that returns the skill distribution given a state. For example, the function f _θ (S) is a probability distribution function that returns a distribution over skills given the input state S. According to some embodiments, f _θ (S) provides the ability to formulate a termination function. The termination function indicates whether the current skill ends (represented by a value 1) or the current skill continues (represented by a value 0) given the state S. .. For example, the termination function is defined as the function β (S) → 0/1.

Some embodiments, for state _{S t} [pi ^h | distribution action or actions given by (A S) is, skills [pi ^h for state _{S t} | to determine (A S), the state _{S t} distribution of actions to be determined (i.e., the distribution of actions to be determined about the state S _t from M finite demonstration) it is based on the recognition that the should match as close as possible. Therefore, the learning module 206d generates a constraint for equation (1), for example, the constraint is

Can be. Action A _t in the constraint may be sampled actions are determined for a state S _t. In some embodiments, the action A _t may be one of the actions on the distribution of actions to be determined about the state S _t.

The skills for state S _t | and actions returned by ^{(π h (A S))} , such as the minimum difference value (e.g. probability value or distance between the distribution of actions to be determined about the state S _t Can be metric). Therefore, equation (1) is converted as follows.

Equation (2) is a constraint

Is a single argument function with. In equation (2), the fact that task demonstrations are stored in a finite number of time instances is omitted for clarity of explanation. To take into account the time instances, equation (2) can be extended by the sum of additions for each time instance t. Further, equation (2) can be extended to the sum of all state and action pairs in demonstration d as follows. d exists in the set of demonstration D.

Equation (2) can be learned in a variety of different ways. In one embodiment, equation (2) can be learned by well-known techniques of expectation-maximization (EM) or some variant of EM. In another embodiment, equation (2) can be learned by a reinforcement learning (RL) technique that has a reward function that minimizes the deviation of the outcome of the task execution. Further, the functions learned based on the RL technique will be described in detail with reference to FIG. Therefore, the function, the state S _t, so as to determine the skills from module 206a corresponding to the highest probability of returning an action to be sampled on the distribution of actions to be determined for the state S _t, is learned ( Trained function 502). For example, in equation (2)

The skills [pi ^h has the highest probability that returns the action to be sampled on the distribution of actions to be determined about the state S _t | a ^(A S), determines the state S _t.

In addition, constraints

Can be given as a soft constraint as follows.

In equation (3), p represents the norm, for example 0 norm or 2 norm, typically n = 1, n = 2. Here, p and n are so-called hyperparameters, that is, parameters selected by the person who designs the final optimization function. In some embodiments, equation (3) can be the output of the learning module 206d as the trained function 502. In some other embodiments, the output of the learning module 206d can be a sequence of skills determined by the trained function 502 for all states in set D. For example, set H sets the determined skills in a sequence, i.e. H = {H ₄ , H ₇ , ... .. .. , H ₁ }, can be generated for storage. In addition, the skills determined in the sequence can be stored in list L. For example, list L can be generated to store an action or distribution of actions in a sequence, i.e. L = {π ₄ (s), π ₇ (s), ..., π _{1 (s)}.} .. Further, in some embodiments, the learning module 206d returns a termination function β (S). Therefore, whenever the termination function β (S) returns 1, the probability function π ^hl associated with the next skill from list L is selected and then executed. In some embodiments, the termination function β (S) determines the time to switch the currently selected skill in list L to the next skill to perform the task.

FIG. 6 shows an exemplary architecture for training a function using selection-based reinforcement learning, according to some embodiments. Reinforcement learning (RL) is an area of machine learning about how software agents should take action in their environment to maximize some concept of cumulative reward. The RL agent interacts with its environment in discrete time steps. At each time t, the agent is typically receive observations o _t containing reward r _t. Then, RL controller, to increase the compensation, select an action a _t the set of available actions, the action a _t is then sent to the environment. However, while many actions are virtually borderless when it comes to learning new tasks for robotic manipulation, RL expects to choose from a finite set of actions. As such, a selection-based RL architecture that replaces action-boundary selection with skill-bounded selection from a finite library of skills is shown in FIG.

In FIG. 6, block 602 may represent the task demonstration module 206b. In block 604, the state observer, in module 206 b, and records the state _{S t} at time t. In block 606, the action observer, a plurality of sets of execution (i.e., M-number of finite task demonstration) to record the actions received the state S _t from. At block 608, to minimize the results deviation of the execution of the task, the reward function of the state S _t are defined. Some embodiments are based on the recognition that the reward function includes a penalty for switching skills. Therefore, the reward function includes a termination function β (S) for switching skills. For example, the termination function specifies in module 206b the time to switch the current skill to another skill. In other embodiments, the reward function includes a penalty based on the difference between the recorded action and the generated action 614 in module 206b.

At block 610, RL controller samples the action from the action received about the state _{S t.} Sampled action corresponding to the action of the highest probability of the state S _t. Further, RL controller, the state S _t, select the skill corresponding to the highest probability from the module 206a, and returns the actions received for sampled action or state S _t. Therefore, in block 612, the RL controller selects a skill from module 206a, which contains a bounded / finite number of skills. Skills, return a certain / finite distribution of the boundary of the action or actions for a given state S _t. At block 614, the action with the highest probability on the finite distribution of actions is selected. As such, selection-based RLs in some embodiments replace borderless selection of actions with bordered selection of skills from a finite library of skills. In fact, such a selection-based RL simplifies the learning of task-specific functions using a library of task agnostic skills. For example, in one embodiment, a deep reinforcement learner (DRL) determines values for each skill and its actions.

Different embodiments may use different methods to train the parameterized functions that form the RL controller. For example, in some embodiments, the parameterized function is a deep deterministic policy gradient method, an advantage-actor critic method, or a proximity-actor critic method (proximal policy optimization method). It is learned using one of the proximal policy optimization method, the deep Q-network method, or the Monte Carlo policy gradient method.

FIG. 7 shows an exemplary architectural diagram of an execution module 206e that executes a task according to some embodiments. As shown in FIG. 7, in block 702, the execution module 206e is configured to accept the current state of the robot 236 for each control step via the input interface. In block 704, the execution module 206e is configured to store the learned functions for executing the task. As shown in FIG. 5, the learned function is the distribution of skills in response to submitting the state of the robot. In block 706, the execution module 206e executes the trained function of the current state for each control step in the sequence of control steps that reach the end condition to select the skill with the highest probability in the skill distribution. It is composed of. As shown in FIG. 3, a skill is a probability function of a robot's state that returns an action or distribution over the action in response to submitting the current state. In block 708, the execution module 206e executes the skill selected for the current state, selects the action with the highest probability in the distribution of actions, and transitions the robot state from the current state to the next state. It is configured as follows. In block 710, the execution module 206e is configured to output the selected robot action via the output interface in order to instruct the robot to perform a task.

FIG. 8 is a flowchart showing the operation executed by the execution module 206e according to some embodiments. In block 802, the execution module 206e is configured to acquire the state of the robot for the control step. In some embodiments, the robot state is acquired via an input interface. In block 804, the execution module 206e is configured to execute a learned function for the acquired state to select the skill with the highest probability on the skill distribution. Alternatively, the execution module 206e executes a learned function on the acquired state to select the skill corresponding to the highest probability in list L. The selected skill can be a single-step skill or a multi-step skill. In some embodiments, a multi-step skill is a skill that can be configured to perform on multiple states of multiple control steps and return their corresponding multiple actions. In block 806, the execution module 206e is configured to execute the selected skill for the acquired state to select the action with the highest probability on the distribution of actions. In block 808, the execution module 206e is configured to output the selected action in order to instruct the robot to perform the selected action.

As should be understood, when the robot performs the selected action, the robot moves from the acquired state to the next state. Further, the execution module 206e acquires the next state for the next control step in block 802. In some embodiments, the execution module 206e executes the termination function β (S) for the next state in block 804, and then executes the learned function for the next state. The termination function β (S) returns 0 indicating that the skill is continued or 1 indicating that the skill is selected for the next state (that is, the termination condition). In some embodiments, the termination function β (S) has the highest probability that the previously selected skill (ie, the skill selected for the acquired state) is a multi-step skill and for the next state. Corresponds to returning a sampled action with, returns 0, otherwise the termination function β (S) returns 1. When the end function returns 0, the execution module 206e skips the execution of the learned function for the next state in block 804. When the end function returns 1, the execution module 206a executes the learned function for the next state in block 804. In some other embodiments, the execution module 206e executes the trained function for the next state in block 804 without executing the termination condition β (S). Therefore, the execution module 206e returns a sampled action in which the previously selected skill (ie, the skill selected for the acquired state) is a multi-step skill and has the highest probability for the next state. Even if it corresponds to the above, the trained function is repeatedly executed for the next state. Execution module 206e executes the selected skill in block 806 to select the action having the highest probability in the distribution of actions. Execution module 206e outputs the selected action in block 808. Further, the flowchart 800 is repeated until all the control steps of the task are executed.

FIG. 9 shows exemplary learning and execution phases for performing tasks according to some embodiments. Part 902 is the preparatory phase for performing the task. As shown in FIG. 9, preparation phase 902 includes skill library module 206a and task demonstration module 206b. Module 206a stores a bounded / finite number of skills in the form of one library or multiple libraries. For example, module 206a stores skill 906a (ie, skill H1), skill 908a (ie, skill H2), and skill 910a (ie, skill H3). Each skill in the module is a probability function of the robot's state that returns an action or distribution of actions in response to submitting the current state. For example, skill 906a returns action A _{0 for state S 0} , skill 908a returns action A _{1 for state S 1} , and skill 910a returns the distribution of actions for states S ₂ and S _3. However, skill 910a returns a distribution of actions in which _{A 2} has the highest probability for state S _{2 and A 3} has the highest probability for state S _3. Some embodiments are based on the recognition that the skills stored in module 206a are task agnostic.

The task demonstration module 206b is a state / action paired demonstration of a task (ie {S _0- A ₀ , S _1- A ₁ , S _2- A ₂ , S _3- A ₃ , S ₄ }). The skill is unknown or undefined, while storing the execution). For example, the task demonstration module 206b is a task demonstration 906b (ie {S _0- A ₀ , S _1- A ₁ , S _2- A ₂ , S _3- A ₃ , S ₄ }), a task demonstration 908b (ie, {. S _0- A ₀ , S _1- A ₁ , S _2- A ₂ , S _3- A ₃ , S ₄ }), and task demonstration 910b (ie {S _0- A ₁ , S _2- A ₂ , Stores S _{3- A 3} , S ₄ }). The state S ₀ is the setting of the robot at the time t _{0 , and the action A 0} represents a vector of values at the _{time t 0} that shifts the robot from the state S ₀ to the next state S _1. As shown in FIG. 9, task demonstrations 906b and 908b are recorded correctly. However, due to some random errors, in the task demonstration 910b, the state _{S 1} is not recorded.

Some embodiments are based on the recognition that task demonstrations 906b, 908b and 910b are a sequence of state / action pairs but do not have instructions regarding skills. Therefore, the learning module 206d trains a function to select a skill from the module 206a in order to perform a task based on the task demonstrations 906b, 908b and 910b in the module 206b.

Part 904 represents a learning phase in which a task is performed. As shown in FIG. 9, the learning module 206d is configured to receive a task demonstration at block 906d. For example, the learning module 206d receives three demonstrations 906b, 908b and 910b. In block 908d, the learning module 206d determines, for each state, from the received task demonstrations (ie, the three demonstrations 906b, 908b, and 910b) the distribution of actions that fit the actions received for that state. It is composed of. Block 908d-0 represents the distribution of actions determined for _{state S 0 , block 908d-1 represents the distribution of actions determined for state S 1} , and block 908d-2 represents the distribution of actions determined for state S _2. It represents the distribution of the action, blocks 908d-3 represents the distribution of the actions determined for the state S _3. For example, state S _{0 receives two A 0} actions from task demonstrations 906b and 908b and _{action A 1} from task demonstration 910b. Therefore, the distribution 908d-0 is determined for the states S _{0 that fit the actions A 0} and A _1. Further, the distribution 908d-0 _{returns action A 0} with a probability of 2/3 and action A ₁ with a probability of 1/3. Similarly, distributions 908d-1, 908d-2 and 908d-3 are determined. In some embodiments, the learning module 206d is configured in block 908d to sample the actions on the distribution of actions determined for the corresponding states for each state in the task demonstration. In some embodiments, the sampled actions correspond to the actions with the highest probability on the distribution of actions. Therefore, the sampled actions for states S ₀ , S ₁ , S ₂ and S _{3 are A 0} , A ₁ , A ₂ and A ₃ , respectively.

In some embodiments, the condition can vary between demonstrations 906b, 908b and 910b. The indexes and notations used here are for clarity of description. The same arguments apply to the action. For example, if the states are different, additional processes outside the scope of the invention may be called to determine the alignment according to the most similar state of the task demonstrations 906b, 908b and 910b. An example of such an alignment could be a well-known process of dynamic time warping.

The learning module 206d is configured in block 910d to determine, for a state, the skill with the highest probability of returning a sampled action on the distribution of actions determined for that state. Therefore, the learning module 206d is a function as described in FIGS. 5 to 6.

To learn. In the trained function

Is the state S _t, determines the skill with the highest probability that return action sampled on the distribution of the actions determined for the state S _t. For example, the learning module 206d determines skills H1 and H2 for _{states S 0} and S ₁ , respectively, and determines skill H _{3 for states S 2} and S 3, respectively. Skill H3 is a multi-step skill. Alternatively, the skills determined in the learning phase can be stored as a sequence of skills in set H. For example, set H is H = {H ₁ , H ₂ , H ₃ }, and when executed in sequence by a robot as in set H, it completes the task in the execution phase.

As it should be understood, the learning phase is completed once. The flow continues in the execution phase. Therefore, an execution module 206e is provided to execute the task.

Execution module 206e is configured to receive the current state. For example, the execution module 206e receives the state _{S 0.} Then, execution module 206e is to select the skill with the highest probability on the skill of the distribution, configured to perform the learned function of the state S _0. For example, the execution module 206e selects the _{skill H1 for the state S 0 because the} skill H1 is the skill determined in the learning phase for the state S ₀ , and returns the _{action A 0} having the highest probability. Further, the execution module 206e is configured to execute the skill H1 selected for the _{state S 0} so as to select the _{action A 0.} The execution module 206e commands the robot to perform action A _{0 for state S 0.}

As should be understood, when the robot performs action A ₀ , the robot transitions to _{state S 1.} Execution module 206e again receives the state S _1, to determine whether or not to select the other skills the state S _1, executes the termination function of the state S ₁ β (S). Skill H1 provides a distribution of the action or the action only for the state S _0. Therefore, the termination function returns _{1 for state S1.} When the end function returns 1, execution module 206e, as described with respect to state S _0, configured to select a different skill for state S _1. This process execution module is repeated until it receives a state S _4. Therefore, execution module 206e selects the skill H2 of the state _{S 1,} and the skills H3 of the the state _{S 2} and _{S 3.} Skill H3 is a multi-step skill executed for _{states S 2} and S ₃ by the execution module 206e to return _{actions A 2} and A _{3, respectively.} When the action A ₃ is performed, the robot moves to a state S ₄ indicating the completion of task execution. This selects the skills to be performed by the robot to perform the task from the library of task agnostic skills, which replaces the learning objective with the selective objective.

The above description provides only exemplary embodiments and is not intended to limit the scope, availability, or configuration of the present disclosure. Rather, the following description of an exemplary embodiment provides one of ordinary skill in the art with an operable description for realizing one or more exemplary embodiments. Various changes can be made in the function and composition of the elements, provided that they do not deviate from the spirit and scope of the subject matter disclosed as described in the appended claims.

Specific details are given in the following description to provide a complete understanding of the embodiments. However, as understood by those skilled in the art, embodiments may be implemented without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other cases, well-known processes, structures, and techniques may be presented without unnecessary details to avoid obscuring embodiments. In addition, similar reference numbers and names in various drawings indicate similar elements.

Further, individual embodiments may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams or block diagrams. Flowcharts can describe actions as sequential processes, but many of the actions can be performed in parallel or simultaneously. Moreover, the order of operations can be reconfigured. The process may be terminated when its operation is complete, but may have additional steps not discussed or included in the figure. Moreover, not all actions in any specifically described process can be performed in all embodiments. Processes can correspond to methods, functions, procedures, subroutines, subprograms, and so on. If the process corresponds to a function, the termination of the function may correspond to the return of the function to the calling function or the main function.

Moreover, embodiments of the disclosed subject matter can be realized, at least in part, either manually or automatically. Manual or automatic realization can be performed, or at least assisted, through the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. Program code or code segments that should perform the required tasks, if implemented in software, firmware, middleware, or microcode, may be stored on machine-readable media. The processor can perform the required tasks.

The various methods or processes outlined herein can be encoded as software that can run on one or more processors using any one of different operating systems or platforms. In addition, such software may be written using any of many suitable programming languages and / or programming or scripting tools, and executable machine language code that runs on a framework or virtual machine. Alternatively, it may be compiled as intermediate code. Typically, the functions of the program modules may be combined or distributed as desired in various embodiments.

The embodiments of the present disclosure can be embodied as the methods provided as an example. The operations performed as part of the method may be ordered in any suitable manner. Thus, although shown as sequential actions in the exemplary embodiments, embodiments may be constructed in which the actions are performed in a different order than shown, which may include performing several actions at the same time. ..

Although this disclosure has been described with reference to certain preferred embodiments, it should be understood that a variety of other indications and modifications may be made within the spirit and scope of this disclosure. Therefore, it is an aspect of the appended claims to cover all such modifications and modifications so that they are within the true spirit and scope of the present disclosure.

Alternatively, execution module 206e is the state S ₁ is received, while running the completion function, executes a learning function for the state S _1.

Claims (22)

A controller for controlling the movement of a robot to execute a task.
The controller
An input interface that accepts the current state of the robot for each control step,
Including memory
The memory is configured to store a library of the robot's skills, where each skill is a probability function of the robot's state that returns a distribution of actions in response to submitting the current state. Yes, the skill is task agnostic,
The memory is configured to store a learned function for the task, which returns the distribution of skills in response to submitting the state of the robot. It is a probability function of the state,
The controller further
The processor includes the trained function for the current state so that for each control step in the sequence of control steps reaching the termination condition, the skill with the highest probability on the distribution of skills is selected. Selected for the current state to perform and to select the action with the highest probability on the distribution of actions in order to transition the state of the robot from the current state to the next state. It is configured to perform the above skills
The controller further
A controller comprising an output interface configured to instruct the robot to perform the selected action to perform the task. The parameters of the learned function minimize the probability of difference between the action returned by the skill for each possible state and the corresponding action of multiple executions of the task for the corresponding state. The controller of claim 1, wherein each execution of the task is represented by a sequence of pairs of states and action values, chosen to maximize the probability of returning the skill from said library. The controller according to claim 1, wherein the learned function is a single argument function. At least some skills are multi-step skills that are configured to be performed for multiple control steps, and the processor is responsible for each control step, even if the currently selected skill is a multi-step skill. The controller of claim 1, which repeats the selection of skills. The controller of claim 1, wherein the function for selecting skills is learned to impose a penalty on switching between skills. The controller according to claim 1, wherein the function is learned based on reinforcement learning (RL) having a reward function including a penalty for switching between skills. The function is learned based on reinforcement learning (RL) having a reward function including a penalty based on the distance between the selected action and one or more demonstrated actions, claim 1. Described controller. The controller according to claim 1, wherein the function is learned by maximizing the expected value. The controller of claim 1, wherein the learned function selects the skill by returning a time to switch the current skill from the sequence of skills. A claim further comprising a learning module configured to train the function to improve skill selection using a reinforcement learning (RL) model with a reward function that minimizes the deviation of the result of performing the task. Item 1. The controller according to item 1. It further includes a learning module configured to train the function for selecting the skill.
In order to train the function, the learning module
It is configured to receive a set of executions of the task, and each execution defines a sequence of state / action pairs.
To train the function, the learning module further
For each state, from the sequence of state / action pairs, determining the distribution of actions that fit the actions received for that state.
The controller of claim 1, wherein for a state, determining the skill with the highest probability of returning an action sampled on said distribution of actions determined for that state. A method for controlling the behavior of a robot to perform a task, the method using a processor coupled to a memory, which stores (1) a library of skills of the robot. Each skill is a stochastic function of the state of the robot that returns a distribution of actions in response to submitting the current state, the skill is task insane, and the memory is (2) the task. The learned function is a stochastic function of the state of the robot that returns a skill distribution in response to submitting the state of the robot, and the processor stores. When combined with the instructions given and executed by the processor to perform the steps of the method,
Accepting the current state of the robot for each control step
For each control step in the sequence of control steps that reach the end condition, executing the learned function for the current state to select the skill with the highest probability on the distribution of skills.
Performing the selected skill for the current state to select the action with the highest probability on the distribution of actions in order to transition the state of the robot from the current state to the next state. To do and
A method comprising outputting a command to the robot to perform the selected action in order to perform the task. The parameters of the learned function are skills that minimize the probability of difference between the action returned by the skill for each possible state and the corresponding action of multiple executions of the task for the corresponding state. 12. The method of claim 12, wherein each execution of the task is represented by a sequence of pairs of states and action values, chosen to maximize the probability of the skill returning from the library. The method of claim 12, wherein the learned function is a single argument function. At least some skills are multi-step skills that are configured to be performed for multiple control steps, and the processor is responsible for each control step, even if the currently selected skill is a multi-step skill. 12. The method of claim 12, wherein the selection of skills is repeated. 12. The method of claim 12, wherein the function is learned to select a skill based on a function that imposes a penalty on switching between skills. 12. The function is learned to select a skill based on the resulting action being penalized according to a function of distance to one or more demonstrated actions. Method. 12. The method of claim 12, wherein the function is learned based on reinforcement learning (RL) having a reward function that includes a penalty for switching between skills. The method of claim 12, wherein the function is learned by maximizing the expected value. 12. The method of claim 12, wherein the learned function selects the skill by returning the time to switch the current skill from the sequence of skills. The memory is further configured to store a learning module and uses an RL with a reward function that minimizes the deviation of the result of performing the task when executed by the processor to perform the steps of the method. 12. The method of claim 12, wherein the function is trained to improve skill selection. The memory is further configured to store a learning module and, when executed by the processor to perform the steps of the method, includes training the function to select the skill.
Learning the function further
Each execution defines a sequence of state / action pairs, including receiving a set of executions of the task.
Learning the function further
For each state, from the sequence of state / action pairs, determining the distribution of actions that fit the actions received for that state.
12. The method of claim 12, wherein for a state, determining the skill with the highest probability of returning an action sampled on said distribution of actions determined for that state.

Images