KR102590411B1 - Control policy for robot agents - Google Patents

Abstract

The method includes receiving, for each of one or more objects, data identifying the respective target location to which a robotic agent interacting with the real environment should move the object; causing the robotic agent to move the one or more objects to one or more target locations by repeatedly performing operations, the operations comprising: receiving a current image of a current state of a real environment; determining from the current image a next sequence of actions to be performed by the robotic agent using a next image prediction neural network that predicts future images based on the current action and the action to be performed by the robotic agent; and instructing the robot agent to perform the next sequence of actions.

Description

Control policy for robot agents

This specification concerns selecting actions to be performed by a robotic agent.

Robotic agents interact with the environment by receiving data characterizing the state of the environment and performing actions to perform robotic tasks. Some robotic agents use neural networks to select an action to perform in response to receiving certain observations.

Neural networks are machine learning models that use one or more layers of nonlinear units to predict an output for received input. Some neural networks are deep neural networks that include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as input to the next layer of the network, that is, the next hidden layer or output layer. Each layer of the network generates an output from the received input according to the current values of each parameter set.

This specification describes a method for a system implemented as a computer program on one or more computers at one or more locations to select an action to be performed by a robotic agent.

In general, the innovative aspect may be implemented in the following way, wherein, for each of one or more objects, a robotic agent interacting with the real environment receives data identifying the respective target location to which the object should be moved. It includes steps to: The method includes causing the robot agent to move the one or more objects to one or more target locations by repeatedly performing the following operations, the operations comprising: receiving a current image of the current state of the real environment; ; Determining from the current image a next sequence of actions to be performed by the robotic agent using a next image prediction neural network that predicts future images based on the current action and the action to be performed by the robotic agent - the next sequence is a sequence among a plurality of candidate sequences that, if performed by a robotic agent starting when the environment is in its current state, would most likely result in one or more objects being moved to the respective target location; and instructing a robotic agent to perform the next sequence of actions. The current image may be an image captured by the robot agent's camera.

The act of instructing the robot agent to perform the next sequence of actions includes interrupting the current sequence of actions being performed by the robot agent and instructing the robot agent to perform the next sequence of actions.

The method may further include providing a user interface that allows the user to specify an object to be moved and a target location for presentation to the user.

A next image prediction neural network receives as input at least a current image and an input action and, if the robot agent performs said input action when the environment is in its current state, generates a next image that is an image of the predicted next state of the environment. It may be a recurrent neural network trained to process the input. As part of generating the next image, the recurrent neural network generates a flow map that identifies, for each of the plurality of pixels in the next image, the respective predicted likelihood of the pixel moved from each of the plurality of pixels in the current image. .

Determining the next sequence of actions uses the flow map generated by the next image prediction neural network to determine for each of the candidate sequences that the performance of the actions within the candidate sequence by the robotic agent will cause the objects to move to the target locations. It may include determining each likelihood.

Determining the next sequence of actions may include determining one or more pixels of the current image that depict one or more objects located in the current environment.

Determining the respective likelihoods for a given candidate sequence may include recursively feeding the neural network as input the actions in the sequence and the next images generated by the neural network for the actions. there is.

The method may further include sampling candidate sequences from a distribution over possible action sequences. Sampling candidate sequences may include performing multiple iterations of sampling using a cross-entropy technique.

Another innovative aspect may be implemented with one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the method described above.

Another innovative aspect may be implemented in one or more non-transitory storage media encoded with instructions that, when executed by one or more computers, cause one or more computers to perform the operations of the method described above.

Certain embodiments of the subject matter described herein may be implemented to realize one or more of the following advantages. By combining a learned predictive model with a model predictive control (MPC)-based controller to select the actions to be performed by the robotic agent, the system described herein allows the robotic agent to use only fully unlabeled training data, i.e., training images. It effectively moves objects to target locations without requiring the use of additional computational resources to label or require training images to be labeled by the user. Moreover, the techniques described here do not require calibrated cameras, instrumented training setups, or accurate sensing and actuation by robotic agents. The techniques described also enable the robot to process new objects that were not seen during training of the next image prediction neural network, allowing the techniques described herein to better generalize to different tasks and objects.

The details of one or more embodiments of the subject matter of the present disclosure are set forth in the accompanying drawings and the description below. Other features, aspects and advantages of the subject matter will become apparent from the detailed description, drawings and claims.

1 is a block diagram of an example of a robot manipulation system.
2 is a flow diagram of an example process in which a robotic agent moves an object to a target location.
3 is a flow diagram of an example process for determining the next sequence of actions to be performed by a robotic agent to move an object to a target location.
Like reference numerals and names in the various drawings indicate like elements.

1 is a block diagram of an example of a robot manipulation system 100. System 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, in which the systems, components and techniques described below may be implemented.

Generally, system 100 includes agents 102 interacting with a real-world environment 104 to move one or more objects from each initial location within environment 104 to each final location within environment 104. It is configured to select actions to be performed by the robot agent 102.

In particular, system 100 provides data 112 that identifies, for each of one or more objects within real environment 104, a respective target location to which the robotic agent 102 should move the object, e.g., target location 114. It includes a controller 106 configured to receive.

In some implementations, system 100 may provide a user interface that allows a user to specify target locations and one or more objects to be moved for presentation to the user. For example, a user may specify a source pixel in an initial image depicting an object as it is currently located in the environment 104 and where the source pixel should be moved, i.e., where another pixel in the initial image should be moved. can be specified. The initial image may be captured by the camera of the robotic agent 102 and may characterize the initial state of the environment 104.

For example, the initial image shows a cup placed on a rectangular tray. The user can specify a source pixel within the image that belongs to a cup and a target location in the image to which the source pixel should be moved, for example to a position near one of the corners of a rectangular tray. Depending on these target specifications, system 100 can control the robotic agent to move the cup to a specific corner of the rectangular tray. In some other implementations, instead of receiving data identifying the object to be moved and a target location from a user, the system may receive data from one or more other systems, such as one or more other robotic systems.

The controller 106 then uses the learned prediction model 108 to control the robotic agent 102 to move the object toward the target location 114.

The learned prediction model 108 includes the following image prediction neural network 110. The next image prediction neural network 110 is trained to receive as input a current image of the current state of the environment 104 and an input action. The input action may be, for example, pushing an object to a target position or rotating the object. The neural network 110 then processes the received input to generate a next image, which is an image of the predicted next state of the environment if the robotic agent performs the input action when the environment is in its current state. In the process of generating the next image, the neural network 110 generates a flow map that identifies each prediction likelihood of a pixel moved from each of the plurality of pixels in the current image for each of the plurality of pixels in the next image.

Next, an example of an image prediction neural network 110 is a recurrent neural network that includes a set of normalized convolutional kernels, one or more convolutional neural network layers, and a stack of convolutional long short-term memory (LSTM) neural network layers. Convolutional LSTM neural network layers are similar to regular LSTM neural network layers, but their gates are implemented by convolutions instead of fully connected neural network layers. Convolutional LSTM neural network layers are described in detail in a 2015 NIPS (Neural Information Processing Systems) paper (X. Shi et al. "Convolutional lstm network: A machine learning approach for precipitation nowcasting."). The above example of neural network 110 is described in detail in the 2016 Neural Information Processing Systems (NIPS) article (C. Finn, I. Goodfellow, and S. Levine, "Unsupervised learning for physical interaction through video prediction,"

At a given time step, given the target location 114, the controller 106 causes the robotic agent 102 to move the object to the target location. Having the robotic agent 102 move an object to a target location includes repeatedly performing the process 200 until the object reaches the target location, as described below with reference to FIG. 2 .

For example, the controller 106 may receive a current image 116 characterizing the current state of the environment 104 and then use the learned prediction model 108 to move the object to the target location 114. Determine the next sequence of actions (118) to be performed by the robotic agent (102). Determining the next sequence of actions using the learned prediction model 108 is described in more detail with reference to FIG. 3 . Controller 106 then instructs robotic agent 102 to perform an operational sequence of actions 118. When the robotic agent is performing a current sequence of actions, the controller 106 causes the robotic agent 102 to interrupt the current sequence of actions being performed and causes the robotic agent 102 to perform the next sequence of actions 118. Start performing. After agent 102 performs one or more actions in the next sequence of actions 118, controller 106 may receive a new image characterizing the new state of the environment. Controller 106 continues to determine another next sequence of actions that will update the current image 116 with a new image. Controller 106 may repeatedly determine next sequences of actions until a predetermined number of next sequences are determined or until robotic agent 102 successfully moves the object to the target location 114.

2 is a flow diagram of an example process that causes a robotic agent to move an object to a target location in the real environment. For convenience, process 200 will be described as being performed by a system of one or more computers located at one or more locations. For example, a robot manipulation system, e.g., robot manipulation system 100 of FIG. 1, or a component of a robot manipulation system, e.g., controller 106 of FIG. 1 appropriately programmed in accordance with the present disclosure, may perform process 200 can be performed.

The system receives a current image of the current state of the real environment, for example as captured by the robotic agent's camera (step 202). If the current image is an initial image (I _t ) depicting an object initially located in the real environment (i.e., depicting the initial state of the real environment), then the goal is a single specified pixel of the initial image (I _t ) (d _t = (xd, yd)) is moved to the target location (g = (xg, yg)) of the initial image (I _t ). If the current image is not the initial image, for example, if the current image is the next image (I _t + 1) representing the next state of the real environment, the system may perform an image observation (I _{t: t} + 1), for example and d _t ), the specified pixel can be updated from d _t to d _t +1 by using the optical flow calculated in d t ). If the target does not change, the target location (g = (xg, yg)) remains the same in the next image predicted by the system.

The system uses the learned prediction model to determine from the current image the next sequence of actions to be performed by the robotic agent to move the object to the target location (step 204). The next sequence is one of a plurality of candidate sequences that, if performed by a robotic agent starting when the environment is in its current state, would most likely result in the object being moved to the target location.

For example, the system uses a learned prediction model to determine future actions ( H future actions) that, when performed by a robotic agent, are most likely to cause a given pixel (dt) to move to the target location (g). ) determine the next sequence.

Determining the next sequence of actions is explained in more detail below with reference to FIG. 3 .

The system then instructs the robotic agent to perform the next sequence of actions (step 206). When a robotic agent is performing a current sequence of actions, the system may cause the robotic agent to interrupt the current sequence of actions being performed and instead have the agent begin performing the next sequence of actions. After the agent performs the first action in the next sequence of actions, the system receives a new image characterizing the new state of the environment. In some implementations, the system may receive a new image only after the robotic agent has performed a predetermined number (> 1) of actions in the next sequence of actions.

For example, the system may execute the determined actions ( ) instructs the agent to perform the following sequence. The agent takes action in the sequence ( ), the system receives a new image (I _t +1). The system then updates the specified pixel from d _t to d _t +1, for example by using the optical flow calculated from the image observations (I _t:t +1 and d _t ). The system then uses the learned prediction model to determine a new next sequence of actions to be performed by the robotic agent based on the new designated source pixel (I _t + 1) and target location (g). The system then interrupts the next sequence of actions being performed and instructs the robotic agent to begin performing one or more actions in the new next sequence of actions. After the robotic agent performs the first action in the next new sequence of actions, the system can receive a new image of the new state of the real environment and update the designated pixel. In some implementations, the system may receive a new image and update a designated pixel only after the robotic agent has performed some actions in the next new sequence of actions.

The system may repeatedly perform steps 202-206 until the robotic agent successfully moves the object to the target location.

3 is a flow diagram of an example process 300 for determining the next sequence of actions to be performed by a robotic agent to move an object to a target location.

For convenience, process 300 will be described as being performed by a system of one or more computers located at one or more locations. For example, a robotic manipulation system, such as robotic manipulation system 100 of FIG. 1 appropriately programmed in accordance with the present disclosure, can perform process 300.

Generally, process 300 involves multiple iterations of sampling candidate action sequences using a cross-entropy technique. In general, the cross-entropy technique involves each iteration (a) generating a random data sample according to a specified mechanism and (b) updating the parameters of the specified mechanism based on this data to generate better samples in the next iteration. It involves an iterative process that can be divided into two steps:

First, at a given time step t, the system initializes a distribution of the set of possible action sequences that can be performed by the robotic agent to move the object to the target location (step 302). The initialized distribution may be, for example, a uniform distribution. Each of the possible action sequences has the same length, eg, each sequence has H actions. The system also specifies the number of iterations J needed to sample candidate action sequences (J>1). That is, J is the number of times the system must repeatedly perform steps 304 to 308, as described below. J can be specified by the system user.

The system then samples M candidate action sequences from the distribution (step 304). For example, in the first iteration, the system generates M action sequences of length H from an initialized uniform distribution ( ) is sampled.

Next, the system uses the learned prediction model to determine, for each of the M candidate sequences, the respective probability that the object will be moved to the target location due to performance of the actions in the candidate sequence by the robotic agent, As a result, the specified pixel (d _t ) in the current image (I _t ) is moved to the target position ( g = ( x _g , y _g )) of the current image (I _t ) (step 306). The learned prediction model includes the following image prediction neural network, which is a recurrent neural network. In particular, for each of the M candidate sequences, the system first feeds the current image characterizing the current state of the environment and the first action in the candidate sequence as input to the next image prediction neural network. The next image prediction neural network then processes the input to generate a first next image, which is an image of the predicted next state of the environment if the robotic agent first performs the first action when the environment is in its current state. As part of generating the first next image, a next image prediction neural network may perform, for each of a plurality of pixels in the first next image, a respective predicted probability (e.g., For example, generate a first flow map that identifies the likelihood.

The system then supplies as input to the next image prediction neural network a second action that follows the first action in the sequence and the first next image generated by the next image prediction neural network. The next image prediction neural network then processes the input to generate a second next image, which is an image of the predicted next state of the environment when the robotic agent performs the second action. While generating a second next image, the next image prediction neural network creates a second flow map that identifies, for each of a plurality of pixels in the second next image, a respective probability of a pixel being moved from each of a plurality of pixels in the current image. Create.

The system recursively feeds subsequent actions in the sequence and the next images generated by the neural network for those actions as input to an image prediction neural network, and the neural network iteratively feeds the subsequent actions in the sequence until all actions in the action sequence have been processed. Processes the input to generate the following images and the corresponding flow map. The flow map generated at each step provides the probability that each pixel of the next image comes from each pixel of the current image. After the last (final) action of the sequence is processed, the system uses _an image prediction neural network to determine, for each of the plurality of pixels in the final next image (I _t + H-1), From each pixel, determine the final flow map that identifies the respective probability of the pixel moving to the final next image (I _t + H-1). That is, the final flow map provides the probability that each pixel of the final next image (I _t + H-1) comes from each pixel of the current image (I _t ). Then, based on the final flow map and the target position (g = (xg, yg) (which remains the same in any next image generated by the next image prediction neural network), the system determines whether the agent will take an action in the candidate sequence. When these are performed, the probability that a specified pixel (d _t ) in the current image (I _t ) will be moved to the target position (g) of the final next image is determined.

After the system determines the respective probabilities for each of the M candidate sequences, the system selects the K candidate action sequences with the highest probability among the M candidate action sequences (step 308).

The system then determines whether the number of iterations for sampling the K candidate action sequences (i.e., steps 304-308) has reached the specified J number of iterations (step 310).

If the number of iterations does not reach J, the system fits a new distribution to the selected K candidate action sequences (step 312). That is, the system determines a new distribution that fits the selected K candidate action sequences. The new distribution may be, for example, a multivariate Gaussian distribution.

The system then repeats steps 304-312, namely resampling a new set of M action sequences from a new distribution, determining respective probabilities for each of the M candidate sequences, and The steps of selecting K candidate action sequences from the M candidate sequences and refitting the new distribution are repeated until the system completes J number of iterations.

At the end of the final iteration, the system selects, from among the K candidate action sequences, the action sequence with the highest probability that performance of the actions within the sequence will cause the object to move toward the target location (step 314).

The system then causes the robotic agent to perform one or more actions in the selected action sequence to move the object toward the target location (step 316). The system may interrupt the current sequence of actions being performed and instruct the robotic agent to begin performing the selected action sequence.

In light of the above description, it will be appreciated that the systems and methods described herein may provide one or more advantages. For example, the system allows a robotic agent to effectively move an object to a target location using a neural network trained using only fully unlabeled training data. Additionally, the system can allow objects to be moved by a robotic agent using calibrated cameras, instrumented training setups, or simpler setups that do not require precise sensing and actuation by the robotic agent. You can. The described technique also allows the robotic agent to process new, unseen objects during the training of the next image prediction neural network, broadening the potential utility of the system.

This specification uses the term “configured” in reference to system and computer program components. One or more computer systems configured to perform a particular operation or action means that the system causes the system to perform the operation or action due to software, firmware, hardware, or a combination thereof. One or more computer programs configured to perform a particular operation or action means that the one or more computer programs, when executed by a data processing device, include instructions that cause the device to perform the operation or action.

Embodiments of the subject matter and functional operations described herein may include the structures disclosed herein and their structural equivalents, or a combination of one or more thereof, in a digital electronic circuit, tangibly-embodied computer software or firmware, or computer hardware. It can be implemented in . Embodiments of the subject matter described herein may comprise one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible temporary storage medium to be executed by or to control the operation of the data processing device. It can be implemented. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be an artificially generated propagated signal, e.g., machine-generated electricity, generated to encode information for transmission to a suitable receiver device for execution by a data processing device. , may be encoded on optical or electromagnetic signals.

The term “data processing apparatus” means data processing hardware and includes all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, or a plurality of processors or computers. The device may also be a special purpose logic circuit, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In addition to hardware, the device may optionally include code that creates an execution environment for computer programs, for example, code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

A computer program (which may be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any form of programming language, including a compiled or interpreted language, or a declarative or procedural language. and may be distributed in any form, including as a stand-alone program, module, component, subroutine, or other device suitable for use in a computing environment. Computer programs can, but do not necessarily, correspond to files in a file system. A program can be a single file dedicated to the program, a number of coordinated files (for example, files storing one or more modules, subprograms, or portions of code), or other programs, such as one or more scripts stored in a markup language document. It may be stored in the part of the file that holds the data. A computer program may be distributed to run on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

The processes and logic flows described herein may be performed by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating output. The above processes and logic flow may also be performed by a special purpose logic circuit, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and the device may also be implemented with the special purpose logic circuit.

A computer suitable for the execution of computer programs may include, or be based on, for example, a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from read-only memory or random access memory, or both. The essential components of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer includes one or more mass storage devices for storing data (e.g., magnetic, magneto-optical disks, or optical disks), or is operable to receive or transmit data from the one or more mass storage devices. will be combined However, a computer need not have such devices. Additionally, the computer may be connected to other devices, such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable storage devices (e.g., Universal Serial Bus (USB) flash drives). ) can be built into.

Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD ROMs. and all forms of non-volatile memory, media and memory devices, including DVD-ROM disks.

In order to provide interaction with a user, embodiments of the subject matter described herein include a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor to provide information to the user, and a display device such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor to provide information to the user, It may be implemented on a computer having a keyboard and a pointing device, such as a mouse or trackball, that can be provided to the computer. Different types of devices may be used to provide interaction with the user, for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and may be used to provide interaction with the user. The input may be received in any form, including acoustic, vocal, or tactile input. The computer may also interact with the user by sending and receiving documents to and from the device used by the user, for example, by sending web pages to a web browser on the user's client device in response to requests received from the web browser. It can be done. Additionally, the computer may interact with the user by sending text messages or other forms of messages to a personal device (eg, a smartphone running a messaging application) and receiving response messages from the user.

Data processing devices for implementing machine learning models may also include special-purpose hardware accelerator units for processing common and computationally intensive parts of the workload, for example, machine learning training or production, i.e., inference.

Machine learning models can be implemented and deployed using machine learning frameworks such as the "TensorFlow" framework, the "Microsoft Cognitive Toolkit" framework, the "Apache Singa" framework, or the "Apache MXNet" framework.

Embodiments of the subject matter described herein include back-end components such as data servers; Middleware components such as application servers; a front-end component, such as a client computer having, for example, a relational graphical user interface or a web browser through which a user can interact with implementations of the subject matter described herein; Or, it may be implemented in a computing system that includes any combination of one or more back end, middleware, and front end components. The components of the system may be interconnected by any form or medium of digital data communication, for example a communication network. Exemplary communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

The computing system may include clients and servers. Clients and servers are usually remote from each other and typically interact through a communications network. The relationship between client and server arises due to computer programs running on each computer and having a client-server relationship with each other. In some embodiments, the server sends data, e.g., an HTML page, to a user device to display data and receive user input from a user interacting with the device acting as a client. Data generated on the user device, for example the results of user interactions, may be received on a server from the device.

Although this specification contains numerous specific implementation details, these should not be construed as limitations on any invention or scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. do. Certain features described herein in relation to separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Moreover, while features may operate in certain combinations and be described as initially claimed, as described above, one or more features of a claimed combination may in some cases be removed from that combination, and the claimed combination may be referred to as a subcombination or You can aim for transformation of that subcombination.

Similarly, although operations are shown in the drawings in a particular order, this is to be understood to require that such operations be performed in the order shown or sequential order, or that all operations shown be performed, to achieve the desired operations. should not be done. Multitasking and parallel processing can be advantageous in certain situations. Additionally, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and the described program components and systems are generally integrated together in a single software product or integrated into multiple software products. It is important to understand that products can be packaged.

Specific embodiments of the invention have been described. Other embodiments are within the scope of the following claims. For example, the operations recited in the claims can be performed in a different order and still achieve the desired result. By way of example, the process depicted in the accompanying drawings does not necessarily require the specific order or sequential order shown to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims (13)

As a method,
Receiving, by the one or more computers, data identifying, for each of the one or more objects, a respective target location to which a robotic agent interacting with a real-world environment should move the object;
causing, by the one or more computers, the robotic agent to move the one or more objects to one or more target locations by repeatedly performing actions,
The above operations are,
Receiving a current image of the current state of the real environment;
Determining from the current image a next sequence of actions to be performed by the robotic agent using a next image prediction neural network that predicts future images based on the current action and the action to be performed by the robotic agent - the next sequence is one candidate sequence among a plurality of candidate sequences that, if performed by the robotic agent starting when the environment is in the current state, would most likely result in the one or more objects being moved to respective target locations, and ; The operation that determines the next sequence of actions is:
For each of the plurality of candidate sequences:
Recursively supplying subsequent actions of the candidate sequence and next images generated by the next image prediction neural network for the subsequent actions as input to the next image prediction neural network, each of the next images being the current associated with a flow map that provides the probability of moving from each pixel in an image to each pixel in the next image,
Using the next image prediction neural network, determine a final flow map for the final next image of the candidate sequence, which is the image of the predicted final state of the environment when the robotic agent performs all actions in the candidate sequence, and
Based on the final flow map and target location, determining a probability that performance of all actions of the candidate sequence by the robotic agent will cause the one or more objects to move to the respective target locations in the final next image. Contains actions-; and
selecting, from the plurality of candidate sequences, the candidate sequence with the highest probability as the next sequence of actions; and
instructing the robotic agent to perform the next sequence of actions, wherein the next image prediction neural network receives as input at least a current image and an input action, and the robotic agent when the environment is in the current state. and is configured to process the input to generate a next image, which is an image of a predicted next state of the environment, upon performing the input action. The method of claim 1, wherein the current image is an image captured by a camera of the robot agent. The method of claim 1, wherein
The method further comprising providing a user interface that allows the user to specify target positions and objects to be moved, for presentation to the user. According to paragraph 1,
The operation of instructing the robot agent to perform the next sequence of actions includes interrupting the current sequence of actions performed by the robot agent and instructing the robot agent to begin performing the next sequence of actions. A method comprising: According to paragraph 1,
The next image prediction neural network is a recurrent neural network,
As part of generating the next images, the recurrent neural network includes a flow map that identifies, for each of a plurality of pixels in the next image, a respective probability of a pixel being moved from each of a plurality of pixels in the current image. A method characterized in that generating for each of the following images. delete In paragraph 1
wherein determining the next sequence of actions comprises determining one or more pixels of the current image that depict one or more objects currently located in the environment. delete The method of claim 1, wherein
The method further comprising sampling said candidate sequences from a distribution over possible action sequences. According to clause 9,
wherein sampling the candidate sequences includes performing multiple iterations of sampling using a cross-entropy technique. As a system,
one or more computers; and
Comprising one or more storage devices, wherein the one or more storage devices, when executed by the one or more computers, cause the one or more computers to perform the operations of claims 1 to 5, 7, 9, and 10. A system characterized by storing instructions that perform the operations of each method of any one of the methods. Instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of each method of any one of claims 1 to 5, 7, 9, and 10. One or more computer-readable storage media encoded with . Comprising computer-readable instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 5, 7, 9, and 10. A computer program contained in a computer-readable storage medium that