Classifications

• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
  • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
  • B25J9/00—Program-controlled manipulators
  • B25J9/16—Program controls
  • B25J9/1628—Program controls characterised by the control loop
  • B25J9/163—Program controls characterised by the control loop learning, adaptive, model based, rule based expert control
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
  • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
  • B25J9/00—Program-controlled manipulators
  • B25J9/16—Program controls
  • B25J9/1602—Program controls characterised by the control system, structure, architecture
  • B25J9/161—Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
  • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
  • B25J9/00—Program-controlled manipulators
  • B25J9/16—Program controls
  • B25J9/1694—Program controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
  • B25J9/1697—Vision controlled systems
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/60—Type of objects
  • G06V20/64—Three-dimensional [3D] objects
  • G06V20/647—Three-dimensional [3D] objects by matching two-dimensional images to three-dimensional objects
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
  • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
  • B25J9/00—Program-controlled manipulators
  • B25J9/16—Program controls
  • B25J9/1612—Program controls characterised by the hand, wrist, grip control
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/30—Nc systems
  • G05B2219/39—Robotics, robotics to robotics hand
  • G05B2219/39082—Collision, real time collision avoidance
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/30—Nc systems
  • G05B2219/39—Robotics, robotics to robotics hand
  • G05B2219/39505—Control of gripping, grasping, contacting force, force distribution
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/30—Nc systems
  • G05B2219/39—Robotics, robotics to robotics hand
  • G05B2219/39531—Several different sensors integrated into hand
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/30—Nc systems
  • G05B2219/39—Robotics, robotics to robotics hand
  • G05B2219/39542—Plan grasp points, grip matrix and initial grasp force
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/30—Nc systems
  • G05B2219/40—Robotics, robotics mapping to robotics vision
  • G05B2219/40053—Pick 3-D object from pile of objects
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/30—Nc systems
  • G05B2219/40—Robotics, robotics mapping to robotics vision
  • G05B2219/40155—Purpose is grasping objects
• • G—PHYSICS
  • G05—CONTROLLING; REGULATING
  • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
  • G05B2219/00—Program-control systems
  • G05B2219/30—Nc systems
  • G05B2219/40—Robotics, robotics mapping to robotics vision
  • G05B2219/40571—Camera, vision combined with force sensor

Definitions

• the disclosed technology relates generally to robotic systems configured for robustly grasping and/or targeting a variety of objects, e.g., in warehouses or homes, e.g., using grippers or suction cup devices and, more particularly, to using libraries of 3D object models that can be analyzed and statistically sampled to train a robot to grasp and/or target objects with robustness to errors in sensing and control.
• Grasping can be simplified when one has an exact model of the object to be grasped and the exact position and location of the object gripper and a gripper that works exactly as expected to work. Sensors are often inaccurate and noisy, and grippers themselves generally have finite amounts of accuracy and precision with which they can be controlled. As such, there is enough uncertainty that consistent robust grasping is a significant, if not unattainable, challenge.
• One approach is to use machine learning.
• One approach is to train a system to predict how robust a particular grasp on a given object will be (e.g., whether the grasp will fail when the object is lifted or moved) using datasets collected from millions of robots grasping millions of objects in millions of physical robot trials. Unfortunately, this is simply not a practical solution.
• Implementations of the disclosed technology can be advantageously used in a variety of industries, such as in connection with robots used for warehouse order fulfillment (e.g., where orders are unique and there can be millions of different products), manufacturing, packing, inspection, marking or otherwise labeling, bin picking, and in homes for menial tasks such as decluttering, for example.
• industries such as in connection with robots used for warehouse order fulfillment (e.g., where orders are unique and there can be millions of different products), manufacturing, packing, inspection, marking or otherwise labeling, bin picking, and in homes for menial tasks such as decluttering, for example.
• FIGURE 1 illustrates an example of a computer-implemented method in accordance with certain embodiments of the disclosed technology.
• FIGURE 2 illustrates an example of a system in accordance with certain embodiments of the disclosed technology.
• FIGURE 1 illustrates an example of a computer-implemented method 100 in accordance with certain embodiments of the disclosed technology.
• the method 100 includes generating a training dataset 140 of labelled examples 150 and 160 of sensor images and grasp configurations using a set of three-dimensional (3D) models of objects 110, one or more analytic mechanical representations of either or both of grasp forces and grasp torques, and statistical sampling to model uncertainty in either or both sensing and control.
• 3D three-dimensional
• the method 100 can also include using the training dataset 140 to train a function approximator 170 that takes as input a sensor image 130 and returns data that is used to select grasp configurations 120 for a robot grasping mechanism.
• grasp configurations may be defined by one or more points relative to the sensor image.
• the generated training dataset 140 may include a measure of grasp quality associated with a grasp configuration 120.
• the measure of grasp quality may include robustness to errors in position or forces.
• the measure of grasp quality may include probability of successful lifting of the object.
• the statistical sampling may include uncertainty in variables related to initial state, contact, physical motion, friction, inertia, object shape, robot control, sensor data, or any suitable combination thereof.
• the function approximator may be a Convolutional Neural Network (CNN), a Random Forest, a Support Vector Machine (SVM), or a linear weight matrix.
• the robotic grasping mechanism can include a robot gripper, a multi-fingered robot hand, one or more suction cups, a magnet, adhesive material, or any suitable combination thereof.
• the set of 3D object models can include mass properties for computation of stable poses for each object.
• the set of 3D object models can include mass properties for computation of resistance to either or both of gravity and inertia for a given grasp and object combination.
• the set of 3D object models can include material properties for computation including frictional properties, deformation, porosity, color textures for photorealistic rendering, or any combination thereof.
• the set of 3D object models may include "adversarial" objects whose geometry makes them difficult to be grasped.
• the set of 3D object models may be augmented by synthetic product packaging models, such as “skin packs” or “blister packs,” for example.
• the set of 3D object models may be augmented using transformations of the initial set of object models using any of the following operations: scaling, stretching, twisting, shearing, cutting, and combining objects.
• the analytic mechanical representations may include wrench mechanics.
• the analytic mechanical representations may include any of the following metrics: force closure grasp quality, Ferrari-Canny grasp quality, suction cup contact quality, wrench resistance quality, magnetic contact quality, and adhesive contact quality.
• Such implementations may include using computer simulation to estimate grasp quality.
• such implementations may include using statistical sampling to compute statistics of a grasp quality metric.
• the grasp quality metric may include average, median, moments, percentiles, or any suitable combination thereof.
• such implementations may include using statistical sampling to compute the statistics of the grasp quality metric includes using numerical integration.
• Certain implementations may include using statistical sampling to generate variations in possible object poses, sensor poses, camera parameters, lighting, material properties, friction, forces, torques, robot hardware parameters to generate the sensor images (e.g., a 3D depth map), or any suitable combination thereof.
• Certain implementations may include using computer simulation to generate variations in possible object poses, sensor poses, camera parameters, lighting, material properties, friction, forces, torques, and robot hardware parameters to generate the sensor images (e.g., a 3D depth map), or any suitable combination thereof.
• configurations for each object model may be stored as a network with one or more computed relationships between object models such as similarity in shape.
• the data in the network of object models and the computed grasp configurations for each object model may be used to efficiently compute one or more desired grasp configurations for one or more new object models.
• Certain implementations may include communicating over a network, e.g., the Internet, to obtain updated function approximator parameters.
• a network e.g., the Internet
• a computer-implemented method may include generating a training dataset of labelled examples of sensor images and target points within those images using a set of three-dimensional (3D) models of objects, one or more analytic evaluation methods of desired target points on an object, and statistical sampling to model uncertainty in either or both sensing and control.
• the method may also include using the training dataset to train a function approximator that takes as input a sensor image and returns data to compute one or more target points for a robot targeting mechanism (e.g., placing a label on the object, affixing a stamp to the object, and inspecting the object).
• the statistical sampling may include uncertainty in variables related to initial state, contact, physical motion, friction, inertia, object shape, robot control, sensor data, or a combination thereof.
• the function approximator may include a Convolutional Neural Network (CNN), a Random Forest, a Support Vector Machine, or a linear weight matrix.
• CNN Convolutional Neural Network
• Random Forest Random Forest
• Support Vector Machine or a linear weight matrix.
• the set of 3D object models can include mass properties for computation of stable poses for each object.
• the set of 3D object models can include mass properties for computation of resistance to either or both of gravity and inertia for a given targeting and object combination.
• the set of 3D object models can include material properties for computation including frictional properties, deformation, porosity, color textures for photorealistic rendering, or any combination thereof.
• the set of 3D object models may include "adversarial" objects whose geometry makes them difficult to be targeted.
• the set of 3D object models may be augmented by synthetic product packaging models, such as “skin packs” or “blister packs,” for example.
• the set of 3D object models may be augmented using transformations of the initial set of object models using any of the following operations: scaling, stretching, twisting, shearing, cutting, and combining objects.
• the analytic mechanical representations may include wrench mechanics.
• Certain implementations may include using statistical sampling to generate variations in possible object poses, sensor poses, camera parameters, lighting, material properties, friction, forces, torques, robot hardware parameters to generate the sensor images (e.g., a 3D depth map), or any suitable combination thereof.
• Certain implementations may include using computer simulation to generate variations in possible object poses, sensor poses, camera parameters, lighting, material properties, friction, forces, torques, and robot hardware parameters to generate the sensor images (e.g., a 3D depth map), or any suitable combination thereof.
• configurations for each object model may be stored as a network with one or more computed relationships between object models such as similarity in shape.
• the data in the network of object models and the computed grasp configurations for each object model may be used to efficiently compute one or more desired grasp configurations for one or more new object models.
• Certain implementations may include communicating over a network, e.g., the Internet, to obtain updated function approximator parameters.
• a network e.g., the Internet
• grasp configurations may be defined by one or more points relative to the sensor image.
• FIGURE 2 illustrates an example of a system 200 for grasping objects 210 in accordance with certain embodiments of the disclosed technology.
• the system 200 includes a sensor 220, a robotic grasping mechanism 240, and one or more processors configured to use sensor images to compute a desired grasp configuration for the robotic grasping mechanism 240 based at least in part on a function approximator 230 that is trained on a training dataset of labelled examples of sensor images and grasp configurations using a set of three-dimensional (3D) models of objects, one or more analytic mechanical representations of either or both of grasp forces and grasp torques, and statistical sampling to model uncertainty in either or both sensing and control.
• the function approximator 230 may be configured to take as input a sensor image and return data that is used to compute robust grasp configurations for the robotic grasping mechanism 240.
• a set of candidate grasp configurations may be computed based on computing potential antipodal grasps from the sensor image, e.g., a 2D image from a camera.
• Certain implementations may include obtaining a 3D depth map using a structured lighting system, a Lidar system, a stereo pair of color cameras, a stereo pair of monochrome cameras, a monocular image, or any suitable combination thereof.
• Certain implementations may include the use of motion planning methods to avoid robot contact with the environment based on collision checking.
• the system may include means for using robot motions to move grasped objects into specific new configurations.
• the system may include means for detecting the outcome of grasp attempts.
• the system may be configured to detect the outcome of grasp attempts includes using one or more load cells, a light sensor, a camera, a force sensor, a tactile sensor, or any suitable combination thereof.
• Certain embodiments may include applying multiple grasping methods in parallel.
• the system may include a sensor, a robotic targeting mechanism, and one or more processors configured to use sensor images to compute a desired target configuration for the robotic targeting mechanism based at least in part on a function approximator that is trained on a training dataset of labelled examples of sensor images and target points using a set of three-dimensional (3D) models of objects, one or more analytic mechanical representations of target points, and statistical sampling to model uncertainty in either or both of sensing and control.
• the function approximator may be configured to take as input a sensor image (e.g., a 2D image from a camera) and return data that is used to compute robust target points for the robotic targeting mechanism.
• Certain implementations may include obtaining a 3D depth map using a structured lighting system, a Lidar system, a stereo pair of color cameras, a stereo pair of monochrome cameras, a monocular image, or any suitable combination thereof.
• Certain implementations may include the use of motion planning methods to avoid robot contact with the environment based on collision checking.
• the system may include means for detecting the outcome of targeting attempts.
• detecting the outcome of targeting attempts may include using one or more load cells, a light sensor, a camera, a force sensor, a tactile sensor, or any suitable combination thereof.
• Certain implementations may include applying multiple targeting methods in parallel.
• Certain implementations may include two or more targeting methods where the outputs of two or more function approximators may be combined to select between these targeting methods for a given sensor image.
• Certain implementations may also include pushing objects to separate them from the environment and create an accessible grasp.
• Grasp and target success can be predicted directly from depth images by training a deep Convolutional Neural Network (CNN) on a massive dataset of parallel-jaw grasps, grasp metrics, and rendered point clouds generated using analytic models of robust grasping and image formation.
• CNN deep Convolutional Neural Network
• a deep neural network can be trained for grasp classification, e.g., from a synthetic dataset of over 6.7 million point clouds, grasps, and robust analytic grasp metrics generated from thousands of three-dimensional (3D) models in randomized poses on a table.
• the resulting dataset can be used to train a Grasp Quality Convolutional Neural Network (GQ-CNN) model that rapidly classifies grasps as robust from depth images and the position, angle, and height of the gripper above a table.
• GQ-CNN Grasp Quality Convolutional Neural Network
• the Grasp Quality Convolutional Neural Network (GQ-CNN) model can be trained to classify robust grasps in depth images using expected epsilon quality as supervision, where each grasp is specified as a 3D pose and depth relative to a camera.
• a grasp planning method can sample antipodal grasp candidates and rank them with a GQ-CNN.
• An enormous dataset can be used to provide the data to train a neural network to develop highly reliable robot grasping across a wide variety of rigid objects.
• the dataset can consist of up to or over 6.7 million point object point clouds, accompanying parallel-jaw gripper poses, along with a robustness estimate of how likely it is that the grasp will be able to lift and carry the object. This can be advantageously used in a robust robotic grasping system.
• Implementations of the disclosed technology can rely on a probabilistic model to generate synthetic point clouds, grasps, and grasp robustness labels from datasets of 3D object meshes using physics-based models of grasping, image rendering, and camera noise, thus leveraging cloud computing to rapidly generate a large training set for a CNN.
• Implementations can include a hybrid approach to machine learning that combines physics with Deep Learning, e.g., by combining a large dataset of 3D object shapes, a physics-based model of grasp mechanics, and sampling statistics to generate many (e.g., 6.7 million) training examples, and then using a Deep Learning network to learn a function that can rapidly find robust grasps when given a 3D sensor point cloud.
• the system can be trained on a very large set of examples of robust grasps, similar to recent results in computer vision and speech recognition.
• the grasp In situations where the CNN estimates the robustness to be high, the grasp generally works as expected. For example, in certain experiments with an ABB YuMi (i.e., a two-arm industrial robot), the planner was 93 percent successful in planning reliable grasps and was also able to successfully predict grasp robustness with 40 novel objects (including tricky things like a can opener and a washcloth) with just one false positive out of 69 predicted successes.
• ABB YuMi i.e., a two-arm industrial robot
• a robot may have a good idea as to when it will succeed, it may also be able to tell when it is likely to fail. In situations where the robot anticipates a failure, the robot could take appropriate action, e.g., by either poking the object to change its orientation or asking a human for help.
• Implementations may be compatible with virtually any 3D camera and parallel-jaw or suction-cup gripper, and may be used to choose a primary grasp axis for multi-fingered grippers.
• certain knowledge specific to the hardware setup such as the focal length and bounds on where the RGB-D sensor will be relative to the robot, the geometry of a parallel-jaw robot gripper (specified as CAD model), and a friction coefficient for the gripper, may be provided as input to generate a new training dataset specific to a given hardware setup.
• a GQ-CNN trained on the dataset may have successful performance.
• Implementations of the disclosed technology may include using robust analytic grasp metrics as supervision, using the gripper's distance from the camera in predictions, and performing extensive evaluations on a physical robot.
• Certain implementations may facilitate development of new architectures for predicting grasp robustness from point clouds, and also to encourage the benchmarking of new methods.
• Certain implementations may include automatically generating training datasets for robotic grasping that can be a useful resource to train deep neural networks for robot grasp planning across multiple different robots.
• a robot may be integrated with an artificial intelligence so that it can figure out how to robustly grip objects it has never seen before or otherwise encountered, for example.
• a robot may use a neural network and a sensor (e.g., a Microsoft Kinect 3D sensor) to see a new object and then determine a robust grasp for successfully grasping the object.
• a sensor e.g., a Microsoft Kinect 3D sensor
• Certain implementations may include household robots performing various chores such as vacuuming, doing dishes, and picking up clutter, for example. Such machines will frequently encounter new objects but, by teaching themselves, they can better adapt to their surroundings.
• robots may be communicating via the cloud, e.g., to share information amongst each other, rather than working and learning in isolation.
• a robot can distribute gained knowledge to other robots that are like it and, in certain embodiments, even entirely different kinds of robots.
• the Dexterity Network generally refers to a research project that includes code, datasets, and algorithms for generating datasets of synthetic point clouds, robot parallel-jaw grasps, and metrics of grasp robustness based on physics for up to or over thousands of 3D object models to train machine learning-based methods to plan robot grasps. Implementations may include developing highly reliable robot grasping across any of a wide variety of rigid objects such as tools, household items, packaged goods, and industrial parts.
• Dex-Net 1.0 may be used for learning predictors of grasp success for new 3D mesh models, e.g., to accelerate generation of new datasets.
• Dex-Net 2.0 may be used for learning Grasp Quality Convolutional Neural Network (GQ-CNN) models that predict the probability of success of candidate grasps on objects from point clouds.
• GQ-CNNs may be useful for quickly planning grasps that can lift and transport a wide variety of objects by a physical robot.
• Certain implementations may include an analytic suction grasp model and metric based on the set of wrenches at the contact interface between the suction cup and rigid object surface and the magnitude of contact wrenches that needed to resist external wrenches due to gravity under perturbations in object pose, center of mass, contact location, friction, and gravity.
• This metric can be used to generate a dataset of up to or over 6.7 million point clouds, suction grasps, and grasp robustness labels generated from up to or over 1,500 3D object models, and train a Grasp Quality Convolutional Neural Network (GQ-CNN) on this dataset to classify grasp robustness from point clouds.
• GQ-CNN Grasp Quality Convolutional Neural Network

Landscapes

• Engineering & Computer Science (AREA)
• Robotics (AREA)
• Mechanical Engineering (AREA)
• Physics & Mathematics (AREA)
• Automation & Control Theory (AREA)
• Orthopedic Medicine & Surgery (AREA)
• General Health & Medical Sciences (AREA)
• Health & Medical Sciences (AREA)
• Theoretical Computer Science (AREA)
• Multimedia (AREA)
• General Physics & Mathematics (AREA)
• Evolutionary Computation (AREA)
• Fuzzy Systems (AREA)
• Mathematical Physics (AREA)
• Software Systems (AREA)
• Artificial Intelligence (AREA)
• Image Analysis (AREA)
• Manipulator (AREA)

Priority Applications (6)

Applications Claiming Priority (2)

Publications (1)

Family

ID=65525862

Family Applications (1)

Country Status (7)

Cited By (8)

Families Citing this family (36)

Citations (5)

Family Cites Families (12)

• 2018
  • 2018-04-04 AU AU2018326171A patent/AU2018326171A1/en not_active Abandoned
  • 2018-04-04 CA CA3073516A patent/CA3073516A1/en active Pending
  • 2018-04-04 JP JP2020512553A patent/JP2020532440A/ja active Pending
  • 2018-04-04 EP EP18851995.3A patent/EP3676766A4/en not_active Withdrawn
  • 2018-04-04 US US16/642,840 patent/US11446816B2/en active Active
  • 2018-04-04 WO PCT/US2018/026122 patent/WO2019045779A1/en not_active Ceased
  • 2018-04-04 CN CN201880067484.8A patent/CN111226237A/zh active Pending

Patent Citations (5)

Non-Patent Citations (3)

Also Published As

Similar Documents

Legal Events