WO2025257594A1 - Vision system for random bin picking applications of unseen objects - Google Patents

Vision system for random bin picking applications of unseen objects

Info

Publication number
WO2025257594A1
WO2025257594A1 PCT/IB2024/055721 IB2024055721W WO2025257594A1 WO 2025257594 A1 WO2025257594 A1 WO 2025257594A1 IB 2024055721 W IB2024055721 W IB 2024055721W WO 2025257594 A1 WO2025257594 A1 WO 2025257594A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
pose estimation
objects
model
pose
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/IB2024/055721
Other languages
French (fr)
Inventor
Jianjun Wang
Haoyan LIU
Yi Chen
Biao Zhang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ABB Schweiz AG
Original Assignee
ABB Schweiz AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ABB Schweiz AG filed Critical ABB Schweiz AG
Priority to PCT/IB2024/055721 priority Critical patent/WO2025257594A1/en
Publication of WO2025257594A1 publication Critical patent/WO2025257594A1/en
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/0014—Image feed-back for automatic industrial control, e.g. robot with camera
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20112—Image segmentation details
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Definitions

  • the present disclosure relates to a method and system for determining a pose of an object that has not previously been observed during training by an artificial intelligence system for motion planning of a robot arm to grasp the object.
  • FIG. 1 Facilities which implement and utilize robotic systems for interacting with objects for further processing may require accurate pose detection of objects within a space.
  • manufacturers may use a number of robotic arms or manipulators to retrieve objects randomly positioned in a container to place it on a conveyer belt or perform subsequent processing such as assembly, blow off, wash, deburring, gauging, and inspection in a facility during manufacture to more efficiently carry out the manufacturing process.
  • the robotic systems may be required to identify particular parts or objects in a given space of a view in order to grasp the object or perform other operations.
  • Conventional methods of identifying objects in a space for robotic manipulation may include a three dimensional (3D) vison camera for predicting a pose of an object within a space.
  • An embodiment of the present disclosure provides a system including two image sensors, a robotic arm, a processor, and a memory including instructions that, when executed with the processor, cause the system to, at least receive an image of a plurality of objects randomly positioned in a bin captured by the two image sensors, the plurality of objects and the bin being a proximal distance from the two image sensors and the robotic arm, generate one or more candidate segmentation proposals using a generalized segmentation model that utilizes the image, a candidate segmentation proposal representing a potential object instance in the image and including a confidence score, select one or more top candidates from the one or more candidate segmentation proposals using associated confidence scores, generate a six degree -of-freedom (6D0F) pose estimation for an object of the plurality of objects using a generalized pose estimation model and the selected one or more top candidates that contain a grasp instance, implement a multi-view triangulation refinement algorithm to refine the 6D0F pose estimation for the object, and instruct the robotic arm to grasp the object based on the 6D0F
  • the instructions are configured to generate an object segmentation mask and a two dimensional (2D) bounding box for the object with a candidate segmentation proposal having an associated confidence score that exceeds a confidence score threshold.
  • the instructions are configured to segment and crop the object from the image using the object segmentation mask and the 2D bounding box.
  • the instructions are configured to generate, during an offline process, one or more sets of synthetic templates at a number of pre-defined viewpoints using a rendering application and a computer aided design (CAD) model of the plurality of objects prior to receiving the image or prior to instructing the robotic arm to grasp the object.
  • CAD computer aided design
  • the instructions are configured to predict a coarse 3D rotation of the object based on using a generalized pose classification model, offline rendered templates, and the segmented and cropped object from the image.
  • the instructions are configured to predict the refined 6DoF pose estimation of the object based on using a generalized pose refiner model using the 6DoF pose estimation, the rendering application, and the segmented and cropped object from the image.
  • the instructions are configured to cache predicted poses from a neural network model and refinement steps using a depth map or multi-view triangulation refinement for the object in the memory during runtime.
  • the instructions are configured to instruct the robot arm to grasp another object of the plurality of objects in the bin using cached pose predictions based at least in part on implementing a structural similarity index measure metric for the image subsequent to the robot arm grasping the object from the bin.
  • Another embodiment of the present disclosure provides a tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more processors, provide for segmenting objects for motion planning with a robotic arm by execution of the following steps: receiving, via two image sensors, an image of a plurality of objects randomly positioned in a bin captured by the two image sensors, the plurality of objects in the bin and the two image sensors being a proximal distance from the robotic arm, generating one or more candidate segmentation proposals using a generalized segmentation model that utilizes the image, a candidate segmentation proposal representing a potential object instance in the image and including a confidence score, selecting one or more top candidates from the one or more candidate segmentation proposals using associated confidence scores, generating a six degree-of-freedom (6DoF) pose estimation for an object of the plurality of objects using a generalized pose estimation model and the selected one or more top candidates that contain a grasp instance, implementing a multi-view triangulation refinement algorithm to refine the 6DoF pose estimation for the object,
  • the instructions upon being executed by the one or more processors, are further configured to execute the following steps: generating an object segmentation mask and a two dimensional (2D) bounding box for the object with a candidate segmentation proposal having an associated confidence score that exceeds a confidence score threshold.
  • instructions upon being executed by the one or more processors, are further configured to execute the following steps: segmenting and cropping the object from the image using the object segmentation mask and 2D bounding box.
  • the instructions, upon being executed by the one or more processors are further configured to execute the following steps: generating, during an offline process, one or more sets of synthetic templates at a number of pre-defined viewpoints using a rendering application and a computer aided design (CAD) model of the plurality of objects prior receiving the image or prior to instructing the robotic arm to grasp the object.
  • CAD computer aided design
  • the instructions, upon being executed by the one or more processors are further configured to execute the following steps: predicting a coarse 3D rotation of the object based on using a generalized pose classification model, offline rendered templates, and the segmented and cropped object from the image.
  • generating the 6DoF pose estimation for the object is further based on using the generalized pose classification model using the predicted coarse 3D rotation, the rendering application, and the segmented and cropped object from the image.
  • the instructions, upon being executed by the one or more processors are further configured to execute the following steps: predicting the refined 6DoF pose estimation of the object based on using a generalized pose refiner model using the 6DoF pose estimation, the rendering application, and the segmented and cropped object from the image.
  • the instructions upon being executed by the one or more processors, are further configured to execute the following steps: caching predicted poses from a neural network model and refinement steps using a depth map or multi-view triangulation refinement for the object in the memory during runtime.
  • FIG. 1 illustrates an example architecture of a random bin picking system with a robotic arm including a computer system, one or more image sensors, a robotic arm, and a plurality of objects within a bin according to embodiments described herein;
  • FIG. 2 illustrates an example of segmenting objects in a container (bin) and generating the corresponding 2D bounding boxes according to embodiments described herein;
  • FIG. 3 illustrates an example of the pose estimation process for segmented objects for motion planning with a robotic arm according to embodiments described herein;
  • FIG. 4 illustrates an example diagram for segmenting and localizing objects for motion planning with a robotic arm according to embodiments described herein;
  • FIG. 5 illustrates a flow chart for segmenting and predicting the 6DoF pose of objects for motion planning with a robotic arm according to embodiments described herein;
  • FIG. 6 illustrates a simplified block diagram of one or more devices or systems for segmenting objects for motion planning with a robotic arm according to embodiments described herein.
  • Embodiments of the present disclosure provide a method and system for segmenting objects for motion planning with a robotic arm associated with a vision-based pose predictor in random bin picking applications. While the present disclosure is described primarily in connection with machines, systems, or components operated in an industrial setting or environment, as would be recognized by a person of ordinary skill in the art, the disclosure is not so limited and inventive features apply to other components or systems of other environments.
  • High speed automated assembly lines involve robots that are responsible for bringing together different industrial parts to build an object.
  • Machine tending robots of highspeed assembly lines may include tool-tips that may be configured to pick up industrial parts (objects) from different bins and place the parts picked up on a conveyer belt for further processing.
  • further processing of industrial parts may include other robot arms picking up the part for assembly, blow off, wash, deburring, gauging, inspection, etc.
  • High speed automated assembly lines are generally used in assembling automobiles, airplanes, printed circuit boards, and the like. For the high speed automated assembly lines to function smoothly, the industrial parts that are randomly positioned in a bin may be required to be correctly picked up.
  • a perception system implemented by a computer system described herein may be required to identify the industrial parts that are randomly placed in the bin and a pose estimation model may precisely predict a six-degree-of-freedom (6DoF) pose in space of the industrial parts in order to correctly grasp the industrial parts using the robot arm and correctly place the industrial parts on a conveyer belt or perform some other subsequent processing.
  • 6DoF may refer to the process of determining the precise position and orientation of an object in a three-dimensional (3D) space. This may involve estimating three translational movements (e.g., forward/back, up/down, left/right) and three rotational movements (pitch, yaw, roll) of the object relative to a given reference frame.
  • This disclosure provides systems and methods for random bin picking applications using pre-trained foundation models that are generalized models to segment objects given an image and determine a pose estimation for an object that it is not been specifically trained for or seen during the training phase.
  • a computer aided design (CAD) for a previously unseen object may be used along with the model to generate a pose estimation (e.g., 6DoF pose) for the object.
  • CAD computer aided design
  • a pose prediction model can be designed with feature-based computer vision techniques, point cloud or depth based registration or pose search techniques, and learningbased artificial intelligence (Al) techniques.
  • learning-based Al techniques require a long training phase that results in a pose estimation model that is only trained to estimate a pose for a specific object or part, not a randomly provided object or part.
  • the training phase contributes a major lead-time before the deployment of Al models.
  • the available computation power plays a role in determining the training time.
  • Generalized foundation Al models provide off-the-shelf solutions to classification, segmentation, and pose estimation problems. However, the inference time is usually higher, and performance are slightly lower compared to their object-specific Al model counterparts. This disclosure improves the pose estimation accuracy using generalized foundation models for object pose estimation and shorten the inference speed.
  • FIG. 1 illustrates an example architecture of a system 100 for segmenting objects for motion planning with a robotic arm including a computer system 102, one or more image sensors 104, a robotic arm 106, and a plurality of objects 108 randomly positioned within a bin 110 according to embodiments described herein.
  • the one or more image sensors 104 e.g., mono color cameras or color cameras
  • the dashed lines in FIG. 1 may represent the capture range or area for each image sensor of the image sensors 104.
  • the image sensors 104 may capture a set of images that are related to real-world random bin scenes.
  • a bin 110 may include a collection of objects 108 such as industrial parts that are randomly distributed throughout the bin 110.
  • the image sensors 104 may be used to photograph the bin 110.
  • the image sensors 104 may include a monochrome camera and/or a color camera for taking pictures of the bin 110 that is filled with randomly distributed objects 108.
  • the image sensors 104 may include multiple cameras arranged at different angles to capture pictures of an object(s) 108 in the bin 110 from different angles.
  • the computer system 102 may implement an object detection model which may be further implemented using a neural network for identifying individual objects 108 from the objects randomly distributed in the bin 110.
  • the objects 108 that are identified in the bin 110 may then be provided as input pose estimation algorithm implemented by the computer system 102.
  • the computer system 102 may implement a generalized foundation segmentation model for generating segmentation masks and 2D (two dimensional) bounding boxes of objects 108 in the bin 110.
  • the computer system 102 may implement a generalized, pre-trained pose estimation algorithm for generating an initial pose estimation for the detected objects 108, which may include performing a 6DoF pose estimation process on the detected individual objects 108.
  • the 6DoF pose estimation process may be refined using deep learning methods, iteratively closet point (ICP) registration, and/or multi-view geometry algorithms.
  • the output of the pose estimation algorithm may be in the form of a six degrees of freedom (6DoF) pose of a given object 108.
  • the computer system 102 may determine a pose estimation for an object 108 using an Al-based deep neural network foundation model.
  • the computer system 102 may use CAD models of the object to determine a pose estimation for an object that has not been seen before (e.g., an object that is not included in the training dataset for the foundation model) using a render and compare process at runtime.
  • the computer system 102 may train the foundation model on a large and diverse dataset of synthetic images containing thousands of different three dimensional (3D) models. This may expose the network of the model to a wide variety of shapes, textures, symmetries, etc., and help the model to learn generalized features and achieve generalized 6D pose estimation.
  • the computer system 102 may use the CAD model of the objects 108 to offline render (render during an offline process) hundreds of templates using calibrated camera intrinsics and pre-defined viewpoints.
  • top kl coarse 3D rotation proposals may be selected among hundreds of offline rendered templates of a given object 108 at predefined viewpoints through a classification neural network as an input for estimating the coarse 6DoF pose.
  • top k2 coarse 6DoF pose proposals may be selected among a limited number of online rendered images of a given object 108 at sampled viewpoints using the estimated 3D rotation through a classification task as an input for refining a pose estimation in an implemented neural network pipeline.
  • the computer system 102 may use multiple rendered views of the object’s 108 CAD model as an input to a refinement network.
  • the CAD model may provide information about the object’s 108 shape and coordinate frame to the network of the model without needing to encode this in the model’s weights during training.
  • the computer system 102 may render views centered on a consistent anchor point. This allows the network of the model to infer the anchor point location at test time on novel objects, such as never before seen objects 108.
  • the process described herein provides sufficient information to a model about novel objects via rendering thereby removing the need to encode object specifics such as weights during training while also leveraging extensive training on diverse synthetic data. This novel combination of techniques allows a foundation pose estimation model implemented by the computer system 102 to estimate accurate 6D poses for objects such as objects 108 that have never been seen before with real-time inference speed by the computer system 102.
  • the present disclosure solves problems such as long runtime inference speed of the render and compare pose estimation foundational model that uses a large number of rendered images as well as lowered pose estimation accuracy using such models in random bin picking applications.
  • All necessary images of the objects 108 may be pre-rendered as the camera intrinsics, field of view, and range of the image sensors 104 to the objects 108 and/or bin 110 are known.
  • the rendered images may be stored locally on the computer system 102 for pose estimation during inference.
  • the computer system 102 may cache the predicted pose of an object 108 for reuse if a local scene around the object does not change from a previous cycle (e.g., no need to re-compute the pose if the object does not move or change its location or alter the location or position of other objects 108 in the bin 110 subsequent to a grasp action by the robotic arm 106).
  • the estimated 6DoF pose from foundation pose estimation models in each camera view are further refined using multi-view triangulation refinement algorithm to improve the accuracy.
  • the computer system 102 implements a part (object) detection phase that is required for the pose estimation foundation model to estimate a 6DoF pose for the object 108.
  • the computer system 102 may implement one or more object segmentation foundation models such as Segment Anything Model (SAM), faster SAM, and mobile SAM, etc.
  • SAM Segment Anything Model
  • the segmentation foundation model generates a set of candidate segmentation regions (proposals) from an input image.
  • the computer system 102 may rank the segmentation proposals by confidence scores and a number of segmentation proposals are used for further process.
  • the computer system 102 may segment and crop the representation of the particular object 108 from the image and use the segmented and cropped object as an input for a pre-trained pose estimation foundation model.
  • the pre-trained pose estimation foundation model may be trained with a large dataset that includes thousands of CAD models that are similar to objects 108.
  • the computer system 102 may implement the pre-trained pose estimation foundation model and generate a pose estimation for an object of the objects 108 within the bin 110 with a highest confidence score as described above.
  • FIG. 3 illustrates an example of generalized pose estimation pipeline for segmented objects for motion planning with a robotic arm according to embodiments described herein.
  • the image crop 302 is generated through the process with reference to FIG.
  • the generalized pose estimation pipeline 300 includes three steps, a 3D coarse rotation estimator 304, a 6DoF coarse pose estimator 306, and a 6DoF pose refiner 308 implemented by a computer system.
  • the generalized pose estimation pipeline 300 implements a render and compare methodology.
  • a set of synthetic images each with a different pose using pre-defined viewpoints are offline rendered via a rendering application before grasp activities and depicted in FIG. 3 as pose hypotheses 310.
  • the computer system may render the different poses 310 of the object in the input crop 302 using known camera intrinsics of the image sensors capturing images of objects.
  • a classifier neural network is used to estimate the coarse 3D rotation of the object by comparing the image crop 302 with the pre-rendered templates 310.
  • the classifier neural network outputs a score that indicates the rotational similarity between the image crop 302 and pre-rendered template 310.
  • Templates describing coarse 3D rotation proposals with a highest score are visualized in 312.
  • Top kl coarse 3D rotation proposals 314 are selected based on the score and used to estimate the 6DoF coarse pose of the object. Additional poses can be sampled using the top kl coarse 3D rotation proposals 314.
  • Based on the estimated coarse 3D rotation proposals, 2D bounding box with reference to FIG. 1-2 and CAD model of the object a limited number of templates 316 are rendered online and incorporate the depth scale information.
  • the coarse pose estimator 306 uses a classifier to compare the image crop 302 and the rendered templates 316 to generate coarse 6D0F pose proposals 318.
  • Top k2 pose proposals 320 can be selected from 318 and used as input to the 6DoF pose refiner network 308 along with the image crop 302.
  • the pose refiner network 308 compares the image crop 302 with a limited number of online multi-view renderings 322 of each coarse pose proposals 320 to refine a 6DoF pose 324.
  • the refined pose estimation 324 may be generated in an iterative process where each iteration gives a AT (representing a 2D image space translation and depth scale update) and a AR (representing a rotational increment). This additional information is provided to the refiner model of the refiner 308 implemented by the computer system to generate the refined pose estimate 324 given the input crop 302.
  • FIG. 4 illustrates an example diagram 400 for segmenting and estimating the 6DoF pose of objects for motion planning with a robotic arm according to embodiments described herein.
  • the diagram 400 includes capturing RGB images of an object(s) with image sensors at 402. As depicted in FIG. 1, image sensors 104 may capture color or mono color images from multiple angles of objects randomly positioned in a bin. The images may be transmitted to a computer system associated with the image sensors and robotic arm for interacting with the objects randomly positioned in the bin.
  • the diagram 400 includes, at 404, generating segmentation proposals of object(s) using a segmentation model, such as SAM, FastSAM, or MobileSAM, to generate a set of candidate segmentation regions (proposals) using the images 402.
  • a segmentation model such as SAM, FastSAM, or MobileSAM
  • the proposals 404 may represent potential object instances in the images 402 including both seen and unseen objects.
  • the proposals are further ranked by confidence score at step 406.
  • Top segmentation proposals are selected for further pose estimation at step 408.
  • the selected segmentation proposals are used to generate part segmentation masks and image crops for the pose estimation networks.
  • CAD mesh models of the objects are received at 412, and pre-rendered images 414 of the CAD mesh models are generated at pre-defined viewpoints and used to estimate a coarse 3D rotation in pose estimation pipeline 416 of the detected object in input images from sensors 402.
  • a 6DoF pose estimation is obtained using generalized pose estimation models and pre-rendered templates that containing the object at various viewpoints 414.
  • the pre-trained pose estimation foundation model 416 is described in more detail above with reference to FIGs. 1 and 3.
  • the pose estimation generated at step 416 is refined at 418 of diagram 400 using pose refinement in a multi-view vision system.
  • the multi-view vision system pose refinement may include using multi-view triangulation refinement (e.g., triangle refinement algorithm) described below.
  • the diagram 400 includes, at 420, proceeding to motion planning for grasping the object using the pose estimation refined at 418.
  • the process implemented by the computer system may proceed to motion planning 420 without using a refinement process 418.
  • the motion planning for grasping the object 420 may include a computer system instructing a robotic arm to grasp the object as well as perform other operations such as moving the object to another conveyer belt for further processing.
  • further processing of objects may include other robot arms picking up the object for assembly, blow off, wash, deburring, gauging, inspection, etc.
  • FIG. 5 illustrates a flow chart for segmenting and predicting the 6DoF pose of objects for motion planning with a robotic arm according to embodiments described herein.
  • FIG. 5 includes exemplary process 500 which may be performed by an environment or architecture such as depicted in FIGs. 1-4 and 6 and by systems and components of FIGs. 1-4 and 6.
  • the computing system 102 of FIG. 1 performs the process 500.
  • any of the following blocks may be performed in any suitable order and that the process 500 may be performed in any environment or architecture and by any suitable computing device and/or controller of robotic arms, sensors, image capture sensors, and systems.
  • the process 500 includes receiving an image of a plurality of objects randomly positioned in a bin captured by two image sensors.
  • the plurality of objects and the bin may be a proximal distance from the two image sensors and a robotic arm.
  • the process 500 includes generating one or more candidate segmentation proposals using a generalized segmentation model that uses the image at 504.
  • a candidate segmentation proposal may represent a potential object instance in the image.
  • the process 500 includes selecting one or more top candidates from segmentation proposals using the associated confidence scores.
  • the process 500 includes generating a 6DoF pose estimation for an object of the plurality of objects using a generalized pose estimation model given the selected segmentation proposals that contain a grasp instance at 508.
  • the generalized 6DoF pose estimation model utilizes hundreds of offline synthetic renderings of the plurality of objects at pre-defined viewpoints using a rendering application that uses CAD models of the plurality of objects.
  • the CAD models for the plurality of objects may have been previously received but the trained pose estimation model for generating a pose estimation has not been trained with images of the plurality of objects.
  • the process 500 includes applying (implementing by the computer system) a multi-view triangulation refinement algorithm to refine the estimated 6DoF pose for an object of the plurality of objects.
  • the process 500 includes instructing the robot arm to grasp the object based on the refined 6DoF pose estimation.
  • a set of 3D points are pre-defined or sampled from a CAD mesh model of each detected mechanical part (object) in the bin.
  • the 3D points of the CAD mesh model are converted to 2-dimensional points in an image space.
  • 2-dimensional (2D) points pixel coordinates
  • a multi-view triangulation is applied to each point in the set of sampled points.
  • the 3D coordinates in the camera view of the selected points are computed.
  • the refined pose of the target part is easily computed with the 3D coordinates in the camera view using a rigid body constraint.
  • the camera view may include the view obtained from the captured images.
  • the computer system may instruct the robot arm to grasp another object of the plurality of objects in the bin based on implementing a structural similarity index measure for the image subsequent to the robot arm grasping the initial object from the bin. For example, the computer system may use a subsequent image of the bin of the plurality of objects after an initial object has been grasped and use the structural similarity index measure between the images to determine if the images are similar. The more similar the images are the more likely that an initial grasp action by the robotic arm of an object in the bin has not shifted, moved, or otherwise altered other objects in the bin.
  • a Structural Similarity Index Measure is a metric used to measure the perceived quality and similarity between two images. It may assess a visual impact of changes in luminance, contrast, and structure between images. SSIM may provide a value between -1 and 1, where 1 indicates perfect similarity.
  • FIG. 6 illustrates a simplified block diagram of one or more devices or systems for segmenting objects for motion planning with a robotic arm according to embodiments described herein.
  • FIG. 6 is a block diagram of an exemplary system or device 600 including a computer, computer device, server, or controller for segmenting objects for motion planning with a robotic arm for picking, moving, or otherwise interacting with objects randomly positioned in a bin as described herein with reference to FIGs. 1-5.
  • the system 600 includes a processor 604, such as a central processing unit (CPU), and/or logic, that executes computer executable instructions for performing the functions, processes, and/or methods described herein.
  • CPU central processing unit
  • the computer executable instructions are locally stored and accessed from a non-transitory computer readable medium, such as storage 610, which may be a hard drive or flash drive.
  • Read Only Memory (ROM) 606 includes computer executable instructions for initializing the processor 604, while the random-access memory (RAM) 608 is the main memory for loading and processing instructions executed by the processor 604.
  • the network interface 612 may connect to a wired network or cellular network and to a local area network or wide area network or BLUETOOTH or other suitable communication methods such as those described herein with reference to the communication component or communication module.
  • the system 600 may also include a bus 602 that connects the processor 604, ROM 606, RAM 608, storage 610, and/or the network interface 612.
  • the components within the system 600 may use the bus 602 to communicate with each other.
  • the components within the system 600 are merely exemplary and might not be inclusive of every component for embodiments described herein.
  • the system 600 might not include a network interface 612.
  • the system 600 may include one or more components for interacting with a machine or system such as image sensors (cameras) or robotic arms.
  • the system 600 may implement a number of algorithms and/or applications described herein for segmenting objects for motion planning with a robotic arm including a segmentation model, feature extraction model, rendering application or rendering engine, matching algorithm, a pose estimation algorithm (generalized pose estimation algorithm), a multi-view triangulation refinement algorithm, or one or more neural network models.
  • the recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise.
  • the recitation of “A, B and/or C” or “at least one of A, B or C” should be interpreted as including any singular entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B and C.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Robotics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

Images of a plurality of objects in a bin from multiple views are received from two image sensors. One or more candidate segmentation proposals are generated using a generalized segmentation model that uses the image. Top candidates are selected from the one or more candidate segmentation proposals using associated confidence scores. A six-degree of freedom (6DoF) pose estimation for an object is generated using a generalized pose estimation model and the selected one or more top candidates that contain a grasp instance. The 6DoF pose estimation is refined by implementing a multi-view triangulation refinement algorithm. A robotic arm grasps the object based on the refined 6DoF pose estimation.

Description

VISION SYSTEM FOR RANDOM BIN
PICKING APPLICATIONS OF UNSEEN OBJECTS
FIELD
[0001] The present disclosure relates to a method and system for determining a pose of an object that has not previously been observed during training by an artificial intelligence system for motion planning of a robot arm to grasp the object.
BACKGROUND
[0002] Facilities which implement and utilize robotic systems for interacting with objects for further processing may require accurate pose detection of objects within a space. For example, manufacturers may use a number of robotic arms or manipulators to retrieve objects randomly positioned in a container to place it on a conveyer belt or perform subsequent processing such as assembly, blow off, wash, deburring, gauging, and inspection in a facility during manufacture to more efficiently carry out the manufacturing process. The robotic systems may be required to identify particular parts or objects in a given space of a view in order to grasp the object or perform other operations. Conventional methods of identifying objects in a space for robotic manipulation may include a three dimensional (3D) vison camera for predicting a pose of an object within a space. However, such camera system rigs may require a large footprint within a facility to properly implement and are expensive. Other conventional systems implementing object pose estimation using a single camera have limited pose prediction accuracy. Those systems may require an object-specific artificial neural network model to estimate a pose of the object in the container, which requires a training lead-time before deployment.
SUMMARY
[0003] An embodiment of the present disclosure provides a system including two image sensors, a robotic arm, a processor, and a memory including instructions that, when executed with the processor, cause the system to, at least receive an image of a plurality of objects randomly positioned in a bin captured by the two image sensors, the plurality of objects and the bin being a proximal distance from the two image sensors and the robotic arm, generate one or more candidate segmentation proposals using a generalized segmentation model that utilizes the image, a candidate segmentation proposal representing a potential object instance in the image and including a confidence score, select one or more top candidates from the one or more candidate segmentation proposals using associated confidence scores, generate a six degree -of-freedom (6D0F) pose estimation for an object of the plurality of objects using a generalized pose estimation model and the selected one or more top candidates that contain a grasp instance, implement a multi-view triangulation refinement algorithm to refine the 6D0F pose estimation for the object, and instruct the robotic arm to grasp the object based on the 6D0F pose estimation.
[0004] In an embodiment of the system, the instructions are configured to generate an object segmentation mask and a two dimensional (2D) bounding box for the object with a candidate segmentation proposal having an associated confidence score that exceeds a confidence score threshold.
[0005] In an embodiment of the system, the instructions are configured to segment and crop the object from the image using the object segmentation mask and the 2D bounding box. [0006] In an embodiment of the system, the instructions are configured to generate, during an offline process, one or more sets of synthetic templates at a number of pre-defined viewpoints using a rendering application and a computer aided design (CAD) model of the plurality of objects prior to receiving the image or prior to instructing the robotic arm to grasp the object.
[0007] In an embodiment of the system, the instructions are configured to predict a coarse 3D rotation of the object based on using a generalized pose classification model, offline rendered templates, and the segmented and cropped object from the image.
[0008] In an embodiment of the system, generating the 6DoF pose estimation for the object is further based on using the generalized pose classification model using the predicted coarse 3D rotation, the rendering application, and the segmented and cropped object from the image.
[0009] In an embodiment of the system, the instructions are configured to predict the refined 6DoF pose estimation of the object based on using a generalized pose refiner model using the 6DoF pose estimation, the rendering application, and the segmented and cropped object from the image.
[0010] In an embodiment of the system, further comprising a depth sensor.
[0011] In an embodiment of the system, the instructions are configured to refine the
6DoF pose estimation for the object using depth information of the object captured by the depth sensor. [0012] In an embodiment of the system, the instructions are configured to cache predicted poses from a neural network model and refinement steps using a depth map or multi-view triangulation refinement for the object in the memory during runtime.
[0013] In an embodiment of the system, the instructions are configured to instruct the robot arm to grasp another object of the plurality of objects in the bin using cached pose predictions based at least in part on implementing a structural similarity index measure metric for the image subsequent to the robot arm grasping the object from the bin.
[0014] Another embodiment of the present disclosure provides a tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more processors, provide for segmenting objects for motion planning with a robotic arm by execution of the following steps: receiving, via two image sensors, an image of a plurality of objects randomly positioned in a bin captured by the two image sensors, the plurality of objects in the bin and the two image sensors being a proximal distance from the robotic arm, generating one or more candidate segmentation proposals using a generalized segmentation model that utilizes the image, a candidate segmentation proposal representing a potential object instance in the image and including a confidence score, selecting one or more top candidates from the one or more candidate segmentation proposals using associated confidence scores, generating a six degree-of-freedom (6DoF) pose estimation for an object of the plurality of objects using a generalized pose estimation model and the selected one or more top candidates that contain a grasp instance, implementing a multi-view triangulation refinement algorithm to refine the 6DoF pose estimation for the object, and instructing the robot arm to grasp the object based on the refined 6DoF pose estimation.
[0015] In an embodiment, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: generating an object segmentation mask and a two dimensional (2D) bounding box for the object with a candidate segmentation proposal having an associated confidence score that exceeds a confidence score threshold.
[0016] In an embodiment, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: segmenting and cropping the object from the image using the object segmentation mask and 2D bounding box.
[0017] In an embodiment, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: generating, during an offline process, one or more sets of synthetic templates at a number of pre-defined viewpoints using a rendering application and a computer aided design (CAD) model of the plurality of objects prior receiving the image or prior to instructing the robotic arm to grasp the object. [0018] In an embodiment, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: predicting a coarse 3D rotation of the object based on using a generalized pose classification model, offline rendered templates, and the segmented and cropped object from the image.
[0019] In an embodiment, wherein generating the 6DoF pose estimation for the object is further based on using the generalized pose classification model using the predicted coarse 3D rotation, the rendering application, and the segmented and cropped object from the image. [0020] In an embodiment, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: predicting the refined 6DoF pose estimation of the object based on using a generalized pose refiner model using the 6DoF pose estimation, the rendering application, and the segmented and cropped object from the image.
[0021] In an embodiment, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: caching predicted poses from a neural network model and refinement steps using a depth map or multi-view triangulation refinement for the object in the memory during runtime.
[0022] Another embodiment of the present disclosure provides a computer-implemented method for segmenting objects for motion planning with a robotic arm including receiving, via two image sensors, an image of a plurality of objects randomly positioned in a bin captured by the two image sensors, the plurality of objects in the bin and the two image sensors being a proximal distance from the robotic arm, generating one or more candidate segmentation proposals using a generalized segmentation model that utilizes the image, a candidate segmentation proposal representing a potential object instance in the image and including a confidence score, selecting one or more top candidates from the one or more candidate segmentation proposals using associated confidence scores, generating a six degree -of-freedom (6DoF) pose estimation for an object of the plurality of objects using a generalized pose estimation model and the selected one or more top candidates that contain a grasp instance, implementing a multi-view triangulation refinement algorithm to refine the 6DoF pose estimation for the object, and instructing the robotic arm to grasp the object based on the refined 6DoF pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The present disclosure will be described in even greater detail below based on the exemplary figures. The disclosure is not limited to the exemplary embodiments. All features described and/or illustrated herein can be used alone or combined in different combinations in embodiments of the disclosure. The features and advantages of various embodiments of the present disclosure will become apparent by reading the following detailed description with reference to the attached drawings which illustrate the following:
[0024] FIG. 1 illustrates an example architecture of a random bin picking system with a robotic arm including a computer system, one or more image sensors, a robotic arm, and a plurality of objects within a bin according to embodiments described herein;
[0025] FIG. 2 illustrates an example of segmenting objects in a container (bin) and generating the corresponding 2D bounding boxes according to embodiments described herein;
[0026] FIG. 3 illustrates an example of the pose estimation process for segmented objects for motion planning with a robotic arm according to embodiments described herein;
[0027] FIG. 4 illustrates an example diagram for segmenting and localizing objects for motion planning with a robotic arm according to embodiments described herein;
[0028] FIG. 5 illustrates a flow chart for segmenting and predicting the 6DoF pose of objects for motion planning with a robotic arm according to embodiments described herein; and
[0029] FIG. 6 illustrates a simplified block diagram of one or more devices or systems for segmenting objects for motion planning with a robotic arm according to embodiments described herein.
DETAILED DESCRIPTION
[0030] Embodiments of the present disclosure provide a method and system for segmenting objects for motion planning with a robotic arm associated with a vision-based pose predictor in random bin picking applications. While the present disclosure is described primarily in connection with machines, systems, or components operated in an industrial setting or environment, as would be recognized by a person of ordinary skill in the art, the disclosure is not so limited and inventive features apply to other components or systems of other environments. [0031] High speed automated assembly lines involve robots that are responsible for bringing together different industrial parts to build an object. Machine tending robots of highspeed assembly lines may include tool-tips that may be configured to pick up industrial parts (objects) from different bins and place the parts picked up on a conveyer belt for further processing. In some embodiments, further processing of industrial parts may include other robot arms picking up the part for assembly, blow off, wash, deburring, gauging, inspection, etc. High speed automated assembly lines are generally used in assembling automobiles, airplanes, printed circuit boards, and the like. For the high speed automated assembly lines to function smoothly, the industrial parts that are randomly positioned in a bin may be required to be correctly picked up. In order for the industrial parts to be picked up in the correct manner, a perception system implemented by a computer system described herein may be required to identify the industrial parts that are randomly placed in the bin and a pose estimation model may precisely predict a six-degree-of-freedom (6DoF) pose in space of the industrial parts in order to correctly grasp the industrial parts using the robot arm and correctly place the industrial parts on a conveyer belt or perform some other subsequent processing. 6DoF may refer to the process of determining the precise position and orientation of an object in a three-dimensional (3D) space. This may involve estimating three translational movements (e.g., forward/back, up/down, left/right) and three rotational movements (pitch, yaw, roll) of the object relative to a given reference frame. This disclosure provides systems and methods for random bin picking applications using pre-trained foundation models that are generalized models to segment objects given an image and determine a pose estimation for an object that it is not been specifically trained for or seen during the training phase. In embodiments, a computer aided design (CAD) for a previously unseen object may be used along with the model to generate a pose estimation (e.g., 6DoF pose) for the object.
[0032] A pose prediction model can be designed with feature-based computer vision techniques, point cloud or depth based registration or pose search techniques, and learningbased artificial intelligence (Al) techniques. Commonly, learning-based Al techniques require a long training phase that results in a pose estimation model that is only trained to estimate a pose for a specific object or part, not a randomly provided object or part. The training phase contributes a major lead-time before the deployment of Al models. The available computation power plays a role in determining the training time. Generalized foundation Al models provide off-the-shelf solutions to classification, segmentation, and pose estimation problems. However, the inference time is usually higher, and performance are slightly lower compared to their object-specific Al model counterparts. This disclosure improves the pose estimation accuracy using generalized foundation models for object pose estimation and shorten the inference speed.
[0033] Embodiments of the present disclosure describe a process to segment objects that are previously unseen for a pose estimation using a vision-based pose prediction systems in robotic-arm-based random bin picking applications. Accurately detecting and segmenting objects as part of a pose estimation process for a previously unseen object is an important step to efficiently generate pose predictions without spending an inordinate amount of time training specific models in highly versatile industrial environment. With appropriate pose estimation for previously unseen objects, robotic machinery may be used to automatically pick up the objects in the proper manner so that they can be efficiently used in further applications such as assembly.
[0034] FIG. 1 illustrates an example architecture of a system 100 for segmenting objects for motion planning with a robotic arm including a computer system 102, one or more image sensors 104, a robotic arm 106, and a plurality of objects 108 randomly positioned within a bin 110 according to embodiments described herein. In embodiments, the one or more image sensors 104 (e.g., mono color cameras or color cameras) may capture one or more images of the objects 108 within bin 110. The dashed lines in FIG. 1 may represent the capture range or area for each image sensor of the image sensors 104. The image sensors 104 may be in communication with the computer system 102 via an Internet connection, BLUETOOTH, WIFI, near field communication (NFC), or other suitable communication methods for transmitting the captured images to the computer system 102. The image sensors 104 may be within a proximal range of the objects 108 and bin 110 that corresponds to the operating range of the sensors or cameras that can vary depending on the type of sensor or camera included in the system 100.
[0035] The image sensors 104 may capture a set of images that are related to real-world random bin scenes. In some embodiments, a bin 110 may include a collection of objects 108 such as industrial parts that are randomly distributed throughout the bin 110. The image sensors 104 may be used to photograph the bin 110. In some cases, the image sensors 104 may include a monochrome camera and/or a color camera for taking pictures of the bin 110 that is filled with randomly distributed objects 108. In some other cases, the image sensors 104 may include multiple cameras arranged at different angles to capture pictures of an object(s) 108 in the bin 110 from different angles. The computer system 102 may implement an object detection model which may be further implemented using a neural network for identifying individual objects 108 from the objects randomly distributed in the bin 110. The objects 108 that are identified in the bin 110 may then be provided as input pose estimation algorithm implemented by the computer system 102. The computer system 102 may implement a generalized foundation segmentation model for generating segmentation masks and 2D (two dimensional) bounding boxes of objects 108 in the bin 110. The computer system 102 may implement a generalized, pre-trained pose estimation algorithm for generating an initial pose estimation for the detected objects 108, which may include performing a 6DoF pose estimation process on the detected individual objects 108. In some embodiments, the 6DoF pose estimation process may be refined using deep learning methods, iteratively closet point (ICP) registration, and/or multi-view geometry algorithms. The output of the pose estimation algorithm may be in the form of a six degrees of freedom (6DoF) pose of a given object 108.
[0036] In an embodiment, the computer system 102 may determine a pose estimation for an object 108 using an Al-based deep neural network foundation model. The computer system 102 may use CAD models of the object to determine a pose estimation for an object that has not been seen before (e.g., an object that is not included in the training dataset for the foundation model) using a render and compare process at runtime. To accomplish this the computer system 102 may train the foundation model on a large and diverse dataset of synthetic images containing thousands of different three dimensional (3D) models. This may expose the network of the model to a wide variety of shapes, textures, symmetries, etc., and help the model to learn generalized features and achieve generalized 6D pose estimation. In embodiments, the computer system 102 may use the CAD model of the objects 108 to offline render (render during an offline process) hundreds of templates using calibrated camera intrinsics and pre-defined viewpoints. In embodiments, top kl coarse 3D rotation proposals may be selected among hundreds of offline rendered templates of a given object 108 at predefined viewpoints through a classification neural network as an input for estimating the coarse 6DoF pose. In embodiments, top k2 coarse 6DoF pose proposals may be selected among a limited number of online rendered images of a given object 108 at sampled viewpoints using the estimated 3D rotation through a classification task as an input for refining a pose estimation in an implemented neural network pipeline.
[0037] The computer system 102 may use multiple rendered views of the object’s 108 CAD model as an input to a refinement network. The CAD model may provide information about the object’s 108 shape and coordinate frame to the network of the model without needing to encode this in the model’s weights during training. In embodiments, the computer system 102 may render views centered on a consistent anchor point. This allows the network of the model to infer the anchor point location at test time on novel objects, such as never before seen objects 108. The process described herein provides sufficient information to a model about novel objects via rendering thereby removing the need to encode object specifics such as weights during training while also leveraging extensive training on diverse synthetic data. This novel combination of techniques allows a foundation pose estimation model implemented by the computer system 102 to estimate accurate 6D poses for objects such as objects 108 that have never been seen before with real-time inference speed by the computer system 102.
[0038] The present disclosure solves problems such as long runtime inference speed of the render and compare pose estimation foundational model that uses a large number of rendered images as well as lowered pose estimation accuracy using such models in random bin picking applications. All necessary images of the objects 108 may be pre-rendered as the camera intrinsics, field of view, and range of the image sensors 104 to the objects 108 and/or bin 110 are known. The rendered images may be stored locally on the computer system 102 for pose estimation during inference. In embodiments, the computer system 102 may cache the predicted pose of an object 108 for reuse if a local scene around the object does not change from a previous cycle (e.g., no need to re-compute the pose if the object does not move or change its location or alter the location or position of other objects 108 in the bin 110 subsequent to a grasp action by the robotic arm 106). The estimated 6DoF pose from foundation pose estimation models in each camera view are further refined using multi-view triangulation refinement algorithm to improve the accuracy.
[0039] In an embodiment, the computer system 102 implements a part (object) detection phase that is required for the pose estimation foundation model to estimate a 6DoF pose for the object 108. Without training the pose estimation algorithm (model) on the unseen parts, such as objects 108, the computer system 102 may implement one or more object segmentation foundation models such as Segment Anything Model (SAM), faster SAM, and mobile SAM, etc. During an inference stage, the segmentation foundation model generates a set of candidate segmentation regions (proposals) from an input image.
[0040] In embodiments, the computer system 102 may rank the segmentation proposals by confidence scores and a number of segmentation proposals are used for further process. The computer system 102 may segment and crop the representation of the particular object 108 from the image and use the segmented and cropped object as an input for a pre-trained pose estimation foundation model. In embodiments, the pre-trained pose estimation foundation model may be trained with a large dataset that includes thousands of CAD models that are similar to objects 108. In embodiments, the computer system 102 may implement the pre-trained pose estimation foundation model and generate a pose estimation for an object of the objects 108 within the bin 110 with a highest confidence score as described above.
[0041] FIG. 2 illustrates an example of segmenting objects for further pose estimation and motion planning with a robotic arm according to embodiments described herein. The input image 202 is captured by an image sensor with reference to FIG. 1. The captured image of a bin with objects is the input to a generalized segmentation model 204 (e.g. FastSAM, MobileSAM, etc.). The outputs of such generalized segmentation models 204 include segmentation proposals 206 and corresponding confidence scores. The segmentation proposals may be further ranked by the confidence score and selected for generating the 2D bounding boxes and image crops that each contain a detected object.
[0042] FIG. 3 illustrates an example of generalized pose estimation pipeline for segmented objects for motion planning with a robotic arm according to embodiments described herein. The image crop 302 is generated through the process with reference to FIG.
2 and is the input to the generalized pose estimation pipeline 300. The generalized pose estimation pipeline 300 includes three steps, a 3D coarse rotation estimator 304, a 6DoF coarse pose estimator 306, and a 6DoF pose refiner 308 implemented by a computer system. To estimate the 6DoF pose of the object, the generalized pose estimation pipeline 300 implements a render and compare methodology. A set of synthetic images each with a different pose using pre-defined viewpoints are offline rendered via a rendering application before grasp activities and depicted in FIG. 3 as pose hypotheses 310. In embodiments, the computer system may render the different poses 310 of the object in the input crop 302 using known camera intrinsics of the image sensors capturing images of objects. A classifier neural network is used to estimate the coarse 3D rotation of the object by comparing the image crop 302 with the pre-rendered templates 310. The classifier neural network outputs a score that indicates the rotational similarity between the image crop 302 and pre-rendered template 310. Templates describing coarse 3D rotation proposals with a highest score are visualized in 312. Top kl coarse 3D rotation proposals 314 are selected based on the score and used to estimate the 6DoF coarse pose of the object. Additional poses can be sampled using the top kl coarse 3D rotation proposals 314. Based on the estimated coarse 3D rotation proposals, 2D bounding box with reference to FIG. 1-2 and CAD model of the object, a limited number of templates 316 are rendered online and incorporate the depth scale information. The coarse pose estimator 306 uses a classifier to compare the image crop 302 and the rendered templates 316 to generate coarse 6D0F pose proposals 318. Top k2 pose proposals 320 can be selected from 318 and used as input to the 6DoF pose refiner network 308 along with the image crop 302. The pose refiner network 308 compares the image crop 302 with a limited number of online multi-view renderings 322 of each coarse pose proposals 320 to refine a 6DoF pose 324. The refined pose estimation 324 may be generated in an iterative process where each iteration gives a AT (representing a 2D image space translation and depth scale update) and a AR (representing a rotational increment). This additional information is provided to the refiner model of the refiner 308 implemented by the computer system to generate the refined pose estimate 324 given the input crop 302.
[0043] FIG. 4 illustrates an example diagram 400 for segmenting and estimating the 6DoF pose of objects for motion planning with a robotic arm according to embodiments described herein. The diagram 400 includes capturing RGB images of an object(s) with image sensors at 402. As depicted in FIG. 1, image sensors 104 may capture color or mono color images from multiple angles of objects randomly positioned in a bin. The images may be transmitted to a computer system associated with the image sensors and robotic arm for interacting with the objects randomly positioned in the bin. The diagram 400 includes, at 404, generating segmentation proposals of object(s) using a segmentation model, such as SAM, FastSAM, or MobileSAM, to generate a set of candidate segmentation regions (proposals) using the images 402. The proposals 404 may represent potential object instances in the images 402 including both seen and unseen objects. The proposals are further ranked by confidence score at step 406. Top segmentation proposals are selected for further pose estimation at step 408. At step 410, the selected segmentation proposals are used to generate part segmentation masks and image crops for the pose estimation networks. As described above, with reference to FIGs. 1 and 3 during an onboarding phase, which may be offline, CAD mesh models of the objects are received at 412, and pre-rendered images 414 of the CAD mesh models are generated at pre-defined viewpoints and used to estimate a coarse 3D rotation in pose estimation pipeline 416 of the detected object in input images from sensors 402. At step 416, with the cropped images that containing a detected object 410, a 6DoF pose estimation is obtained using generalized pose estimation models and pre-rendered templates that containing the object at various viewpoints 414. The pre-trained pose estimation foundation model 416 is described in more detail above with reference to FIGs. 1 and 3.
[0044] The pose estimation generated at step 416 is refined at 418 of diagram 400 using pose refinement in a multi-view vision system. For example, the multi-view vision system pose refinement may include using multi-view triangulation refinement (e.g., triangle refinement algorithm) described below. Once a pose estimation is generated at 418 the diagram 400 includes, at 420, proceeding to motion planning for grasping the object using the pose estimation refined at 418. In embodiments, the process implemented by the computer system may proceed to motion planning 420 without using a refinement process 418. The motion planning for grasping the object 420 may include a computer system instructing a robotic arm to grasp the object as well as perform other operations such as moving the object to another conveyer belt for further processing. In some embodiments, further processing of objects may include other robot arms picking up the object for assembly, blow off, wash, deburring, gauging, inspection, etc.
[0045] FIG. 5 illustrates a flow chart for segmenting and predicting the 6DoF pose of objects for motion planning with a robotic arm according to embodiments described herein. FIG. 5 includes exemplary process 500 which may be performed by an environment or architecture such as depicted in FIGs. 1-4 and 6 and by systems and components of FIGs. 1-4 and 6. As will be described below, the computing system 102 of FIG. 1 performs the process 500. However, it will be recognized that any of the following blocks may be performed in any suitable order and that the process 500 may be performed in any environment or architecture and by any suitable computing device and/or controller of robotic arms, sensors, image capture sensors, and systems.
[0046] At step 502, the process 500 includes receiving an image of a plurality of objects randomly positioned in a bin captured by two image sensors. In embodiments, the plurality of objects and the bin may be a proximal distance from the two image sensors and a robotic arm. The process 500 includes generating one or more candidate segmentation proposals using a generalized segmentation model that uses the image at 504. A candidate segmentation proposal may represent a potential object instance in the image. At step 506, the process 500 includes selecting one or more top candidates from segmentation proposals using the associated confidence scores. The process 500 includes generating a 6DoF pose estimation for an object of the plurality of objects using a generalized pose estimation model given the selected segmentation proposals that contain a grasp instance at 508. In some embodiment, the generalized 6DoF pose estimation model utilizes hundreds of offline synthetic renderings of the plurality of objects at pre-defined viewpoints using a rendering application that uses CAD models of the plurality of objects. The CAD models for the plurality of objects may have been previously received but the trained pose estimation model for generating a pose estimation has not been trained with images of the plurality of objects. [0047] At step 510, the process 500 includes applying (implementing by the computer system) a multi-view triangulation refinement algorithm to refine the estimated 6DoF pose for an object of the plurality of objects. At step 512, the process 500 includes instructing the robot arm to grasp the object based on the refined 6DoF pose estimation.
[0048] In an embodiment, in order to implement multi-view triangulation refinement (e.g., triangle refinement algorithm), a set of 3D points are pre-defined or sampled from a CAD mesh model of each detected mechanical part (object) in the bin. The 3D points of the CAD mesh model are converted to 2-dimensional points in an image space. With the initial pose of the mechanical part in each view, 2-dimensional (2D) points (pixel coordinates) in an image space of the 3D points selected from the CAD model are computed via back reprojection. A multi-view triangulation is applied to each point in the set of sampled points. Using the multi-view triangulation refinement on the 2D points, the 3D coordinates in the camera view of the selected points are computed. The refined pose of the target part is easily computed with the 3D coordinates in the camera view using a rigid body constraint. The camera view may include the view obtained from the captured images. In embodiments, the computer system may instruct the robot arm to grasp another object of the plurality of objects in the bin based on implementing a structural similarity index measure for the image subsequent to the robot arm grasping the initial object from the bin. For example, the computer system may use a subsequent image of the bin of the plurality of objects after an initial object has been grasped and use the structural similarity index measure between the images to determine if the images are similar. The more similar the images are the more likely that an initial grasp action by the robotic arm of an object in the bin has not shifted, moved, or otherwise altered other objects in the bin. Thus, the computer system can instruct the robotic arm to grasp another object in the bin, perhaps at a similar level, depth, or layer within the bin, without needing to recalculate the pose estimation for the another object. However, if the structural similarity index measure between the images indicates a difference than the computer system may recalculate or determine an updated pose estimation for one or more objects in the bin subsequent to an initial grasp action by the robotic arm. In embodiments, a Structural Similarity Index Measure (SSIM) is a metric used to measure the perceived quality and similarity between two images. It may assess a visual impact of changes in luminance, contrast, and structure between images. SSIM may provide a value between -1 and 1, where 1 indicates perfect similarity.
[0049] FIG. 6 illustrates a simplified block diagram of one or more devices or systems for segmenting objects for motion planning with a robotic arm according to embodiments described herein. FIG. 6 is a block diagram of an exemplary system or device 600 including a computer, computer device, server, or controller for segmenting objects for motion planning with a robotic arm for picking, moving, or otherwise interacting with objects randomly positioned in a bin as described herein with reference to FIGs. 1-5. The system 600 includes a processor 604, such as a central processing unit (CPU), and/or logic, that executes computer executable instructions for performing the functions, processes, and/or methods described herein. In some examples, the computer executable instructions are locally stored and accessed from a non-transitory computer readable medium, such as storage 610, which may be a hard drive or flash drive. Read Only Memory (ROM) 606 includes computer executable instructions for initializing the processor 604, while the random-access memory (RAM) 608 is the main memory for loading and processing instructions executed by the processor 604. [0050] The network interface 612 may connect to a wired network or cellular network and to a local area network or wide area network or BLUETOOTH or other suitable communication methods such as those described herein with reference to the communication component or communication module. The system 600 may also include a bus 602 that connects the processor 604, ROM 606, RAM 608, storage 610, and/or the network interface 612. The components within the system 600 may use the bus 602 to communicate with each other. The components within the system 600 are merely exemplary and might not be inclusive of every component for embodiments described herein. For instance, in some examples, the system 600 might not include a network interface 612. In embodiments the system 600 may include one or more components for interacting with a machine or system such as image sensors (cameras) or robotic arms. The system 600 may implement a number of algorithms and/or applications described herein for segmenting objects for motion planning with a robotic arm including a segmentation model, feature extraction model, rendering application or rendering engine, matching algorithm, a pose estimation algorithm (generalized pose estimation algorithm), a multi-view triangulation refinement algorithm, or one or more neural network models.
[0051] While the disclosure has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. It will be understood that changes and modifications may be made by those of ordinary skill within the scope of the following claims. In particular, the present disclosure covers further embodiments with any combination of features from different embodiments described above and below. Additionally, statements made herein characterizing the disclosure refer to an embodiment of the disclosure and not necessarily all embodiments.
[0052] The terms used in the claims should be construed to have the broadest reasonable interpretation consistent with the foregoing description. For example, the use of the article “a” or “the” in introducing an element should not be interpreted as being exclusive of a plurality of elements. Likewise, the recitation of “or” should be interpreted as being inclusive, such that the recitation of “A or B” is not exclusive of “A and B,” unless it is clear from the context or the foregoing description that only one of A and B is intended. Further, the recitation of “at least one of A, B and C” should be interpreted as one or more of a group of elements consisting of A, B and C, and should not be interpreted as requiring at least one of each of the listed elements A, B and C, regardless of whether A, B and C are related as categories or otherwise. Moreover, the recitation of “A, B and/or C” or “at least one of A, B or C” should be interpreted as including any singular entity from the listed elements, e.g., A, any subset from the listed elements, e.g., A and B, or the entire list of elements A, B and C.

Claims

CLAIMS What is claimed is:
1. A system comprising: two image sensors; a robotic arm; a processor; and a memory including instructions that, when executed with the processor, cause the system to, at least: receive an image of a plurality of objects randomly positioned in a bin captured by the two image sensors, the plurality of objects and the bin being a proximal distance from the two image sensors and the robotic arm; generate one or more candidate segmentation proposals using a generalized segmentation model that utilizes the image, a candidate segmentation proposal representing a potential object instance in the image and including a confidence score; select one or more top candidates from the one or more candidate segmentation proposals using associated confidence scores; generate a six degree-of-freedom (6D0F) pose estimation for an object of the plurality of objects using a generalized pose estimation model and the selected one or more top candidates that contain a grasp instance; implement a multi-view triangulation refinement algorithm to refine the 6D0F pose estimation for the object; and instruct the robotic arm to grasp the object based on the refined 6D0F pose estimation.
2. The system according to claim 1, wherein the instructions are configured to generate an object segmentation mask and a two dimensional (2D) bounding box for the object with a candidate segmentation proposal having an associated confidence score that exceeds a confidence score threshold.
3. The system according to claim 2, wherein the instructions are configured to segment and crop the object from the image using the object segmentation mask and the 2D bounding box.
4. The system according to claim 3, wherein the instructions are configured to generate, during an offline process, one or more sets of synthetic templates at a number of pre-defined viewpoints using a rendering application and a computer aided design (CAD) model of the plurality of objects prior to receiving the image or prior to instructing the robotic arm to grasp the object.
5. The system according to claim 4, wherein the instructions are configured to predict a coarse 3D rotation of the object based on using a generalized pose classification model, offline rendered templates, and the segmented and cropped object from the image.
6. The system according to claim 5, wherein generating the 6DoF pose estimation for the object is further based on using the generalized pose classification model using the predicted coarse 3D rotation, the rendering application, and the segmented and cropped object from the image.
7. The system according to claim 6, wherein the instructions are configured to predict the refined 6DoF pose estimation of the object based on using a generalized pose refiner model using the 6DoF pose estimation, the rendering application, and the segmented and cropped object from the image.
8. The system according to claim 1, further comprising a depth sensor.
9. The system according to claim 8, wherein the instructions are configured to refine the 6DoF pose estimation for the object using depth information of the object captured by the depth sensor.
10. The system according to claim 9, wherein the instructions are configured to cache predicted poses from a neural network model and refinement steps using a depth map or multi-view triangulation refinement for the object in the memory during runtime.
11. The system according to claim 10, wherein the instructions are configured to instruct the robot arm to grasp another object of the plurality of objects in the bin using cached pose predictions based at least in part on implementing a structural similarity index measure metric for the image subsequent to the robot arm grasping the object from the bin.
12. A tangible, non-transitory computer-readable medium having instructions thereon which, upon being executed by one or more processors, provide for segmenting objects for motion planning with a robotic arm by execution of the following steps: receiving, via two image sensors, an image of a plurality of objects randomly positioned in a bin captured by the two image sensors, the plurality of objects in the bin and the two image sensors being a proximal distance from the robotic arm; generating one or more candidate segmentation proposals using a generalized segmentation model that utilizes the image, a candidate segmentation proposal representing a potential object instance in the image and including a confidence score; selecting one or more top candidates from the one or more candidate segmentation proposals using associated confidence scores; generating a six degree-of-freedom (6D0F) pose estimation for an object of the plurality of objects using a generalized pose estimation model and the selected one or more top candidates that contain a grasp instance; implementing a multi-view triangulation refinement algorithm to refine the 6D0F pose estimation for the object; and instructing the robot arm to grasp the object based on the refined 6D0F pose estimation.
13. The tangible, non-transitory computer-readable medium according to claim
12, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: generating an object segmentation mask and a two dimensional (2D) bounding box for the object with a candidate segmentation proposal having an associated confidence score that exceeds a confidence score threshold.
14. The tangible, non-transitory computer-readable medium according to claim
13, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: segmenting and cropping the object from the image using the object segmentation mask and 2D bounding box.
15. The tangible, non-transitory computer-readable medium according to claim
14, where the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: generating, during an offline process, one or more sets of synthetic templates at a number of pre-defined viewpoints using a rendering application and a computer aided design (CAD) model of the plurality of objects prior receiving the image or prior to instructing the robotic arm to grasp the object.
16. The tangible, non-transitory computer-readable medium according to claim
15, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: predicting a coarse 3D rotation of the object based on using a generalized pose classification model, offline rendered templates, and the segmented and cropped object from the image.
17. The tangible, non-transitory computer-readable medium according to claim
16, wherein generating the 6DoF pose estimation for the object is further based on using the generalized pose classification model using the predicted coarse 3D rotation, the rendering application, and the segmented and cropped object from the image.
18. The tangible, non-transitory computer-readable medium according to claim
17, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: predicting the refined 6DoF pose estimation of the object based on using a generalized pose refiner model using the 6DoF pose estimation, the rendering application, and the segmented and cropped object from the image.
19. The tangible, non-transitory computer-readable medium according to claim
18, wherein the instructions, upon being executed by the one or more processors, are further configured to execute the following steps: caching predicted poses from a neural network model and refinement steps using a depth map or multi-view triangulation refinement for the object in the memory during runtime.
20. A computer-implemented method for segmenting objects for motion planning with a robotic arm comprising: receiving, via two image sensors, an image of a plurality of objects randomly positioned in a bin captured by the two image sensors, the plurality of objects in the bin and the two image sensors being a proximal distance from the robotic arm; generating one or more candidate segmentation proposals using a generalized segmentation model that utilizes the image, a candidate segmentation proposal representing a potential object instance in the image and including a confidence score; selecting one or more top candidates from the one or more candidate segmentation proposals using associated confidence scores; generating a six degree-of-freedom (6D0F) pose estimation for an object of the plurality of objects using a generalized pose estimation model and the selected one or more top candidates that contain a grasp instance; implementing a multi-view triangulation refinement algorithm to refine the 6D0F pose estimation for the object; and instructing the robotic arm to grasp the object based on the refined 6D0F pose estimation.
PCT/IB2024/055721 2024-06-11 2024-06-11 Vision system for random bin picking applications of unseen objects Pending WO2025257594A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/IB2024/055721 WO2025257594A1 (en) 2024-06-11 2024-06-11 Vision system for random bin picking applications of unseen objects

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/IB2024/055721 WO2025257594A1 (en) 2024-06-11 2024-06-11 Vision system for random bin picking applications of unseen objects

Publications (1)

Publication Number Publication Date
WO2025257594A1 true WO2025257594A1 (en) 2025-12-18

Family

ID=91617268

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2024/055721 Pending WO2025257594A1 (en) 2024-06-11 2024-06-11 Vision system for random bin picking applications of unseen objects

Country Status (1)

Country Link
WO (1) WO2025257594A1 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022015807A1 (en) * 2020-07-14 2022-01-20 Vicarious Fpc, Inc. Method and system for object grasping
US20220405506A1 (en) * 2021-06-22 2022-12-22 Intrinsic Innovation Llc Systems and methods for a vision guided end effector

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022015807A1 (en) * 2020-07-14 2022-01-20 Vicarious Fpc, Inc. Method and system for object grasping
US20220405506A1 (en) * 2021-06-22 2022-12-22 Intrinsic Innovation Llc Systems and methods for a vision guided end effector

Similar Documents

Publication Publication Date Title
CN112652016B (en) Generation method of point cloud prediction model, pose estimation method and device thereof
US11845194B2 (en) Robot system and workpiece picking method
CN109255813B (en) Man-machine cooperation oriented hand-held object pose real-time detection method
CN110634161B (en) Rapid high-precision estimation method and device for workpiece pose based on point cloud data
CN116249607B (en) Method and device for robotically gripping three-dimensional objects
CN114387513B (en) Robot grasping methods, devices, electronic equipment and storage media
RU2700246C1 (en) Method and system for capturing an object using a robot device
US20200311855A1 (en) Object-to-robot pose estimation from a single rgb image
US20210114209A1 (en) Robot control device, and method and non-transitory computer-readable storage medium for controlling the same
CN117769724A (en) Synthetic dataset creation for object detection and classification using deep learning
CN112801977A (en) Deep learning-based relative pose estimation and monitoring method for assembly parts
US12061481B2 (en) Robot navigation using a high-level policy model and a trained low-level policy model
CN110335314A (en) The method of two dimensional image prediction target object real-time pose based on target object
CN108367436B (en) Active camera movement determination for object position and extent in 3D space
JP2023519725A (en) Image processing system and method
CN115222810A (en) Target pose estimation method and device, computing equipment and storage medium
CN117377987A (en) Transformation of covariate offsets for grasping neural networks
KR20240057354A (en) Method and apparatus for 6dof object pose estimation using self-supervision learning
Ge et al. Vipose: Real-time visual-inertial 6d object pose tracking
US20230298203A1 (en) Method for selecting surface points from a cad model for locating industrial 3d objects, application of this method to the location of industrial 3d objects, and augmented reality system usi
US11657506B2 (en) Systems and methods for autonomous robot navigation
JP2010184300A (en) Attitude changing device and attitude changing method
Wolnitza et al. 6D pose estimation and 3D object reconstruction from 2D shape for robotic grasping of objects
Sun et al. Panelpose: A 6d pose estimation of highly-variable panel object for robotic robust cockpit panel inspection
WO2025257594A1 (en) Vision system for random bin picking applications of unseen objects

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24735021

Country of ref document: EP

Kind code of ref document: A1