WO2020006433A1 - Segmentation de boîtes à partir d'un nuage de points 3d destiné à un déchargement et à un chargement automatiques - Google Patents

Segmentation de boîtes à partir d'un nuage de points 3d destiné à un déchargement et à un chargement automatiques Download PDF

Info

Publication number
WO2020006433A1
WO2020006433A1 PCT/US2019/039863 US2019039863W WO2020006433A1 WO 2020006433 A1 WO2020006433 A1 WO 2020006433A1 US 2019039863 W US2019039863 W US 2019039863W WO 2020006433 A1 WO2020006433 A1 WO 2020006433A1
Authority
WO
WIPO (PCT)
Prior art keywords
point cloud
points
package
planes
box
Prior art date
Application number
PCT/US2019/039863
Other languages
English (en)
Inventor
Zoran IVANOVSKI
Elena VASILEVA
Original Assignee
Fast Global Solutions, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fast Global Solutions, Inc. filed Critical Fast Global Solutions, Inc.
Publication of WO2020006433A1 publication Critical patent/WO2020006433A1/fr

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • B25J9/1697—Vision controlled systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1679—Programme controls characterised by the tasks executed
    • B25J9/1687—Assembly, peg and hole, palletising, straight line, weaving pattern movement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
    • B65G—TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS
    • B65G2203/00—Indexing code relating to control or detection of the articles or the load carriers during conveying
    • B65G2203/04—Detection means
    • B65G2203/041—Camera
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B65—CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
    • B65G—TRANSPORT OR STORAGE DEVICES, e.g. CONVEYORS FOR LOADING OR TIPPING, SHOP CONVEYOR SYSTEMS OR PNEUMATIC TUBE CONVEYORS
    • B65G67/00—Loading or unloading vehicles
    • B65G67/02—Loading or unloading land vehicles
    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00—Program-control systems
    • G05B2219/30—Nc systems
    • G05B2219/40—Robotics, robotics mapping to robotics vision
    • G05B2219/40067—Stack irregular packages
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

  • the present disclosure relates to automatic unloading and loading of packages, and in particular, to an algorithm for the segmentation of boxes for automatic unloading and loading of packages.
  • a robot vision system for recognizing and localizing differently shaped and sized objects in piled packaging scenarios using range data is presented in“A 3D-Robot Vision System for Automatic Unloading of Containers.” H. Thamer, H. Kost, D. Weimer, and B. Scholz-Reiter,“A 3D-Robot Vision System for Automatic Unloading of Containers,” IEEE International Conference on Emerging Technologies and Factory Automation, ETFA, 2013. Segmentation is performed in point cloud data based on geometric properties of boxes, bags, and barrels and the segmented surfaces are combined to complete packages. Different approaches for segmenting point clouds for other uses based on topology graphs, Gauss maps or segmentation of 2D range images have previously been presented. R. Schnabel, R. Wessel, R.
  • a system for automatic loading and/or unloading of packages includes an automatic package loading and/or unloading machine, a vision system that is configured to capture a 3D point cloud image of a package wall, and a controller that is configured to process the 3D point cloud image using segmentation software to generate an output, and wherein the controller is configured to command the automatic package loading and/or unloading machine, based on the output from the segmentation software, to place a package in the package wall and/or remove a package from the package wall.
  • a method of automatically unloading packages includes capturing a 3D point cloud image of a package wall; processing the 3D point cloud image using segmentation software; inputting an output from the segmentation software into a controller for an automatic package unloader machine; determining, based on the output from the segmentation software, which box in the package wall is highest and closest to the automatic package unloader machine; commanding the automatic package unloader machine to retrieve the box that is determined to be the highest and closet for unloading; and unloading the highest and closest box using the automatic package unloader machine.
  • a method of automatic loading packages includes capturing a 3D point cloud image of a package wall; processing the 3D point cloud image using segmentation software; inputting an output from the segmentation software into a controller for an automatic package loader machine; capturing weight and size details for a box on a conveyor; inputting the weight and size details for the box on the conveyor into the controller for the automatic package loader machine; determining, based on the output from the segmentation software and the weight and size details of the box on the conveyor, where to place the box on the conveyor in the package wall; commanding the automatic package loader machine to place the box on the conveyor in the package wall; and loading the box from the conveyor into the package wall using the automatic package loader machine.
  • a method of segmenting includes loading the 3D point cloud; segmenting planes from a scene of the 3D point cloud; segmenting edge points from the planes; segmenting edges from the edge points of each plane; obtaining groups of orthogonal planes; obtaining indices of planes which comprise boxes; and saving points of each box.
  • FIG. 1 is an isometric view of a forward end of a gantry unloader and an extendable conveyor.
  • FIG. 2 is an isometric view of a back end of the gantry unloader.
  • FIG. 3 is a block diagram of a control system for the gantry unloader.
  • FIG. 4 is a flow chart showing a method for automatic unloading.
  • FIG. 5 is a flow chart showing a method for automatic loading.
  • FIG. 6 is a flow chart showing a box segmentation algorithm.
  • FIG. 7 is a graph showing an example of a generated scene.
  • FIG. 8 is a graph showing an example of segmented boxes.
  • FIG. 9 is a graph showing an example of a generated scene.
  • FIG. 10 is a graph showing an example of a generated scene.
  • FIG. 11 is a graph showing a scene containing perfectly aligned boxes.
  • FIG. 12 is a graph showing the results of the segmentation of the scene shown in FIG. 11.
  • FIG. 13 is a flow chart showing the segmentation of planes from the input point cloud.
  • FIG. 14 is a graph showing separation of coplanar disconnected components.
  • FIG. 15 is a graph showing coplanar points belonging to different planes before separation process.
  • FIG. 16 is a flow chart describing the separation of multiple coplanar connected components.
  • FIG. 17 is a graph showing separated border points.
  • FIG. 18 is a flow chart showing the segmentation of the border points of a plane.
  • FIG. 19 is a graph showing segmented plane edges.
  • FIG. 20 is a flow chart showing the segmentation of plane edges.
  • FIG. 21 is a flow chart showing how to get orthogonal planes.
  • FIG. 22 is a flow chart showing the process of segmenting boxes from the surfaces.
  • FIG. 23 is a graph showing an example of segmented boxes.
  • FIG. 24 is a graph showing an example of segmented boxes.
  • FIG. 25 is a flow chart showing the surface segmentation algorithm.
  • FIG. 26 is a photo showing an example scene.
  • FIG. 27 is a graph showing an example point cloud representation of the scene shown in FIG. 26.
  • FIG. 28 is a graph showing separated true outliers.
  • FIG. 29 is a flow chart showing cloud filtering to separate the true outliers.
  • FIG. 30 is a graph showing erroneously assigned points in the RANSAC phase.
  • FIG. 31 is a graph showing correctly rearranged points.
  • FIG. 32 is a flow chart showing a points rearrangement algorithm.
  • FIG. 33 is a graph showing over- segmented surfaces.
  • FIG. 34 is a graph showing the result of surface merging of the scene shown in FIG. 33.
  • FIG. 35A is a flow chart showing a first portion of the merging of over-segmented surfaces.
  • FIG. 35B is a flow chart showing a second portion of the merging of over segmented surfaces.
  • FIG. 36 is a flow chart showing the reassigning of outliers to planes in a first version.
  • FIG. 37 is a graph showing an example of too many rejected outliers.
  • FIG. 38 is a graph showing a result of reassigning the outliers from the scene shown in FIG. 37.
  • FIG. 39 is a flow chart showing the reassigning of outliers to planes in a second version.
  • FIG. 40 is a photo showing an example of segmentation of box surfaces.
  • FIG. 41 is a flow chart showing the surface segmentation algorithm.
  • FIG. 42 is a photo showing an example of a packaging scenario.
  • FIG. 43 is a graph showing an example of a point cloud representation of the packaging scenario shown in FIG. 42.
  • FIG. 44 is a graph showing separated true outliers.
  • FIG. 45 is a graph showing erroneously assigned points in the RANSAC phase.
  • FIG. 46 is a graph showing correctly rearranged points.
  • FIG. 47 is a graph showing an example of segmented visible sides of boxes.
  • FIG. 48 is a photo showing results of the segmentation of box surfaces.
  • the present disclosure presents an algorithm for localizing, recognizing and segmenting packaging boxes from a 3D point cloud representation of a scene, intended to be used for automatic trailer unloading and loading.
  • a typical packaging scenario contains piles of boxes of different sizes and shapes placed in arbitrary configurations, which results in a high degree of occlusion.
  • the proposed algorithm relies on the structure of the packaging boxes which represent cuboid shapes comprised of orthogonal planes.
  • RANSAC Random SAmple Consensus
  • a sequence of post-processing techniques is applied to the output of RANSAC to resolve the incorrectly assigned points, and the conditions for planes to form a box which are implied from the cuboid shape of packaging boxes - orthogonality and sharing a common edge, are exploited for combining the resulting planes into boxes.
  • the algorithm is evaluated with simulated and real 3D point cloud data of different packaging scenarios.
  • the high degree of occlusion along with the similar color and texture of the packages makes segmenting instances of packages in 2D images a complicated problem. Exploiting the defined shape of types of packaging for segmentation of separate packages is a more intuitive approach. Existing approaches use range data acquired by laser scanners for recognition of packages in containers.
  • the present disclosure presents an algorithm for segmentation of cuboid boxes in a 3D point cloud representation of a scene containing piles of packages of different shapes and sizes. The present disclosure is based on describing a box as a set of mutually orthogonal planes sharing a common edge. The proposed algorithm is described in the context of systems for both automatic unloading and automatic loading below.
  • FIG. 1 is an isometric view of a forward end of gantry unloader 100 and extendable conveyor 102.
  • FIG. 2 is an isometric view of a back end of gantry unloader 100.
  • FIG. 3 is a block diagram of a control system for gantry unloader 100.
  • FIG. 1 shows gantry unloader 100 and extendable conveyor 102.
  • FIG. 2 shows gantry unloader 100.
  • Gantry unloader 100 includes gantry frame 110, platform 112, and gripper assembly 114.
  • Gantry frame 110 includes gantry frame drives 120.
  • Platform 112 includes platform drive 130.
  • Gripper assembly 114 includes gripper 140, which includes gripper arm 142, and gripper head 144.
  • Gripper assembly 114 further includes gripper rotational drive 146, gripper y-axis drive 148, gripper z-axis drive 150, and gripper x-axis drive 152.
  • Gantry unloader 100 further includes control system 160, which includes vision system 162 and controller 164.
  • FIGS. 1-2 also show gantry x-axis XGA, platform y-axis Yp, gripper x-axis XGR, gripper y-axis YGR, gripper z-axis ZGR, and gripper a- axis AGR.
  • Gantry unloader 100 can be aligned with an end of extendable conveyor 102. Gantry unloader 100 is not connected to extendable conveyor 102. The motion of gantry unloader 100 and extendable conveyor 102 can be coordinated so that extended conveyor 102 follows gantry unloader 100 to be aligned with gantry unloader 100. Extendable conveyor 102 includes a belt that is configured to move packages. Gantry unloader 100 includes gantry frame 110, platform 112, and gripper assembly 114. Gantry frame 110 is configured to sit around extendable conveyor 102 and a portion of platform 112. Gantry frame 110 is configured to move gantry unloader 100 through a semi-trailer.
  • Platform 112 is supported by gantry frame 110 and can be aligned with the end of extendable conveyor 102. Platform 112 is configured to move packages from the semi-trailer onto extendable conveyor 102.
  • Gripper assembly 114 is supported by gantry frame 110 and is configured to pull packages in the semi-trailer onto platform 112.
  • Gantry frame 110 includes gantry frame drives 120.
  • Gantry frame drives 120 are positioned on a back end of gantry frame 110 in the embodiment shown, but can be positioned on gantry frame 110 in any position in alternate embodiments.
  • Gantry frame drives 120 control the movement of gantry frame 110, and thus gantry unloader 100, through the semi-trailer.
  • Gantry frame 110 moves along gantry x-axis XGA as it moves through the semi-trailer.
  • Gantry frame drives 120 drive gantry frame 110 along gantry x- axis XGA-
  • Platform 112 is connected to gantry frame 110. Platform 112 is configured to move up and down on gantry frame 110 along platform y-axis Yp. Platform 112 is controlled by platform drive 130. Platform drive 130 is positioned on gantry frame 110 in the embodiment shown, but can be positioned anywhere on gantry unloader 100 in alternate embodiments. Platform drive 130 drives platform 112 up and down along platform y-axis Yp on gantry frame 110.
  • Gripper assembly 114 includes gripper 140, which is configured to move forwards and backwards along gripper x-axis XGR, up and down on gripper y-axis YGR, side to side along gripper z-axis ZGR, and rotate around gripper a-axis AGR with respect to gantry unloader 100.
  • Gripper 140 includes gripper head 144 that is connected to a forward end of gripper arm 142.
  • Gripper head 144 includes grasping fingers and suction cups in the embodiment shown.
  • Gripper assembly 114 further includes gripper rotational drive 146, gripper y-axis drive 148, gripper z-axis drive 150, and gripper x-axis drive 152.
  • Gripper rotational drive 146 rotates gripper head 144 on gripper arm 142 around gripper a-axis AGR.
  • Gripper y- axis drive 148 moves gripper assembly 114, and thus gripper 140, up and down on gantry frame 110 along gripper y-axis YGR.
  • Gripper z-axis drive 150 moves gripper 140 side to side along gripper z-axis ZGR.
  • Gripper x-axis drive 152 moves gripper 140 forwards and backwards along gripper x-axis XGR.
  • Gantry unloader 100 is designed to automatically unload packages, parcels, and parcel bags from semi-trailers that commercially-haul packages, parcels and parcel bags.
  • Gantry unloader 100 includes gripper 140 that extracts one package, parcel, or parcel bag at a time onto platform 112.
  • Platform 112 then conveys the extracted package onto extendable conveyor 102, which conveys the package out of the semi-trailer and into the building.
  • Platform 112 is capable of moving up and down along a prescribed path, while gripper 140 is capable of moving side to side, up and down, forward and rearward, and rotationally all along prescribed paths. Platform 112 and gripper 140 are supported and transported by gantry frame 110.
  • Gantry unloader 100 is controlled by control system 160.
  • Control system 160 includes vision system 162 to create near-real-time, electronic, 3D images of the packages within the semi-trailer as the semi-trailer is being unloaded.
  • Controller 164 is programmed to use the 3D image to control the movements of gripper 140 that is used to extract one package at a time from the packages in the semi-trailer.
  • controller 164 can send signals to gripper rotational drive 146, gripper y-axis drive 148, gripper z-axis driver 150, and gripper x-axis drive 152 to control the movements of gripper 140. Movement of the entire gantry unloader 100 and extendable conveyor 102 is also controlled based on the 3D image.
  • controller 164 can send a signal to gantry frame drives 120 to control movement of gantry frame 110 and to platform drive 130 to control movement of platform 112.
  • All of the motors used on gantry unloader 100 for positioning are electric stepper motors with built-in rotational position sensors.
  • the rotational position signal from each motor indicates to controller 164 where that particular function is located within its range of motion.
  • the position signal is input into the programmed algorithms within the controller.
  • gantry unloader 100 utilizes vision system 162.
  • Vision system 162 includes a Light Detection and Ranging (LIDAR) sensor and a high-resolution camera, or a 3D Scanner, to create an electronic‘vision’ of the packages within the semi-trailer as well as the semi-trailer walls, ceiling, and floor. Both the LIDAR sensor and camera, or the 3D Scanner, face toward the package wall, and these vision system 162 components are connected to controller 164 via electrical and ethernet cables.
  • LIDAR Light Detection and Ranging
  • the LIDAR sensor uses a laser beam reflection technique to send an array of laser beams, of known strength and frequency, out from the LIDAR sensor. If objects are present in the path of the laser beams, the laser beam will reflect back to the LIDAR sensor, and the LIDAR sensor measures the time and frequency of the reflected laser beam. Algorithms within controller 164 then calculate the distance based on the reflected laser beams. A 3D Scanner may be used in place of the LIDAR sensor and the camera. Since each laser beam is a discrete light beam, the calculation for the distance to the object generates only one point in 3D space. Simultaneously calculating the distance for each laser beam produces a point cloud which can then be used to determine general shapes of the packages in 3D space.
  • the LIDAR sensor is programmed to take measurements every 2 seconds for near-real-time, positional determination of the packages in the semi-trailer.
  • the resolution of the point cloud is limited by the number of laser beams available in the LIDAR sensor so a high- resolution camera is used to augment the resolution.
  • the high-resolution camera takes images every 2 seconds synchronized with the LIDAR sensor.
  • Algorithms within controller 164 then use the camera image and the point cloud to make an improved 3D image of the packages. Controller 164 then uses the improved 3D image of the packages to determine which package to remove from the package wall.
  • a 3D Scanner uses infrared light instead of laser light to accomplish much the same electronic vision of packages. Algorithms within controller 164 then calculate the distance based on the reflected infrared light.
  • Controller 164 determines if platform 112 is within the physical reach of the package to be extracted; if not within reach, controller 164 sends commands to move gantry unloader 100 forward or backward to a position that makes the package to be extracted within the range of motion of gantry unloader 100. Simultaneously, controller 164 moves platform 112 to the correct height, determines if suction cups or grasping fingers will be used for this particular extraction, and positions gripper 140. Once in position, controller 164 commands gripper 140 to move forward to the proper position for engaging the package, commands the suction cups or grasping fingers to engage the package, commands gripper 140 to move rearward and upward slightly to aid in extraction all while maintaining hold on the package. Once the package is extracted onto platform 112, controller 164 commands gripper 140 to release the package and moves gripper 140 up and over as-needed to allow movement of the package across platform 112 to extendable conveyor 102.
  • Extendable conveyor 102 conveys the packages to the rear of the semi-trailer and into the building for further processing of the packages.
  • the desired rate of package unloading is at least 2000 packages per hour.
  • LIDAR and 3D Scanner scan times allow for near-real-time scanning and generation of the 3D images which allows for significantly reduced wait times while controller 164 is processing information and subsequent delivery of movement commands.
  • Gantry unloader 100 can also be used to load packaged into a semi-trailer.
  • Gantry unloader 100 is one example of an automatic unloader or loader that can utilize the methods and algorithms described below. In alternate embodiments, the following methods and algorithms can be performed using any suitable automatic unloader or loader.
  • FIG. 4 is a flow chart showing a method for automatic unloader.
  • FIG. 4 includes steps 200-212.
  • Step 200 includes capturing a 3D point cloud image of a package wall.
  • vision system 162 shown in control system 160 of automatic unloader 100 can be used to capture the 3D point cloud.
  • a LIDAR sensor or a 3D scanner can be used to capture the 3D point cloud.
  • Step 202 includes processing the 3D point cloud image using segmentation software.
  • the segmentation software will undergo a series of processing steps to segment the boxes in the 3D point cloud.
  • a first embodiment of a segmentation algorithm includes the following steps: the 3D point cloud in loaded, planes from the scene are segmented using RANSAC, edge points are segmented from the planes, edges from the edge points of each plane are segmented using RANSAC, a group of orthogonal planes are obtained, indices of planes which comprise boxes are obtained, and points of each box are saved.
  • the first embodiment of the segmentation algorithm is discussed in further detail with respect to FIGS. 6-24.
  • a second embodiment of a segmentation algorithm includes the following steps: the 3D point cloud is loaded, planes from the scene are segmented using RANSAC, coplanar disconnected surfaces are separated, the cloud is prefiltered to separate true outliers, points are rearranged, surfaces are combined, and outliers are reassigned.
  • the second embodiment of the segmentation algorithm is discussed in further detail with respect to FIGS. 25-40.
  • a third embodiment of a segmentation algorithm includes the following steps: the 3D point cloud is loaded, the cloud is prefiltered to separate true outliers, planes from the scene are segmented using RANSAC, coplanar disconnected surfaces are separated, points are rearranged, surfaces are combined, and outliers are reassigned.
  • the third embodiment of the segmentation algorithm is discussed in further detail with respect to FIGS. 41-48.
  • Step 204 includes inputting an output from the segmentation software into a controller for an automatic unloader.
  • a controller for an automatic unloader For example, with respect to automatic unloader 100 described above with respect to FIGS. 1-3, an output from the segmentation software can be input into controller 164 of control system 160 of automatic unloader 100.
  • Step 206 includes determining, based on the output from the segmentation software, which box in the package wall is highest and closest to the automatic unloader. For example, with respect to automatic unloader 100 described above with respect to FIGS. 1-3, controller 164 of control system 160 can determine which box in the package wall is highest and closed to automatic unloader 100 based on the output from the segmentation software.
  • Step 208 includes commanding the automatic unloader to retrieve the highest and closest box for unloader.
  • controller 164 of control system 160 can command gantry frame drives 120 to move gantry frame 110 front and back to the right position with respect to the package wall. Controller 164 of control system 160 can then command platform drive 130 to move platform 112 to move up and down to the right position with respect to the package wall. Finally, controller 164 of control system 160 can command gripper rotational drive 146, gripper y-axis drive 148, gripper z-axis drive 150, and gripper x-axis drive 152 to move gripper assembly 114 to grasp and remove a box.
  • Step 210 includes unloading the highest and closest box.
  • gripper assembly 114 can grasp and remove the highest and closest box from the package wall.
  • Steps 200-210 can be repeated as needed to remove all boxes from the package wall.
  • Using segmentation software to segment the boxes in the package wall allows an automatic unloader to remove boxes from the package wall without user intervention.
  • FIG. 5 is a flow chart of a system for automatic loading.
  • FIG. 5 includes steps 300-314.
  • Step 300 includes capturing a 3D point cloud image of a package wall.
  • a vision system in a control system of an automatic loader can be used to capture the 3D point cloud.
  • a LIDAR sensor or a 3D scanner can be used to capture the 3D point cloud.
  • Step 302 includes processing the 3D point cloud image using segmentation software.
  • the segmentation software will undergo a series of processing steps to segment the boxes in the 3D point cloud.
  • a first embodiment of a segmentation algorithm includes the following steps: the 3D point cloud in loaded, planes from the scene are segmented using RANSAC, edge points are segmented from the planes, edges from the edge points of each plane are segmented using RANSAC, group of orthogonal planes are obtained, indices of planes which comprise boxes are obtained, and points of each box are saved.
  • the first embodiment of the segmentation algorithm is discussed in further detail with respect to FIGS. 6-24.
  • a second embodiment of a segmentation algorithm includes the following steps: the 3D point cloud is loaded, planes from the scene are segmented using RANSAC, coplanar disconnected surfaces are separated, the cloud is prefiltered to separate true outliers, points are rearranged, surfaces are combined, and outliers are reassigned.
  • the second embodiment of the segmentation algorithm is discussed in further detail with respect to FIGS. 25-40.
  • a third embodiment of a segmentation algorithm includes the following steps: the 3D point cloud is loaded, the cloud is prefiltered to separate true outliers, planes from the scene are segmented using RANSAC, coplanar disconnected surfaces are separated, points are rearranged, surfaces are combined, and outliers are reassigned.
  • the third embodiment of the segmentation algorithm is discussed in further detail with respect to FIGS. 41-48.
  • Step 304 includes inputting an output from the segmentation software into a controller for an automatic loader.
  • Step 306 includes capturing weight and size details for a box on a conveyor.
  • the weight and size details can be captured using any suitable means.
  • Step 308 includes inputting the weight and size details for the box on the conveyor into the controller for the automatic loader.
  • Step 310 includes determining, based on the output from the segmentation software and the weight and size details of the box on the conveyor, where to place to box in the package wall.
  • Step 312 includes commanding the automatic loader to place the box on the conveyor in the package wall.
  • a controller of a control system can command drives to move a frame, a platform, a belt, a gripper, or any other suitable component into the right position with respect to the package wall.
  • Step 314 includes loading the box from the conveyor into the package wall.
  • An automatic loader can load the box from the conveyor in any suitable position in the package wall based on the weight and size details of the box.
  • Steps 300-314 can be repeated as needed to load as many boxes as possible and/or needed into the package wall.
  • Using segmentation software to segment the boxes in the package wall allows an automatic loader to load boxes into the package wall without user intervention.
  • FIGS. 3-46 describe embodiments of the segmentation algorithm in more detail.
  • FIG. 6 is a flow chart showing a box segmentation algorithm.
  • FIG. 6 includes steps 400-412.
  • Step 400 includes loading an input point cloud.
  • the input point cloud is represented as an unordered list of points, with each point represented by three coordinates in space.
  • Step 402 is plane segmentation, where the input points are separated in groups of points which are part of a plane and outlier points. Points at a great distance from their nearest neighbors and groups of points with less than 15 points are declared outliers. Due to the noisy data, a predefined tolerance for the distance of a point to a plane is empirically set. The input point clouds contain outliers due to low scanner resolution or occlusions in the stereo image pairs from which the depth is extracted. The good performance of RANSAC in datasets with a large percent of outliers is the main reason for choosing this algorithm for plane segmentation.
  • the plane segmentation is done using the Point Cloud Library implementation of RANSAC. RANSAC is used in a loop to segment all planes from the input point cloud which contain enough points. RANSAC can’t separate multiple connected components (like multiple box sides) which lie on the same plane. Therefore, a region growing procedure is used to separate the multiple connected components and remove coplanar points from other surfaces.
  • Step 404 includes performing segmentation of the border points of each plane in order to simplify the process of getting the edges of each plane.
  • the border points of every plane are determined by the number of neighbors belonging to the plane - every point having fewer neighbors within a spherical environment than a predefined threshold is declared a border point.
  • Step 406 includes using RANSAC to segment the plane edges from the border points for modeling a line in 3D space.
  • Step 408 includes grouping the segmented planes into boxes based on two basic criteria: the planes comprising a box have to be orthogonal and share an edge.
  • Step 410 includes finding groups of orthogonal planes. Orthogonal planes sharing an edge are declared a box.
  • Step 412 includes saving the points of each box.
  • the points of each box form the output that can be input into a controller of an automatic unloader or automatic loader, such as controller 164 of automatic unloader 100.
  • the controller of the automatic unloader or automatic loader can then determine which box to unload or load, respectively, as described above in reference to FIGS. 4-5.
  • the minimum visible sides of a box are one, and the maximum is three. Because a decision with a single plane is not reliable, only boxes with two or three detected sides are accepted. Starting the search for boxes from the top avoids selecting more orthogonal planes which are part of different boxes; therefore, the highest plane is selected as a starting plane.
  • the output of the algorithm is a list of planes belonging to a single box.
  • the flow chart of the described algorithm is shown in FIG. 6.
  • the algorithm shown in FIG. 6 performs well when applied to simulated data representing piles of packages - it is able to correctly segment packages of different sizes and orientations.
  • a sequence of adaptations to the algorithm is introduced for segmenting packages from a point cloud obtained using depth estimation from stereo image pairs. These changes are presented below in reference to FIGS. 25-48.
  • the algorithm was tested using simulated data.
  • the input for the algorithm is a point cloud representing a scene consisting of a pile of randomly orientated boxes (cuboids).
  • the generated boxes are fully described in 3D - all 6 sides of a box are generated.
  • a viewpoint Tenderer is used to simulate real data. In data received from a range scanner, there are occlusions and changes in point density depending on the distance and orientation of the box in relation to the scanner. After the viewpoint rendering, uniform noise is added to the points.
  • An example of the generated scenes is shown in the graph of FIG. 7.
  • the proposed method performs well when segmenting boxes in simulated data of scenes containing a relatively small number of boxes and is able to handle a high level of random noise, any position and rotation of the boxes, and the resulting occlusions.
  • An example of segmented boxes in simulated data is shown in the graph of FIG. 8.
  • the following is a description of the first embodiment of the box segmentation algorithm, discussed above with respect to FIG. 6, for segmenting separate boxes from an input point cloud.
  • the point cloud segmentation is performed by implementing standard methods of digital signal processing like RANSAC for segmenting lines or planes and statistical filtering of the intermediate results.
  • the experiment described herewith describes the box segmentation algorithm applied to artificially generated 3D scenes.
  • the software used was Python 2.7 and the libraries used were pel (Point Cloud Library) (3D point cloud library, implemented filtering algorithms and shape segmentation (lines, planes, cylinders%)); numpy (basic mathematical functions and working with matrices); and matplotlib.py plot (3D scene plotting).
  • pel Point Cloud Library
  • numpy basic mathematical functions and working with matrices
  • matplotlib.py plot (3D scene plotting).
  • the input for step 400 of the algorithm is a point cloud representing a scene of randomly placed and rotated boxes (cuboids), where occlusion of objects is present.
  • the generated boxes are fully described in 3D - all six sides of a box are generated.
  • a viewpoint Tenderer is used to remove the occluded points and decrease the point density as distance from the point is increased. After the viewpoint rendering, random noise is added to the points. Examples of the generated scenes are shown in the graphs of FIGS. 9 and 10.
  • the input point cloud is represented as an unordered list of points, with each point represented by three coordinates in space.
  • the list is converted to a numpy array object for easier manipulation.
  • Step 402 is plane segmentation, where the input points are separated in groups of points which are part of a plane (the tolerance for the distance of a point to a plane is empirically set), and outlier points. Outliers are points far away from their neighbors or groups of less than 15 points.
  • RANSAC is chosen as the algorithm for plane segmentation because of the good performance in datasets with a large percent of outliers.
  • the border points of every plane are determined by the number of neighbors - every point having fewer neighbors than a predefined threshold (min_num_sosedi) is declared a border point.
  • a predefined threshold min_num_sosedi
  • RANSAC with a 3D line model is used to get the plane edges from the segmented border points.
  • a box is comprised by 2 or 3 orthogonal planes sharing an edge. Therefore, at step 408, orthogonal planes for every plane are found. Finally, orthogonal planes sharing an edge are declared a box at step 410.
  • the described algorithm is implemented in a box_check.py file, and the flow chart of the algorithm is shown in FIG. 6.
  • Each block in the diagram on FIG. 6 represents a separate function call which performs the given task.
  • the detailed description of the algorithms implemented for each block is provided below.
  • Step 402 is described herewith in more detail.
  • RANSAC is used in a loop to segment a plane from the input point cloud while there are enough points to form a plane.
  • the minimum number of points to define a plane is a predefined threshold (min_no_plane), empirically set at 15. If there are more than 15 points, but they are not coplanar, the algorithm will continue to the next iteration without returning a plane. Therefore, if a plane hasn’t been returned in three successive iterations, the process stops and the found planes are saved.
  • RANSAC is the minimum normal distance of a point to the plane for the point to be accepted as a part of the plane.
  • the distance threshold (distance_thr) is empirically set at 2.0.
  • the angular distance of the surface normal in a point didn’t change the outcome of the algorithm, so the final version of the algorithm doesn’t use it.
  • RANSAC can’t separate multiple connected components (multiple box sides) which lie on the same plane, therefore a separate function (f_multiple_surfaces) is called to separate the possible multiple connected components.
  • An example of a scene with aligned boxes is shown in the graph of FIG. 11 , and the result containing multiple box sides segmented as one is shown in the graph of FIG. 12.
  • the described algorithm is implemented in a f_ransac_planes.py file and the flow chart of the algorithm is shown in the flow chart of FIG. 13.
  • FIG. 13 shows the flow chart for the plane segmentation algorithm.
  • an input point cloud is converted to pel datatype.
  • step 510 If the answer to step 510 is yes, then a plane is segmented with RANSAC at step 512. Then at step 514, it is determined if the plane contains enough points. If the answer to step 514 is no, then the process continues at step 504. If the answer to step 514 is yes, then all spatially separated connected components from the plane are obtained (f_multiple_surfaces.py) at step 516. Then at step 518, all spatially separated connected components from the plane are saved. Then the process continues at step 504.
  • the separation of multiple coplanar connected components is performed using region growing in a 3D space.
  • the input is a group of coplanar points outputted from RANSAC.
  • a starting point for region growing is randomly selected. All points within a cubic volume of size 25 are a potential point of the region and are assigned to the current neighborhood. If a point from the neighborhood has more than a predefined threshold (min_num_sosedi) of points empirically set at five, it is added to the region. The threshold is added to help remove small parts of another surface connected to the current component.
  • the resulting components are accepted as a plane in a f_ransac_planes function if they have more than 15 points. All rejected points are returned to the unprocessed point cloud to be possibly assigned to another plane.
  • this algorithm successfully removes parts of other orthogonal surfaces that satisfy the minimum distance from plane criterion (example: points from line L on plane P on the graph of FIG. 15).
  • the described algorithm is implemented in a f_multiple_surfaces.py file, and the flow chart of the algorithm is shown in FIG. 16.
  • FIG. 16 shows the flow chart for a connected components algorithm.
  • step 602 it is determined if there are unprocessed points. If the answer to step 602 is no, then a list of connected components is returned at step 604 and the process is complete. If the answer to step 602 is yes, then a starting point for the next connected component is chosen at step 606. Then at step 608, an empty list is created which will contain the candidate points of the connected component and the starting point is appended to it. Then at step 610, it is determined if the list of candidate points is empty. If the answer to step 610 is no, then the process continues at step 602. If the answer to step 610 is yes, then a candidate point to process is chosen at step 612.
  • step 614 all points within a radius of the current point is obtained. Then at step 616, the neighbor points to the list of candidates is added. Then at step 618, it is determined if the chosen candidate point has high enough neighbors. If the answer to step 618 is no, the candidate point is removed from the list of candidates at step 620 and the process continues at step 610. If the answer to step 618 is yes, then the point is assigned to the final connected component at step 622 and the process continues to step 620.
  • Step 404 is described herewith in more detail.
  • segmentation of the border points of each plane is performed by counting the number of neighbors of each point (see the graphs of FIG. 17). Points with fewer than 40 neighbors within a cubic volume of size 30 are declared border points.
  • the described algorithm is implemented in a f_plane_edges.py file, and the flow chart of the algorithm is shown in FIG. 18.
  • FIG. 18 shows the algorithm for getting edge points.
  • the point is added to the edge points list.
  • RANSAC with a mathematical model of a line is used to segment plane edges from the border points of each plane (see the graph of FIG. 19). Each edge must contain more than 10 points to be accepted (the threshold (min_no_line) is empirically chosen). Other RANSAC parameter used is the minimum normal distance of a point to the line for the point to be accepted as a part of the line, empirically set at 1.5. The algorithm ends after all lines with more than 10 points are found. The described algorithm is implemented in a f_plane_edges.py file, and the flow chart of the algorithm is shown in FIG. 20.
  • FIG. 20 shows the algorithm for edge segmentation.
  • a list containing border points of a surface is created.
  • Step 408 is described herewith in more detail.
  • the planes For a set of planes to form a box, two basic criteria need to be met: the planes have to be orthogonal and they have to share an edge. Getting orthogonal planes is a helping step for box segmentation.
  • RANSAC returns the four coefficients for each plane.
  • Two planes are declared orthogonal if the dot product of the equations is close to zero (empirically set threshold (normal_threshold) - 0.01).
  • the described algorithm is implemented in a f_find_normal_planes.py file, and the flow chart of the algorithm is shown in FIG. 21.
  • FIG. 21 shows the algorithm for getting orthogonal planes.
  • an empty list is created which will contain sublists of orthogonal planes for each plane.
  • step 914 it is determined if the dot product of the planes’ coefficients is less than the threshold (close to 0). If the answer to step 914 is no, then the process continues at step 910. If the answer to step 914 is yes, then the compared plane is added to the sublist of orthogonal planes at step 916 and the process continues at step 910. If the answer to step 910 is yes, then the process continues at step 904.
  • Step 410 is described in more detail herewith.
  • the basic criterion for a set of planes to form a box is orthogonal planes sharing an edge.
  • the minimum visible sides of a box are one, and the maximum is three.
  • This algorithm accepts boxes with two or three detected sides because a decision with one plane is not reliable. Starting the search for boxes from the top avoids selecting more orthogonal planes which are part of different boxes.
  • the highest plane is selected as a starting plane. If there are other planes at a less than ten steps lower height, the starting plane is the one closest to horizontal. All edges of the starting plane are compared to all the edges of the planes orthogonal to the starting plane.
  • RANSAC returns six coefficients of the segmented edges (lines). A sharing edge is defined if all coefficients of both lines differ by less than 0.5. If two planes belonging to the same box are found, a next box is searched. The output of this function is lists containing the indices of the planes belonging to a box.
  • the described algorithm is implemented in a f_get_box.py file, and the flow chart of the algorithm is shown in FIG. 22. An example of segmented boxes is shown in the graphs of FIGS. 23 and 24.
  • FIG. 22 shows the algorithm for box segmentation.
  • an empty list is created which will contain sublists of indices of surfaces belonging to the same box.
  • step 1016 it is determined if there are planes orthogonal to the current plane not yet checked for a common edge. If the answer to step 1016 is no, it is determined if the box sublist contains more than one plane at step 1018. If the answer to step 1018 is no, then the process continues to step 1006. If the answer to step 1018 is yes, then a box sublist is appended to the boxes list at step 1020 and the process continues to step 1006. If the answer to step 1016 is yes, then all edge coefficients are compared to the orthogonal plane at step 1022. Then at step 1024, it is determined if the difference is less than the threshold. If the answer to step 1024 is no, then the process continues at step 1016.
  • step 1024 determines if the plane index is added to the box sublist at step 1026. Then at step 1028, it is determined if the box sublist contains less than three planes. If the answer to step 1028 is no, then the process continues at step 1004. If the answer to step 1028 is yes, then the process continues at step 1016. If the answer to step 1010 is yes, then the most horizontal plane is selected at step 1030 and the process continues to step 1014.
  • the box segmentation algorithm described above shows the possibilities of the standard digital signal processing methods for 3D box segmentation.
  • the proposed method works well with simulated data of scenes containing a small number of boxes (tests were performed with at most ten boxes).
  • FIG. 25 is a flow chart showing a surface segmentation algorithm.
  • FIG. 25 includes steps 1100-1112.
  • Step 1100 includes loading the point cloud.
  • Step 1102 includes segmenting planes from the scene.
  • Step 1104 includes separating coplanar disconnected surfaces. Steps 1100, 1102, and 1104 are the same as steps 400, 402, and 404 discussed above in reference to FIG. 6 for the first segmentation algorithm.
  • Step 1106 includes cloud prefiltering.
  • the number of outliers is increased due to the different occlusions appearing in each of the images, so the cloud is filtered.
  • points whose distance from the cameras differs significantly from the distance of the neighbor points are excluded from the point cloud.
  • Step 1108 includes point rearrangement. Because no perspective correction is performed to the input point cloud, some planar surfaces are distorted, which results in some points being wrongly assigned to different surfaces. In some situations, after the plane segmentation phase, some points have more neighboring points belonging to another surface than the one chosen by RANSAC. That results in addition of line like shapes to otherwise rectangular surfaces. Therefore a post-processing step of rearranging the points reassigns each point to the surface in which the point has the largest number of neighbors.
  • Step 1110 includes combining surfaces.
  • a surface merging step based on adjacency and similar plane coefficients is applied.
  • Step 1112 includes reassigning outliers. In order to reduce the number of points which are declared outliers, reassignment of outlier points is performed. In this step, the depth value for each outlier is recalculated using the median depth value of the neighbor points before it is returned to the point cloud.
  • the software used was Python 2.7 and the libraries used were pel (Point Cloud Library) (3D point cloud library, implemented filtering algorithms and shape segmentation (lines, planes, cylinders%)); numpy (basic mathematical functions and working with matrices); and matplotlib.pyplot (3D scene plotting).
  • pel Point Cloud Library
  • numpy basic mathematical functions and working with matrices
  • matplotlib.pyplot (3D scene plotting).
  • the input for the second part of the algorithm is a point cloud representing a real scene of randomly placed and rotated boxes (cuboids), cylindrical boxes and bags, where occlusion of objects is present.
  • a 3D point cloud representation of the scene is generated from a stereo image pair of each scene. No perspective correction is applied.
  • An example of the real scenes used for testing is shown in the photo of FIG. 26 and the corresponding 3D point cloud of the scene is shown in the graph of FIG. 27.
  • the first part of the segmentation process is used unchanged - the input point cloud is represented as a list of points, with each point represented by 3 coordinates in space and the first step is plane segmentation using RANSAC.
  • the only parameter value that is changed is the tolerance threshold (distance_thr) for the distance of a point to a plane - the threshold is decreased to 1.5.
  • a separate function is used for separating the coplanar disconnected surfaces (see FIG. 14). Multiple disconnected coplanar surfaces returned by RANSAC are then separated using region growing in a 3D space.
  • a separate function cloud_prefiltering py iterates through the outliers rejected by RANSAC and separates true outliers - points at a great distance from their nearest neighbors.
  • points rearrangement reassigns each point to the surface which has the largest number of neighbors.
  • the function preraspredelba.m implements points rearrangement.
  • the scene’s change in perspective causes combined parts of surfaces or a lot of disconnected components when using a distance threshold of 2.0, so the threshold had to be decreased to 1.5.
  • the threshold decrease causes two significant side effects - some box sides are separated in multiple surfaces, and the number of outliers that RANSAC discards is increased.
  • a new post-processing step is introduced - checking whether multiple surfaces are neighbors and have similar plane coefficients. If the two conditions are met, the surfaces are merged.
  • a second post processing step examines the deviation of the outliers from the environment. If a predefined threshold is reached, the outlier is repositioned and assigned to a surface; otherwise, it is simply reassigned to the closest surface.
  • the proposed algorithm is described by the flow chart shown in FIG. 25. Each of the blocks in the diagram describing the post-processing steps represents a separate script which performs the given task.
  • Step 1106 is described herewith in detail.
  • pre-filtering is performed to detect the true outliers - points at a great distance from their nearest neighbors.
  • the z coordinate of each point (represents depth in the scene) is compared to the mean z coordinates of the neighbor points. If the distance is greater than a threshold (outlier_thr) empirically set at 10, the point is declared a true outlier.
  • a list of declared outliers is saved. An example of separated true outliers is shown in the graph of FIG. 28.
  • the described algorithm is implemented in a cloud_prefiltering.py file, and the flow chart of the algorithm is shown in FIG. 29.
  • FIG. 29 shows the algorithm for cloud prefiltering.
  • an empty list is created which contains the outliers.
  • step 1216 it is determined if the distance is smaller than the threshold. If the answer to step 1216 is no, then the point is added to the outliers list at step 1218 and the process continues at step 1204. If the answer to step 1216 is yes, then the process continues at step 1210. If the answer to step 1210 is yes, then the distance between points is calculated at step 1220. Then at step 1222, it is determined if the distance is smaller than the threshold. If the answer to step 1222 is no, then the process continues at step 1210. If the answer to step 1222 is yes, then the point is declared a neighbor at step 1224 and the process continues at step 1210.
  • Step 1108 is described herewith in detail.
  • the distance criterion used to calculate surfaces from points in RANSAC phase of the algorithm could assign points to different surfaces. In some situations some points have more neighboring points belonging to another surface than the one chosen by RANSAC. That results in addition of line like shapes to otherwise rectangular surfaces. Example of this is given in the graph of FIG. 30.
  • the points rearrangement procedure reassigns each point to the surface which has the largest number of neighbors out of all surfaces that the point is close enough to.
  • the function preraspredelba.m implements points rearrangement.
  • FIG. 31 shows the effect of applied rearrangement.
  • FIG. 32 shows a flow chart of the algorithm for points rearrangement.
  • thresholds, variables, and color pallet is initialized.
  • the points are read from txt files and plane coefficients are read from txt files.
  • the process is started with the first source plane.
  • step 1312 determines if the distance from the current point to all planes. If the answer to step 1312 is no, then the distance from the current point to all planes is calculated, and the distance to planejhr is compared to set flags (0-too far, l-close). Then at step 1316, it is determined if the point is close to at least one plane. If the answer to step 1316 is no, then the point is discarded as an outlier at step 1318. Then at step 1320, the next point is obtained, and the process continues at step 1312. If the answer to step 1316 is yes, then the process is further started with the first destination plane at step 1322. Then at step 1324, it is determined if all planes are checked. If the answer to step 1324 is no, then it is determined if the point is close to the destination plane at step 1326.
  • step 1326 If the answer to step 1326 is no, then the next destination plane is obtained at step 1328 and the process continues at step 1324. If the answer to step 1326 is yes, then all points from the destination plane are compared to the current point at step 1330. Then at step 1332, all that are closer than distjhr are counted and the process continues at step 1328. If the answer to step 1324 is yes, then the maximum number of neighboring points and index of the plane is found at step 1334. Then at step 1336, it is determined if the number is bigger than num_pnt_thr. If the answer to step 1336 is no, then the point is discarded as an outlier at step 1338 and the process continues at step 1320.
  • step 1336 it is determined if the point is closest to the source plane at step 1340. If the answer to step 1340 is no, then the point is deleted from the source plane and added to the destination plane at step 1342 and then process continues at step 1320. If the answer to step 1340 is yes, then the process continues at step 1320. If the answer to step 1312 is yes, then the next source plane is obtained at step 1344 and the process continues at step 1308. If the answer to step 1308 is yes, then the points are written into txt files at step 1346 and the process is complete.
  • Step 1110 is described herewith in detail.
  • the check for possible surfaces belonging to the same box side uses two criteria: coplanarity and adjacency of the surfaces. Only the planes with less than 200 points are considered for merging. Firstly, the surfaces whose direction coefficients don’t differ more than 0.1, and the distance coefficients don’t differ more than 30 are declared candidates for merging. Surfaces which have some points within a distance of 30 are considered adjacent. The list of pairs of surfaces to be merged will contain pairs of coplanar surfaces which were separated by the coplanar connected component separation function f_multiple_surfaces.py. Those pairs of surfaces are removed from the list. Only surface pairs meeting all previous conditions are further considered.
  • the first check is iterating through the list to detect whether multiple surfaces need to be merged (ex. the list contains (1,2), (2, 3), (3,1)). If a surface can be merged to two other surfaces, but there is no pair containing the other two surfaces, the current surface is merged with one of them. The surface with a point centroid closer to the centroid is selected for merging. All selected surfaces are merged into the largest one.
  • An example case where merging is needed is shown in the graph of FIG. 33, and the result of the merging algorithm shows significant improvement, as shown in the graph of FIG. 34.
  • the described algorithm is implemented in a combine_surfaces.py file, and the flow chart of the algorithm is shown in FIGS. 35A-35B.
  • FIGS. 35A-35B show the algorithm for surface merging.
  • point clouds of all segmented surfaces are loaded.
  • plane coefficients of all segmented surfaces are loaded.
  • a list which will contain pairs of indices of surfaces that should be merged is created.
  • step 1412 If the answer to step 1412 is yes, the pair of surfaces’ indices is added to the merging list at step 1414 and the process continues at step 1410. If the answer to step 1410 is yes, the process continues at step 1408. If the answer to step 1408 is yes, it is determined if all surfaces have been checked for merging by adjacency criterion at step 1416. If the answer to step 1416 is no, it is determined if the surface contains a small enough number of points at step 1418. If the answer to step 1418 is no, the process continues at step 1416. If the answer to step 1418 is yes, then it is determined if the surface has been checked against all other surfaces at step 1420.
  • step 1420 determines if any two points (one from each plane) are close enough (neighbors) at step 1422. If the answer to step 1422 is no, then the process continues at step 1420. If the answer to step 1422 is yes, then the pair of surfaces’ indices are added to the merging list at step 1424 and the process continues at step 1420. If the answer to step 1420 is yes, then the process continues at step 1416. If the answer to step 1416 is yes, then duplicate pairs are removed from the merging list at step 1426. Then at step 1428, pairs of surfaces already separated in previous processing steps are removed from the merging list. Then at step 1430, it is determined if all pairs have been merged.
  • step 1434 it is determined if there is another pair containing a surface from the current pair. If the answer to step 1434 is no, then the two surfaces are merged - add the points of the smaller surface to the larger one at step 1436. Then at step 1438, the index of the smaller surface is added to the list of surfaces to be removed. Then at step 1440, surfaces which have been merged into another surface are removed from the list of surfaces. Then at step 1442, a new list of surfaces is returned. If the answer to step 1434 is yes, then it is determined if there is a third pair such that the three pairs contain only three surfaces at step 1444.
  • step 1444 If the answer to step 1444 is no, then the surface present in both pairs is found at step 1446. Then at step 1448, the centroid of all three surfaces is calculated. Then at step 1450, the common surface is merged with the surface which has a centroid closer to the common surface. Then at step 1452, the pair containing the farther centroid from the merging list is removed. Then at step 1454, the indices of the surfaces of the discarded pair is added to the list of surfaces to be removed. Then at step 1456, the index of the merged surface is added to the list of surfaces to be removed and the process continues at step 1440. If the answer to step 1444 is yes, the three surfaces are merged - add the points of the smaller surfaces to the largest one at step 1458. Then at step 1460, the indices of the smaller surfaces are added to the list of surfaces to be removed and the process continues at step 1440. If the answer to step 1430 is yes, then the process continues to step 1440.
  • Step 1112 is described herewith in detail.
  • Version 1 of step 112 of reassigning outliers is as follows.
  • a starting threshold (distance_thr) for the distance between the point and a plane is set at 3.0.
  • the distance of every outlier rejected by RANSAC to the segmented surfaces is calculated. If the distance is below the threshold, the outlier has to be close enough to any point of the surface to be assigned to it.
  • the outliers are assigned to the surfaces unchanged.
  • the steps are repeated increasing the distance threshold to 10.
  • the described algorithm is implemented in a reassign_outliers_vl.py file, and the flow chart of the algorithm is shown in FIG. 36.
  • FIG. 36 shows Version 1 of the algorithm for reassigning outliers.
  • an input point cloud is loaded.
  • segmented surfaces are loaded.
  • a list which contains all points declared outliers by RANSAC is created.
  • a starting distance threshold (from point to plane) is set.
  • step 1514 If the answer to step 1514 is no, then the next outlier is obtained at step 1516. Then at step 1518, it is determined if the distance from the point to every surface is calculated. If the answer to step 1518 is no, then the distance between the point and the next surface is calculated at step 1520. Then at step 1522, it is determined if the distance is less than the threshold. If the answer to step 1522 is no, then the process continues at step 1518. If the answer to step 1522 is yes, then the point is added to the surface at step 1524 and the process continues at step 1518. If the answer to step 1518 is yes, then the process continues at step 1514. If the answer to step 1514 is yes, then the process continues at step 1510.
  • Version 2 of step 1112 of reassigning outliers is as follows.
  • the first version of the script adds only the unchanged points falsely excluded from the surface, so an additional experiment is made in an effort to increase the output point density of the cloud to improve subsequent processing methods. Therefore, the second version additionally uses the true outliers outputted from the cloud pre-filtering step. For every true outlier, a new z coordinate (depth in scene) is computed - median of the z coordinates of the neighbor points.
  • the reassignment process includes one change - the distance threshold from point to plane (distance_thr) remains fixed to three. If a point is close enough to the surface and has a neighbor point in the surface, it is assigned.
  • FIG. 37 An example scene where too many points are rejected of reassigning part of the outliers is shown in FIG. 37.
  • the result of reassigning the outliers is shown in FIG. 38.
  • the described algorithm is implemented in a reassign_outliers_v2.py file, and the flow chart of the algorithm is shown in FIG. 39.
  • An example result of segmented box surfaces is shown in FIG. 40, where the bounding box of each surface is formed based on the left-most, right-most, upper and lower points of the surface.
  • FIG. 39 shows Version 2 of the algorithm for reassigning outliers.
  • an input point cloud is loaded.
  • segmented surfaces are loaded.
  • a list which contains all points declared outliers by RANSAC is created.
  • true outliers points at large distance from any plane, are loaded.
  • step 1610 it is determined if all true outliers have been checked for repositioning. If the answer to step 1610 is no, then the next outlier is obtained at step 1612.
  • a list which will contain the inlier neighbors of the outlier is created.
  • it is determined if the distance to all inliers is calculated.
  • step 1616 determines if the distance between the point and the next inlier is calculated at step 1618. Then at step 1620, it is determined if the distance is less than the threshold. If the answer to step 1620 is no, then the process continues at step 1616. If the answer to step 1620 is yes, then the point is added to the neighborhood list at step 1622 and the process continues at step 1616. If the answer to step 1616 is yes, then it is determined if the neighborhood list is empty at step 1624. If the answer to step 1624 is no, then the z coordinate is replaced with the median z coordinate of the points in the environment at step 1626. Then at step 1628, it is determined if the distance of the point to all surfaces is calculated.
  • step 1630 If the answer to step 1628 is no, then the distance between the point and the next surface is calculated at step 1630. Then at step 1632, it is determined if the distance to all points in the surface is calculated. If the answer to step 1632 is no, then the distance between the point and the next point of the surface is calculated at step 1634. Then at step 1636, it is determined if the distance is less than a threshold. If the answer to step 1636 is no, then the number of neighbors to the current plane is increased at step 1638 and the process continues at step 1632. If the answer to step 1636 is yes, then the process continues at step 1632. If the answer to step 1632 is yes, then the process continues at step 1628.
  • step 1628 If the answer to step 1628 is yes, then the point is assigned to the plane with the most neighbor points at step 1640. Then at step 1642, a list of new surfaces is returned and the process is complete. If the answer to step 1624 is yes, then the process continues at step 1628. If the answer to step 1610 is yes, then the process continues at step 1642.
  • FIG. 41 is a flow chart showing a surface segmentation algorithm.
  • FIG. 41 includes steps 1700-1712.
  • FIG. 41 show an alternate embodiment of a surface segmentation algorithm than that shown in FIG. 25.
  • cloud prefiltering step 1106 is completed after the point cloud is loaded (step 1100), the planes are segmented from the scene (step 1102), and the coplanar disconnected surfaces are separated (step 1104).
  • cloud prefiltering step 1702 is completed after the point cloud is loaded (step 1700) and before the planes are segmented from the scene (step 1704) and the coplanar disconnected surfaces are separated (step 1706).
  • Step 1700 includes loading the point cloud.
  • Step 1704 includes segmenting planes from the scene.
  • Step 1706 includes separating coplanar disconnected surfaces. Steps 1700, 1704, and 1706 are the same as steps 400, 402, and 404 discussed above in reference to FIG. 6 for the first segmentation algorithm.
  • Step 1702 includes cloud prefiltering.
  • the number of outliers is increased due to the different occlusions appearing in each of the images, so the cloud is filtered before processing with RANSAC.
  • Step 1702 is the same as step 1106 discussed above in reference to FIG. 25 for the second segmentation algorithm, it is just performed at a different point in the sequence of steps. Specific details regarding the cloud prefiltering algorithm are discussed with respect to FIG. 29.
  • Step 1708 includes points rearrangement. Because no perspective correction is performed to the input point cloud, some planar surfaces are distorted, which results in some points being wrongly assigned to different surfaces. In some situations, after the plane segmentation phase, some points have more neighboring points belonging to another surface than the one chosen by RANSAC. That results in addition of line like shapes to otherwise rectangular surfaces. Therefore a post-processing step of rearranging the points reassigns each point to the surface in which the point has the largest number of neighbors. Step 1708 is the same as step 1108 discussed above in reference to FIG. 25 for the second segmentation algorithm. Specific details regarding the points rearrangement algorithm are discussed with respect to FIG. 32.
  • Step 1710 includes combining surfaces. To handle the over- segmentation, a surface merging step based on adjacency and similar plane coefficients is applied. Step 1710 is the same as step 1110 discussed above in reference to FIG. 25 for the second segmentation algorithm. Specific details regarding the surface merging algorithm are discussed with respect to FIGS. 35A-35B.
  • Step 1712 includes reassigning outliers. In order to reduce the number of points which are declared outliers, reassignment of outlier points is performed. In this step, the depth value for each outlier is recalculated using the median depth value of the neighbor points before it is returned to the point cloud. Step 1712 is the same as step 1112 discussed above in reference to FIG. 25 for the second segmentation algorithm. Specific details regarding the first version of the reassigning outliers algorithm are discussed with respect to FIG. 36, and specific details regarding the second version of the reassigning outliers algorithm are discussed with respect to FIG. 39.
  • the algorithm was also tested using real range data.
  • the input for the algorithm is a point cloud representing a real scene consisting of randomly placed and rotated boxes (cuboids), cylindrical boxes and bags, where occlusion of objects is present.
  • a 3D point cloud representation of the scene is generated from a stereo image pair of each scene. No perspective correction is applied.
  • An example of the real scenes used for testing is shown in the photo of FIG. 42.
  • the corresponding 3D point cloud of the scene is shown in FIG. 43, where the shade of the points encodes the distance of the points to the cameras.
  • the algorithm When applied to real data acquired from stereo image pairs, the algorithm gives satisfactory results of segmenting the front visible sides of boxes which are represented with a large number of points. Correct plane segmentation is possible for surfaces containing more than 15 points. By reducing the tolerance threshold for the segmentation of planes, the number of wrongly segmented points is reduced and correct segmentation of boxes in the presence of differently shaped packaging in the scene is enabled.
  • the strict threshold used in the RANSAC phase to reduce merging of close surfaces causes the rejection of a lot of points as outliers, even though some of them are caused by the perspective of the scene or imperfections in the shape of a package.
  • By reassigning the outliers which were not rejected in the initial filtering phase information on the edges or top sides of packages is preserved.
  • FIG. 47 An example of segmented boxes in simulated data is shown in the graph of FIG. 47.
  • the front sides of the packages are correctly segmented.
  • the sides of the boxes positioned at an angle to the image plane contain too few points for a reliable decision, and therefore they are rejected.
  • the bag consists of multiple smaller planar surfaces, which causes some of them to be merged with package sides, such as surface S in the graph of FIG. 47.
  • the proposed method based on segmentation of planes and combining them into boxes based on adjacency and orthogonality performs well when applied to simulated scenes containing a relatively small number of boxes are used as the input for segmentation.
  • the algorithm is able to correctly segment boxes with any position and rotation and the occlusions from the unordered packaging.
  • RANSAC is able to reject the true outliers and handle the random noise, but post-processing is required for separating multiple coplanar connected components. Additional information from RGB images can be used to help separate multiple perfectly aligned boxes. Points which are connected to a plane, but belong to another nearby plane can’t be rejected based on the number of neighbors as the solution is not reliable. Those cases are fixed by reassigning the points to the plane with the largest number of neighbor points.

Abstract

L'invention concerne un système destiné au chargement et/ou au déchargement automatiques d'emballages, le système comprenant une machine de chargement et/ou de déchargement automatiques d'emballages, un système de vision qui est configuré pour capturer une image de nuage de points 3D d'une paroi d'emballage, et un dispositif de commande qui est configuré pour traiter l'image de nuage de points 3D à l'aide d'un logiciel de segmentation pour générer une sortie, le dispositif de commande étant configuré pour commander à la machine de chargement et/ou de déchargement automatiques d'emballages, sur la base de la sortie provenant du logiciel de segmentation, de placer un emballage dans la paroi d'emballage et/ou de retirer un emballage de la paroi d'emballage.
PCT/US2019/039863 2018-06-29 2019-06-28 Segmentation de boîtes à partir d'un nuage de points 3d destiné à un déchargement et à un chargement automatiques WO2020006433A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201862692239P 2018-06-29 2018-06-29
US62/692,239 2018-06-29

Publications (1)

Publication Number Publication Date
WO2020006433A1 true WO2020006433A1 (fr) 2020-01-02

Family

ID=68987617

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2019/039863 WO2020006433A1 (fr) 2018-06-29 2019-06-28 Segmentation de boîtes à partir d'un nuage de points 3d destiné à un déchargement et à un chargement automatiques

Country Status (1)

Country Link
WO (1) WO2020006433A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115984195A (zh) * 2022-12-16 2023-04-18 秦皇岛燕大滨沅科技发展有限公司 一种基于三维点云的车厢轮廓检测方法及系统
EP4159643A4 (fr) * 2020-06-12 2023-11-22 Hai Robotics Co., Ltd. Extraction de caisse de chargement et dispositif, système, robot et support d'informations

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6694217B2 (en) * 2001-05-24 2004-02-17 Breakthrough Logistics Corporation Automated system for efficient article storage and self-service retrieval
US20130093852A1 (en) * 2011-10-12 2013-04-18 Board Of Trustees Of The University Of Arkansas Portable robotic device
US20160203387A1 (en) * 2015-01-08 2016-07-14 GM Global Technology Operations LLC Vision system and analytical method for planar surface segmentation
US20180037412A1 (en) * 2016-08-04 2018-02-08 Opex Corporation Automated storage and retrieval system with detector for detecting items extending beyond dimensional threshold

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6694217B2 (en) * 2001-05-24 2004-02-17 Breakthrough Logistics Corporation Automated system for efficient article storage and self-service retrieval
US20130093852A1 (en) * 2011-10-12 2013-04-18 Board Of Trustees Of The University Of Arkansas Portable robotic device
US20160203387A1 (en) * 2015-01-08 2016-07-14 GM Global Technology Operations LLC Vision system and analytical method for planar surface segmentation
US20180037412A1 (en) * 2016-08-04 2018-02-08 Opex Corporation Automated storage and retrieval system with detector for detecting items extending beyond dimensional threshold

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4159643A4 (fr) * 2020-06-12 2023-11-22 Hai Robotics Co., Ltd. Extraction de caisse de chargement et dispositif, système, robot et support d'informations
CN115984195A (zh) * 2022-12-16 2023-04-18 秦皇岛燕大滨沅科技发展有限公司 一种基于三维点云的车厢轮廓检测方法及系统
CN115984195B (zh) * 2022-12-16 2023-09-29 秦皇岛燕大滨沅科技发展有限公司 一种基于三维点云的车厢轮廓检测方法及系统

Similar Documents

Publication Publication Date Title
AU2016243617B2 (en) Imager for detecting visual light and infrared projected patterns
JP7433609B2 (ja) 物体識別のための方法および計算システム
KR101772367B1 (ko) 스테레오와 구조화된 광 처리의 결합
US9327406B1 (en) Object segmentation based on detected object-specific visual cues
CA3066502A1 (fr) Determination de positions et d'orientations d'objets
JP7394347B2 (ja) 候補エッジを処理するための方法及び計算システム
WO2020006433A1 (fr) Segmentation de boîtes à partir d'un nuage de points 3d destiné à un déchargement et à un chargement automatiques
CN114897974B (zh) 一种目标物空间定位方法、系统、存储介质及计算机设备
JP7272568B2 (ja) ロボット運動計画およびリポジトリ検出を実行するための方法および計算システム
CN112734783B (zh) 用于处理候选边缘的方法和计算系统
US11900652B2 (en) Method and computing system for generating a safety volume list for object detection
CN111191083B (zh) 用于对象标识的方法和计算系统
KR20220111074A (ko) 적재된 객체 인식 장치 및 객체 인식 방법
EP4332900A1 (fr) Détection automatique de bac pour des applications robotiques
JP7191352B2 (ja) 物体検出を実施するための方法および計算システム
WO2021246476A1 (fr) Procédé et système informatique pour effectuer ou faciliter une détection de bord physique
WO2023140266A1 (fr) Dispositif de saisie et programme de génération d'image
JP2021071420A (ja) 情報処理装置、情報処理方法、プログラム、システム、物品の製造方法、計測装置及び計測方法
CN116157828A (zh) 机器学习装置
Park et al. Depth Evaluation from Pattern Projection Optimized for Automated Electronics Assembling Robots

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19826549

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19826549

Country of ref document: EP

Kind code of ref document: A1