CN114096331A - Method and computing system for performing or facilitating physical edge detection - Google Patents

Abstract

A method and computing system for facilitating edge detection is presented. The computing system may include at least one processing circuit configured to receive image information representing a set of objects in a field of view of a camera and identify, from the image information, a plurality of candidate edges associated with the set of objects. The at least one processing circuit may also determine whether the image information satisfies a defined darkness condition at a first candidate edge when the plurality of candidate edges includes the first candidate edge formed based on a boundary between the first image region and the second image region. The at least one processing circuit may also select a subset of the plurality of candidate edges to form a selected subset of candidate edges for representing physical edges of a set of objects.

Description

Method and computing system for performing or facilitating physical edge detection

Cross reference to related applications

The present application claims priority from U.S. application No. 17/331,878 entitled METHOD AND COMPUTING SYSTEM FOR PERFORMING OR facilitating physical edge DETECTION (METHOD AND COMPUTING SYSTEM FOR PERFORMANCING OR FACILITATING PHYSICAL EDGE DETECTION), filed on 27/5/2021, which claims priority from U.S. provisional application No. 63/034,403 entitled "robot System with Vision MECHANISM (ROBOTIC SYSTEM WITH VISION MECHANISM"), filed on 4/6/2020, which are incorporated herein by reference in their entirety.

Technical Field

The present disclosure relates to computing systems and methods for performing or facilitating physical edge detection.

Background

As automation becomes more prevalent, robots are being used in more environments, such as in warehousing and retail environments. For example, a robot may be used to interact with objects in a warehouse. The movement of the robot may be fixed or may be based on input, such as information generated by sensors in the warehouse.

Disclosure of Invention

One aspect of the present disclosure relates to a computing system or a method performed by the computing system. The computing system may include a communication interface and at least one processing circuit. The communication interface may be configured to communicate with the robot and with a camera having a camera field of view. The at least one processing circuit may be configured to, when a group of objects is in the camera field of view, perform the following: receiving image information representing the set of objects, wherein the image information is generated by a camera; identifying, from the image information, a plurality of candidate edges associated with the set of objects, wherein the plurality of candidate edges are or comprise respective image locations or sets of physical locations forming respective candidates for representing physical edges of the set of objects; when the plurality of candidate edges includes a first candidate edge formed based on a boundary between a first image area and a second image area, wherein the first image area is darker than the second image area, determining whether the image information satisfies a defined darkness condition at the first candidate edge, wherein the first image area and the second image area are respective areas described by the image information; selecting a subset of the plurality of candidate edges to form a selected candidate edge subset for representing physical edges of the set of objects, wherein the selecting comprises: determining whether to reserve the first candidate edge as a candidate for representing at least one of the physical edges of the set of objects by including the first candidate edge in the selected subset of candidate edges based on whether the image information satisfies a defined darkness condition at the first candidate edge.

Drawings

1A-1D illustrate a system for performing or facilitating physical edge detection consistent with embodiments herein.

Fig. 2A-2D provide block diagrams illustrating a computing system configured to perform or facilitate physical edge detection consistent with embodiments herein.

3A-3C illustrate environments in which physical edge detection may be performed according to embodiments herein.

Fig. 4 provides a flow diagram illustrating a method of processing candidate edges according to embodiments herein.

Fig. 5A and 5B illustrate example image information from which candidate edges may be identified according to embodiments herein.

Fig. 6A and 6B illustrate candidate edges identified from image information according to embodiments herein.

Fig. 7A-7C illustrate candidate edges identified from image information according to embodiments herein.

Fig. 8 illustrates an environment in which an image may be generated based on diffuse reflection in which incident light is reflected from an object in a camera field of view according to embodiments herein.

Fig. 9A-9C illustrate an image intensity profile around a candidate edge, and fig. 9D illustrates depth values of an area around a position corresponding to the candidate edge according to an embodiment herein.

10A-10C illustrate image intensity distributions around a candidate edge according to embodiments herein.

11A-11C illustrate depth values for locations around a physical edge according to embodiments herein, while FIG. 11D illustrates an image intensity distribution of an image representing the physical edge according to embodiments herein.

12A-12C illustrate image intensity distributions of images representing an object having image areas that are darker than adjacent image areas according to embodiments herein.

13A-13C illustrate image intensity distributions of images representing two objects separated by a physical gap and in which one object is darker than the other object, according to embodiments herein.

Detailed Description

One aspect of the present disclosure relates to using image information representing a set of objects to detect or otherwise identify physical edges of the set of objects. For example, the 2D image may represent a set of boxes and may include candidate edges that potentially represent physical edges of the set of boxes. The computing system may use the candidate edges in the image information to distinguish a single object represented in the image information. In some cases, the computing system may use information identifying a single cassette to control robotic interactions involving the single cassette. For example, the robotic interaction may include an unstacking operation in which an end effector device of the robot approaches one of the objects, picks up the object, and moves the object to a destination location.

In some cases, the 2D image or other image information may include candidate edges that are false edges, which may be candidate edges that do not correspond to any actual physical edges of objects in the field of view of the camera. Accordingly, one aspect of the present disclosure relates to evaluating candidate edges to determine a confidence level that the candidate edges correspond to actual physical edges rather than false edges. In embodiments, such a determination may be based on an expectation or prediction as to how likely certain physical edges will appear in the image. More particularly, such a determination may be based on an expectation that if a physical edge is associated with a physical gap between the object (e.g., the physical edge forms one side of the physical gap), such a physical gap may appear very dark in the image and/or may have an image intensity distribution characterized by a spike (spiked) reduction in image intensity in an image region corresponding to the physical gap. Thus, the method or computing system of the present disclosure may operate based on the expectation that a physical gap between objects (particularly a narrow physical gap) will likely be represented by an image having certain characteristics related to how dark the physical gap may appear in the image. Such image characteristics or features may be referred to as dark priors (dark priors), and the present disclosure may relate to detecting dark priors, where the presence of a dark prior may increase a confidence level as to whether a candidate edge corresponds to an actual physical edge.

In an embodiment, a method or system of the present disclosure may determine whether an image satisfies a defined darkness condition at a candidate edge, where the defined darkness condition may relate to detecting a dark prior. More particularly, the defined darkness conditions may be defined by a darkness threshold criterion and/or a spike intensity distribution criterion, which will be discussed in more detail below. In this embodiment, if the computing system or method determines that the image satisfies the defined darkness condition at the candidate edge, there may be a greater level of confidence that the candidate edge corresponds to an actual physical edge (such as a physical edge that forms one side of a physical gap between two objects). In some cases, the candidate edge is more likely to be a false edge if the image does not satisfy the defined darkness condition at the candidate edge.

An aspect of the present disclosure relates to using 2D image information to compensate for limitations in 3D image information and vice versa. For example, if multiple objects (such as two or more boxes) are placed next to each other and separated by a narrow physical gap, the 3D image information may not have a sufficiently high level of resolution to capture or otherwise represent the physical gap. Thus, the 3D image information may have limitations in the ability to distinguish a single object of the multiple objects, especially if the multiple objects have the same depth relative to the camera that generated the 3D image information. In such an example, the physical gaps between the plurality of objects may be represented in the 2D image information. More particularly, the physical gap may be represented by an image area that may satisfy a defined darkness condition. Candidate edges associated with such image regions may thus represent physical edges of objects at a high confidence level. In such a case, the candidate edges in the 2D image information may be useful in distinguishing individual objects in a set of objects. Thus, in some cases, the 2D image information may provide enhanced ability to distinguish individual objects.

In some cases, the 3D image information may compensate for limitations in the 2D image information. For example, a 2D image may not meet a defined darkness condition at some candidate edge in the 2D image. In such a case, the candidate edge may have a low confidence level for any actual physical edge object corresponding to the camera field of view. If the candidate edge in the 2D image information corresponds to a candidate edge in the 3D image information, the 3D image information may be used to compensate for this limitation in the 2D image information. More particularly, a candidate edge in the 2D image information may be mapped to a location or set of locations in the 3D image information where there is a sharp change in depth. In such a case, the 3D image information may be used to increase the confidence level that the candidate edge in the 2D image information corresponds to the actual physical edge.

In an embodiment, the 3D image information may be used to identify a surface (e.g., a top surface) of the object, and the candidate edge may be identified based on a location where a transition exists between the two surfaces. For example, the surface may be identified based on a set of locations having corresponding depth values in the 3D image information that deviate from each other by no more than a defined measurement variance threshold. The defined measurement variance threshold may describe the effects of imaging noise, manufacturing tolerances, or other factors that may introduce random fluctuations in depth measurements in the 3D image information. The identified surface may be associated with a depth value that is an average of the respective depth values. In some implementations, the candidate edges may be detected in the 3D image information based on identifying depth transitions between two surfaces identified in the 3D image information that are greater than a defined depth difference threshold.

Fig. 1A illustrates a system 1000 for performing or facilitating physical edge detection, which may involve using image information representing one or more objects to detect or otherwise identify physical edges of the one or more objects. More particularly, the system 1000 may include a computing system 1100 and a camera 1200. In this example, the camera 1200 may be configured to generate image information that describes or otherwise represents the environment in which the camera 1200 is located, or more specifically the environment in the field of view of the camera 1200 (also referred to as the camera field of view). The environment may be, for example, a warehouse, a manufacturing facility, a retail space, or some other location. In such a case, the image information may represent an object, such as a box, cabinet, box, crate, or other container, located at such a location. The system 1000 may be configured to generate, receive, and/or process image information, such as by distinguishing individual objects in a camera field of view using the image information to perform object recognition or object registration based on the image information, and/or to perform robotic motion planning based on the image information, as discussed in more detail below (the terms "and/or" and "or" are used interchangeably in this disclosure). Robot motion planning may be used, for example, to control a robot at a site to facilitate robot interaction between the robot and a container or other object. The computing system 1100 and the camera 1200 may be located in the same facility or may be remote from each other. For example, the computing system 1100 may be part of a cloud computing platform hosted in a data center remote from a warehouse or retail space, and may communicate with the camera 1200 via a network connection.

In an embodiment, the camera 1200 (also referred to as an image sensing device) may be a 2D camera and/or a 3D camera. For example, fig. 1B shows a system 1000A (which may be an embodiment of system 1000), the system 1000A including a computing system 1100 and a camera 1200A and a camera 1200B, both of which may be embodiments of camera 1200. In this example, the camera 1200A may be a 2D camera configured to generate 2D image information that includes or forms a 2D image describing the visual appearance of the environment in the field of view of the camera. The camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or spatial structure sensing device) configured to generate 3D image information, the 3D image information comprising or forming spatial structure information about the environment in the field of view of the camera. The spatial structure information may include depth information (e.g., a depth map) that describes respective depth values relative to various positions of the camera 1200B, such as positions on the surfaces of various objects in the field of view of the camera 1200. These locations in the field of view of the camera or on the surface of the object may also be referred to as physical locations. In this example, the depth information may be used to estimate how the objects are spatially arranged in a three-dimensional (3D) space. In some cases, the spatial structure information may include or may be used to generate a point cloud that describes locations on one or more surfaces of objects in the field of view of the camera 1200B. More specifically, the spatial structure information may describe various locations on the structure of the object (also referred to as an object structure).

In an embodiment, the system 1000 may be a robot operating system for facilitating robotic interaction between a robot and various objects in the environment of the camera 1200. For example, fig. 1C shows a robot operating system 1000B, which may be an embodiment of the system 1000/1000a of fig. 1A and 1B. The robot operating system 1000B may include a computing system 1100, a camera 1200, and a robot 1300. As described above, the robot 1300 may be used to interact with one or more objects in the environment of the camera 1200 (such as with a box, crate, cabinet, or other container). For example, robot 1300 may be configured to pick containers from one location and move them to another location. In some cases, robot 1300 may be used to perform an unstacking operation in which a group of containers or other objects is unloaded and moved to, for example, a conveyor. In some implementations, the camera 1200 may be attached to the robot 1300, such as to a robotic arm of the robot 1300. In some implementations, the camera 1200 may be separate from the robot 1300. For example, the camera 1200 may be mounted to a ceiling of a warehouse or other structure and may remain fixed relative to the structure.

In an embodiment, the computing system 1100 of fig. 1A-1C may form or be part of a robot control system (also referred to as a robot controller) that is part of the robot operating system 1000B. The robot control system may be a system configured to, for example, generate movement commands for the robot 1300 (such as robot-interactive movement commands for controlling robot interaction between the robot 1300 and a container or other object). In such embodiments, the computing system 1100 may be configured to generate such commands based on, for example, image information generated by the cameras 1200/1200 a/1200B. For example, the computing system 1100 may be configured to determine a motion plan based on the image information, where the motion plan may be intended for grasping or otherwise picking up an object. The computing system 1100 may generate one or more robot-interactive movement commands to execute the motion plan.

In embodiments, the computing system 1100 may form or be part of a vision system. The vision system may be a system that generates, for example, visual information describing the environment in which the robot 1300 is located, or more specifically, the environment in which the camera 1200 is located. The visual information may comprise 3D image information and/or 2D image information as discussed above, or some other image information. In some cases, if the computing system 1100 forms a vision system, the vision system may be part of the robotic control system discussed above, or may be separate from the robotic control system. If the vision system is separate from the robot control system, the vision system may be configured to output information describing the environment in which the robot 1300 is located. The information may be output to a robotic control system, which may receive such information from the vision system and, based on the information, execute a motion plan and/or generate robot interactive movement commands.

In embodiments, the computing system 1100 may communicate with the camera 1200 and/or the robot 1300 via a direct connection, such as a connection provided via a dedicated wired communication interface, such as an RS-232 interface, a Universal Serial Bus (USB) interface, and/or via a local computer bus, such as a Peripheral Component Interconnect (PCI) bus. In embodiments, the computing system 1100 may communicate with the camera 1200 and/or with the robot 1300 via a network. The network may be any type and/or form of network, such as a Personal Area Network (PAN), a Local Area Network (LAN) (e.g., an intranet), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), or the internet. The network may utilize different technologies and protocol layers or protocol stacks including, for example, ethernet protocol, internet protocol suite (TCP/IP), ATM (asynchronous transfer mode) technology, SONET (synchronous optical network) protocol, or SDH (synchronous digital hierarchy) protocol.

In embodiments, the computing system 1100 may communicate information directly with the camera 1200 and/or with the robot 1300, or may communicate via an intermediate storage device or more generally via an intermediate non-transitory computer-readable medium. For example, fig. 1D illustrates a system 1000C, which may be an embodiment of the system 1000/1000a/1000B, the system 1000C including a non-transitory computer-readable medium 1400, which non-transitory computer-readable medium 1400 may be external to the computing system 1100, and may serve as a repository or external buffer for storing image information, e.g., generated by the camera 1200. In such an example, the computing system 1100 may retrieve or otherwise receive image information from the non-transitory computer-readable medium 1400. Examples of the non-transitory computer-readable medium 1400 include an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination thereof. The non-transitory computer readable medium may form, for example, a computer diskette, a Hard Disk Drive (HDD), a solid state drive (SDD), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), and/or a memory stick.

As described above, the camera 1200 may be a 3D camera and/or a 2D camera. The 2D camera may be configured to generate a 2D image, such as a color image or a grayscale image. The 3D camera may be, for example, a depth sensing camera, such as a time of flight (TOF) camera or a structured light camera, or any other type of 3D camera. In some cases, the 2D camera and/or the 3D camera may include an image sensor, such as a Charge Coupled Device (CCD) sensor and/or a Complementary Metal Oxide Semiconductor (CMOS) sensor. In embodiments, the 3D camera may include a laser, a LIDAR device, an infrared device, a light/dark sensor, a motion sensor, a microwave detector, an ultrasound detector, a RADAR detector, or any other device configured to capture depth information or other spatial structure information.

As described above, the image information may be processed by the computing system 1100. In embodiments, the computing system 1100 may include or be configured as a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smartphone, a tablet computing device, and/or any other computing system. In embodiments, any or all of the functionality of the computing system 1100 may be performed as part of a cloud computing platform. Computing system 1100 may be a single computing device (e.g., a desktop computer), or may include multiple computing devices.

Fig. 2A provides a block diagram that illustrates an embodiment of a computing system 1100. The computing system 1100 includes at least one processing circuit 1110 and a non-transitory computer-readable medium (or media) 1120. In an embodiment, the processing circuitry 1110 includes one or more processors, one or more processing cores, a programmable logic controller ("PLC"), an application specific integrated circuit ("ASIC"), a programmable gate array ("PGA"), a field programmable gate array ("FPGA"), any combination thereof, or any other processing circuitry.

In an embodiment, the non-transitory computer-readable medium 1120 that is part of the computing system 1100 may be an alternative or addition to the intermediate non-transitory computer-readable medium 1400 discussed above. The non-transitory computer-readable medium 1120 may be a storage device, such as an electronic, magnetic, optical, electromagnetic, semiconductor storage device, or any suitable combination thereof, such as, for example, a computer diskette, a Hard Disk Drive (HDD), a Solid State Drive (SSD), a Random Access Memory (RAM), a Read Only Memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, any combination thereof, or any other storage device. In some cases, the non-transitory computer-readable medium 1120 may include a plurality of storage devices. In certain implementations, the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200 and received by the computing system 1100. In some cases, the non-transitory computer-readable medium 1120 may store one or more object recognition templates for performing object recognition operations. The non-transitory computer-readable medium 1120 may alternatively or additionally store computer-readable program instructions that, when executed by the processing circuit 1110, cause the processing circuit 1110 to perform one or more methods described herein, such as the operations described with respect to fig. 4.

Fig. 2B depicts a computing system 1100A that is an embodiment of computing system 1100 and that includes a communication interface 1130. The communication interface 1130 may be configured to receive image information generated by the camera 1200 of fig. 1A-1D, for example. The image information may be received via the intermediary non-transitory computer-readable medium 1400 or network discussed above, or via a more direct connection between the camera 1200 and the computing system 1100/1100 a. In an embodiment, the communication interface 1130 may be configured to communicate with the robot 1300 of fig. 1C. If the computing system 1100 is external to the robotic control system, the communication interface 1130 of the computing system 1100 may be configured to communicate with the robotic control system. The communication interface 1130 may also be referred to as a communication component or communication circuitry, and may include, for example, communication circuitry configured to perform communications via wired or wireless protocols. By way of example, the communication circuit may include an RS-232 port controller, a USB controller, an Ethernet controller, a USB interface, a USB,

A controller, a PCI bus controller, any other communication circuit, or a combination thereof.

In an embodiment, as shown in fig. 2C, the non-transitory computer-readable medium 1120 may store edge detection information 1126, the edge detection information 1126 may describe a plurality of candidate edges identified from image information generated by the camera 1200. As discussed in more detail below, if the image information represents a set of objects, each of the candidate edges may be or may form a candidate for at least one of a plurality of physical edges representing the set of objects. In some cases, the computing system 1100/1100A/1100B may determine whether a particular candidate edge in the edge detection information 1126 should be used to represent at least one of the physical edges of the set of objects. Such a determination may involve evaluating a confidence level associated with whether the candidate edge actually represents a physical edge rather than a false edge. In one example, such an evaluation may be based on whether the candidate edge is associated with an image characteristic resulting from representing a physical edge. Such characteristics may be associated with image features known as dark priors, which will be discussed in more detail below. In some cases, the computing system 1100 may select a subset of candidate edges from the plurality of candidate edges that have a sufficiently high confidence level that actually represents a physical edge of the set of objects, while candidate edges excluded from the subset may not have a sufficiently high confidence level that actually represents a physical edge of the set of objects. Thus, if the computing system 1100/1100A/1100B determines to use a particular candidate edge to represent at least one of the physical edges, the computing system may include the candidate edge in the subset. If the computing system 1100/1100A/1100B determines not to use a particular candidate edge to represent at least one of the physical edges, the computing system may determine not to include the candidate edge in the subset. Candidate edges not included in the subset may be filtered out from the edge detection information 1126 or, more generally, excluded from further consideration as candidates for at least one of the physical edges representing a set of objects.

In an embodiment, the processing circuit 1110 may be programmed by one or more computer-readable program instructions stored on the non-transitory computer-readable medium 1120. For example, fig. 2D illustrates a computing system 1100C that is an embodiment of the computing system 1100/1100a/1100B in which the processing circuitry 1110 is programmed by one or more modules including a physical edge detection module 1125, an object identification/registration module 1128, and/or an exercise planning module 1129.

In an embodiment, the physical edge detection module 1125 may be configured to determine which candidate edge(s) should be used to represent physical edges of a set of objects from a plurality of candidate edges occurring in image information representing the set of objects. In some implementations, the physical edge detection module 1125 may make such a determination based on whether a defined darkness condition is satisfied and/or a depth discontinuity condition is satisfied, as discussed in more detail below. In some cases, the physical edge detection module 1125 may also be configured to identify a plurality of candidate edges from the image information. In some cases, the physical edge detection module 1125 may be configured to perform image segmentation (e.g., point cloud segmentation), which may involve distinguishing a single object represented by image information. For example, the module 1125 may extract or otherwise identify an image segment (also referred to as an image portion) representing image information of an object in a set of objects. In some implementations, image segmentation may be performed based on, for example, the module 1125 having determined candidate edges that should be used to represent physical edges of a set of objects.

In embodiments, the object identification/registration module 1128 may be configured to perform an object identification operation or an object registration module based on the results from the physical edge detection module 1125. For example, if the physical edge detection module 1125 identifies an image segment representing an object in a set of objects, the object identification/registration module 1128 may be configured to, for example, determine whether the image segment sufficiently matches an object identification template, and/or generate a new object identification template based on the image segment.

In an embodiment, the movement plan module 1129 may be configured to execute a robot movement plan based on the results of the physical edge detection module 1125 and/or based on the results of the object recognition/registration module 1128. As described above, a robot motion plan may be used for robot interaction between a robot (e.g., 1300) and at least one object of a set of objects. In some cases, a robot motion plan may involve, for example, determining the movement of a component of the robot (e.g., an end effector device) for picking up an object, and/or the trajectory that the component is to follow after picking up the object.

In various embodiments, the terms "computer-readable instructions" and "computer-readable program instructions" are used to describe software instructions or computer code that are configured to perform various tasks and operations. In various embodiments, the term "module" broadly refers to a collection of software instructions or code configured to cause the processing circuit 1110 to perform one or more functional tasks. When a processing circuit or other hardware component is executing a module or computer readable instructions, the module and computer readable instructions may be described as performing various operations or tasks.

3A-3C illustrate example environments in which processing of candidate edges, or more particularly physical edge detection, may be performed. More particularly, fig. 3A depicts an environment with a system 3000 (which may be an embodiment of the systems 1000/1000a/1000B/1000C of fig. 1A-1D) including a computing system 1100, a robot 3300, and a camera 3200. The camera 3200 may be an embodiment of the camera 1200 and may be configured to generate image information representing a scene in the camera field of view 3210 of the camera 3200, or more specifically objects in the camera field of view 3210, such as objects 3510, 3520, 3530, 3540, and 3550. In one example, each of the objects 3510-3540 can be, for example, a container such as a box or crate, and the object 3550 can be, for example, a tray having the container disposed thereon.

Objects 3510-3540 are also depicted in fig. 3B, which fig. 3B shows the physical edges of the objects in more detail. More particularly, the figure shows physical edges 3510A-3510D of the top surface of object 3510, physical edges 3520A-3520D of the top surface of object 3520, physical edges 3530A-3530D of the top surface of object 3530, and physical edges 3540A-3540D of the top surface of object 3540. The physical edges in FIG. 3B (e.g., 3510A-3510D, 3520A-3520D, 3530A-3530D, and 3540A-3540D) can be the outer edges of the respective top surfaces of the object 3510 and 3540. In some cases, a physical edge of the surface of the object (e.g., 3510A-3510D) may define the contour of the surface. If an object forms a polyhedron (e.g., a cube) with multiple non-coplanar surfaces (also referred to as faces), the physical edges of one surface may form boundaries where that surface intersects another surface of the object.

In an embodiment, an object in the camera field of view may have visual details (also referred to as visible details) such as visual markers on an outer surface of the object. For example, as shown in fig. 3A and 3B, objects 3510, 3520, 3530, 3540 can have visual indicia 3512, 3522, 3532, 3542, respectively, that are printed or otherwise disposed on respective exterior (e.g., top) surfaces of the object 3510 and 3540. As examples, the visual indicia may include visible lines (e.g., straight lines or curved lines), visible shapes (such as polygons), visual patterns, or other visual indicia. In some cases, a visual indicia (e.g., a visible line) may form or appear as part of a symbol or a figure on the outer surface of the object. The symbol may include, for example, a logo or written character (e.g., an alphanumeric character). In some cases, the visual details on the outer surface of the container or other object may be formed by the outline of a layer of material (e.g., a strip of wrapping tape or a mailing label) disposed on the outer surface of the container.

In an embodiment, the system 3000 of fig. 3A may include one or more light sources, such as light source 3600. The light source 3600 may be, for example, a Light Emitting Diode (LED), a halogen lamp, or any other light source, and may be configured to emit visible light, infrared radiation, or any other form of light toward the surface of the object 3510-3550. In some implementations, the computing system 1100 may be configured to communicate with the light source 3600 to control when the light source 3600 is activated. In other implementations, the light source 3600 may operate independently of the computing system 1100.

In an embodiment, as shown in fig. 3C, system 3000 may include multiple cameras, including camera 3200A (which may be an embodiment of camera 1200A) having camera field of view 3210A, and including camera 3200B (which may be an embodiment of camera 1200B) having camera field of view 3210B. The camera 3200A may be, for example, a 2D camera configured to generate 2D images or other 2D image information, while the camera 3200B may be, for example, a 3D camera configured to generate 3D image information. The 2D image (e.g., a color image or a grayscale image) may describe the appearance of one or more objects (such as object 3510 and 3550) in the camera field of view 3210/3210 a. For example, the 2D image may capture or otherwise represent visual details, such as visual markers 3512 and 3542 disposed on respective exterior surfaces (e.g., top surfaces) of the object 3510 and 3540, and/or contours of those exterior surfaces. In an embodiment, the 3D image information may describe a structure of one or more of the objects 3510-3550, where the structure of the objects may also be referred to as an object structure or a physical structure of the objects. For example, the 3D image information may include a depth map, or more generally depth information, which may describe respective depth values for various locations in the camera field of view 3210/3210B relative to the camera 3200B or relative to some other reference point. The locations corresponding to the respective depth values may be locations (also referred to as physical locations) on various surfaces in the camera field of view 3210/3210B, such as locations on respective top surfaces of the objects 3510 and 3550. In some cases, the 3D image information may include a point cloud that may include a plurality of 3D coordinates that describe various locations on one or more exterior surfaces of the object 3510 and 3550 or some other object in the object field of view 3210/3210B.

In the example of fig. 3A and 3B, robot 3300 (which may be an embodiment of robot 1300) may include a robot arm 3320, the robot arm 3320 having one end attached to a robot base 3310 and having another end, such as a robot hand grip, attached to or formed by an end effector device 3330. Robot base 3310 may be used to mount robot arm 3320, while robot arm 3320, or more specifically end effector device 3330, may be used to interact with one or more objects (e.g., 3510/3520/3530/3540) in the environment of robot 3300. The interaction (also referred to as robotic interaction) may include, for example, grasping or otherwise picking up at least one of the objects 3510 and 3540. For example, the robotic interaction may be part of an unstacking operation in which the robot 3300 is used to pick up objects 3510 and 3540 (e.g., boxes) from objects 3550 (e.g., pallets or other platforms) and move the objects 3510 and 3540 to a destination location.

As discussed above, one aspect of the present disclosure relates to performing or facilitating detection of one or more physical edges of a set of objects (such as a set of boxes) based on image information representing the one or more objects. FIG. 4 depicts a flow diagram of an example method 4000 for performing or facilitating physical edge detection, or more particularly for determining whether a candidate edge should be used to represent at least one of a set of physical edges of an object. More particularly, the method may involve receiving image information having candidate edges that may represent physical edges or may be false edges. The false edge may be a candidate edge representing, for example, a visible line or other visual marker appearing on the surface of one object in a set of objects. The visual indicia may have an appearance similar to the physical edges but not actually corresponding to any physical edges. Thus, in embodiments, method 4000 may be used to assess a confidence level or likelihood of whether a candidate edge corresponds to an actual physical edge or whether a candidate edge is likely to be a false edge. If the candidate edge is likely to be a false edge, and/or does not have a sufficiently high confidence level corresponding to an actual physical edge, then method 4000 may filter out or more generally exclude the candidate edge from further consideration for any physical edge representing a set of objects in embodiments.

In an embodiment, the method 4000 may be performed by, for example, the computing system 1100 of fig. 2A-2D or 3A or 3C, or more specifically by the at least one processing circuit 1110 of the computing system 1100. In some cases, the at least one processing circuit 1100 may perform the method 4000 by executing instructions stored on a non-transitory computer-readable medium (e.g., 1120). For example, the instructions may cause processing circuitry 1110 to perform one or more of the modules illustrated in fig. 2D that may perform method 4000. As an example, one or more of steps 4002 and 4008 discussed below may be performed by the physical edge detection module 1125. If the method 4000 includes steps to perform object identification and/or object registration, the steps may be performed by, for example, the object identification/registration module 1128. If the method 4000 involves planning robot interactions or generating robot-interactive movement commands, such steps may be performed by, for example, the motion planning module 1129. In embodiments, method 4000 may be performed in an environment where computing system 1100 communicates with robots and cameras (such as robot 3300 and cameras 3200/3200a/3200B in fig. 3A and 3C) or any other camera or robot discussed in this disclosure. In some cases, such as shown in fig. 3A and 3C, a camera (e.g., 3200) may be mounted to a fixed structure (e.g., a ceiling of a room). In other cases, the camera may be mounted on a robotic arm (e.g., 3320), or more specifically on an end effector device (e.g., 3330) of a robot (e.g., 3300).

In an embodiment, one or more steps of method 4000 may be performed when a group of objects (e.g., 3510-3550) is currently in the camera field of view (e.g., 3210/3210A/3210B) of the camera (e.g., 3200/3200A/3200B). For example, one or more steps of method 4000 may be performed immediately after a set of objects is in the camera field of view (e.g., 3210/3210a/3210B), or more generally, while a set of objects is in the camera field of view. In some cases, one or more steps of method 4000 may be performed when a group of objects is already in the field of view of the camera. For example, when a set of objects is in a camera field of view (e.g., 3210/3210a/3210B), a camera (e.g., 3200/3200a/3200B) may generate image information representing the set of objects and may transmit the image information to a computing system (e.g., 1100). When a set of objects is still in the camera field of view, or even when some or all of the set of objects are no longer in the camera field of view, the computing system may perform one or more steps of method 4000 based on the image information.

In an embodiment, method 4000 may begin at step 4002 or otherwise include step 4002, where computing system 1100 receives image information representing a set of objects in a camera field of view (e.g., 3210/3210a/3210B) of a camera (e.g., 3200/3200 a/3200B). Image information may be generated by a camera (e.g., 3200/3200a/3200B) when a set of objects is (or is already) in the camera field of view and may include, for example, 2D image information and/or 3D image information. For example, FIG. 5A depicts 2D image information, or more specifically, a 2D image 5600, generated by the camera 3200/3200A and representing the objects 3510 and 3550 of FIGS. 3A-3C. More specifically, the 2D image 5600 (e.g., a grayscale or color image) may describe the appearance of the object 3510 and 3550 from the viewpoint of the camera 3200/3200 a. In an embodiment, the 2D image 5600 may correspond to a single color channel (e.g., red, green, or blue channel) of a color image. If the camera 3200/3200A is disposed above the object 3510-. In the example of fig. 5A, the 2D image 5600 may include respective portions 5610, 5620, 5630, 5640, and 5650 (also referred to as image portions) representing respective surfaces (e.g., top surfaces) of the object 3510-. In fig. 5A, each image portion of the image portion 5610-. More particularly, the image region may be a region of an image, and the pixel region may be a region of pixels. One or more of the image portions 5610-5550 may capture or otherwise represent visual indicia or other visual details that are visible or appear on the surface of the object. For example, the image portion 5610 may represent the visual indicia 3612 of fig. 3B, which may be printed or otherwise disposed on the top surface of the object 3610.

Fig. 5B depicts an example in which the image information in step 4002 includes 3D image information 5700. More particularly, the 3D image information 5700 may include, for example, a depth map or point cloud indicating respective depth values for various locations on one or more surfaces (e.g., a top surface or other outer surface) of the object 3510 and 3550. For example, the 3D image information 5700 may include a set of locations 5710 on a surface indicating an object 3510₁-5710_nA first portion 5710 (also referred to as an image portion) of corresponding depth values (also referred to as physical locations); indicating a set of locations 5720 on a surface of object 3520₁-5720_nA second portion 5720 of the respective depth values; indicating a set of locations 5730 on a surface of an object 3530₁-5730_nA third portion 5730 of the corresponding depth value; indicating a set of locations 5740 on a surface of object 3540₁-5740_nA fourth portion 5740 of the respective depth values; and indicating a set of locations 5750 on the surface of object 3550₁-5750_nA fifth portion 5750 of the corresponding depth value. The respective depth value may be relative to a camera (e.g., 3200/3200B) that generated the 3D image information, or may be relative to some other reference point. In some implementations, the 3D image information may include a point cloud including respective coordinates of various locations on the structure of the object in the camera field of view (e.g., 3210/3210B). In the example of fig. 5B, the point cloud may include a respective set of coordinates describing locations on respective surfaces of the object 3510 and 3550. The coordinates may be 3D coordinates, such as [ X Y Z ]]Coordinates, and may have values relative to the camera coordinate system or some other coordinate system. As an example, the camera coordinate system may be defined by the X, Y, and Z axes shown in fig. 3A, 3C, and 5B.

In an embodiment, step 4002 may involve receiving both 2D image information and 3D image information. In some cases, the computing system 1100 may use the 2D image information to compensate for limitations of the 3D image information, and vice versa. For example, when multiple objects in a camera field of view are placed close to each other and have substantially equal depths relative to a camera (e.g., 3200B), the 3D image information (e.g., 5700) may describe multiple locations having substantially equal depth values and may lack details for distinguishing a single object represented in the 3D image information, particularly if the spacing between the objects is too narrow for the resolution of the 3D image information. In some cases, the 3D image information may have erroneous or missing information due to noise or other error sources, which may further increase the difficulty of distinguishing individual objects. In this example, the 2D image information may compensate for this lack of detail by capturing or otherwise representing physical edges between individual objects. However, in some cases, as discussed below, the 2D image information may include false edges that may be candidate edges that do not correspond to any actual physical edges. In some implementations, the computing system 1100 may evaluate the likelihood that the candidate edge is a false edge by determining whether the 2D image information satisfies a defined darkness condition at the candidate edge, as discussed below with respect to step 4006. In some implementations, such as when the candidate edge corresponds to a physical location where the 3D image information describes a sharp change in depth, the computing system 1100 may determine whether the candidate edge corresponds to a physical edge in the 3D image information. In such a case, the 3D image information may be used to check whether the candidate edge is a false edge, and may supplement or replace the use of defined darkness conditions, thereby providing a more robust way of determining whether the candidate edge is a false edge or whether the candidate edge corresponds to an actual physical edge.

Returning to FIG. 4, in an embodiment, the method 4000 may include a step 4004 in which the computing system 1100 identifies a plurality of candidate edges associated with a set of objects (e.g., 3510 and 3550) from the image information of the step 4002. In embodiments, the candidate edge may be or may comprise an image location or set of physical locations that form candidates for representing a physical edge of an object or a group of objects. In one example, if the image information includes a 2D image representing one or more objects, the candidate edges may refer to a set of image locations, such as pixel locations (e.g., pixel location [ u ] s)₁ v₁]To [ u ]_kv_k]). The set of pixel locations may correspond to a group of pixels that collectively resemble a physical edge. For example, fig. 6A depicts a scenario in which the computing system 1100 has identified a candidate edge 5601 from a 2D image 5600₁、5601₂、5601₃、5601₄、5601₅、5601₆、...5601_nExamples of (2). Candidate edge 5601₁-5601_nMay be, may comprise or may be formed by a respective set of pixel locations, for example lines or line segments defining that the 2D image has a sharp change in image intensity. A sharp change in image intensity may occur, for example, between two image areas that are immediately adjacent to each other, where one image area is darker than the other. As discussed in more detail below, candidate edges may be formed based on a boundary between two image regions. In such an example, the boundary may be formed by a line or line segment as discussed above. Identification from 2D images (e.g., 5600)The candidate edges that are out may be referred to as 2D candidate edges or 2D edges.

In an embodiment, the image information may include some candidate edges corresponding to actual physical edges, and may include some candidate edges that are false edges. For example, candidate edge 5601 in fig. 6A₁、5601₂、5601₅、5601₆The actual physical edge of the set of objects 3510-₃And 5601₄Possibly a false edge. Candidate edge 5601₃And 5601₄Can represent, for example, visible lines or other visual indicia appearing on the surface of object 3510. These visible lines may be similar to physical edges, but do not correspond to any actual physical edges of the objects 3510-3550. Thus, as discussed below with respect to step 4008, in an embodiment method 4000 may involve determining whether a particular candidate edge should be used to represent at least one of the physical edges of a set of objects.

In one example, if the image information includes 3D information, the candidate edge may refer to a set of image locations or a set of physical locations. As an example, if the image locations are pixel locations, they may correspond to a set of pixels that appear as physical edges. In another example, if the 3D image information comprises a depth map, the candidate edges may comprise a set of pixel locations defining, for example, a line or line segment, which may form a boundary in the depth map where there is a sharp change in depth. If the 3D image information describes 3D coordinates of physical locations on the surface of the object (e.g., via a point cloud), the candidate edges may include a set of physical locations that also define, for example, imaginary lines or line segments, which may also form boundaries where there is a sharp change in depth in the point cloud or other 3D image information. For example, FIG. 6B depicts the computing system 1100 having identified a candidate edge 5701 from 3D image information 5700₁、5701₂、5701₃、5701_nExamples of (2). Candidate edge 5701₁-5701_nMay include, for example, a set of physical locations [ X ] that define a boundary at which a sharp change in depth occurs₁ Y₁ Z₁]To [ X ]_p Y_pZ₁]. The candidate edges identified from the 3D image information may be referred to as 3D candidate edges or 3D edges.

In an embodiment, if the computing system 1100 identifies both a 2D candidate edge and a 3D candidate edge, the computing system 1100 may be configured to determine any 2D candidate edge (e.g., 5601) of the 2D candidate edges₅) Whether to match one of the 3D candidate edges (e.g., 5701)₁) Representing a common physical edge and vice versa. In other words, the computing system 1100 may determine whether any of the 2D candidate edges map to one of the 3D candidate edges, and vice versa. The mapping may be based on, for example, converting coordinates of the 2D candidate edges from being expressed in a coordinate system of the 2D image information to being expressed in a coordinate system of the 3D image information, or converting coordinates of the 3D candidate edges from being expressed in a coordinate system of the 3D image information to being expressed in a coordinate system of the 2D image information. Mapping from 2D candidate edges to 3D candidate edges is discussed in more detail in U.S. patent application No. 16/791,024 (attorney docket MJ0049-US/0077-0009US1), entitled "METHOD AND COMPUTING System FOR PROCESSING candidate edges (METHOD AND COMPATING SYSTEM FOR PROCESSING CANDIDATE EDGES)", the entire contents of which are incorporated herein by reference.

As described above, the computing system 1100 may identify candidate edges from a 2D image or other 2D image information by identifying image locations (e.g., pixel locations) in the 2D image information where there is a sharp change in image intensity (e.g., pixel intensity). In some implementations, the abrupt change may occur at a boundary between two image areas, where one image area is darker than the other. For example, the two image regions may include a first image region and a second image region. The first image region may be a region of the 2D image that is darker than one or more immediately adjacent regions that may include the second image region. The darkness of an image region may indicate how much reflected light from the corresponding physical region was detected or otherwise sensed by a camera (e.g., 3200/3200a) that generated the image information (e.g., 5600). More particularly, darker image regions may indicate that the camera senses a relatively small amount of reflected light (or no reflected light) from the corresponding physical region. In some implementations, the darkness of an image region may indicate how close the image intensity in the image region is to the smallest possible image intensity value (e.g., zero). In these implementations, darker image regions may indicate that the image intensity value(s) of the image region are closer to zero, while less dark image regions may indicate that the image intensity value(s) of the image region are closer to the maximum possible image intensity value.

In an embodiment, the second image area may have an elongated shape, such as a rectangular strip or a line segment. As an example, fig. 7A shows image regions 5605 that are respectively more immediately adjacent₁、5605₂、5605₃、5605₄、5605₅、5605₆Darker example image region 5603₁、5603₂、5603₃、5603₄、5603₅、5603₆. The image region in fig. 7A may be a pixel region if the image 5600 includes pixels. Image region 5603₁-5603₄Each can be or form a band of pixels, such as a rectangular band, and can have a width of more than one pixel, while image region 5603₅And 5603₆Each may be or form a row of pixels having a width of one pixel. As described above, the candidate edge may be defined by a first image region (e.g., image region 5603)₁-5603₆One of) and a second image region (e.g., image region 5605)₁-5605₆One of) or based on the boundary, wherein the first image area may be immediately adjacent to the second image area and may be darker than the second image area, and wherein a sharp change in image intensity may occur at the boundary between the two image areas. For example, FIG. 7B shows a region composed of an image region 5603₁-5603₆And corresponding immediate image region 5605₁-5605₆ Candidate edges 5601 defined or formed by respective boundaries therebetween₁-5601₆. As an example, computing system 1100 may assign candidate edges 5601₁Is identified as defining oneImage region 5605₁And another darker image region 5603₁A set of pixel locations of the boundary therebetween. As an additional example, computing system 1100 may assign candidate edges 5601₅Identified as defining an image region 5605₅And a darker image region 5603₅A set of pixel locations of the boundary therebetween. In some cases, candidate edge 5601₅May be located in a darker image region 5603₅In (1). More particularly, candidate edge 5601₅May be the image region 5603₅Or may be associated with the image region 5603₅Overlap, the image region 5603₅May be, for example, a row of pixels having a single pixel width.

In embodiments, the computing system 1100 may detect or otherwise identify candidate edges, such as candidate edge 5601, based on image edge detection techniques that may detect, for example, sharp changes in image intensity₁-5601₆One of them. For example, the computing system 1100 may be configured to detect candidate edges in a 2D image or other image information by applying a Sobel operator, a Prewitt operator, or other techniques for determining intensity gradients in the 2D image, and/or by applying a Canny edge detector or other edge detection techniques.

In an embodiment, if the computing system 1100 identifies an image region in the 2D image as a darker band of pixels than one or more immediately adjacent image regions, the image region may be wide enough to form more candidate edges in some cases. For example, FIG. 7C illustrates that the computing system 1100 is based on an image region 5603₁-5603₄To identify additional candidate edges 5601₇-5601₁₀. More particularly, additional candidate edge 5601₇-5601₁₀Can be a defined image region 5603₁-5603₄And an immediate image region 5607₁-5607₄A respective set of pixel locations of a respective boundary therebetween. As a more specific example, the image region 5603 in this example₂Can be wide enough so that image edge detection techniques can identify the region 5603 imaged₂And an immediately adjacent region 5605₂ Candidate edge 5601 formed of boundary therebetween₂As shown in fig. 7B, and further identifies an image region 5603₂Opposite side (e.g., left side) and immediately adjacent region 5607₂Another candidate edge 5601 formed of a boundary therebetween₈As shown in fig. 7C. In embodiments, the image region may be very narrow, such that the image edge detection technique may identify only a single candidate edge from the image region. In such embodiments, the image area may have a width of a single pixel or a few pixels. For example, as described above, the image region 5603₅The pixel rows may be formed and may have a width of one pixel. In this example, the computing system 1100 can be based on the image region 5603₅Identifying only a single candidate edge 5601₅Where the edge candidate 5601₅Can be connected with the image region 5603, for example₅Overlap so that image candidate edge 5601₅May be to form an image region 5603₅Or may form an image area 5603₅Overlap.

Returning to fig. 4, in an embodiment, method 4000 may include step 4006, which step 4006 may be performed when the plurality of candidate edges in step 4004 includes a first candidate edge formed by a boundary between a first image area and a second image area, wherein an image intensity of the first image area may be darker than the second image area and may be immediately adjacent to the second image area. In this example, the first image region and the second image region may be regions described by a 2D image (e.g., 5600) or other image information. For example, fig. 7A-7C provide multiple candidate edges 5601₁-5601_nAnd the first candidate edge of step 4006 may be the plurality of candidate edges 5601₁-5601_nAny one of the above. As described above, the first candidate edge may be formed by or based on a boundary between the first image region and the brighter second image region. As an example, if the first candidate edge is candidate edge 5601₁Then the first image region can be image region 5603₁And the second image region may be 5605₁. As another example, ifThe first candidate edge is candidate edge 5601₂Then the first image region can be image region 5603₂And the second image region may be image region 5605₂。

In step 4006, the computing system 1100 can determine that image information (e.g., 2D image 5600) is at a first candidate edge (e.g., 5601)₂) Whether a defined darkness condition is satisfied. Such a determination may more specifically include, for example, determining a first image region (e.g., 5603)₂) Whether a defined darkness condition is met. In an embodiment, a defined darkness condition may be used to determine a first candidate edge (e.g., 5601)₂) Whether it is likely to correspond to an actual physical edge of an object (e.g., 3510) in the camera field of view (e.g., 3210/3210a), or whether the first candidate edge is likely to be a false edge.

In an embodiment, the defined darkness condition may be used to detect an image prior, or more specifically, a dark prior. An image prior may refer to an image feature that has some likelihood of appearing in an image in some cases, or an image feature in an image may be expected. More specifically, the image priors may correspond to expectations, or predictions of what image feature(s) will be present in an image generated under such circumstances (such as where the image is generated to represent a set of boxes or other objects placed adjacent to one another in the field of view of the camera). In some cases, a dark prior may refer to an image feature that has a high level of darkness and/or has a peaky-shaped image intensity distribution (e.g., a pixel intensity distribution). A peaky image intensity distribution may involve a peaked increase in darkness and/or a peaked decrease in image intensity. Dark priors may correspond to a situation where a set of boxes or other objects in the camera field of view are placed close enough to each other such that there is only a narrow physical gap between some or all of the objects. More particularly, the dark prior may correspond to an expectation, or prediction that when an image is generated to represent a set of objects in this case, the physical gap will appear very dark in the image. More specifically, the dark prior may correspond to an expectation or prediction that an image region representing a physical gap in the image will have a high level of darkness and/or may have a peaky image intensity distribution, as discussed in more detail below. In some implementations, a dark prior can be used to determine whether a candidate edge is likely to correspond to a physical edge by evaluating whether an image region associated with the candidate edge corresponds to a physical gap between two objects.

In embodiments, the darkness condition, which in some cases may be a definition of a condition for detecting dark priors, may be based on a model of how a physical gap between two objects (e.g., 3510 and 3520 of fig. 3A-3C) should appear in a 2D image or likely in a 2D image, especially if the physical gap is narrow (e.g., less than 5 millimeters or less than 10 millimeters). For example, the defined darkness conditions may be based on a lambertian model of diffuse reflection. Such a reflection model may estimate how light is reflected from one or more surfaces or areas, especially surfaces or areas that cause diffuse reflection of incident light. The model may thus estimate the intensity of reflected light from the surface or surface area, and this intensity may indicate how much lighter or darker the surface or surface area will appear in the image generated by the camera (e.g., 3200/3200a) sensing the reflected light.

As an example of how a lambertian model may be applied to a set of objects (e.g., 3510-. In some cases, the reflected light may be a reflection of light emitted from the light source 3600. More particularly, the light source 3600 may be directed toward the objects 3510, 3520 at least along a vector

Light is emitted. In this example, objects 3510 and 3520 may be placed adjacent to each other and may be separated by a narrow physical gap g, which may be the space between physical edge 3510B of object 3510 and physical edge 3520D of object 3520. In the case of FIG. 8, the image intensity of the physical gap g in the image (e.g., 5600) may depend on how much reflected light will come from the various surfaces forming the physical gapSuch as surface 3510E (e.g., a side surface) of object 3510. Under a Lambertian model of diffuse reflection, the intensity of reflected light from a particular surface may be proportional to or otherwise based on cos α, where α is the incident light vector

Normal vector to surface (e.g., 3510E)

The angle therebetween. As shown in fig. 8A, the vector

Normal vector to surface 3510E

The angle a therebetween can be substantially 90 degrees so that the intensity of reflected light from surface 3510E can be very low, or in some cases zero. Thus, an image area representing a physical gap between two objects may have an extremely high level of darkness. Thus, in some implementations, the defined darkness condition of step 4006 may include a defined darkness threshold criterion that may be used to evaluate whether an image area is dark enough to indicate that it represents a physical gap between two objects.

In some cases, the physical gap may appear darker at its middle than at its periphery. That is, if any reflected light leaves the physical gap, more reflected light may be from the periphery of the physical gap than from the middle of the physical gap. The periphery can refer to, for example, a location in the physical gap that is closer to physical edge 3520D or physical edge 3510B. In some cases, the darkness of the peak level may occur in the middle of the physical gap. Thus, an image region representing a physical gap may have a peaky-shaped image intensity distribution (e.g., a pixel intensity distribution) with a peaked increase in darkness or a peaked decrease in image intensity within the image region. Thus, in some cases the defined darkness conditions in step 4006 may include defined spike intensity distribution criteria to evaluate whether an image region has an image intensity distribution such as a spike (rather than an image intensity distribution such as a staircase).

In an embodiment, the defined darkness conditions may be defined by one or more rules, criteria, or other information stored, for example, in the non-transitory computer-readable medium 1120 or elsewhere. For example, the information may define whether the darkness condition is satisfied only by satisfying the darkness threshold criteria, only by satisfying the spike intensity distribution criteria, only by satisfying both criteria, or by satisfying either of the darkness threshold criteria or the spike intensity distribution criteria. In some cases, the information may be predefined manually or otherwise and stored in the non-transitory computer-readable medium 1120, such that the defined darkness condition may be a predefined darkness condition. In some cases, the information of the darkness conditions may be dynamically defined.

In an embodiment, the defined darkness threshold criteria and/or the defined spike intensity distribution criteria may be defined by information stored on the non-transitory computer-readable medium 1120 or elsewhere. This information may be predefined such that the defined darkness threshold criteria and/or the defined spike intensity distribution criteria may be a predefined criterion or criteria. In embodiments, various defined thresholds or other defined values in the present disclosure may be defined as stored values on the non-transitory computer readable medium 1120 or elsewhere. For example, the defined darkness threshold or the defined depth difference threshold discussed below may be values stored on the computer-readable medium 1120. They may be predefined values or may be dynamically defined.

FIGS. 9A-9C illustrate evaluation of image information (e.g., 5600) at candidate edge 5601₂Whether or not a defined darkness condition is satisfied, or more specifically, the image region 5603₂Examples of whether the defined darkness conditions are met. Image region 5603₂A physical gap between a first object (such as object 3510 of fig. 8) and a second object (such as object 3520) may be represented. In an embodiment, candidate edge 5601₂Can represent a pairLike the physical edge 5610B of 3510, and may be defined by the image region 5603₂And an immediately adjacent image region 5605₂Or formed based on the boundary. In this example, image region 5603₂Can be the first image region, and the image region 5605₂May be the second image area. More specifically, image region 5603₂Can be the first pixel region forming a pixel band, and image region 5605₂May be the second pixel region, such that the candidate edge 5601₂May comprise or may be formed by, for example, a set of pixel locations defining a boundary between the first pixel region and the second pixel region. As described above, the computing system 1100 may detect the image region 5603 by, for example₂、5605₂Sharp change in image intensity (e.g., pixel intensity) therebetween to identify candidate edge 5601₂。

In an embodiment, if the candidate edge is formed based on a boundary between a first image region and a second image region, wherein the first image region is darker than the second image region, the computing system 1100 may determine that the defined darkness condition is satisfied if the first image region satisfies the defined spike intensity distribution criteria. More particularly, if the first image region has a particular shape for its image intensity distribution (e.g., pixel intensity distribution), such as a shape in which the darkness of the image intensity within the first image region increases toward a peak level of the darkness at a location within the first image region, then the darkness decreases, the computing system may determine the first image region (e.g., 5603)₂) Meeting the defined peak intensity distribution criteria. Such a criterion may be consistent with a peaky intensity distribution, where the image intensity distribution has a peaky increase in darkness or a peaky decrease in intensity within the image area. Such a criterion may be associated with detecting a dark prior, where any physical gap that is expected to appear in the image appears darker in the middle of the gap relative to the periphery of the gap.

FIG. 9B illustrates an image intensity distribution 9001, or more specifically, a pixel, that can satisfy a defined spike intensity distribution criterionThe intensity distribution. More particularly, the image intensity distribution may include information describing the image intensity, or more particularly how the pixel intensity changes as a function of image location, such as pixel location. In some implementations, the image intensity distribution may be represented by a curve or graph that describes the value of the image intensity as a function of position in the image. For example, FIG. 9B depicts a curve or graph of image intensity distribution 9001 that describes values of image intensity, or more specifically pixel intensity values, as a function of pixel position in a particular direction along axis 5609. Axis 5609 may be across image region 5603₂And along an axis extending along the width dimension, while a direction along axis 5609 may be a particular direction along axis 5609. In the example of fig. 9B, the width dimension can be aligned with, for example, coordinate axis u of image 5600 of fig. 5A, and the direction along axis 5609 can be a positive direction with pixel coordinates [ u, v ] along that direction]With an increasing u-value.

In an embodiment, the computing system 1100 may determine the image region 5603 by₂Whether or not there is a pixel intensity distribution (e.g., 9001) to determine image region 5603₂Whether a defined spike intensity distribution criterion is met, the pixel intensity distribution comprising: (i) a first distribution portion (e.g., 9011) in which an image intensity (e.g., pixel intensity) increases in darkness in the first image region as a function of position along a first direction (e.g., a positive direction along axis 5609) to reach position u in the first image region₁A peak level of darkness (e.g., 9002) followed by (ii) a second distribution portion (e.g., 9012) where the image intensity decreases in darkness within the first image region, away from the peak level of darkness, as a function of position along the same direction (e.g., the positive direction). Image intensity distribution 9001 in fig. 9B may more specifically be a peaked intensity distribution, which is in image region 9503₂With a spike reduction in image intensity.

In some implementations, the image intensity distribution with increased darkness may correspond to an image intensity distribution with a value of image intensity reduction. For example, an image (e.g., 5600) may have pixel intensity values ranging from a minimum possible pixel intensity value (e.g., zero) to a maximum possible pixel intensity value (e.g., 255 for pixel intensity values encoded with 8 bits). In this example, lower pixel intensity values may represent a lower level of luminance, and therefore a higher level of darkness, while higher pixel intensity values may represent a higher level of luminance, and therefore a lower level of darkness. Also in this example, the peak level of darkness of the image intensity distribution (e.g., 9002) may correspond to the minimum image intensity value of the image intensity distribution (e.g., 9001).

In the above example, the computing system 1100 may determine whether the image region satisfies the defined spike intensity distribution criteria by determining whether the image intensity distribution has a shape that: where the image intensity values (e.g., pixel intensity values) begin by decreasing the image intensity toward the minimum image intensity value, and then switching to increasing the image intensity away from the minimum image intensity value. For example, the image intensity distribution 9001 in FIG. 9B may depict across image region 5603₂Of the series of pixels extending in the width dimension. The computing system 1100 can determine the image region 5603 by determining whether the image intensity distribution has a shape such as₂Whether the peak intensity distribution criterion is met: the corresponding pixel intensity value in the shape is toward the image region 5603₂Then switches to the image region 5603₂Away from the minimum pixel intensity value. In this example, the minimum pixel intensity value may correspond to the peak level 9002 of darkness in the image intensity distribution 9001.

In an embodiment, if the candidate edge is formed based on a boundary between a first image region and a second image region, wherein the first image region is darker than the second image region, satisfying the defined darkness threshold criterion may involve a comparison with a defined darkness threshold. Such a criterion may correspond to detecting a dark prior in which any physical gap appearing in the image is expected to be extremely dark in appearance. Fig. 9C shows an image region 5603₂And another image intensity distribution 9003 of an immediately adjacent image region. In this exampleThe first image region may be an immediately brighter second image region (e.g., 5605)₂) Image region 5603 of₂And darker image region 5603₂May be the first image area. As described above, the image region 5603₂Pixel strips may be formed. In FIG. 9C, the computing system 1100 can determine an image region 5603₂Whether or not there is a darkness threshold τ defined by the image intensity ratio_{dark_prior}Determining an image region 5603 using at least one darker portion₂Whether a defined darkness threshold criterion is met. As described above, in some cases, a higher level of darkness may correspond to a lower image intensity value. In such a case, the computing system 1100 may determine the image region 5603₂Whether or not to have a darkness threshold τ less than a defined value_{dark_prior}An image intensity distribution of image intensity values (e.g., pixel intensity values). In some cases, computing system 1100 may more specifically determine whether the minimum intensity value of image intensity distribution 9003 is less than or equal to a defined darkness threshold τ_{dark_prior}Where the minimum intensity value may correspond to the peak level 9004 of the darkness of intensity distribution 9003. In an embodiment, if image region 5603₂With image intensity distribution 9003, it can satisfy both the defined darkness threshold criteria and the defined spike intensity distribution criteria.

In an embodiment, if at least one of the defined darkness threshold criteria or the defined spike intensity distribution criteria is satisfied, the computing system 1100 may determine that the defined darkness condition is satisfied for the candidate edge and/or image region, such that any of the above criteria may be used to satisfy the defined darkness condition. In embodiments, computing system 1100 may determine that the defined darkness condition is satisfied only in response to determining that the defined spike intensity distribution criteria are satisfied (whether or not the defined darkness threshold criteria are satisfied), only in response to determining that the defined darkness threshold criteria are satisfied (whether or not the defined spike intensity distribution criteria are satisfied), or only in response to determining that both the defined darkness threshold criteria and the defined spike intensity distribution criteria are satisfied.

In the examplesIn some embodiments, the computing system 1100 may identify candidate edges (e.g., 5601) based on 2D image information (such as the 2D image 5600)₂) And determining whether the candidate edge satisfies a defined darkness condition. As described above, if the computing system 1100 receives both 2D image information and 3D image information, the computing system 1100 may use the 2D image information to compensate for limited or missing 3D image information in the 3D image information, and vice versa. For example, when a camera (e.g., 3200B) generates 3D image information to represent a group of objects, the 3D image information may lack information to distinguish individual objects within the group, particularly if the group of objects have the same depth values relative to the camera. More particularly, the 3D image information may lack information for detecting narrow physical gaps between objects and thus may have limited usefulness in identifying physical edges associated with the physical gaps.

As an example, fig. 9D shows depth values associated with a portion 5715 of the 3D image information 5700 of fig. 5B. More specifically, the portion 5715 may describe a physical location 5720 on the top surface of the object 3520_aTo 5720_a+5And a position 5710 on the top surface of the object 3510_bTo 5710_b+4To the corresponding depth value. These physical locations may map or otherwise correspond to the image region 5603₂Middle or surrounding image position, or candidate edge 5601₂The surrounding image locations. As shown in fig. 8, an image region 5603₂A physical gap g between the objects 3510, 3520 may be represented. As described above, the computing system 1100 may use the 3D image information to attempt to detect one or more locations where there is a sharp change in depth. However, a physical gap (such as the physical gap of fig. 8) may be too narrow or otherwise too small relative to the resolution of the 3D image information to be captured by the 3D image information 5700. Thus, in the example of fig. 9D, computing system 1100 can determine location 5720_aTo 5720_a+5And 5710_bTo 5710_b+4There is no sharp change in depth and it is therefore determined that the 3D image information does not indicate a candidate edge at any of those locations. Furthermore, in some cases, the 3D image information may lack the location 5720_aTo 5720_a+5And 5710_bTo 5710_b+4(or more specifically, corresponding to candidate edge 5601)₁One or more locations of) the depth information. In some cases, mapped or otherwise corresponding to candidate edge 5601₁May be affected by an imaging noise level that is greater than a defined noise margin threshold, which may be a value defined in the non-transitory computer readable medium 1120. In the above example, the 2D image information may compensate for these limitations of the 3D image information, as the 2D image information may include the candidate edge 5601 representing the physical edge 3510B of the object 3510₁And includes an image region 5603 that does represent a physical gap between the object 3510 and the object 3520₂. In the above example involving the limitation of 3D image information, the computing system 1100 may also determine the candidate edge 5601 using the defined darkness condition₂Whether it should be used to represent one of the physical edges of a set of objects (e.g., 3510B).

10A-10C depict methods for determining candidate edges 5601₄And/or image region 5603₄Examples of whether a defined darkness condition is met. Image region 5603₄A portion of the visual marker 3512 of fig. 3B may be represented, such as a visible line printed on the top surface of the object 3510. In this example, image region 5603₄Can be compared with the immediate image region (such as image region 5605)₄And 5607₄) And is darker. Candidate edge 5601₄Can be based on a darker image region 5603₄And immediately adjacent region 5605₄The boundary therebetween is formed.

Fig. 10B depicts an image region 5603 having an image intensity distribution 10001 that does not meet the defined spike intensity distribution criteria₄Because the image intensity distribution 10001 is in the image region 5603₄It was not changed in the following manner: in this manner the image intensity distribution 10001 increases as a function of position toward the peak level darkness of darkness and then decreases away from the peak level darkness of darkness. Image intensity distribution 10001 in this example can describe pixel intensity values as pixel locations along axis 5608As a function, axis 5608 can be associated with image region 5603₄Are aligned in the width dimension. As described above, the spike intensity distribution criteria may correspond to a desire that a physical gap between physical edges of the object may appear darker in the middle of the physical gap relative to the periphery of the physical gap. Thus, the image area representing the physical gap may have an image intensity distribution that varies as a function of image position within the image area, and more specifically, increases in darkness as a function of image position along a particular direction from image positions corresponding to the periphery of the physical gap to image positions corresponding to the middle of the physical gap, and then decreases in darkness as a function of position along the same direction. More particularly, in an image region representing a physical gap, the image intensity distribution may have a spike increase in darkness or a spike decrease in image intensity. In embodiments, the image area that instead represents a visible line or other visual indicia may lack such an image intensity distribution and may instead have a more uniform darkness level within the image area. Thus, as shown in fig. 10B, an image region 5603 representing a part of the visual marker 3512₄In image region 5603₄May have a more uniform image intensity distribution 10001 such that distribution 10001 is in an image region 5603₄The inner portions are not substantially changed. Further, the image intensity distribution 10001 can be in the image region 5603₄Has a sub-image area of image intensity at the boundary with a brighter neighboring image area (e.g., 5605)₄) Associated image intensity to image region 5603₄A step-like change in uniform image intensity. Thus, the image intensity distribution 10001 does not pass through the image region 5603₄Starts towards a peak horizontal darkness increase of darkness and then switches to a shape in which darkness decreases along that direction. More particularly, the image intensity distribution 10001 exhibited no spike reduction in image intensity. Thus, in this example, the computing system 1100 may determine the image region 5603₄Not satisfying the defined spike intensity distribution criteria, which may result in a determination that the 2D image 5600 is at the candidate edge 5601₄And/or in image region 5603₄Does not satisfy the defined darkness conditions.

FIG. 10C depicts an image region 5603 having an image intensity distribution 10003 that may not meet a defined darkness threshold criterion₄Since the image intensity distribution 10003 may indicate the image region 5603₄Not dark enough. More specifically, the computing system 1100 can determine an image region 5603₄Of image intensity distribution 10003 is at a defined darkness threshold τ_{dark_prior}Above. Thus, the computing system 1100 in FIG. 10C can determine the image region 5603₄Not satisfying the defined darkness threshold criteria, which may result in a determination that the image 5600 is at the candidate edge 5601₄And/or image region 5603₄Does not satisfy the defined darkness conditions. The image intensity distribution 10003 may also fail to meet the defined peak intensity distribution criteria, as discussed above with respect to fig. 10B.

In an embodiment, the image region may have a width that is too small for a reliable evaluation of whether the image region meets the defined criterion of a spike intensity distribution. For example, the image area may have a width of only a single pixel or only a few pixels. In some cases, the computing system 1100 may determine that such image regions do not satisfy a defined darkness condition. In other cases, the computing system 1100 may determine whether the image region satisfies the defined darkness condition based on whether the image region satisfies the defined darkness threshold criteria. In some cases, the computing system 1100 may decide not to evaluate such image regions or associated candidate edges for defined darkness conditions.

As described above, an aspect of the disclosure relates to a case where the computing system 1100 identifies a plurality of candidate edges based at least on 2D image information (such as the 2D image 5600). In such embodiments, the plurality of candidate edges may include at least a first candidate edge based on 2D image recognition (e.g., 5601)₁/5601₂/5601₃/5601₄). For example, the first candidate edge may be formed based on a boundary between two image regions of the 2D image. In some cases, the computing system 1100 may identify a plurality of candidate edges based on the 2D image information and the 3D image information. In such a case, multiple candidate edges may beIncluding a first candidate edge from the 2D image information, as discussed above, and further including a second candidate edge identified based on the 3D image information (e.g., 5701 of fig. 6B)₁)。

11A-11B illustrate the computing system 1100 to identify candidate edges 5701 based on 3D image information 5700₁Is identified as a second candidate edge of the plurality of candidate edges. In this example, the computing system 1100 can be based on a candidate edge 5701 between a first portion 5710A and a second portion 5750A of 3D image information 5700₁To identify candidate edges 5701 where a sharp change in depth is detected₁. The first portion 5710A may represent a location area on the top surface of the object 3510, e.g., of fig. 3A-3C, and the second portion 5750A may represent a location area on the top surface of the object 3550, e.g., of fig. 3A-3C. More particularly, as depicted in fig. 11B, the first portion 5710A of 3D image information may include a location 5710 on the top surface of the object 3510_cTo 5710_c+5And the second portion 5750A may comprise a location 5750 on the top surface of the object 3550_dTo 5750_d+4To the corresponding depth value. In this example, position 5710_cTo 5710_c+5And 5750_dTo 5750_d+4There may be a series of positions aligned along the Y-axis of fig. 11A.

In embodiments, the computing system 1100 can identify the candidate edge 5701 of fig. 11B based on detecting a sharp change in depth between two consecutive or otherwise adjacent locations of a series of locations described by the 3D image information 5700₁. Such a sharp change may be referred to as a depth discontinuity condition. For example, a sharp change may be detected when the difference between the respective depth values of the two locations exceeds a defined depth difference threshold. For example, the computing system 1100 may determine the location 5710_c+5Depth value and location 5750_dThe difference between the depth values of (a) exceeds a defined depth difference threshold. Thus, the computing system 1100 can be based on these two locations 5710_c+5、5750_dTo identify candidate edges 5701₁. For example, candidate edge 5701₁Can be identified as including a position 5710 on the Y-axis_c+5、5750_dIn the middle position.

In an embodiment, the computing system 1100 may identify candidate edges by identifying, based on the 3D image information, two surfaces having a depth difference that exceeds a defined depth difference threshold. For example, as shown in fig. 11C, the computing system 1100 can identify a first surface of a set of objects 3510 and 3550 based on a first set of locations described by the 3D image information 5700, wherein the first set of locations have respective depth values that deviate from each other by no more than a defined measurement variance threshold. Similarly, the computing system 1100 can identify a second surface of the set of objects 3510 and 3550 based on a second set of locations described by the 3D image information 5700, wherein the second set of locations have respective depth values that deviate from each other by no more than a defined measurement variance threshold. In the example of FIG. 11C, the first set of locations can include location 5710_cTo 5710_c+5Which may represent the top surface of object 5710, and the second set of locations may include location 5750_dTo 5750_d+4Which may represent the top surface of object 5750.

In this embodiment, the defined measurement variance threshold may describe the effects of imaging noise, manufacturing tolerances, or other factors that may introduce random fluctuations in the depth measurements made by the camera (e.g., 3200B). Such a source of random fluctuations causes the depth values of the various locations to have some natural variance (natural variance), even though the various locations are part of a common surface and actually have the same depth relative to the camera. In some cases, the defined measurement variance threshold may be equal to or based on a nominal standard deviation describing expected random fluctuations in the depth measurements, or more generally, how sensitive the camera is to noise or other sources of error. The nominal standard deviation may describe a baseline standard deviation or other form of variance desired for the camera-generated depth values or other depth information. The nominal standard deviation, or more generally the defined measurement variance threshold, may be a value stored in, for example, the non-transitory computer readable medium 1120, and may be a predefined value or a dynamically defined value. In an embodiment, the computing system is configured to determine a measure variance threshold for a set of locations based on the depth values of the set of locationsSystem 1100 can determine that the set of locations is part of a common surface. In a more specific embodiment, if the standard deviation of the respective depth values of the location set (e.g., Std)₅₇₁₀Or Std₅₇₅₀) Less than the defined measurement variance threshold, the computing system 1100 may determine that the set of locations is part of a common surface.

In the above embodiment, the computing system 1100 may identify candidate edges from the 3D image information based on two surfaces having sufficient depth differences. For example, the first set of locations in FIG. 11C that describe the top surface of the object 5710 may have an average depth value of Avg₅₇₁₀Or otherwise associated with the average depth value Avg₅₇₁₀And (4) associating. Similarly, the second set of locations describing the top surface of the object 5750 may have an average depth value Avg₅₇₅₀Or otherwise associated with the average depth value Avg₅₇₅₀And (4) associating. Computing system 1100 can determine Avg₅₇₁₀And Avg₅₇₅₀Whether the difference between is greater than or equal to a defined depth difference threshold. In some cases, the defined depth difference threshold may be determined as a multiple of the defined measurement variance threshold (e.g., two times the defined measurement variance threshold, or five times the defined measurement variance threshold). If Avg associated with two surfaces₅₇₁₀And Avg₅₇₅₀The difference between is greater than or equal to the defined depth difference threshold, then the computing system 1100 may determine that the depth discontinuity condition is satisfied. More particularly, the computing system 1100 may determine candidate edges (e.g., 5701)₁) A candidate edge is identified that exists at a location between the two surfaces, and more particularly, based on the location where a transition exists between the two surfaces.

As described above, an aspect of the present disclosure relates to using 2D image information and 3D image information to compensate for each other so that the 3D image information can compensate for a restriction in the 2D image information (and vice versa). In some cases, physical edges detected from 3D image information may be associated with a higher confidence level than physical edges detected from 2D image information only. In some cases, the computing system 1100 may be able to determine whether a physical edge (e.g., 3510A of fig. 3B) is represented in both the 2D image information and the 3D image informationTo identify candidate edges (e.g., 5601) representing physical edges in 2D image information₅) And identifying corresponding candidate edges (e.g., 5701) representing physical edges in the 3D image information₁). As described above, the corresponding candidate edges may be mapped to each other. For example, candidate edges in 3D image information (e.g., 5701)₁) Can map to a candidate edge in 2D image information (e.g., 5601)₅). Candidate edges (e.g., 5601)₅) May be based on two image regions (such as 5605)₅And 5650) is formed. However, in some cases, the computing system 1100 may not be able to determine candidate edges from 2D image information at a high confidence level (e.g., 5601)₅) Corresponding to the actual physical edge. For example, as shown in fig. 11D, the 2D image 5600 has a candidate edge 5601₅May have a step-like variation in image intensity. In this example, the computing system 1100 may determine that the 2D image 5600 is at the candidate edge 5601₅To (or more specifically to) two image regions 5605₅And 5650) do not meet the defined darkness condition. Thus, the computing system 1100 can determine that there are no candidate edges 5601 that represent physical edges₅A sufficiently high confidence level of the association.

In such a case, the computing system 1100 may use the 3D image information to provide additional input. More particularly, the computing system 1100 may identify candidate edges 5701 based on 3D image information₁11A-11C, as discussed above with respect to FIGS. 11A-11C, where the candidate edge 5601 in the 2D image information₅May map or otherwise correspond to candidate edges 5701 in 3D image information₁. In some cases, because of candidate edge 5701₁Is identified based on depth information, the computing system 1100 may determine that there is a candidate edge 5701₁Representing a sufficiently high probability of a physical edge (i.e., physical edge 3510A of fig. 3B). Thus, the 2D image information may not result in detection of the physical edge 3510A, or may result in detection of the physical edge 3510A with a low confidence level, while the 3D image information may be used by the computing system 1100 to detect the physical edge 3510A with a higher confidence level.

Returning to FIG. 4, in an embodiment, method 4000 may include step 4008, where computing system 1100 may select a subset of the plurality of candidate edges (e.g., plurality of candidate edges 5601)₁To 5601_nTo form a selected candidate edge subset for representing a physical edge of a set of objects (e.g., 3510-3540). In an embodiment, this step may involve excluding from the subset one or more candidate edges that are each likely to be false edges. One or more candidate edges that are likely to be false edges may be filtered out of the subset of candidate edges or, more generally, ignored from further consideration of the physical edges used to represent a group of objects (e.g., 3510-3540). In one example, the computing system 1100 may select a subset of the plurality of candidate edges by determining which candidate edge(s) to filter from the plurality of candidate edges, wherein the plurality of candidate edges form the resulting subset after being filtered. In one example, if multiple candidate edges are represented or described by the edge detection information 1126 of FIG. 2C, filtering a candidate edge may involve deleting information about the candidate edge from the edge detection information 1126.

As described above, a plurality of candidate edges (e.g., 5601)₁To 5601_nOr 5601₁To 5601_nAnd 5701₁To 5701_n) At least a first candidate edge (e.g., 5601) formed based on a boundary between a first image region and a second image region darker than the first image region may be included₁Or 5601₄). Furthermore, a first candidate edge may be identified from the 2D image information. In an embodiment, step 4008 may involve determining whether to include the first candidate edge in a subset (also referred to as a subset of candidate edges). The first candidate edge (e.g., 5601)₁) Inclusion in the subset may allow the first candidate edge to be used to represent at least one physical edge (e.g., 3510B) of a set of objects in the camera field of view. More specifically, if the first candidate edge (e.g., 5601)₁) Is included in the subset, such inclusion may indicate a first candidate edge (e.g., 5601)₁) At least one of the physical edges still being considered for representing a set of objects remainsA candidate for an edge. In other words, the computing system 1100 may determine whether to retain the first candidate edge as a candidate for representing at least one physical edge. If the computing system 1100 determines to retain the first candidate edge as such a candidate, it may include the first candidate edge in a subset (which may also be referred to as a selected subset of candidate edges). This determination may be part of the step of selecting a subset of the plurality of candidate edges, and may be performed based on whether the image satisfies a defined darkness condition at the first candidate edge. In some cases, including the first candidate edge in the subset may be an indication that the first candidate edge has a sufficiently low likelihood of being a false edge. In some cases, the inclusion of the first candidate edge in the subset may be an indication as follows: i.e., the first candidate edge (e.g., 5601)₁) With a sufficiently high confidence level corresponding to the actual physical edges of the set of objects, such that the computing system 1100 will use the first candidate edge to represent at least one physical edge of the set of objects, or will at least continue to consider the first candidate edge for representing at least one physical edge of the set of objects. If the computing system 1100 determines not to remove the first candidate edge (e.g., 5601)₄) Included in the subset such that a first candidate edge (e.g., 5601)₄) Filtered or otherwise excluded from the subset, such exclusion may be a first candidate edge (e.g., 5601)₄) Is no longer an indication of a candidate for representing at least one of the physical edges of the set of objects. In some cases, excluding the first candidate edge from the subset may indicate the first candidate edge (e.g., 5601)₄) Possibly a false edge.

In an embodiment, determining whether to include the first candidate edge in the selected subset of candidate edges may be based on whether the image information (e.g., 5600) satisfies a defined darkness condition at the first candidate edge, as described above. In some implementations, if the image information satisfies a defined darkness condition at the first candidate edge, such a result may indicate that the first candidate edge has a sufficiently low likelihood of being a false edge, because the first candidate edge in such a caseLikely associated with an image region representing a physical gap between two objects. Thus, the first candidate edge is likely to represent the physical edge of the side forming the physical gap. In such a case, the computing system 1100 may determine to include the first candidate edge in the selected subset. In some cases, if the image information does not satisfy the defined darkness condition at the first candidate edge, the computing system 1100 may determine not to include the first candidate edge in the selected subset. In some cases, if the computing system determines that the 2D image information does not satisfy the defined darkness condition at the first candidate edge, the computing system 1100 may further evaluate the first candidate edge using the 3D image information. For example, if the computing system 1100 determines that the 2D image 5600 is at the candidate edge 5601₅Where the defined darkness condition is not satisfied, the computing system 1100 may determine the candidate edge 5601₅Whether to map to a candidate edge 5701 described by 3D image information₁And candidate edges 5701 in 3D image information₁Whether a change in depth greater than a defined depth difference threshold is indicated, as discussed above with respect to fig. 11A-11D.

In an embodiment, method 4000 may perform steps 4006 and/or 4008 multiple times (e.g., via multiple iterations) to determine whether the image information satisfies a defined darkness condition at multiple candidate edges, and select the subset discussed above based on these determinations. As an example, if the plurality of candidate edges includes at least candidate edge 5601₁To 5601_nThe computing system 1100 can perform step 4006 multiple times to determine that the 2D image 5600 is at, for example, the candidate edge 5601₁To 5601_nWhether a defined darkness condition is satisfied. The computing system 1100 may also perform step 4008 multiple times to determine which of these candidate edges are to be included in the subset and remain candidates for representing a physical edge, and which of these candidate edges are to be excluded from the subset and are therefore no longer candidates for representing a physical edge. For example, the computing system 1100 may determine that the subset is to include the candidate edge 5601₁And 5601₂Because the 2D image 5600 satisfies the defined darkness condition at those candidate edgesAnd the subset will not include candidate edge 5601₃And 5601₄Because the 2D image does not satisfy the defined darkness condition at those candidate edges. In some cases, computing system 1100 may determine not to discard candidate edge 5601₅Included in the subset because the 2D image 5600 does not satisfy the defined darkness condition at the candidate edge. In some cases, if candidate edge 5601₅Mapping to candidate edges 5701 in 3D image information indicating a depth change that exceeds a depth difference threshold₁Then the computing system 1100 can determine that the candidate edge 5601 is still to be₅Included in the subset.

In an embodiment, the method 4000 may include a step in which the computing system 1100 outputs a robot interaction movement command. The robot interaction movement command may be for robot interaction between a robot (e.g., 3300) and at least one object of a set of objects (e.g., 3510-3550). The robotic interaction may involve, for example, a robot (e.g., 3300) performing an unstacking operation or other operation in which the robot picks up an object (e.g., a box) from a pallet and moves the object to a destination location.

In an embodiment, the robot interaction movement command may be generated based on the selected subset of candidate edges of step 4008. For example, the computing system 1100 may use the selected subset of candidate edges to distinguish a single object from a set of objects described by the image information. In some cases, the computing system 1100 may perform segmentation of the image information using the selected subset. For example, if the image information includes a point cloud, the computing system may perform point cloud segmentation using the selected subset of candidate edges, which may involve identifying a portion of the point cloud that corresponds to a single object in a set of objects. Point cloud segmentation is discussed in more detail in U.S. patent application No. 16/791,024 (attorney docket MJ0049-US/0077-0009US1), which is incorporated herein by reference in its entirety. In one example, if the image information includes 2D image information, the computing system 1100 may use the selected subset of candidate edges to isolate a portion of the 2D image information that corresponds to a single object in the set of objects. The isolated portion may, for example, be used as a target image or target image portion for performing an object recognition operation or an object registration operation (e.g., by module 1128). Object registration and object identification are discussed in more detail in U.S. patent application No. 16/991,466 (attorney docket No. mj0054-US/0077-0012US1) and 17/193,253 (attorney docket No. MJ0060-US/0077-0017US1), which are incorporated herein by reference in their entirety. In such an example, the robot interaction movement command may be generated based on a result of an object recognition operation or an object registration operation. For example, the object recognition operation may produce a detection hypothesis, which may be an estimate of what object or object type the image information or a portion thereof represents. In some cases, the detection hypothesis may be associated with an object recognition template, which may, for example, include information describing the physical structure of one of the objects 3510-3540. This information may be used by computing system 1100 to plan the movements of a robot (e.g., 3300) for picking and moving objects (e.g., via module 1129).

While the above steps of method 4000 are described with respect to objects 3510 and 3550 of FIGS. 3A-3C, FIG. 12A illustrates the above steps with respect to object 12510, and FIG. 13A illustrates the above steps with respect to object 13510 and 13520. In an embodiment, object 12510 of fig. 12A may be a box having a top surface with a first physical area 12512 that is darker than a second, immediately adjacent physical area 12514. For example, the first physical area 12512 may have more ink printed thereon relative to the second physical area 12514. The object 12510 may be disposed on the object 12520, and the object 12520 may be a tray on which a box is disposed. Fig. 12B illustrates a 2D image 12600 that may be generated to represent the object 12510. More particularly, the 2D image 12600 may include a first image region 12603 representing the first physical region 12512 and include a second image region 12605 representing the second physical region 12514. The computing system 1100 in this embodiment can identify the first candidate edge 12601 based on the boundary between the first image region 12603 and the second image region 12605₁。

In an embodiment, the computing system 1100 may determine the 2D image 12600 at a first candidate edge 12601₁Does not satisfy the defined darkness conditions. For example, the computing system 1100 may determine that the 2D image 12600 has an image intensity distribution 12001, the image intensity distribution 12001 at the first candidate edge 12601₁With a step-like variation in image intensity. The image intensity distribution may be measured along an axis 12609 extending along the u-axis of the image. In some implementations, the computing system 1100 may determine the image intensity distribution 12001 or, more specifically, the image regions 12603 and 12605 that do not meet the spike intensity distribution criteria. Computing system 1100 can also determine that there is a first candidate edge 12601₁Does not satisfy the defined darkness conditions. Thus, the computing system 1100 may filter out the first candidate edge 12601 from the edge detection information 1126₁。

In the embodiment of fig. 13A, objects 13510 and 13520 may each be a box and may be disposed on object 13530, and object 13530 may be a pallet or other platform. In this embodiment, object 13510 may be darker (e.g., due to being made of darker cardboard or other material) than object 13520. Furthermore, the two objects may be separated by a narrow physical gap g. Fig. 13B shows a 2D image 13600 including a first image region 13603 representing a first object 13510 and a second image region 13605 representing a second object 13530. The computing system 1100 in this embodiment may identify a candidate edge 13601 based on a boundary between two image regions 13605, 13605₁。

As shown in fig. 12B and 13B, images 12600 and 13600 may have similar appearances. However, as shown in fig. 13C, the image 13600 may have an image intensity distribution that includes a spike reduction in image intensity. More particularly, the image region 13603 of the image 13600 may more particularly include the image region 13603 to represent the physical gap g between the objects 13510, 13520₁And includes an image region 13603 for representing the object 13520₂. In this embodiment, image region 13603₁A spike reduction in image intensity may be included and may have a minimum pixel intensity value less than a defined darkness threshold. Thus, computing system 1100 may determine image region 13603₁Meet a defined peakIntensity distribution criteria and/or defined darkness threshold criteria. Thus, the computing system 1100 may determine that the image 13600 is at the first candidate edge 13603₁Meets the defined darkness conditions. The computing system 1100 may therefore determine to use the first candidate edge 13603₁To represent one of the physical edges of the objects 13510, 13520.

Additional discussion of various embodiments:

embodiment 1 includes a computing system or a method performed by the computing system. The computing system may include a communication interface and at least one processing circuit. The communication interface may be configured to communicate with the robot and with a camera having a camera field of view. The at least one processing circuit may be configured to, when a group of objects is in the camera field of view, perform the following: receiving image information representing a set of objects, wherein the image information is generated by a camera; identifying, from the image information, a plurality of candidate edges associated with the set of objects, wherein the plurality of candidate edges are or include respective image locations or sets of physical locations that form respective candidates for representing physical edges of the set of objects; when the plurality of candidate edges includes a first candidate edge formed based on a boundary between a first image area and a second image area, wherein the first image area is darker than the second image area, determining whether the image information satisfies a defined darkness condition at the first candidate edge, wherein the first image area and the second image area are respective areas described by the image information; selecting a subset of the plurality of candidate edges to form a selected candidate edge subset for representing physical edges of the set of objects, wherein the selecting comprises: determining whether to reserve the first candidate edge as a candidate for at least one of the physical edges representing the set of objects by including the first candidate edge in the selected subset of candidate edges based on whether the image information satisfies a defined darkness condition at the first candidate edge; and outputting a robot-interactive movement command, wherein the robot-interactive movement command is for a robot interaction between the robot and at least one object of the set of objects and is generated based on the selected subset of candidate edges. In this embodiment, the at least one processing circuit is configured to determine that the image information satisfies the defined darkness condition at the first candidate edge in response to determining that the first image region satisfies at least one of the defined darkness threshold criteria or the defined spike intensity distribution criteria. Further in this embodiment, the at least one processing circuit is configured to determine whether the first image region meets the defined darkness threshold criteria by determining whether the first image region has at least a portion of an image intensity that is darker than the defined darkness threshold. Further in this embodiment, the at least one processing circuit is configured to determine whether the first image region satisfies a spike intensity distribution criterion by determining whether the first image region has an image intensity distribution comprising: (i) a first distribution portion in which the image intensity increases as a function of position within the first image area to reach a peak level of darkness at a position within the first image area, followed by (ii) a second distribution portion in which the image intensity decreases as a function of position within the first image area away from the peak level of darkness.

Embodiment 2 includes the computing system of embodiment 1, wherein the first image region is a first pixel region forming a band of pixels representing a physical gap between a first object and a second object in the set of objects, wherein the second image region is a second pixel region immediately adjacent to the first pixel region such that a boundary forming the first candidate edge is located between the first pixel region and the second pixel region.

Embodiment 3 includes the computing system of embodiment 2, wherein the at least one processing circuit is configured to determine whether the first image region meets the defined darkness threshold criteria by determining whether the first image region has pixel intensity values that are less than the defined darkness threshold.

Embodiment 4 includes the computing system of embodiment 2 or 3, wherein the image intensity distribution of the first image region describes respective pixel intensity values of a series of pixels extending across a width dimension of the first image region, and wherein the at least one processing circuit is configured to determine whether the first image region satisfies the spike intensity distribution criterion by determining whether the image intensity distribution has a shape in which the respective pixel intensity values decrease towards a minimum pixel intensity value in the first image region and then switch to increasing away from the minimum pixel intensity value, wherein the minimum pixel intensity value is associated with a peak level of darkness in the first image region.

Embodiment 5 includes the computing system of any of embodiments 1 to 4, wherein the at least one processing circuit is configured to determine that the first image region satisfies the defined darkness condition only in response to determining that the first image region satisfies the spike intensity distribution criteria.

Embodiment 6 includes the computing system of any of embodiments 1 to 5, wherein the at least one processing circuit is configured to determine that the first image region satisfies the defined darkness condition only in response to determining that the first image region satisfies the defined darkness threshold criteria.

Embodiment 7 includes the computing system of embodiment 1, wherein the at least one processing circuit is configured to determine that the first image region satisfies the defined darkness condition only in response to determining that the first image region satisfies both the defined darkness threshold criteria and the defined spike intensity distribution criteria.

Embodiment 8 includes the computing system of any of embodiments 1-7, wherein the at least one processing circuit is configured to identify a first candidate edge formed based on a boundary between the first image region and the second image region based on the 2D image information when the image information includes the 2D image information and the 3D image information, wherein the 3D image information includes depth information for a location in the field of view of the camera.

Embodiment 9 includes the computing system of embodiment 8, wherein when: (i) when the 3D image information lacks depth information corresponding to one or more locations of the first candidate edge, or (ii) a portion of the 3D image information corresponding to the first candidate image is affected by an imaging noise level that is greater than a defined noise tolerance threshold, the at least one processing circuit is configured to perform a determination of whether to retain the first candidate edge as a candidate for representing at least one physical edge of the set of physical edges of the object.

Embodiment 10 includes the computing system of embodiment 8 or 9, wherein when the 3D image information does not satisfy the defined depth discontinuity condition at one or more locations corresponding to the first candidate edge, the at least one processing circuit is configured to perform the determination of whether to retain the first candidate edge as a candidate for representing at least one of the physical edges of the set of objects.

Embodiment 11 includes the computing system of embodiment 10, wherein the at least one processing circuit is configured to determine that the 3D image information does not satisfy the defined depth discontinuity condition at the one or more locations corresponding to the first candidate edge in response to determining that the 3D image information does not describe a depth change at the one or more locations that exceeds the defined depth difference threshold.

Embodiment 12 includes the computing system of any of embodiments 8-11, wherein the at least one processing circuit is configured to identify a second candidate edge of the plurality of candidate edges based on the 3D image information.

Embodiment 13 includes the computing system of claim 12, wherein the at least one processing circuit is configured to identify the second candidate edge based on the 3D image information by: identifying a first surface of the set of objects based on a first set of locations described by the 3D image information, the first set of locations having respective depth values that deviate from each other by no more than a defined measurement variance threshold; identifying a second surface of the set of objects based on a second set of locations described by the 3D image information, the second set of locations having respective depth values within a defined measurement variance threshold; determining an average depth value associated with the first surface as a first average depth value; determining an average depth value associated with the second surface as a second average depth value; in response to determining that a difference between the first average depth value and the second average depth value exceeds a defined depth difference threshold, a second candidate edge is identified based on a location at which a transition exists between the first surface and the second surface.

Embodiment 14 includes the computing system of embodiment 12 or 13, wherein the at least one processing circuit is configured to: when the second candidate edge is mapped to a candidate edge in the 2D image information and formed based on a boundary between two image areas that do not satisfy the defined darkness condition, the second candidate edge is identified based on the 3D image information.

Embodiment 15 includes the computing system of embodiments 1-14, wherein the at least one processing circuit is configured to perform an object recognition operation or an object registration operation based on the selected subset of candidate edges, wherein the robot interactive movement command is generated based on a result of the object recognition operation or the object registration operation.

Embodiment 16 includes the computing system of any of embodiments 1-15, wherein the at least one processing circuit is configured to select the subset of the plurality of candidate edges by determining which candidate edges to filter from the plurality of candidate edges, wherein the plurality of candidate edges form the subset of candidate edges after being filtered.

It will be apparent to one of ordinary skill in the relevant art that other suitable modifications and adaptations to the methods and applications described herein may be made without departing from the scope of any of the embodiments. The above-described embodiments are illustrative examples, and the present invention should not be construed as being limited to these specific embodiments. It is to be understood that the various embodiments disclosed herein may be combined in different combinations than those specifically presented in the description and drawings. It will also be understood that, depending on the example, certain acts or events of any process or method described herein can be performed in a different order, may be added, merged, or omitted altogether (e.g., all described acts or events may not be necessary for performing the method or process). For example, method 4000 may be modified to omit step 4002. Although various embodiments discussed above relate to steps 4002 and 4008 of method 4000, another method of the present disclosure may include identifying candidate edges based on 3D image information, as discussed above with respect to fig. 11B or 11C, and steps 4002 and 4008 may be omitted. Furthermore, although certain features of the embodiments herein are described as being performed by a single component, module or unit for clarity, it should be understood that the features and functions described herein may be performed by any combination of components, units or modules. Accordingly, various changes and modifications may be effected therein by one skilled in the art without departing from the spirit or scope of the invention as defined in the appended claims.

Claims (20)

1. A computing system, comprising:

a communication interface configured to communicate with a robot and with a camera having a camera field of view;

at least one processing circuit configured to, when a set of objects is in the camera field of view:

receiving image information representing the set of objects, wherein the image information is generated by the camera;

identifying, from the image information, a plurality of candidate edges associated with the set of objects, wherein the plurality of candidate edges are or include respective image locations or sets of physical locations that form respective candidates for representing physical edges of the set of objects;

when the plurality of candidate edges includes a first candidate edge formed based on a boundary between a first image area and a second image area, wherein the first image area is darker than the second image area, determining whether the image information satisfies a defined darkness condition at the first candidate edge, wherein the first image area and the second image area are respective areas described by the image information;

selecting a subset of the plurality of candidate edges to form a selected subset of candidate edges for representing the physical edges of the set of objects, wherein the selecting comprises: determining whether to retain a first candidate edge as a candidate for at least one of the physical edges representing the set of objects by including the first candidate edge in the selected subset of candidate edges based on whether the image information satisfies the defined darkness condition at the first candidate edge; and

outputting a robot-interactive movement command, wherein the robot-interactive movement command is for a robot interaction between the robot and at least one object of the set of objects, and is generated based on the selected subset of candidate edges,

wherein the at least one processing circuit is configured to determine that the image information satisfies the defined darkness condition at a first candidate edge in response to determining that a first image region satisfies at least one of a defined darkness threshold criterion or a defined spike intensity distribution criterion,

wherein the at least one processing circuit is configured to determine whether the first image area meets a defined darkness threshold criterion by determining whether the first image area has at least a portion with an image intensity that is darker than the defined darkness threshold, and

wherein the at least one processing circuit is configured to determine whether the first image region satisfies the spike intensity distribution criterion by determining whether the first image region has an image intensity distribution that: the image intensity distribution comprises (i) a first distribution portion in which the image intensity increases as a function of position within the first image area to reach a peak level of darkness at a position within the first image area, followed by (ii) a second distribution portion in which the image intensity decreases as a function of position within the first image area away from the peak level of darkness.

2. The computing system of claim 1, wherein the first image area is a first pixel area forming a band of pixels representing a physical gap between a first object and a second object in the set of objects, wherein the second image area is a second pixel area immediately adjacent to the first pixel area such that the boundary forming the first candidate edge is located between the first pixel area and the second pixel area.

3. The computing system of claim 2, wherein the at least one processing circuit is configured to determine whether the first image region meets the defined darkness threshold criteria by determining whether the first image region has pixel intensity values that are less than the defined darkness threshold.

4. The computing system of claim 2, wherein the image intensity distribution of the first image region describes respective pixel intensity values for a series of pixels extending across a width dimension of the first image region, and wherein the at least one processing circuit is configured to determine whether the first image region satisfies a spike intensity distribution criterion by determining whether the image intensity distribution has a shape in which the respective pixel intensity values decrease towards a minimum pixel intensity value in the first image region and then switch to increasing away from the minimum pixel intensity value, wherein the minimum pixel intensity value is associated with the peak level of darkness in the first image region.

5. The computing system of claim 1, wherein the at least one processing circuit is configured to determine that a first image region satisfies the defined darkness condition only in response to determining that the first image region satisfies the spike intensity distribution criteria.

6. The computing system of claim 1, wherein the at least one processing circuit is configured to determine that the first image region satisfies the defined darkness condition only in response to determining that the first image region satisfies the defined darkness threshold criteria.

7. The computing system of claim 1, wherein the at least one processing circuit is configured to determine that the first image region satisfies the defined darkness condition only in response to determining that the first image region satisfies both the defined darkness threshold criteria and the defined spike intensity distribution criteria.

8. The computing system of claim 1, wherein the at least one processing circuit is configured to: when the image information includes 2D image information and 3D image information, identifying a first candidate edge formed based on the boundary between a first image region and a second image region based on the 2D image information, wherein the 3D image information includes depth information for a location in the field of view of the camera.

9. The computing system of claim 8, wherein when: (i) the 3D image information lacks depth information corresponding to one or more locations of a first candidate edge, or (ii) a portion of the 3D image information corresponding to a first candidate image is affected by an imaging noise level that is greater than a defined noise tolerance threshold, the at least one processing circuit is configured to perform the determination of whether to retain a first candidate edge as a candidate for representing at least one of the physical edges of the set of objects.

10. The computing system of claim 8, wherein, when the 3D image information does not satisfy a defined depth discontinuity condition at one or more locations corresponding to a first candidate edge, the at least one processing circuit is configured to perform the determination of whether to retain the first candidate edge as a candidate for at least one of the physical edges representing the set of objects.

11. The computing system of claim 10, wherein the at least one processing circuit is configured to: determining that the 3D image information does not satisfy the defined depth discontinuity condition at one or more locations corresponding to a first candidate edge in response to determining that the 3D image information does not describe a depth change at the one or more locations that exceeds a defined depth difference threshold.

12. The computing system of claim 8, wherein the at least one processing circuit is configured to identify a second candidate edge of the plurality of candidate edges based on the 3D image information.

13. The computing system of claim 12, wherein the at least one processing circuit is configured to identify a second candidate edge based on the 3D image information by:

identifying a first surface of the set of objects based on a first set of locations described by the 3D image information, the first set of locations having respective depth values that deviate from each other by no more than a defined measurement variance threshold;

identifying a second surface of the set of objects based on a second set of locations described by the 3D image information, the second set of locations having respective depth values within the defined measurement variance threshold;

determining an average depth value associated with the first surface as a first average depth value;

determining an average depth value associated with the second surface as a second average depth value;

in response to determining that a difference between the first average depth value and the second average depth value exceeds a defined depth difference threshold, a second candidate edge is identified based on a location at which a transition exists between the first surface and the second surface.

14. The computing system of claim 12, wherein the at least one processing circuit is configured to: identifying a second candidate edge based on the 3D image information when the second candidate edge maps to a candidate edge in the 2D image information and formed based on a boundary between two image regions that do not satisfy the defined darkness condition.

15. The computing system of claim 1, wherein the at least one processing circuit is configured to perform an object recognition operation or an object registration operation based on the selected subset of candidate edges, wherein the robot-interactive movement command is generated based on a result of the object recognition operation or object registration operation.

16. The computing system of claim 1, wherein the at least one processing circuit is configured to select the subset of the plurality of candidate edges by determining which candidate edges to filter from the plurality of candidate edges, wherein the plurality of candidate edges form the subset of candidate edges after being filtered.

17. A non-transitory computer-readable medium having instructions that, when executed by at least one processing circuit of a computing system, cause the at least one processing circuit to:

receiving, by the at least one processing circuit of the computing system, representative image information, wherein the computing system is configured to: (i) a robot, and (ii) a camera communication having a camera field of view, wherein the image information is representative of a set of objects in the camera field of view and is generated by the camera;

identifying, from the image information, a plurality of candidate edges associated with the set of objects, wherein the plurality of candidate edges are or include respective image locations or sets of physical locations that form respective candidates for representing physical edges of the set of objects;

when the plurality of candidate edges includes a first candidate edge formed based on a boundary between a first image area and a second image area, wherein the first image area is darker than the second image area, determining whether the image information satisfies a defined darkness condition at the first candidate edge, wherein the first image area and the second image area are respective areas described by the image information;

selecting a subset of the plurality of candidate edges to form a selected subset of candidate edges for representing the physical edges of the set of objects, wherein the selecting comprises: determining whether to retain a first candidate edge as a candidate for at least one of the physical edges representing the set of objects by including the first candidate edge in the selected subset of candidate edges based on whether the image information satisfies the defined darkness condition at the first candidate edge; and

outputting a robot-interactive movement command, wherein the robot-interactive movement command is for a robot interaction between the robot and at least one object of the set of objects, and is generated based on the selected subset of candidate edges,

wherein the instructions cause the at least one processing circuit to determine that the image information satisfies the defined darkness condition at a first candidate edge in response to determining that a first image region satisfies at least one of a defined darkness threshold criterion or a defined spike intensity distribution criterion,

wherein the instructions cause the at least one processing circuit to determine whether the first image region meets a defined darkness threshold criterion by determining whether the first image region has at least a portion with an image intensity that is darker than the defined darkness threshold, and

wherein the instructions cause the at least one processing circuit to determine whether the first image region satisfies the spike intensity distribution criterion by determining whether the first image region has an image intensity distribution that: the image intensity distribution comprises (i) a first distribution portion in which the image intensity increases as a function of position within the first image area to reach a peak level of darkness at a position within the first image area, followed by (ii) a second distribution portion in which the image intensity decreases as a function of position within the first image area away from the peak level of darkness.

18. The non-transitory computer-readable medium of claim 17, wherein the first image region is a first pixel region forming a band of pixels representing a physical gap between a first object and a second object in the set of objects, wherein the second image region is a second pixel region immediately adjacent to the first pixel region such that the boundary forming the first candidate edge is located between the first pixel region and the second pixel region.

19. The non-transitory computer-readable medium of claim 18, wherein the instructions cause the at least one processing circuit to determine whether the first image region meets the defined darkness threshold criteria by determining whether the first image region has one or more pixel intensity values that are less than the defined darkness threshold.

20. A method performed by a computing system, the method comprising:

receiving, by the computing system, image information, wherein the computing system is configured to: (i) a robot, and (ii) a camera communication having a camera field of view, wherein the image information is representative of a set of objects in the camera field of view and is generated by the camera;

identifying, from the image information, a plurality of candidate edges associated with the set of objects, wherein the plurality of candidate edges are or include respective image locations or sets of physical locations that form respective candidates for representing physical edges of the set of objects;

when the plurality of candidate edges includes a first candidate edge formed based on a boundary between a first image area and a second image area, wherein the first image area is darker than the second image area, determining whether the image information satisfies a defined darkness condition at the first candidate edge, wherein the first image area and the second image area are respective areas described by the image information;

selecting a subset of the plurality of candidate edges to form a selected subset of candidate edges for representing the physical edges of the set of objects, wherein the selecting comprises: determining whether to retain a first candidate edge as a candidate for at least one of the physical edges representing the set of objects by including the first candidate edge in the selected subset of candidate edges based on whether the image information satisfies the defined darkness condition at the first candidate edge; and

outputting a robot-interactive movement command, wherein the robot-interactive movement command is for a robot interaction between the robot and at least one object of the set of objects, and is generated based on the selected subset of candidate edges,

wherein the image information satisfies the defined darkness condition at a first candidate edge when a first image region satisfies at least one of a defined darkness threshold criterion or a defined spike intensity distribution criterion,

wherein said determination of whether the first image area meets said defined darkness threshold criteria is performed by determining whether the first image area has at least a portion of an image intensity that is darker than a defined darkness threshold, an

Wherein the determination of whether the first image region satisfies the spike intensity distribution criterion is performed by determining whether the first image region has an image intensity distribution that: the image intensity distribution comprises (i) a first distribution portion in which the image intensity increases as a function of position within the first image area to reach a peak level of darkness at a position within the first image area, followed by (ii) a second distribution portion in which the image intensity decreases as a function of position within the first image area away from the peak level of darkness.

Images