JP7418762B2 - Computing systems, methods and non-transitory computer-readable media - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is filed in U.S. Provisional Patent Application No. 63/230,93, filed August 9, 2021, entitled “A ROBOTIC SYSTEM FOR FACILITATION TEMPLATE MATCHING AND DETECTION FOR OBJECT PICKING” Asserting the interests of No. 1, Incorporated herein by reference in its entirety.

TECHNICAL FIELD The present technology is generally directed to robotic systems, and more particularly to systems, processes, and techniques for identifying and detecting objects. More particularly, the technology can be used to identify and detect objects in containers.

Many robots (e.g. machines configured to perform physical actions automatically/autonomously) are now widely used in a variety of different fields as their performance continues to improve and costs decrease. . Robots may be used to perform various tasks (eg, manipulating or conveying objects through space), for example, in manufacturing and/or assembly, packaging and/or packaging, transportation and/or shipping, and the like. In performing tasks, robots can reproduce human actions, thereby replacing or reducing human involvement required to perform otherwise dangerous or repetitive tasks. be able to.

However, despite advancements in technology, robots often lack the sophistication necessary to replicate human interaction, which is required to perform larger and/or more complex tasks. lack. Accordingly, there remains a need for improved techniques and systems for managing motion and/or interaction between robots.

In one embodiment, a computing system is provided that is configured to generate a set of object recognition templates for identifying objects in a scene. The computing system includes at least one processing circuit, the at least one processing circuit acquiring registration data for the object, including an object model representing the object, and determining multiple viewpoints of the object model in three-dimensional space. estimating a plurality of appearances of the object model at each of the plurality of viewpoints; and generating a plurality of object recognition templates according to the plurality of appearances, each corresponding to a respective one of the plurality of appearances. and communicating the plurality of object recognition templates to a robot control system as an object recognition template set. Each of the plurality of object recognition templates represents a pose that an object may have with respect to an optical axis of a camera that generates image information of the object in the scene.

In another embodiment, a method is provided for generating a set of object recognition templates for identifying objects in a scene. The method includes obtaining registration data of an object including an object model representing the object, determining multiple viewpoints of the object model in three-dimensional space, and determining multiple appearances of the object model at each of the multiple viewpoints. estimating a plurality of object recognition templates according to the plurality of appearances, each object recognition template corresponding to a respective one of the plurality of appearances, and transmitting the plurality of object recognition templates to a robot control system for object recognition. and communicating as a set. Each of the plurality of object recognition templates represents a pose that an object may have with respect to an optical axis of a camera that generates image information of the object in the scene.

In another embodiment, a non-transitory computer-readable medium is provided operable by at least one processing circuit through a communication interface configured to communicate with a robotic system, the non-transitory computer-readable medium being operable within a scene. The method comprises executable instructions for performing a method for generating an object recognition template for identifying an object. The method includes receiving registration data of an object including an object model representing the object, performing processing to generate a plurality of viewpoints of the object model in a three-dimensional space, and, at each of the plurality of viewpoints, performing processing for estimating a plurality of appearances of an object model; performing processing for generating a plurality of object recognition templates, each of which corresponds to one of the plurality of appearances, according to the plurality of appearances; and a robot. The method includes outputting a plurality of object recognition templates to the system as an object recognition template set. Each of the plurality of object recognition templates represents a pose that an object may have with respect to an optical axis of a camera that generates image information of the object in the scene.

In another embodiment, a computing system is provided that is configured to generate an object recognition template for identifying objects in a scene. The computing system includes at least one processing circuit. The processing circuit includes a step of acquiring object information including a digitally represented object, a step of extracting two-dimensional measurement information from the object information, a step of extracting three-dimensional measurement information from the object information, and a step of extracting two-dimensional measurement information from the object information. and generating an object recognition template according to the three-dimensional measurement information.

In another embodiment, a method is provided for generating an object recognition template for identifying objects in a scene. The method includes acquiring object information including a digitally represented object, extracting two-dimensional measurement information from the object information, extracting three-dimensional measurement information from the object information, and extracting two-dimensional measurement information and and generating an object recognition template according to the three-dimensional measurement information.

In another embodiment, a non-transitory computer readable medium operable by at least one processing circuit through a communication interface configured to communicate with a robotic system for identifying an object in a scene. A non-transitory computer-readable medium having executable instructions for performing a method for generating a recognition template is provided. The method includes receiving object information including a digitally represented object, performing an operation for extracting two-dimensional measurement information from the object information, and performing an operation for extracting three-dimensional measurement information from the object information. and outputting an object recognition template to the robot system according to the two-dimensional measurement information and the three-dimensional measurement information.

In another embodiment, a computing system is provided. The computing system includes at least one processing circuit in communication with a robot having an arm and an end effector connected to the arm, and in communication with a camera having a field of view. At least one processing circuit is configured to execute instructions stored on the non-transitory computer-readable medium when there is or has been one or more objects within the field of view. The instructions executed are to obtain object image information for an object in the scene, obtain a detection hypothesis containing a corresponding object recognition template representing the template object, and detect the discrepancy between the template object and the object image information. identifying a set of template positions within the template object that correspond to the set of object positions of the object image information; and adjusting the set of template positions to converge to the set of object positions. , generating an adjusted detection hypothesis including an adjusted corresponding object recognition template according to the set of adjusted template positions.

In another embodiment, a method is provided. The method includes obtaining object image information of an object in a scene, obtaining a detection hypothesis including a corresponding object recognition template representing the template object, and identifying mismatches between the template object and the object image information. identifying a set of template positions in the template object that correspond to the set of object positions in the object image information; adjusting the set of template positions to converge to the set of object positions; generating a tailored detection hypothesis that includes a tailored corresponding object recognition template according to the set of template positions of the object recognition template.

In another embodiment, a non-transitory computer-readable medium operable by at least one processing circuit through a communication interface configured to communicate with a robotic system includes a method for refining a detection hypothesis. A non-transitory computer readable medium having executable instructions for execution is provided. The method includes receiving object image information of an object in a scene, receiving a detection hypothesis including a corresponding object recognition template representing the template object, and identifying a mismatch between the template object and the object image information. performing an operation, identifying a set of template positions within the template object that correspond to a set of object positions in the object image information; and adjusting the set of template positions to converge to the set of object positions. and outputting an adjusted detection hypothesis to the robotic system including an adjusted corresponding object recognition template according to the adjusted set of template positions.

In another embodiment, a computing system is provided. The computing system includes at least one processing circuit in communication with a robot having an arm and an end effector connected to the arm and in communication with a camera having a field of view. The object is or is configured to execute instructions stored on the non-transitory computer-readable medium. The instructions executed are to obtain object image information for objects in the scene, obtain a set of detection hypotheses each containing a corresponding object recognition template representing a template object, and obtain a set of detection hypotheses for each detection in the set of detection hypotheses. testing a hypothesis; What is verified is the occlusion validator score, point cloud validator score, hole matching validator, based on the comparison between the 3D information of the object recognition template of the detection hypothesis and the 3D information of the object image information corresponding to the object. generating a plurality of three-dimensional validation scores including at least one of a score and a normal vector validator score; and comparing two-dimensional information of a corresponding object recognition template of a detection hypothesis and three-dimensional information of object image information. generating a plurality of two-dimensional validation scores including at least one of a rendered matching validator score and a template matching validator score based on the plurality of three-dimensional validation scores and the plurality of two-dimensional validation scores. Accordingly, by filtering detection hypotheses from the set of detection hypotheses and detecting objects in the scene according to the unfiltered detection hypotheses remaining in the set of detection hypotheses after validation.

In another embodiment, a method is provided. The method includes obtaining object image information of objects in a scene, obtaining a set of detection hypotheses each containing a corresponding object recognition template representing a template object, and validating each detection hypothesis in the set of detection hypotheses. and include. What is verified is the occlusion validator score, point cloud validator score, hole matching validator, based on the comparison between the 3D information of the object recognition template of the detection hypothesis and the 3D information of the object image information corresponding to the object. generating a plurality of 3D verification scores including at least one of a score and a normal vector validator score; and 2D information of a corresponding object recognition template of the detection hypothesis and 3D information of the object image information. generating a plurality of two-dimensional validation scores, including at least one of a rendered matching validator score and a template matching validator score, based on a comparison of a plurality of three-dimensional validation scores and a plurality of two-dimensional validation scores; This is done by filtering detection hypotheses from the set of detection hypotheses according to the dimensional validation score and detecting objects in the scene according to the unfiltered detection hypotheses remaining in the set of detection hypotheses after validation.

In another embodiment, a non-transitory computer readable medium operable by at least one processing circuit through a communication interface configured to communicate with a robotic system to perform a method for testing a detection hypothesis. A non-transitory computer-readable medium is provided having executable instructions to do so. The method includes receiving object image information for objects in a scene and receiving a set of detection hypotheses, each detection hypothesis including a corresponding object recognition template representing a template object. Based on receiving and comparing the three-dimensional information of the object recognition template of the detection hypothesis and the three-dimensional information of the object image information corresponding to the object, an occlusion validator score, a point cloud validator score, a hole matching validator are calculated. performing a process of generating a plurality of three-dimensional verification scores including at least one of a score and a normal vector validator score, and generating two-dimensional information and object image information of an object recognition template corresponding to a detection hypothesis. performing a process of generating a plurality of two-dimensional validation scores, including at least one of a rendered matching validator score and a template matching validator score, based on the comparison with the three-dimensional information; performing a process of filtering a detection hypothesis from a set of detection hypotheses according to a 3D validation score and a plurality of 2D validation scores; and outputting the detected objects in the scene to a robotic system.

1 illustrates a system for performing or facilitating object detection, identification, and retrieval according to embodiments herein;

1 illustrates one embodiment of a system for performing or facilitating object detection, identification, and retrieval according to embodiments herein.

3 illustrates another embodiment of a system for performing or facilitating object detection, identification, and retrieval according to embodiments herein.

3 illustrates yet another embodiment of a system for performing or facilitating object detection, identification, and retrieval according to embodiments herein.

FIG. 1 is a block diagram illustrating a computing system configured to perform or facilitate object detection, identification, and retrieval consistent with embodiments herein.

FIG. 1 is a block diagram illustrating one embodiment of a computing system configured to perform or facilitate object detection, identification, and retrieval consistent with embodiments herein.

FIG. 2 is a block diagram illustrating another embodiment of a computing system configured to perform or facilitate object detection, identification, and retrieval consistent with embodiments herein.

FIG. 2 is a block diagram illustrating yet another embodiment of a computing system configured to perform or facilitate object detection, identification, and retrieval consistent with embodiments herein.

2 is an example of image information processed by the system and consistent with embodiments herein;

2 is another example of image information processed by the system and consistent with embodiments herein.

1 illustrates an example environment for operating a robotic system, according to embodiments herein.

1 illustrates an example environment for object detection, identification, and retrieval by a robotic system consistent with embodiments herein.

1 provides a flow diagram illustrating the overall flow of methods and processes for object detection, identification, and retrieval according to embodiments herein.

3 illustrates an example of object registration data consistent with embodiments herein.

3 illustrates a method for generating object recognition templates consistent with embodiments herein.

3 illustrates aspects of a method for generating object recognition templates consistent with embodiments herein. 3 illustrates aspects of a method for generating object recognition templates consistent with embodiments herein.

4 illustrates a method for generating object recognition templates consistent with embodiments herein.

4 illustrates aspects of a method for generating object recognition templates consistent with embodiments herein. 3 illustrates aspects of a method for generating object recognition templates consistent with embodiments herein. 2 illustrates aspects of a method for generating object recognition templates consistent with embodiments herein. 2 illustrates aspects of a method for generating object recognition templates consistent with embodiments herein.

3 illustrates a method of object identification and hypothesis generation via template matching consistent with embodiments herein. 2 illustrates a method of object identification and hypothesis generation via template matching consistent with embodiments herein.

FIG. 11 illustrates a method for refining detection hypotheses consistent with embodiments herein.

4 illustrates aspects of a method for refining a detection hypothesis consistent with embodiments herein. 4 illustrates aspects of a method for refining a detection hypothesis consistent with embodiments herein. 3 illustrates aspects of methods for refining detection hypotheses consistent with embodiments herein.

3 illustrates a method for testing a detection hypothesis consistent with embodiments herein.

3 illustrates aspects of methods for refining detection hypotheses consistent with embodiments herein.

Systems and methods related to object detection, identification, and retrieval are described herein. Specifically, the systems and methods of the present disclosure may facilitate detection, identification, and retrieval of objects where they are located within a container. As discussed herein, the object may be metal or other material and may be located within a container such as a box, bottle, crate, or the like. The objects may be located within the container in an unorganized or irregular manner, such as, for example, a box filled with screws. Detection and identification of objects in such situations can be difficult due to the irregular arrangement of objects, but the systems and methods discussed herein are useful for detecting and identifying objects that are arranged in a regular or semi-regular manner. Detection, identification, and object retrieval may be similarly improved. Accordingly, the systems and methods described herein are designed to identify individual objects among a plurality of objects, and the individual objects may be located at different positions, at different angles, etc. The systems and methods discussed herein may include robotic systems. A robotic system configured according to embodiments herein may autonomously perform integrated tasks by coordinating the operations of multiple robots. The robotic system controls, issues commands, receives information from robotic devices and sensors, and accesses and analyzes data generated by robotic devices, sensors, and cameras, as described herein. and of robotic devices, actuators, sensors, cameras, and computing systems configured to process and generate data or information usable for controlling the robotic system and to plan actions of the robotic devices, sensors, and cameras. Any suitable combination may be included. As used herein, a robotic system does not require immediate access to or control of robotic actuators, sensors, or other devices. A robotic system may be a computing system configured to improve the performance of such robotic actuators, sensors, and other devices through receiving, analyzing, and processing information, as described herein. .

The techniques described herein provide technological improvements to robotic systems configured for use in object identification, detection, and retrieval. The technical improvements described herein increase the speed, accuracy, and precision of these tasks, making it easier to detect, identify, and remove objects from containers. The robotic and computational systems described herein address the technical problem of identifying, detecting, and removing objects from containers, where objects may be randomly located. Addressing this technical problem improves object identification, detection, and retrieval techniques.

This application refers to systems and robotic systems. A robotic system may include robotic actuator components (eg, robotic arms, robotic grippers, etc.), various sensors (eg, cameras, etc.), and various computing or control systems, as discussed herein. As discussed herein, a computing system or control system may be referred to as "controlling" various robot components, such as a robot arm, robot gripper, camera, etc. Such "control" may refer to the direct control and interaction of various actuators, sensors, and other functional aspects of the robotic components. For example, a computing system may control a robotic arm by issuing or providing various motors, actuators, and sensors with all of the necessary signals to cause robotic movement. Such "control" may also refer to the issuance of abstract or indirect commands to further robot control systems that convert such commands into the signals necessary to cause robot movement. For example, a computing system may control a robotic arm by issuing commands that describe a trajectory or a destination location for the robotic arm to travel, and a further robotic control system associated with the robotic arm may control such commands. may be received and interpreted and then provide the necessary direct signals to the various actuators and sensors of the robot arm to cause the required movement.

Specifically, the techniques described herein assist a robotic system in interacting with a target object among a plurality of objects within a container. Detection, identification, and retrieval of objects from containers involves several steps, including generation of appropriate object recognition templates, extraction of features that can be used for identification, and generation, refinement, and validation of detection hypotheses. is necessary. For example, due to the possibility of irregular placement of objects, objects can be recognized and It may be necessary to identify.

Specific details are set forth below to provide an understanding of the techniques of this disclosure. In embodiments, the techniques introduced herein may be practiced without each specific detail disclosed herein. In other instances, well-known features, such as particular functions or routines, are not described in detail to avoid unnecessarily obscuring the present disclosure. Reference herein to "an embodiment," "an embodiment," or the like means that the particular feature, structure, material, or characteristic described is included in at least one embodiment of the present disclosure. . Therefore, the appearances of such phrases herein are not necessarily all referring to the same embodiment. However, such references are not necessarily mutually exclusive. Furthermore, the specific features, structures, materials, or characteristics described with respect to any one embodiment may be combined with those of any other embodiment in any appropriate manner, unless such items are mutually exclusive. Can be combined with It is to be understood that the various embodiments shown in the figures are merely exemplary representations and are not necessarily drawn to scale.

A few details describing structures or processes that are well known and often associated with robotic systems and subsystems may unnecessarily obscure some important aspects of the disclosed techniques. , are not included in the description below for clarity. Additionally, while the following disclosure describes several embodiments of different aspects of the technology, several other embodiments may have different configurations or different components than those described in this section. You may. Accordingly, the disclosed technology may have other embodiments with additional elements or without some of the elements described below.

Many embodiments or aspects of the present disclosure described below may take the form of computer- or controller-executable instructions, including routines executed by a programmable computer or controller. Those of ordinary skill in the relevant art will appreciate that the disclosed techniques may be practiced on or with computer or controller systems other than those shown and described below. The techniques described herein may be implemented on a special purpose computer or data processor that is specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. can be realized. Accordingly, the terms "computer" and "controller" as used generally herein refer to any data processor, Internet appliances and handheld devices (such as palmtop computers, wearable computers, cellular or mobile phones, multi-processor systems, processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information processed by these computers and controllers may be presented on any suitable display medium, including liquid crystal displays (LCDs). Instructions for performing computer- or controller-executable tasks may be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. The instructions may be included in any suitable memory device, including, for example, a flash drive, a USB device, and/or other suitable media.

The terms "coupled" and "connected", along with their derivatives, may be used herein to describe structural relationships between components. It should be understood that these terms are not intended as synonyms for each other. Rather, in certain embodiments, "connected" can be used to indicate that two or more elements are in direct contact with each other. Unless the context clearly indicates otherwise, the term "coupled" means that two or more elements are in direct or indirect contact with each other (with other intervening elements between them) or used to indicate that elements cooperate or interact with each other (e.g., in a causal relationship, such as for sending/receiving signals or for function calls), or both. obtain.

Any reference herein to image analysis by a computational system may be performed in accordance with or using spatial structure information, which may include depth information that describes depth values for each of the various locations relative to a selected point. . Depth information may be used to identify objects or estimate how objects are spatially located. In some instances, the spatial structure information may include or be used to generate a point cloud that describes locations on one or more surfaces of an object. Spatial structure information is only one form of possible image analysis; other forms known to those skilled in the art may be used in accordance with the methods described herein.

FIG. 1A shows a system 1000 for performing object detection, or more specifically, object recognition. More particularly, system 1000 may include a computing system 1100 and a camera 1200. In this example, camera 1200 describes or otherwise represents the environment in which camera 1200 is located, or more specifically represents the environment within the field of view of camera 1200 (also referred to as camera field of view). The image information may be configured to generate image information. The environment may be, for example, a warehouse, manufacturing plant, retail space, or other facility. In such instances, the image information may represent objects located in such facilities, such as boxes, bins, cases, crates, pallets, or other containers. System 1000 uses the image information to distinguish between individual objects within the camera field of view, perform object recognition or object registration based on the image information, and/or as discussed in more detail below. May be configured to generate, receive, and/or process image information, such as executing a robot interaction plan based on the image information (the terms “and/or” and “or” are used interchangeably in this disclosure). used). A robot interaction plan may be used to control a robot at a facility, for example, to facilitate robot interaction between the robot and a container or other object. Computing system 1100 and camera 1200 may be located in the same facility or may be located remotely from each other. For example, computing system 1100 may be part of a cloud computing platform hosted in a data center remote from a warehouse or retail space and may communicate with camera 1200 via a network connection.

In one embodiment, camera 1200 (which may also be referred to as an image sensing device) may be a 2D camera and/or a 3D camera. For example, FIG. 1B depicts a system 1500A, which may be an embodiment of system 1000, including a computing system 1100, and a camera 1200A and a camera 1200B, both of which may be embodiments of camera 1200. In this example, camera 1200A may be a 2D camera configured to generate 2D image information that includes or forms a 2D image that describes the visual appearance of the environment within the camera's field of view. Camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or spatial structure sensing device) that is configured to generate 3D image information that includes or forms spatial structure information about the environment within the camera's field of view. . The spatial structure information may include depth information (eg, a depth map) that describes depth values for each of various locations relative to camera 1200B, such as locations on the surface of various objects within the field of view of camera 1200. These locations on the field of view of the camera or on the surface of the object may also be referred to as physical locations. Depth information in this example may be used to estimate how an object is spatially positioned in three-dimensional (3D) space. In some instances, the spatial structure information may include or be used to generate a point cloud that describes locations on one or more surfaces of an object that are within the field of view of camera 1200B. . More specifically, the spatial structure information may describe various positions on the structure of the object (also referred to as the object structure).

In one embodiment, system 1000 may be a robotic manipulation system to facilitate robotic interaction between the robot and various objects in the environment of camera 1200. For example, FIG. 1C shows a robotic manipulation system 1500B, which may be an embodiment of the systems 1000/1500A of FIGS. 1A and 1B. Robot manipulation system 1500B may include computing system 1100, camera 1200, and robot 1300. As mentioned above, robot 1300 may be used to interact with one or more objects within the environment of camera 1200, such as boxes, crates, bins, pallets, or other containers. For example, robot 1300 may be configured to pick up containers from one location and move them to another location. In some cases, robot 1300 may be used to perform a depalletization operation in which groups of containers or other objects are unloaded and transferred to, for example, a belt conveyor. In some embodiments, camera 1200 may be attached to robot 1300 or robot 3300, discussed below. This is also known as a handheld camera or handheld camera solution. Camera 1200 may be attached to robot arm 3320 of robot 1300. Robotic arm 3320 may then move to various lifting areas and generate image information regarding those areas. In some embodiments, camera 1200 may be remote from robot 1300. For example, camera 1200 may be mounted to the ceiling of a warehouse or other structure and remain stationary relative to the structure. In some embodiments, multiple cameras 1200 may be used, including multiple cameras 1200 separate from robot 1300 and/or separate cameras 1200 from robot 1300 used with handheld camera 1200. In some embodiments, the camera 1200 or cameras 1200 are separate from the robot 1300 used for object manipulation, such as a robotic arm, gantry, or other automated system configured for camera movement. , may be attached to a dedicated robotic system, or may be fixed. Throughout this specification, there may be discussion of "controlling" or "controlling" camera 1200. For handheld camera solutions, controlling the camera 1200 also includes controlling the robot 1300 to which the camera 1200 is mounted or attached.

In one embodiment, the computing system 1100 of FIGS. 1A-1C may form or be incorporated into a robot 1300, which may also be referred to as a robot controller. A robot control system may be included in system 1500B to generate commands for robot 1300, such as, for example, robot interaction movement commands to control robot interaction between robot 1300 and a container or other object. It is configured as follows. In such embodiments, computing system 1100 may be configured to generate such commands based on image information generated by camera 1200, for example. For example, computing system 1100 may be configured to determine a motion plan based on the image information, where the motion plan may be intended to, for example, grasp or otherwise pick up an object. Computing system 1100 may generate one or more robot interaction movement commands to execute the motion plan.

In embodiments, computing system 1100 may form or be part of a vision system. The vision system may be, for example, a system that generates visual information that describes the environment in which the robot 1300 is located, or alternatively or additionally, describes the environment in which the camera 1200 is located. The visual information may include the 3D image information discussed above, and/or the 2D image information, or some other image information. In some scenarios, where computing system 1100 forms a vision system, the vision system may be part of the robot control system discussed above or may be separate from the robot control system. If the vision system is separate from the robot control system, the vision system may be configured to output information that describes the environment in which the robot 1300 is located. Information can be output to a robot control system that can receive such information from the vision system and execute a motion plan and/or generate robot interaction movement commands based on the information. Further information regarding the vision system is detailed below.

In one embodiment, computing system 1100 connects to a local computer bus, such as a peripheral component interconnect (PCI) bus, through a dedicated wired communications interface, such as an RS-232 interface, a universal serial bus (USB) interface, and/or a peripheral component interconnect (PCI) bus. The camera 1200 and/or the robot 1300 may communicate with the camera 1200 and/or the robot 1300 by a direct connection, such as a connection provided through. In one embodiment, computing system 1100 may communicate with camera 1200 and/or robot 1300 via a network. The network can be any type and/or form of network, such as a personal area network (PAN), a local area network (LAN), e.g. an intranet, a metropolitan area network (MAN), a wide area network (WAN), or the Internet. obtain. The network uses, for example, Ethernet protocol, Internet protocol group (TCP/IP), ATM (Asynchronous Transfer Mode) technology, SONET (Synchronous Optical Networking) protocol, or SDH (Synchronous Digital Hiera). different techniques of protocols, including rchy) protocols, and Layers or stacks may be utilized.

In embodiments, computing system 1100 may communicate information directly with camera 1200 and/or robot 1300, or may communicate via intermediate storage or, more broadly, an intermediate non-transitory computer-readable medium. . For example, FIG. 1D may be an embodiment of a system 1000/1500A/1500B that includes a non-transitory computer-readable medium 1400 that may be external to computing system 1100, e.g. A system 1500C is shown that can act as an external buffer or repository for storage. In one such example, computing system 1100 may retrieve or otherwise receive image information from non-transitory computer-readable medium 1400. Examples of non-transitory computer-readable media 1400 include electronic storage, magnetic storage, optical storage, electromagnetic storage, semiconductor storage, or any suitable combination thereof. Non-transitory computer-readable media may include, for example, a computer diskette, hard disk drive (HDD), solid state drive (SDD), random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), or (flash memory), static random access memory (SRAM), portable compact disc read only memory (CD-ROM), digital versatile disc (DVD), and/or memory stick.

As mentioned above, camera 1200 can be a 3D camera and/or a 2D camera. A 2D camera may be configured to generate 2D images, such as color or grayscale images. The 3D camera may be a depth sensing camera, or any other type of 3D camera, such as a time-of-flight (TOF) camera or a structured light camera. In some cases, the 2D camera and/or 3D camera may include an image sensor, such as a charge coupled device (CCD) sensor and/or a complementary metal oxide semiconductor (CMOS) sensor. In one embodiment, the 3D camera includes a laser, a LIDAR device, an infrared device, a light/dark sensor, a motion sensor, a microwave detector, an ultrasound detector, a radar detector, or a 3D camera that captures depth information or other spatial structure information. It may include any other device configured to capture.

As described above, image information may be processed by computing system 1100. In one embodiment, computing system 1100 includes a server (e.g., having one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smartphone, a tablet computing device, and/or It may also include or be configured as any other computing system. In one embodiment, any or all of the functionality of computing system 1100 may be performed as part of a cloud computing platform. Computing system 1100 may be a single computing device (eg, a desktop computer) or may include multiple computing devices.

FIG. 2A provides a block diagram illustrating one embodiment of a computing system 1100. Computing system 1100 in this embodiment includes at least one processing circuit 1110 and non-transitory computer readable medium(s) 1120. In some instances, processing circuitry 1110 includes a processor (e.g., a central processing unit) configured to execute instructions (e.g., software instructions) stored on non-transitory computer-readable medium 1120 (e.g., computer memory). unit (CPU), dedicated computer, and/or on-board server). In some embodiments, the processor may be included in a separate/standalone controller that is operably coupled to other electronic/electrical devices. The processor may execute program instructions to control/interface with other devices, thereby causing computing system 1100 to perform actions, tasks, and/or processing. In one embodiment, processing circuitry 1110 includes one or more processors, one or more processing cores, a programmable logic controller (“PLC”), an application specific integrated circuit (“ASIC”), a programmable gate array (“PGA”) ), field programmable gate arrays (“FPGAs”), any combination thereof, or any other processing circuitry.

In one embodiment, non-transitory computer-readable media 1120 that is part of computing system 1100 may be an alternative to or in addition to intermediate non-transitory computer-readable media 1400 discussed above. Non-transitory computer readable medium 1120 can be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or, for example, a computer diskette, a hard disk drive (HDD), a solid state drive (SSD), a random storage device, etc. Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable Read Only Memory (EPROM or Flash Memory), Static Random Access Memory (SRAM), Portable Compact Disk Read Only Memory (CD-ROM), Digital Versatile Memory The storage device may be any suitable combination thereof, such as a disc (DVD), a memory stick, any combination thereof, or any other storage device. In some instances, non-transitory computer-readable medium 1120 may include multiple storage devices. In certain embodiments, non-transitory computer-readable medium 1120 is configured to store image information generated by camera 1200 and received by computing system 1100. In some instances, non-transitory computer-readable medium 1120 may store one or more object recognition templates used to perform the methods and processes discussed herein. Non-transitory computer-readable medium 1120 may alternatively or additionally contain computer-readable program instructions that, when executed by processing circuitry 1110, cause processing circuitry 1110 to perform one or more methodologies described herein. Can be memorized.

FIG. 2B depicts computing system 1100A, which is one embodiment of computing system 1100, and includes communication interface 1131. FIG. Communication interface 1131 may be configured to receive image information generated by camera 1200 of FIGS. 1A-1D, for example. Image information may be received via the intermediate non-transitory computer readable medium 1400 or network discussed above, or via a more direct connection between camera 1200 and computing system 1100/1100A. In one embodiment, communication interface 1131 may be configured to communicate with robot 1300 of FIG. 1C. If the computing system 1100 is external to the robot control system, a communication interface 1131 of the computing system 1100 may be configured to communicate with the robot control system. Communication interface 1131 may also be referred to as a communication component or communication circuit, and may include, for example, communication circuitry configured to perform communications over a wired or wireless protocol. By way of example, the communication circuitry may include an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth controller, a PCI bus controller, any other communication circuitry, or a combination thereof.

In one embodiment, as illustrated in FIG. 2C, non-transitory computer-readable medium 1120 may include a storage space 1125 configured to store one or more data objects discussed herein. For example, the storage space may store object recognition templates, detection hypotheses, image information, object image information, robot arm movement commands, and any additional data objects that the computing systems discussed herein may need access to. .

In one embodiment, processing circuitry 1110 may be programmed with one or more computer readable program instructions stored on non-transitory computer readable medium 1120. For example, FIG. 2D shows computing system 1100C, which is one embodiment of computing system 1100/1100A/1100B, in which processing circuitry 1110 includes an object recognition module 1121, a motion planning module 1129, and an object manipulation planning module 1126. Programmed by one or more modules. Processing circuitry 1110 may be further programmed with hypothesis generation module 1128, object registration module 1130, template generation module 1132, feature extraction module 1134, hypothesis refinement module 1136, and hypothesis verification module 1138. Each of the modules described above is a computer readable device configured to perform a particular task when instantiated on one or more of the processors, processing circuits, computing systems, etc. described herein. May represent program instructions. Each of the modules described above may cooperate with each other to accomplish the functionality described herein. Various aspects of the functionality described herein may be performed by one or more of the software modules described above, and the software modules and their descriptions are representative of the systems disclosed herein. It is not understood as limiting structure. For example, although a particular task or function may be described with respect to a particular module, the task or function may be performed by different modules as appropriate. Additionally, the system functions described herein may be performed by different sets of software modules configured with different functional subdivisions or assignments.

In one embodiment, object recognition module 1121 may be configured to acquire and analyze image information as discussed throughout this disclosure. The methods, systems, and techniques discussed herein regarding image information may use object recognition module 1121. The object recognition module may be further configured for object recognition tasks related to object identification, as discussed herein.

Motion planning module 1129 may be configured to plan and execute robot movements. For example, motion planning module 1129 may interact with other modules described herein to plan motion of robot 3300 for object retrieval operations and camera placement operations. The methods, systems, and techniques discussed herein regarding robot arm movements and trajectories may be implemented by motion planning module 1129.

Object manipulation planning module 1126 is configured to plan and execute object manipulation activities of the robotic arm, such as grasping and releasing objects, and executing robotic arm commands to assist and facilitate such grasping and releasing. You can.

Hypothesis generation module 1128 may be configured to perform template matching and recognition tasks to generate detection hypotheses, eg, as described with respect to FIGS. 10A-10B. Hypothesis generation module 1128 may be configured to interact or communicate with any other necessary modules.

Object registration module 1130 may be configured to obtain, store, generate, and otherwise process object registration information that may be needed for various tasks discussed herein. Object registration module 1130 may be configured to interact or communicate with any other necessary modules.

Template generation module 1132 may be configured to complete object recognition template generation tasks, for example, as discussed herein in connection with FIGS. 6-9D. Template generation module 1132 may be configured to interact with object registration module 1130, feature extraction module 1134, and any other necessary modules.

Feature extraction module 1134 may be configured, for example, to complete the feature extraction and generation tasks discussed herein in connection with FIGS. 8-9D. Feature extraction module 1134 may be configured to interact with object registration module 1130, template generation module 1132, hypothesis generation module 1128, and any other necessary modules.

Hypothesis refinement module 1136 may be configured to complete hypothesis refinement tasks, for example, as discussed herein in connection with FIGS. 11-12C. Hypothesis refinement module 1136 may be configured to interact with object recognition module 1121 and hypothesis generation module 1128, as well as any other necessary modules.

Hypothesis testing module 1138 may be configured to complete hypothesis testing tasks, for example, as discussed herein in connection with FIGS. 13-14. Hypothesis validation module 1138 may be configured to interact with object registration module 1130, feature extraction module 1134, hypothesis generation module 1128, hypothesis refinement module 1136, and any other necessary modules.

2E, 2F, 3A, and 3B, methods associated with object recognition module 1121 that may be implemented for image analysis are described. 2E and 2F illustrate example image information associated with an image analysis method, while FIGS. 3A and 3B illustrate an example robot environment associated with an image analysis method. References herein relating to image analysis by a computing system may be performed in accordance with or using spatial structure information, which may include depth information describing respective depth values of various locations relative to a selected point. . Depth information may be used to identify objects or estimate how objects are spatially located. In some instances, the spatial structure information may include or be used to generate a point cloud that describes locations on one or more surfaces of an object. Spatial structure information is only one form of possible image analysis; other forms known to those skilled in the art may be used in accordance with the methods described herein.

In embodiments, computing system 1100 may obtain image information representative of objects within the camera field of view (eg, 3200) of camera 1200. The steps and techniques described below for acquiring image information may hereinafter be referred to as image information capture process 3001. In some examples, the object may be one object 5012 from a plurality of objects 5012 within the scene 5013 of the field of view 3200 of the camera 1200. The image information 2600, 2700 may be generated by a camera (e.g., 1200) when the object 5012 is (or has been) in the camera field of view 3200, and may capture one or more of the individual objects 5012 or the scene 5013. May be written. Object appearance describes the appearance of object 5012 from the perspective of camera 1200. If there are multiple objects 5012 in the camera field of view, the camera optionally displays image information representing multiple objects or a single object (such image information regarding a single object may be referred to as object image information). can be generated. Image information may be generated by a camera (e.g., 1200) when a group of objects is (or has been) in camera field of view, and may include, for example, 2D image information and/or 3D image information.

As an example, FIG. 2E illustrates a first set of image information, more specifically 2D image information 2600, which is generated by camera 1200 and which is generated by object 3410A/3410B of FIG. 3A, as described above. /3410C/ 3410D . More specifically, 2D image information 2600 may be a grayscale or color image and may describe the appearance of object 3410A/3410B/3410C/3410D from the perspective of camera 1200. In one embodiment, 2D image information 2600 may correspond to a single color channel (eg, a red, green, or blue channel) of a color image. When camera 1200 is positioned above objects 3410A/3410B/3410C/ 3410D , 2D image information 2600 may represent the appearance of the respective upper surfaces of objects 3410A/3410B/3410C/3410D. In the example of FIG. 2E, 2D image information 2600 includes respective portions 2000A/2000B/2000C/2000D/2550, also referred to as image portions or object image information, representing respective surfaces of objects 3410A/3410B/ 3410C /3410D. may be included. In FIG. 2E, each image portion 2000A/2000B/2000C/2000D/2550 of 2D image information 2600 may be an image region or, more specifically, a pixel region (if the image is formed by pixels). Each pixel within the pixel region of 2D image information 2600 may be characterized as having a position described by a set of coordinates [U,V], and may be defined in the camera coordinate system or as shown in FIGS. 2E and 2F. It may have values for some other coordinate system. Each pixel may also have an intensity value, such as a value between 0 and 255 or between 0 and 1023. In further embodiments, each of the pixels may include any additional information associated with the pixel in various formats (e.g., hue, saturation, intensity, CMYK, RGB, etc.).

As mentioned above, the image information may be all or a portion of an image, such as 2D image information 2600, in some embodiments. In an example, computing system 1100 may be configured to extract image portion 2000A from 2D image information 2600 to obtain only image information associated with corresponding object 3410A. If an image portion (such as image portion 2000A) is directed toward a single object, it may be referred to as object image information. Object image information need not only contain information about the object at which it is directed. For example, the object toward which it is directed may be located near, below, above, or otherwise near one or more other objects. In such cases, the object image information may include information about the object toward which it is directed, as well as one or more adjacent objects. For example, computing system 1100 may extract image portion 2000A by performing image segmentation or other analysis or processing operations based on 2D image information 2600 and/or 3D image information 2700 shown in FIG. 2F. In some embodiments, image segmentation or other operations include detecting locations in images where physical edges of objects (e.g., edges of objects) appear in 2D image information 2600, and detecting locations in such images. Use may include identifying object image information that is limited to representing individual objects within the camera field of view (eg, 3200), as well as substantially excluding other objects. "Substantially exclude" means that the image segmentation or other processing technique is designed and configured to exclude non-target objects from object image information, but may be subject to errors and noise may be present. It is meant to be understood that various other factors may result in the inclusion of portions of other objects.

FIG. 2F illustrates an example where the image information is 3D image information 2700. More specifically, the 3D image information 2700 includes, for example, the depth of each of various locations on one or more surfaces (e.g., the top surface, or other external surface) of the object 3410A/3410B/3410C/3410D. It may include a depth map or point cloud showing the values. In some embodiments, image segmentation operations for extracting image information include detecting image locations where physical edges of objects (e.g., edges of a box) appear in 3D image information 2700, and Image locations may include identifying image portions (eg, 2730) that are limited to representing individual objects within the camera field of view (eg, 3410A).

Each depth value may be relative to the camera 1200 generating the 3D image information 2700, or may be relative to some other reference point. In some embodiments, 3D image information 2700 may include a point cloud that includes respective coordinates for various structural locations of an object within the camera field of view (eg, 3200). In the example of FIG. 2F, the point cloud may include respective sets of coordinates that describe the locations of respective surfaces of objects 3410A/3410B/3410C/3410D . The coordinates may be 3D coordinates, such as [X Y Z] coordinates, and may have values relative to the camera coordinate system, or some other coordinate system. For example, the 3D image information 2700 may include a first image portion 2710, also referred to as an image portion, that indicates respective depth values for a set of locations 2710 ₁ -2710 _n , also referred to as physical locations on the surface of the object 3410D. . Furthermore, the 3D image information 2700 may further include a second portion, a third portion, a fourth portion, and a fifth portion 2720, 2730, 2740, and 2750. These parts then further indicate respective depth values for the set of locations, which may be represented by 2720 ₁ to 2720 _n , 2730 ₁ to 2730 _n , 2740 ₁ to 2740 _n , and 2750 ₁ to 2750 _n , respectively. obtain. These figures are merely examples; any number of objects with corresponding image portions may be used. As mentioned above, the acquired 3D image information 2700 may, in some instances, be part of a first set of 3D image information 2700 generated by a camera. In the example of FIG. 2F , if the acquired 3D image information 2700 represents object 3410A of FIG. 3A, the 3D image information 2700 may be narrowed to refer only to image portion 2710. Similar to the discussion of 2D image information 2600, identified image portions 2710 may be associated with individual objects and may be referred to as object image information. Accordingly, object image information as used herein may include 2D and/or 3D image information.

In one embodiment, image normalization operations may be performed by computing system 1100 as part of obtaining image information. An image normalization operation may involve transforming an image or image portion produced by camera 1200 to produce a transformed image or transformed image portion. For example, the acquired image information, which may include 2D image information 2600, 3D image information 2700, or a combination of the two, may cause the image information to change in perspective, object pose, and lighting conditions associated with visual descriptive information. This is the case when an image can be subjected to a normalization operation in order to attempt to do so. Such normalization may be performed to facilitate more accurate comparisons between image information and model (eg, template) information. A viewpoint may refer to the pose of an object relative to the camera 1200 and/or the angle at which the camera 1200 views the object when the camera 1200 generates an image representing the object.

For example, image information may be generated during an object recognition operation in which a target object is within the camera field of view 3200. Camera 1200 may generate image information representative of the target object when the target object has a particular pose with respect to the camera. For example, the target object may have an orientation with its top surface perpendicular to the optical axis of camera 1200. In such examples, the image information generated by camera 1200 may represent a particular perspective, such as a top view of the target object. In some instances, when camera 1200 is generating image information during an object recognition operation, image information may be generated at particular lighting conditions, such as illumination intensity. In such instances, the image information may represent a particular illumination intensity, illumination color, or other illumination condition.

In one embodiment, the image normalization operation includes an image or image portion of a scene generated by the camera to better match the image or image portion to viewpoint and/or lighting conditions associated with information in the object recognition template. This may involve adjusting the Adjustment may involve transforming the image or image portion to produce a transformed image that matches at least one of an object pose or a lighting condition associated with visual description information of the object recognition template.

Perspective adjustment may involve processing, warping, and/or shifting images of a scene so that the images represent the same viewpoint as the visual description information that may be included within the object recognition template. Processing may include, for example, changing the color, contrast, or illumination of the image, warping the scene may include changing the size, dimensions, or proportions of the image, and shifting the image may include changing the position of the image. , orientation, or rotation. Exemplary embodiments use processing, warping, and/or shifting to orient objects in images of a scene to orientations and orientations that match or better correspond to the visual description information of the object recognition template. /or may be changed to have a size. If the object recognition template describes a front view (eg, a top view) of some object, the image of the scene may be warped to also represent a front view of the object in the scene.

Further aspects of the object recognition methods implemented herein are described in U.S. Patent Application No. 16/991,510, filed August 12, 2020; , 466, each of which is incorporated herein by reference.

In various embodiments, the terms "computer readable instructions" and "computer readable program instructions" are used to describe software instructions or computer code that are configured to perform various tasks and operations. . In various embodiments, the term "module" broadly refers to a collection of software instructions or code that is configured to cause processing circuitry 1110 to perform one or more functional tasks. Modules and computer-readable instructions may be described as processing circuits or other hardware components that, when executing the modules or computer-readable instructions, perform various operations or tasks.

3A-3B illustrate that computer readable program instructions stored on non-transitory computer readable media 1120 are utilized via computing system 1100 to increase the efficiency of object identification, detection, and retrieval operations and methods. An example environment is shown. The image information obtained by the computing system 1100 and illustrated in FIG. 3A influences the system's decision-making procedures and command output to the robot 3300 present within the object environment.

3A-3B illustrate example environments in which the processes and methods described herein may be implemented. FIG. 3A shows an environment having a system 3000 (which may be an embodiment of the systems 1000/1500A/1500B/1500C of FIGS. 1A-1D) including at least a computing system 1100, a robot 3300, and a camera 1200. Camera 1200 may be an embodiment of camera 1200 representing a scene 5013 within camera field of view 3200 of camera 1200, or more specifically objects 3410A , 3410B , 3410C , 3410D , etc. may be configured to generate image information representative of an object (such as a box) within the camera field of view 3200 . In one example, each of objects 3410A - 3410D may be a container, such as a box or crate, while object 3400 may be, for example, a pallet on which the containers are disposed. obtain. Additionally, each of objects 3410A - 3410D may be a container containing an individual object 5012. Each object 5012 may be, for example, a rod, bar, gear, bolt, nut, screw, nail, rivet, spring, linkage, gear, or any other type of physical object, as well as an assembly of multiple objects. good. 3A depicts an embodiment that includes multiple containers of object 5012, whereas FIG. 3B depicts an embodiment that includes a single container of object 5012.

In one embodiment, system 3000 of FIG. 3A may include one or more light sources. The light source may be, for example, a light emitting diode (LED), a halogen lamp, or any other light source, such as visible light, infrared light, or any other form of light directed toward the surface of the object 3410A - 3410D . may be configured to emit. In some embodiments, computing system 1100 may be configured to communicate with a light source to control when the light source is activated. In other embodiments, the light source may operate independently of computing system 1100.

In one embodiment, system 3000 includes a plurality of cameras 1200 or 2D cameras configured to generate 2D image information 2600 and 3D cameras configured to generate 3D image information 2700. A camera 1200 may be included. The camera 1200 or cameras 1200 may be mounted on or attached to the robot 3300, may be stationary within the environment, and/or may be mounted on a robot arm, gantry, or for camera movement. The robot 3300 may be fixed to a dedicated robotic system separate from the robot 3300 used for object manipulation, such as other automated systems configured to operate the object. 3A shows an example with a still camera 1200 and a handheld camera 1200, while FIG. 3B shows an example with only a still camera 1200. 2D image information 2600 (eg, a color or grayscale image) may describe the appearance of one or more objects, such as objects 3410 A/ 3410 B/ 3410 C/ 3410 D or object 5012, in camera field of view 3200. For example, the 2D image information 2600 may include visual details located on the respective external surfaces (e.g., top surfaces) of the objects 3410 A/ 3410 B/ 3410 C/ 3410 D and/or the The contour may be captured or otherwise represented. In one embodiment, the 3D image information 2700 may describe the structure of one or more of the objects 3410 A/ 3410 B/ 3410 C/ 3410 D/ 3400 and 5012, and the structure for the object It may also be referred to as a structure or physical structure of an object. For example, 3D image information 2700 may include a depth map, and more generally, may describe depth values for each of various locations in camera field of view 3200, relative to camera 1200 or relative to some other reference point. It may also include depth information. The locations corresponding to each depth value may be located on various surfaces of the camera field of view 3200 (physical (also called a place). In some instances, the 3D image information 2700 is on the outer surface of one or more of the objects 3410 A/ 3410 B/ 3410 C/ 3410 D/ 3400 and 5012, or some other object within the camera field of view 3200. It may include a cloud of points, which may include multiple 3D coordinates, describing various locations. The point cloud is shown in Figure 2F.

In the example of FIGS. 3A and 3B, a robot 3300 (which may be an embodiment of robot 1300) has one end attached to a robot base 3310 and attached to an end effector device 3330, such as a robot gripper, or or another end formed thereby. The robot base 3310 may be used to mount a robot arm 3320, and more specifically an end effector device 3330, for interacting with one or more objects in the environment of the robot 3300. can be used. The interaction (also referred to as robot interaction) may include, for example, grasping or otherwise picking up at least one of objects 3410A - 3410D and 5012. For example, the robot interaction may be part of an object picking operation to identify, detect, and remove object 5012 from a container. End effector device 3330 may have a suction cup or other component for grasping or grasping object 5012. The end effector device 3330 is configured to grip or grip an object through contact with a single side or surface of the object, e.g., through the top surface, using a suction cup or other gripping component. can be done.

Robot 3300 may further include additional sensors configured to obtain information used to perform tasks, such as to manipulate structural members and/or to transport robotic units. The sensors are configured to detect or measure one or more physical characteristics of the robot 3300 and/or the surrounding environment (e.g., the state, condition, and/or location of one or more structural members/joints thereof). may include a device. Some examples of sensors may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, and the like.

FIG. 4 provides a flow diagram illustrating the overall flow of methods and operations for object detection, identification, and retrieval according to embodiments herein. Object detection, identification, and retrieval method 4000 may include any combination of sub-methods and operational features described herein. The method 4000 includes an object registration process 5000, an object recognition template generation method 6000, a feature generation method 8000, an image information capture process 3001, a hypothesis generation process 10000, a hypothesis refinement method 11000, a hypothesis verification method 13000, and obstacle detection and motion planning. , and robot control processing 15000 including motion execution. In embodiments, the object registration process 5000, object recognition template generation method 6000, and feature generation method 8000 may be performed in a pre-processing or offline environment outside the context of robot motion. These processes and methods may therefore be performed in advance to facilitate later actions by the robot. Image information capture processing 3001, hypothesis generation processing 10000, hypothesis refinement method 11000, hypothesis verification method 13000, and robot control processing 15000 are each performed in the context of robot operation for detecting, identifying, and removing objects from containers. You can.

FIG. 5 illustrates object registration data related to object types that may be generated, acquired, received, or otherwise obtained during an object registration process 5000. As mentioned above, the methods and systems described herein obtain and use object registration data 5001, e.g., known, previously stored information associated with object 5011, to create an image in a physical scene. The method is configured to generate object recognition templates for use in identifying and recognizing similar objects. Object registration data 5001 may include any type of computer readable information that identifies, associates with, and/or describes object model 4200. The object registration data 5001 of the object model 4200 may represent an object 5011, and the object model 4200 provides measurements and dimensions of the represented object 5011 and provides secondary data that may or may not be interactive. or in three-dimensional format. Object registration data 5001 may include, for example, CAD (ie, computer-aided design) data or other modeling data describing object model 4200 and stored in any suitable format. The registration data may be a solid CAD model, a wireframe CAD model, or a surface CAD model. In one embodiment, the registration data may be in any type of three-dimensional file format, such as FBX, OBJ, USD, STL, STEP, COLLADE, etc. Object model 4200 represents one or more physical objects. Object model 4200 is a modeled (ie, computer-stored) version of one or more corresponding objects 5011 that physically exist in the world. As shown in FIG. 5, object 5011 is a physical object existing in the physical world, and object model 4200 is a digital representation of object 5011 described by object registration data 5001. The objects 5011 shown include, for example, rods, bars, gears, bolts, nuts, screws, nails, rivets, springs, chains, gears, or any other type of physical object, as well as assemblies of multiple objects. It can be any object. In an embodiment, the object 5011 is an object accessible from a container (e.g., a bottle, box, bucket, etc.), for example, having a mass in the range of a few grams to a few kilograms, and a size in the range of, for example, 5 mm to 500 mm. It can be pointed out. Object model 4200 may be specific to an exact version of real-world object 5011, such as, for example, a screw with a particular length, number of threads, thread size, head size, etc. For example, and for exemplary purposes, this description refers to a thread-shaped object as object 5011. It is presented for convenience only and is not intended to limit the scope of the description in any way.

In some embodiments, the present disclosure relates to generating a set of object recognition templates to identify objects 5012 within scene 5013. Object registration data 5001 may be based on physical object 5011 to facilitate recognition of other physical objects 5012 that are similar to physical object 5011 (and may be copies or versions thereof). may be used. Identifying an object 5012 in a scene includes identifying the object model 4200 to which the object 5012 corresponds (e.g., identifying what the object 5012 is), identifying the pose of the object 5012 (e.g., identifying what the object 5012 is) position, angle, and orientation).

FIG. 6 shows a flow diagram of an example object recognition template generation method 6000 for generating an object recognition template set. In one embodiment, the object recognition template generation method 6000 includes, for example, the computing system 1100 (or 1100A/1100B/1100C) of FIGS. 2A-2D, or the computing system 1100 of FIGS. 3A- 3B , or more specifically, This may be performed by at least one processing circuit 1110 of system 1100. In some scenarios, computing system 1100 may perform object recognition template generation method 6000 by executing instructions stored on a non-transitory computer-readable medium (eg, 1120). For example, the instructions may cause computing system 1100 to perform one or more of the modules illustrated in FIG. 2D that may perform object recognition template generation method 6000. For example, in embodiments, the steps of object recognition template generation method 6000 may be performed by object registration module 1130, and template generation module 1132 may operate to collaboratively generate object recognition templates. .

The steps of object recognition template generation method 6000 may be utilized to accomplish object recognition template generation, which may later be used in conjunction with a particular sequential robot trajectory to perform a particular task. As a general overview, object recognition template generation method 6000 provides a set of object recognition templates for computing system 1100 for use in identifying objects in a scene for processing related to object picking. It may be processed so as to be generated. The object recognition template generation method 6000 is described below with further reference to FIGS. 7A and 7B.

At least one processing circuit 1110 may perform certain steps of an object recognition template generation method 6000 to generate an object recognition template set 4301 that may include a plurality of object recognition templates 4300. The object recognition template generation method 6000 may begin with or include operation 6001, which includes obtaining object registration data 5001 for an object model 4200 representing an object 5011.

In the process 6001, the object recognition template generation method 6000 may include obtaining object registration data 5001 representing the object 5011, and the object registration data 5001 may include an object model 4200 representing the object 5011. At least one processing circuit 1110 may determine multiple viewpoints 4120 of the object model in three-dimensional space 4100. At least one processing circuit 1110 may further estimate multiple appearances 4140 of object model 4200 at each of multiple viewpoints 4120. The robot system may further generate a plurality of object recognition templates 4300 (e.g., 4300A/4300B/4300C/4300D) according to the plurality of appearances, each of the plurality of object recognition templates 4300 corresponding to a respective one of the plurality of appearances 4140. corresponds to one of the At least one processing circuit 1110 may then communicate the plurality of object recognition templates 4300 as an object recognition template set 4301 to a robotic system or storage system for later use. Each of the plurality of object recognition templates 4300 may represent a pose that the object model 4200 may have with respect to the optical axis 4130 of the virtual camera 4110. Each object recognition template 4300 represents the field of view of the object 5011 corresponding to the object model 4200 from the line of sight of the camera 1200, which has a line of sight corresponding to the line of sight of the virtual camera 4110 during generation of the object recognition template 4300.

At least one processing circuit 1110 receives object registration data 5001 from within its own hardware storage component (i.e., HDD, SSD, USB, CD, RAID, etc.) or software storage component (i.e., cloud, VSP, etc.). can be obtained. In one embodiment, at least one processing circuit 1110 may obtain registration data from an external processor (i.e., an external laptop, desktop, mobile phone, or any other separate device with its own processing system).

The object recognition template generation method 6000 may further include a process 6003 that may include determining multiple viewpoints 4120 of the object model 4200 in the three-dimensional space 4100. This may be called a spatial subsampling procedure. A three-dimensional space 4100 surrounding the object model 4200 may be surrounded by a surface 4101. Three-dimensional space 4100 and surface 4101 are virtual entities surrounding object model 4200, which is also a virtual entity. Each of the plurality of viewpoints 4120 determined in the process 6003 corresponds to the position of the virtual camera 4110 on the surface 4101 surrounding the three-dimensional space 4100 and the rotational angle of the virtual camera 4110 around the optical axis 4130 of the virtual camera 4110. or may represent them. Thus, each location on surface 4101 may correspond to multiple viewpoints 4120.

The virtual camera 4110 used in the spatial subsampling procedure may capture the appearance of the object from a viewpoint 4120 from which the virtual camera 4110 is located. For example, as shown in FIG. 7A, a virtual camera 4110 located at a respective viewpoint 4120A may capture an exterior view 4140 of an object model 4200. Appearance 4140 includes information that describes the appearance of object model 4200 to virtual camera 4110 based on the viewing angle and rotation of virtual camera 4110 about the optical axis of virtual camera 4110. Object model 4200 may be fixed within this three-dimensional space 4100. In one embodiment, the three-dimensional space may be substantially spherical. Object model 4200 may further be fixed at or near the center of a substantially spherical three-dimensional space. In another embodiment, the three-dimensional space may be any other suitable three-dimensional shape, such as an ellipsoid or a parallelepiped. Object model 4200 may be fixed at either a central or non-central point in three-dimensional space. Each of the individual object recognition templates 4300A/4300B/4300C/4300D, etc. generated (e.g., via process 6007 as discussed further below) is a representation of the object model 4200 from one viewpoint 4120 of the plurality of viewpoints 4120. May correspond to one captured appearance 4140. Each object recognition template 4300 may include an appearance 4140 of the object model 4200 from a viewpoint 4120 that captures the object's pose, ie, the object's orientation and visible surface. In one embodiment, each of the plurality of viewpoints 4120 may further correspond to a rotation angle of the virtual camera 4110 in the three-dimensional space 4100, ie, from 1 to 360 degrees with respect to its optical axis 4130.

Process 6003 may include a spatial subsampling procedure performed to select viewpoints 4120 whose corresponding object recognition templates 4300 are included in object recognition template set 4301. The efficiency of object recognition template generation method 6000 may be increased or maximized by reducing or otherwise optimizing the space (eg, number of viewpoints 4120 and appearances 4140) in which object recognition template 4300 is generated. In embodiments, superfluous viewpoints 4120 may be eliminated after initially capturing object appearances 4140 at those viewpoints 4120. For example, superfluous viewpoints 4120 may be eliminated if they are determined to contain information that is substantially similar to other viewpoints 4120 (eg, due to symmetry). In embodiments, superfluous viewpoints 4120 may be eliminated prior to acquisition of object features 4140 based on predetermined decisions regarding pose, spacing, etc., as discussed below. In embodiments, the number of selected viewpoints 4120 and the distance of the spacing between adjacent viewpoints 4120 are determined based on the required object recognition, e.g., the complexity and/or symmetry of the object model 4200 in question. It may depend on the number of templates 4300.

Multiple viewpoints 4120 may be selected or determined according to several different methods. For example, at least one processing circuit 1110 may determine viewpoint 4120 according to the intersection of longitude circle 4170 and latitude circle 4180. Viewpoint 4120 may be located at the intersection of longitude circle 4170 and latitude circle 4180 across surface 4101 of three-dimensional space 4100. In such a selection scheme, high-density viewpoints 4120 may be clustered at or near the poles of surface 4101, and low-density viewpoints may be clustered from the poles around intersecting longitude and latitude circles . They may be formed further apart (eg, closer to the equator of surface 4101). Such a non-uniform distribution of sample locations may cause multiple object recognition templates 4300 to overrepresent one range or set of ranges of relative pose/orientation between virtual camera 4110 and object model 4200 and overrepresent another range. or may cause the set of ranges to be underrepresented. Such a choice may be advantageous in some scenarios with some object models 4200 and less advantageous in other scenarios.

In further embodiments, the plurality of viewpoints 4120 may be selected according to an even distribution across the surface 4101 surrounding the three-dimensional space 4100. Even distribution may refer to viewpoints 4120 that are distributed at equal distances from each other across surface 4101. An even distribution may provide more consistent template generation than an uneven distribution and may be preferred for objects lacking symmetry.

In some embodiments, multiple viewpoints 4120 may be selected to reduce the total number of viewpoints 4120 and/or to weight or bias the viewpoint distribution in favor of particular viewpoints.

In one embodiment, multiple viewpoints 4120 may be determined based on a range of predictable poses that are expected to be observed for multiple objects 5011 in a physical situation. For example, in a container holding several tapered bottles, the orientation of the bottles may be expected to be such that the wider or proximal end faces downward. Therefore, the viewpoint distribution may be biased or weighted to have more viewpoints 4120 in the upper half of the surface 4101.

In another embodiment, multiple viewpoints 4120 may be determined based on symmetry (or lack thereof) of object model 4200. The symmetry of object model 4200 may be determined based on whether the appearance 4140 of object model 4200 changes by degrees about an axis of object model 4200 after rotation of object model 4200. For example, an object model 4200 that looks substantially the same after a 180 degree rotation has two-way symmetry. Object model 4200, which looks substantially the same after a 120 degree rotation, has three-way symmetry. Object model 4200, which looks substantially the same after a 90 degree rotation, has four-way symmetry. Object model 4200, which looks substantially the same after a 60 degree rotation, has hexagonal symmetry. Other symmetries may be possible for different objects. Substantially the same appearance may be determined according to a similarity threshold.

The object recognition template generation method 6000 may further include an operation 6005 that includes estimating or capturing a plurality of appearances 4140 of the object model 4200 at each of the plurality of viewpoints 4120. Estimation of multiple appearances 4140 may be performed at each viewpoint 4120 of multiple viewpoints 4120. Each appearance 4140 includes a pose or orientation of the object model 4200 as seen from a respective viewpoint 4120. Each of the object recognition templates 4300 corresponds to each viewpoint 4120 of the plurality of viewpoints 4120 and includes information representing the appearance 4140 of the object model 4200 from each viewpoint 4120. For example, the object recognition templates 4300 may be configured to represent each appearance of the object model 4200 from a respective viewpoint 4120 corresponding to a virtual camera 4110 placed directly above the object model (i.e., along the Y-axis of the three-dimensional plane). 4140 or can be represented. In another example, the object recognition template 4300 includes a view of the object model from a respective viewpoint 4120 corresponding to a virtual camera 4110 positioned directly to the left of the object model 4200 (i.e., along the X-axis of the three-dimensional plane). It may correspond to each appearance 4140. In one embodiment, each of the object recognition templates 4300 of the object recognition template set 4301 includes a virtual camera 4110 positioned at a number of different positions and orientations around the object model 4200 (i.e., multiple positions in a three-dimensional plane). may correspond to or represent a respective appearance 4140 of a plurality of appearances 4140 of the object model 4200 from a respective viewpoint 4120 of a plurality of viewpoints 4120 corresponding to the object model 4200 . Accordingly, estimating multiple appearances 4140 may include determining or estimating how object model 4200 looks when viewed from a particular viewpoint and in a particular orientation. For example, the viewpoint may be a direct downward look, an upward look, a leftward look, a rightward look of the object model 4200, or the principal axes X, Y, and Z of the surface 4101 surrounding the three-dimensional space 4100 as well as the principal axes X, Y, and Z on the surface 4101. may include any angle/position between. As discussed above, each viewpoint 4120 may also include a rotation angle of the virtual camera 4110 with respect to the camera's optical axis 4130 from 1 to 360 degrees. Thus, each camera position may correspond to a set of viewpoints 4120, and each viewpoint 4120 of the set of viewpoints may further correspond to a different rotation angle of the virtual camera 4110. For example, two separate viewpoints 4120 of the set of viewpoints 4120 may be estimated or captured from the same angle/location between the principal axes X, Y, and Z of the surface 4101, but the rotation angle of the first viewpoint 4120 is It is rotated by 45° with respect to the rotation angle of the second viewpoint 4120.

The object recognition template generation method 6000 may further include a process 6007 in which a plurality of object recognition templates 4300 are generated based on the plurality of appearances 4140. Each of the plurality of object recognition templates 4300 corresponds to a respective one of the plurality of appearances 4140. Thus, the generated object recognition template 4300 may include information representing the object model 4200 at a particular pose and at a particular angle and/or rotation of the virtual camera 4110 relative to the object model 4200. Thus, each of the plurality of object recognition templates 4300 may be different from the other of the plurality of object recognition templates 4300 (although in some scenarios, two different object recognition templates 4300 may contain substantially the same information due to symmetries of the object model 4200 that are not accounted for).

Each object recognition template 4300 may include a 2D appearance 4302 and a 3D appearance 4303 generated according to a respective captured or estimated appearance 4140. 2D appearance 4302 may include a rendered two-dimensional image, which may be rendered according to ray tracing and discontinuity detection techniques, for example. 3D appearance 4303 includes, for example, a rendered 3D point cloud similar to 3D image information 2700 described in connection with FIG. 2F.

In some embodiments, 2D appearance 4302 and/or 3D appearance 4303 may be generated via ray tracing techniques. The ray-tracing process may simulate various light rays from the perspective of the virtual camera 4110 hitting the surface of the object model 4200. This further determines the angle at which the ray hits the surface of the object model 4200, the distance the ray travels to the surface of the object model 4200, and/or the effects of diffuse reflection (rays deflected at multiple angles are thus implemented) or specular surfaces. The effect of reflection (the deflected ray is so performed at a single angle) can be determined. The angle of the deflected ray reflected from the surface of object model 4200 may indicate a change in the angle of the object's surface normal. Such changes in the angle of the object's surface normal can occur at the edges of the object.

The total number of multiple object recognition templates 4300 generated for the object model 4200 may range from approximately 100 templates to 3200 templates, and the larger the number of templates, the more the multiple object recognition templates 4300 may be correlated to the complexity of the object model 4200 being generated. Although the numbers cited are common to some applications and some object types, more or fewer templates may be used without departing from the scope of the invention. For example, an object model 4200 that exhibits a substantially symmetrical appearance (eg, a threaded nut) will generate multiple redundant templates (ie, matching templates) or templates that are substantially the same. Thus, such a simple object model 4200 can generate as few as 100 templates, or any number of templates in the lower half of the range of 100 to 3200 templates. Conversely, an object model 4200 that lacks symmetry may require more object recognition templates 4300 to adequately represent the object model 4200 at a larger feasible angle.

The object recognition template generation method 6000 may further include an operation 6009 that includes communicating the plurality of object recognition templates 4300 to the robot control system as an object recognition template set 4301. Object recognition template set 4301 may be communicated to a robot control system, such as computing system 1100, any other type of robot control system, and/or any other system that may employ object recognition template 4300. In embodiments, communicating the object recognition template set 4301 for later access by a robot control system, or other system that can employ the object recognition templates, via any suitable networking protocol and/or storage device. may include direct transmission to memory or other storage device for any amount of time. Each of the plurality of object recognition templates 4300 in the object recognition template set 4301 represents a pose that the object model 4200 may have with respect to the optical axis 4130 of the virtual camera 4110 when located at a particular viewpoint 4120. As mentioned above, a pose can include any position angle and rotation angle.

As described above, the object recognition template generation method 6000 of the present invention includes generating the object recognition template set 4301 from the object registration data 5001. Object recognition template set 4301 may be used to identify one or more objects 5011 in a scene that grab, pick up, or otherwise interact with one or more objects 5011 during physical processing. Object registration data 5001 of object model 4200 representing object 5011 is acquired. A plurality of viewpoints 4120 of the object model 4200 in the three-dimensional space 4100 are determined. The appearance 4140 of the object model at each of the plurality of viewpoints 4120 is estimated or captured. The plurality of object recognition templates 4300 are generated according to the plurality of appearances 4140, and each of the plurality of object recognition templates 4300 corresponds to a respective one of the plurality of appearances 4140. The plurality of object recognition templates 4300 are transmitted to the robot control system as an object recognition template set 4301. Each of the plurality of object recognition templates 4300 represents a posture that the object model 4200 may have with respect to the optical axis 4130 of the virtual camera 4110. Accordingly, each of the plurality of object recognition templates 4300 represents the object in the physical scene relative to the optical axis of the camera (such as camera 1200) that generates image information (e.g., image information 2600/2700) of the object 5011 in the physical scene. It can correspond to 5011 potential poses.

In further embodiments, additional or alternative methods may be used to generate object recognition template set 4301 from object registration data 5001, where object recognition template 4300 is additional or alternative to 2D appearance 4302 and 3D appearance 4303. The 2D appearance 4302 and the 3D appearance 4303 may include different information. Specifically, object recognition template 4300 may include two-dimensional (2D) measurement information 4304 and three-dimensional (3D) measurement information 4305.

2D measurement information 4304 may refer to a gradient feature map. The gradient feature map may include gradient information 9100 captured or extracted from the digital representation of the object at one or more gradient extraction locations 5100 on the surface of the digital object, as described below. 3D measurement information 4305 may refer to a surface normal feature map. The surface normal feature map may include surface normal vectors 9101 captured or extracted from the digital representation of the object at one or more surface normal locations 5101 on the surface of the digital object, as described below. The generation and/or extraction of 2D measurement information 4304 and 3D measurement information 4305 is described in more detail below with respect to FIGS. 8-9C.

FIG. 8 shows a flow diagram of an example feature generation method 8000. In embodiments, feature generation method 8000 may be used to generate a set of object recognition templates and/or a plurality of object recognition templates. In further embodiments, the feature generation method 8000 may be used to extract features from object image information in a hypothesis generation, refinement, and verification method, as discussed in more detail below. In one embodiment, the feature generation method 8000 is performed by, for example, the computing system 1100 (or 1100A/1100B/1100C) of FIGS. 2A-2D, or the computing system 1100 of FIGS. 3A-3B, or more specifically, the computing system 1100 may be performed by at least one processing circuit 1110 of 1100. In some scenarios, computing system 1100 may perform feature generation method 8000 by executing instructions stored on a non-transitory computer-readable medium (eg, 1120). For example, the instructions may cause computing system 1100 to execute one or more of the modules shown in FIG. 2D that may perform feature generation method 8000. For example, in embodiments, the steps of feature generation method 8000 may be performed by object registration module 1130, object recognition module 1121, feature extraction module 1134, and template generation module 1132 operating together.

In embodiments, the steps of the feature generation method 8000 may be utilized to accomplish object recognition template generation, e.g., through feature generation and/or extraction methods, which may include specific may later be used in conjunction with continuous robot trajectories. In embodiments, the steps of feature generation method 8000 may be applied to extract or generate features from object image information for use in hypothesis generation, refinement, and validation. As a general overview, the feature generation method 8000 can be applied to a computing system (e.g., computing system 1100 or a similar computing system) for use in identifying objects in a scene for processing related to object elevation. An object recognition template, a feature map, and/or a set of extracted/generated features may be generated by the computing system 1100. Feature generation method 8000 is described below with further reference to FIGS. 7A and 7B and FIGS. 9A-9C.

Feature generation method 8000 generates 2D measurement information 4304 and 3D measurement information 4305 that may be used to generate object recognition template 4300 and/or to characterize object 5012 (see, e.g., FIG. 3B) within physical scene 5013. may include. At least one processing circuit 1110 may obtain object information 9121. As shown in FIG. 9A, the object information 9121 includes an object 9200 represented digitally, for example, object registration data 5001 of the object model 4200, appearance 4140 of the object model 4200, object recognition template 4300, and/or scene information 9131. may be included. Scene information 9131 may include captured 2D or 3D image information of a physical scene 5013 including a plurality of objects 5012, similar to 2D image information 2600 and/or 3D image information 2700, for example. Scene information 9131 may also include image information 12001, discussed below with respect to hypothesis generation, verification, and refinement methods and processes. At least one processing circuit 1110 may further extract or generate 2D measurement information 4304 and/or 3D measurement information 4305 from object information 9121. In embodiments, at least one processing circuit 1110 may further generate object recognition template 4300 according to 2D measurement information 4304 and 3D measurement information 4305. In embodiments, 2D measurement information 4304 and 3D measurement information 4305 may be used or utilized for alternative purposes, such as hypothesis generation, validation, and refinement. At least one processing circuit 1110 may perform certain steps of the feature generation method 8000 for generating the object recognition template set 4301 and/or for use in hypothesis refinement and validation.

In process 8001, feature generation method 8000 may include obtaining object information 9121. Object information 9121 may include a digitally represented object 9200. Object information 9121 and digitally represented object 9200 may represent object 5015 that physically exists in the world. Objects 5015 include, for example, objects 5011 (e.g., physical objects represented by object model 4200) and/or objects 5012 (e.g., physical objects represented by captured image information of physical scene 5013). obtain. In embodiments, object information 9121 may include one or more of object recognition template 4300, object registration data 5001, object appearance 4140, and/or scene information 9131. At least one processing circuit 1110 receives object information 9121 from within a hardware storage component (i.e., HDD, SSD, USB, CD, RAID, etc.) or a software storage component (i.e., cloud, VSP, etc.) of computing system 1100. can be obtained. At least one processing circuit 1110 may obtain object information 9121 as part of internal processing, for example, as object recognition template 4300. At least one processing circuit 1110 may obtain object information 9121 from a camera 1200 associated with computing system 1100. At least one processing circuit 1110 receives object information about the object from an external processor (i.e., an external laptop, desktop, mobile phone, or any other separate device with its own processing system) or an external storage device. 9121 can be obtained.

In operation 8003, the feature generation method 8000 may further include selecting feature locations that include gradient extraction locations 5100 (shown in FIG. 9B) and surface normal locations 5101 (shown in FIG. 9C). Gradient extraction location 5100 is a location selected for extraction or generation of 2D measurement information 4304. Surface normal position 5101 is the position selected for extraction or generation of 3D measurement information 4305. Each of the gradient extraction position 5100 and the surface normal position 5101 is a position on the surface 9122 of the digitally represented object 9200.

In embodiments, gradient extraction location 5100 and surface normal location 5101 may correspond to each other. In embodiments, some gradient extraction locations 5100 may correspond to some surface normal locations 5101, while other gradient extraction locations 5100 do not correspond to surface normal locations 5101. In further embodiments, gradient extraction locations 5100 and surface normal locations 5101 may be selected such that they do not overlap each other. Thus, gradient extraction location 5100 and surface normal location 5101 may have any amount of overlap, including complete overlap and no overlap.

In embodiments, the location of the gradient extraction location 5100 and the surface normal location 5101 on the surface 9122 of the digitally represented object 9200 is used to store the extracted or generated 2D measurement information 4304 and 3D measurement information 4305. A limited set may be selected to limit the amount of memory required. This memory preservation practice ensures that, regardless of the size (in bytes) of the object information 9121 of the digitally represented object 9200, a fixed number of total features (as described below, information 9100 and/or surface normal vector 9101, etc.). The number of features captured for 2D measurement information 4304 may be the same as the number of features captured for 3D measurement information 4305, or may be different.

In embodiments, a limited number of gradient extraction locations 5100 and surface normal locations 5101 may be located to produce efficient results. For example, the gradient extraction location 5100 may be located along an identified edge of the digitally represented object 9200, as shown in FIG. 9B, while the surface normal location 5101 9200 may be located away from the edge. In embodiments, edges of the digitally represented object 9200 may be identified according to, for example, ray tracing, pixel intensity discontinuities, or other analysis techniques. This is because, as explained below, gradient information 9100 may be more significant in hypothesis generation, validation, and refinement when captured close to an object edge, whereas surface normal vector 9101 is captured away from an edge. It may prove efficient as it may be more significant when captured. The number of combinations of selected gradient extraction positions 5100 and surface normal positions 5101 may range from 100 to 1000, 50 to 5000, and/or 10 to 1000, but may also be greater or less. may be suitable as well. In certain embodiments, the number of gradient extraction locations 5100 and surface normal locations 5101 may be 256 each or 256 in total.

In operation 8005, the feature generation method 8000 may further include extracting 2D measurement information 4304 from object information 9121. 2D measurement information 4304 may be added to object 5015 to conserve memory or other resources and/or to improve the speed at which object recognition template set 4301 is generated or hypothesis testing and refinement is performed. may represent a smaller set of information (eg, compared to 2D appearance 4302). As mentioned above, object recognition template 4300 from object recognition template set 4301 may include 2D measurement information 4304 (and/or 3D measurement information 4305) describing object 5015.

2D measurement information 4304 may include two-dimensional features extracted or generated from object information 9121. In one embodiment, extracting or generating 2D measurement information 4304 may include extracting slope information 9100 from object information 9121. Accordingly, 2D measurement information 4304 may include a slope feature map that includes slope information 9100 as described herein. Gradient information 9100 indicates the direction or orientation of edge 5110 of digitally represented object 9200. Gradient information 9100 may be extracted at multiple gradient extraction locations 5100 of a digitally represented object 9200. Gradient extraction location 5100 may represent any or all internal and external edges identified within digitally represented object 9200.

Extracting the gradient information 9100 involves analyzing the pixel intensity of the two-dimensional image information of the object information 9121, and determining the direction in which the pixel intensity of the two-dimensional image information at each gradient extraction position changes in a process called gradient extraction. For example, it may include measuring (represented by arrow 9150). Changes in pixel intensity may represent the contour and orientation of the surfaces and edges of the digitally represented object 9200, thus providing information that may be useful in comparing two digitally represented objects 9200. Locations near each other along edge 5110 are likely to have similar slope information 9100, e.g., pixel intensities near such adjacent locations change in a similar manner with increasing distance from edge 5110. . In some examples, a portion of the digitally represented object 9200 that presents a higher than average pixel intensity may exhibit an edge 5110 or other discernible feature. As mentioned above, in some examples, the gradient extraction location 5100 may be located along the edge 5110 of the digitally represented object 9200.

In one embodiment, extracted gradient information 9100 may be used to improve a template matching process, hypothesis generation, or hypothesis testing process because object recognition template 4300 from object recognition template set 4301 is It may be determined whether there is a match to object 5012 in the scene. For example, if the 2D feature 4302 has a particular portion that overlaps or intersects the digitally represented object 9200 from the scene, the at least one processing circuit 1110 may determine whether the matching portion also matches or has a similar gradient (e.g. Whether a portion of the 2D measurement information 4304 matches) can be determined. If the slopes are different or do not match, at least one processing circuit 1110 may determine that the different particular portion is a result of the mismatch or is a coincidence. The mismatch may be the result of the 2D appearance 4302 overlapping portions of the scene by a small amount.

For example, referring now to FIG. 9D, the 2D appearance 4302A of the object recognition template 4300A is represented by a rectangle, and the 2D measurement information 4304A (eg, slope information 9100) of the object recognition template 4300A is represented by an arrow. 2D image information 2600B of an object 5012 in the scene (physical object not shown) is represented by an L-shaped solid. Object 5012 is further represented by 2D measurement information 4304B represented by an arrow. A portion of the 2D appearance 4302A may be compared and superimposed with a patterned portion 9309 of the 2D image information 2600B (physical object not shown) representing an object 5012 in the scene. However, the slopes represented by 2D measurement information 4304A and 4304B do not match, and therefore object recognition template 4300A is determined to be non-conforming for object 5012 in scene 5013 by at least one processing circuit 1110. Good too.

In operation 8007, the feature generation method 8000 may further include extracting or generating 3D measurement information 4305 from object information 9121. Referring now to FIG. 9C, process 8007 may include determining a surface normal vector 9101 at surface normal position 5101. 3D measurement information 4305 may include a surface normal feature map that includes extracted or generated surface normal vectors 9101.

The extracted 3D measurement information 4305 includes surface normal vector information, for example, a normal vector taken at a surface normal position 5101 on the surface 9122 of the digitally represented object 9200 (a vector perpendicular to the surface). ) may include measurements that describe the surface normal vector 9101. In one embodiment, extracting or generating 3D measurement information 4305 includes extracting or generating surface normal vector 9101 and/or surface normal vector information from object information 9121. Surface normal vector 9101 describes a plurality of vectors normal to surface 9122 of digitally represented object 9200. Surface normal vectors 9101 may be extracted or generated at multiple surface normal positions 5101 of digitally represented object 9200. Extracting the surface normal vector 9101 may include identifying a plurality of surface normal vectors 9101 of the digitally represented object 9200 at each of the surface normal vector positions 5101.

In operation 8009, the feature generation method 8000 may include generating an object recognition template set 4301 or a plurality of object recognition templates 4300. At least one processing circuit 1110 may generate one or more object recognition templates 4300 that include 2D measurement information 4304 and 3D measurement information 4305 described above. One or more object recognition templates 4300 may form an object recognition template set 4301. As described above, object recognition template 4300 may include one or more of 2D measurement information 4304, 3D measurement information 4305, 2D appearance 4302, and 3D appearance 4303. Accordingly, in some embodiments, the feature generation method 8000 may augment or further develop the previously established object recognition template 4300 and object recognition template set 4301. The extracted or generated 3D measurement information 4305 and 2D measurement information 4304 may be used to identify objects 5012 in the scene during a real-time or near real-time lifting process, as discussed below. The feature generation method 8000 uses the object recognition template described above when generating an object recognition template set 4301 in order to later perform matching (hypothesis refinement and verification) processing on a scene (or an object in the scene). It may be operated or processed in parallel with or subsequent to the generation method 6000. Feature generation method 8000 may serve as a final step towards creating an object recognition template set 4301 for use in subsequent hypothesis processing (eg, method 11000 and method 13000, described in further detail below).

10A and 10B illustrate aspects of a template matching and hypothesis generation method 10000 consistent with embodiments herein. The hypothesis generation techniques discussed herein may be generally consistent with lineMod techniques.

In process 10001, the template matching and hypothesis generation method 10000 may include obtaining image information. In one embodiment, obtaining image information 12001 may include capturing images of a scene 5013 and one or more objects 5012 within the scene. In these examples, image information 12001 may represent an object 5012 located within a box, bin, case, crate, pallet, or other container. Image information 12001 may be acquired by camera 1200, as discussed herein.

At least one processing circuit 1110 generates, receives, and generates image information 12001, such as by using image information 12001 to distinguish individual objects within the field of view of camera 1200 and perform object recognition based on image information 12001. and/or may be configured to process. In one embodiment, image information 12001 may include two-dimensional image information (eg, similar to 2D image information 2600) that describes the visual appearance of an environment or scene 5013 in the field of view of camera 1200. In one embodiment, image information 12001 includes three-dimensional image information (e.g., 3D image information 2700 similar to). This example three-dimensional image information may be used to estimate how object 5012 is spatially positioned within three-dimensional space (eg, scene 5013). Obtaining image information 12001 may include generating or obtaining image information 12001 representing a scene 5013, and optionally generating or obtaining object image information 12002 representing an individual object 5012 or multiple objects 5012 within scene 5013. May include acquisition. Image information 12001 may be generated by camera 1200 when object 5012 is (or was) in the field of view of camera 1200 and may include, for example, two-dimensional image information and/or three-dimensional image information.

In one embodiment, image information 12001 may include a two-dimensional grayscale or color image and may describe the appearance of scene 5013 (and/or objects 5012 within the scene) from the perspective of camera 1200. In one embodiment, image information 12001 may correspond to a single color channel (eg, a red, green, or blue channel) of a color image. If camera 1200 is positioned above object 5012, the two-dimensional image information may represent the appearance of the respective upper surface of object 5012. Further, image information 12001 may include, for example, various object positions 6220 on one or more surfaces (e.g., a top surface, or other outer surface) of object 5012 or along one or more edges of object 5012. It may include three-dimensional image information, which may include a depth map or point cloud, indicating respective depth values. The two-dimensional image information and three-dimensional image information of the object image information 12002 may be referred to as 2D image information 12600 and 3D image information 12700, respectively. In some embodiments, object positions 6220 representing physical edges of objects 5012 may be used to identify object image information 12002 that is limited to representing individual objects 5012.

Object image information 12002 may include image information related to a particular physical object 5012 within scene 5013. Object image information 12002 may include 2D image information 12600 representing object 5012, similar to image information 2600. Object image information 12002 may include 3D image information 12700 representing object 5012, similar to image information 2700. The object image information 12002 may include an object position 6220, where the object position 6220 is, for example, a gradient extraction position representing the position at which the respective gradient information 8102 and surface normal vector 8103 are obtained via the feature generation method 8000. 8100 and a surface normal position 8101 may be further included. As described above, the gradient extraction position 8100, the surface normal position 8101, the gradient information 8102, and the surface normal vector 8103 are , may be similar to gradient extraction location 5100, surface normal location 5101, gradient information 9100, and surface normal vector 9101, except that they are obtained from image information obtained from a physical object.

The template matching and hypothesis generation process discussed below may be performed by comparing the object recognition template to image information 12001 and/or object image information 12002. In embodiments, object image information 12002 may be generated from image information 12001 based on, for example, image segmentation or other techniques as well as feature generation method 8000, as described above.

In process 10003, the template matching and hypothesis generation method 10000 may include matching a template to object image information. The types of objects 5012 present in scene 5013 (whether a single type or multiple types) may be known. Accordingly, an object recognition template set 4301 corresponding to a known object type may be obtained, for example, via any method described herein. The information of each object recognition template 4300 of the object recognition template set 4301 representing information about how the object 5012 should appear in various poses is compared to the object image information 12002 representing the object 5012. It may be determined whether 4300 is a matching candidate. Good matching candidates may then be selected for generation of detection hypotheses.

Any relevant information in object recognition template 4300 may be compared with corresponding information in object image information 12002. For example, gradient information 8102 and gradient extraction position 8100 of object image information 12002 may be compared with gradient information 9100 and gradient extraction position 5100 of object recognition template 4300. Surface normal vector 8103 and surface normal position 8101 of object image information 12002 may be compared with surface normal vector 9101 and surface normal position 5101 of object recognition template 4300. 2D information 12600 and 3D information 12700 may be compared to 2D appearance 4302 and 3D appearance 4303, respectively.

The above information from object recognition template 4300 and from object image information 12002 can be understood as a map in that the information can be attributed to a series of two-dimensional locations. The template map (representing any of the object recognition template 4300 information) can be slid laterally relative to the object map (representing any of the object image information 12002) until a match that exceeds a threshold is found. . Template matching may involve comparing respective gradient information, respective 2D image information, respective 3D information, and/or respective surface normal vector information.

Thresholds may be used and tolerances may be allowed to account for potential changes in pose between object recognition template 4300 and object image information 12002. It can be understood and explained that the spatial subsampling procedure described above is not capable of capturing all possible poses in the object recognition template 4300, and therefore some deformation is allowed. Such tolerance techniques may include, for example, spreading, whereby gradient information 9100 is spread between adjacent gradient extraction locations 5100 of object recognition template 4300 to increase the likelihood of a match. Another tolerance technique may include finding a match based on a threshold level of match, such as when slope information or surface normal vectors are close to each other but do not match perfectly. Template matching may generate a template matching score to indicate the quality of the match.

In operation 10005, the template matching and hypothesis generation method 10000 may include clustering and grouping matching templates to reduce the total number of matches. The template matching process can find a plurality of object recognition templates 4300 that match the object represented by the object image information 12002. In some embodiments, the template matching process may be limited by time or computational resources in terms of how many matches can be identified. In these situations, process 10005 may avoid concentrating matches on a single portion or set of portions within scene 5013. Accordingly, matched templates with good quality matches (eg, above a threshold) may be clustered, grouped, and filtered to maintain good scene coverage. Object recognition templates 4300 that are identified as corresponding to the same object image information 12002 may be clustered or grouped. Within each cluster or group, the best match may be selected and the rest may be removed. Therefore, the remaining matches may represent object 5012 throughout scene 5013 rather than clustering in a single region. In one example, if object 5012 in scene 5013 is near the top of the container and is very easily recognizable, it may generate more matches than a partially hidden object. By selecting only the best match for each object image information 12002, more objects can be identified.

In operation 10007, the template matching and hypothesis generation method 10000 may include generating a set of one or more detection hypotheses. The object recognition templates 4300 remaining after clustering and grouping may be selected as detection hypotheses. These object recognition templates 4300 may be stored with pose information 6301 indicating information about where each object recognition template 4300 in scene 5013 should be located to match the corresponding object image information. Posture information 6301 may further include information associating each object recognition template 4300 with corresponding object image information 12002. Detection hypotheses 6300 may be combined in groups and/or sets. For example, detection hypothesis set 8309 may include multiple detection hypotheses 8300 associated with object image information 12002 representing a single object 5012, but a group of detection hypotheses 8300 may represent multiple different objects 5012 in scene 5013. A plurality of detection hypotheses 8300 related to the represented object image information 12002 may be included.

FIG. 11 shows a flow diagram of an example hypothesis refinement method 11000 for refining detection hypotheses. In one embodiment, the hypothesis refinement method 11000 may be applied to, for example, the computing system 1100 of FIGS. 2A-2D (i.e., 1100A/1100B/1100C), or the computing system 1100 of FIGS. 1100 may be performed by at least one processing circuit 1110 of 1100. In some scenarios, computing system 1100 may perform hypothesis refinement method 11000 by executing instructions stored on a non-transitory computer-readable medium (eg, 1120). For example, the instructions may cause computing system 1100 to execute one or more of the modules shown in FIG. 2D that may perform method 11000. For example, in embodiments, the steps of method 11000 may be performed by hypothesis generation module 1128 and hypothesis refinement module 1136 operating in conjunction.

Hypothesis refinement method 11000 may also be used to refine one or more detection hypotheses 6300 (e.g., as described above) generated to identify objects 5012 physically located within scene 5013. good. Hypothesis refinement method 11000 may operate on image information 12001 obtained from scene 5013. Image information 12001 may be similar to 2D image information 2600 and 3D image information 2700. Within image information 12001 there may be one or more object image information 12002 representing objects 5012 within scene 5013. Identifying the object 5012 includes identifying a type of object, or identifying dimensions of the object from corresponding object image information 12002, and/or matching object image information 12002 to object recognition template 4300. obtain. Accordingly, detection hypothesis 6300 may be a hypothesis regarding which of one or more object recognition templates 4300 may match object image information 12002 of image information 12001 representing scene 5013. Object image information 12002 may include 2D image information 12600 representing object 5012. 2D image information 12600 may be similar to image information 2600 and/or may include rendered 2D image information generated according to rendering techniques such as ray tracing and discontinuity detection. Object image information 12002 may include 3D image information 12700 representing object 5012, similar to image information 2700. Detection hypothesis 6300 may be generated according to a template matching procedure, as described above. For example, in one embodiment, detection hypothesis 6300 may be generated via the lineMod algorithm and/or procedure described above. The hypothesis refinement method 11000 may perform processing to refine the match between the object recognition template 4300 and the object image information 12002 even in a scenario where the object recognition template 4300 does not exactly match the object image information 12002.

In the hypothesis refinement method 11000, at least one processing circuit 1110 includes a robot 3300 having a robot arm 3320 and an end effector device 3330 connected to the arm, and a field of view with one or more objects 5012 within the field of view. or when the camera 1200 is configured to execute instructions stored on a non-transitory computer-readable medium. In embodiments, at least one processing circuit 1110 may not communicate directly with robot 3300, but may send information to and receive information from robot 3300 via a network and/or storage device. In embodiments, at least one processing circuit 1110 may communicate directly with robot 3300. At least one processing circuit 1110 may obtain image information 12001 of one or more objects 5012 within a scene 5013. At least one processing circuit 1110 may also obtain a detection hypothesis 6300. Detection hypothesis 6300 may include information associating object recognition template 4300 (e.g., a corresponding object recognition template 4300B selected from a plurality of object recognition templates 4300) and object image information 12002, and is represented by object image information 12002. It may also include posture information 6301 of the object 5012. Pose information 6301 for object 5012 may refer to the position and orientation of object 5012. In embodiments, a detection hypothesis 6300 may include a corresponding object recognition template 4300B or may include a reference to a corresponding object recognition template 4300B. At least one processing circuit 1110 may process to identify a mismatch between the corresponding object recognition template 4300B and the corresponding object image information 12002. At least one processing circuit 1110 may process to identify a corresponding set of template positions 6210 of object recognition template 4300B that correspond to a set of object positions 6220 of object image information 12002. At least one processing circuit 1110 may further process the set of template positions 6210 to adjust it to converge on the set of object positions 6220. At least one processing circuit 1110 is configured to generate an adjusted detection hypothesis 6300' or a plurality of iterative adjusted detection hypotheses 6300' that include an object recognition template adjusted according to a set of adjusted template positions 6210. May be processed.

At least one processing circuit 1110 may perform certain steps of hypothesis refinement method 11000 to refine detection hypothesis 6300. In process 11001, hypothesis refinement method 11000 may include obtaining image information 12001 of one or more objects 5012 in scene 5013. In one embodiment, obtaining image information 12001 may include capturing an image of scene 5013. In these examples, image information 12001 may represent an object 5012 located within a box, bin, case, crate, pallet, or other container. Image information 12001 may be acquired by camera 1200, as discussed herein.

At least one processing circuit 1110 generates image information 12001, such as using image information 12001 to distinguish individual objects within the field of view of camera 1200 and perform object recognition or object registration based on image information 12001; The information may be configured to receive and/or process. In one embodiment, image information 12001 may include two-dimensional image information (eg, similar to 2D image information 2600) that describes the visual appearance of an environment or scene 5013 in the field of view of camera 1200. In one embodiment, image information 12001 includes three-dimensional image information (e.g., 3D image information 2700 similar to). This example three-dimensional image information may be used to estimate how object 5012 is spatially positioned within three-dimensional space (eg, scene 5013). With respect to process 11001, obtaining image information 12001 may include generating or obtaining image information 12001 representing a scene 5013, optionally a single object representing an individual object 5012 or multiple objects 5012 within scene 5013. It may also include generating or acquiring more than one object image information 12002. Image information 12001 may be generated by camera 1200 when object 5012 is (or was) in the field of view of camera 1200 and may include, for example, two-dimensional image information and/or three-dimensional image information.

In one embodiment, image information 12001 may include a two-dimensional grayscale or color image and may describe the appearance of scene 5013 (and/or objects 5012 within the scene) from the perspective of camera 1200. In one embodiment, image information 12001 may correspond to a single color channel (eg, a red, green, or blue channel) of a color image. If camera 1200 is positioned above object 5012, the two-dimensional image information may represent the appearance of the respective upper surface of object 5012.

Object image information 12002 may include image information related to a particular physical object 5012 within scene 5013. Object image information 12002 may include 2D image information 12600 representing object 5012, similar to image information 2600. Object image information 12002 may include 3D image information 12700 representing object 5012, similar to image information 2700. The object image information 12002 may include an object position 6220, where the object position 6220 is, for example, a gradient extraction position representing the position at which the respective gradient information 8102 and surface normal vector 8103 are obtained via the feature generation method 8000. 8100 and a surface normal position 8101 may also be included. As described above, the gradient extraction position 8100, the surface normal position 8101, the gradient information 8102, and the surface normal vector 8103 are , may be similar to gradient extraction location 5100, surface normal location 5101, gradient information 9100, and surface normal vector 9101, except that they are obtained from image information obtained from a physical object.

In operation 11003, the hypothesis refinement method 11000 may further include obtaining a detection hypothesis 6300. Detection hypothesis 6300 may include multiple pieces of information. For example, the detection hypothesis 6300 includes the corresponding object recognition template 4300B and object pose information 6301 indicating the position and orientation of the corresponding object recognition template 4300B necessary to overlay the corresponding object image information 12002 in the image information 12001. may include. The corresponding object recognition template 4300B may include one or more of 2D appearance 4302B, 3D appearance 4303B, 2D measurement information 4304B, and 3D measurement information 4305B. As described above, 2D measurement information 4304B may include gradient information 9100B and gradient extraction position 5100B, whereas 3D measurement information 4305B may include surface normal vector 9101B and surface normal position 5101B. The corresponding object recognition template 4300B may further include template locations 6210, which may include gradient extraction locations 5100B and surface normal locations 5101B or a subset thereof.

In operation 11005, the hypothesis refinement method 11000 may further include identifying a mismatch between the corresponding object recognition template 4300B and the object image information 12002 that the template matched according to the detection hypothesis 6300. The two-dimensional information of the corresponding object recognition template 4300B (eg, 2D appearance 4302B) may be compared to the object image information 12002 to identify discrepancies. Mismatches may be identified or quantified according to areas of non-alignment or other mismatch between 2D appearance 4302B and object image information 12002.

Upon identification of a mismatch or inconsistency between the corresponding object recognition templates 4300B, the two-dimensional information of the corresponding object recognition templates 4300B (e.g., 2D appearance 4302B) is used for comparison and alignment with the object image information 12002. A two-dimensional space may be converted to a three-dimensional space. In some instances, a 3D transformation of 3D appearance 4303B or 2D appearance 4302B may be used for comparison with object image information 12002 to identify discrepancies. In some embodiments, a mismatch may be identified or quantified according to the mismatch between object position 6220 and template position 6210. Object position 6220 represents a point on a digital representation of object 5012 (eg, object image information 12002), while template position 6210 represents a point on template object 6290 (as discussed below).

The transformation from two-dimensional space to three-dimensional space may be based on calibration parameters or other parameters of the camera 1200 or other image sensor, which may be determined during a camera calibration process or may be predefined. As mentioned above, the corresponding object recognition template 4300B may be derived from the object registration data 5001 and have a coordinate system associated therewith. In the transformation to three-dimensional space, the coordinate system of the corresponding object recognition template 4300B may be mapped to the coordinate system of the scene 5013, as captured in the image information 12001. Therefore, calibration parameters or other parameters of the camera 1200 that captured the image information 12001 may be utilized for the conversion. The detection hypothesis 6300 information may define a digital representation of the object, referred to herein as a template object 6290. The three-dimensional transformation may be referred to as template object 6290 and may represent information of detection hypothesis 6300 in three-dimensional space in the coordinate system of image information 12001 for comparison with object image information 12002.

In operation 11007, the hypothesis refinement method 11000 may further include identifying a set of template positions in the corresponding object recognition template that correspond to the set of object positions on the corresponding object. Object position 6220 represents a point on the digital representation of object 5012 (eg, object image information 12002), while template position 6210 represents a point on template object 6290. Therefore, aligning object location 6220 with template location 6210 may help refine detection hypothesis 6300.

As mentioned above, template location 6210 may correspond to gradient extraction location 5100B and surface normal location 5101B or a subset thereof. In further embodiments, template location 6210 may include additional or different locations used for alignment of object image information 12002 with object location 6220.

Template positions 6210 (and corresponding object positions 6220) may be selected according to positions that have a high impact on hypothesis refinement (eg, alignment between object image information 12002 and template object 6290). In some examples, template position 6210 and object position 6220 may be selected as positions around the edges of respective template object 6290 and object image information 12002. Such locations may be more useful for performing hypothesis refinement because they are less susceptible to noise and may provide an outline of the object's shape.

In operation 11009, the hypothesis refinement method 11000 may further include adjusting the set of template positions 6210 to converge to the set of object positions 6220. At least one processing circuit 1110 may be further configured to adjust the set of template positions 6210. If a mismatch is identified, the at least one processing circuit 1110 may make an adjustment to improve the alignment value between the template position 6210 of the template object 6290 and the corresponding object position 6220 of the object image information 12002. .

The alignment procedure may be performed using the iterative closest point (ICP) technique, shown in FIGS. 12B and 12C. ICP techniques may include adjusting template positions 6210 to converge on a set of object positions 6220. A set of vectors 6215 between template positions 6210 and their corresponding object positions 6220 may be determined. Each vector 6215 may represent direction and magnitude. In one embodiment, the direction and magnitude of the vector may be used to adjust template position 6210 to converge on object position 6220. A vector 6215 extending from template position 6210 to object position 6220 has a direction and a magnitude. If the set of vectors 6215 is understood mathematically as a force having the direction and magnitude of the vector and operating on the template object 6290 at the template position 6210, then the template object 6290 is applied at each template position 6210. or may be adjusted or moved according to the direction and magnitude of the acting vector 6215. Therefore, vectors 6215 representing template positions 6210 and object positions 6220 that have larger magnitudes and are further apart (e.g., have larger deltas or offsets) are understood to apply more "force" to the template adjustment. obtain. For example, referring to FIG. 12B, template object 6290 may overlay object image information 12002. Vector 6215 extends between template position 6210 and object position 6220. When vectors 6215 are collectively applied as a “force” to template object 6290 based on their direction and magnitude, template object 6290 (shown in FIG. There is a tendency to achieve close alignment. After applying vectors 6215, a new set of vectors 6215 may be generated and applied iteratively. In another example, applying vector 6215 may cause a translation of template object 6290 to result in alignment with object image information 12002, as shown in FIG. 12C. In some embodiments, through iterative generation and application of vectors 6215, template object 6290 moves into better alignment with object image information 12002 until the remaining vectors 6215 cancel each other out and no further movement is produced. do. If no further movement can be generated, alignment quality may be evaluated. In some embodiments, iterative adjustments may be made until the alignment quality exceeds a threshold.

The quality of alignment (or level of misalignment) can be evaluated or determined in a number of different ways. For example, the quality of alignment may be evaluated or determined according to the level of misalignment defined by the direction and magnitude of vector 6215. The quality of the alignment may also be evaluated or determined according to distance measurements between the new, updated, or adjusted set of template positions 6210 and the set of object positions 6220. The quality of alignment may be evaluated or determined according to the convergence rate. Embodiments may use any combination of these alignment quality measurements.

The quality of alignment may be determined based on the level of misalignment defined by the direction and magnitude of each new or updated vector 6215. As mentioned above, vector 6215 can be interpreted mathematically as a force acting on template object 6290 according to its direction and magnitude. When an object is at rest and subjected to a force, it will experience stress. In embodiments, the level of misalignment may be calculated according to mathematically treating vector 6215 as a force that creates internal stress in template object 6290. Thus, for example, equal and opposite vectors 6215 do not cancel each other out (as if vectors 6215 were simply added together), but create "stress" on template object 6290. When the level of alignment quality is good (and the level of misalignment is low), vector 6215 has a relatively small magnitude, thereby corresponding to low internal stresses. If the alignment quality is poor (and the level of misalignment is high), vector 6215 will be larger, thereby corresponding to more significant internal stresses. This internal stress calculation can be considered as an indication of alignment quality.

In one embodiment, the quality of alignment may be determined based on distance measurements between the new or updated set of template locations 6210 and the set of object locations 6220. The distance measure may be a Euclidean distance measure or the length of a line segment between two points in Euclidean space. Euclidean distance (or Pythagorean distance) can be expressed via the following formula:

In the formula, it is as follows.

d=distance

p = first point with 3D coordinates p ₁ , p ₂ , p ₃

q = second point with 3D coordinates q ₁ , q ₂ , q ₃

The distance measurements generated via the above equation will output a distance value (typically greater than or equal to zero), and an output value close to zero will correspond to the point p, q (zero represents no distance or the same /represents an overlapping point). The distance measurements between each new or updated set of template locations 6210 and the set of object locations 6220 may be combined, for example, by taking an arithmetic or geometric mean. The combined distance values may then be compared to a predetermined threshold, with distance values equal to or below the predetermined threshold (i.e., zero and the predetermined threshold) indicating a good match ( That is, between the template object 6290 and the object image information 12002), a distance output value greater than a predetermined threshold value indicates a mismatch.

In one embodiment, the distance measurement is the cosine between the new set of template positions 6210 (i.e., template vector 6260) and the surface normal vector associated with the set of object positions 6220 (i.e., object vector 6270). It may be distance. Template vector 6260 may include some or all of the previously determined surface normal vectors 9101 associated with the corresponding object recognition template 4300B. Object vector 6270 may include surface normal vector 8101 associated with object image information 12002. The measured cosine distance may indicate an angle between a surface normal vector (e.g., template vector 6260 and object vector 6270), and the indicated angle may indicate an angle between a surface normal vector (e.g., template vector 6260 and object vector 6270). 6270). The cosine distance can be expressed by the following formula.

Cosine distance = 1 - cosine similarity,

Here, the cosine similarity is expressed by the following formula,

where x _i and y _i are the components of vectors X and Y.

Or, in the alternative:

The distance measurement produced by the above equation outputs a value indicating the distance between two surface normal vectors (ie, as a cosine distance). This output value may further indicate the angle between the template vector 6260 and the object vector 6270, or more specifically, the angle between the planar portion of the template object 6290 and the planar portion of the object image information 12002. good. A planar portion refers to a surface whose surface normal vectors extend and are parallel. An output that provides a small angle may indicate a good match (ie, good convergence or alignment) between a planar portion of template object 6290 and a planar portion of object image information 12002. The cosine distances between each corresponding pair of template vector 6260 and object vector 6270 may be combined to generate a distance measurement, for example, by taking an arithmetic or geometric mean.

In another embodiment, the distance measurement may be a planar distance measurement, measured from one of the template positions 6210 to a plane containing the corresponding point from the object position 6220, or vice versa. It may be. The planar distances between each corresponding pair of template vector 6260 and object vector 6270 may be combined to generate a distance measurement, for example, by taking an arithmetic or geometric mean.

The quality of alignment between template object 6290 and object image information 12002 may be further determined according to a profile that shows decreasing distance over successive iterations of the ICP technique. As discussed above, ICP techniques may be used to align template object 6290 and object image information 12002 by converging template position 6210 with object position 6220 . During successive iterations, distance measurements (eg, cosine distance, Euclidean distance, planar distance, etc.) between template object 6290 and object image information 12002 may be obtained. A profile may show changes in these distances over successive iterations.

For example, a profile that exhibits a consistent decrease in distance over successive iterations of the ICP may indicate high alignment quality in terms of convergence between template object 6290 and object image information 12002. Conversely, if the profile indicates that there are successive repetitions of the ICP, indicating that the distance is increasing or otherwise decreasing very rapidly between successive repetitions. , the profile may indicate that template object 6290 and object image information 12002 do not exhibit high quality convergence and the final alignment between template object 6290 and object image information 12002 may be of low quality.

In operation 11011, the hypothesis refinement method 11000 may include generating an adjusted detection hypothesis. Adjusted detection hypothesis 6300' may be generated according to adjustments made to template position 6210, as described above. Adjustments may represent adjusted versions of various pieces of information stored in detection hypothesis 6300. For example, the adjusted detection hypothesis 6300' may include information associating the object image information 12002 with the adjusted corresponding object recognition template 4300B', and may include adjusted pose information 6301'. The adjusted corresponding object recognition template 4300B' is one of adjusted 2D appearance 4302B', adjusted 3D appearance 4303B', adjusted 2D measurement information 4304B', and adjusted 3D measurement information 4305B'. may contain more than one. Adjusted 2D measurement information 4304B' may include adjusted slope information 9100B' and adjusted gradient extraction positions 5100B', while adjusted 3D measurement information 4305B' includes adjusted surface normals. It may include vector 9101B' and adjusted surface normal position 5101B'. The adjusted object recognition template 4300B' may further include adjusted template positions 6210', which may include adjusted gradient extraction positions 5100B' and adjusted surface normal positions 5101B' or a subset thereof. There is no requirement that all "adjusted" versions of the information included in the adjusted detection hypothesis 6300' be different from the corresponding information in the detection hypothesis 6300. For example, in embodiments, a position may be adjusted while information associated with that position (slope and surface normal) may remain the same. In embodiments, the adjusted information may be captured by storing information about the adjustment in conjunction with storing the original detection hypothesis 6300.

The present disclosure further relates to testing detection hypotheses. FIG. 13 shows a flow diagram of an example detection hypothesis verification method 13000 for testing a detection hypothesis. The following description of detection hypothesis verification refers to FIG. 14. The detection hypothesis verification method 13000 operates on one or more previously obtained detection hypotheses to verify a particular detection hypothesis as corresponding to a specifically detected physical object in a scene. Good too. As discussed above, through template matching and detection hypothesis generation and refinement, multiple detection hypotheses may be proposed as relating to or explaining a particular physical object in a scene. Detection hypothesis verification method 13000 may receive object image information for a particular physical object as well as a set of detection hypotheses associated therewith, and may test the plurality of detection hypotheses to determine an optimal or best-fitting detection hypothesis. The set of detection hypotheses may be initial detection hypotheses (such as detection hypothesis 6300) and/or may be adjusted detection hypotheses (such as adjusted detection hypothesis 6300'), or a combination thereof. There may be. At least one processing circuit 1110 may perform certain steps of detection hypothesis verification method 13000 to verify detection hypothesis 8300, as described below.

In one embodiment, detection hypothesis validation method 13000 is performed, for example, by computing system 1100 (or 1100A/1100B/1100C) of FIGS. 2A-2D or computing system 1100 of FIGS. 3A- 3B , or more specifically, by The processing may be performed by at least one processing circuit 1110 of computing system 1100. In some scenarios, computing system 1100 may perform detection hypothesis verification method 13000 by executing instructions stored on a non-transitory computer-readable medium (eg, 1120). For example, the instructions may cause computing system 1100 to execute one or more of the modules illustrated in FIG. 2D that may perform detection hypothesis verification method 13000. For example, in embodiments, the steps of method 13000 may be performed by hypothesis generation module 1128, hypothesis refinement module 1136, and hypothesis verification module 1138 operating together.

Detection hypothesis testing method 13000 may also be used to test one or more detection hypotheses 8300 of detection hypothesis set 8309 generated to identify one or more objects 5012 physically located within scene 5013. good. The detection hypothesis verification method 13000 may process image information 12001 acquired from the scene 5013. Image information 12001 may be similar to 2D image information 2600 and 3D image information 2700. Within image information 12001 there may be one or more object image information 12002 representing objects 5012 within scene 5013.

In the following discussion, detection hypothesis verification method 13000 is discussed according to the use of detection hypothesis set 8309 for a single object 5012 to be identified. As discussed below, detection hypothesis validation method 13000 operates to identify the best detection hypothesis 8300 that corresponds to a single object 5012. In other embodiments, detection hypothesis set 8309 may include detection hypotheses 8300 for multiple objects 5012 in scene 5013. Each individual object 5012 may have a corresponding group of detection hypotheses 8300 from the detection hypothesis set 8309, which may be verified according to the methods described with respect to the corresponding individual object 5012. In this way, the detection hypothesis verification method 13000 can be used to test and identify the best detection hypothesis 8300 for a single object 5012 or to identify multiple best detection hypotheses, each corresponding to a different individual object 5012. 8300. By testing multiple detection hypotheses 8300, complex lifting processes involving sequential lifting of multiple objects 5012 can be planned and executed.

In the detection hypothesis verification method 13000, at least one processing circuit 1110 includes a robot 3300 having a robot arm 3320 and an end effector device 3330 connected to the arm, and a field of view in which one or more objects 5012 are present. or when the camera 1200 is configured to execute instructions stored on a non-transitory computer-readable medium. In embodiments, at least one processing circuit 1110 may not communicate directly with robot 3300, but may send information to and receive information from robot 3300 via a network and/or storage device. In embodiments, at least one processing circuit 1110 may communicate directly with robot 3300. At least one processing circuit 1110 may obtain image information 12001 of one or more objects 5012 within a scene 5013. At least one processing circuit 1110 may also obtain one or more detection hypotheses 8300 and/or detection hypothesis sets 8309.

Each detection hypothesis 8300 may include information that associates an object recognition template 4300 (e.g., a corresponding object recognition template 4300C selected from a plurality of object recognition templates 4300) with object image information 12002, and is represented by object image information 12002. It may also include posture information 6301 of the object 5012. Pose information 6301 for object 5012 may refer to the position and orientation of object 5012. In embodiments, the detection hypothesis 8300 may include a corresponding object recognition template 4300C or may include a reference to a corresponding object recognition template 4300C.

In process 13001, the detection hypothesis verification method 13000 includes obtaining image information of one or more objects in a scene. Process 13001 may be similar to process 11001 discussed above. Obtaining image information 12001 may include capturing an image of scene 5013. In these examples, image information 12001 may represent an object 5012 located within a box, bin, case, crate, pallet, or other container. Image information 12001 may be acquired by camera 1200, as discussed herein. At least one processing circuit 1110 generates image information 12001, such as using image information 12001 to distinguish individual objects within the field of view of camera 1200 and perform object recognition or object registration based on image information 12001; The information may be configured to receive and/or process. In one embodiment, image information 12001 may include two-dimensional image information (eg, similar to 2D image information 2600) that describes the visual appearance of an environment or scene 5013 in the field of view of camera 1200. In one embodiment, image information 12001 includes three-dimensional image information (e.g., 3D image information 2700 similar to). This example three-dimensional image information may be used to estimate how object 5012 is spatially positioned within three-dimensional space (eg, scene 5013). Obtaining image information 12001 may include generating or obtaining image information 12001 representing a scene 5013 and, optionally, one or more object image information 12002 representing an individual object 5012 or multiple objects 5012 of scene 5013. It may also include the generation or acquisition of. Object image information 12002 may include 2D image information 12600 representing object 5012. 2D image information 12600 may be similar to image information 2600 and/or may include rendered 2D image information generated according to rendering techniques such as ray tracing and discontinuity detection. Object image information 12002 may include 3D image information 12700 representing object 5012, similar to image information 2700. Image information 12001 may be generated by camera 1200 when object 5012 is (or was) in the field of view of camera 1200 and may include, for example, two-dimensional image information and/or three-dimensional image information.

Object image information 12002 may include image information related to a particular physical object 5012 within scene 5013. Object image information 12002 may include 2D image information 12600 representing object 5012, similar to image information 2600. Object image information 12002 may include 3D image information 12700 representing object 5012, similar to image information 2700. The object image information 12002 may include an object position 6220, where the object position 6220 is, for example, a gradient extraction position representing the position at which the respective gradient information 8102 and surface normal vector 8103 are obtained via the feature generation method 8000. 8100 and a surface normal position 8101 may be further included. As described above, the gradient extraction position 8100, the surface normal position 8101, the gradient information 8102, and the surface normal vector 8103 are , may be similar to gradient extraction location 5100, surface normal location 5101, gradient information 9100, and surface normal vector 9101, except that they are obtained from image information obtained from a physical object.

Image information 12001 may be the same image information 12001 obtained during implementation of hypothesis refinement method 11000. Accordingly, computing system 1100 may obtain image information 12001 for performance of hypothesis refinement method 11000, store image information 12001, and access image information 12001 for performance of detection hypothesis verification method 13000. In embodiments, the image information 12001 may be newly acquired, particularly for implementing the detection hypothesis verification method 13000.

As discussed above, image information 12001 may include two-dimensional grayscale and/or color images that describe the appearance of scene 5013 (and/or objects 5012 within the scene) from the perspective of camera 1200. Good too. In one embodiment, image information 12001 may correspond to a single color channel (eg, a red, green, or blue channel) of a color image. If camera 1200 is positioned above object 5012, the two-dimensional image information may represent the appearance of the respective upper surface of object 5012. Further, image information 12001 may include, for example, various object positions 6220 on one or more surfaces (e.g., a top surface, or other outer surface) of object 5012 or along one or more edges of object 5012. It may include three-dimensional image information, which may include a depth map or point cloud, indicating respective depth values. The two-dimensional image information and three-dimensional image information of the object image information 12002 may be referred to as 2D image information 12600 and 3D image information 12700, respectively. In some embodiments, object positions 6220 representing physical edges of objects 5012 may be used to identify object image information 12002 that is limited to representing individual objects 5012.

In operation 13003, the detection hypothesis verification method 13000 may further include obtaining one or more detection hypotheses 8300 and/or detection hypothesis set 8309. For ease of explanation, the described attributes and qualities of a particular detection hypothesis 8300 may be understood to apply to each of the detection hypotheses 8300 of the detection hypothesis set 8309, unless otherwise specified. Detection hypothesis 8300 may be obtained as adjusted detection hypothesis 6300' following implementation of hypothesis refinement method 11000. Detection hypothesis 8300 may be obtained as initial detection hypothesis 8300 from the template matching process, as described above.

The detection hypothesis 8300 includes a corresponding object recognition template 4300C and object pose information 8301 indicating the position and orientation of the corresponding object recognition template 4300C necessary to overlay the corresponding object image information 12002 in the image information 12001. But that's fine. The corresponding object recognition template 4300C may include one or more of 2D appearance 4302C, 3D appearance 4303C, 2D measurement information 4304C, and 3D measurement information 4305C. As mentioned above, 2D measurement information 4304C may include slope information 9100C and gradient extraction position 5100C, while 3D measurement information 4305C may include surface normal vector 9101C and surface normal position 5101C. The corresponding object recognition template 4300C may further include template locations 8210, which may include gradient extraction locations 5100B and surface normal locations 5101B or a subset thereof. The detection hypothesis 8300 information may define a digital representation of the object, referred to herein as a template object 8290. Template object 8290 represents information of detection hypothesis 6300 in three-dimensional space in the coordinate system of image information 12001 for comparison with object image information 12002.

The detection hypothesis set 8309, and in particular the size of the set, may be selected or determined to balance proportion and completeness. Selecting more detection hypotheses 8300 may provide a higher chance of achieving a good match, but may also take longer to process. As discussed above with respect to hypothesis refinement method 11000, the quality of the alignment may be measured or determined during the steps associated with refinement. Exceeding a quality threshold may be a marker that causes hypothesis refinement method 11000 to be determined as complete. Similarly, exceeding a quality threshold may be considered as a marker that allows for inclusion of an adjusted detection hypothesis 6300' within the detection hypothesis set 8309. Failure to exceed the quality threshold may result in the adjusted detection hypothesis 6300' being rejected. Therefore, the size of the detection hypothesis set 8309 may be determined by how stringent the quality threshold is. In some embodiments, the size of detection hypothesis set 8309 may be limited and includes only the highest quality alignment adjustment detection hypotheses 6300'. In embodiments, both a quality threshold and a ranked order may be used. Embodiments use template matching and hypothesis refinement techniques to generate large detection hypothesis sets 8309 (e.g., more than 500, 1000, or 10,000 total detection hypotheses) with the understanding that many false positives will be generated. It can be beneficial to do so. Such embodiments may rely on the detection hypothesis validation method 13000 to filter false positives, as discussed below.

In operation 13005, the detection hypothesis verification method 13000 includes testing each detection hypothesis of the set of detection hypotheses. A plurality of detection hypotheses 8300 are obtained and compared to object image information 12002 of image information 12001 to determine which detection hypothesis 8300 is the best estimate or Identify compliance. Selecting the best detection hypothesis from the detection hypothesis set 8309 includes testing each of the detection hypotheses according to processes 13007-13011 described below. Hypothesis testing may include generating three-dimensional and two-dimensional testing scores and filtering the detected hypothesis set 8309 accordingly.

In operation 13007, the detection hypothesis verification method 13000 includes generating a plurality of three-dimensional verification scores. Each three-dimensional validation score may be based on a comparison of the three-dimensional information of the detection hypothesis 8300 and the corresponding three-dimensional information of the image information corresponding to the object from the scene (eg, object image information 12002). The plurality of three-dimensional validation scores may include at least one of an occlusion validator score, a point cloud validator score, a hole match validator score, and a normal vector validator score. The three-dimensional information of detection hypothesis 8300 may include 3D appearance 4303C and 3D measurement information 4305C including surface normal vector 9101C and surface normal position 5101C. The three-dimensional information of the object image information 12002 may include 3D image information 12700, surface normal position 8101, and surface normal vector 8103.

A validator score may be a score or number that represents how well a particular detection hypothesis corresponds to or aligns with object image information 12002, as discussed herein. The validator score may be a penalty score applied to the hypothesis confidence score, with lower values representing a better fit, as discussed herein. Alternatively, the validator score may be a bonus score where higher values represent a better fit. For ease of explanation, the validator scores discussed herein may be penalty scores, but it is understood that all of the same concepts and techniques can be applied using bonus scores.

The occlusion validator score and the point cloud validator score each compare the object position 6220 of the object image information 12002 with the surface of the template object 8290 represented by the detection hypothesis 8300, and calculate the difference between the object position 6220 and the surface of the template object 8290. may be obtained by identifying discrepancies between and obtaining an occlusion validator score and a point cloud validator score. The three-dimensional information of the detection hypothesis may indicate the position of the surface of the template object 8290. If the three-dimensional information of detection hypothesis 8300 actually represents object 5012 in scene 5013, object position 6220 associated with 3D image information 12700 should be on or near the surface. If not near the surface, the match determined by the template matching process may be a false positive. Comparing object position 6220 to the surface of template object 8290 may identify two types of mismatches: valid points and occlusion and invalid points. Discrepancies that place object position 6220 above or otherwise outside the surface of template object 8290 are referred to as occlusions and may be used to calculate an occlusion validator score. Discrepancies that place object positions 6220 below the surface of template object 8290 are referred to as invalid points and may be used to calculate point cloud validator scores. Object locations 6220 that are above or near the surface of template object 8290 (within a threshold distance, also referred to as the skin depth parameter) may be referred to as valid points. Some deviation between object position 6220 and the surface of template object 8290 is expected. Such deviations may be accounted for by skin depth parameters, the size of which determines the amount of allowable deviation.

The occlusion validator score is obtained by identifying discrepancies that place the object location 6220 above or outside the surface of the template object 8290. These mismatches are called occlusions. The occlusion validator score provides a weighted penalty to the hypothesis confidence score, with the weight depending on the distance of the object location 6220 from the surface of the template object 6290. The occlusion validator score may be calculated as a function of distance from the surface of template object 6290. The function may be, for example, a log-normal function, where the peak of the curve of the log-normal function represents a distance from the surface that coincides with a 3D point near the surface of the template object 8290, but It is unlikely that some of them are. In embodiments, a function with a peak may be selected at a distance just beyond the point at which a sensor or camera capturing image information 12001 loses accuracy. For example, object position 6220 that exceeds the surface of template object 6290 by a very large distance is not an actual point on object 5012 represented by object image information 12002, but rather a combination of a matching portion of scene 5013 and a matching portion of camera 1200. Due to the possibility that such an object position 6220 results from occlusion from another object 5012 in between or from noise in the image information 12001, the penalty applied to it may be low. Therefore, the occlusion validator score penalty for a particular object location 6220 may initially increase with distance, reducing the confidence of the detection hypothesis. After the distance increases beyond the peak, it becomes increasingly likely that a particular object position 6220 is not generated by the object 5012 represented by the object image information 12002 and the penalty decreases.

In embodiments, a shielding confidence score may be determined relative to a shielding validator score. The shielding confidence score represents the level of confidence that the shielding validator score provides good information about what decisions can be made. Object locations 6220 may represent points or locations where there is confidence that they belong to an object. However, object image information 12002 may include additional points that are not confidently identified as belonging to object 5012. The occlusion confidence score may be based on the ratio of the object position 6220 to the total number of visible points in the object image information 12002. Therefore, if object locations 6220 that are confident to belong to an object represent a lower percentage of the total visibility points, then the occlusion validator score based on object locations 6220 has less confidence in providing accurate information, and the associated occlusion Confidence scores decrease as well. In some embodiments, the final occlusion validator score may be represented by the initial occlusion validator score modified according to the occlusion confidence score.

Point cloud validator scores are obtained by identifying discrepancies that place object location 6220 within or below the surface of template object 8290. These discrepancies are called invalid points. The point cloud validator score becomes a penalty to the hypothesis confidence score. An object location 6220 identified as an invalid point, eg, below the surface of template object 8290, may be a strong indicator that detection hypothesis 8300 is inaccurate and may result in a correspondingly high penalty score. In embodiments, the point cloud validator score may be based on the number of invalid points or the ratio of invalid points to an invalid point cutoff value.

The point cloud validator score may have a point cloud confidence score determined in the same manner as described above with respect to the occlusion confidence score, e.g., according to the ratio of the object position 6220 to the total number of visible points in the object image information 12002. good. In embodiments, the final point cloud validator score may be represented by a point cloud validator score that is modified according to the point cloud confidence score.

In embodiments, the point cloud validator score and the occlusion validator score may be combined into a single surface validator score. The surface validator score may be determined as a combination of the point cloud validator score and the occlusion validator score, for example by adding, averaging, or performing another mathematical operation to combine the two. .

The normal vector validator score determines whether a valid point, identified according to object position 6220 on or near the surface of template object 6290, has a surface normal vector 8103 that matches the orientation of the surface of template object 6290. can be obtained by determining. Such determination is made by comparing the surface normal vector 8103 associated with the object position 6220 to the corresponding surface normal vector 9101C associated with the corresponding surface normal position 5101C of the corresponding object recognition template 4300C. You can. If the surface normal vector 8103 does not align or match the orientation with the corresponding surface normal vector 9101C, the normal vector validator score may be applied as a penalty to the detection hypothesis confidence score. In embodiments, the amount of mismatch or misalignment may affect the size of the penalty applied.

In embodiments, some tolerance is provided for situations in which, even if the detection hypothesis is accurate, surface normal vector 8103 is not expected to align or match orientation with corresponding surface normal vector 9101C. It's okay to be hit. For example, an object such as a gear with many teeth may have portions that exhibit abrupt changes in edge and surface normal vectors. Such an object structure can generate surface normal vectors 8103 and corresponding surface normal vectors 9101C even if there is only a slight misalignment between object image information 12002 and template object 8290 overlaid on the scene. can cause large deviations. To account for such a scenario, at least one processing circuit 1110 may check whether the corresponding object recognition template 4300C or image information 12001 has a region with high variation in surface normal vectors 9101C/8103. If the result is positive, the at least one processing circuit 1110 determines the difference in the high variation region between the corresponding surface normal vector 9101C of the corresponding object recognition template 4300C and the surface normal vector 8103 of the object image information 12002. A higher amount of tolerance can be applied by lowering the normal vector validation score for .

In embodiments, a surface normal validator score may have a surface normal confidence level associated with it. The surface normal confidence level may represent the confidence level of the information provided by the surface normal validator score. In one embodiment, the surface normal confidence level may be determined according to the quality of the extracted edges. In one embodiment, the surface normal validator score may be adjusted according to the surface normal confidence level.

The object position 6220 obtained from the object image information 12002 is compared to the structure of the template object 8290, as represented by the corresponding object recognition template 4300C, and mismatches between the object position 6220 and the structure are identified and the structure The hole match validator score is obtained by identifying invalid hole or empty coordinates (referred to as hole invalidity) according to the object position 6220, which corresponds to an empty volume of the structure in the template object or position where the hole is absent. Since the object position 6220 of the object image information 12002 represents the position on the surface of the physical structure of the object 5012, the object 5012 in the scene has a structure in a space that the corresponding object recognition template 4300C indicates is empty. isn't it. The presence of an object location 6220 in a portion that the corresponding object recognition template 4300C indicates as empty may be due to noise, but may also indicate an inaccurate detection hypothesis. Therefore, the hole match validator score may be determined as a penalty score for the detection hypothesis confidence level for invalidity of all identified holes.

In embodiments, a hole match validator score may have a hole match confidence level associated therewith. The hole match confidence level may represent the confidence level of the information provided by the hole match validator score. In one embodiment, the hole match confidence level may be determined according to the quality of the extracted edges. In one embodiment, the hole match validator score may be adjusted according to the hole match confidence level.

In embodiments, tolerances may be provided to account for noise or other situations that may cause invalidity of holes even with correct detection hypotheses. For example, if object image information 12002 includes an object position 6220 that corresponds to an empty space in template object 8290 (e.g., a hole or opening in the object), then that object position 6220 corresponds to another position that is coincidentally located in the space. can correspond to a part of an object. Such a scenario provides accurate detection for an object 5012 in a scene 5013 when, in estimation, the object position 6220 in empty space does not belong to the object 5012 represented by the corresponding object recognition template 4300C, but instead belongs to another object. This may be consistent with hypothesis 8300. In one embodiment, the hole match validator score determines when the coordinates of a hole, opening, or void in template object 8290 are relatively large in size, and irregularities that affect the measurement of that space (e.g., hole or Greater tolerances may be provided because the likelihood of an object intersecting the opening or protruding through the hole or opening and another object extending into that space is increased.

In embodiments, the point cloud validator score, occlusion validator score, hole match validator score, and surface normal validator score may be combined into a single 3D validator score. The 3D validator score may be, for example, a combination of point cloud validator score, occlusion validator score (or combined surface validator score), hole match validator score, and surface normal validator score. It may be determined by adding, averaging, or performing another mathematical operation to combine the two.

In operation 13009, the detection hypothesis validation method 13000 includes generating a plurality of two-dimensional validation scores, which validation scores may include at least one of a rendered match validator score and a template matching validator score. .

The rendered match validator score is obtained by comparing the rendered 2D image information 12600 of the image information 12001 to the corresponding 2D appearance 4302C. The rendered match validator score may be further processed to extract edge information from both the rendered 2D image information 12600 and the corresponding 2D appearance 4302C. The rendered match validator score may be based on determining whether edges extracted from the 2D image information 12600 align with edges extracted from the corresponding 2D appearance 4302C. The rendered match validator score may be based on the amount of overlap between regions defined by extracted edges, average distance between extracted edges, or any other suitable metric. The rendered match validator score may be used as a penalty score applied to the detection hypothesis confidence score. In some instances, using rendering (e.g., ray tracing) to generate and extract edge information may introduce noise and other artifacts such as glare from light reflecting off metal objects or shadows. can be compensated for. In some instances, process 13009 may also process to re-render information from corresponding object recognition template 4300C to extract edges from corresponding object recognition template 4300C.

In embodiments, a rendered match validator score may have a rendered match confidence level associated therewith. The rendered match confidence level may represent the confidence level in the information provided by the rendered match validator score. In one embodiment, the rendered match confidence level may be determined according to the quality of the extracted edges. In one embodiment, the rendered match validator score may be adjusted according to the rendered match confidence level.

The template matching validator score is obtained by comparing the edges extracted from the object image information 12002 and the object image derived from the corresponding object recognition template 4300C (e.g., template object 8290 or 2D appearance 4302C, etc.). Ru. Edge detection algorithms, such as a Canny edge detector, may be utilized to identify object edges directly from the object image information 12002 and template edges from the image information stored in the corresponding object recognition template 4300C. Template matching validator scores match object edges and template edges by sliding the template edge against the object edge and determining how much sliding (if any) results in a peak response or overlap. The offset between can be determined accordingly. The template matching validator score may be based on the amount of sliding, movement, offset, or adjustment required to achieve peak response or overlap. The greater the amount of movement or sliding required, the higher the template matching validator score and the greater the penalty applied. In other words, more moves required indicate a poorer match.

In embodiments, a template matching validator score may have a template matching confidence level associated therewith. The template matching confidence level may represent the level of confidence in the information provided by the template matching validator score. In one embodiment, the template matching confidence level may be determined according to the quality of the extracted edges. In embodiments, the template matching validator score may be adjusted according to the template matching confidence level.

The three-dimensional validator score and the two-dimensional validator score may be combined to determine an overall validation score that can be used in further processing to determine an overall level of confidence in the detection hypothesis. The total validation score may be based on a combination of each of the three-dimensional and two-dimensional validator scores and the confidence value associated with each validator score. For example, a validator score with a higher confidence value and/or a higher score weight may have a greater impact on the total validation score, whereas a validator score with a lower confidence value and/or a lower score weight may have a greater impact on the total validation score. The validator score may have a smaller impact on the total validation score.

In embodiments, the process 13005 further includes an additional verification step of determining whether the corresponding object recognition template 4300C is globally consistent with other structures or objects within the image information 12001 corresponding to the scene 5013. may be included. For example, such other structures and objects may include containers in which workpieces or other objects are located. For example, the process 13005 determines whether the template object 8290 fits completely within such a container (e.g., based on the position of the template object 8290 as determined by the pose information 6301), or whether the template object 8290 fits completely within such a container. It may further be determined whether it extends outwardly or protrudes. If template object 8290 or a portion thereof is outside the container, such a situation may be indicative of an incorrect detection hypothesis. In such a situation, the total validation score may be adjusted accordingly, with a weighted penalty according to how far outside the container the template object 8290 is. In embodiments, if template object 8290 or a portion thereof is outside the container by more than a threshold amount, the total validation score may be adjusted to indicate an incorrect detection hypothesis. Some tolerances are provided to account for situations in which an accurate detection hypothesis may still correspond to a template object 8290 that extends outside the container or extends beyond the plane that defines the interior surface of the container. It's okay to be hit. Such a situation may arise, for example, if the container is a mesh container or if the object is a metal object that is sufficiently hard to roughen or otherwise deform the inner surface of the container.

In operation 13011, the detection hypothesis validation method 13000 further includes filtering the detection hypothesis from the set of detection hypotheses by a plurality of three-dimensional validation scores and a plurality of two-dimensional validation scores.

In embodiments, multiple validator scores may be combined to generate a total validator score that may be used to determine a detection hypothesis confidence level. The total validator score and detection hypothesis confidence level may indicate how well the corresponding object recognition template 4300C matches the object image information 12002 obtained from the scene 5013. The detection hypothesis confidence level may be used to decide whether to exclude the detection hypothesis or use the detection hypothesis to plan a robot motion to lift the object 5012 in the scene 5013.

In embodiments, the filtering of detection hypotheses may be performed according to a sequential filtering technique, where each validator score retains a given detection hypothesis 8300 from the detection hypothesis set 8309 compared to a corresponding threshold. or filtering. Each successive validator score may be compared to a threshold, and if the validator score exceeds the threshold, the detection hypothesis 8300 may be excluded. In one example, filtering detection hypotheses 8300 from detection hypothesis set 8309 includes occlusion validator score, point cloud validator score, hole match validator score, normal vector validator score, rendered match validator score, and template It may include comparing the matching validator score to a corresponding threshold and removing validator scores that exceed the corresponding threshold and any detection hypothesis 8300. The above comparisons may be performed sequentially. The above order is for illustrative purposes only; any order may be used. If continuous filtering is utilized, the efficiency of the process may be increased by not calculating additional validator scores for excluded detection hypotheses.

In embodiments, comparing the validator score to a corresponding threshold may take into account the confidence level associated with the validator score. For example, the associated threshold may be adjusted according to the validator score confidence level, and/or the validator score may be adjusted according to the confidence level. Therefore, validator scores with low confidence indicating a bad match may be excluded as detection hypotheses, while validator scores with high confidence may have a greater impact.

Therefore, if one or more of the three-dimensional validation score or the two-dimensional validation score exceeds the corresponding threshold (considering the confidence level if necessary), the detection hypothesis 8300 is removed or filtered from the detection hypothesis set 8309. can be done. A detection hypothesis 8300 may remain within the detection hypothesis set 8309 if all of the 3D and 2D validation scores cannot exceed all of the corresponding thresholds (optionally taking into account confidence levels). .

The filtering process may continue until a single detection hypothesis 8300 remains for each particular object 5012 corresponding to object image information 12002. This is done by selecting the detection hypothesis with the highest detection hypothesis confidence level (and lowest total validation score) and/or until only a single detection hypothesis 8300 is successful for each object 5012. This can occur by repeating the filtering process with increasingly lowered filter thresholds. Single detection hypothesis 8300 may be unfiltered detection hypothesis 8300. In embodiments, a minimum confidence level may be set to detection hypothesis 8300. In such embodiments, if the best-fitting detection hypothesis 8300 for an object 5012 cannot exceed the confidence threshold, the system may not return a detection hypothesis 8300 for that object.

In operation 13013, the hypothesis testing method 13000 includes detecting one or more objects in the scene according to the unfiltered detection hypotheses remaining in the set of detection hypotheses after validation. After filtering, the best detection hypothesis 8300 corresponding to the object 5012 associated with the object image information 12002 is identified to detect the object 5012 in the scene. As mentioned above, hypothesis testing method 13000 may also be utilized to identify multiple different objects 5012 according to multiple different associated detection hypotheses 8300.

In some embodiments, hypothesis testing method 13000 may further include a duplicate detection process, whereby one or more detection hypotheses 8300 are compared to each other to determine whether their corresponding template objects 8290 have duplicates. judge whether Such overlaps may indicate that one or both of the detection hypotheses 8300 with overlaps are inaccurate. Detection hypotheses 8300 may be compared for overlap after filtering process 13011. Prior to the filtering process 13011, multiple detection hypotheses 8300 may remain for each object 5012, so overlap is expected. After the filtering process 13011, the remaining detection hypotheses 8300 represent the best fit for each object 5012 and no overlap is expected. In response to detecting a duplicate, the system may be configured to discard one or both of the duplicate detection hypotheses 8300, e.g., based on their confidence scores, or perform additional analysis or processing with respect to the duplicate detection hypothesis 8300. It may be configured to perform. The decision to discard, keep, or reanalyze a duplicate detection hypothesis 8300 may further consider the degree of overlap.

Following detection of one or more objects 5012 in scene 5013 , at least one processing circuit 1110 is operable to perform robot control processing 15000 for retrieval of one or more objects 5012 and to control robot 3300 . A command may be output that causes the move to retrieve one or more objects 5012. Robot control processing 15000 may include obstacle detection, motion planning, and motion execution.

Obstacle detection may include detecting and describing obstacles in the vicinity of the retrieved object 5012. As discussed herein, object 5012 may be in a container with other items and objects. Accordingly, other items and objects, as well as the container itself, may represent obstacles to robotic movement of robot 3300. Such obstacles may be captured in image information 12001 and/or object image information 12002, which may be processed to determine the location of the obstacle in the vicinity of the object.

Motion planning may include, for example, planning robot motions, such as plotting a trajectory that robot 3300 will take to retrieve object 5012. A trajectory may be plotted to account for and avoid identified obstacles. Motion execution may include sending commands related to the motion plan to the robot or robot control system to cause the robot 3300 to perform the planned motion.

The methods discussed herein, e.g., methods 6000, 8000, 10000, 11000, and 13000, create object recognition templates, utilize the object recognition templates to generate, refine, and refine detection hypotheses for objects in a scene. and may be jointly processed for verification. Accordingly, methods 6000, 8000, 10000, 11000, and 13000 may be utilized to facilitate robotic processes for detecting, identifying, and removing objects from within a container.

It will be apparent to those skilled in the relevant art that other suitable modifications and adaptations to the methods and uses described herein can be made without departing from the scope of any of the embodiments. Dew. The embodiments described above are illustrative examples and the disclosure should not be construed as limited to these particular embodiments. It is to be understood that the various embodiments disclosed herein may be combined in different combinations than those specifically presented in the description and accompanying figures. It should also be understood that, by way of example, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, and may be added, combined, or omitted entirely. (For example, not all described acts or events may be necessary to carry out a method or process). Additionally, certain features of the embodiments herein are described herein, for clarity, as being implemented by a single component, module, or unit. It should be understood that the features and functions may be performed by any combination of components, units, or modules. Accordingly, various changes and modifications may be made by those skilled in the art without departing from the spirit or scope of the invention as defined in the appended claims.

Further embodiments include the following embodiments.

Embodiment 1 is a computing system configured to generate a set of object recognition templates for identifying objects in a scene, the system comprising: obtaining object registration data including an object model representing the object; determining a plurality of viewpoints of the object model in three-dimensional space; estimating, at each of the plurality of viewpoints, a plurality of appearances of the object model; and according to the plurality of appearances, each one of the plurality of appearances. and transmitting the plurality of object recognition templates to a robot control system as an object recognition template set; Each of the plurality of object recognition templates is a computational system that represents a pose that the object may have with respect to an optical axis of a camera that generates image information of the object in the scene.

In the second embodiment, the three-dimensional space is surrounded by a surface, each of the plurality of viewpoints corresponds to a camera position on the surface, and each of the object recognition templates corresponds to one of the plurality of viewpoints, 3 is a calculation system according to embodiment 1, including an appearance of an object from one viewpoint.

Embodiment 3 is the calculation system according to Embodiment 1, in which each of the plurality of viewpoints further corresponds to a camera rotation angle.

Embodiment 4 is the calculation system according to Embodiment 2, in which the object model is fixed in three-dimensional space.

Embodiment 5 is the calculation system according to Embodiment 2, in which the three-dimensional space is substantially spherical and the object model is fixed at the center of the three-dimensional space.

Embodiment 6 is the computing system according to Embodiment 2, in which the multiple viewpoints are selected according to an even distribution over the surface.

Embodiment 7 is the computing system described in Embodiment 3, where each camera position corresponds to a set of viewpoints, and each viewpoint of the set of viewpoints corresponds to a different camera rotation angle.

Embodiment 8 is the computing system according to Embodiment 3, wherein a subset of the set of object recognition templates includes object recognition templates corresponding to viewpoints corresponding to different positions and different camera rotation angles.

Embodiment 9 is the calculation system according to Embodiment 2, further including determining a plurality of viewpoints based on predicted ranges of postures observed for a plurality of object recognition templates.

Embodiment 10 is the computing system of Embodiment 2, further comprising determining multiple viewpoints based on symmetry of the object.

Embodiment 11 is the computing system of embodiment 10, further comprising determining symmetry of the object according to at least one of determining that an object appearance of the object changes after rotation and identifying an axis of the object. It is.

Embodiment 12 is a method for generating an object recognition template set for identifying an object in a scene, which includes acquiring object registration data including an object model representing the object, and generating object recognition template set for identifying an object in a three-dimensional space. determining a plurality of viewpoints, estimating a plurality of appearances of the object model at each of the plurality of viewpoints, and a plurality of object recognitions according to the plurality of appearances, each corresponding to a respective one of the plurality of appearances. generating a template; and communicating the plurality of object recognition templates to a robot control system as an object recognition template set, each of the plurality of object recognition templates generating image information of an object in the scene. A method of representing the pose that an object can have with respect to the optical axis of a camera.

Embodiment 13 further comprises: the three-dimensional space is surrounded by a surface, the plurality of viewpoints correspond to camera positions on the surface, and each of the object recognition templates corresponds to one of the plurality of viewpoints. This is the method described in Embodiment 12.

Embodiment 14 is the method of Embodiment 13, further comprising associating each of the plurality of viewpoints with a camera rotation angle.

Embodiment 15 is the method of embodiment 13, further comprising fixing the object model in three-dimensional space.

Embodiment 16 is the method of embodiment 13, further comprising selecting the plurality of viewpoints according to an even distribution across the surface.

Embodiment 17 is the method described in Embodiment 13, further comprising determining a plurality of viewpoints based on predicted ranges of postures observed for a plurality of object recognition templates.

Embodiment 18 is the method of embodiment 13, further comprising determining multiple viewpoints based on symmetry of the object.

Embodiment 19 is the method of embodiment 18, further comprising determining symmetry of the object according to at least one of determining that an object appearance of the object changes after rotation and identifying an axis of the object. be.

Embodiment 20 is a non-transitory computer-readable medium operable by at least one processing circuit via a communication interface configured to communicate with a robotic system, the non-transitory computer readable medium operable by at least one processing circuit to perform object recognition for identifying objects in a scene. A non-transitory computer readable medium having executable instructions for performing a method for generating a template. The method includes receiving registration data of an object including an object model representing the object, performing an operation to generate a plurality of viewpoints of the object model in a three-dimensional space, and performing an operation for generating a plurality of viewpoints of the object model in a three-dimensional space. , performing an operation of estimating a plurality of appearances of an object model; performing an operation of generating a plurality of object recognition templates according to the plurality of appearances, each corresponding to a respective one of the plurality of appearances; outputting the object recognition templates to the robot system as a set of object recognition templates, each of the plurality of object recognition templates including an image of the object in the scene relative to an optical axis of the camera that generates image information of the object in the scene. Represents the attitude to obtain.

Embodiment 21 is a computing system configured to generate an object recognition template for identifying an object in a scene, the computing system comprising: obtaining object information including a digitally represented object; At least one configured to extract two-dimensional measurement information, extract three-dimensional measurement information from the object information, and generate an object recognition template according to the two-dimensional measurement information and the three-dimensional measurement information. A computing system that includes one processing circuit.

Embodiment 22 provides that the digitally represented object is an object model, and extracting the two-dimensional measurement information and three-dimensional measurements is performed to generate a feature map of the object model at a selected viewpoint. , the computing system described in Embodiment 21.

Embodiment 23 provides that the at least one processing circuit acquires image information of a scene, accesses an object recognition template, and compares the image information with the two-dimensional measurement information and the three-dimensional measurement information to digitally 22. The computing system of embodiment 21, further configured to identify an object corresponding to the represented object.

Embodiment 24 provides that extracting the two-dimensional measurement information includes extracting gradient information from the object information, the gradient information indicating a direction or orientation of a candidate edge of the digitally represented object; extracting the information includes extracting surface normal vector information from the object information, the surface normal vector information describing a plurality of vectors normal to a surface of the digitally represented object; This is the calculation system described in Embodiment 21.

Embodiment 25 is the calculation system according to Embodiment 21, wherein the object information includes object registration data, and the digitally represented object includes an object model.

Embodiment 26 is the calculation system according to Embodiment 21, wherein the object information includes at least one of two-dimensional image information and three-dimensional image information.

Embodiment 27 provides that gradient information is extracted at a plurality of gradient extraction positions of a digitally represented object, and that extracting the gradient information includes analyzing pixel intensities of two-dimensional image information of the object information to extract each gradient. 25. The computing system of embodiment 24, comprising measuring the direction in which pixel intensities of the two-dimensional image information change at the extraction location.

In embodiment 28, surface normal vector information is extracted at a plurality of surface normal positions of a digitally represented object, and extracting the surface normal vector information is performed by digitally representing the object at each surface normal position. 25. The computational system of embodiment 24, comprising identifying a plurality of vectors normal to a surface of an object.

In Embodiment 29, gradient information is extracted at a plurality of gradient extraction positions of a digitally represented object, surface normal vector information is extracted at a plurality of surface normal positions of a digitally represented object, and the surface normal vector information is extracted at a plurality of surface normal vector positions of a digitally represented object. 25. The calculation system according to embodiment 24, wherein the gradient extraction position of is different from a plurality of surface normal positions.

Embodiment 30 is the calculation system according to Embodiment 29, wherein the plurality of gradient extraction positions do not overlap with the plurality of surface normal positions.

Embodiment 31 is Embodiment 29, wherein the plurality of gradient extraction positions are located at the edges of the digitally represented object, and the plurality of surface normal positions are located away from the edges of the digitally represented object. This is a calculation system described in .

Embodiment 32 is a method for generating an object recognition template for identifying objects in a scene, the method comprising acquiring object information including a digitally represented object, and extracting two-dimensional measurement information from the object information. The method includes: extracting three-dimensional measurement information from the object information; and generating an object recognition template according to the two-dimensional measurement information and the three-dimensional measurement information.

Embodiment 33 is the method of embodiment 32, further comprising generating a feature map of the object model at the selected viewpoint.

Embodiment 34 corresponds to a digitally represented object by obtaining image information of a scene, accessing an object recognition template, and comparing two-dimensional measurement information and three-dimensional measurement information with image information. 33. The method of embodiment 32, further comprising identifying the object as an object.

Embodiment 35 is an embodiment in which extracting the two-dimensional measurement information further comprises extracting gradient information from the object information, the gradient information indicating a direction or orientation of a candidate edge of the digitally represented object. This is the method described in No. 32.

Embodiment 36 provides that extracting the three-dimensional measurement information further includes extracting surface normal vector information from the object information, wherein the surface normal vector information is normal to the surface of the digitally represented object. 33. The method of embodiment 32, which describes multiple vectors of lines.

Embodiment 37 extracts gradient information at a plurality of gradient extraction positions of a digitally represented object, analyzes the pixel intensity of two-dimensional image information of the object information, and extracts a two-dimensional image at each gradient extraction position. 36. The method of embodiment 35, further comprising measuring the direction in which the pixel intensity of the information changes.

Embodiment 38 includes extracting surface normal vector information at a plurality of surface normal positions of a digitally represented object, and extracting surface normal vector information at each surface normal position of a digitally represented object. 37. The method of embodiment 36, further comprising identifying a plurality of vectors of .

Embodiment 39 is a non-transitory computer-readable medium operable by at least one processing circuit via a communication interface configured to communicate with a robotic system, the non-transitory computer-readable medium operable by at least one processing circuit to perform object recognition for identifying objects in a scene. A non-transitory computer-readable medium configured with executable instructions for performing a method for generating a template. The method includes receiving object information including a digitally represented object, performing an operation of extracting two-dimensional measurement information from the object information, and performing an operation of extracting three-dimensional measurement information from the object information. and outputting an object recognition template to the robot system according to the two-dimensional measurement information and the three-dimensional measurement information.

Embodiment 40 includes receiving image information of a scene, accessing an object recognition template, and outputting a comparison of two-dimensional measurement information and three-dimensional measurement information for the image information to a robot system to identify an object. 40. The embodiment of embodiment 39, further comprising identifying as corresponding to a digitally represented object.

Embodiment 41 includes at least one processing circuit in communication with a robot having an arm and an end effector connected to the arm, and in communication with a camera having a field of view, the at least one processing circuit communicating with a robot having an arm and an end effector connected to the arm, the at least one processing circuit communicating with a camera having a field of view. is configured to execute instructions stored on the non-transitory computer-readable medium when the is or has been within the field of view, the instructions comprising: obtaining object image information of an object in the scene; obtaining a detection hypothesis comprising a corresponding object recognition template representing the template object; identifying mismatches between the template object and object image information; and adjusting the set of template positions to converge to the set of object positions, and including a corresponding object recognition template adjusted according to the adjusted set of template positions. , generating an adjusted detection hypothesis.

Embodiment 42 improves the template position by identifying respective vectors that extend between the set of template positions and a corresponding one of the set of object positions, and iteratively adjusting the set of template positions according to the respective vectors. 42. The computing system of embodiment 41, further comprising adjusting the set of positions.

Embodiment 43 provides that iteratively adjusting the set of template positions includes repeatedly generating an adjusted set of template positions according to the magnitude and direction of each vector acting on the template object; and identifying new respective vectors according to the set of adjusted template positions until the quality of the alignment exceeds a threshold. It is a system.

Embodiment 44 is the computing system of embodiment 43, wherein the quality of alignment is determined based on the level of misalignment defined by each new vector.

Embodiment 45 is the computing system of embodiment 43, wherein the quality of the alignment is determined based on distance measurements between the set of adjusted template positions and the set of object positions.

Embodiment 46 is the computing system of embodiment 45, wherein the distance measurements include Euclidean distance measurements.

Embodiment 47 is the computing system of embodiment 45, wherein the distance measurement comprises a cosine distance between surface normal vectors associated with the set of adjusted template positions and the set of object positions.

Embodiment 48 is the calculation system of embodiment 47, wherein the cosine distance indicates an angle between surface normal vectors, and the size of the angle correlates with the quality of the alignment.

Embodiment 49 is the computing system of embodiment 45, wherein the distance measurement is a measurement from a first position of the set of adjusted template positions to a plane of a second position of the set of object positions. It is.

Embodiment 50 is the computational system of embodiment 43, wherein the quality of alignment is determined by the percentage of convergence between the set of adjusted template positions and the set of object positions.

Embodiment 51 overlays image information of a scene on an object recognition template, and combines template gradient information and template surface normal vector information of the object recognition template with object gradient information and object surface normal vector information extracted from the image information. 42. The system of embodiment 41, further comprising obtaining a detection hypothesis by identifying object image information based on a comparison with the object image information.

Embodiment 52 includes obtaining object image information for an object in a scene, obtaining a detection hypothesis that includes a corresponding object recognition template representing the template object, and detecting a mismatch between the template object and the object image information. identifying a set of template positions within the template object that correspond to a set of object positions of the object image information; and adjusting the set of template positions to converge to the set of object positions; generating an adjusted detection hypothesis that includes an adjusted corresponding object recognition template according to the set of adjusted template positions.

Embodiment 53 provides that adjusting the set of template positions includes identifying respective vectors extending between the set of template positions and a corresponding one of the set of object positions, and adjusting the template positions according to the respective vectors. 53. The method of embodiment 52, further comprising iteratively adjusting the set.

Embodiment 54 includes: iteratively generating an adjusted set of template positions according to the magnitude and direction of each vector acting on the template object; and adjusting each vector according to the adjusted set of template positions; 54. The method of embodiment 53, further comprising identifying each new vector according to the adjusted set of template positions until the quality of the alignment exceeds a threshold.

Embodiment 55 is the method of embodiment 54, further comprising determining the quality of the alignment based on the level of misalignment defined by each new vector.

Embodiment 56 is the method of embodiment 54, further comprising determining the quality of alignment based on a distance measurement between the set of adjusted template positions and the set of object positions.

Embodiment 57 is the method of embodiment 54, further comprising determining the quality of alignment by a percentage of convergence between the set of adjusted template positions and the set of object positions.

In embodiment 58, obtaining a detection hypothesis includes overlaying image information of a scene on an object recognition template, and combining template gradient information and template surface normal vector information of the object recognition template with object gradient extracted from the image information. 53. The method of embodiment 52, further comprising identifying object image information based on a comparison of the information and object surface normal vector information.

Embodiment 59 is a non-transitory computer readable medium operable by at least one processing circuit through a communication interface configured to communicate with a robotic system to perform a method for refining a detection hypothesis. a non-transitory computer-readable medium having executable instructions for: receiving object image information of an object in a scene; and a detection hypothesis comprising a corresponding object recognition template representing the template object. and performing a process of identifying a mismatch between the template object and the object image information, and a process of identifying a set of template positions within the template object that correspond to a set of object positions of the object image information. and adjusting a set of template positions to converge to a set of object positions, and adjusting a corresponding object recognition template according to the adjusted set of template positions. and outputting the hypothesis to a robotic system.

Embodiment 60 provides that the process for adjusting the set of template positions includes: performing a process of identifying respective vectors extending between the set of template positions and a corresponding one of the set of object positions; 60. The method of embodiment 59, comprising performing the process of adjusting the set of template positions after iteratively adjusting the set of template positions according to a vector of .

Embodiment 61 includes at least one processing circuit in communication with a robot having an arm and an end effector connected to the arm and in communication with a camera having a field of view, the at least one processing circuit communicating with a robot having an arm and an end effector connected to the arm; The instructions are configured to execute instructions stored on the non-transitory computer-readable medium when being or were within the field of view, the instructions comprising: obtaining object image information of an object in the scene; obtaining a set of detection hypotheses including a corresponding object recognition template representing the template object; and validating each detection hypothesis of the set of detection hypotheses, wherein the validating the object recognition template of the detection hypothesis generating a plurality of three-dimensional verification scores based on a comparison between three-dimensional information of the object and three-dimensional information of object image information corresponding to the object, the plurality of three-dimensional verification scores being an occlusion validator score, including at least one of a point cloud validator score, a hole matching validator score, and a normal vector validator score, and two-dimensional information of the object recognition template and three-dimensional information of the object image information corresponding to the detection hypothesis. generating a plurality of two-dimensional validation scores based on the comparison of the plurality of two-dimensional validation scores, the plurality of two-dimensional validation scores including at least one of a rendered matching validator score and a template matching validator score. and filtering detection hypotheses from the set of detection hypotheses according to the plurality of 3D validation scores and the plurality of 2D validation scores, and filtering the detection hypotheses in the scene according to the unfiltered detection hypotheses remaining in the set of detection hypotheses after validation. is a computational system that detects objects in

Embodiment 62 is as described in embodiment 61, wherein the instructions further include: performing a robot motion planning procedure to retrieve the object from the scene; and outputting a command to move the robot to retrieve the object. It is a calculation system.

Embodiment 63 provides that the plurality of three-dimensional verification scores include a point cloud validator score, and the point cloud validator score compares an object position obtained from object image information with a surface of a template object; 62. The computational system of embodiment 61, wherein the point cloud validator score is obtained by identifying discrepancies with the surface.

Embodiment 64 is the computational system of embodiment 63, wherein the invalid object position is identified according to a discrepancy that places the object position below a surface of the template object, and the point cloud validator score is based on the invalid object position. It is.

Embodiment 65 provides that the plurality of three-dimensional validation scores include an occlusion validator score, the occlusion validator score compares an object position obtained from object image information with a surface of a template object, and the object position and the surface. 62. The computational system of embodiment 61, wherein the occlusion validator score is obtained by identifying discrepancies between the occlusion validator scores.

Embodiment 66 is the computational system of embodiment 65, wherein occlusion is identified according to a discrepancy that places the corresponding object position above or outside a surface of the template object, and the occlusion validator score is based on the occlusion.

In embodiment 67, the plurality of three-dimensional verification scores include a normal vector validator score, and the normal vector validator score applies a surface normal vector obtained from object image information to a corresponding surface method of the template object. 62. The computing system of embodiment 61, obtained by comparing with a line vector, identifying a mismatch between a surface normal vector and a corresponding surface normal vector to obtain a normal vector validator score. It is.

Embodiment 68 provides that the plurality of three-dimensional validation scores include a hole match validator score, the hole match validator score compares an object position obtained from object image information with a structure of a template object; The calculation of embodiment 61, obtained by identifying a mismatch between an object position and said structure, in order to identify the invalidity of a hole according to an object position at a position corresponding to an empty volume in the structure of said structure. It is a system.

Embodiment 69 provides that the rendered match validator score generates a two-dimensional rendering of an object in the scene, and that the rendered edges of the two-dimensional rendering of the object are compared to the extracted edges of the template object. 62. The computing system according to embodiment 61, wherein the calculated edge is obtained by identifying invalid edges.

Embodiment 70 is as in embodiment 61, wherein validating each detection hypothesis of the set of detection hypotheses further comprises comparing the corresponding object recognition template to a scene element other than an object corresponding to the template object. It is a calculation system.

Embodiment 71 is described in embodiment 70, wherein comparing a corresponding object recognition template representing the estimated object with the scene element includes determining whether an object corresponding to the template object is within the container. It is a calculation system.

Embodiment 72 provides that filtering a detection hypothesis from the set of detection hypotheses includes an occlusion validator score, a point cloud validator score, a hole match validator score, a normal vector validator score, a rendered match validator score, and comparing the template matching validator score with a corresponding threshold, and if either the three-dimensional validation score or the two-dimensional validation score cannot exceed the corresponding threshold, the detection hypothesis is selected from the set of detection hypotheses. 72. The computing system of embodiment 71, wherein the detection hypothesis remains in the set of detection hypotheses if it is deleted and the three-dimensional validation score and the two-dimensional validation score exceed all of the corresponding thresholds.

Embodiment 73 includes obtaining object image information for objects in a scene, obtaining a set of detection hypotheses each including a corresponding object recognition template representing a template object, and detecting each detection hypothesis of the set of detection hypotheses. generating a plurality of three-dimensional verification scores based on a comparison between three-dimensional information of an object recognition template of a detection hypothesis and three-dimensional information of object image information corresponding to the object, generating a plurality of three-dimensional validation scores, the validation scores including at least one of an occlusion validator score, a point cloud validator score, a hole match validator score, and a normal vector validator score; generating a plurality of two-dimensional validation scores based on a comparison of two-dimensional information of a corresponding object recognition template of a hypothesis and three-dimensional information of object image information, the plurality of two-dimensional validation scores being rendered; generating a plurality of two-dimensional validation scores including at least one of a matched validator score and a template-matching validator score; and a set of detection hypotheses according to the plurality of three-dimensional validation scores and the plurality of two-dimensional validation scores. A method comprising: filtering detection hypotheses from a set of detection hypotheses; and validating by detecting objects in a scene according to unfiltered detection hypotheses remaining in the set of detection hypotheses after validation.

Embodiment 74 is the method of embodiment 73, further comprising: performing a robot motion planning procedure to retrieve the object from the scene; and outputting a command to move the robot to retrieve the object. It is.

Embodiment 75 provides that generating a plurality of three-dimensional verification scores includes obtaining a normal vector validator score, and the normal vector validator score is a surface normal vector obtained from object image information. obtained by comparing with a corresponding surface normal vector of the template object and identifying discrepancies between the surface normal vector and the corresponding surface normal vector to obtain a normal vector validator score. 74. The method of embodiment 73, further comprising:

Embodiment 76 provides that obtaining the hole match validator score comprises: comparing the object position obtained from the object image information to the structure of the template object; and identifying a mismatch between the object position and the structure; 74. The method of embodiment 73, comprising identifying invalidity of the hole due to an object position at a location corresponding to an empty volume in the structure of the template object.

Embodiment 77 provides that obtaining a rendered match validator score includes: generating a two-dimensional rendering of an object in a scene; and determining invalid edges of the rendered match validator to identify invalid edges. 74. The method of embodiment 73, comprising comparing the edges to extracted edges of the template object.

Embodiment 78 is as in embodiment 73, wherein validating each detection hypothesis of the set of detection hypotheses further comprises comparing the corresponding object recognition template to a scene element other than an object corresponding to the template object. It's a method.

Embodiment 79 provides that filtering a detection hypothesis from the set of detection hypotheses includes an occlusion validator score, a point cloud validator score, a hole match validator score, a normal vector validator score, a rendered match validator score, and comparing the template matching validator score with a corresponding threshold and removing the detection hypothesis from the set of detection hypotheses if either the three-dimensional validation score or the two-dimensional validation score cannot exceed the corresponding threshold. and maintaining the detection hypothesis within the set of detection hypotheses if the three-dimensional validation score and the two-dimensional validation score exceed all of the corresponding thresholds.

Embodiment 80 is a non-transitory computer readable medium operable by at least one processing circuit through a communication interface configured to communicate with a robotic system to perform a method for testing a detection hypothesis. A non-transitory computer-readable medium having executable instructions for: receiving object image information of objects in a scene; and detecting detection hypotheses each including corresponding object recognition templates representing template objects. Based on the comparison of the three-dimensional information of the object recognition template of the detection hypothesis and the three-dimensional information of the object image information corresponding to the object, the occlusion validator score, point cloud validator score, hole performing a process of generating a plurality of three-dimensional verification scores including at least one of a match validator score and a normal vector validator score; and two-dimensional information of an object recognition template corresponding to the detection hypothesis and the object. performing a process of generating a plurality of two-dimensional validation scores, including at least one of a rendered matching validator score and a template matching validator score, based on comparing three-dimensional information of the image information; and filtering detection hypotheses from the set of detection hypotheses according to the plurality of three-dimensional validation scores and the plurality of two-dimensional validation scores; and after the validation, unfiltered detection hypotheses remaining in the set of detection hypotheses. According to a non-transitory computer-readable medium, the method includes detecting an object in a scene and outputting the detected object in the scene to a robotic system.

Embodiment 81 is the method of embodiment 13, wherein the three-dimensional space is substantially spherical and the object model is fixed at the center of the three-dimensional space.

Embodiment 82 is the method of embodiment 14, wherein each camera position corresponds to a set of viewpoints, and each viewpoint of the set of viewpoints corresponds to a different camera rotation angle.

Embodiment 83 is the method of embodiment 14, wherein the subset of the set of object recognition templates includes object recognition templates corresponding to viewpoints corresponding to different positions and different camera rotation angles.

Embodiment 84 provides that the digitally represented object is an object model, and extracting two-dimensional measurement information and three-dimensional measurements is performed to generate a feature map of the object model at a selected viewpoint. , the method described in Embodiment 32.

Embodiment 85 is the method of embodiment 32, wherein the object information includes object registration data and the digitally represented object includes an object model.

Embodiment 86 is the method according to Embodiment 32, wherein the object information includes at least one of two-dimensional image information and three-dimensional image information.

Embodiment 87 is such that gradient information is extracted at a plurality of gradient extraction positions of a digitally represented object, surface normal vector information is extracted at a plurality of surface normal positions of a digitally represented object, and surface normal vector information is extracted at a plurality of surface normal vector positions of a digitally represented object. 37. The method of embodiment 36, wherein the gradient extraction position of is different from the plurality of surface normal positions.

Embodiment 88 is the method of embodiment 87, wherein the plurality of gradient extraction positions do not overlap the plurality of surface normal positions.

Embodiment 89 is the embodiment 87 in which the plurality of gradient extraction positions are located at the edge of the digitally represented object and the plurality of surface normal positions are located away from the edge of the digitally represented object. This is the method described in .

Embodiment 90 is the method of embodiment 56, wherein the distance measurements include Euclidean distance measurements.

Embodiment 91 is the method of embodiment 56, wherein the distance measurement comprises a cosine distance between surface normal vectors associated with the set of adjusted template positions and the set of object positions.

Embodiment 92 is the method of embodiment 91, wherein the cosine distance indicates an angle between surface normal vectors, and the size of the angle correlates with the quality of the alignment.

Embodiment 93 is the method of embodiment 56, wherein the distance measurement is a measurement from a first position of the set of adjusted template positions to a plane of a second position of the set of object positions. be.

Embodiment 94 provides that the plurality of three-dimensional validation scores include a point cloud validator score, the point cloud validator score comparing an object position obtained from object image information with a surface of a template object; 74. The method of embodiment 73, wherein the point cloud validator score is obtained by identifying mismatches between the points and the surface.

Embodiment 95 is the method of embodiment 94, wherein the invalid object location is identified according to a discrepancy that places the object location below a surface of the template object, and the point cloud validator score is based on the invalid object location. be.

Embodiment 96 provides that the plurality of three-dimensional validation scores include an occlusion validator score, and the occlusion validator score compares an object position obtained from object image information with a surface of a template object; 74. The method of embodiment 73, wherein the occlusion validator score is obtained by identifying discrepancies between the occluding validator scores.

Embodiment 97 is the method of embodiment 96, wherein the occlusion is identified according to a discrepancy that places the corresponding object position above or outside a surface of the template object, and the occlusion validator score is based on the occlusion.

Embodiment 98 is described in embodiment 78, wherein comparing a corresponding object recognition template representing the estimated object with the scene element includes determining whether an object corresponding to the template object is within the container. This is the method.

Claims (20)

A computational system configured to generate an object recognition template for identifying objects in a scene, the computational system comprising:
comprising at least one processing circuit;
The at least one processing circuit includes:
Obtaining object information including digitally represented objects;
Extracting two-dimensional measurement information from the object information;
Extracting three-dimensional measurement information from the object information;
generating an object recognition template including the two-dimensional measurement information and the three-dimensional measurement information;
A computing system configured to perform The digitally represented object is an object model,
2. The computing system of claim 1, wherein extracting the two-dimensional measurement information and the three-dimensional measurement information is performed to generate a feature map of the object model at a selected viewpoint. The at least one processing circuit includes:
obtaining image information of the scene;
accessing the object recognition template;
comparing the two-dimensional measurement information and the three-dimensional measurement information with the image information to identify an object corresponding to the digitally represented object;
The computing system of claim 1, further configured to perform. Extracting the two-dimensional measurement information includes extracting slope information from the object information,
the gradient information indicates a direction or orientation of a candidate edge of the digitally represented object;
Extracting the three-dimensional measurement information includes extracting surface normal vector information from the object information,
The computing system according to claim 1, wherein the surface normal vector information describes a plurality of vectors normal to the surface of the digitally represented object. The object information includes registration data of the object,
2. The computing system of claim 1, wherein the digitally represented object includes an object model. The calculation system according to claim 1, wherein the object information includes at least one of two-dimensional image information and three-dimensional image information. the gradient information is extracted at a plurality of gradient extraction positions of the digitally represented object;
Extracting the gradient information includes analyzing pixel intensities of two-dimensional image information of the object information to determine a direction in which the pixel intensities of the two-dimensional image information change at each gradient extraction position. , the calculation system according to claim 4. the surface normal vector information is extracted at a plurality of surface normal positions of the digitally represented object;
5. The method of claim 4, wherein extracting the surface normal vector information includes identifying, at each surface normal position, a plurality of vectors normal to the surface of the digitally represented object. calculation system. the gradient information is extracted at a plurality of gradient extraction positions of the digitally represented object;
the surface normal vector information is extracted at a plurality of surface normal positions of the digitally represented object;
The calculation system according to claim 4, wherein the plurality of gradient extraction positions are different from the plurality of surface normal positions. The calculation system according to claim 9, wherein the plurality of gradient extraction positions do not overlap with the plurality of surface normal positions. The plurality of gradient extraction positions are arranged at edges of the digitally represented object,
10. The computing system of claim 9, wherein the plurality of surface normal positions are located away from edges of the digitally represented object. A method for generating an object recognition template for identifying objects in a scene, the method comprising:
Obtaining object information including digitally represented objects;
Extracting two-dimensional measurement information from the object information;
Extracting three-dimensional measurement information from the object information;
generating an object recognition template including the two-dimensional measurement information and the three-dimensional measurement information;
method including. 13. The method of claim 12, further comprising generating a feature map of the object model at the selected viewpoint. further comprising using the object recognition template to identify the object;
Using the object recognition template includes:
obtaining image information of the scene;
accessing the object recognition template;
13. The method of claim 12, comprising comparing the two-dimensional measurement information and the three-dimensional measurement information with the image information to identify an object as corresponding to the digitally represented object. Extracting the two-dimensional measurement information further includes extracting slope information from the object information,
13. The method of claim 12, wherein the gradient information indicates a direction or orientation of a candidate edge of the digitally represented object. Extracting the three-dimensional measurement information further includes extracting surface normal vector information from the object information,
13. The method of claim 12, wherein the surface normal vector information describes a plurality of vectors normal to the surface of the digitally represented object. extracting the gradient information at a plurality of gradient extraction positions of the digitally represented object;
analyzing the pixel intensity of the two-dimensional image information of the object information and measuring the direction in which the pixel intensity of the two-dimensional image information changes at each gradient extraction position;
16. The method of claim 15, further comprising: extracting the surface normal vector information at a plurality of surface normal positions of the digitally represented object;
identifying, at each surface normal position, a plurality of vectors normal to the surface of the digitally represented object;
17. The method of claim 16, further comprising: a non-transitory computer-readable medium operable by at least one processing circuit to generate an object recognition template for identifying objects in a scene through a communication interface configured to communicate with a robotic system; a non-transitory computer-readable medium having executable instructions for performing a method for
The method includes:
receiving object information including a digitally represented object;
performing a process of extracting two-dimensional measurement information from the object information;
performing a process of extracting three-dimensional measurement information from the object information;
outputting an object recognition template including the two-dimensional measurement information and the three-dimensional measurement information to the robot system. receiving image information of the scene;
accessing the object recognition template;
outputting a comparison between the two-dimensional measurement information and the three-dimensional measurement information for the image information to a robot system to identify the object as corresponding to the digitally represented object;
20. The non-transitory computer-readable medium of claim 19, further comprising: