JP6697204B1 - Robot system control method, non-transitory computer-readable recording medium, and robot system control device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a high degree of cooperation between units including a robot and sufficiently enhance the storage efficiency of an operation target. An end effector acquires an approach position for gripping an operation target, acquires a scan position for scanning an identifier of the operation target, creates or acquires a control sequence based on the approach position and the scan position, A control method including instructing a robot to execute the operation. The control sequence is (1) grasping the operation target from the start position, (2) scanning the identifier of the operation target with a scanner located between the start position and the work position, (3) satisfying a predetermined condition. At this time, the operation target is once released from the end effector in the holding position, and is gripped again by the end effector so as to be held again, and (4) the operation target is moved to the work position. [Selection diagram] Fig. 4B

Description

  The present disclosure generally relates to a robot system, and more particularly to a controller, a control method, a physical distribution system, a program, and a recording medium for a robot system that operates an operation target such as an article.

  Many robots (eg, machines configured to perform physical actions automatically / independently) are now widely used in many fields due to their ever-increasing performance and diminishing costs. . For example, the robot can be used in manufacturing, assembling, packing, transferring, transporting, and the like to perform various tasks such as manipulating and moving an operation target. In performing work, the robot can repeat human movements, thereby replacing or reducing human danger or repetitive work.

  As a system (robot system) using such a robot, for example, Patent Document 1 discloses a transport container storage mechanism that temporarily stores a transport container in order to automate and save labor from loading and unloading of articles. An automatic physical distribution system has been proposed which includes an automatic article delivery mechanism in which articles in a transport container are automatically collected in a shipping container based on shipping information.

JP, 2018-167950, A

  However, despite technological advances, robots often lack the sophistication necessary to duplicate human-involved tasks to perform larger and / or more complex tasks. Therefore, automation and high functionality in the robot system are still insufficient, and there are many tasks in which it is difficult to replace human involvement, and the robot system has a granularity in the operation to be executed. Lack of fine control and flexibility. Accordingly, there remains a need for technical improvements to manage various movements and / or interactions between robots and further facilitate automation and enhanced functionality of robotic systems. Therefore, it is an object of the present disclosure to provide a controller, a control method, and the like of a robot system that can realize a high degree of cooperation between units including a robot and can sufficiently enhance the storage efficiency of an operation target, for example.

  The present invention adopts the following configurations in order to solve the problems described above.

[1] That is, a control method of a robot system including a robot having a robot arm and an end effector according to the present disclosure includes acquiring an approach position at which the end effector grips an operation target, and Acquiring a scan position for scanning the identifier, creating or acquiring a control sequence based on the approach position and the scan position, and instructing the robot to execute the control sequence. Then, the control sequence is the following (1) to (4);
(1) Grasping the operation target from the start position,
(2) A scanner in which identification information of the operation target (for example, a computer-readable identifier such as a bar code or a quick response (QR) code (registered trademark)) is located between the start position and the work position. To scan with,
(3) When the predetermined condition is satisfied, the operation target is once released from the end effector at the holding position, and is gripped again by the end effector so that the holding target is held again; and
(4) moving the operation target to a work position,
including.

  Here, the “operation target” refers to an object operated by a robot included in the robot system, and includes, for example, one or more items (items) and containers such as bottles, containers, and boxes on which they are placed or accommodated. The container may be packaged or unpacked, and a portion of the container (eg, the top surface) may be open. In addition, in other embodiments and examples, the “operation target” may be a concept including shelves, pallets, conveyors, and other temporary storage areas. Further, the “control sequence” refers to an order of operations set in advance when performing control for causing one or more units such as a robot included in the robot system to execute individual work.

[2] In the above configuration, the control sequence is the following (5) and (6);
(5) It is set as the predetermined condition that the storage efficiency of the operation target at the work position is enhanced when the operation target is changed and the gripping direction of the end effector is changed, and
(6) calculating the storage efficiency in the work position before the operation target is changed and the storage efficiency in the work position after the operation target is changed,
May be included.

[3] In the above configuration, the control sequence is the following (7) and (8);
(7) Obtaining the height of the operation target, and
(8) calculating the storage efficiency based on the height of the operation target,
May be included.

[4] In the above configuration, the height of the operation target is the height position (level) of the top surface of the operation target and the height of the bottom surface of the operation target measured in a state of being gripped by the end effector. It may be configured to calculate from the position (level).

[5] In the above configuration, the height of the measurement target may be measured when the operation target is scanned by the scanner.

[6] In the above-mentioned configuration, the control sequence is (9) that when the predetermined condition is satisfied, the operation target is placed on a temporary placing table at the holding position and temporarily released from the end effector. It may be configured to include.

[7] In the above configuration, the imaging data indicating the pickup area including the operation target is acquired, the first pose of the operation target is determined based on the imaging data, and the first pose of the operation target is determined. It may be configured to further include calculating a confidence criterion indicating that the pose is likely to be accurate, and acquiring the approach position and the scan position based on the confidence criterion.

  Here, the “pose” indicates the position and / or orientation (for example, the posture including the orientation of the stopped state) of the operation target, and the component of parallel movement in the grid system used by the robot system and / or Including the rotation component. Further, the “pose” is indicated by a vector, a set of angles (for example, Euler angles and / or roll-pitch-yaw angles), a homogeneous transformation, or a combination thereof, and the “pose” of the operation target These coordinate transformations and the like may include translational components, rotational components, changes thereof, or a combination thereof.

  In addition, the “confidence criterion” refers to a quantified criterion indicating the degree of matching (certainty or possibility) of the determined pose of the operation target with the actual pose of the operation target in the real world. In other words, the “confidence criterion” is a criterion indicating the accuracy of the determined pose of the operation target, or an index indicating the possibility that the determined pose matches the actual pose of the operation target. Can also be said. The “confidence criterion” is stored in, for example, one or more visible characteristics (for example, shape, color, image, design, logo, text, etc.) of the operation target in the image data of the pickup area including the operation target and the master data. It can be quantified based on the result of matching with the information regarding the visible characteristic of the operated target.

[8] In the above configuration, the control sequence selects (10) the approach position and the scan position according to a performance metric and / or a scan metric based on a result of comparing the confidence criterion with a sufficiency threshold. Metric of the scan, the metric of the scan being related to the possibility that the identifier of the operation target is not covered by the end effector regardless of whether the first pose of the operation target is accurate or not. It may be configured to do so.

[9] In the above configuration, the approach position and the scan position are acquired based on the metric of the scan when the confidence criterion does not satisfy the sufficiency threshold, or the scan position is more than the performance metric. May be prioritized and acquired based on the metric of the scan.

[10] Alternatively, in the above configuration, the approach position and the scan position may be configured to be acquired based on the performance metric when the confidence criterion satisfies the sufficiency threshold.

[11] In the above configuration, the control sequence is the following (11) and (12);
(11) A first scan position for providing the identification information of the operation target to the scanner, and a second scan position for providing the alternative identification information of the operation target to the scanner. To obtain, and
(12) If the scan result indicates a successful scan after the operation target is moved to the first scan position, the operation target is moved to the work position, and the second scan position is set. , Or moving the operation target to the second scan position if the scan result indicates a failed scan.
May be included.

[12] The non-transitory computer-readable recording medium having processor instructions recorded thereon for implementing a control method of a robot system including a robot having a robot arm and an end effector according to the present disclosure is the processor instructions. However, a control sequence is created based on the command for acquiring the approach position where the end effector grips the operation target, the command for acquiring the scan position for scanning the identifier of the operation target, and the approach position and the scan position. Or a command for instructing the robot to execute the control sequence. Then, the control sequence is the following (1) to (4);
(1) Grasping the operation target from the start position,
(2) scanning the identification information of the operation target with a scanner located between the start position and the work position,
(3) When the predetermined condition is satisfied, the operation target is once released from the end effector at the holding position, and is gripped again by the end effector so that the holding target is held again; and
(4) moving the operation target to a work position,
including.

[13] In the above configuration, the control sequence is the following (5) and (6);
(5) It is set as the predetermined condition that the storage efficiency of the operation target at the work position is enhanced when the operation target is changed and the gripping direction of the end effector is changed, and
(6) calculating the storage efficiency in the work position before the operation target is changed and the storage efficiency in the work position after the operation target is changed,
May be included.

[14] In the above configuration, the control sequence is the following (7) and (8);
(7) Obtaining the height of the operation target, and
(8) calculating the storage efficiency based on the height of the operation target,
May be included.

[15] In the above configuration, the height of the measurement target is the height position (level) of the top surface of the operation target and the height of the bottom surface of the measurement target measured while being held by the end effector. It may be configured to calculate from the position (level).

[16] Further, a control device for a robot system including a robot having a robot arm and an end effector according to the present disclosure executes the control method according to any one of [1] to [11].

FIG. 3 illustrates an exemplary environment in which a robot system according to an embodiment of the present disclosure may operate. FIG. 3 is a block diagram showing an example of a hardware configuration of a robot system according to an embodiment of the present disclosure. It is a perspective view which shows typically the 1st pose of an operation target. It is a perspective view which shows typically the 2nd pose of an operation target. It is a perspective view which shows typically the 3rd pose of an operation target. FIG. 6 is a top view showing an exemplary work performed by a robot system according to an embodiment of the present disclosure. FIG. 6 is a front view showing an exemplary work performed by the robot system according to the embodiment of the present disclosure. FIG. 9 is a flowchart showing an example of a procedure in an operation of the robot system according to the embodiment of the present disclosure. FIG. 9 is a flowchart showing an example of a procedure in an operation of the robot system according to the embodiment of the present disclosure.

    According to the present disclosure, a robot system in which a plurality of units (for example, various robots, various devices, and a control device provided integrally or separately with them) is highly integrated, the control device thereof, and the same A physical distribution system and a method therefor are provided. That is, the robot system according to the embodiment of the present disclosure is, for example, an integrated system capable of autonomously executing one or more tasks. Further, the robot system according to the embodiment of the present disclosure can significantly enhance the storage efficiency when storing the operation target in the storage container or the like, based on the shape and size of the operation target and the spatial volume of the storage container. Operates to include advanced handling that can be done. In addition, a sophisticated scanning operation of the operation target is provided by creating or acquiring and executing the control sequence based on the confidence criterion related to the first pose of the operation target.

  The robot system according to the embodiment of the present disclosure may be configured to perform work based on operating an operation target (for example, physical movement and / or orientation). More specifically, the robot system operates from, for example, a pickup area including a start position (for example, a large box, a bin, a container, a pallet, a storage container, a bucket, a cage, a belt conveyor, which is a supply source of an operation target). Pick up an object, move it to the drop area that contains the desired work position (for example, the destination box, bin, container, pallet, storage container, bucket, cage, belt conveyor, etc.) and perform various operations. The objects can be rearranged or replaced.

  In addition, the control sequence executed by the robot system includes one or more specific positions to be operated and / or one or more identifiers located on the surface (for example, a bar code or a quick response (QR) code (registered trademark). ) Etc.) may be included in the scan during the transfer. Therefore, the robot system grips and picks up the operation target, scans the identifier at an appropriate position / orientation, adjusts the pose, changes the pose and holds (releases the grip, re-grasps again, and picks up the object. ), It is possible to perform various works such as transferring to the work position, releasing the grip, and disposing at the work position.

  In addition, the robot system may include an imaging device (for example, a camera, an infrared sensor / camera, a radar, a rider, etc.) for identifying the position and pose of the operation target and the environment around the operation target. In addition, the robot system can calculate a confidence criterion for the pose of the manipulated object. Further, the robot system includes a pickup area including a start position, a drop area including a work position, and an area including a holding position in the middle of a movement path of an operation target (for example, an appropriate work table such as a temporary table, other It is possible to acquire an image showing the position and the orientation of the operation target when it is transferred to the robot etc.).

  Further, the robot system can perform image processing so as to identify or select the operation target according to a predetermined order (for example, top to bottom, outer to inner, inner to outer, etc.). Furthermore, the robot system identifies the outline of the operation target based on, for example, the color, the brightness, the depth / position of the pixel in the pattern image of the imaging data, and / or the combination thereof or the change of the values thereof. Further, by grouping them, the first pose of the operation target in the pickup area can be determined from the image. In determining the initial pose, the robot system can calculate a confidence metric according to a predetermined process and / or equation.

  In addition, the robot system changes the holding target of the operation target (holding position of the operation target at the holding position of the operation target, if necessary, at the holding position provided on the way from the pickup area including the start position to the drop area including the work position. Change). Then, while the operation target is moved from the pickup area including the start position to the drop area including the work position, the robot system may increase the height of the operation target by an imaging device having a distance measuring function, if necessary. Can be obtained.

  In addition, the robot system controls to execute each work according to the position, pose, height, confidence standard, or a combination thereof of the operation target and / or the position, pose, or a combination thereof of the robot. The sequence can be executed. Such a control sequence can be created or acquired by, for example, machine learning such as motion planning or deep learning. The control sequence is, for example, rearrangement of operation objects, holding, replacement, etc., at the start position and / or at an arbitrary position during movement, to grasp the operation objects, to steer the operation objects, It corresponds to placing the operation target in the intended work position.

  Here, in the conventional robot system, a control sequence is performed in which the operation target is gripped in a pickup area including the start position and the operation target is moved to a drop area including the work position in the gripped state and released. To do. Therefore, in the conventional system, the gripped operation target is only moved in the gripped state and released from the gripped state, and the space in which the operation target is stacked or stored can be used sufficiently effectively. It couldn't be said. For this reason, human intervention (adjustment, redo, supplement, system stop, etc.) and an operation input therefor may be required from the viewpoint of packing or storage efficiency of operation targets.

  On the other hand, unlike the related art, the robot system according to the present disclosure can create or acquire the control sequence based on the shape information of the operation target and the stowage or storage information of the operation target and execute the control sequence. In other words, the robot system according to the present disclosure can optimize the packing or storage efficiency of further operation targets based on the shape information of the operation targets and the packing or storage information of the operation targets. Further, the robot system according to the present disclosure changes the grip position of the operation target to a grip position suitable for optimizing the packing efficiency or the storage efficiency of the operation target at the holding position which is located in the middle of the work position from the start position. be able to.

  Further, unlike the conventional system, the robot system according to the present disclosure creates or acquires a control sequence that is suitable for optimizing the packing efficiency or storage efficiency of the operation target according to the actual height of the operation target, as necessary. Can be executed. For example, one or more specific positions of scanned objects and / or one or more identifiers located on the surface may be the same, but actually have different geometric dimensions. obtain. Therefore, on the upstream side (previous stage) of the control sequence with respect to the holding position, for example, the distance from the vertically positioned imaging device (camera or distance measuring device) to the operation target whose supported position is known. The height of the operation target is measured based on the information. Then, the packing or storage efficiency of the operation targets at the work position can be calculated based on the actually measured height of the operation target, and the control sequence can be further optimized based on the result.

  Moreover, unlike conventional systems, the robotic system according to the present disclosure can optionally create or acquire and execute control sequences according to confidence criteria. For example, the approach to the operation target can be changed, the grip position on the operation target can be changed, the pose / position of the operation target can be changed, and / or a part of the movement path can be changed according to a confidence criterion.

  In addition, the pose of the measurement target held in the pickup area or the like is generally exposed with its top surface facing horizontally (upward) and the side surface of the operation target facing vertically (horizontal direction). ) Can be exposed. Therefore, in the robot system according to the present disclosure, in the master data, the operation target has one identifier on the bottom surface of the operation target (that is, on the side opposite to the top surface of the operation target), and operates another identifier. Having on one of the side surfaces of the object.

  In addition, the robot system according to the present disclosure can calculate the confidence criterion as necessary when processing the image of the pickup area in identifying the operation target. If the confidence criterion exceeds the adequacy threshold and a sufficient certainty that the top surface to be manipulated is exposed is recognized, the robot system places an end effector on the exposed top surface and The manipulation target can be rotated to grip the surface and provide the bottom surface of the manipulation target in place in front of the scanner. On the other hand, if the confidence criterion is less than the sufficiency threshold and it is not recognized whether the top or bottom surface of the operation target is exposed, then the robot system may move the end effector along one of the side surfaces of the operation target. Can be arranged to grasp the side surface of the operation target and rotate the operation target, for example, so as to pass it between the sets of opposed scanners.

  In this case, by scanning the operation target within the movement path of the operation target, for example, between the pickup area including the start position and the drop area including the work position, the work efficiency and work speed are improved. At that time, the robot system according to the present disclosure can effectively combine the moving operation of the operation target and the scanning operation of the operation target by creating or acquiring the control sequence that also cooperates with the scanner at the scan position. Furthermore, by creating or acquiring a control sequence based on the confidence criterion of the first pose of the operation target, efficiency, speed and accuracy regarding the scanning work can be further improved.

  In addition, the robot system according to the present disclosure can create or acquire a control sequence corresponding to the case where the first pose of the operation target is not accurate. As a result, the operation target can be accurately and surely set even when accompanied by an error in determining the pose of the operation target (for example, an error in determining a result such as a calibration error, an unexpected pose, or an unexpected light condition). The likelihood of scanning can be increased. As a result, the overall throughput of the robot system can be increased and the labor / intervention of the operator can be further reduced.

  It should be noted that although specific details have been set forth herein in order to provide a thorough understanding of the present disclosure, the present disclosure is not limited thereto. In addition, the embodiments of the present disclosure can be carried out with respect to the technology described in the present specification without accompanying those specific details. In addition, well-known specific functions or routines, etc. have not been described in detail to avoid unnecessarily obscuring the present disclosure. References to “an embodiment”, “one embodiment” and the like in this specification are made in accordance with at least one embodiment of the disclosure in which the particular feature, structure, material, or characteristic described is at least one embodiment. Means included in. As such, references to such phrases in this specification are not necessarily all referring to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, the particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments. In addition, it should be understood that the various embodiments shown are for illustration purposes only and are not necessarily shown to scale.

  Also, structures or processes that are well known and often associated with robotic systems and subsystems that may unnecessarily interfere with some salient aspects of the present disclosure are described in the clarifications of this disclosure. The description is omitted for the purpose. Further, although various embodiments of the disclosure are described herein, the disclosure includes other embodiments having different configurations or different components than those described in this section. You can As such, the disclosure may include other embodiments with additional elements or without some of the elements described below.

  Also, each embodiment of the present disclosure includes a routine executed by a programmable computer or controller, and can take the form of instructions executable by the computer or controller. It should be noted that those skilled in the art to which the present disclosure belongs may understand that the technique of the present disclosure can be implemented in a system including various computers or controllers. The techniques of this disclosure may be implemented in a special purpose computer or data processor that is programmed, configured, or constructed to execute one or more instructions on various computers. Thus, the terms "computer" and "controller" as used herein may be any data processor, Internet devices and handheld devices (palmtop computers, wearables, etc.). Computer, cellular or mobile telephone, multi-processor system, processor-based or programmable consumer electronics, network computer, minicomputer, etc.). The information handled by these computers and controllers can be provided on any suitable display medium such as a liquid crystal display (LCD). The instructions for performing computer- or controller-executable tasks may be stored in any suitable computer-readable storage medium, including hardware, firmware, or a combination of hardware and firmware. Also, the instructions may be recorded in any suitable memory device, including, for example, a flash drive and / or other suitable medium.

  Also, the terms "coupled" and "connected", as well as their derivatives, may be used herein to describe structural relationships between components. It should be understood that these terms are not intended to be synonymous with respect to each other. Rather, in certain embodiments, "connected" can be used to indicate that two or more elements are in direct contact with each other. Unless otherwise indicated in context, the term "coupled" refers to two or more elements either directly or indirectly contacting each other with other intervening elements between them. Or two or more elements cooperate or interact with each other in a causal relationship, such as, for example, with respect to sending / receiving signals or calling functions, or It can be used to indicate both.

[Suitable environment]
FIG. 1 is a diagram illustrating an exemplary environment in which a robot system 100 according to an embodiment of the present disclosure may operate. The robot system 100 comprises units such as one or more robots configured to perform one or more tasks.

  With respect to the example shown in FIG. 1, the robot system 100 may comprise an unloading unit 102, a transfer unit 104, a transportation unit 106, a loading unit 108, or a combination thereof in a warehouse or distribution / transport hub. In these units, examples of the robot for operating the operation target include a robot for operating the operation target by a robot arm and an end effector such as a devanning robot, a piece picking robot, and a fetching robot. Further, each unit in the robot system 100 unloads an operation target from a truck, a van, or the like, or unloads an operation target from a storage position in order to store one or more operations, for example, a warehouse. Perform a control sequence that combines multiple operations, such as moving multiple operations between containers and loading multiple operations onto a truck or van for transportation. be able to. That is, the “work” here is a concept including various operations and actions for the purpose of transferring an operation target from “a certain position” to “another certain position”.

  More specifically, “work” includes operations of the operation target 112 from the start position 114 of the operation target 112 to the work position 116 (for example, movement, orientation, change of pose, etc.), work positions from the start position 114. Reholding of the operation target 112 at the holding position 118 provided in the middle of the movement path of the operation target 112 to 116, scanning of the operation target 112 for acquiring identification information of the operation target 112, and the like are included.

  Further, for example, the unloading unit 102 can be configured to transfer the operation target 112 from a certain position in the carrier (for example, a truck) to a certain position on the belt conveyor. Further, the transfer unit 104 transfers the operation target 112 from one position (for example, a pickup area including a start position) to another position (for example, a drop area including a work position on the transportation unit 106), and The operation target 112 can be changed over in the middle of the movement route. Further, the transport unit 106 can transfer the operation target 112 from the area associated with the transfer unit 104 to the area associated with the loading unit 108. Furthermore, the loading unit 108 moves the operation target 112, for example, a pallet or the like on which the operation target 112 is placed, so that the loading unit 108 moves from the transfer unit 104 to a storage position (for example, a predetermined position on a shelf such as a rack in a warehouse). Can be transferred to.

  It should be noted that although the robot system 100 is described herein as an example applied to a transportation center, the robot system 100 may be used in manufacturing, assembly, packaging, healthcare, and / or other types of automation. It should be appreciated that, etc., can be configured to perform work in other environments / for other purposes. It should also be appreciated that the robot system 100 may include other units not shown in FIG. 1, such as manipulators, service robots, modular robots, etc. For example, the robot system 100 may, for example, transfer a target 112 from a cage cart or pallet to a conveyor or other pallet, a pallet unloading unit, and a container switch to transfer the target 112 between containers. A unit, a packaging unit for wrapping the operation target 112, a sorting unit for grouping according to the characteristics of the operation target 112, various operations of the operation target 112 according to the characteristics of the operation target 112 (for example, sorting , Grouping, and / or transfer), a self-propelled carriage unit that moves a pallet or rack for storing the operation target 112 (for example, an automated guided vehicle, an automated guided vehicle, etc.), or any combination thereof. Can be included.

[Appropriate system]
FIG. 2 is a block diagram showing an example of the hardware configuration of the robot system 100 according to an embodiment of the present disclosure. The robot system 100 may include, for example, one or more processors 202, one or more storage devices 204, one or more communication devices 206, one or more input-output devices 208, one or more actuating devices 212, one. An electronic or electrical device such as the transfer motor 214 above, one or more sensors 216, or a combination thereof may be included. The various electronic or electrical devices can be coupled to each other via wired and / or wireless connections.

  Further, the robot system 100 includes, for example, a system bus, a peripheral component interconnection (PCI) bus or a PCI-express bus, a HyperTransport or an industrial standard configuration (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB). ), An IIC (I2C) bus, or a Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also referred to as "Firewire"). Further, the robot system 100 can include bridges, adapters, controllers, or other signaling devices, for example, to provide wired connections between electronic or electrical devices. The wireless connection may be, for example, a cellular communication protocol (for example, 3G, 4G, LTE, 5G, etc.), a wireless local area network (LAN) protocol (for example, a faithful wireless communication environment (WIFI)), peer-to-peer or Device-to-device communication protocol (for example, Bluetooth (registered trademark), Near-Field communication (NFC), etc.), Internet of Things (IoT) protocol (for example, NB-IoT, LTE-M, etc.), and / or other It can be based on the wireless communication protocol.

  Processor 202 is a data processor (eg, a central processing unit (CPU)) configured to execute instructions (eg, software instructions) stored in storage device 204 (eg, computer memory), a special purpose computer, and And / or an onboard server). The processor 202 can implement program instructions to control / interact with other devices, causing the robotic system 100 to perform control sequences including various actions, tasks, and / or operations.

  Storage device 204 may include a non-transitory computer readable recording medium having program instructions (eg, software) stored thereon. Examples of the storage device 204 include, for example, volatile memory (eg, cache and / or random access memory (RAM)) and / or non-volatile memory (eg, flash memory and / or magnetic disk drive), portable memory drive and / or Examples include cloud storage devices. The storage device 204 may also be used to further store and provide access to processing results and / or predetermined data / thresholds, for example, storing master data 252 including information about the manipulated object 112. You can

  The master data 252 is, for example, the size, shape, mass, center of gravity, position of the center of mass, template regarding poses and outlines, model data for recognizing different poses, stock management unit (SKU: Stock Keeping Unit), etc., color scheme, image, identification information, logo, expected position of operation target, expected sensor measurement value (eg force, torque, pressure, physical quantity related to contact reference value), or those Can be included.

  The storage device 204 can also store the tracking data 254 of the operation target 112, for example. The tracking data 254 includes a log of an operation target to be scanned or operated, imaging data of the operation target 112 at one or more positions (for example, an appropriate start position, work position, holding position, etc.) (for example, a photograph, a point). Cloud, live video, etc.), the position and / or pose of the operation target 112 at one or more positions thereof.

  The communication device 206 includes, for example, a circuit, a receiver, a transmitter, a conditioner / demodulator (modem), a signal detector, a signal encoder / decoder, which is configured to communicate with an external or remote device via a network. It can include connector ports, network cards and the like. The communication device 206 can also be configured to send, receive, and / or process electrical signals in response to one or more communication protocols (eg, Internet Protocol (IP), wireless communication protocols, etc.). The robot system 100 exchanges information between units and exchanges information with external systems or devices for appropriate purposes such as reporting, data collection, analysis, and troubleshooting. , The communication device 206 can be used.

  The input-output device 208 is, for example, a keyboard, a mouse, a touch screen, a microphone, as a user interface device configured to input information and instructions from an operator, communicate and present information to the operator, and the like. It may include a user interface (UI) sensor (eg, a camera for receiving motion commands), an input device such as a wearable input device, and an output device such as a display 210, a speaker, a haptic circuit, a tactile feedback device. The robot system 100 may also use the input-output device 208 to communicate with an operator when performing an action, task, operation, or combination thereof.

  The robot system 100 includes, for example, a physical or structural member (for example, a robot manipulator, a robot arm, or the like) connected by a link or a joint in order to execute a control sequence including displacement such as movement or rotation of the operation target 112. Hereinafter, simply referred to as "structural member"). Such physical or structural members and links or joints may be end effectors (eg, gripping, rotating, welding, assembling, etc.) configured to perform one or more tasks in the robotic system 100 (eg, Grippers, hands, etc.). The robot system 100 also includes an actuation device 212 (eg, motor, actuator, wire) configured to drive or steer (eg, displace and / or reorient) structural members around or at the joint. , Artificial muscles, electroactive polymers, etc.) and a transfer motor 214 configured to transfer the unit from one location to another.

  The robot system 100 can also include sensors 216 configured to obtain information used to steer structural members and / or perform operations such as transferring units. The sensor 216 detects or detects one or more physical properties of the robot system 100 (eg, one or more structural members, link or joint conditions, conditions, positions, etc.) and / or properties of the surrounding environment. It may include devices configured to measure, such as accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders and the like.

  Further, the sensor 216 may include one or more imaging devices 222 (eg, visible and / or infrared cameras, two-dimensional and / or three-dimensional imaging cameras, lidars, radars, etc.) configured to detect the surrounding environment. Distance measuring device, etc.) can be included. The imaging device 222 may generate a display of the detection environment, such as a digital image and / or a point cloud, to obtain visual information for automated inspection, robot guidance, and other robot applications, for example.

  Further, the robot system 100 processes a digital image, a point cloud, distance measurement data, and the like via, for example, the processor 202, and the operation target 112 of FIG. 1, the start position 114 of FIG. 1, the work position 116 of FIG. The holding position 118 between the start position 114 and the work position 116, the pose of the operation target 112, the confidence criterion regarding the pose of the operation subject at the start position 114, the confidence criterion regarding the height of the operation subject 112, or a combination thereof is identified. be able to.

  Further, the robot system 100 operates the operation target 112 in a designated area (for example, a pickup area in a truck or on a belt conveyor, a drop area for arranging the operation target 112 on the belt conveyor, an operation area). An image of an area for holding the target 112, an area for arranging the operation target in the container, an area on a pallet for stacking the operation target 112, etc.) is acquired and analyzed through various units, The operation target 112, its start position 114, its work position 116, the holding position 118, etc. can be identified. The imaging device 222 also includes, for example, one or more cameras configured to generate images such as a pickup area, a drop area, and an area for holding the operation target 112 set therebetween. be able to.

  In addition, the imaging device 222 is one of a rider, a radar, and the like configured to measure the distance to the operation target 112 supported at a predetermined position on the upstream side (previous stage) of the holding position 118. The above distance measuring device can be included. The robot system 100 determines a start position 114, a work position 116, a holding position 118, a related pose, an actual height of the operation target 112, a confidence standard, and the like based on the acquired image and / or distance measurement data. be able to.

  In addition, the imaging device 222 operates in order to scan the operation target 112 during transportation or movement of the operation target, for example, between the start position 114 and the work position 116 (preferably before the holding position 118). One or more scanners 412, 416 (eg, bar code scanners, QR code scanners (registered trademark)) configured to scan the identification information of the target 112 (eg, identifier 332 of FIGS. 3A and / or 3C described below). ) Etc .: See FIG. 4A and FIG. 4B described later). Then, the robot system 100 can create or obtain a control sequence for providing one or more portions of the operation target 112 to the one or more scanners 412.

  Further, the sensor 216 may include, for example, a position sensor 224 (eg, position encoder, potentiometer, etc.) configured to detect the position of structural members or links or joints. The position sensor 224 may be used to track the position and / or orientation of structural members or links or joints during the performance of work.

  Also, the sensor 216 may include, for example, a contact sensor 226 (eg, pressure sensor, force sensor, strain gauge, piezoresistive / piezoresistive / piezoresistive sensor configured to measure properties associated with direct contact between physical structures or surfaces). Sensors, capacitive sensors, elastic resistance sensors, other tactile sensors, etc.). The contact sensor 226 can measure a characteristic corresponding to the grip of the end effector of the operation target 112. Thereby, the contact sensor 226 outputs a contact reference indicating a quantified measurement value (for example, measured force, torque, position, etc.) corresponding to the degree of contact between the end effector and the operation target 112. be able to. Here, the “contact reference” can include, for example, one or more force or torque readings related to the force applied to the actuated object 112 by the end effector.

[Judgment of trust criteria for the first pose]
3A, 3B, and 3C schematically illustrate a first pose 312, a second pose 314, and a third pose 316 as examples of various poses (positions and orientations) of the operation target 302. It is a perspective view shown in FIG. The robot system 100 can process, for example, a 2D image, a 3D image, a point cloud, and / or other imaging data from the imaging device 222 to identify a pose of the manipulation target 302. The robot system 100 can also analyze imaging data from one or more imaging devices 222 directed to the pickup area, for example, to identify the initial pose of the operation subject 302.

  In order to identify the pose of the operation target 302, the robot system 100 first analyzes a pattern image of the operation target 302 in the imaging data based on a predetermined recognition mechanism, a recognition rule, and / or a template regarding a pose and an outline. However, the outline (for example, the peripheral edge or surface) of the operation target 302 can be identified, or it can be grouped. More specifically, the robot system 100, for example, based on the outline or pose template in the master data 252, color, brightness, depth / position, and / or a combination thereof, and the like over the outline to be operated. It is possible to identify the grouping of the outlines that correspond to the pattern in the change in the value of (e.g., the same value, change in a known ratio / pattern, etc.).

  When the outlines of the operation target 302 are grouped, the robot system 100 may, for example, one or more surfaces, edges, and / or points of the operation target 302 in a grid or coordinate system used in the robot system 100. Also, the pose can be identified.

  The robot system 100 can also identify one or more exposed surfaces (eg, the first exposed surface 304, the second exposed surface 306, etc.) of the manipulation target 302 in the imaging data. Further, the robot system 100 uses, for example, the outline of the operation target 302 and the imaging data relating to the calibration, or the mapping data of the imaging device 222 to determine the shape of the outline of the operation target 302, one or more dimensions (for example, the length). , Width, and / or height) and compare the determined dimensions with corresponding data in the master data 252 to identify the operation target 302. Further, the robot system 100 determines whether the exposed surface corresponds to the top surface 322, the bottom surface 324, or the outer peripheral surface 326 based on the length, width, and height of the operation target 302 whose exposed surface dimensions are identified. Can be identified.

  The robot system 100 may also include, for example, one or more markings (eg, letters, numbers, shapes, visible images, logos, or combinations thereof) displayed on one or more exposed surfaces in the master data 252. The operation target 302 can be identified by comparing with one or more predetermined images. In this case, the master data 252 may include, for example, one or more images of a product name, logo, design / image, or a combination thereof on the package surface of the operation target 302. The robot system 100 also compares a portion of the imaged data (eg, the portion within the outline of the operation target 302) with the master data 252 to identify the operation target 302, and likewise, is unique to the surface. The pose (particularly the orientation) of the operation target 302 can be identified based on the predetermined image pattern.

  Here, in FIG. 3A, the first exposed surface 304 (eg, the exposed surface facing up) is the top surface 322 of the operation target 302, and the second exposed surface 306 (eg, generally as a source of imaging data). The first pose 312 is shown when the exposed exposed surface) is one of the outer peripheral surfaces 326 of the operation target 302.

  In identifying exposed surfaces, the robotic system 100 processes the imaging data of FIG. 3A to determine a measurement of the dimensions (eg, number of pixels) of the first exposed surface 304 and / or the second exposed surface 306. Calibration or mapping functions can be used to map to real world dimensions. The robot system 100 can also compare the mapped dimensions to the known / expected dimensions of the manipulated object 302 in the master data 252 and identify the manipulated object 302 based on the results. Further, the robot system 100 ensures that the pair of intersecting edges that bound the first exposed surface 304 match the length and width of the identified manipulation target 302, so that the first exposed surface 304 is It can be identified whether it is the top surface 322 or the bottom surface 324. Similarly, the robot system 100 identifies the second exposed surface 306 as the outer peripheral surface 326 because one of the edges defining the second exposed surface 306 matches the height of the identified operation target 302. can do.

  The robot system 100 can also process the imaging data of FIG. 3A to identify one or more markings unique to the surface of the manipulation target 302. In this case, the master data 252 includes one or more images and / or other visual characteristics (eg, color, size, size, etc.) of the surface and / or unique markings of the manipulation target 302 as described above. You can As shown in FIG. 3A, since the operation target 302 has “A” on the top surface 322, the robot system 100 can identify the operation target 302 as the operation target stored in the master data 252. , The first exposed surface 304 can be identified as the top surface 322 of the operation target 302.

  Further, the master data 252 can include an identifier 332 as identification information of the operation target 302. More specifically, the master data 252 includes an image of the identifier 332 of the operation target 302 and / or a coded message, a position 334 of the identifier 332 with respect to a set of surfaces and / or edges, one or more visuals thereof. Characteristics, or a combination thereof. As shown in FIG. 3A, the robot system 100 identifies the second exposed surface 306 as a peripheral surface 326 based on the presence of the identifier 332 and / or their position matching the position 334 of the identifier 332. You can

  3B shows the second pose 314 in which the operation target 302 is rotated 90 degrees about the vertical axis along the direction B in FIG. 3A. For example, the reference point “α” of the operation target 302 can be the lower left corner of FIG. 3A and the lower right corner of FIG. 3B. Therefore, in comparison with the first pose 312, the top surface 322 of the operation target 302 is recognized as a different orientation in the imaging data and / or the outer peripheral surface 326 of the operation target 302 having the identifier 332 is not visible.

  The robot system 100 can identify various poses of the manipulation target 302 based on the particular orientation of the one or more visual feature identifiers 332. For example, a dimension matching a known length of the operation target 302 extends horizontally in the imaging data, a dimension matching a known height of the operation target 302 extends vertically in the imaging data, and / or a dimension of the operation target 302. The first pose 312 and / or the third pose 316 can be determined if a dimension that matches a known width extends along the depth axis in the imaging data. Similarly, the robot system 100 has a width-matching dimension that extends horizontally in the imaging data, a height-matching dimension that extends vertically in the imaging data, and / or a length-matching dimension that extends deep in the imaging data. The second pose 314 can be determined if it extends along the length axis.

  In addition, the robot system 100 determines that the operation target 302 is in the first pose 312 or the second pose 314 based on the direction of the visual marking such as “A” shown in FIGS. 3A and 3B. Can be determined. In addition, the robot system 100 provides visual markings that should be visible on each surface combination, such as when the identifier 332 of the operational object 302 is viewed with the marking “A” (ie, on different surfaces). Based on this, it can be determined that the operation target 302 is in the first pose 312.

  Further, FIG. 3C shows the third pose 316 in which the operation target 302 is rotated 180 degrees about the horizontal axis along the direction C in FIG. 3A. For example, the reference point “α” of the operation target 302 can be the lower left front corner of FIG. 3A and the upper left rear corner of FIG. 3C. Therefore, in comparison to the first pose 312, the first exposed surface 304 is the bottom surface 324 of the operation target, and both the top surface 322 and the outer peripheral surface 326 having the identifier 332 of the operation target 302 are visible. Not done.

  As described above, the robot system 100 can identify that the operation target 302 is in the first pose 312 or the third pose 316 based on the dimensions determined from the image data, and the top surface 322. When the marker (eg, “A”) is visible, it is possible to identify that the operation target 302 is in the first pose 312. Further, the robot system 100 can identify that the operation target 302 is in the third pose 316 when the marker on the bottom surface (for example, the example of the operation target identifier 332) is visually recognized.

  Further, when determining the pose of the operation target 302, the situation in the real world may affect the accuracy of the determination. For example, the visibility of the marking on the surface may be reduced due to, for example, reflection and / or shading of the light conditions. Furthermore, the actual orientation of the manipulator 302 may reduce the exposure or viewing angle of one or more surfaces, which may cause any markings on the surface to be indistinguishable. Therefore, the robot system 100 can calculate the confidence criterion regarding the pose of the determined operation target 302.

  In addition, the robot system 100 can calculate the confidence criterion based on the certainty interval related to the dimension measurement in the image in the captured data. In this case, the certainty interval may vary as the distance between the manipulation target 302 and the imaging source (eg, the imaging device 222) decreases, and / or the measured edge of the manipulation target 302 may radiate from the imaging source. It may increase as it moves away from the direction parallel to the radial direction as it approaches the direction orthogonal to the direction of rotation. Further, the robot system 100 can calculate a confidence metric, for example, based on the degree of match between the marker or design in the imaging data and the known marker / design in the master data 252. Furthermore, the robot system 100 can measure the degree of overlap or deviation between at least a part of the imaging data and a predetermined marker / image.

  In this case, the robotic system 100 identifies the manipulator 302 and / or its orientation according to the measurement of the largest overlap and / or the smallest deviation, such as for the least mean squared error (MMSE) mechanism. Furthermore, a confidence criterion can be calculated based on the degree of overlap / deviation obtained. Then, the robot system 100 can calculate the movement path of the operation target 302 in the control sequence based on the obtained reliability standard. In other words, the robot system 100 can calculate the movement path based on the acquired reliability standard. The operation target 302 can be appropriately moved.

[System operation]
FIG. 4A is a top view illustrating an exemplary operation 402 performed by the robot system 100 according to one embodiment of the present disclosure. As mentioned above, task 402 is an example of a control sequence performed by robot system 100 (eg, performed by the units such as shown in FIG. 1). As shown in FIG. 4A, for example, in the work 402, the operation target 112 is moved from the pickup area including the start position 114 to the drop area including the work position 116 via the holding position 118. Scanning the operation target 112 while moving from the work position to the work position 116 and changing the operation target 112 (changing the grip position) at the holding position 118 can be included. Thereby, the robot system 100 adds the scanned operation target 112 to the tracking data 254 of the operation target 112, removes the operation target 112 from the tracking data 254, and / or evaluates the operation target 112. The tracking data 254 can be updated at any time.

  The robot system 100 also uses a pickup area (more specifically, for example, a designated area for a pallet or bin for parts procurement and / or a belt) to identify and / or identify the start position 114. It may include a scanner 412 (an example of an imaging device 222) such as a 3D vision aimed at a pick-up area for imaging the conveyor receiving area, etc., whereby imaging data of a designated area is obtained. Can be obtained. Then, the robot system 100 can perform image processing (visual field processing) by the computer on the imaged data in order to identify the various operation targets 112 located in the designated area via the processor 202, for example.

  Further, the robot system 100 selects the operation target 112 for executing the work 402 from among the recognized operation targets 112, for example, a predetermined selection criterion and / or a selection rule and / or a pose or an outline. Imaging data can be further processed to determine a start position 114 and / or an initial pose for the selected and manipulated object 112).

  The robot system 100 may identify a drop area and another predetermined area (more specifically, for example, a sorted pallet or bin, and / or the like) in order to identify and / or identify the work position 116 and the holding position 118. Alternatively, it may include another scanner 416 (one example of imaging device 222) that is directed to image areas (such as areas designated with respect to the area on the feed side of the belt conveyor), whereby It is possible to acquire image pickup data of a different area. Then, the robot system 100 uses the computer of the imaging data to identify the work position 116 for placing the operation target 112, the holding position 118, and / or the pose of the operation target 112 via the processor 202, for example. Image processing (visual field processing) can be performed. Further, the robot system 100 determines the work position 116 and the holding position 118 (based on the imaging result or the imaging result based on a predetermined standard or rule for loading and / or arranging the plurality of operation targets 112). Can be identified and selected.

  Here, the scanner 416 is configured to scan a mark on the surface of the manipulator 112 that is adjacent to it (eg, at a height corresponding to the height of the corresponding scanner (s)) and is vertically oriented. It can be arranged horizontally. The scanner 416 may also be vertically oriented so as to scan the marks above / below it and on the surface of the operation target 112 facing horizontally. Further, the scanners 416 may be arranged opposite to each other so that both sides of the operation target 112 placed between the scanners 416 can be scanned.

  The robot system 100 also positions the operation target 112 at the providing position and / or scans one or more surfaces / portions of the operation target 112 with the scanner 416 according to the position and / or scanning direction of the scanner 416. As described above, the operation target 112 can be operated. Further, the robot system 100 includes an imaging device 222 configured to measure the height position of the bottom surface 324 of the operation target 112, which is scanned by the scanner 416 and whose support position is known, for example. (See FIG. 4B).

  Using the start position 114, the pick-up position 118, and / or the work position 116 thus identified, the robot system 100 performs one or more structural members (eg, robots) of each unit to perform the work 402. The arm 414 and / or the end effector) can be manipulated. Accordingly, the robot system 100 can create or obtain a control sequence corresponding to one or more actions performed by the corresponding unit to perform the task 402, eg, via the processor 202.

  For example, the control sequence for the transfer unit 104 includes placing the end effector in an approach position (eg, a position / place for placing the end effector for gripping the manipulation target 112), gripping the manipulation target 112, Lifting the operation target 112, moving the operation target 112 from above the start position 114 to a provision position / pause for scan operation, holding the operation target 112 at the holding position 118 (changing the gripping position), and operating Including moving the object 112 from the start position 114, optionally via the holding position 118, to the work position 116, lowering the operation target 112, and releasing the grip of the operation target 112. be able to.

  The robot system 100 may also create or obtain a control sequence by determining a sequence of commands and / or settings for one or more actuating devices 212 that operate the robot arm 414 and / or the end effector. it can. In this case, the robot system 100 uses, for example, the processor 202 to place the end effector at the approach position around the start position 114, grip the operation target 112 with the end effector, and move the end effector around the scan position or the holding position 118. Of the actuation device 212 for manipulating the end effector and the robot arm 414 to position the end effector around the working position 116 and release the actuated object 112 from the end effector. Settings can be calculated. This allows the robot system 100 to perform an operation to complete the task 402 by operating the actuating device 212 according to the determined control sequence of commands and / or settings.

  In addition, the robot system 100 can create or acquire a control sequence based on a confidence criterion regarding the pose of the operation target 112. In this case, the robot system 100 may place the end effector in various positions for pickup, such as to grip or cover different surfaces, or in various provided positions / poses with respect to the manipulator 112, eg, according to a confidence criterion for poses. Can be calculated or a combination thereof can be performed.

  As an example, the operation target 112 is the operation target 302 placed in the first pose 312 of FIG. 3A (where the top surface 322 of the operation target 302 is generally facing upwards and exposed). If the confidence criterion for the pose is high (ie, the degree of certainty exceeds the sufficiency threshold and the determined pose is likely to be accurate), then the robotic system 100 determines that the first approach position 432 and the first A first control sequence 422 can be created or obtained that includes the serving location 442. At this time, for example, there is sufficient certainty that the top surface 322 of the operation target 302 faces upward (that is, the bottom surface 324 having the target identifier 332 of FIG. 3C faces downward). The robot system 100 may calculate a first control sequence 422 that includes a first approach position 432 for placing the end effector directly on the top surface 322 of the manipulator 302.

  As a result, the robot system 100 can grip the operation target 112 with the end effector that contacts / covers the top surface 322 of the operation target 302 so that the bottom surface 324 of the operation target 302 is exposed. The robot system 100 also includes a first control sequence 422 that includes a first providing position 442 for the operation target 112 to be directly above the upward facing scanner 416 that scans the identifier 332 located on the bottom surface 324. Can be calculated.

  On the other hand, if the confidence criterion for the pose is low (ie, the degree of certainty is less than the adequacy threshold and the determined pose is unlikely to be accurate), then the robot system 100 determines that the second approach position 434 and A second control sequence 424 (ie, different from the first control sequence 422) can be created or obtained that includes one or more second provision locations 444. At this time, the robot system 100 measures, for example, the dimensions of the operation target 112 and compares them with the master data 252, and the operation target 302 is in the first pose 312 of FIG. 3A or the third pose 316 of FIG. 3C. Can be determined (eg, if the level of certainty of the measurement exceeds a predetermined threshold).

  However, the robot system 100 may have difficulty in capturing / processing the mark printed on the surface of the operation target 112, and as a result, the confidence criterion regarding the determined pose may be less than the sufficiency threshold. In other words, in the robot system 100, the exposed surface facing upward of the operation target 302 is its top surface 322 (eg, the first pose 312) or its bottom surface 324 (eg, the third pose 316). ) May not be fully convinced.

  In this case, the robot system 100 may be adjacent to one of the outer peripheral surfaces 326 of the operation target 302 of FIG. And / or a second control sequence 424 may be calculated that includes a second approach position 434 for positioning the end effector aligned and / or facing in a direction parallel to the bottom surface 324. ..

  As a result, the robot system 100 grips the operation target 112 with an end effector that contacts / covers one of the outer peripheral surfaces 326 of the operation target 302 and exposes both the top surface 322 and the bottom surface 324 of the operation target 302. can do. The robot system 100 may also provide or place the top surface 322 and the bottom surface 324 of the operation target 302 in front of the scanner 416 (eg, in and / or facing the scan field) simultaneously or sequentially. be able to. When the manipulator 112 is in the scan position, the robot system 100 uses the scanner 416 (eg, at least the scanner 416 facing the top surface 322 and the bottom surface 324 of the manipulator 302) to simultaneously scan the provided surface. It is possible to scan continuously and / or obtain the identifier 332 (s) of the operation target 302 on it.

  The second control sequence 424 also positions the initially downwardly facing surface (the bottom surface 324 of the operation subject 302) horizontally and directly above the upwardly facing scanner 416, and / or , Including second providing position (s) 444 for placing the initially upwardly facing surface (top surface 322 to be manipulated) vertically and just in front of the horizontally oriented scanner 416. There is. The second control sequence 424 may include a reorientation / rotation action (eg, as indicated by the dashed hollow circle) to provide the two delivery positions / pauses, which Scans both top surface 322 and bottom surface 324 using orthogonally oriented scanner 416. Further, the robot system 100 may, for example, continuously provide and scan the top surface 322 of the manipulator 302 to an upward facing scanner, and then rotate the manipulator 302 90 degrees to reveal its bottom surface 324. , Can be provided to a horizontally oriented scanner 416 for scanning. The reorientation / rotation action can then be conditional, such that if the reading of the identifier 332 of the operation subject 302 fails, the robot system 100 will execute the corresponding command.

  Alternatively, as an example, the robot system 100 may create or control sequences (not shown) for gripping / covering one of the outer peripheral surfaces 326 along the width of the manipulator 302 when confidence criteria are low. Can be obtained. In this case, the robot system 100 moves the operation target 302 between a pair of horizontally opposed scanners 416 and provides an outer peripheral surface 326 of the operation target 302 along its length, for example, as shown in FIG. 3A. As shown, the identifier 332 on one of the peripheral surfaces 326 can be scanned. Note that details regarding the control sequence based on the confidence criterion will be described later with reference to FIGS. 5A and 5B described later.

  In addition, the robot system 100 is placed at the work position 116 and the two-dimensional or three-dimensional shape of the operation target 112 (hereinafter, referred to as the “operation target 112” instead of the “operation target 302”) held by the end effector. The control sequence can be reacquired based on the information about the operation target 112 in the storage container 450 (for example, a large box, a bucket, etc.).

  As an example, the robot system 100 grasps the size of the operation target 112 in both the first control sequence and the second control sequence described above. Further, since the other operation targets 112 already stored in the storage container 450 placed at the work position 116 and their dimensions are also known, the robot system 100 uses the empty space in the storage container 450. You can ask for information. The robot system 100 can calculate the spatial shape parameter of the operation target 112 when various poses of the operation target 112 grasped by the end effector are changed two-dimensionally or three-dimensionally. Therefore, by comparing those spatial shape parameters with the spatial information in the storage container 450, the pattern or plan capable of storing the operation target 112 at a higher packing density in the storage container 450 is optimized. Can be selected.

  In this case, the robot system 100 can consider the presence or absence of interference between the end effector and the storage container 450 or the operation target 112 already stored when the end effector accesses the storage container 450. The filling rate of the operation target 112 into the storage container 450 is higher when the pose of the operation target 112 is changed than when the operation target 112 currently held is stored in the storage container 450 in the same orientation. At times, the robot system 100 can create or acquire a control sequence that includes an operation of holding the operation target 112 in a pose optimized for storage.

  FIG. 4B is a front view illustrating an exemplary operation 404 performed by the robot system 100 according to one embodiment of the present disclosure. In this example, a plurality of operation targets 112 are mixedly mounted on a pallet 464, and the pallet 464 is mounted on a self-propelled trolley 462 such as an AGV (Automated Guided Vehicle), and the start position 114 is set. It is transported to the pickup area including. Although FIG. 4B shows a state in which a plurality of operation objects 112 having the same shape are orderly mixed, a plurality of operation objects 112 having different sizes and shapes are randomly placed on the pallet 464 depending on the actual unloading situation. Note that it is often loaded.

  The pickup area to which the pallet 464 has been conveyed is imaged by the scanner 412, and the operation target 112 is selected in the same manner as described with reference to FIG. 4A. The selected operation target 112 is gripped on the top surface 322 of the operation target 112 in this example by the end effector installed at the tip of the robot arm 414 of the transfer unit 104, and is scanned by the scanner 416 to acquire the identifier 332. To be done. The robot system 100 can grasp the information including the dimension of the operation target 112 by comparing the information of the identifier 332 of the operation target 112 with the master data 252, for example.

  On the other hand, even the operation targets 112 having the same identifier 332 may actually have different dimensions (especially height). Therefore, the robot system 100 uses the distance measuring device 466 (an example of the imaging device 222) installed on or near the floor surface of the work space to scan the bottom of the operation target 112 while scanning the operation target 112, for example. The distance to surface 324 is measured. At this time, when the movement of the operation target 112 is suspended during scanning, the distance to the bottom surface 324 of the operation target 112 can be measured during the suspension. In FIG. 4B, the operation target 112 is shown to be measured by the distance measuring device 466 immediately after being unloaded (depalletized) from the pallet 464, but the timing of the measurement is higher than that of the holding position 118 in the control sequence. There is no particular limitation as long as it is performed at the upstream position.

  In this example, the robot system 100 can grasp the height position (grasping level) of the top surface 322 of the operation target 112 at the time of measurement by a control sequence or an appropriate position measurement. Therefore, the height 112h of the operation target 112 can be obtained by acquiring the measurement value of the distance to the bottom surface 324 of the operation target 112. That is, the robot system 100 receives the measurement data of the bottom surface 324 of the operation target 112 by the distance measuring device 466, and from the received measurement data and the height position (grasping level) of the top surface 322 of the operation target 112, The height 112h can be calculated. When the height 112h is different from the value stored as the master data 252 of the operation target 112, the robot system 100 can replace the master data 252 or add it to the master data 252 to update it.

  After the actual size of the operation target 112 is known in this way, the robot system 100 can calculate the space shape parameter of the pose when the operation target 112 is gripped from various directions. Then, the robot system 100 compares the spatial shape parameters with the spatial information in the storage container 450 placed at the work position 116, so that the operation target 112 is stored in the storage container 450 at a higher packing density. It is possible to optimize and select a plan or a pattern that can store the data.

  At this time, when the end effector accesses the storage container 450, the robot system 100 calculates the presence / absence of interference between the end effector and the storage container 450 or the operation target 112 that has already been stored. Can exclude that pattern. The filling rate of the operation target 112 in the storage container 450 is higher when the pose of the operation target 112 is changed than when the operation target 112 currently held is stored in the storage container 450 in the same orientation. At times, the robot system 100 changes the control sequence 472 (corresponding to the first control sequence 422 or the second control sequence 424 in FIGS. 4A and 4B) up to that point, and the pose is optimized for storage. It is possible to recreate a control sequence including an operation of changing the operation target 112 so that

  On the contrary, when the pose of the operation target 112 currently gripped is optimal from the viewpoint of storage efficiency, the robot system 100 does not change the control sequence 472 and moves the gripped operation target 112 to the work position 116. It is stored in a storage container 450 such as a bucket placed on the storage conveyor or the like of the transportation unit 106 in FIG.

  Further, when changing the operation target 112, the robot system 100 operates the operation target 112 according to the control sequence 474 after the recalculation. For example, the operation target 112 after scanning is moved to the area around the holding position 118, the operation target 112 is turned into a pose for temporary placement with the end effector facing in a predetermined direction, and the operation target 112 is placed on the temporary placement table 468 in that state. Then, the grip is released. The temporary stand is not particularly limited, and for example, the operation target 112 can be placed so that at least two surfaces thereof are exposed, and in particular, from the viewpoint of ease of gripping and stability at the time of gripping, the operation target 112 can be operated. A preferable example is a pedestal capable of supporting the object 112 while holding it in an inclined state. The robot system 100 can change the orientation of the end effector, grasp a surface different from the surface grasped before the temporary placement, and change the operation target 112.

  The robot system 100 stores the re-held operation target 112 in a storage container 450 such as a bucket placed on the storage conveyor or the like of the transportation unit 106 at the work position 116. At this time, instead of directly positioning the end effector at one time, it may be operated so as to swing forward / backward / left / right / vertically with respect to the target position. Further, a plurality of end effectors or a plurality of end effectors may be provided, and control may be performed such that each end effector is selectively used in relation to the size of the operation target 112.

  It should be noted that, in the above, in order to execute the operation related to the work 402, the robot system 100 has a current position of the operation target 112 (for example, a set of coordinates corresponding to a grid used by the robot system 100), and / or You can track your current pose. For example, the robot system 100 can track the current position / pose according to the data from the position sensor 224 of FIG. 2, eg, via the processor 202. Robot system 100 can position one or more portions (eg, links or joints) of robot arm 414 according to data from position sensor 224. The robot system 100 may further calculate the position / pose of the end effector based on the position and orientation of the robot arm 414, thereby calculating the current position of the manipulator 112 held by the end effector. it can. The robot system 100 also processes other sensor readings (eg, force readings or accelerometer readings), executed actuation commands / settings, and / or associated timing according to a speculation mechanism. Or, the current position can be tracked based on a combination thereof.

[Operation flow (control sequence based on trust criteria)]
FIG. 5A is a flowchart of a method 500 illustrating an example of a procedure in an operation of the robot system 100 according to an embodiment of the present disclosure. Method 500 includes procedures for obtaining / calculating and performing a confidence criteria-based control sequence to perform task 402 of FIG. 4A in accordance with confidence criteria associated with determining an initial pose of manipulator 112. The method 500 can also be implemented based on executing instructions stored on one or more storage devices 204 on one or more processors 202.

  At block 501, the robot system 100 can identify a scan field of one or more imaging devices 222 of FIG. For example, the robot system 100 can identify a space that can be scanned by one or more imaging devices 222, such as the scanners 412 and 416 of FIGS. 4A and 4B, via one or more processors 202, for example. The robot system 100 can identify scan fields oriented in opposite directions and / or orthogonal directions according to the orientation of the scanner 416. As shown in FIGS. 4A and 4B, the scanners 416 can be positioned on opposite sides and / or facing each other, such as horizontally on both sides or vertically on both sides. The scanners 416 can also be arranged vertically with one facing up or down and another facing horizontally.

  The robot system 100 can identify the scan field according to the master data 252, for example. The master data 252 can include grid positions, coordinates, and / or other markers that indicate the imaging device 222 and / or the corresponding scan fields. The master data 252 is pre-determined according to the layout and / or physical placement of the imaging device 222, the capabilities of the imaging device 222, environmental factors (eg, light conditions, and / or obstructions / structures), or combinations thereof. can do. The robot system 100 can also perform a calibration process to identify the scan field. For example, the robotic system 100 can use the transfer unit 104 to place a known mark or code in a set of locations and determine whether the corresponding imaging device 222 has accurately scanned the known mark. The robot system 100 can identify the scan field based on the position of the known mark that resulted in the accurate scan result.

  At block 502, the robotic system 100 can scan a designated area. The robot system 100 uses one or more imaging devices 222 (eg, the scanner 412 of FIGS. 4A and 4B and / or other area scanners), eg, via commands / prompts sent by the processor 202. , Imaging data (eg, acquired digital images and / or point clouds) of one or more designated areas, such as pickup areas and / or drop areas can be generated. Imaging data can be communicated from the imaging device 222 to one or more processors 202. Accordingly, the one or more processors 202 may further include a pickup area (e.g., including the operation target 112 prior to performing the work), a holding area, and / or a drop area (e.g., after performing the work) for further processing. It is possible to receive imaging data indicating the operation target 112).

  At block 504, the robot system 100 poses the manipulator 112 and associated positions (eg, the start position 114 of FIG. 1 and / or the work position 116 of FIG. 1) and / or the initial pose of the manipulator 112. Can be identified. Via the robot system 100, eg, the processor 202, the imaging data is analyzed based on pattern recognition mechanisms and / or recognition rules to identify outlines (eg, peripheral edges and / or surfaces) of the manipulated object 112. Can be analyzed. The robot system 100 may further include an outline and / or a surface of the operation target 112 corresponding to various operation targets 112 based on, for example, a predetermined recognition mechanism, a recognition rule, and / or a template regarding a pose or an outline. Grouping can be identified.

  The robot system 100 may, for example, span the outline of the target 112, pattern in color, brightness, depth / position, and / or combinations thereof (e.g., are they the same value or change at a known rate / pattern? It is possible to identify the outline grouping of the operation target 112 corresponding to (). Further, for example, the robot system 100 can identify the outline and / or the surface grouping of the operation target 112 according to a template of a predetermined shape / pose defined in the master data 252, an image, or a combination thereof.

  From the operation target 112 recognized in the pickup area, the robot system 100 selects one as the operation target 112 (for example, according to a predetermined sequence or rule set and / or an outline template of the operation target). You can The robot system 100 can select the operation target 112, for example, according to a point cloud indicating the distance / position of the scanner 412 with respect to the known position. In addition, the robot system 100 can select an operation target 112 having two or more surfaces located at corners / edges and exposed / shown in an imaging result, for example. Further, the robot system 100 can select the operation target 112 according to a predetermined pattern or sequence (eg, left to right, closest to farthest from the reference position, etc.).

  With respect to the selected control target 112, the robot system 100 can further process the imaging data to determine the start position 114 and / or the initial pose. For example, the robot system 100 maps the position of the operation target 112 (for example, a predetermined reference point regarding the determined pose) in the imaging data to a position in the grid used by the robot system 100, so that the start position 114. Can be determined and the position can be mapped according to a predetermined calibration map.

  The robot system 100 can process the imaging data of the drop area to determine the space between the operation targets 112. The robot system 100 maps the outline of the operation target 112 according to a predetermined calibration map that maps the position of the image to the actual position and / or the coordinates used by the system, and based on that, determines the free space. can do. The robot system 100 can determine a vacant space as a space between outlines (and, by extension, surfaces of the operation targets 112) of the operation targets 112 that belong to different groupings. In addition, the robot system 100 measures one or more dimensions of the vacant space and the measured dimensions are one or more dimensions of the operation target 112 (eg, as stored in master data 252). By comparing with, it is possible to determine an appropriate empty space for the operation target 112. The robot system 100 may also select one of the appropriate / empty spaces according to a predetermined pattern (eg, left to right, closest to farthest, bottom to top, etc. relative to a reference position). It can be selected as the work position 116.

  The robot system 100 can determine the work position 116 without or in addition to processing the imaging data. For example, the robot system 100 can arrange the operation target 112 in the arrangement area according to a predetermined control sequence and position without imaging the area. Also, for example, the robot system 100 may use imaging data to perform multiple tasks (eg, moving multiple operation targets 112, such as those associated with operation targets 112 located in a common layer / column of a stack). Can be processed.

  In block 522, for example, the robot system 100 determines a first pose (eg, estimation of a stopped pose of the operation target 112 in the pickup area) based on processing of the imaging data (eg, imaging data from the scanner 412). be able to. The robot system 100 compares the outline of the operation target 112 with the outline of the template of the predetermined pose of the master data 252 (for example, comparing pixel values) to determine the first pose of the operation target 112. Can be judged. The template for a given pose may include, for example, different possible placements of the outline of the manipulator 112, depending on the corresponding orientation of the manipulator 112 expected. The robot system 100 may include a set of outlines of the selected target 112 and the previously associated target 112 (eg, the first exposed surface 304 of FIGS. 3A and / or 3C and / or the first exposed surface 304 of FIG. 3A). Edges of exposed surfaces, such as two exposed surfaces 306) can be identified. The robot system 100 can determine the initial pose by selecting one of the pose templates that corresponds to the measurement of the least difference between the outlines of the compared targets 112.

  For a further example, the robot system 100 can determine the initial pose of the manipulator 112 based on the physical dimensions of the manipulator 112. The robot system 100 can roughly estimate the physical size of the operation target 112 based on the size of the exposed surface acquired by the imaging data. The robot system 100 measures the length and / or angle for each of the outlines of the manipulated object 112 in the imaging data and then uses a calibration map, a conversion table or process, a predetermined equation, or a combination thereof. The defined length can be mapped or converted to a real world length or a standard length. The robot system 100 can use the measured dimensions to identify the manipulated object 112 and / or exposed surface (s) that correspond to the physical dimensions.

  The robot system 100 compares the estimated physical dimensions to a set of known dimensions (eg, height, length, and / or width) of the manipulated object 112 in the master data 252 and its surface, The manipulation target 112 and / or the exposed surface (s) can be identified. The robot system 100 can use the set of matched dimensions to identify exposed surface (s) and corresponding poses. For example, the robot system 100 may define the exposed surface as the top surface 322 of the manipulator 302 of FIG. 3A, or of FIG. 3B, if the dimensions of the exposed surface match the expected length and width for the manipulator 112. It can be identified as the bottom surface 324 of the operation target 302 (eg, a pair of opposite surfaces). Based on the orientation of the exposed surface, the robot system 100 determines a first pose of the operation target 112 (eg, the first pose 312 or the third pose 316 of the operation target 302 when the exposed surface is facing up). can do.

  For example, the robot system 100 can determine a first pose of the manipulation target 112 based on a visible image of one or more surfaces of the manipulation target 112 and / or one or more markings thereof. The robot system 100 can compare the pixel values of the set of connected outlines with a template of a given marking-based pose of the master data 252. The marking-based pose template may include, for example, one or more unique markings of the expected manipulator 112 in a variety of different orientations. The robot system 100 determines the initial pose of the manipulator 112 by selecting one of the surface, the surface orientation, and / or the corresponding pose that results in the least difference measurement for the compared images. can do.

  At block 524, the robot system 100 may calculate a confidence metric for the initial pose of the manipulator 112. The robot system 100 can calculate confidence criteria as part of determining the initial pose. For example, the confidence criterion may correspond to a difference criterion between the outline of the operation target 112 and the outline of the selected template described above. Also, for example, a confidence metric may correspond to a tolerance level associated with the approximated physical dimensions and / or angles described above. Also, for example, the confidence criterion may correspond to the criterion of the difference between the visible marking in the imaging data and the image of the template described above.

  At block 506, the robot system 100 may perform a control sequence (eg, the first control sequence 422 of FIG. 4A, the second control sequence 424 of FIG. 4A, the control sequence of FIG. 4B) for performing the task 402 on the operation target 112. 472), and the control sequence 474 including the holding operation of the operation target 112 shown in FIG. 4B can be calculated.

  For example, the robot system 100 creates a control sequence based on calculating a sequence of commands or settings for the actuating device 212 that operates the robot arm 414 and / or the end effector of FIGS. 4A and 4B, or a combination thereof. Or you can get it. For some tasks, the robot system 100 steers the robot arm 414 and / or the end effector to move the operation target 112 from the start position 114 to the work position 116, optionally via the hold position 118. Control sequences and setpoints can be calculated. The robot system 100 implements a control sequence mechanism (eg, process, function, equation, algorithm, computer generated / readable model, or combination thereof) configured to calculate a path of travel in space. can do.

  For example, the robot system 100 may move the operation target 112 from the start position 114 to the work position 116, optionally via the repositioning position 118, to one or more provided poses / positions (eg, 1 for end effectors). An A * algorithm, a D * algorithm, and / or other grid-based searches may be used to compute a path of travel through space to move through one or more corresponding scan positions). The control sequence mechanism may use further processing, functions, or equations, and / or mapping tables to translate the travel path into a sequence of commands or settings for the actuating device 212, or a combination thereof. When using the control sequence mechanism, the robot system 100 can calculate a control sequence that steers the robot arm 414 and / or the end effector to cause the operation target 112 to follow the calculated movement path.

  The robot system 100 can selectively create or acquire a control sequence based on a confidence criterion. The robot system 100 may use one or more scan positions (eg, the first approach position 432 of FIG. 4A and / or the second approach position 434 of FIG. 4A), one or more scan positions (eg, the first approach position 434 of FIG. 4A) according to a confidence criterion. A control sequence can be calculated that includes one serving position 442 and / or the second serving position 444 of FIG. 4, or a combination thereof. For example, the robot system 100 calculates an approach position and / or a scan position for a metric (eg, performance metric and / or scan metric) based on a result of comparing the confidence criterion to a sufficiency threshold. be able to. The scan position is one or more surfaces of the manipulation target 112 in front of (ie, its scan field) one or more corresponding target scanners 416 that scan one or more identifiers 332 of the manipulation target 112. As provided, the end effector may be for placement.

  At block 532, the robot system 100 may calculate a set of available approach positions, eg, via the processor 202. The available approach positions may correspond to a space that is open or unoccupied around the start position 114 sufficient to position the end effector. The robot system 100 can also place the end effector at the selected approach position to contact and grip the operation target 112 without interfering with other operation targets 112.

  For example, the robot system 100 can calculate the set of available approach positions by calculating the separation distance between the outline of the operation target 112 and the outline of the adjacent operation target 112. The robot system 100 can compare the separation distance to a predetermined set of distances that correspond to the physical size / shape of the end effector and / or their various orientations. The robot system can identify each of the available approach positions if the corresponding separation distance exceeds a predetermined set of distances corresponding to the size of the end effector.

  At decision block 534, the robot system 100 can compare the confidence criteria to one or more sufficiency thresholds to determine if they are met. If the confidence criterion meets the sufficiency threshold, such as that shown at block 536 (eg, if the confidence criterion exceeds the required sufficiency threshold), the robotic system 100 may base the control sequence on the performance metric (eg, , A first control sequence 422) can be calculated. If the confidence criterion meets the sufficiency threshold, the robot system 100 deduces that the first pose is suitable and may involve scanning at least one identifier 332 of the manipulator 112 and / or the first pose. The control sequence can be calculated without considering the metric of the scan, which corresponds to the possibility that may be inaccurate.

  As an example, robot system 100 may calculate candidate plans at block 542. Each of the candidate plans may be an example of a control sequence corresponding to a unique combination of available approach positions and scan positions (eg, corresponding serving position / orientation for the target 112). The robot system 100 can calculate the position 334 of the identifier 332 according to the initial pose, such as by rotating the position (s) 334 of the identifier 332 or the corresponding model / pose in the master data 252. The robotic system 100 can eliminate available approach positions that cause the end effector to cover the position 334 of the identifier 332 (eg, directly above, in front of, and / or within a threshold distance).

  Robot system 100 may calculate a candidate plan for each of the remaining available approach positions in the set (eg, the results of the calculations in block 532). For each of the candidate plans, the robot system 100 can further calculate the unique scan position according to the available approach positions. The robot system 100 can calculate the scan position based on the rotation and / or movement of the model of the operation target 112 so that the surface corresponding to the position 334 of the identifier 332 is within the scan field and corresponding. Facing scanner 416. The robot system 100 can rotate and / or move the model according to predetermined processes, equations, functions, etc.

  At block 544, the robot system 100 may calculate performance metrics for each candidate plan. The robot system 100 can calculate a performance metric corresponding to the throughput (rate) for completing the task 402. For example, the performance metric may be the distance traveled by the operation target 112, the estimated travel time, the number of command and / or setting changes relating to the actuating device 212, the achievement rate (ie, the amount of piece loss and the complement of the piece loss complementary to the candidate plan. Target) or a combination thereof. The robot system 100 includes one or more measured or known data (e.g., acceleration / velocity associated with settings / commands and / or percentage of piece loss associated with gripping surfaces and / or movements) and a predetermined amount. Can be used to calculate the corresponding values for the candidate control sequences.

  At block 546, the robot system 100 may select a candidate plan with the metric of maximum performance (ie, with a corresponding approach position) as the control sequence. For example, the robot system 100 may use the highest success rate, the shortest movement distance, the smallest number of command and / or setting changes, the fastest movement duration, or the shortest movement distance in the set of candidate plans as the control sequence. Candidate plans corresponding to those combinations can be selected. Therefore, the robot system 100 can select an available approach position in the set corresponding to the highest performance metric as the approach position.

  By comparison, the robot system 100 may calculate candidate plans according to different criteria if the confidence criteria do not meet the sufficiency threshold (eg, the confidence criteria is less than the required sufficiency threshold). As indicated by block 538, the robot system 100 may calculate a control sequence (eg, the second control sequence 424) based on the scan metric. The metric of the scan corresponds to the possibility that at least one of the identifiers 332 of the operated object 112 remains uncovered by the end effector and is scannable, regardless of whether the initial pose is correct. A value (eg, a two-element value or a score / percentage that is not two-element).

  For example, the robotic system 100 may prioritize (eg, meet first and / or give more weight) scan metrics over performance metrics if the confidence criteria do not meet the sufficiency threshold. Accordingly, the robotic system 100 is in front of the one or more scanners 416 one or more scan positions (ie, within the scan field) to provide at least one uncovered identifier 112 of the operation target 112. And / or suitable control sequences (including for a corresponding scanner) can be calculated.

  FIG. 5B is a flow diagram illustrating an example of a procedure in the operation of the robot system according to the embodiment of the present disclosure, which determines a control sequence (for example, one or more positions for an end effector) based on a scan metric. FIG. 6 shows a flow diagram 538 for selectively calculating.

  In this example, calculating the control sequence based on the scan metric may include calculating a set of exposed identifier 332 positions, as indicated at block 552. The robot system 100 may calculate a set of exposed identifier 332 positions (eg, a position 334 of the identifier 332 that may remain scannable by the end effector in the gripping position) for the initial pose of the manipulator 112. You can The robot system 100 can calculate the position 334 of the exposed identifier 332 for each of the available approach positions. The position 334 of the exposed identifier 332 shall correspond to the position 334 of the identifier 332 of the operation target 112 that remains uncovered by the end effector in the corresponding approach position, according to the assumption that the initial pose is correct. be able to.

  As described above with respect to block 542, the master data 252 may include a computer model or template (eg, edges of one or more operating targets 112 and / or templates) that describe the position 334 of the identifier 332 for each of the expected operating targets 112. Measuring the offset with respect to the image). The robot system 100 can calculate a set of exposed identifier 332 positions based on rotating and / or moving a given model / template in the master data 252 to match the initial pose. The robot system 100 can cause the end effector to cover the position 334 of the identifier 332 (eg, directly above, in front of, and / or within a threshold distance) and remove the approach position. In other words, the robot system 100 can eliminate available approach positions directly above, in front of, and / or within a threshold distance of the position 334 of the identifier 332.

  At block 554, the robot system 100 may calculate a set of positions 334 of alternative identifiers 332. The robot system 100 can calculate a set of positions 334 of alternative identifiers 332 for the alternative poses of the initial pose. For each of the available approach positions, the robot system 100 can calculate an alternative pose, and for each of the alternative poses, the robot system 100 can calculate the position of the alternative identifier 332. Therefore, the position of the alternative identifier 332 corresponds to the position 334 of the identifier 332 of the operation target 112 that remains uncovered by the end effector in the corresponding approach position, according to the assumption that the initial pose is not correct. Can be As described above with respect to the position 334 of the exposed identifier 332, the robot system 100 may rotate and / or move a predetermined model / template in the master data 252 according to the alternative pose. The position 334 of 332 can be calculated.

  At block 556, the robot system 100 may calculate the exposure potential for each approach position, each alternative pose, each identifier 332 of the manipulator 112, or a combination thereof. The possibility of exposure is such that the identifiers of one or more operation targets 112 may remain exposed and remain scannable from the corresponding approach position with the end effector gripping the operation target 112. Shows. Possibility of exposure can indicate both scenarios where the initial pose is correct and scenarios where the initial pose is incorrect. In other words, the exposure probability indicates that the identifiers of one or more targets 112 may remain exposed and scannable, even if the initial pose is not accurate. You can

  For example, the robotic system 100 may provide certainty about a condition, such as a probabilistic value corresponding to a particular condition (eg, approach position, alternative pose, identifier of the target 112, or a combination thereof). , The possibility of exposure can be calculated. The robot system 100 may (eg, add and / or multiply) the certainty of a condition with the certainty / likelihood (eg, close to a confidence criterion) that a particular condition is true. ), The likelihood of exposure can be calculated. The robotic system 100 is based on adding a certainty for each of the identifiers that it considers to be exposed when multiple identifiers are exposed with respect to the considered approach position and / or the considered pose. Possibility can be calculated.

  The robotic system 100 determines the likelihood of exposure based on combining certainty values based on the position of the exposed identifier and the position of the alternative identifier, such as each of the potential poses for the considered approach position. Can be calculated. For example, the robot system 100 may use certainty regarding the location of the exposed identifier and the location of the alternative identifier with opposite signs (eg, positive and negative) to calculate the likelihood of exposure. it can. The robot system 100 can calculate the likelihood of exposure based on adding two confidence magnitudes and / or adding the confidence with a sign. The overall size may indicate the overall likelihood that one or more identifiers 332 of the manipulated object 112 remain scannable, and the signed / vector likelihood is 1 of the manipulated object 112. One or more identifiers may indicate the possibility of remaining scannable, even if the initial pose is not accurate. Therefore, the approach position is ideal when the overall size is larger, such as to show a similar possibility that the identifier 332 of the operation target 112 can be scanned regardless of the accuracy of the initial pose. And the likelihood of being signed / vector is close to zero.

  At block 558, the robotic system 100 may select an approach position. The robot system 100 may use a set of exposed identifiers 332 (eg, a set of approximate positions of the identifier 332 of the manipulator 112 as an approach position, subject to the assumption that the initial pose is correct) and alternative identifiers. Position 334 of the uncovered identifier 332, both with the set of 332 (eg, one or more sets of approximate positions of the identifier 332 of the target 112, subject to the assumption that the initial pose is not accurate). Available approach positions can be selected to include. In other words, the robot system 100 can select an approach position that leaves at least one identifier 332 of the exposed scannable operation target 112 regardless of the accuracy of the initial pose. The robot system 100 matches the targeted condition such as the approach position having the largest overall size and / or the signed / vector probability closer to zero. And / or an available approach position can be selected that corresponds to the closest possible exposure.

  The robot system 100 may calculate a scan possibility (eg, the identifier 332 of the exposed operation target 112 may be successfully scanned) based on the exposure possibility. For example, the robot system 100 may evaluate the exposure probability to a corresponding evaluation value (eg, percentage of successful scans tracked, physical size, and / or type of identifier 332) associated with the identifier 332 of the corresponding exposed manipulation target 112. ) Can be combined with The robot system 100 can select an available approach position corresponding to the highest scanning possibility as the approach position.

  The robot system 100 compares the set of exposed identifiers 332 with the set of alternative identifiers 332 such that the set of exposed identifiers 332 and the set of alternative identifiers 332 are located on opposite surfaces of the manipulation target 112. (Eg, between the first pose 312 and the third pose 316) may be included. Therefore, the robot system 100 can select an available approach position corresponding to a third surface (eg, one of the outer peripheral surfaces 326 of the operation target 302) orthogonal to the two opposing surfaces.

  At block 560, the robot system 100 may create or obtain a candidate control sequence based on the selected approach position, such as if the confidence criterion does not meet the adequacy threshold. The robot system 100 places one or more identifiers 332 of the manipulated object 112 on both the set of exposed identifiers 332 and the set of alternative identifiers 332 with respect to one or more end positions corresponding to one or more provided positions / orientations. Candidate control sequences including scan positions can be calculated. In other words, the robot system 100 can calculate a candidate control sequence that can scan the operation target 112 regardless of the accuracy of the initial pose.

  The robot system 100 can create or obtain candidate control sequences that address the location 334 of the identifier 332 in both the exposed set of identifiers 332 and the alternative set of identifiers 332. For example, the robot system 100 can calculate candidate control sequences that address possible identifier 332 locations on opposing and / or orthogonal surfaces. Therefore, in addition to the initial pose, the robot system 100 may perform a reverse pose (for example, a reverse pose in which the outline of the operation target 112 is located at the same position from the view position / angle) and / or a rotation. Can deal with other poses that have been made. Referring back to FIGS. 3A and 3C as an example of the description, the robot system 100 has a first pose 312 and a third pose 316 when the grip position corresponds to one of the outer peripheral surfaces 326 of the operation target 302. Candidate control sequences can be calculated that address both.

  To address multiple possible poses (eg, estimate the error in the first pose), the robot system 100 sets the identifier 332 of the manipulator 112 to a set of exposed identifiers 332 and an alternative identifier 332. You can calculate the scan pose to put on both the set and. As indicated by block 562, the robotic system 100 may calculate a set of candidate poses for the manipulator 112 within or through the scan field. Once the approach position is selected, the robot system 100 may rotate and / or move the model of the position 334 of the identifier 332 to position the position 334 of the identifier 332 within the scan field, etc., as described above with respect to block 542. As described above, the candidate scan position can be calculated.

  At block 564, the robot system 100 may map the exposed set of identifiers 332 and the alternative set of identifiers 332 to each of the candidate scan positions. The robot system 100 can map the exposed set of identifiers 332 based on rotating the model at the position 335 of the identifier 332 starting from the first pose. The robot system 100 may map a set of alternative identifiers 332 based on rotating the model at position 334 of identifiers 332 starting from one of the alternative poses (eg, the reverse pose). it can.

  Once the location 334 of the identifier 332 has been mapped, at block 568, the robot system 100 causes the location 334 and / or the location 334 of the identifier 332 of the operation target 112 in both the set of exposed identifiers 332 and the set of alternative identifiers 332. Alternatively, the orientation can be compared to the scan field. At decision block 570, the robot system 100 determines whether the candidate pose simultaneously provided the scanner with the identifier 332 of the manipulated object 112 in both the set of exposed identifiers 332 and the set of alternative identifiers 332. be able to.

  At block 572, the robot system 100 scans for candidate poses that simultaneously provide the identifiers 332 of the target 112 in both the set of exposed identifiers 332 and the set of alternative identifiers 332 to different scanner / scan fields. Can be identified as a pose. For example, when the position of the operation target 112 in the set of exposed identifiers 332 of the operation target 112 and the set of alternative identifiers 332 is on the opposite surface, the grip position is one of the outer peripheral surfaces 326 of the operation target 112. In the scan pose, the robot system 100 places the operation target 112 between a pair of facing / facing scanners, with the surfaces on both sides of the operation target 112 facing one of the scanners. Can be identified.

  At block 574, if none of the candidate poses provide the identifier 332 of the manipulated object 112 at the same time in both the set of identifiers 332 of the exposed manipulated object 112 and the set of alternative identifiers 332, the robot system 100 determines. , A plurality of scan positions (eg, a first scan position and a second scan position) each of which provides at least one identifier 332 of the manipulation target 112 from the set of identifiers 332 of the manipulation target 112 and the set of alternative identifiers 332. Can be calculated). For example, the first scan position may provide one of the target scanners with a position 334 of one or more identifiers 332 in the set of identifiers 332 of the exposed manipulation target 112, and a second scan position may be , The position 334 of one or more identifiers 332 within the set of identifiers 332 of the alternative operation target 112 may be provided to one of the scanners. The second scan position may be associated with rotating the end effector about an axis, translating the end effector, or a combination thereof from the first scan position.

  Referring again to the example shown in FIGS. 4A and 4B, the second control sequence 424 is orthogonal to two opposing surfaces (eg, for the first pose 312 and the third pose 316), as described above. It may correspond to the second approach position 434, which corresponds to the third surface (eg, one of the outer peripheral surfaces 326 of the operation target 112). Thus, the first scan position is a scanner 416 with its surface (e.g., estimated to be the bottom surface 324 of the manipulator 112) corresponding to the first pose (e.g., the first pose 312) facing up. Corresponding to a first position of one of the second providing positions 444 above and facing the scanner. The second scanning position is the second providing position 444 obtained by rotating the operation target 112 by 90 degrees, for example, in the counterclockwise direction with respect to the general movement direction from the start position 114 to the working position 116. It may correspond to one second position. Thus, the second scan position is a scanner with the surface (eg, determined to be the bottom surface 324 of the operation target 112) corresponding to the alternative pose (eg, the third pose 316) oriented horizontally. It may correspond to a second providing position 444, which is arranged in front of 416 and in a vertical orientation towards this scanner 416.

  According to the resulting set of scan poses and / or scan positions, the robot system 100 can create or obtain candidate control sequences. The robot system 100 uses one or more of the mechanisms described above (eg, the A * mechanism) to position the end effector at a selected approach position and contact and grip the manipulator 112 accordingly and also manipulate. Candidate plans for lifting and moving the subject 112 to a set of identified scan poses and / or scan positions can be calculated. For example, once a scan pose is identified, the robot system 100 can calculate candidate plans and establish a scan pose for the manipulator 112 within or through the scan field. If the robot system 100 does not identify a scan pose, the robot system 100 can calculate a candidate plan and move / orient the end effector to sequentially pass through a set of scan positions, Thus, the operation target 112 is continuously moved / rotated according to the plurality of providing positions / directions.

  At block 576, the robot system 100 may recreate or update the scan possibilities for each of the candidate control sequences. Robot system 100 may perform various possibilities and / or priorities (eg, approach position, scan position, scanner 416 utilized, suspected exposed identifier 332, associated error, and / or various possibilities and / or priorities described above with respect to block 544. Alternatively, the scan likelihood can be updated with respect to the metric of the scan instead of the performance metric, but based on a combination of probability and / or score), or loss rate, or a combination thereof.

  At block 578, the robotic system 100 may create or obtain a control sequence based on the selection of candidate plans according to the scan likelihood. The robot system 100 can select, as a control sequence, a candidate plan having the highest possibility of scanning among the candidate plans. For example, the robot system 100 may, for example, move one or more scan fields (i.e., one or more scanners 416) during movement of the manipulator 112 for scanning the space between the start position 114 and the work position 116. In the previous), the candidate plan most likely to place at least one of the exposed identifier 332 positions 334 and at least one of the alternative identifier 332 positions 334 can be selected.

  If more than one candidate plan corresponds to a scan probability within a relatively small difference value (eg, a predetermined threshold), the robot system 100 determines a performance metric corresponding to the corresponding candidate plan. It can be calculated and evaluated (eg, as described above with respect to blocks 544 and 546). The robot system 100 can select a candidate plan that is the closest to the targeted condition as the control sequence.

  The robot system 100 may deviate from the exemplary flow shown. For example, the robot system 100 can select the approach position, as described above. Based on the selected approach position, the robot system 100 performs a predetermined set of movements, such as grasping, lifting, reorienting, moving horizontally, returning and releasing, or a combination thereof. can do. During or after the predetermined set of movements, the robotic system 100 re-images or scans the pickup area (eg, via looping back to block 502) to determine the initial pose and confidence criteria (eg, block). 522 and via block 524) can again be determined.

  Returning to FIG. 5A, at block 508, the robot system 100 can begin performing the resulting control sequence. The robot system 100 may send control sequence commands and / or settings to other devices (eg, corresponding actuating devices 212 and / or other processors) to perform one or more tasks 402, 404. A control sequence based on operating the processor 202 may be implemented. Accordingly, the robot system 100 can execute the control sequence by manipulating the actuating device 212 in response to a sequence of commands or settings, or a combination thereof. For example, the robot system 100 can operate the actuation device 212 to position the end effector in an approach position around the start position 114, contact and grip the operation target 112, or a combination thereof.

  At block 582, the robotic system 100 may move the end effector to the scan position, thereby moving the manipulator 112 to the serving position / orientation. For example, the robot system 100 may move the end effector to establish a scan pose for the operation target 112 after or after the operation target 112 is lifted from the start position 114. In addition, the robot system 100 can move the end effector to the first scan position.

  At block 584, the robot system 100 may operate the scanner 416 to scan the operation target 112. For example, one or more processors 202 may send commands to scanner 416 to perform a scan and / or send queries to scanner 416 to receive scan status and / or scanned values. You can If the control sequence includes a scan pose, such as at block 585, the robotic system 100 may perform the control sequence to move the scan pose target 112 across the scan field in a direction orthogonal to the scan field orientation. You can While the manipulator 112 is moved, the scanner 416 can scan multiple surfaces (simultaneously and / or sequentially) for a plurality of possible locations 334 of the identifier 332 of the manipulator 112.

  At decision block 586, the robot system 100 may evaluate the scan result (eg, status and / or scanned value) to determine if the manipulator 112 has been scanned. For example, the robot system 100 can evaluate the scan result after performing the control sequence up to the first scan position. At block 588, the scan result indicates that the scan of the operation target 112 was successful (eg, the status corresponds to detection of a valid code / identifier and / or the scanned value has been identified. (/ Matches the expected operation target 112), the robot system 100 can move the operation target 112 to the work position 116. Based on the successful scan, the robot system 100 can move the operation target 112 directly to the work position 116, ignoring any subsequent scan position (eg, the second scan position).

  If the scan result indicates that the scan of the manipulator 112 was not successful, the robot system 100 may determine at decision block 590 whether the current scan position is the last one in the control sequence. it can. If not the last control sequence, the robot system 100 may move the manipulator 112 to the next serving position / orientation, as indicated by the loop back to block 582.

  If the current scan position is the last one in the control sequence, the robot system 100 may perform one or more remediation tasks, as indicated by block 592. The robot system 100 may stop and / or cancel the control sequence if the scan results for all of the scan positions in the control sequence indicate a scan failure. The robot system 100 can generate error status / messages to inform the operator. The robot system 100 can arrange the operation target 112 in an area (that is, a position different from the start position 114 and the work position 116) specified for the operation target 112 for which the scan has failed.

  Based on whether the work 402, 404 (that is, the scan of the operation target 112 is successful and the work target 112 is placed at the work position 116) has been completed successfully or the improvement work has been performed, the robot system 100 determines the next work / It is possible to move to the operation target 112. The robot system 100 may rescan the designated area, as indicated by the loop back to block 502, and may use the existing imaging data to move to the next area as indicated by the loop back to block 504. The operation target 112 can be selected.

  Scanning the operation target 112 in space (eg, a position between the start position 114 and the work position 116) improves efficiency and speed with respect to performing the works 402, 404. By calculating a control sequence that includes a scan position and that also cooperates with the scanner 416, the robot system 100 performs a task of moving the operation target 112 and a task of scanning the operation target 112. Can be effectively combined with. Moreover, creating or obtaining the control sequence according to the confidence criteria of the initial pose further improves efficiency, speed, and accuracy with respect to scanning operations. As described above, the robot system 100 can create or obtain a control sequence that addresses alternative orientations that correspond to scenarios where the initial pose is not accurate. Therefore, the robot system 100 accurately / successfully scans the operation target 112 even when there is a pose determination error such as a calibration error, an unexpected pose, an unexpected light condition, or the like. The possibility of doing so can be increased. The increased likelihood of accurate scanning can lead to an increase in overall throughput for the robot system 100 and can further reduce operator effort / intervention.

[Summary]
The forms for carrying out the above invention of the embodiments of the present disclosure are not intended to be exclusive or to limit the present disclosure to the explicit forms disclosed above. Although particular embodiments of the present disclosure have been described above for purposes of illustration, various equivalent variations are possible within the scope of the present disclosure, as those skilled in the relevant art will recognize. For example, although processes or blocks are provided in a given order, alternative implementations may implement routines with steps or employ systems with blocks in different orders, and Some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternative or sub-combinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are performed sequentially at the times shown, these processes or blocks may instead be performed or performed in parallel, or at different times. May be carried out. Moreover, any particular numbers provided herein are examples only, and alternative embodiments may employ different values or ranges.

  These and other changes can be made to the present disclosure in light of the above-described modes for carrying out the invention. While the mode for carrying out the invention describes specific embodiments of the present disclosure and the best modes envisioned, the present disclosure describes how detailed the above description is in the text. Regardless, it can be implemented in many ways. The details of the system, which may vary significantly in its particular implementation, are still encompassed by the technology disclosed herein. As mentioned above, a particular term used in describing a particular feature or aspect of this disclosure refers to any particular characteristic, feature, or aspect of this disclosure to which the term is associated. It is not meant to be redefined herein as limiting. Therefore, the present invention is not limited, except by the appended claims. In general, the terminology used in the appended claims discloses the present disclosure in the specification, unless the section of the Detailed Description of the Invention sets out such term explicitly. It should not be construed as limiting to any particular embodiment of.

  While particular aspects of the invention are provided in the accompanying claims in the form of specific claims, Applicants contemplate the various aspects of the invention in the form of any number of the claims. is doing. Accordingly, Applicants reserve the right, in this application or in its continuation application, to carry out further claims after filing this application and to carry out such additional claim forms.

100 ... Robot system, 102 ... Unloading unit, 104 ... Transfer unit, 106 ... Transport unit, 108 ... Loading unit, 112 ... Operation target, 114 ... Start position, 116 ... Working position, 118 ... Holding position, 202 ... Processor , 204 ... Storage device, 206 ... Communication device, 208 ... Input-output device, 210 ... Display, 212 ... Actuating device, 214 ... Transfer motor, 216 ... Sensor, 222 ... Imaging device, 224 ... Position sensor, 226 ... Contact sensor , 252 ... Master data, 254 ... Tracking data, 302 ... Operation target, 304 ... First exposed surface, 306 ... Second exposed surface, 312 ... First pose, 314 ... Second pose, 316 ... Third Pose, 322 ... Top surface, 324 ... Bottom surface, 326 ... Peripheral surface, 332 ... Identifier, 334 ... Identifier position, 402, 404 ... Work, 412, 416 ... Scanner, 414 ... Robot arm, 422 ... First Control sequence, 424 ... Second control sequence, 432 ... First approach position, 434 ... Second approach position, 442 ... First providing position, 444 ... Second providing position, 450 ... Storage container, 464 ... Self-propelled trolley, 464 ... Pallet, 466 ... Distance measuring device, 468 ... Temporary stand, 472, 474 ... Control sequence.

Claims (14)

A method for controlling a robot system including a robot having a robot arm and an end effector, comprising:
Acquiring an approach position at which the end effector grips an operation target,
Acquiring a scan position for scanning the identifier of the operation target;
Creating or acquiring a control sequence based on the approach position and the scan position, and instructing the robot to execute the control sequence;
Including,
The control sequence is the following (1) to ( 6 );
(1) Grasping the operation target from the start position,
(2) scanning the identification information of the operation target with a scanner located between the start position and the work position,
(3) When the predetermined condition is satisfied, the operation target is temporarily released from the end effector at the holding position and is re-gripped by the end effector so as to be held again.
(4) moving the operation target in the working position,
(5) It is set as the predetermined condition that the storage efficiency of the operation target at the work position is enhanced when the operation target is changed and the gripping direction of the end effector is changed, and
(6) calculating the storage efficiency in the work position before the operation target is changed and the storage efficiency in the work position after the operation target is changed,
A method for controlling a robot system, including: The control sequence is the following (7) and (8);
(7) Obtaining the height of the operation target, and
(8) calculating the storage efficiency based on the height of the operation target,
The control method of the robot system according to claim 1 , further comprising: The height of the operation target is calculated from the height position of the top surface of the operation target and the height position of the bottom surface of the operation target measured in a state gripped by the end effector,
The control method of the robot system according to claim 2 . The height of the operation target is determined when scanning the operation target by the scanner,
Control method of a robot system according to claim 2 or 3. The control sequences may (9) when the predetermined condition is satisfied includes the operation target in the re-holding position, temporarily released from the end effector is placed on the temporary placement table, according to claim 1-4 7. A method for controlling a robot system according to any one of 1. Acquiring imaging data indicating a pickup area including the operation target,
Determining a first pose of the operation target based on the imaging data,
Calculating a confidence criterion for the first pose of the manipulator;
Obtaining the approach position and the scan position based on the confidence criterion,
Further comprising, a control method of a robot system according to any one of claims 1 to 5. (10) selectively calculating the approach position and the scan position according to a performance metric and / or a scan metric based on a result of comparing the confidence criterion with a sufficiency threshold;
Including,
The scan metric is related to the likelihood that the identifier of the manipulator will not be covered by the end effector, regardless of the confidence criterion regarding the initial pose of the manipulator.
The control method of the robot system according to claim 6 . The approach position and the scan position are obtained based on the metric of the scan when the confidence criterion does not meet a sufficiency threshold, or prioritize the metric of the scan over the metric of the performance, Obtained based on the metric of
The control method of the robot system according to claim 6 . The approach position and the scan position are obtained based on the performance metric if the confidence criterion meets a sufficiency threshold,
The control method of the robot system according to claim 6 . The control sequence is the following (11) and (12);
(11) A first scan position for providing the identification information of the operation target to the scanner, and a second scan position for providing the alternative identification information of the operation target to the scanner. To obtain, and
(12) After the operation target is moved to the first scan position, if the scan result indicates a successful scan, the operation target is moved to the work position and the second scan position is changed. Ignoring, or moving the operation target to the second scan position if the scan result indicates a failed scan,
Including, control method according to any one of claims 1-9. A non-transitory computer-readable recording medium having processor instructions recorded thereon for carrying out a control method of a robot system including a robot having a robot arm and an end effector,
The processor instructions are
A command to obtain an approach position at which the end effector grips an operation target;
An instruction to acquire a scan position for scanning the identification information of the operation target;
A command for creating or acquiring a control sequence based on the approach position and the scan position, and for instructing the robot to execute the control sequence,
Including,
The control sequence is the following (1) to ( 6 );
(1) Grasping the operation target from the start position,
(2) scanning the identifier of the operation target with a scanner located between the start position and the work position,
(3) When the predetermined condition is satisfied, the operation target is temporarily released from the end effector at the holding position and is re-gripped by the end effector so as to be held again.
(4) moving the operation target in the working position,
(5) It is set as the predetermined condition that the storage efficiency of the operation target at the work position is increased when the operation target is changed and the direction of gripping by the end effector is changed, and
(6) calculating storage efficiency at the work position before changing the operation target and storage efficiency at the work position after changing the operation target,
A non-transitory computer-readable recording medium including. The control sequence is the following (7) and (8);
(7) Obtaining the height of the operation target, and
(8) calculating the storage efficiency based on the height of the operation target,
The non-transitory computer-readable recording medium according to claim 11 , comprising: The height of the operation target is calculated from the height position of the top surface of the operation target and the height position of the bottom surface of the operation target measured in a state gripped by the end effector,
The non-transitory computer-readable recording medium according to claim 12 . It includes a robot having a robot arm and the end effector and the robot controller system for performing a control method according to any one of claims 1-10.

Images