DE102020127343A1 - Method and control system for verifying and updating calibration information for the robot controller - Google Patents

Abstract

A robotic control system and method for calibration verification is presented. The computer system is configured to perform a first calibration process and control a robotic arm to move a verification symbol to a reference location. The robot control system further receives a reference image of the verification symbol from a camera and determines a reference image coordinate for the verification symbol. The robot control system also controls the robot arm to move the verification symbol to the reference location again during a rest period, receives an additional image of the verification symbol, and determines a verification image coordinate. The robot control system determines a deviation parameter value based on the reference image coordinate and the verification image coordinate, and whether the deviation parameter value exceeds a defined threshold value, and performs a second calibration process if the threshold value is exceeded.

Description

REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of the United States patent application 16 / 732,832 with the title "Method and Control System for Verifying and Updating Camera Calibration for Robot Control," filed January 2, 2020 , which is a continuation of the US patent application 16 / 525.004 with the title "Method and Control System for Verifying and Updating Camera Calibration for Robot Control," filed July 29, 2019 which is a continuation of the US patent application 16 / 369,630 with the title "Method and Control System for Verifying and Updating Camera Calibration for Robot Control," filed March 29, 2019 the entire contents of which are incorporated herein by reference. This application also claims priority to U.S. Provisional Application No. 62 / 916,798 entitled "Method and Control System for Verifying Calibration for Robot Control," the entire contents of which are also incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to a method and a control system for verifying and updating calibration information for robot control.

BACKGROUND

As automation increases, robots are used in more environments, such as warehouse and manufacturing environments. For example, robots can be used to load and unload items on a pallet in a warehouse, or to pick up items from a conveyor belt in a factory. The movement of the robot can be fixed or can be based on an input such as an image that is captured by a camera in the warehouse or the factory. In the latter situation, a calibration process can be performed to determine a property of the camera and to determine a relationship between the camera and an environment in which the robot is located. The calibration process can generate calibration information that is used to control the robot. In some implementations, the calibration process can involve manual operation by a person who can manually control movements of the robot or the camera to capture an image of the robot.

SHORT REPRESENTATION

One aspect of the embodiments herein relates to performing a calibration verification for robot control, e.g. B. the verification of the camera calibration or another system calibration. The calibration verification can be carried out by a robotic control system comprising a communication interface and a control circuit. The communication interface can be configured to communicate with a robot having a base and a robot arm with a verification symbol arranged thereon and to communicate with a camera with a camera field of view. The control circuitry of the robot control system can be configured to perform the camera calibration verification by: a) performing a first calibration process (e.g., a first camera calibration) to determine calibration information (e.g., camera calibration information), b) issuing a first movement command to the Communication interface, wherein the communication interface is configured to communicate the first movement command to the robot to cause the robot arm to move the verification symbol to a location within the camera's field of view during or after the first calibration process, the location being a reference location of one or more Reference locations for verifying the first calibration process are, c) receiving an image of the verification symbol via the communication interface from the camera, which is configured to capture the image of the verification symbol at the reference location, the image being a reference image d is for the verification, d) determining a reference image coordinate for the verification symbol, the reference image coordinate being a coordinate at which the verification symbol appears in the reference image, and e) outputting a second movement command based on the calibration information to the communication interface, wherein the communication interface is configured to communicate the second movement command to the robot in order to cause a movement of the robot arm to carry out a robot operation.

In one embodiment, the control circuit is configured to further carry out the verification of the calibration by: f) detecting a rest time during robot operation, g) outputting a third movement command to the communication interface, wherein the communication interface is configured to communicate the third movement command to the robot in order to cause the robot arm to move the verification symbol to at least the reference location during the rest time, h) receiving, via the communication interface, an additional one Image of the verification symbol from the camera which is configured to capture the additional image of the verification symbol during the rest time at least at the reference location, the additional image being a verification image for the verification, i) determining a verification image coordinate which is used for the verification, wherein the verification image coordinate is a coordinate at which the verification symbol appears in the verification image, j) determining a deviation parameter value based on an amount of deviation between the reference image coordinate and the verification image coo rdinate, wherein the reference image coordinate and the verification image coordinate are both connected to the reference location, the deviation parameter value being a change since the first calibration process in the camera with which the communication interface is configured to communicate, or a change since the first calibration process in a relationship between the camera and indicates to the robot with which the communication interface is configured to communicate, k) determining whether the deviation parameter value exceeds a defined threshold value, and 1) executing, in response to a determination that the deviation parameter value exceeds the defined threshold value, a second calibration process ( z. A second camera calibration event) to determine updated calibration information (e.g. updated camera calibration information).

Figure list

• Die 1A und 1B stellen Blockdiagramme von Systemen dar, bei denen eine Verifizierung von Kalibrierinformationen gemäß Ausführungsformen hierin ausgeführt wird.
• 1C stellt ein Blockdiagramm eines Robotersteuersystems dar, das dazu konfiguriert ist, eine Kalibrierungsverifizierung gemäß einer Ausführungsform hierin auszuführen.
• 1D stellt ein Blockdiagramm einer Kamera dar, für die eine Kamerakalibrierung gemäß einer Ausführungsform hierin ausgeführt wird.
• 2 stellt ein System dar, das einen Roboter veranschaulicht, der gemäß einer Ausführungsform hierin basierend auf Kalibrierinformationen, die von einem Kalibriervorgang erlangt sind, gesteuert wird.
• 3 zeigt ein System zum Ausführen eines Kalibriervorgangs gemäß einer Ausführungsform hierin.
• Die 4A und 4B stellen ein Ablaufdiagramm bereit, das ein Verfahren zum Ausführen einer Verifizierung von Kalibrierinformationen gemäß einer Ausführungsform hierin veranschaulicht.
• Die 5A und 5B veranschaulichen ein System, in dem ein Verifizierungssymbol an einem Roboter angeordnet ist, wobei das Verifizierungssymbol verwendet wird, um eine Verifizierung von Kalibrierinformationen gemäß einer Ausführungsform hierin auszuführen.
• 5C zeigt ein Beispiel für ein Verifizierungssymbol gemäß einer Ausführungsform hierin.
• Die 6A-6D zeigen Beispiele gemäß einer Ausführungsform hierin von Referenzorten, an denen entsprechende Bilder eines Verifizierungssymbols erfasst werden.
• 7A zeigt ein Beispiel des Bestimmens einer Referenzbildkoordinate gemäß einer Ausführungsform hierin.
• 7B zeigt ein Beispiel des Bestimmens einer Verifizierungsbildkoordinate gemäß einer Ausführungsform hierin.
• 8 veranschaulicht eine beispielhafte Zeitachse für eine Verifizierung von Kalibrierinformationen gemäß einer Ausführungsform hierin.
• 9 stellt ein Ablaufdiagramm bereit, das ein beispielhaftes Verfahren zum Ausführen einer Verifizierung von Kalibrierinformationen gemäß einer Ausführungsform hierin veranschaulicht.
• Die 10A-10C veranschaulichen ein System, in dem eine Gruppe von Verifizierungssymbolen an einem Roboter angeordnet ist, wobei die Gruppe von Verifizierungssymbolen verwendet wird, um eine Verifizierung von Kalibrierinformationen gemäß einer Ausführungsform hierin auszuführen.
• Die 11A-11B veranschaulichen ein System, in dem eine Gruppe von Verifizierungssymbolen an einem Roboter angeordnet ist, wobei die Gruppe von Verifizierungssymbolen verwendet wird, um eine Verifizierung von Kalibrierinformationen gemäß einer Ausführungsform hierin auszuführen.
• 11C veranschaulicht eine Gruppe von Verifizierungssymbolen, die gemäß einer Ausführungsform davon unterschiedliche entsprechende Größen aufweisen.
• Die 12A und 12B stellen ein Ablaufdiagramm bereit, das ein Verfahren zum Ausführen einer Verifizierung von Kalibrierinformationen gemäß einer Ausführungsform hierin veranschaulicht.
• Die 13A und 13B veranschaulichen entsprechend ein Referenzbild und ein Verifizierungsbild, die mit einer ersten Pose eines Roboterarms verbunden sind.
• Die 14A und 14B veranschaulichen entsprechend ein Referenzbild und ein Verifizierungsbild, die mit einer zweiten Pose eines Roboterarms verbunden sind.

The foregoing and other features, objects, and advantages of the invention will become apparent from the following description of embodiments as illustrated in the accompanying drawings. The accompanying drawings, which are incorporated herein and form a part of the specification, also serve to explain the principles of the invention and to enable any person skilled in the relevant art to make and use the invention. The drawings are not to scale.

• The 1A and 1B FIG. 10 depicts block diagrams of systems in which verification of calibration information is performed in accordance with embodiments herein.
• 1C FIG. 10 illustrates a block diagram of a robotic control system configured to perform calibration verification according to an embodiment herein.
• 1D FIG. 10 illustrates a block diagram of a camera for which camera calibration is performed in accordance with an embodiment herein.
• 2 FIG. 10 depicts a system illustrating a robot controlled based on calibration information obtained from a calibration process, according to an embodiment herein.
• 3 Figure 12 shows a system for performing a calibration process according to an embodiment herein.
• The 4A and 4B provide a flow chart illustrating a method for performing verification of calibration information according to an embodiment herein.
• The 5A and 5B illustrate a system in which a verification icon is placed on a robot, the verification icon being used to perform verification of calibration information in accordance with an embodiment herein.
• 5C Figure 12 shows an example of a verification icon according to an embodiment herein.
• The 6A-6D show examples according to an embodiment herein of reference locations at which corresponding images of a verification symbol are captured.
• 7A Figure 12 shows an example of determining a reference image coordinate according to an embodiment herein.
• 7B Figure 10 shows an example of determining a verification image coordinate according to an embodiment herein.
• 8th FIG. 11 illustrates an exemplary timeline for verification of calibration information according to an embodiment herein.
• 9 provides a flow diagram illustrating an example method for performing verification of calibration information according to an embodiment herein.
• The 10A-10C illustrate a system in which a group of verification symbols is disposed on a robot, the group of verification symbols being used to perform verification of calibration information according to an embodiment herein.
• The 11A-11B illustrate a system in which a group of verification symbols is disposed on a robot, the group of verification symbols being used to perform verification of calibration information according to an embodiment herein.
• 11C Figure 3 illustrates a group of verification symbols having different corresponding sizes according to one embodiment thereof.
• The 12A and 12B provide a flow chart illustrating a method for performing verification of calibration information according to an embodiment herein.
• The 13A and 13B Figures 10, respectively, illustrate a reference image and a verification image associated with a first pose of a robotic arm.
• The 14A and 14B Figures 10, respectively, illustrate a reference image and a verification image associated with a second pose of a robotic arm.

DETAILED DESCRIPTION

The following detailed description is merely exemplary in nature and is not intended to limit the invention or the application and uses thereof. Furthermore, there is no intention to be bound by any expressed or implied theory presented in the preceding technical field, prior art, brief description or the following detailed description.

Embodiments described herein relate to verifying and / or updating calibration information used to control a robot, such as a robot used in a warehouse, manufacturing facility, or other environment. In particular, the calibration information can be carried out to facilitate the control of a robot operating system, and it can be determined by carrying out a calibration process which can be referred to as system calibration. System calibration can include calibration of a camera (which may be referred to as a camera calibration or camera calibration process), calibration of a robot (which may be referred to as robot calibration), calibration of another element of the robot operating system, or any combination thereof. The system calibration can e.g. B. executed by a robot control system (also referred to as a robot controller) to generate calibration information (e.g. camera calibration information) that facilitate an ability of the robot control system to target the robot based on images captured (e.g. photographed) by the camera Taxes. For example, the robot can be used to pick up a package in a warehouse, where the placement of a robot arm or other component of the robot can be based on images of the package captured by the camera. In this case, if the calibration information includes camera calibration information, the camera calibration information can be used together with the images of the package, for example to determine a location and an orientation of the package relative to the robot arm of the robot. If the system calibration includes camera calibration, the camera calibration may involve determining appropriate estimates of intrinsic camera parameters of the camera (which may also be referred to as internal parameters) and determining an estimate of a relationship between the camera and its external environment. An intrinsic parameter of the camera can have one or more parameter values, such as a matrix, a vector or a scalar value. Further examples of an intrinsic parameter are a projection matrix and a distortion parameter. In one case, camera calibration may include determining the camera position with respect to a fixed position in an external environment, which can be expressed as a transformation function representing the relationship between the camera and the fixed position in the external environment. In some cases, the camera calibration can be carried out with the aid of a calibration pattern, which can have pattern elements that can be arranged at defined locations on the calibration pattern. The camera can capture an image of the pattern elements of the calibration pattern (also referred to as a calibration image) and the camera calibration can be carried out based on the comparison of an image of the pattern elements with the defined locations of the pattern elements. Camera calibration is described in U.S. Application No. 16 / 295,940 (File Number MJ0021US1) entitled "METHOD AND DEVICE FOR PERFORMING AUTOMATIC CAMERA CALIBRATION FOR ROBOT CONTROL," the entire contents of which are incorporated herein by reference.

As mentioned above, one aspect of the present disclosure relates to verifying whether an earlier camera calibration or other calibration process is still correct at a later time. The camera calibration performed at the earlier point in time can generate camera calibration information reflecting a property of the camera at that point in time, such as an intrinsic parameter of the camera or a relationship between the camera and its external environment at that point in time. In some cases, a previous camera calibration may lose accuracy over time as the characteristics of the camera can change over time. In a first example, an intrinsic parameter of the camera can change over time. Such a change can e.g. B. caused by a temperature change, which changes a shape of a housing and / or a lens of the camera. In a second example, a relationship between the camera and its external environment can change over time. The camera can, for example, shift in its position or orientation relative to, for example, a base of the robot or a location in a warehouse. Such a change can e.g. B. from a change in temperature that expands or contracts a component used to mount the camera, from a person or other object striking the camera, from vibration in the external environment of the camera (e.g. B. a warehouse), by a force caused by the weight of the camera (ie by gravity) or by some other factor. These changes could render the camera calibration information or other calibration information out of date, and later using this camera calibration information or other calibration information to position a robotic arm or other component of the robot could result in errors. In other words, if a property associated with the camera has changed over time, but the camera calibration information is not updated to reflect such a change, the robot may operate based on outdated or otherwise incorrect camera calibration information, causing undesirable errors in the robot's operation. To address the possibility that changes in one or more properties of the camera may occur, a robot control system can automatically perform a verification that detects when camera calibration information from a camera calibration is no longer sufficiently accurate (or more generally when calibration information from a calibration process is no longer accurate are sufficiently accurate). Detecting such a condition can provide an indication of a change in a property of the camera or other element of a robot operating system. If the verification detects that the calibration information is no longer sufficiently accurate, the robot control system can perform a calibration process to determine updated calibration information that may reflect a more recent property or properties of the camera or other element of the robot operating system. The updated calibration information can be used to control the placement of the robot arm or any other aspect of robot operation. Accordingly, an automatic check of the calibration information and / or an update of the calibration information is carried out to ensure that the robot is operating based on correct information about one or more properties associated with the camera or another element of the robot operating system.

One aspect of the embodiments herein relates to verifying calibration information for a camera by comparing a reference image captured by the camera with a verification image captured by the camera. In some cases, the reference image can be an image of an object captured when the object was previously in a particular location, and the verification image can be an image of the object captured in the same location at a later time . During the verification, it can be determined whether there is too great a deviation between the reference image and the verification image, such as, for example, whether the deviation exceeds a certain threshold value. In some implementations, the object can be a verification icon. More specifically, a robot arm or other component of the robot may have a verification icon that is used to verify the calibration information. Both the reference image and the verification image can capture or otherwise include the verification symbol, and the robotic control system can compare the two images by comparing an appearance of the verification symbol in the reference image with an appearance of the verification symbol in the verification image. For example, after the robot control system has performed a calibration process that generates calibration information at a certain point in time, the robot control system can control the robot arm (e.g. via a movement command) in such a way that it places the verification symbol in a number of predefined locations within the field of view of the camera ( also known as the camera's field of view), which locations can be used as a set of reference locations for verification. The camera can take corresponding reference images of the verification symbol at the series of reference locations. In some cases, the reference images can be captured immediately after the calibration process. The movement of the robot arm, or in particular a movement command that is used to move the robot arm, can be based on the calibration information be based on the calibration process being carried out or be independent of the calibration information. In some cases, the reference images can be captured before the robot starts robot operation. After the reference images have been captured, the robot can be considered ready to start robot operation to perform a task, and the robot control system can e.g. B. control the positioning of the robot arm based on images subsequently captured by the camera.

As mentioned above, the reference images can be compared with subsequently acquired verification images. In one embodiment, the verification images may be captured during one or more rest periods that are detected by the robotic control system. In particular, at the beginning of robot operation, the robot can begin performing robot tasks (e.g. by interacting with packages or other objects). While the robot is performing the robot operation, the robot control system can detect one or more rest periods for the robot. In some cases, the rest time may be a period of time in which the robot is free from performing a robot task during robot operation. In some cases, the robot control system may schedule robot operation based on detecting or otherwise anticipating objects with which the robot must interact and detect a rest time based on detecting or otherwise anticipating the absence of objects with which the robot must interact.

During the rest period (s), the robot control system can control the robot arm or other components of the robot (e.g. via a movement command) in such a way that it moves to the reference locations and (e.g. via a camera command) at each of the reference locations corresponding verification image captured. When the robot has a verification icon, the robot control system can more specifically control the robot arm to move the verification icon to the reference locations and capture the verification images. The robot control system can then determine how much the corresponding verification images deviate from corresponding reference images at corresponding reference points. In some cases, the deviation between the verification images and corresponding reference images can be expressed as a deviation parameter. If a value of the deviation parameter (also referred to as a deviation parameter value) exceeds a defined threshold value for the deviation parameter (which can also be referred to as a defined deviation threshold value), the robot control system can perform an additional calibration process (e.g. an additional camera calibration) in order to obtain updated calibration information ( e.g. updated camera calibration information) for the camera. When a deviation parameter value exceeds the defined deviation threshold, this condition can indicate that the use of previously generated calibration information can lead to an undesirable amount of error in robot operation. Therefore, in some cases, robot operation can be paused or stopped while the additional calibration process is in progress (the pause can be viewed as another rest period). After the additional calibration process is complete, a new set of reference images can be captured and robot operation can continue with the updated camera calibration information. During one or more subsequent rest periods, a new set of verification images can be acquired and the robot control system can carry out the verification of the additional calibration process by comparing the new set of reference images with the new set of verification images.

As mentioned above, the robot control system can carry out an additional calibration process if a deviation parameter value exceeds the defined deviation threshold value. If the deviation parameter value does not exceed the deviation threshold value, the robot operation can be continued after a rest period without the robot control system performing an additional calibration process. In this scenario, the camera can capture a new set of verification images at the appropriate reference locations during one or more subsequent rest periods. When the new set of verification images is acquired, the robot control system can again carry out a verification of the calibration information by determining how much the new set of verification images deviates from the corresponding reference images at the corresponding reference locations.

As mentioned above, the robot arm can have a verification symbol, such as a ring pattern, and the verification symbol can be captured from or otherwise contained in the reference images and the verification images. In one embodiment, the robot control system may determine a deviation between the reference images and the corresponding verification images based on respective locations where the verification symbol appears in the reference images and based on corresponding locations where the verification symbol appears in the verification images. The robot control system can, for example, determine a reference image coordinate for each reference location. The reference image coordinate for a particular location can be a coordinate where the Verification Icon appears in a reference image captured when the Verification Icon was placed at that reference location. In particular, the reference image coordinate can be linked to a specific reference location and relate to an image coordinate at which a verification symbol appears in a reference image, the reference image being captured by a camera when the verification symbol was placed at this reference location. In the above example, the image coordinate can refer to a coordinate in an image, such as a pixel coordinate. If the robot control system then places the verification symbol at a specific reference location at a later point in time and receives a corresponding verification image, the robot control system can determine a verification image coordinate. The verification image coordinate can also be associated with the reference location and refer to an image coordinate (e.g. a pixel coordinate) at which the verification symbol appears in the verification image, the verification image being captured by the camera when the verification symbol was placed at the reference location. The robot control system can compare the reference image coordinate associated with a certain reference location with a verification image coordinate associated with the same reference location. This comparison can be carried out for each reference location at which a verification image and a reference image were captured.

In one case, a reference image coordinate at which a verification symbol appears in a reference image may be a coordinate of a center of the verification symbol in the reference image (which is also referred to as a center coordinate of the verification symbol in the reference image). Similarly, a verification image coordinate at which the verification symbol appears in a verification image may be a coordinate of a center of the verification symbol in the verification image (which is also referred to as a center coordinate of the verification symbol in the verification image). For each reference location at which the robot arm and / or the verification symbol was when a corresponding verification image was captured, the robot control system can determine a deviation between the reference image coordinate associated with the reference location and the verification image coordinate associated with the same reference location. If the robot arm and / or the verification symbol have been placed at a plurality of reference locations, the robot control system can determine corresponding deviation amounts between corresponding reference image coordinates and corresponding verification image coordinates for the plurality of reference locations. The robot control system may further determine a value of a deviation parameter based on the respective deviation amounts between the reference image coordinates and the respective verification image coordinates for respective reference locations.

In one case, the verification symbol may include multiple shapes that are concentric with one another such that the respective centers of the multiple shapes are in the same or substantially the same location in the verification symbol. For example, the verification symbol can be a ring pattern that includes two or more concentric circles. In some cases, when the reference image coordinate of a verification symbol is a center coordinate of the verification symbol in a reference image, the robot control system can determine a center coordinate of the verification symbol based on corresponding center coordinates of the plurality of shapes in the reference image, the center coordinate of a certain shape being a coordinate of a center of that shape. When the verification symbol is a ring pattern, the center coordinate of the ring pattern in a reference image can be determined as an average of a center coordinate of a first circle forming the ring pattern and a center coordinate of a second circle forming the ring pattern in the reference image. Similarly, a center coordinate of a verification symbol in a verification image can be determined based on the corresponding center coordinates of the plurality of shapes that make up the verification symbol in the verification image. In some cases, using multiple shapes to form the verification symbol can improve verification accuracy. For example, using the corresponding center coordinates of the plurality of shapes in an image to determine a center coordinate of the verification symbol can improve robustness of the verification against image noise. In particular, when an image of the verification symbol includes image noise, the image noise can reduce an accuracy with which the robot control system detects a center coordinate of a certain shape of the verification symbol. However, when the center coordinate of this shape is averaged with a center coordinate of another shape to determine a center coordinate of the verification symbol, the averaged center coordinate can reduce an effect of the image noise. As a result, accuracy in determining the center coordinate of the verification symbol can be improved.

In one case, the verification symbol can have a plurality of regions with different corresponding colors, wherein corresponding areas of the plurality of regions can have a certain and defined relationship. For example, the verification symbol may have a first region with a first color (e.g. black) and a second region with a second color (e.g. white), with a The ratio of an area of the first region to an area of the second region is defined or otherwise known. The unique relationship can facilitate the identification of the verification symbol in an image, in particular if the image captures or otherwise includes other features, such as points of a calibration pattern. The robot arm that moves the verification symbol can, for example, also have the calibration pattern arranged on the robot arm. The robot control system can use the ratio to distinguish the verification symbol from the points of the calibration pattern. Since the ratio of the areas of the various regions of the verification symbol is defined as a unique ratio, the robot control system can identify the verification symbol in an image based on the defined ratio. In identifying the verification symbol appearing in an image, the robot control system can distinguish the verification symbol from the calibration pattern or other feature based on the defined relationship. In some cases, the verification symbol in the image can be identified as a part of the image having the plurality of regions with different corresponding colors and having the defined relationship between corresponding areas of the plurality of regions. If the robotic control system or another system or device determines that a certain part of the image does not have multiple regions each having different colors, or that corresponding areas of the multiple regions have a different ratio than the defined ratio, the robotic control system can determine that the part of the image is not the verification icon.

In one case, the robot control system can perform the verification based on a temperature surrounding the robot. For example, the robot control system may adjust the defined deviation threshold based on temperature (i.e., define a new value for the deviation threshold). For example, a temperature can affect different parts in the camera and / or in the robot, as some materials are sensitive and / or can expand / contract based on temperature. A change in temperature can cause one or more intrinsic parameters of the camera to change and / or to change the relationship between the camera and its external environment. In one embodiment, the deviation threshold can be set to have a first value when the temperature is outside a defined range, while the deviation threshold can be set to have a second value that is lower than the first value when the temperature is within the defined range. If the temperature is e.g. B. is within a defined normal operating temperature range (e.g. within 10 degrees of room temperature), then the deviation threshold can be the first value. If the temperature is outside the normal operating temperature range, then the deviation threshold can have the second value lower than the first value. The second value can be lower than the first value so that an additional calibration process can be more easily triggered if the temperature is outside the normal operating range, since the temperature outside the normal operating temperature range is more likely to change the camera or its relationship to the external environment and therefore more likely to cause errors in robot operation with previously generated camera calibration information or any other calibration information.

In one embodiment, the verification of the calibration information can be based on only a single reference location. Alternatively, the verification of the calibration information can be based on multiple reference locations. The reference locations can be any locations in the field of view of a camera or specific, defined locations. The reference locations can be defined, for example, as locations on a surface of at least one imaginary sphere that is concave with respect to the camera. At each reference location in this scenario, the robotic arm can be controlled to position the verification symbol such that the verification symbol is positioned tangential to the surface of the at least one imaginary sphere while facing the camera. This positioning may allow the verification icon to be better photographed or otherwise viewed head-on by the camera (with the verification icon facing directly towards the camera) so that an image of the verification icon will resemble a top view rather than a perspective view of the verification icon. For example, if the verification symbol is a ring pattern, positioning the ring pattern tangent to the surface of the imaginary sphere may allow the resulting image of the ring pattern to still appear circular rather than elliptical. The resulting image may have less or no perspective distortion (relative to a scenario where the ring pattern in the image appears elliptical). The lack of perspective distortion can facilitate an accurate determination of a central coordinate of the ring pattern. In some cases, the reference locations can be divided into several imaginary spheres, all of which are concave with respect to the camera. The plurality of imaginary spheres can have a common center and can be of different sizes, so that each imaginary sphere has a spherical surface, each of which is at a different distance from the camera. In some cases the camera can be a common center for all imaginary spheres.

1A Figure 3 shows a block diagram of a robot operating system 100 (also as a system 100 for performing automatic camera calibration and automatic camera calibration verification. While some of the following examples discuss performing an automatic camera calibration and verifying the camera calibration information determined by the automatic camera calibration, these examples apply more generally to any type of automatic calibration process and to verifying any type of calibration information determined by the automatic calibration process, hold true. The robot operating system 100 includes a robot 150 , a robot control system 110 (also known as a robot controller) and a camera 170 . In one embodiment, the system can 100 are in a warehouse, production facility or other premises. The robot control system 110 may be configured to perform camera calibration, described in more detail below, to determine calibration information that will later be used to identify the robot 150 in such a way that it can operate a robot, e.g. B. the picking up of packages in the warehouse. The robot control system 110 can also be configured to perform a camera calibration verification, also discussed in more detail below, to verify that the camera calibration information is still sufficiently accurate. In some cases the robot control system is 110 configured to perform camera calibration and the robot 150 to perform a robot operation based on the camera calibration information. In some cases the robot control system can 110 form a single device (e.g. a single console or a single computer) that works with the robot 150 and the camera 170 communicates. In some cases the robot control system can 110 comprise multiple devices.

In some cases the robot control system can 110 specifically designed to perform camera calibration and / or review camera calibration, and can communicate the latest camera calibration information to another control system (also referred to as another controller, not shown) which then controls the robot 150 controls to perform a robot operation based on the latest camera calibration information. The robot 150 can be based on images taken by the camera 170 and positioned on the camera calibration information. In particular, the robot control system 110 in one embodiment, be configured to generate movement commands based on the images and the camera calibration information, and the movement commands to the robot 150 communicates to control the movement of his robotic arm. In some cases the robot control system is 110 configured to perform the camera calibration verification during a rest period of the robot operation. In some cases the robot control system is 110 configured to perform verification while performing a robot operation with the robot 150 to execute.

In one embodiment, the robotic control system can 110 configured to have wired or wireless communication with the robot 150 and the camera 170 to communicate. The robot control system 110 can for example be configured to work with the robot 150 and / or the camera 170 Communicate via an RS-232 interface, a universal serial bus (USB) interface, an Ethernet interface, a Bluetooth® interface, an IEEE 802.11 interface or any combination thereof. In one embodiment, the robotic control system can 110 to be configured with the robot 150 and / or the camera 170 communicate over a local computer bus such as a Peripheral Component Interconnect (PCI) bus.

In one embodiment, the robotic control system can 110 separate from the robot 150 and communicate with the robot via the wireless or wired connection described above. The robot control system 110 can for example be a single computer configured to work with the robot 150 and the camera 170 communicate over a wired connection or a wireless connection. In one embodiment, the robotic control system can 110 an integral part of the robot 150 his and with other components of the robot 150 communicate over the local computer bus discussed above. In some cases the robot control system can 110 be a dedicated control system (also known as a dedicated controller) that only controls the robot 150 controls. In other cases, the robot control system 110 configured to have multiple robots including the robot 150 to control. In one embodiment, the robotic control system is located 110 , the robot 150 and the camera 170 in the same premises (e.g. a warehouse). In one embodiment, the robotic control system can 110 away from the robot 150 and the camera 170 located and it can be configured to work with the robot 150 and the camera 170 Communicate over a network connection (e.g. a local area network (LAN) connection).

In one embodiment, the robotic control system can 110 be configured to take images of a calibration pattern 160 and / or one on the robot 150 arranged verification symbol 165 (e.g. on a Robot arm) from the camera 170 retrieve or otherwise receive. In some cases the robot control system can 110 be configured to use the camera 170 to control to capture such images. The robot control system 110 For example, it can be configured to generate a camera command that causes the camera 170 an image in a field of view of the camera 170 (also known as the camera's field of view) and the camera command over the wired or wireless connection to the camera 170 communicates. The same command can cause the camera 170 the image also to the robot control system 110 or more generally communicated to a storage device maintained by the robot control system 110 is accessible. The robot control system 110 can alternatively generate another camera command that causes the camera 170 after receiving the camera command, an image or images that it has captured to the robot control system 110 communicates. In one embodiment, the camera 170 Capture an image in your camera's field of view either periodically or automatically in response to a defined trigger condition, without a camera command from the robot control system 110 to need. In such an embodiment, the camera 170 also be configured to send the image to the robot control system 110 or more generally to a storage device maintained by the robot control system 110 is accessible, automatically, without a camera command from the robot control system 110 , to communicate.

In one embodiment, the robotic control system can 110 be configured to move the robot 150 to be controlled via movement commands issued by the robot control system 110 generated and via the wired or wireless connection to the robot 150 communicated. The robot 150 can be configured to use either or both of the calibration pattern 160 and the verification symbol 165 on the robot 150 available. 1B shows, for example, a robot operating system 100A with the verification symbol 165 without the presence of the calibration pattern 160 of 1A on the robot 150 is arranged. In one case, the verification icon 165 part of the robot 150 and permanently on the robot 150 be arranged. The verification symbol 165 can for example permanently on the robot 150 painted or part of a sticker or board that is permanent on the robot 150 is appropriate. In another case, the verification icon 165 be a separate component attached to the robot 150 can be attached and removed from it. The verification symbol 165 can be permanent on the robot 150 or may be a separate component attached thereto and from the robot 150 can be removed.

In one embodiment, the only images that are in the system 100 used to be the robot 150 to control, be the ones through the camera 170 are recorded. In another embodiment, the system 100 include multiple cameras and the robot 150 can be controlled by images from the multiple cameras.

1B further illustrates an embodiment in which the robotic control system 110 with a user interface device 180 communicates. The user interface device 180 can be configured to interface with an operator of the robot 150 forms, such as an employee in a warehouse in which the robot is 150 is located. The user interface device 180 can e.g. B. comprise a tablet or desktop computer that provides a user interface, the information related to the robot operation 150 indicates. As mentioned above, the robot control system can 110 be configured such that it detects when a deviation parameter value exceeds a defined deviation threshold value. In one embodiment, the user interface device 180 issue an alarm or other warning to inform the operator of the value of the deviation parameter exceeding the defined deviation threshold.

1C Figure 10 shows a block diagram of the robot control system 110 As illustrated in the block diagram, the robot control system comprises 110 a control circuit 111 , a communication interface 113 and a non-transitory computer readable medium 115 (e.g. memory). In one embodiment, the control circuit 111 one or more processors, a programmable logic circuit (PLC) or a programmable logic arrangement (PLA), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other control circuit.

In one embodiment, the communication interface can 113 comprise one or more components that are configured to work with the camera 170 of 1A or 1B and the robot 150 of 1A or 1B to communicate. The communication interface 113 For example, may include communication circuitry configured to perform communication over a wired or wireless protocol. The communication circuit may, for example, be an RS-232 port controller, a USB controller, an Ethernet controller, a Bluetooth® controller, a PCI bus controller, any other communication circuitry or a combination thereof. In one embodiment, the control circuit 111 be configured such that it generates a movement command (e.g. a motor movement command) and the movement command to the communication interface 113 issues. In this embodiment, the communication interface 113 be configured to send the move command to the robot 150 communicates to the movement of a robotic arm or other component of the robot 150 to control. In one embodiment, the control circuit 111 be configured to generate a camera command and the camera command (e.g. image capture command) to the communication interface 113 to spend. In this embodiment, the communication interface 113 be configured to send the camera command to the camera 170 communicates to the camera 170 in such a way that it photographs an object in the camera's field of view or otherwise captures an image of it. In one embodiment, the communication interface can 113 be configured to take a picture or other data from the camera 170 to receive, and the control circuit 111 can be configured to take the image from the communication interface 113 to recieve.

In one embodiment, the non-transitory computer readable medium can 115 comprise a computer memory. The computer memory can e.g. B. Dynamic Random Access Memory (DRAM), integrated solid-state memory and / or a hard disk drive (HDD). In some cases, camera calibration can be performed by computer-executable instructions (e.g., computer code) residing on the non-transitory computer-readable medium 115 are stored, be implemented. In such cases, the control circuit 111 comprise one or more processors configured to execute the computer-executable instructions to perform camera calibration verification (e.g., those described in FIGS 4A , 4B and 9 illustrated steps).

1D Figure 3 shows a block diagram of the camera 170 who have one or more lenses 171 , an image sensor 173 and a communication interface 175 includes. The communication interface 175 can be configured to do so with the robot control system 110 of 1A , 1B or 1C to communicate, and they can be the communication interface 113 of 1C of the robot control system 110 be similar to. In one embodiment, the one or more lenses 171 Focus light coming from outside the camera 170 on the image sensor 173 falls. In one embodiment, the image sensor 173 comprise an array of pixels configured to represent an image via corresponding pixel intensity values. The image sensor 173 may include a charge coupled device (CCD) sensor, a complementary metal oxide semiconductor (CMOS) sensor, a quantum image sensor (QIS), or any other image sensor.

The camera calibration can be performed as stated above to facilitate control of a robot based on images captured by a camera. For example poses 2 a robot operating system 200 dar (also as a system 200 is referred to) in which the images are used to represent a robot 250 to control a robot operation, such as picking up an object 292 in a warehouse. The system 200 can in particular an embodiment of the system 100 of 1A and includes a camera 270 , the robot 250 and the robot control system 110 . The camera 270 can be an embodiment of the camera 170 of 1A , 1B or 1D his and the robot 250 can be an embodiment of the robot 150 of 1A or 1B be. The camera 270 can be configured to take an image of the object 292 (e.g. a package for shipping) that is on a conveyor belt 293 located in the warehouse, and the robot control system 110 can be configured to use the robot 250 to control to the object 292 to record. When there is one or more objects on the conveyor belt 293 the robot control system 110 be configured to control the movement of the robot 250 to plan for capturing the objects. The robot control system 110 can in some cases be configured to detect a dormant time for robot operation by detecting when there are no objects on the conveyor belt 293 or when there are no objects on the conveyor belt 293 that are within reach of the robot 250 are located.

In the embodiment of 2 can the robot 250 One Base 252 and having a robotic arm relative to the base 252 is movable. In particular, the robot arm can have several links 254A to 254E and one on the limb 254E attached robotic hand 255 include. The multiple of the limbs 254A to 254E may be rotatable relative to one another and / or may be prismatic members that are linearly movable with respect to one another. There 2 the robot 250 that is used to pick up objects, the robotic hand can 255 the grippers 255A and 255B include those for gripping the object 292 be used. In one embodiment, the robotic control system can 110 configured to communicate a movement command to one or more of the limbs 254A to 254E to turn. The move command can be a low level command, such as motor move commands, or a high level command. When the movement command from the robot control system 110 is a high-level command, the robot can 150 be configured to convert the high-level command into a low-level command.

In one embodiment, the camera calibration information determined by the camera calibration describes a relationship between the camera 270 and the robot 250 or in particular a relationship between the camera 270 and a world point 294 that is relative to the base 252 of the robot 250 is stationary. The world point 294 can represent a world or other environment in which the robot is 250 and can be any imaginary point relative to the base 252 is stationary. In other words, the camera calibration information may include information relating to a relationship between the camera 270 and the world point 294 describe. In one embodiment, this relationship can include a location of the camera 270 relative to the world point 294 as well as an orientation of the camera 270 relative to a reference orientation for the robot 250 describe. The above relationship between the camera 270 and the world point 294 can be referred to as a camera-to-world relationship and can be used to represent a relationship between the camera 270 and the robot 250 be used. In some cases, the camera-to-world relationship can be used to establish a relationship between the camera 270 and the object 292 (also known as a camera-to-object relationship) and a relationship between the object 292 and the world point 294 (also known as the object-to-world relationship). The camera-to-object relationship and the object-to-world relationship can be used to control the robot 250 to control and the object 292 to record.

In one embodiment, the camera calibration information can be an intrinsic parameter of the camera 270 describe, where the intrinsic parameter can be any parameter, its value from a location and an orientation of the camera 270 is independent. The intrinsic camera parameters can be a property of the camera 270 such as their focal length, a size of their image sensor or a lens distortion effect caused by the camera 270 is introduced, characterize.

An example showing a detailed structure of an example of a robot 350 shows is in 3 illustrated having a robot operating system 300 represents which the robot operating system 110 in communication with a camera 370 and the robot 350 includes. The camera 370 can accordingly be an embodiment of the camera 170/270 of 1A , 1B , 1D or 2 his and the robot 350 can accordingly be an embodiment of the robot 150/250 of 1A , 1B or 2 be. The camera 370 may be able to take pictures within a field of view 330 capture. The robot 350 can be a base 352 and one relative to the base 352 include movable robotic arm. The robotic arm includes one or more links, such as the links 354A to 354E , and a robotic hand 355 . In one embodiment, the members 354A-354E be rotatably attached to each other. The Member 354A can for example at the robot base 352 by means of a connection 356A be rotatably mounted. The remaining links 354B to 354E can by means of the connections 356B to 356E be rotatably attached to each other. In one embodiment, the base 352 used to be the robot 350 at z. B. to attach a mounting frame or a mounting surface (z. B. the floor of a warehouse). In one embodiment, the robot can 350 comprise multiple motors configured to drive the robotic arm by rotating the links 354A to 354E to move. For example, one of the motors can be configured to be the first link 354A in relation to the compound 356A and the base 302 to rotate as indicated by the dotted arrow in 3 is shown. Similarly, other motors of the plurality of motors can be configured to use the links 354B to 354E to turn. The multiple motors can be controlled by the robot control system 110 being controlled. 3 also shows the robot hand 355 that is firmly attached to the fifth link 354E is arranged. The robot hand 355 can be a calibration pattern 320 have so that the robot control system 110 Images of the calibration pattern 320 about the camera 370 capture and a camera calibration based on the captured images of the calibration pattern 320 can perform. The robot control system 110 can for example move the robot arm in such a way that the calibration pattern 320 within the camera's field of view 330 and for the camera 370 can be visible when the camera 370 for capturing the images of the calibration pattern 320 (also known as calibration images) is used. After the camera calibration is done, the robot hand can 355 removed and replaced with a different robotic hand, such as a robotic hand with a verification symbol attached, as described in more detail below.

As mentioned above, according to one embodiment, calibration verification may involve comparing a reference image coordinate at which a verification symbol appears in a reference image with a verification image coordinate at which the verification symbol appears in a verification image. The comparison can determine a deviation between the verification image coordinate and the reference image coordinate, and the deviation can be used to determine whether an additional calibration process should be carried out. The verification image can be used during a rest period Robot operation can be recorded. The 4A and 4B represent flow charts showing a procedure 400 for camera calibration verification according to one embodiment. While some of the embodiments below describe verifying camera calibration information determined from the camera calibration, the method may 400 can be used to more generally verify calibration information obtained from any calibration operation performed for a robot operating system. In one embodiment, the method can 400 by the control circuit 111 of the robot control system 110 are executed. As mentioned above, the robot control system can 110 the communication interface 113 of 1C include those needed to communicate with the robot 150 of 1A or 1B and with the camera 170 of 1A , 1B or 1D is configured. In one embodiment, the robot can have a base (e.g., the base 252 of 2 or the base 352 of 3 ) and a robot arm (e.g. the robot arm from 2 or 3 ) with a verification symbol arranged thereon, wherein the robot arm is movable relative to the base.

An exemplary environment in which the procedure 400 is running is in 5A and 5B illustrated having a robot operating system 500 / 500A represent each of the robot control systems 110 in communication with the camera 570 and a robot 550 includes. The camera 570 can be an embodiment of the camera 170/270/370 of 1 , 2 respectively 3 and the robot 550 can be an embodiment of the robot 150/250/350 of 1A , 1B , 2 be 3 respectively. The robot 550 can be a base 552 and a robotic arm (which is in the 5A and 5B With 553 which is relative to the base 552 is movable. The robotic arm includes one or more links, such as the links 554A up to limb 554E . The limbs 554A-554E can also be examples of arm parts of the robot arm 553 be that are movably attached to each other. In one embodiment, the members 554A-554E be rotatably attached to each other. For example, the limb 554A at the robot base 552 be rotatably attached. The remaining links 554B to 554E can be rotatably attached to one another by means of several connections. In one embodiment, the base 552 used to be the robot 552 at z. B. to attach a mounting frame or a mounting surface (z. B. the floor of a warehouse). The robot 550 can be done in a similar way to the robot 350 of 3 work. The robot 550 for example, may include multiple motors configured to drive the robotic arm by rotating the limbs 554A-554E move relative to each other. The robotic arm can also include a robotic hand attached to the limb 554E is attached. 5A shows, for example, a first robot hand 555 , a second robotic hand 557 and a third robotic hand 559 each attached to the fifth link 554E attached and detached from it. The robot hand 555/557/559 can e.g. B. comprise a gripper or a suction device that is configured to pick up objects (e.g. 582A , 582B , 582C ) from a conveyor belt 573 to record. When the robot hand 555/557/559 on the fifth link 554E is attached, the attachment may be in mixed ways. Attachment and detachment can be done manually or automatically. In one example, the fifth link 554E on the first robot hand 555 attached as in and and the robot control system 110 can the robot 550 so control that the fifth link 554E the first robotic hand 555 releases and the fifth link 554E on the second robot hand 557 attached. In another embodiment, the fifth link 554E permanently on a robot hand (e.g. robot hand 559 ) be attached.

In one embodiment, the robot can 550 a verification symbol arranged on it 530 exhibit. In some cases the verification icon 530 permanently on the robot 550 be arranged. In some cases the verification icon 530 on a robot arm of the robot 550 such as on one of the links 554A-554E , or be arranged on a robot hand. For example shows 5A the verification icon 530 that on the first robot hand 555 and the third robotic hand 559 is arranged while 5B the verification icon 530 that on the limb 554E is arranged represents. The verification symbol 530 can directly on the robot 550 painted or on the robot 550 z. B. be attached by a sticker or a flat board. In the in 5A The example shown can be the second robot hand 557 or the third robot hand 559 can be used to perform the camera calibration, as they each have a corresponding calibration pattern 520/527 has while the first robot hand 555 or the third robot hand 559 can be used for camera calibration verification as each has a corresponding verification icon 530 having.

Referring again to FIG 4A in one embodiment, the method 400 with step 401 begin in which the control circuit 111 performs an initial camera calibration to determine camera calibration information associated with the camera (e.g., camera 170/270/370/570 of 1 , 2 , 3 or 5). In particular, the camera calibration information can also include camera calibration values for the camera. In this embodiment, the control circuit 111 carry out the first camera calibration based on images of a calibration pattern (also called calibration images).

For example, to perform the first camera calibration, the robot can 550 of 5A on the second robot hand 557 with the calibration pattern 520 or the third robot hand 559 with the calibration pattern 527 be attached. 3 shows a similar environment in which the first camera calibration can be performed. During this step, the first robot hand can 555 from the fifth link 554E , in which the calibration pattern 320 used to perform camera calibration. The first camera calibration can be performed before starting any robot operation. The robot operation can, for example, with a robot task such as the first robot hand 555 that with a first object 582A interacting on a conveyor belt, begin. During the first camera calibration, the robot can 550 with the second robot hand 557 be equipped. The robot control system 110 can use the robot arm of the robot 550 control via movement commands in such a way that it uses the calibration pattern 520 to different locations within a camera field of view 510 the camera 570 moved and corresponding images of the calibration pattern 520 be recorded in these locations. The robot control system 110 can perform the first camera calibration to get the camera calibration information for the camera 570 based on the captured images of the calibration pattern 520 to determine. The calibration information may include, for example, information relating to a relationship between the camera 570 and the robot 550 describe. In one example, the camera calibration information can be the intrinsic parameters of the camera 570 describe. Camera calibration is described in U.S. Application No. 16 / 295,940 (File Number MJ0021US1) entitled "METHOD AND DEVICE FOR PERFORMING AUTOMATIC CAMERA CALIBRATION FOR ROBOT CONTROL," the entire contents of which are incorporated herein by reference.

Referring again to FIG 4A can do the procedure 400 also the step 403 include in which the control circuit 111 controls the robot arm in such a way that it displays the verification symbol (e.g. 530 of 5 ) during or after the first camera calibration to a location within a camera's field of view (e.g. 510 ) the camera (e.g. 570 ) moved by sending a first movement command to the communication interface 113 of the robot control system 110 is issued. The communication interface 113 can be configured to communicate the move command to the robot to cause the robot arm to display the verification icon (e.g. 530 ) during or after the first camera calibration to the location within the camera's field of view (e.g. 510 ) to move. The move command can also cause the robot arm to orient the verification icon so that it faces the camera (e.g. 570 ), or more generally that it is visible to the camera. The location can be used as a reference location for one or more reference locations for verifying the first camera calibration. For example, if the verification process captures images of the verification icon over time, the control circuit may 111 control the robot arm in such a way that the verification symbol (e.g. 530 ) is consistently positioned at the same or multiple locations so that the one or more locations can be used as one or more reference locations. In addition, the verification process can be as follows in relation to the steps 405 to 459 later images of the verification symbol with a set of earlier images of the verification symbol (e.g. 530 ) compare, e.g. B. with images that were obtained immediately after performing the first camera calibration. The later images can be used as verification images, while the images with which the later images are compared can be used as reference images.

In step 405 can control circuit 111 via the communication interface 113 from the camera (e.g. 170/270/370/570 ) an image of the verification symbol (e.g. 530 ) receive (e.g. retrieve), the image being a reference image for verification. The image may have been captured by the camera while the verification symbol was or was at the reference location. In one embodiment, the communication interface can 113 first receive the reference image from the camera, and the control circuit 111 can then take the reference image from the communication interface 113 receive. In one embodiment, step 405 executed without the control circuit 111 generates a camera command for the camera. In one embodiment, the step 405 that involve the control circuit 111 a camera command is generated and the camera command is generated via the communication interface 113 communicates to the camera. The camera command can control the camera to capture an image of the verification symbol at the reference location.

The 5A to 6B illustrate aspects of the steps 403 and 405 . In the embodiment of 5A can the second robot hand 557 by the third robot hand 559 at which the verification symbol 530 is arranged to be replaced after the first camera calibration z. B. with the second robot hand 557 was executed. In this case, the robot control system controls 110 the robot arm of the robot 550 (e.g. by means of one or more movement commands) to the verification symbol 530 to one or more reference locations within the camera's field of view 510 the camera 570 to move. The one or more reference locations can be any location within the field of view of the camera 510 or may be a series of one or more specific locations, such as locations located on the face of an imaginary sphere, as discussed in more detail below. In another case, in the embodiment of 5B the robot control system 110 during or after the first camera calibration, control the robot arm in such a way that it displays the verification symbol 530 to one or more reference locations within the camera's field of view 510 emotional. In this case, the one or more reference locations can include any location where the verification symbol 530 (together with the calibration sample 520 ) was photographed during the initial camera calibration, or it can be a series of one or more specific locations that the verification icon 530 is moved after performing the first camera calibration. The robot control system 110 can move the robot arm of the robot 550 in this step with the guidance of the camera 570 based on the camera calibration information obtained from the first camera calibration, or can do so without such guidance. In one embodiment, the reference locations can be defined locations that are stored in a local or remote storage device and can be accessed. They can be in the form of coordinates (e.g. Cartesian coordinates) or as motor commands to rotate the limbs 554A to 554E or stored in some other way.

In one embodiment, the one or more reference locations to which the robotic arm places the verification icon (e.g. 530 ) includes multiple reference locations, each of the multiple reference locations being a location located on a surface of an imaginary sphere that is concave with respect to the camera. In such an embodiment, the control circuit 111 further configured to control the robot arm to move the verification symbol such that it is tangent to the surface of the imaginary sphere at each reference location of the plurality of reference locations. For example, as shown in the 6A , 6B , 6C and 6D illustrates the robot control system 110 the robot arm of the robot 550 control so that it has the verification symbol 530 to the reference locations 610A to 6101 moves, and the camera 570 control so that they are at each of the reference locations 610A to 6101 captured a corresponding reference image. The reference locations 610A to 6101 in 6A and 6B can point to several imaginary spheres within the camera's field of view 510 be divided. The reference locations 610A and 610B can on a first spherical surface 621 a first imaginary sphere 620 be arranged where the first spherical surface 621 within the camera's field of view 510 is located. The reference locations 610C , 610D and 610E can on a second spherical surface 631 a second imaginary sphere 630 be arranged where the second spherical surface 631 within the camera's field of view 510 is located. The reference locations 610F , 610G , 610H and 6101 can on a third spherical surface 641 a third imaginary sphere 640 be arranged where the third spherical surface is 641 within the camera's field of view 510 is located. As in the 6A and 6B illustrated are the first, second and third spherical surfaces 621 , 631 and 641 in terms of the camera 570 concave. Although the examples in the 6A and 6B show three spherical surfaces based on three spheres, a number of different spherical surfaces on which reference locations can be arranged can be larger than three or smaller than three. In one embodiment, the camera 570 a center of each of the imaginary spheres 620 , 630 , 640 be.

In one embodiment, as shown in 6A to 6D illustrated, the robot control system can 110 when the verification icon 530 is moved to a reference location, the robot arm 553 of the robot 550 Control (e.g. via a movement command) in such a way that the verification symbol 530 is positioned tangentially to the spherical surface on which the reference location is arranged. The 6A and 6B illustrate, for example, the verification icon 530 the second sphere surface 631 at the reference location 610D affected, while the 6C and 6D the verification icon 530 illustrate which is the second spherical surface 631 at the reference location 610C affects. In particular, the verification symbol 530 be placed on a flat surface (e.g. on a sticker) and the flat surface of the verification symbol 530 can at the reference location 610D in the 6A and 6B the second spherical surface 631 tangent and at the reference location 610C in the 6C and 6D the second spherical surface 631 affect. In one embodiment, the robotic arm can 553 in the 6A and 6B in a first pose and in the 6C and 6D are in a second pose. A pose of the robotic arm 553 can z. B. refer to a shape or more generally to a geometry that is created by limbs or other arm parts of the robot arm 553 is formed. The pose of a robotic arm 553 can refer, for example, to a certain permutation of angles or distances around which limbs of the robotic arm move 553 relative to previous limbs of the robot arm 553 rotated or translated (e.g. extended or retracted). As an example, the in 6A The first pose shown correspond to a first permutation of angles that exist between successive members of the series of members of the robot arm 553 is formed, while the in 6C illustrated second pose of a second permutation of angles between successive links of the series of links of the robot arm 553 can correspond. In such an example, the reference location can be 610D for the verification icon 630 with the first pose for the robotic arm 553 as in 6A is illustrated, be connected while the reference location 610C for the verification icon 630 with the second pose for the robotic arm 553 as in 6C is shown, can be connected.

In one embodiment, the control circuit is 111 configured to control the robot arm in such a way that it displays the verification symbol (e.g. 530 ) is moved in such a way that it is directly facing the camera when the verification symbol is moved to a reference location. For example, as in 6A illustrates the robot control system 110 the robotic arm 553 of the robot 550 control so that it has the verification symbol 530 so too moved that it is directly facing the camera 570 is facing when the verification icon 530 to the reference location 610D is moved. In this example, the robot control system 110 the robot hand 555 Rotate so that the verification icon 530 directly from the camera 570 is facing. In some cases the verification icon can be directly on the camera 570 face by placing a spherical surface in the camera's field of view 510 affects. When the verification icon 530 directly from the camera 570 facing can the camera 570 be able to see the verification icon 530 photograph frontally so that in a resulting image of the verification symbol 530 no or a reduced perspective effect occurs.

In one embodiment, the verification icon (e.g. 530 ) a first area having a first color and a second area having a second color, defining a ratio of an area of the first region to an area of the second region, and in a non-transitory, computer-readable medium (e.g., storage device) of the robot control system 110 is stored. In such an embodiment, the control circuit 111 be configured to identify the verification symbol in a reference image or a verification image based on the defined relationship. For example, as in 5C illustrates the verification icon 530 a first region 531 that is ring-shaped and has a first color (e.g., a black region) and a second region 533 (for example, a white region) by the first region 531 is enclosed and has a second color. The ratio of an area of the first region 531 in black to an area of the second region 533 in white in the verification symbol 530 can be a clearly defined value. By analyzing the colors within a captured image, the robot control system can 110 be able to identify part of the image that corresponds to the verification symbol 530 corresponds by determining whether this part has an annular region enclosing a circular region and whether a ratio between an area of the annular area and an area of the circular area corresponds to the defined ratio. It can do this to the robot control system 110 enable the verification icon 530 to distinguish from other features captured in an image. For example, the robot can 550 as in 5A illustrated to be set so that he is the third robot hand 559 uses a combination of a calibration pattern 527 and a verification symbol 530 having. In this example, a reference image can have both the verification symbol 530 as well as the calibration pattern 527 demonstrate. In this example the calibration pattern 527 do not have a ring pattern or a ring pattern with any other than the defined ratio described above. The control circuit 111 can determine whether part of the reference image has the verification symbol 530 or the calibration pattern 527 is by determining whether the part of the reference image has a first image region having the first color and a second image region having the second color, and whether a ratio between an area of the first image region and an area of the second image region is equal to the defined ratio.

In some cases the robot control system can 110 determine whether a certain part of a captured image has a first area with a first color and a second area with a second color and whether a ratio between an area of the first region and an area of the second region is within a defined range. In one example, the robotic control system 110 If the defined ratio is 1.5, determine that the particular region corresponds to the verification symbol 530 if the ratio falls in the range between 1.4 and 1.6 in the particular region. The two colors of the first and second regions are not limited to black and white and can be any two different colors used by the robot control system 110 can be distinguished.

und ein Zentrum der zweiten Form 537 bei einer Koordinate $\left(u_{symbol}^{inner},{\nu }_{symbol}^{inner}\right)$

befindet, können die Koordinate $\left(u_{symbol}^{outer},{\nu }_{symbol}^{outer}\right)$

und die Koordinate $\left(u_{symbol}^{inner},{\nu }_{symbol}^{inner}\right)$

im Wesentlichen gleich sein.In one aspect, the verification symbol (e.g. 530 ) comprise a first shape and a second shape that are concentric with each other, with the respective centers of the first shape and the second shape being at substantially the same location. For example, the verification symbol may be in the form of a circular ring comprising first and second circles that are concentric with one another. In particular, the verification symbol 530 as in 5C illustrates a first form 535 (e.g. outer circle) and a second shape 537 (e.g. inner circle). The first form 535 and the second form 537 can be concentric to each other so that a center of the first form 535 and a center of the second shape 537 are essentially in the same place. For example, if there is a center of the first shape 535 at a coordinate $\left(u_{symbOl}^{Outer},{\nu }_{symbOl}^{Outer}\right)$

and a center of the second shape 537 at a coordinate $\left(u_{symbOl}^{inner},{\nu }_{symbOl}^{inner}\right)$

can be the coordinate $\left(u_{symbOl}^{Outer},{\nu }_{symbOl}^{Outer}\right)$

and the coordinate $\left(u_{symbOl}^{inner},{\nu }_{symbOl}^{inner}\right)$

essentially be the same.

Referring again to FIG 4A can do the procedure 400 also the step 407 include in which the control circuit 111 determines a reference image coordinate for the verification symbol, the reference image coordinate being a coordinate at which the verification symbol (e.g. 530 ) appears in the reference image. For example, as in 6A illustrates an image of the verification icon 530 at the reference location 610D captured and used as a reference image. The verification symbol 530 may appear within the reference image at a particular coordinate that can be referred to as the reference image coordinate.

In one embodiment, the verification symbol (e.g. 530 ) comprise a first shape and a second shape that are concentric with each other with respective centers of the first shape and the second shape being at substantially the same location. In such an embodiment, the control circuit 111 in step 407 be configured to determine the reference image coordinate for such a verification symbol by: determining a first coordinate of a center of the first shape in the reference image, determining a second coordinate of a center of the second shape in the reference image and determining the reference image coordinate as an average of the first coordinate and the second Coordinate in the reference image.

eines Zentrums der ersten Form 735 (d. h. eine Mittenkoordinate der ersten Form 735), die im Referenzbild 710 gezeigt ist, und eine zweite Koordinate $\left(u_{ref\_N}{}^{inner},{\nu }_{ref\_N}{}^{inner}\right)$

eines Zentrums der zweiten Form 737 (d. h. eine Mittenkoordinate der zweiten Form 737), die im Referenzbild 710 gezeigt ist, bestimmen. Um die Referenzbildkoordinate $\left(u_{ref\_N},{\nu }_{ref\_N}\right)$

für das Referenzbild 710 als Ganzes zu bestimmen, wobei das Referenzbild 710 dem Verifizierungssymbol 530 an dem Referenzort N entspricht, kann das Robotersteuersystem 110 einen Mittelwert der ersten Koordinate $\left(u_{ref\_N}{}^{outer},{\nu }_{ref}{}_N^{outer}\right)$

und der zweiten Koordinate $\left(u_{ref\_N}{}^{inner},{\nu }_{ref\_}{}_N^{inner}\right)$

im Referenzbild 710 wie folgt berechnen: $$\left(u_{ref\_N},{\nu }_{ref\_N}\right)=\left(\frac{u_{ref\_N}{}^{outer}+u_{ref\_N}{}^{inner}}{2},\frac{v_{ref\_N}{}^{outer}+v_{ref\_N}{}^{inner}}{2}\right)$$

For example shows 7A a reference image 710 recorded at a reference location N (where N is an integer) of the reference locations. The reference image 710 includes a verification section 730 , which is an image section in the reference image 710 is that of the verification icon 530 of 5A , 5B or 5C shows. The robot control system 110 of 1A or 1B can be configured from the verification section 730 a first form 735 Identify (e.g. outer circle) that is equal to the first shape 535 of the verification symbol 530 of 5C is or is substantially the same. The robot control system 110 can be configured from the verification section 730 a second form 737 Identify (e.g. inner circle) that is equal to the second shape 537 of the verification symbol 530 in 5C or is essentially the same. Then the robot control system can 110 a first coordinate for the reference location N. $\left(u_{ref\_N}{}^{Outer},{\nu }_{ref}{}_N^{Outer}\right)$

a center of the first form 735 (ie a center coordinate of the first shape 735 ) in the reference image 710 and a second coordinate $\left(u_{ref\_N}{}^{inner},{\nu }_{ref\_N}{}^{inner}\right)$

a center of the second shape 737 (ie a center coordinate of the second shape 737 ) in the reference image 710 is shown. To the reference image coordinate $\left(u_{ref\_N},{\nu }_{ref\_N}\right)$

for the reference image 710 as a whole to be determined, the reference image 710 the verification symbol 530 at the reference location N, the robot control system 110 an average of the first coordinate $\left(u_{ref\_N}{}^{Outer},{\nu }_{ref}{}_N^{Outer}\right)$

and the second coordinate $\left(u_{ref\_N}{}^{inner},{\nu }_{ref\_}{}_N^{inner}\right)$

in the reference image 710 calculate as follows: $$\left(u_{ref\_N},{\nu }_{ref\_N}\right)=\left(\frac{u_{ref\_N}{}^{Outer}+u_{ref\_N}{}^{inner}}{2},\frac{v_{ref\_N}{}^{Outer}+v_{ref\_N}{}^{inner}}{2}\right)$$

In one embodiment, the reference image coordinate for the verification symbol can be its center coordinate, and the determination of the center coordinate of the verification symbol 530 based on the corresponding center coordinates of the first shape 735 and the second form 735 can improve a robustness of the verification process against image noise. Image noise can e.g. B. to errors in determining a center coordinate for the first shape 735 lead, but not when determining the center coordinate for the second shape 737 . In some cases it can be the second form 737 in reality the same center as the first form 735 have, but image noise can cause the center coordinate of the second shape 737 appears different in an image than the center coordinate of the first shape 735 . In this Scenario can be to simply use the center coordinate of the second shape 737 as the center coordinate of the verification symbol 530 lead to an undesirable amount of errors. The amount of error can be reduced by taking an average of the center coordinate for the first shape 735 and the center coordinate for the second shape 737 as the center coordinate of the verification symbol 530 is used.

In one embodiment, the one or more reference locations discussed above may be multiple reference locations each corresponding to multiple reference image coordinates. In this embodiment, the reference image coordinate can be one of a plurality of reference image coordinates. For example, as in 6A and 6B illustrated, give several reference locations, such as the reference locations 610A to 6101 to which the verification symbol 530 moved or otherwise placed. For each of the reference locations 610A to 6101 of the verification symbol 530 can the robot control system 110 retrieve or otherwise receive a corresponding reference image from the camera 570 of the verification symbol 530 was captured at this location and can determine a corresponding reference image coordinate that indicates where the verification symbol 530 appears in the corresponding reference image.

Referring again to FIG 4A can do the procedure 400 also the step 409 include in which the control circuit 111 controls the movement of the robot arm based on the camera calibration information to perform a robot operation. In one embodiment, this step may involve the control circuit 111 generates a second movement command based on the camera calibration information and the second movement command to the communication interface 113 issues. The communication interface 113 can in turn communicate the second movement command to the robot in order to control the movement of the robot arm. As in 5A shown controls the robot control system 110 after the first camera calibration, for example, the robot 550 such that it performs a robot operation that performs robot tasks such as picking up objects 582A , 582B and 582C involves. The movement of the robot 550 can access the camera calibration information obtained from the initial camera calibration and that from the camera 570 captured images of the objects 582A , 582B , 582C based.

In step 411 the control circuit detects 111 a rest period during robot operation. In one aspect, the robot idle time may be a period of time during which the robot is free from performing a robot task during robot operation. When the robot operates on picking up objects from the conveyor belt 573 In some cases, the idle time may be based on an absence of objects on the conveyor belt 573 based. In particular, the conveyor belt 573 from the robotic arm 553 be accessible and the control circuit 111 is configured to detect the idle time by determining that there are no objects on the conveyor belt 573 or that there is a gap between the robot 550 and a closest object on the conveyor belt 573 exceeds a defined distance threshold. In some cases the control circuit can 111 receive a signal indicating that a rest period is imminent, which signal may be received by another device or component that is monitoring robot operation. As in 5A illustrated, the robot control system can 110 for example, a rest period between a robot task in which the second object 582B is recorded, and a robot task in which the third object 582C is recorded, detect during the robot operation, as a large distance between the second object 582B and the third object 582C exists. During this rest period after the robot 550 the second object 582B has recorded, he can have a rest period in which he is free from performing a robot task because the object 582C for the robot 550 is not yet available. In one example, the robot control system can 110 Detect the idle time when there is no object on the conveyor belt 573 for the robot 550 is reachable, and / or when the robot control system 110 that determines a distance between the robot 550 and the closest object (e.g. the third object 582C ) upstream on the conveyor belt 573 exceeds a certain threshold.

Referring again to FIG 4A and 4B can do the procedure 400 also a step 451 include where the control circuit 111 the robotic arm 553 so controls that it controls the verification symbol 530 during the rest period at least to that in step 403 used reference location (which was used to acquire the reference image). In one embodiment, step 451 involve that control circuit 111 a third movement command is generated and the third movement command is sent to the communication interface 113 issues. The communication interface 113 can be configured to then communicate the third movement to the robot to the robot arm 553 cause to move based on the move command. In some cases, the third move command may include a set of stored motor commands that correspond to the reference location. In some cases, the third move command may be based on the camera calibration information from step 401 be generated. In other cases, the third movement command is based in step 451 not on the camera calibration information from step 401 .

In step 453 calls the control circuit 111 an additional image of the verification symbol (e.g. 530 ) from the camera (e.g. 570 ) during the idle time or otherwise receives it, wherein the additional image is a verification image for the verification and an image of the verification symbol is at least at the reference location during the idle time. That is, the verification image for the reference location is captured while the verification symbol (e.g. 530 ) is or has found at the reference location. In one embodiment, step relates 453 one that the control circuit 111 generates a camera command that the camera (e.g. 570 ) controls to capture the verification image. The control circuit 111 can send the camera command to the communication interface 113 output that sends the camera command to the camera (e.g. 570 ) can communicate. In one embodiment, step 451 involve controlling the robotic arm to move the verification icon to multiple reference locations and receiving multiple corresponding verification images captured by the camera. As for example in the 6A and 6B illustrated, the robot control system can 110 the robot arm during a rest period 553 of the robot 550 control so that it has the verification symbol 530 to one of the reference locations 610A to 6101 moved and an image of the verification icon 530 captured at the location as a verification image. If the rest period has not yet expired, or in particular if a sufficient amount of time remains in the rest period, the robot control system can 110 the robotic arm 553 of the robot 550 control so that it has the verification symbol 530 to another of the reference locations 610A to 6101 moved and an image of the verification icon 530 captured at this other location as another verification image. When the rest period ends, the robot control system can 110 stop capturing verification images. Therefore, the robot control system can 110 the robot arm during each rest period 553 of the robot 550 control so that it has the verification symbol 530 to one or more of the reference locations 610A to 6101 moved and at each of the one or more reference locations 610A to 6101 captures a verification image.

Referring again to FIG 4B can do the procedure 400 also the step 455 include in which the control circuit 111 determines a verification image coordinate used for the verification, the verification image coordinate being a coordinate at which the verification symbol appears in the verification image. If the verification symbol (e.g. 530 ) to several reference locations (e.g. 610A to 610I ) is moved, the camera (e.g. 570 ) take a plurality of verification images each corresponding to the plurality of reference locations and the control circuit 111 can determine a plurality of verification image coordinates each corresponding to the plurality of verification images and each corresponding to the plurality of reference locations. The multiple verification images can be captured by the camera (e.g. 570 ) in a single rest period (e.g. if the single rest period is long enough to allow the robot arm to move the verification symbol (e.g. 530 ) to all reference locations 610A to 610I to allow) or in several different rest periods (e.g. if each of the rest periods is not long enough for the robot arm to display the verification symbol 530 to all reference locations 610A to 610I can move).

der ersten Form 785, die im Verifizierungsabschnitt 780 des Verifizierungsbilds 760 gezeigt wird, und eine Mittenkoordinate $\left(u_{verifyy\_N}^{inner},v_{verifyy\_N}^{inner}\right)$

der zweiten Form 787, die im Verifizierungsabschnitt 780 des Verifizierungsbilds 760 gezeigt wird, zu bestimmen. Das Robotersteuersystem 110 kann ferner die Verifizierungsbildkoordinate $\left(u_{verify\_N},{\nu }_{verify\_N}\right)$

für das Verifizierungsbild 760 als einen Mittelwert der Mittenkoordinate der ersten Form 785 und der Mittenkoordinate der zweiten Form 787 im Verifizierungsbild 760 wie folgt bestimmen: $$\left(u_{verify\_N},{\nu }_{verefy\_N}\right)=\left(\frac{u_{verify\_N}^{outer}+u_{verify\_N}^{inner}}{2},\frac{v_{verify\_N}^{outer}+v_{verify\_N}^{inner}}{2}\right)$$

In one embodiment, the verification image coordinate can be determined in a manner similar to the reference image coordinate. The verification image coordinate can, for example, be a center coordinate of the verification symbol (e.g. 530 ) and as an average of a center coordinate of a first form of the verification symbol (e.g. 530 ) and a center coordinate of a second form of the verification symbol in the verification image (e.g. 760 ) can be determined. For example shows 7B a verification picture 760 that was recorded at the reference location N of the reference locations. The verification picture 760 shows a verification section 780 that is an image portion in the verification image 760 is that of the verification icon 530 shows. The robot control system 110 can from the verification section 780 a first form 785 identify the same as the first shape 585 of the verification symbol 530 of 5C or is essentially the same. The robot control system 110 can also from the verification section 780 a second form 787 identify the same or substantially the same as the second shape 587 of the verification symbol 530 is. The robot control system 110 can also be configured to have a center coordinate $\left(u_{verifyy\_N}^{Outer},v_{verifyy\_N}^{Outer}\right)$

the first form 785 that is in the verification section 780 of the verification image 760 is shown, and a center coordinate $\left(u_{verifyy\_N}^{inner},v_{verifyy\_N}^{inner}\right)$

the second form 787 that is in the verification section 780 of the verification image 760 is shown to determine. The robot control system 110 can also be the verification image coordinate $\left(u_{verify\_N},{\nu }_{verify\_N}\right)$

for the verification picture 760 as an average of the center coordinate of the first shape 785 and the center coordinate of the second shape 787 in the verification image 760 determine as follows: $$\left(u_{verify\_N},{\nu }_{verefy\_N}\right)=\left(\frac{u_{verify\_N}^{Outer}+u_{verify\_N}^{inner}}{2},\frac{v_{verify\_N}^{Outer}+v_{verify\_N}^{inner}}{2}\right)$$

und die Verifizierungsbildkoordinate am Referenzort N als $\left(u_{verify\_n},{\nu }_{verify\_N}\right)$

v_{verify_N}) ausgedrückt wird, kann eine Abweichung (z. B. Abstand) am Referenzort N beispielsweise als $\sqrt{{\left(u_{ref\_n}-u_{verify\_N}\right)}^2+{\left({\nu }_{ref\_n}-{\nu }_{verify\_N}\right)}^2}.$

ausgedrückt werden.Referring again to FIG 4B can do the procedure 400 also the step 457 include in which the control circuit 111 a deviation parameter value based on an amount of deviation between the reference image coordinate of step 403 and the verification image coordinate of step 455 is determined, the reference image coordinate and the verification image coordinate both being connected to the reference location N. In one example, the deviation between the reference image coordinate and the verification image coordinate may be a distance between the reference image coordinate and the verification image coordinate. Since the reference image coordinate at the reference location N as $\left(u_{ref\_N},{\nu }_{ref\_N}\right)$

and the verification image coordinate at the reference location N as $\left(u_{verify\_n},{\nu }_{verify\_N}\right)$

v _{verify_N} ) is expressed, a deviation (e.g. distance) at the reference location N can be, for example, as $\sqrt{{\left(u_{ref\_n}-u_{verify\_N}\right)}^2+{\left({\nu }_{ref\_n}-{\nu }_{verify\_N}\right)}^2}.$

be expressed.

As described above, in one aspect where the one or more reference locations are multiple reference locations, the control circuit 111 be configured to determine the plurality of verification image coordinates each corresponding to the plurality of reference locations at which the verification image coordinate described above is one of the plurality of verification image coordinates. In such an aspect, the deviation parameter value is based on corresponding deviation amounts between the plurality of reference image coordinates and the plurality of verification image coordinates for the plurality of reference locations, wherein each deviation amount of the respective deviation amounts is between: (a) a reference image coordinate corresponding to a corresponding reference location of the plurality of reference locations and (b ) a verification image coordinate that corresponds to the same reference location. The plurality of verification image coordinates may be corresponding coordinates at which the verification symbol appears in a plurality of verification images, with the verification image discussed above being one of the plurality of verification images. The control circuit 111 can be configured to control the camera to capture all of the plurality of verification images in a rest period, and / or it can be configured to control the camera to capture the plurality of verification images in different rest periods.

For example, if there are multiple reference locations, as in the 6A and 6B shown, the robot control system can 110 determine a plurality of corresponding reference image coordinates that correspond to the plurality of reference locations and determine a plurality of corresponding verification image coordinates that correspond to the plurality of reference locations, and determine respective deviation amounts between the plurality of reference image coordinates and the plurality of verification image coordinates. The deviation parameter value may be based on the respective deviation amounts between the plurality of reference image coordinates and the plurality of verification image coordinates. The deviation parameter can, for example, be an average value of the corresponding deviation amounts as follows. $$\mathrm{A.}bweicHunGsparameterwert=\frac{{\displaystyle {\sum }_{N=1}^{\mathrm{M.}}\sqrt{{\left(u_{ref\_n}-u_{verify\_N}\right)}^2+{\left({\nu }_{ref\_n}-{\nu }_{verify\_N}\right)}^2.}}}{\mathrm{M.}}$$

In the above expression, N may refer to the Nth reference location, while M may refer to a total number of reference locations.

Referring again to FIG 4B can do the procedure 400 also the step 459 include in which the control circuit 111 determines whether the deviation parameter value exceeds a defined threshold (which can also be referred to as a defined deviation threshold). The control circuit 111 can also be done in step 461 in response to a determination that the deviation parameter value exceeds the defined threshold, perform a second camera calibration to determine updated camera calibration information for the camera. For example, the deviation parameter value that exceeds the defined threshold may indicate that the camera's camera calibration information is out of date and / or likely to cause an undesirable amount of error in robot operation. Therefore, if the deviation parameter value exceeds the defined threshold, a second camera calibration can be performed for the camera to update the camera calibration information for the camera (e.g. 570 ). The second camera calibration may use the same techniques as the first camera calibration, but may be based on images recently captured by the camera. In one example, robot operation can be stopped or paused when step 459 indicates that the deviation parameter value exceeds the defined threshold and then continues to perform the second camera calibration, which can begin capturing images for the second camera calibration. After the second camera calibration is complete and the camera calibration information for the camera is updated, the robot control system can 110 resume robot operation using the updated camera calibration information.

In one embodiment, the control circuit 111 in response to determining that the deviation parameter value does not exceed the defined threshold value, be configured to control the robot to continue robot operation after the rest period without performing additional camera calibration (e.g. by issuing a fourth movement command to the robot via the communication interface). Such a condition may indicate that the camera calibration information is from step 401 are still sufficiently accurate and that robot operation can continue without an undesirable amount of error.

In one embodiment, the control circuit 111 be configured to determine the temperature of an environment in which the robot is located and to adjust at least one of the defined deviation threshold (also referred to as redefining the deviation threshold) or the camera calibration information for the camera based on the measured temperature. For example, the control circuit 111 determine the temperature of the environment by measuring the temperature or receiving temperature data from another device or component. In such an embodiment, the control circuit 111 be configured to adjust the defined threshold based on the measured temperature by: adjusting the defined threshold to have a first value when the measured temperature is outside a defined range and adjusting the threshold to have a second value, which is lower than the first value if the measured temperature is within the defined range. For example, an excessively high or excessively low temperature can cause changes in the camera. In particular, a change in temperature can influence the intrinsic parameters of the camera. For example, the components in the camera can expand when the temperature increases and contract when the temperature decreases, which can affect the intrinsic parameters of the camera. It can therefore be advantageous to adapt the defined deviation threshold value based on the temperature or the extent of the temperature change. For example, if the temperature is in a range of normal operating temperature (e.g. a defined range around room temperature), then the defined deviation threshold value can be lower, since the temperature does not adversely affect the camera. On the other hand, if the temperature is outside the range of a normal operating temperature, the deviation threshold may be higher because a cold or hot temperature will adversely affect the camera. In an alternative example, the deviation threshold can be defined with a lower value if the temperature is outside of a normal operating temperature in order to trigger additional camera calibration more frequently. In this example, the deviation threshold can be set to a higher value when the temperature is within normal operating temperature in order to trigger additional camera calibration less frequently.

8th shows an exemplary timeline 800 where camera calibration and camera calibration verification are performed. While the following examples describe verifying camera calibration, they can be applied more generally to verifying any type of calibration process performed on a robot operating system. Before a robot operation begins, the robot control system performs 110 of 1A or 1B the first camera calibration to display the camera calibration information for the camera (e.g. camera 570 of 5A or 5B) during a calibration period 811 to determine. After the first camera calibration is complete, the robot control system captures 110 Reference images of the verification symbol (e.g. verification symbol 530 ) at different reference locations and determined during a reference recording period 813 Reference image coordinates at which the verification symbol in corresponding reference images (e.g. reference image 710 of 7A) appears. The robot can operate after the end of the reference detection period 813 begin after determining the reference image coordinates.

After the robot starts operating, during a task period 815 , controls the robot control system 110 the robot (e.g. robot 550 of 5A or 5B) such that it is performing one or more robotic tasks and therefore, in one embodiment, may not be able to collect verification images (e.g., verification image 760 of 7B) . The robot control system 110 detected a rest period 817 during which the robot after the task period 815 is free from performing a robot task. Therefore, the robot control system detects 110 during the rest period 817 correspondingly one or more verification images of the verification symbol in a first row of one or more locations (e.g. 610A-610B ) of the reference locations. After the end of the rest period 817 , during a task period 819 , adopts the robot control system 110 regains control of the robot to perform one or more robot tasks and therefore may not collect verification images. The robot control system 110 detects a rest period 821 during which the robot after the task period 817 is free from performing a robot task. During the rest period 821 captures the robot control system 110 corresponding one or more verification images of the verification symbol in a second row of one or more locations (e.g. 610C-610E ) of the reference locations. After the rest period 821 , during a robot task period 823 , adopts the robot control system 110 regains control of the robot to perform one or more robot tasks and therefore may not collect verification images. The robot control system 110 detects a rest period 825 during which the robot after the task period 823 is free from performing a robot task. During the rest period 825 captures the robot control system 110 corresponding one or more verification images of the verification symbol in a third row of one or more locations (e.g. 610F-610I ) of the reference locations.

The verification images (e.g. 760 ) during rest periods 817 , 821 and 825 can be recorded at different locations of the reference locations. For example, the first, second and third rows of one or more locations can differ from one another and cannot overlap at the locations. Furthermore, the robot control system 110 during the rest period 825 determine that the verification image acquisition is complete, which may indicate that a sufficient number of verification images have been acquired for camera calibration verification. In one embodiment, the robotic control system can 110 determine that verification image capture is complete when all reference locations (e.g. 610A-610I ) Verification images were captured. In one embodiment, the robotic control system can 110 determine that the verification image acquisition is complete when a number of the verification images reaches a defined target number.

After determining that the verification image acquisition is complete, the robot control system determines 110 the verification image coordinates at which the verification symbol appears in the corresponding verification images. The robot control system then determines 110 a deviation parameter value based on the respective deviation amounts of the verification image coordinates from the reference image coordinates. If the deviation parameter exceeds a defined threshold value, the robot control system leads 110 another camera calibration. In this example, however, the deviation parameter does not exceed the defined threshold value, so the robot control system 110 during a task duration 827 after the rest period 825 continues to perform a robot task without performing additional camera calibration.

9 shows an exemplary flow chart 900 which has a verification process related to the timeline in 8th shows. In step 901 runs the robot control system 110 of 1A , 1B or 1C the first camera calibration of the camera (e.g. camera 570 of 5A or 5B) to determine the camera's camera calibration information. In step 903 controls the robot control system 110 the robot (e.g. robot 550 of 5A or 5B) to select the verification icon (e.g. verification icon 530 of 5A or 5B) to move to reference locations, and uses the camera to capture corresponding instances of the reference images (e.g. reference image 710 of 7A) the verification symbol at appropriate reference locations. In step 905 the robot control system begins 110 a robot operation of the robot based on the camera calibration information obtained from the first camera calibration.

In step 907 detects the robot control system 110 a rest period during robot operation. In step 909 controls the robot control system 110 the robot (e.g. robot 550 of 5A or 5B) so that he has the verification icon (e.g. the verification icon 530 of 5A or 5B) moved to one or more locations of the reference locations during the rest period and one or more verification images (e.g. the verification image 760 of 7B) recorded accordingly at the one or more locations of the reference locations. In some cases the robot control system can 110 control the robot in such a way that it moves the verification symbol to as many reference locations as the duration of the rest period allows. In step 911 determines the robot control system 110 whether a total number of the recorded verification images has reached a defined target number. If the total number of verification images captured has not reached the target number, the robot control system tries 110 to detect another subsequent idle time during robot operation by going to step 907 returns to capture more verification images.

When the total number of verification images captured has reached the target number, the robot control system leads 110 in step 913 a camera calibration verification based on the reference images (e.g. 710 ) and the verification images (e.g. 760 ) out. The camera calibration verification generates a deviation parameter. In step 915 determines the robot control system 110 whether the deviation parameter exceeds a defined threshold value. If the deviation parameter does not exceed the threshold, the robot control system can 110 the total number of verification images captured in step 919 Reset to zero and continue robot operation after the sleep time while it tries to detect another sleep time to capture a new set of verification images by going to step 907 returns.

If the deviation parameter exceeds the threshold, the robot control system can 110 stop the robot operation and at step 917 perform a second camera calibration. After the second camera calibration at 917 can the robot control system 110 the total number of verification images captured 921 reset to zero. After step 921 can step the flowchart too 903 return to where the robot control system 110 the robot (e.g. 550 ) controls in such a way that it displays the verification symbol (e.g. 530 ) moved to the reference locations and using the camera (e.g. 570 ) a new set of reference images (e.g. 710 ) of the verification symbol is recorded at the corresponding reference locations so that the new set of reference images can later be used for verification.

As indicated above, one aspect of the present disclosure includes receiving a reference image (e.g., reference image 710 of 7A) and a verification image (e.g., verification image 760 of 7B) that both have a common verification symbol (e.g. the verification symbol 530 the 6A to 6D ) capture or otherwise represent that in a physical location on a robotic arm such as the robotic arm 553 of the robot 550 the 6A to 6D is arranged. In one embodiment, the reference image (e.g. 710 ) at a first point in time, such as E.g. an earlier point in time, while the verification image (e.g. 760 ) z. B. can be a new image that at a second point in time, such as. B. at a later point in time. In some cases, the reference image can be generated based on one or more commands (also referred to as one or more commands) provided by a computer system, such as a computer system. B. the robot control system 110 of 1C , be generated. In some cases, the one or more commands may include a motion command and / or a camera command. The move command can serve to cause the robot arm to move the verification symbol to a reference location. For example, the move command can cause the robot arm to assume a particular pose, which results in the verification symbol being moved to a reference location associated with the particular pose of the robot arm. The camera command can be used to set a camera such as B. the camera 570 the 6A to 6D to cause the reference image to be generated while the robot arm is in the particular pose and / or while the verification symbol is at the reference location associated with the particular pose. The reference image can therefore correspond to the reference location. Similarly, the verification image can be generated based on one or more commands, e.g. B. a movement command and / or camera command, which can be the same or similar to the movement command and / or camera command described above with respect to the reference image. The one or more commands for the verification image can be issued by the same computer system that caused the generation of the reference image, e.g. B. the robot control system 110 , or generated by another computer system. In this example, the one or more commands for generating the verification image can cause the verification symbol to be moved back to the reference location. The one or more commands can, for example, cause the robot arm to assume the specific pose that is associated with the reference location again.

In the above example, the computer system receiving the verification image can compare it to the reference image to determine whether the calibration information is still sufficiently accurate. The calibration information can be determined through a calibration process performed for a robot operating system. The calibration process, which can also be referred to as a system calibration, can include a camera calibration, a robot calibration or any other calibration for controlling one or more components of the robot operating system. In one embodiment, the calibration information can include camera calibration information, robot calibration information, or other calibration information. The comparison described above can e.g. Include determining a deviation parameter value based on a difference between the location where the verification symbol appears in the reference image and the location where the verification symbol appears in the verification image. If the deviation parameter value is too large, e.g. For example, if it exceeds a defined deviation threshold (e.g., a predefined deviation threshold), then the computer system can determine that the calibration information is no longer sufficiently accurate are. In such a situation, the calibration information may be referred to as reflective or including mis-calibration or misalignment. If the camera calibration information includes the mis-calibration or misalignment, it may no longer accurately describe an inherent property of the camera (e.g. a projection property or a lens distortion property) and / or no longer adequately describe a relationship between the camera and its external environment ( e.g. a spatial relationship between the camera and a base of the robot). If the robot calibration information includes mis-calibration or misalignment, it may no longer be reliable for accurately moving a robot arm or other component of the robot to a desired location and / or orientation.

In one embodiment, the above-described method of comparing the reference image and the verification image may include multiple verification symbols. The 10A to 10C for example represent a group of several verification symbols 530A to 530C In particular, the group of verification symbols 530A to 530C Part of a robot operating system 500A be that one embodiment of the robot operating system 500 can be. The robot operating system 500A can be a robot 550A (the one embodiment of the robot 550 can be), a camera 570 with a camera field of view 510 and a computer system such as B. the robot controller 110 , include. Like the robot 550 also instructs the robot 550A a robotic arm 553 comprising a plurality of arm parts which are movably attached to one another. The multiple arm parts can, for. B. the limbs 554A to 554E and a robot end effector such as B. a robot hand 555 that attached to the limb 554E is attached, include. In some cases, as in the 10A to 10C as shown, the multiple arm portions can be formed from a base 552 of the robot 550A up to the robot end effector (e.g. robot hand 555 ) connected or otherwise arranged in series. In such cases, the series of arm parts can form a kinematic chain in which the movement of a particular arm part in the series can cause movement of some or all of the arm parts after that particular arm part. The arm parts after the particular arm part may refer to arm parts that follow the particular arm part in the series of arm parts. The limbs 554B to 554E and the robotic hand 555 can for example look after the limb 554A are located. The Member 554A in other words, can stand in front of the limbs 554B to 554E and in front of the robot hand 555 are located. In this example, each arm portion in the row of arm portions or in a subset thereof may be rotatable, extendable, retractable or otherwise movable relative to a corresponding arm portion immediately preceding the arm portion in the row of arm portions. The Member 554C can for example be relative to the limb 554B be rotatable, the limb 554B can be an arm part that the limb 554C in the in 10A immediately precedes the series of arm parts shown. In the example of the 10A to 10C can the limbs 554A to 554E and the robotic hand 555 about the joints 556A to 556D be movably attached to each other.

As described above, a robotic arm (e.g. 553 ) are moved into different poses, wherein a pose can refer to a shape or more generally to a geometry that is formed by arm parts (e.g. limbs) of the robot arm. The 10A to 10C for example, pose three different poses for the robotic arm 553 In one embodiment, the robot 550A in the 10A to 10C one or more actuators (e.g., motors) configured to operate the members 554A to 554E and the robotic hand 555 to rotate relative to each other, to translate (e.g. extend or retract) or to move in some other way. In such an embodiment, any of the poses in the 10A to 10C be connected to a certain permutation of a movement output by one or more actuators. The permutation can e.g. B. Describe angle values around the different arm parts of the robot arm 553 have been rotated relative to the corresponding arm parts immediately in front of the various arm parts, and / or distances by which the various arm parts have been translated relative to the corresponding arm parts immediately in front of the various arm parts. As an example, the poses in the 10A to 10C be associated with different permutations of five angle values, each having a direction of rotation and an amount of rotation of the link 554B relative to the limb 554A , a rotating direction and a rotating amount of the link 554C relative to the limb 554B , a rotating direction and a rotating amount of the link 554D relative to the limb 554C , a rotating direction and a rotating amount of the link 554E relative to the limb 554D and a direction and a rotation of the robot hand 555 relative to the limb 554E describe.

As indicated above, the 10A to 10C a group of multiple verification symbols 530A to 530C represents that on one or more arm parts of the robot arm 553 are arranged. In particular, the verification symbol 530A on the limb 554B be placed while the verification icon 530B on the limb 554C and the verification icon 530C on the robot hand 555 can be arranged. In some cases a robot (e.g. robot 550 ) have any number of verification symbols and these can be arranged on the robot in any way. The verification symbols 530A to 530C can, for example, happen at appropriate locations on the robot 550A be arranged. In some cases the number and relative placement of the verification symbols (e.g. 530A to 530C ) are subject to one or more defined restrictions. For example, their relative placement may be subject to a defined constraint that requires neighboring verification symbols (e.g. 530B and 530C ) by a defined minimum distance (also known as symbol distance), such as B. a defined minimum distance of 5 cm, are separated from each other. Such a restriction can reduce a likelihood that a computer system (e.g. the robot controller 110 ) a specific verification symbol (e.g. 530B ) with an adjacent verification symbol (e.g. 530C ) mistaken.

The 11A to 11C represent another example that is a group of multiple verification symbols 1130A to 1130C includes. In particular, the verification symbols 1130A to 1130C Part of a robot operating system 1100 be that a robot 1150 , a camera 1170 with a camera field of view 1110 and a computer system such as B. the robot controller 110 , includes. The robot operating system 1100 and the camera 1170 can e.g. B. each corresponding to an embodiment of the robot operating system 500 and the camera 570 be. The robot 1150 can use a robotic arm 1153 have comprising several arm parts, such as. B. the limbs 1154A to 1154E and a robotic hand 1155 (or another robot end effector). The multiple arm parts can be movable e.g. B. over the joints 1156A to 1156D be attached to each other. 11A illustrates the robotic arm 1153 in a first pose while 11B the robotic arm 1153 in a second pose. As in the 10A to 10C can use the multiple arm parts from one base 1152 of the robot 1150 to the robot hand 1155 connected or otherwise arranged in series. The series of arm parts can form a kinematic chain in which the movement of an arm part in the series can be propagated to downstream arm parts in the chain. As in 11A can be the verification symbol 1130A on the limb 1154C be placed while the verification icon 1130B on the limb 1154D and the verification icon 1130C on the robot hand 1155 can be arranged.

In one embodiment, one or more verification symbols from the group of verification symbols (e.g. 530A to 530C or 1130A to 1130C ) have a circular shape. Each of the verification symbols 530A to 530C the 10A to 10C or 1130A to 1130C of the 11A to 11C can for example be a ring pattern or in particular a circular ring, as described above in relation to FIG 5C has been described. In the example of 5C a ring pattern can have concentric circular regions (e.g. 531 and 533 ) or concentric circles (e.g. 535 and 537 ) exhibit. The concentric circular regions or concentric circles may e.g. Comprise an inner circular region and an outer circular region or an inner circle and an outer circle. In the in 11C example shown can be the verification symbol 1130A be shaped, for example, as a circular ring with an inner circle with radius r _1.1130A and an outer circle with radius r _2.1130A . In this example, the verification icon 1130B be another circular ring with an inner circle of radius r _1.1130B and an outer circle of radius r _2.1130B . The verification symbol 1130C can also be shaped as a circle with an inner circle with radius r _1.1130C and an outer circle with radius r _2.1130C.

In one embodiment, the set of verification symbols (e.g. 530A to 530C or 1130A to 1130C ) be shaped as corresponding circular rings with different corresponding sizes. As in 11C can be the verification symbols 1130A to 1130C for example, have different radii for their respective outer circular regions or outer circles. That is, they can have r _2.1130A , r _2.1130B, and r _2.1130C , all of which are different from each other. In one embodiment, the circular rings for the verification symbols 1130A to 1130C have different ratios between the corresponding radii of their inner circular regions or inner circles and the corresponding radii of their outer circular regions or outer circles. That is, the ratio r _2.1130A / r _1.1130A , the ratio r _2.1130B / r _1.1130B, and the ratio r _2.1130C / r _1.1130C may all be different from each other. As described in more detail below, a computer system (e.g., the robot control system 110 ) be configured to use the verification icon 1130A / 1130B / 1130C based on the size of a corresponding circular ring that represents the verification symbol 1130A / 1130B / 1130C and / or identified based on a corresponding ratio between a radius of an inner circle of the circular ring and a radius of an outer circle of the circular ring.

In one embodiment, some or all of the verification symbols (e.g. 530A to 530C the 10A to 10C or 1130A to 1130C of the 11A to 11B) on a robot arm (e.g. 533 or 1130 ) be permanently attached or otherwise arranged on it. Such an embodiment can be a calibration pattern (e.g. 520 the 5A and 5B) include that to perform a camera calibration is used and is of a size large enough to accommodate a pattern that is sufficiently complex and / or large enough to accommodate a sufficient number of pattern elements to provide a sufficiently accurate result for camera calibration. Such a large size for the calibration pattern (e.g. 520 ) however, can make it too large to be a permanent or otherwise permanent part of the robotic arm because the large size of the calibration pattern (e.g. 520 ) can cause it to interfere with normal robot operation. In such an example, the calibration pattern 520 z. B. from the robot arm (e.g. 553 ) before normal robot operation resumes. In this embodiment, some or all of the verification symbols (e.g. 530A to 530C or 1130A to 1130C ) less complex and / or smaller than the calibration pattern (e.g. 520 ) be. The smaller size of the verification icons can allow them to remain on the robot arm during normal robot operation (e.g. 553 or 1153 ), while they have little or no impact on normal robot operation. Therefore, in such an example, some or all of the verification symbols (e.g. 530A to 530C or 1130A to 1130C ) in some cases permanently or otherwise permanently on a robot arm (e.g. 533 or 1133 ) be arranged. Such an arrangement offers the advantage that a robot controller or another computer system is able to carry out a calibration verification more frequently and / or more quickly and to assess whether an updated camera calibration needs to be determined.

The 12A and 12B illustrate a procedure 1200 to use multiple verification symbols to verify calibration information such as B. Camera calibration information. In one embodiment, the method can 1200 by a computer system such as B. the robot control system 110 the 10A to 10C or the 11A to 11B , or in particular by a control circuit of the computer system, such as. B. the control circuit 111 of the robot control system 110 in the 1C to be executed. As in 1C illustrated, the computer system may have a communication interface 113 that is configured to work with a camera with a camera field of view, such as. B. the camera 570 of 10A or the camera 1170 of 11A corresponding to the camera's field of view 510 or 1110 has to communicate. The communication interface 113 can also be configured to communicate with a robot, such as a robot. B. the robot 550A the 10A to 10C or the robot 1150 the 11A to 11C . As stated above, the robot can 550A / 1150 a robotic arm 553/1153 comprising a plurality of arm parts movably attached to one another, and it may comprise a group of verification symbols arranged on the corresponding arm parts of the plurality of arm parts. In the example of the 10A to 10C the group of verification symbols can be the verification symbols 530A to 530C include that corresponding to the limb 554B , the limb 554C and the robotic hand 555 are arranged. In the example of 11A to 11C the group of verification symbols can be the verification symbols 1130A to 1130C include that corresponding to the limb 1154C , the limb 1154D and the robotic hand 1155 can be arranged.

In one embodiment, the method can 1200 one step 1201 include in which the control circuit 111 of the robot control system or other computer system, a movement command to control the movement of the robot arm (e.g. the robot arm 533/1153 ) to perform a robot operation. The movement command can e.g. B. based on calibration information. The calibration information, such as B. camera calibration information, can from a first calibration process, such. B. a first camera calibration can be determined. In some cases it can step 1201 same or similar to step 409 of the procedure 400 be where the control circuit 111 controls the movement of the robot arm to perform a robot operation. Robot operation may include picking up boxes or other items in a warehouse, for example. In this example the control circuit 111 be configured based on one of the camera (e.g. camera 570 the 10A to 10C or camera 1170 the 11A to 11B) generated image of the boxes and, based on the calibration information, a spatial relationship between the camera and the boxes and / or a spatial relationship between the robot (e.g. robot 550A 1150 ) and determine the boxes. In one embodiment, the method can 1200 comprise a step in which the control circuit 111 performs the first calibration process to determine the calibration information. Such a step for performing the first calibration process can be similar to step 401 of procedures 400 of 4A be and can step ahead 1201 are executed. As an example, the first calibration process can be a camera calibration, which includes determining an estimate for a camera calibration parameter based on calibration images generated by the camera.

In one embodiment, the method can 1200 one step 1203 include in which the control circuit 111 or another component of the computer system determines a set of reference image coordinates. The group of reference image coordinates can e.g. B. be corresponding coordinates at which the group of verification symbols (e.g. 530A to 530C / 1130A to 1130C ) appears in a reference image, the reference image being an image representing the group of verification symbols (e.g. 530A to 530C / 1130A to 1130C ) can be. In one embodiment, the set of reference image coordinates can be used to verify the calibration information.

For example poses 13A a reference image 1120 represents the group of verification symbols 1130A to 1130C of 11A represents. In the example of 13A the group of reference image coordinates can comprise a first reference image coordinate, a second reference image coordinate and a third reference image coordinate. In such an example, the first reference image coordinate can identify where the verification symbol 1130A in the reference image 1120 appears while the second reference image coordinate can identify where the verification icon 1130B in the reference image 1120 appears, and the third reference image coordinate can indicate where the verification symbol 1130C in the reference image 1120 appears. In a more specific example, each of the first reference image coordinate, the second reference image coordinate, and the third reference image coordinate can be a pixel coordinate [uv] ^T. In particular, represents 13A the three reference image coordinates for the verification symbols 1130A , 1130B and 1130C correspondingly represented as [u _{ref_1} V _{ref_1} ] ^T , _1130A , [ _{uref_1} V _{ref_1} ] ^T , 1130B, [u _{ref_1} v _{ref_1} ] ^T _1130C . In this example, the term ref_N (e.g. ref 1) can refer to coordinates associated with a reference image that is generated when the robot arm is moving 1153 is in an Nth pose, such as B. a first pose (if N = 1). As described in more detail below, the reference image 1120 of 13A correspond to a first pose or, more generally, be associated with a first pose that corresponds to the in 11A Illustrated exemplary pose of the robotic arm 1153 can be. Different poses can use the verification symbols 1130A to 1130C at different groups of corresponding locations in the camera's field of view 1110 place. The pose in 11A can for example the verification symbols 1130A to 1130C _Place accordingly at the 3D locations [x ref_1 y _{ref_1} z _{ref_1} ] ^T _1130A , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130B , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130C . In such a case, these 3D locations [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130A , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130B , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _{1130C can be linked} to the pixel coordinates [ _{uref_1} v _{ref_1} ] ^T _1130A , [u _{ref_1} v _{ref_1} ] ^T _1130B , [u _{ref_1} v _{ref_1} ] ^T _1130C in the reference image 1120 projected or otherwise assigned to these. In one embodiment, each of the pixel coordinates [u _{ref_1} v _{ref_1} ] ^T _1130A , [u _{ref_1} v _{ref_1} ] ^T _1130B , [u _{ref_1} v _{ref_1} ] ^T _{1130C can identify} a center in which its corresponding verification symbol 1130A / 1130B / 1130C in the reference image 1120 appears.

In one embodiment, the step 1203 that involve the control circuit 111 the reference image (e.g. 1120 ) from the communication interface 113 and / or the non-transitory computer readable medium 115 of the robot control system 110 or another computer system. The reference image 1120 of 13A can for example from the camera 1170 generated when the robot arm moves 1153 in the pose of 11A is located. The computer system 110 can use the reference image 1120 from the camera 1170 via the communication interface 113 received and the reference image 1120 in the non-transitory computer readable medium 115 to save. In step 1203 can control circuit 111 in one example the reference image 1120 from the non-transitory computer readable medium 115 retrieve or otherwise receive. The non-transitory computer-readable medium 115 can also store a verification image (which will be described below), and the control circuit 111 can transfer the verification image from the non-transitory computer readable medium 115 receive.

In one embodiment, the control circuit 111 be configured to display at least one verification symbol (e.g. 1130A ) from the group of verification symbols (e.g. 1130A to 1130C ) in the reference image (e.g. 1120 ) based on a defined model that shows a geometry of the robot arm (e.g. 1153 ) describes. The defined model can e.g. For example, describe which limbs, robot end effector or other arm parts make up the robot arm, their corresponding sizes (e.g. lengths) and how they are connected, and / or describe which arm parts have at least one verification symbol (e.g. 1130A ) exhibit on it. In such an embodiment, the control circuit can be configured to determine a region within the reference image (e.g. 1120 ) to determine in which the at least one verification symbol is expected, and after the at least one verification symbol (e.g. 1130A ) within the region of the reference image. In one embodiment, the model may include a location of a verification symbol (e.g. 1130A ) on the robot arm (e.g. 1153 ) save or describe more generally. The position of the verification symbol (also known as the symbol position) may be an approximate location of the verification symbol on the robot arm (e.g. 1153 ) be.

If the reference image (e.g. 1120 ) is stored, for example, together with one or more parameter values of a movement command that is used to generate the pose associated with the reference image or in particular the pose of the robot arm that appears in the reference image, the one or more parameter values and the model can be used to create a pose for the robot arm (e.g. 1153 ) estimate when the robot arm is moved according to the move command. As described in more detail below, the one or more parameter values in a Embodiment belong to one or more actuator parameters that are used to control one or more actuators (e.g. one or more motors) that are used to move the robot arm (e.g. 1153 ) be used. The estimated pose can be used to indicate the location of a verification symbol (e.g. 1130A ) on the robot arm (e.g. 1153 ), and the estimated location can be used to estimate where the verification symbol is likely to be in the reference image (e.g. 1120 ) appears.

In the above examples, the control circuit 111 be configured to refer to a region or regions of a reference image (e.g. 1120 ) where a verification icon or icons is expected to appear. Such a technique can enable the control circuit 111 it avoids the entire reference image (e.g. 1120 ) according to verification symbols (e.g. 1130A to 1130C ) and therefore the verification symbols (e.g. 1130A to 1130C ) in the reference image (e.g. 1120 ) identified more quickly. The model in the previous examples can also be used to search for the verification symbols in a verification image (described below).

As indicated above, the set of verification symbols (e.g. 1130A to 1130C ) be shaped in one embodiment as corresponding circular rings. In such an embodiment, the control circuit 111 be configured to display a verification icon (e.g. 1130A ) in the reference image (e.g. 1120 ) or otherwise identify it by identifying a circular ring that forms the verification symbol. When the set of verification symbols is shaped as corresponding circular rings of different sizes, as above in relation to FIG 11C illustrated, the control circuit 111 be configured to use a verification symbol (e.g. 1130A ) based on a size ( _e.g., a radius r 2.1130A) of a corresponding circular ring that forms the verification symbol. If in some cases a verification symbol (e.g. 1130A ) is shaped as a circular ring with at least a first circular region or a first circle and a second circular region or a second circle, the control circuit 111 be configured to identify the verification icon based on a ratio between a radius of the first circular region or the first circle and a radius of the second circular region or the second circle. The control circuit 111 can for example be configured to use the verification icon 1130A based on identifying a circular ring and confirming that the ring has the ratio r _{2.1130A / r1.1130A} .

As further explained above, the reference image (e.g. 1120 of 13A) by step 1203 from the camera (e.g. 1170 of 11A) generated when the robot arm (e.g. 1153 ) is in a first pose, such as B. the in 11A illustrated pose. The reference image can be generated by the camera during a first time period or, more generally, at a first point in time. The first period of time may refer to a period of time (e.g., a period of milliseconds, seconds, or minutes) that is used or otherwise associated with the generation of the reference image. The first time period can include, for example, the time for a camera to operate to capture the reference image and, in some cases, also a time for a robot movement to position the verification symbol in the camera's field of view. In some cases, the first period of time that the reference image is generated, e.g. B. hours, days or weeks before step 1201 and / or step 1203 are located. In one embodiment, the computer system or, in particular, the control circuit 111 showing the steps 1201 and 1203 performs, on the generation of the reference image (e.g. 1120 ) not be involved. In one embodiment, the computer system or, in particular, the control circuit 111 showing the steps 1201 and 1203 performs, on the generation of the reference image (e.g. 1120 ) be involved. The computer system can, for example, in a step that corresponds to the step 1201 and or 1203 precedes, issue a movement command during the first time period to cause the robot arm (e.g. 1153 ) moved into the first pose. In such a case, this previous step may be part of the procedure 1200 be. The first pose can cause the set of verification symbols (e.g. 1130A to 1130C ) is moved to a first group of corresponding locations associated with the first pose, e.g. The locations discussed above [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130A , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130B , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130C . In some cases, such a step may be similar or identical to the step 403 in 4A be. In the above example, the computer system, or in particular the control circuit 111 , in this step also issue a camera command that activates the camera (e.g. 1170 ) causes the reference image (e.g. 1120 ) while the robot arm (e.g. 1153 ) is in the first pose. In one embodiment, the computer system can use the reference image (e.g. 1120 ) received by the camera and the reference image in the non-transitory computer readable medium 115 of 1C or store in some other non-transitory computer readable medium. In some cases, the computer system can also store information that it sends to the robotic arm (e.g. 1153 ) allow to return to the first pose and / or the verification symbols (e.g. 1130A to 1130C ) to the first group of appropriate places. The computer system can, for example, store the first group of the corresponding locations or, in particular, store their 3D coordinates, and / or it can store parameter values of the movement command in order to move the robot arm (e.g. 1153 ) to move into the first pose. In one embodiment, the coordinates and / or the movement command can be in the non-transitory computer readable medium 115 be stored in a way that matches the stored information with the reference image described above (e.g. 1120 ) connect.

In one embodiment, the move command discussed above to move the robot arm into the first pose may include one or more parameter values indicative of movement for the robot arm (e.g. 1153 ) describe. As indicated above, in some cases, the one or more parameter values may belong to one or more actuator parameters that control one or more actuators that cause movement for the robot arm (e.g. 1153 ) produce. In such cases, the one or more parameter values can be referred to as one or more actuator parameter values (also referred to as robot joint values). The one or more actuator parameter values can e.g. B. corresponding amounts of rotation of the arm parts of the robot arm relative to one another, the corresponding locations and / or orientations of the arm parts relative to one another and / or the corresponding locations of the joints (e.g. 1156A to 1156D ), which connect the arm parts. The one or more actuator parameter values of the movement command can, for example, describe corresponding angle values around which the motors in the robot 1150 different arm parts (e.g. the limbs 1154A to 1154E and the robotic hand 1155 ) should rotate relative to their immediately preceding arm parts.

In one embodiment, the move command discussed above can have any or any parameter value (e.g., a random or random parameter value (s)) and the first pose caused by the move command can be any pose. In one embodiment, the move command discussed above may be a move command that causes some or all of the verification symbols (e.g. 1130A to 1130C ) a desired appearance in the reference image (e.g. 1120 ) exhibit. For example, as discussed above, some or all of the set of verification symbols (e.g. 1130A to 1130C ) have a circular shape, such as. B. a ring shape formed by concentric circular regions or concentric circles. In such an example, if at least one of the verification symbols (e.g. 1130A ) in certain orientations relative to the camera (e.g. 1170 ) is located, the at least one verification symbol (e.g. 1130A ) appear elliptical rather than completely circular in the resulting reference images. The elliptical appearance can result in inaccurate calibration verification. For example, if the reference image coordinate for the verification icon 1130A is a place where the center of the symbol appears in the reference image, that reference image coordinate may be more difficult to determine precisely when the verification symbol 1130A appears elliptical in the reference image. When identifying the verification symbol 1130A in the reference image (e.g. distinguishing the verification symbol 1130A of other features in the reference image) is based on identifying a ring pattern in the reference image and verifying that the ring pattern has a certain ratio between concentric circles associated with the symbol ( _{e.g. the ratio r 2,1130A / r1,1130A} , the one with the verification symbol 1130A this identification may be more difficult to perform if the ring pattern appears elliptical in the reference image. Therefore, when the control circuit 111 is involved in the outputting of the movement command associated with the generation of the reference image, the control circuit can 111 try to generate a movement command that causes the group of verification symbols (e.g. 1130A to 1130C ) are positioned in such a way that they appear completely or substantially circular in the reference image.

The control circuit 111 can, for example, generate a movement command that the robot arm (e.g. 1153 ) causes them to move into a pose in which the group of verification symbols (e.g. 1130A to 1130C ) directly to the camera (e.g. 1170 ) is facing. The pose can e.g. B. cause at least one verification symbol of the group of verification symbols (e.g. 1130A to 1130C ) is tangent to a surface of an imaginary sphere that is in relation to the camera (e.g. 1170 ) is concave. In 11A can do the pose shown for the robotic arm 1153 cause the verification icon 1130A and the verification icon 1130B have corresponding orientations in which they are a surface of an imaginary sphere 1121 affect that in relation to the camera 170 is concave and it can cause the verification symbol 1130C has an orientation in which there is a surface of an imaginary sphere 1123 that affects that in relation to the camera 1170 is also concave. In such a pose, which may be the first pose discussed above, the set of verification symbols (e.g. 1130A to 1130C ) as corresponding circular shapes in the reference image (e.g. 1120 ) appear. In particular, the group of verification symbols (e.g. 1130A to 1130C ) in such an example must be positioned so that they appear without eccentricity in the reference image or appear with corresponding amounts of eccentricity which are below a defined eccentricity threshold value. In the example above, the imaginary circles 1121 and 1123 on the camera (e.g. 1170 ) be centered. In one embodiment, the control circuit 111 be configured to generate random move commands and search through the random move commands to find one that can generate the alignments discussed above for the set of verification symbols. The movement command found can be obtained from the control circuit 111 output to display the reference image (e.g. 1120 ) to create.

With reference to the 12A to 12B includes the procedure 1200 in one embodiment, a step 1205 in which the control circuit 111 issues a move command to control the robot arm so that it moves into the first pose, such as: B. those in the 10A or 11A shown first pose, wherein the first pose is a pose in which the reference image (e.g. 1120 ) is generated during the first period of time, as discussed above. In some cases the movement command can be sent via the communication interface 113 to the robot (e.g. 550A / 1150 ) are output. In one embodiment, the movement command can be in step 1205 can be referred to as an additional movement command, since it is in addition to the movement command of step 1201 is available. The additional movement command can be issued during a second time period or, more generally, during a second point in time following the first time period. The second period of time may refer to a period of time (e.g., a period of milliseconds, seconds, or minutes) used or otherwise associated with the generation of the verification image. The second time period can include, for example, a time for a robot movement to position the verification symbol in the camera field of view and / or a time for a camera operation to capture the verification image. In some cases, the second time period (or more generally the second point in time) can follow the first time period (or more generally the first point in time) by hours, days, or weeks. As described in more detail below, the additional move command can be used to generate a verification image during the second time period. In one embodiment, the additional movement command can move the robot arm (e.g. 1153 ) induce oneself into a pose (e.g. the first pose of 11A) in which the group of verification symbols (e.g. 1130A to 1130C ) the camera (e.g. 1170 ) is facing directly. The group of verification symbols can, for example, have corresponding orientations in which they are tangent to one or more imaginary spheres which are concave with respect to the camera. In such a pose, the group of verification symbols (e.g. 1130A to 1130C ) as corresponding circular shapes in the verification image (e.g. 1160 of 13B) appear.

If, in one embodiment, the method 1200 as described above comprises a step of issuing a movement command for generating the reference image during the first time period, the movement command for generating the reference image during the first time period can be a first additional movement command, during the movement command in step 1205 for generating the verification image during the second time period can be a second additional movement command. In some cases, the first additional move command can be an earlier move command while the second additional move command can be a later move command. In some cases, the first additional move command can include one or more actuator parameter values (or robot joint values) for controlling the robot arm to move into the first pose, and the second additional move command can also include the one or more actuator parameter values. In particular, the first additional movement command and the second additional movement command can have the same actuator parameter values. For example, if a robot (e.g. 1150 ) includes several motors that support different arm parts (e.g. the limbs 1154A to 1154E and the robotic hand 1155 ) rotate relative to one another, the one or more actuator parameter values can include several corresponding angle values that control how much rotation is to be output by the several motors. In this example, the first additional move command and the second additional move command may both include the same multiple corresponding angle values that control how much rotation is output from the multiple motors.

In one embodiment, the first additional movement command and / or the second additional movement command can be issued during a corresponding rest time, such as the rest time described above in relation to step 411 For 4A was discussed. In some situations, when the calibration information is determined by performing a calibration process, the first additional movement command may be issued immediately after the calibration process is performed or during an earliest rest period following the calibration process. In some situations, the second additional movement command can be issued in response to a defined trigger condition, such as e.g. B. a defined period of time that has elapsed since the calibration process was carried out, a collision event involving the robot or another event (e.g. a natural disaster such as an earthquake) that is due to a possible shift or misalignment between the camera and robot or parts thereof, or another trigger condition. If the generation of the verification image includes that the control circuit 111 outputs a camera command, in some examples the camera command can also be output as a reaction to the defined trigger condition.

In one embodiment, the first pose of the step 1205 for the robot arm (e.g. 1153 ) with a specific group of corresponding reference locations for the group of verification symbols (e.g. 1130A to 1130C ) attached to the robotic arm as discussed above. The group of corresponding reference locations can e.g. B. the 3D locations [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130A , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130B , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130C . If the reference image (e.g. 1120 ) is generated, the group of verification symbols (e.g. 1130A to 1130C ) are at the group of corresponding reference locations associated with the first pose. If in step 1205 the additional movement command causes the robot arm to return to the first pose, the group of verification symbols (e.g. 1130A to 1130C ) return to the group of reference _locations (e.g. [x ref_1 y _{ref_1} Z _{ref_1} ] ^T _1130A , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130B , [x _{ref_1} y _{ref_1} z _{ref_1} ] ^T _1130C ). In such an example, the set of verification symbols (e.g. 1130A to 1130C ) when generating the reference image (e.g. 1120 ) are arranged in the group of the corresponding reference locations and when generating the verification image (e.g. 1160 ) can be arranged again in the group of the corresponding reference locations.

Referring again to the 12A to 12B can do the procedure 1200 in one embodiment, a step 1207 include in which the control circuit 111 a verification image, such as B. the verification image 1160 in 13B receives. As indicated above, the reference image (e.g. 1160 ) an image to represent the group of verification symbols (e.g. 1130A to 1130C ) be. In this example, the verification image (e.g. 1160 ) be an additional image to include the group of verification symbols (e.g. 1130A to 1130C ) and it can be generated when the robot arm (e.g. 1153 ) as a result of the additional movement command from step 1205 moved into the first pose, such as B. in the in 11A shown first pose.

In one embodiment, the method can 1200 one step 1209 include in which the control circuit 111 determines a set of verification image coordinates. In this embodiment, the set of verification image coordinates may be corresponding coordinates at which the set of verification symbols (e.g. 1130A to 1130C ) in the verification image (e.g. 1160 of 13B) appears. In one embodiment, the set of verification image coordinates can be used to verify the calibration information, such as B. camera calibration information can be used. Similar to the discussion above regarding the reference image coordinates in step 1203 For example, the set of verification image coordinates may include a first verification image coordinate, a second verification image coordinate, and a third verification image coordinate. In a more specific example, each of the first verification image coordinate, the second verification image coordinate, and the third verification image coordinate may be a pixel coordinate, e.g. B. a center of their corresponding verification symbol is identified. 13B _{represents, for example, three verification image} coordinates or in particular pixel coordinates [u Verify_1 v _{verify_1} ,] ^T _1130A , [u _{verify_1} v _{verify_11]} ^T _1130B , [u _{verify_1} v _{verify_1} ] ^T _1130C , at which the corresponding centers of the verification symbols 1130A to 1130C in the verification image 1160 appear. Similar to the discussion of reference image coordinates, the term verify N (e.g., verify_1) may refer to coordinates associated with a verification image generated when the robot arm is in an Nth pose, such as, for example. B. the first pose (N = 1).

In one embodiment, the control circuit 111 the steps 1203 to 1209 for multiple reference images and multiple verification images (e.g., five reference images and five verification images) multiple times. The multiple reference images and multiple verification images may correspond to multiple corresponding poses. For example, the 10A to 10C a series of three poses for the robotic arm 553 In this example, the control circuit 111 the robot controller 110 or any other computer system a first reference image and a first verification image, both as in FIG 10A illustrated associated with the first pose are a second reference image and a second verification image, both as in FIG 10B illustrated are associated with a second pose, and a third reference image and a third verification image as shown in FIG 10C illustrated associated with a third pose. In another example, the reference image 1120 of 13A a first reference image and the verification image 1160 of 13B be a first verification image, both with the in 11A for the robot arm 1153 illustrated first pose can be connected. In this example the control circuit 111 also a second reference image 1122 of 14A and a second verification image 1162 of 14B received, both with an in 11B shown second pose can be connected. In this example, the coordinates of the reference _image [uref_1 v _{ref_1} ] ^T _1130A , [ _{uref_1} v _{ref_1} ] ^T _1130B , [u _{ref_1} v _{ref_1} ] ^T _1130C from 13A be a first group of reference _{image coordinates} , while the verification image coordinates [u verify_1 v _{verify_11} ] ^T _1130A , u _{verify_1} v _{verify_1} ] ^T _1130B , [u _{verify_1} v _{verify_11} ] ^T _1130C of 13B may be a first set of verification image coordinates. The control circuit 111 In this example, a second group of reference _{image coordinates} [u ref_2 v _{ref_2} ] ^T _1130A , [u _{ref_2} v _{ref_2} ] ^T _1130B , [u _{ref_} 2 v _{ref_2} ] ^T _1130C for the second reference image can also be configured 1122 of 14A to determine and a second group of verification _{image coordinates} [u verify_2 v _{verify_12} ] ^T _1130A , u _{verify_2} v _{verify_2} ] ^T _1130B , [u _{verify_2} v _{verify_12} ] ^T _1130C for the second verification image 1162 of 14B to determine. In the above examples, the control circuit 111 in some cases be configured to issue various appropriate movement commands to move the robot arm (e.g. 1153 ) to move to the plurality of corresponding poses, wherein the plurality of reference images and / or the plurality of verification images are associated with the plurality of corresponding poses.

Referring again to the 12A to 12B can do the procedure 1200 in one embodiment, a step 1211 include in which the control circuit 111 determines a group of corresponding deviation parameter values based on corresponding deviation _{amounts between the group of reference image coordinates} (e.g. [u ref_1 v _{ref_1} ] ^T _1130A , [Uref _{_1} v _{ref_1} ] ^T _1130B , [u _{ref_1} v _{ref_1} ] ^T _1130C from 13A) and the group of verification _{image coordinates} (e.g. [u verify_1 v _{verify_11} ] ^T _1130A , [u _{verify_1} v _{verify_11} ] ^T _1130B , [u _{verify_1} v _{verify_11} ] ^T _1130C of 13B) based. The set of corresponding deviation parameter values in the context of the 13A and 13B may include a first variance parameter value, a second variance parameter value, and a third variance parameter value. The first deviation parameter value can be based on a deviation _{amount between a first reference image coordinate} [uref_1 v _{ref_1} ] ^T _1130A for the verification symbol 1130A in the reference image 1120 and a first verification _{image coordinate} [u Verify_1 v _{verify_11} ] ^T _1130A for the verification symbol 1130A in the verification image 1160 based. The first deviation parameter value can, for example, be equal to the distance between the first reference image coordinate and the first verification image coordinate or, more generally, be based on it, as described above in relation to step 457 of 4B was discussed. Similarly, the second deviation parameter value can be based on a deviation _{amount between a second reference image coordinate} [uref_1 v _{ref_1} ] ^T _1130B for the verification symbol 1130B in the reference image 1120 and a second verification _{image coordinate} [u verify_1 v _{verify_11} ] ^T _1130B for the verification symbol 1130B in the verification image 1160 based (e.g. be the same). Furthermore, the third deviation parameter value can be based on a deviation _{amount between a third reference image coordinate} [uref_1 v _{ref_1} ] ^T _1130C for the verification symbol 1130C in the reference image 1120 and a third verification _{image coordinate} [u verify_1 v _{verify_1} ] ^T _1130C for the verification symbol 1130C in the verification image 1160 based. In the example above, the group of corresponding deviation parameter values is with the group of verification symbols 1130A to 1130C connected. That is, the first deviation parameter value is with the verification symbol 1130A connected, while the second deviation parameter value is connected to the verification symbol 1130B and the third deviation parameter value with the verification symbol 1130C connected is.

In one embodiment, some or all of the set of corresponding deviation parameter values may be based on a single pair of a reference image and a verification image, both of which may be associated with a common pose for the robotic arm. The first deviation parameter value described above can be, for example, with the verification symbol 1130A be connected and on a single pair of the reference image 1120 and the verification image 1160 based both of which are associated with a common pose, such as B. the pose in 11A .

In one embodiment, some or all of the set of respective deviation parameter values may be based on multiple pairs of respective reference images and respective verification images, each pair associated with a corresponding one of multiple poses for the robotic arm. As an example, the first deviation parameter value discussed above, associated with the verification symbol 1130A is connected, on a first pair of the reference image 1120 and the verification image 1160 (the 13A and 13B) and a second pair of the reference image 1122 and the verification image 1162 (the 14A and 14B) based. The first pair can match the first pose of 11A be connected while the second pair with the second pose of 11B can be connected. _{In particular, the first deviation parameter value} in this example may be based on a deviation amount (e.g., the distance) between [u ref_1 v _{ref_1} ] ^T _1130A and [u _{verify_1} v _{verify_11} ] ^T _1130A (which are associated with the first pose), and on _{a deviation amount} between [u ref_2 v _{ref_2} ] ^T _1130A and [u _{verify_2} v _{verify_12} ] ^T _1130A (which are associated with the second pose). In one example, the first deviation parameter value can be equal to or based on a mean value of the two deviation amounts. In more general terms, the group of corresponding deviation parameter values in this example can refer to the corresponding deviation amounts between the first group of reference image coordinates and the first group of verification image coordinates (associated with the first pose) and further on the corresponding deviation amounts between the second group of reference image coordinates and the second group of verification image coordinates (associated with the second pose) based.

In some cases, a reference image coordinate in a first reference image and a reference image coordinate in a second reference image may be part of a set of reference image coordinates indicating where a common verification symbol (e.g. 1130A ) appears in a set of reference images, such as The two reference images discussed above 1120 , 1122 . The set of reference images and the set of reference image coordinates in this example may correspond to a set of corresponding poses for the robot arm (e.g. 1153 ) correspond, such as B. corresponding to the two poses in the 11A and 11B . Similarly, a verification image coordinate in a first verification image and a verification image coordinate in a second verification image may be part of a set of verification image coordinates indicating where the verification symbol appears in a set of verification images, e.g. The verification images discussed above 1160 , 1162 . The set of verification images and the set of verification image coordinates can also correspond to the set of poses. In such cases, a deviation parameter value associated with the verification symbol may be based on the corresponding deviation amounts between the set of reference image coordinates and the set of verification image coordinates.

Referring again to the 12A to 12B can do the procedure 1200 in one embodiment, a step 1213 include in which the control circuit 111 determines whether at least one deviation parameter value of the group of corresponding deviation parameter values exceeds a defined deviation threshold value. In some cases it can step 1213 similar to the step 459 of 4B be. In one embodiment, step 1213 include determining whether each deviation parameter value of the group of corresponding deviation parameter values has exceeded a corresponding deviation threshold. The control circuit 111 can for example determine whether a with the verification symbol 1130A associated deviation parameter value has exceeded a defined deviation threshold value, whether the one with the verification symbol 1130B associated deviation parameter value has exceeded a defined deviation threshold value, and / or whether the one with the verification symbol 1130C associated deviation parameter value has exceeded the defined deviation threshold value. In some cases, the corresponding deviation thresholds can have the same value and therefore form a common deviation threshold for the verification symbols, or they can have different values.

In one embodiment, the method can include a step 1215 include in which the control circuit 111 in response to determining that at least one deviation parameter value of the group of corresponding deviation parameter values exceeds the defined deviation threshold value, can perform at least one of the following: (a) issue a notification that at least one of the group of corresponding deviation parameter values exceeds the defined deviation threshold value, or ( b) Performing a calibration process to determine updated calibration information (e.g., updated camera calibration information). step 1215 can e.g. B. issuing the notification to a user interface device, such as e.g. B. an electronic display in communication with the robot controller 110 , include. The electronic display can e.g. B. the at least one value of the deviation parameter or an indication that the at least one value of the deviation parameter exceeds the defined deviation threshold value. In one embodiment, the calibration process can be carried out in step 1215 similar or equal to the step 461 in 4B be. If, in one embodiment, the calibration information from step 1201 can be determined by performing a first calibration process, then the calibration process from step 1215 be a second calibration process following the first calibration process.

In one embodiment, the at least one deviation parameter value that exceeds the defined deviation threshold can be based on a change in the camera (e.g. 1170 the 11A to 11B) and / or a change in the environment of the camera or the robot operating system (e.g. 1100 ) Clues. In some cases the change to the camera may be an internal change, e.g. B. a lens or an image sensor of the camera has changed its shape or size due to a change in temperature or physical damage. In some cases, the change to the camera may also include a change to the location where the camera (e.g. 170 ) is appropriate, such as Due to vibrations in a structure (e.g. the ceiling) on which the camera is attached. In some cases, a change in the environment of the camera or the robot operating system can result in a change in the location or orientation of a base (e.g. 1152 ) of a robot (e.g. 1150 ) include, such as B. due to a vibration in a structure (e.g. the floor) on which the robot is attached. In some cases, the change in the environment of the camera or the robot operating system may be a change in a relationship between arm parts of the robot arm (e.g., between the limbs 1154A to 1154E of the robot arm 1153 ) or in the arm parts themselves. For example, one of the arm parts (e.g. the limb 1154D or the robot hand 1155 ) by an event generated by the computer system 110 was not planned, bent or otherwise deformed or damaged. The unplanned event can be a collision event or some other unexpected event that leads to a possible change to the robot (e.g. 1150 ) or another element of the robot operating system. The calibration verification described above can be a quick and efficient technique for detecting the change of a camera (e.g. 1170 ) and / or a change in an environment of the camera or the robot operating system 1100 provide. A change in the camera (e.g. 1170 ) and / or the robot (e.g. 1150 ) can be detected by comparing one or more reference image coordinates with one or more corresponding verification image coordinates to determine a difference therebetween. In many cases, such a comparison can be made without using the computer resources of the computer system 110 overuse. For example, the comparison can be made with calculations that only require a limited amount of processor execution time and / or limited storage capacity. In this way, the comparison can facilitate precise monitoring of the accuracy of calibration information in a computationally efficient manner.

In one embodiment, the at least one deviation parameter value that exceeds the defined deviation threshold value may indicate that there is a calibration error, such as e.g. B. a camera calibration error in which the camera calibration information from the first camera calibration is no longer sufficiently accurate. In one embodiment, the method can 1200 that involve the control circuit 111 determines a type of calibration error (also referred to as a type of alignment error) that causes the at least one deviation parameter value to exceed the defined deviation threshold value. The type of calibration error can e.g. For example, indicate whether a loss of accuracy in the calibration information (e.g. camera calibration information) is caused by a change in the robot or is otherwise representative of it, or whether the loss of accuracy is caused by a change in the camera or is otherwise representative thereof is. The change to the camera (e.g. 1170 ) may include, for example, changing the camera internally and / or changing the location where the camera is attached, as described above. The change to the robot can include changing a location or orientation of a base (e.g. 1152 ) of the robot (e.g. 1150 ), a change in the relationship between the arm parts of the robot arm (e.g. 1153 ) and / or a change to the arm parts themselves.

In one embodiment, the determination of the type of calibration error can be based on a comparison between the group of corresponding deviation parameter values and in particular on whether the group of corresponding deviation parameter values exceeds a defined deviation threshold value in a substantially uniform manner. When the group of deviation parameter values associated with the various verification symbols (e.g. 1130A to 1130C ) completely exceeds the defined deviation threshold and does so in a substantially uniform manner, the control circuit can 111 determine that the loss of accuracy caused by the camera (e.g. 1170 ) or is caused in particular by a change to the camera. The reason is that the appearance of each of the verification symbols (e.g. 1130A to 1130C ) in a reference or verification image of an internal property and / or of the positioning of the camera (e.g. 1170 ) is dependent. In particular, the coordinates of the reference image and the coordinates of the verification image for the verification symbols (e.g. 1130A to 1130C ) in the reference or verification image all be dependent on the internal property or on the positioning of the camera. Therefore, a change in the internal properties or the positioning of the camera (e.g. 1170 ) often to corresponding deviation parameter values for all verification symbols (e.g. 1130A to 1130C ) and in particular can increase all deviation parameter values substantially uniformly. In comparison, if there is damage, malfunction or any other change to any part of the robot (e.g. 1150 ), such as An arm part, all other parts of the robot would change in exactly the same way (e.g. damaged or all malfunctioning). Therefore, if there were a change in a part of the robot that would increase at least one deviation parameter above the defined deviation threshold value, other deviation parameter values of the group of deviation parameters can still remain below the defined deviation threshold value or the group of deviation parameters can completely exceed the defined deviation threshold value, but does so in an inconsistent manner. Thus, in one embodiment, if at least one of the group of corresponding deviation parameter values associated with the different verification symbols (e.g. 1130A to 1130C ), exceeds the defined deviation threshold, but the group of corresponding deviation parameter values does not completely exceed the defined deviation threshold value in a substantially uniform manner, the control circuit can 111 determine that the loss of accuracy caused by the robot (e.g. 1150 ) or is caused in particular by a change to the robot.

In one embodiment, the control circuit 111 use a defined uniformity threshold to assess whether the set of deviation parameter values completely exceeds the defined deviation threshold substantially uniformly. The control circuit 111 For example, it can determine whether at least one of the deviation parameter values exceeds the defined deviation threshold value, and it can further determine whether differences between the deviation parameter values (or between the corresponding amounts by which they exceed the defined deviation threshold value) lie within the defined uniformity threshold value. The uniformity threshold can be defined in a dynamic manner (e.g. based on a current operating state of the robot operating system) or it can be predefined. As an example of the use of the defined uniformity threshold when the control circuit 111 determines the appropriate deviation parameter values for the verification symbols 1130A to 1130C all exceed the defined deviation threshold, but that one with the verification symbol 1130C associated deviation parameter value from one with the verification symbol 1130A associated deviation parameter value differs by more than the defined uniformity threshold value, and / or from one with the verification symbol 1130B associated deviation parameter value differs by more than the defined uniformity threshold value, then the control circuit can 111 determine that the loss of accuracy due to a change in the robot 1150 is caused, such as B. by changing at least the robot hand 1155 or another part of the arm with the verification symbol 1130C is arranged. In the above example, the control circuit compares 111 the values of the deviation parameters directly. In other examples, the control circuit 111 compare corresponding amounts by which the deviation parameter values exceed the defined deviation threshold value and whether these corresponding amounts differ by more than the defined uniformity threshold value. If in another example the control circuit 111 determines that a first deviation parameter value of the group of deviation parameter values exceeds the defined deviation threshold value, but one or more of the group of deviation parameter values does not exceed the defined deviation threshold value, the control circuit can 111 also determine that the calibration error (also known as misalignment) is due to a change to the robot (e.g. 1150 ) is caused, such as B. on an arm part on which the verification symbol associated with the first deviation parameter value is arranged. When the control circuit 111 in a further example determines that the group of deviation parameter values completely exceeds the defined deviation threshold value and that they do not differ from one another by more than the defined uniformity threshold value, then the control circuit can 111 determine that the calibration error was caused by a change to the camera (e.g. 1170 ) is caused.

In some cases the control circuit can 111 the robot controller 110 or another computer system with a conveyor belt, e.g. B. with the conveyor belt 1173 the 11A and 11B to be in communication. In such cases, the control circuit 111 in response to a determination that the at least one deviation parameter value exceeds the defined deviation threshold, be configured to cause the conveyor belt 1173 stops. Stopping the conveyor belt 1173 can prevent the robot arm 1153 unwanted interaction with an object on the conveyor belt based on inaccurate calibration information 1173 Has.

In one embodiment, the control circuit 111 be configured to determine that when the calibration information for a particular arm part (e.g., limb 1154C where the verification symbol 1130A is arranged, or limb 1154D where the verification symbol 1130B is arranged, or the robot hand 1155 at which the verification symbol 1130C is arranged) are sufficiently accurate, the calibration information is also sufficiently accurate for one or more arm parts that are located in front of this particular arm part. As stated above, the plurality of arm parts may be arranged as a series of arm parts from a base of the robot to a robot end effector. An arm part can be located in front of another arm part if the first-mentioned arm part precedes the last-mentioned arm part in the row of arm parts. The Member 1154D in the 11A to 11B can for example be in front of the limb 1154E and the robotic hand 1155 are located. When the control circuit 111 in one example, determines that the calibration information (e.g., camera calibration information) for e.g. B. the robot hand 1155 are sufficiently accurate, it can determine that the calibration for upstream arm parts, such as B. limb 1154E , 1154D , 1154C , 1154B and 1154A , is sufficiently accurate. In this example the control circuit 111 determine that the calibration information for an arm part is sufficiently accurate if a deviation parameter value associated with a verification symbol arranged on the arm part is below the defined deviation threshold value. When the control circuit 111 In one embodiment, if it determines that there is a calibration error for a particular arm part such that the calibration information for that arm part is not sufficiently accurate, the control circuit may 111 in one embodiment, determine that there is a calibration error for some or all of the downstream arm parts. When the control circuit 111 for example, determines that the calibration information for a particular robot part, such as. B. the limb 1154D , are not sufficiently accurate, the control circuit 111 determine that the calibration information for downstream arm parts, such as B. the limb 1154E and the robotic hand 1155 , are not sufficiently accurate.

In one embodiment, one or more steps of the method 1200 such as B. the steps 1203 to 1215 , in response to a user command from the robot controller 110 or another computer system. For example, a user (e.g., a system operator) can manually initiate a calibration verification process that includes the steps 1203 to 1215 includes. In one embodiment, the steps 1203 to 1215 run during a rest period. The rest period can e.g. B. be a period of time in which a robot operation, such as picking up objects from a conveyor belt or a pallet, is not performed. In one embodiment, one or more steps of the method 1200 such as B. the steps 1203 to 1215 , as a reaction to a defined trigger condition from the robot controller 110 or another computer system. As described above, the trigger condition can e.g. B. an unplanned event such as a collision with the robot (e.g. 1150 ), an earthquake or other natural disaster that leads to a change in the robot (e.g. 1150 ) and / or on the camera (e.g. 1170 ) can lead. In some cases, the trigger condition may include a certain period of time that elapses after a previous calibration process, such as B. a calibration process to determine the calibration information from step 1201 . In such an example, the earlier calibration process can be a first calibration process, during the calibration process of step 1215 can be a second calibration process.

Additional discussion of various embodiments

Embodiment A1 relates to a robot control system comprising a communication interface configured to communicate with a robot having a base and a robot arm with a verification symbol placed thereon and with a camera with a camera field of view. The robotic control system further includes a control circuit configured to perform a first camera calibration (or more generally a calibration process) to determine camera calibration information associated with the camera (or more generally calibration information associated with the robotic control system). The control circuit is further configured to: a) control the robot arm in such a way that it moves the verification symbol to a location within the camera field of view during or after the first camera calibration by issuing a first movement command to the robot via the communication interface, the location being a The reference location of one or more reference locations for verification of the first camera calibration is, b) to receive an image of the verification symbol from the camera via the communication interface, wherein the camera is configured to capture the image of the verification symbol at the reference location, the image being a reference image for the verification, c) determining a reference image coordinate for the verification, the reference image coordinate being a coordinate at which the verification symbol appears in the reference image; d) based on the camera calibration information, controlling the movement of the robot arm to perform a robot operation by outputting a second movement command based on the camera calibration information to the robot via the communication interface; e) detect a rest period during robot operation; f) controlling the robot arm in order to move the verification symbol to at least the reference location during the rest period by issuing a third movement command to the robot via the communication interface; g) to receive an additional image of the verification symbol from the camera via the communication interface during the idle time, the camera being configured to capture the additional image of the verification symbol at least at the reference location, the additional image being a verification image for the verification; h) determine a verification image coordinate used for verification, the verification image coordinate being a coordinate at which the verification symbol appears in the verification image; i) determine a deviation parameter value based on an amount of deviation between the reference image coordinate and the verification image coordinate, the reference image coordinate and the verification image coordinate both being associated with the reference location, the deviation parameter value being a change in the camera since the first camera calibration or a change in a relationship between the Camera and robot has indicated i) closed since the first camera calibration determine whether the deviation parameter value exceeds a defined threshold, and j) in response to a determination that the deviation parameter value exceeds the defined threshold, to perform a second camera calibration to determine updated camera calibration information (or more generally to perform a second calibration process to update calibration information to be determined).

Embodiment A2 comprises the robot control system of Embodiment A1, wherein the control circuit is configured to, in response to a determination that the deviation parameter value does not exceed the defined threshold value, control the robot to continue the robot operation after the rest period without performing additional camera calibration by issuing a fourth movement command to the robot via the communication interface.

Embodiment A3 comprises the robot control system of embodiment A1 or A2, wherein the one or more reference locations are multiple reference locations each corresponding to multiple reference image coordinates, the reference image coordinate being one of the multiple reference image coordinates. In this embodiment, the control circuit is further configured to determine a plurality of verification image coordinates each corresponding to the plurality of reference locations, wherein the verification image coordinate is one of the plurality of verification image coordinates, and wherein the deviation parameter value is based on corresponding deviation amounts between the plurality of reference image coordinates and the plurality of verification image coordinates for the plurality of reference locations wherein each deviation amount of the corresponding deviation amounts is between: (a) a reference image coordinate corresponding to a corresponding reference location of the plurality of reference locations, and (b) a verification image coordinate corresponding to the same reference location.

Embodiment A4 includes the robot control system of Embodiment A3, wherein the plurality of verification image coordinates are corresponding coordinates at which the verification symbol appears in a plurality of verification images, the verification image being one of the plurality of verification images, and wherein the control circuit is configured to control the camera, all of them to capture the multiple verification images during the rest period.

Embodiment A5 comprises the robot control system of embodiment A3, wherein the plurality of verification image coordinates are corresponding coordinates at which the verification symbol appears in a plurality of verification images, the verification image being one of the plurality of verification images, and wherein the control circuit is configured to control the camera such that it captures the multiple verification images in different rest periods, the rest period being one of the various rest periods.

Embodiment A6 comprises the robot control system of one of the embodiments A1 to A5, wherein the verification symbol comprises a first region with a first color and a second region with a second color, a ratio of an area of the first region to an area of the second region being defined and in a storage device of the robot control system is stored as a defined relationship.

Embodiment A7 includes the robot control system of embodiment A6, wherein the control circuit is configured to identify the verification symbol in the reference image or in the verification image based on the defined relationship.

The embodiment A8 comprises the robot control system of the embodiment A7, wherein the robot arm has a calibration pattern arranged thereon, the reference image comprising the verification symbol and the calibration pattern, the control circuit being configured to determine whether a part of the reference image is the verification symbol or the calibration pattern by determining whether the part of the reference image has a first image region having the first color and a second image region having the second color, and whether a ratio between an area of the first image region and an area of the second image region is equal to the defined ratio.

The embodiment A9 includes the robot control system of any one of the embodiments A1 to A8, wherein the verification symbol includes a first shape and a second shape that are concentric with each other, with the respective centers of the first shape and the second shape being substantially at the same location.

The embodiment A10 includes the robot control system of the embodiment A9, wherein the control circuit is configured to determine the reference image coordinate by: a) determining a first coordinate of a center of the first shape in the reference image; b) determining a second coordinate of a center of the second shape in the reference image; and c) determining the reference image coordinate as a mean value of the first coordinate and the second coordinate in the reference image. In this embodiment, the control circuit is configured to determine the verification image coordinate by: d) determining a first coordinate of a center of the first shape in the verification image; e) determining a second coordinate of a center of the second shape in the verification image; and f) determining the verification image coordinate as an average of the first coordinate and the second coordinate in the verification image.

The embodiment A11 includes the robot control system of any one of the embodiments A1 to A10, wherein the control circuit is configured to identify the verification symbol in the reference image or the verification image by identifying a circular ring, the verification symbol being shaped as the circular ring.

The embodiment A12 includes the robot control system of any one of the embodiments A1 to A11, wherein the control circuit is further configured to determine a temperature of an environment in which the robot is located; and adjust at least one of the defined threshold or the camera calibration information based on the measured temperature.

The embodiment A13 includes the robot control system of the embodiment A12, wherein the control circuit is configured to adjust the defined threshold based on the temperature by: setting the defined threshold to have a first value when the temperature is outside a defined range; and setting the threshold value to have a second value that is lower than the first value when the temperature is within the defined range.

The embodiment A14 comprises the robot control system of one of the embodiments A1 to A13, wherein the one or more reference locations to which the control circuit is configured to cause the verification symbol to be moved by means of the robot arm comprise a plurality of reference locations which are located on a surface of a Sphere are arranged, which is concave with respect to the camera.

The embodiment A15 includes the robot control system of embodiment A14, wherein the control circuit is further configured to control the robot arm so that the verification symbol is moved such that it is tangent to the surface of the sphere at each reference location of the plurality of reference locations.

The embodiment A16 comprises the robot control system of any one of the embodiments A1 to A15, wherein the control circuit is configured to control the robot arm to move the verification symbol so that it directly faces the camera when the verification symbol is moved to the reference location.

The embodiment A17 includes the robot control system of any one of the embodiments A1 to A16, wherein the control circuit is configured to detect the idle time of the robot operation by detecting a time period during which the robot is free from performing a robot task during the robot operation.

The embodiment A18 comprises the robot control system of embodiment A17, wherein the control circuit is configured to control the robot arm to interact with objects on a conveyor belt that can be reached by the robot arm, the control circuit being configured to detect the idle time, by detecting that there is no object on the conveyor belt, or by detecting that a distance between the robot and a closest object on the conveyor belt exceeds a defined distance threshold.

Embodiment B1 relates to a computer system comprising a communication interface and a control circuit. The communication interface is configured to communicate with: (i) a camera with a camera field of view and (ii) a robot with a robotic arm with a plurality of arm parts movably attached to one another and with a group of verification symbols arranged on corresponding arm parts of the plurality of arm parts are. The control circuit is configured to, when the robot arm is in the camera's field of view, perform a method comprising: issuing a movement command for controlling movement of the robot arm to perform a robot operation, the movement command based on calibration information; Determine a group of Reference image coordinates, the group of reference image coordinates being corresponding coordinates at which the group of verification symbols appear in a reference image, the reference image being an image for representing the group of verification symbols and being generated by the camera during a first period of time when the robot arm is in a first pose is located; Issuing an additional move command during a second period of time after the first period of time to control the robot arm to move into the first pose; Receiving a verification image which is an additional image for representing the group of verification symbols and is generated by the camera when the robot arm has been moved into the first pose as a result of the additional movement command; Determining a group of verification image coordinates, the group of verification image coordinates being corresponding coordinates at which the group of verification symbols appear in the verification image; Determining a group of corresponding deviation parameter values based on corresponding deviation amounts between the group of reference image coordinates and the group of verification image coordinates, the group of corresponding deviation parameter values being associated with the group of verification symbols; Determining whether at least one deviation parameter value of the group of corresponding deviation parameter values exceeds a defined deviation threshold value; and executing at least one of the following in response to a determination that at least one deviation parameter value of the group of corresponding deviation parameter values exceeds the defined deviation threshold value: issuing a notification that at least one of the group of corresponding deviation parameter values exceeds the defined deviation threshold value, or executing a calibration process for Determination of updated calibration information. The control circuit can use the method e.g. B. by executing commands on a non-transitory computer readable medium.

Embodiment B2 comprises the computer system of embodiment B1, wherein the first pose is associated with a first additional movement command output during the first time period, the first additional movement command having one or more actuation parameter values for controlling the robot arm so that it moves into the first pose , wherein the reference image is generated by the camera when the robot arm is in the first pose as a result of the first additional movement command. Furthermore, in this embodiment, the additional movement command issued during the second time period is a second additional movement command and also includes the one or more actuation parameter values.

The embodiment B3 comprises the computer system of embodiment B2, wherein each verification symbol of the group of verification symbols has a circular shape, and wherein the one or more actuation parameter values of the first additional movement command and the second additional movement command cause the group of verification symbols to be positioned such that the group of verification symbols appears in the reference image and in the verification image without eccentricity or with corresponding amounts of eccentricity which are smaller than a defined eccentricity threshold value.

The embodiment B4 comprises the computer system of one of the embodiments B1 to B3, wherein the control circuit is configured to determine one or more actuation parameter values for the additional movement command that cause each verification symbol of the group of verification symbols to be moved such that it is directly the Facing the camera.

Embodiment B5 includes the computer system of embodiment B4, wherein the one or more actuation parameter values cause the set of verification symbols to tangent to one or more imaginary spheres that are concave with respect to the camera.

The embodiment B6 comprises the computer system of one of the embodiments B 1 to B5, wherein if at least one verification symbol of the group of verification symbols is shaped as a circular ring, the control circuit is configured to carry out the at least one verification symbol in the reference image and in the verification image Identify the circular ring to identify.

The embodiment B7 comprises the computer system of the embodiment B6, wherein when the group of verification symbols is shaped as corresponding circular rings with different corresponding sizes, the control circuit is configured to determine the at least one verification symbol based on a size of a corresponding circular ring which has the at least forms a verification symbol to identify.

The embodiment B8 comprises the computer system of one of the embodiments B1 to B7, wherein the control circuit is configured to identify at least one verification symbol of the group of verification symbols in the reference image based on a defined model that describes a geometry of the robot arm.

Embodiment B9 comprises the computer system of embodiment B8, wherein the control circuit is configured to determine, based on the model, a region within the reference image in which the at least one verification symbol is expected, and according to the at least one verification symbol within the region of the reference image search.

The embodiment B10 comprises the computer system of one of the embodiments B1 to B9, the control circuit being configured to determine, based on a comparison between the group of corresponding deviation parameter values, a type of calibration error that causes the at least one deviation parameter value to exceed the defined deviation threshold value , the type of calibration error indicating whether a loss of accuracy in the calibration information is representative of a change in the robot or whether the loss of accuracy is representative of a change in the camera.

Embodiment B 11 comprises the computer system of embodiment B10, wherein the control circuit is configured to determine whether the group of corresponding deviation parameter values completely exceeds the defined deviation threshold value, and whether the group of corresponding deviation parameters differ from one another by more than a defined uniformity threshold value. The control circuit in this embodiment is further configured to, in response to determining that the group of corresponding deviation parameter values completely exceeds the defined deviation threshold value and that they do not differ from one another by more than the defined uniformity threshold value, determine that the type of calibration error a Is a calibration error that is representative of a change in the camera.

The embodiment B12 comprises the computer system of embodiment B11, wherein the control circuit is further configured, in response to a determination that one or more of the group of corresponding deviation parameter values do not exceed the defined deviation threshold value, or that the group of corresponding deviation parameter values differ from one another by more than the defined uniformity threshold to determine that the type of calibration error is a calibration error representative of a change in the robot.

Embodiment B13 comprises the computer system of one of the embodiments B1 to B12. In this embodiment, the calibration information is connected to a first calibration process and the calibration process for generating the updated calibration information is a second calibration process that follows the first calibration process, wherein the control circuit is configured to output the additional movement command and in response to a defined triggering condition Issue camera command to receive the verification image. The defined trigger condition comprises at least one of: a defined time period that has elapsed since the first calibration process, or an event that was not planned by the computer system and results in a change to the robot or the camera.

Embodiment B14 includes the computer system of any one of Embodiments B1 to B13, wherein when the plurality of robot arm parts are arranged as a series of arm parts from a base of the robot to a robot end-effector, the control circuit is configured to: determine whether a deviation parameter value for a first verification symbol the set of verification symbols exceeds the defined deviation threshold; Identifying, from the plurality of arm parts, a first arm part on which the first verification symbol is arranged; and in response to a determination that the deviation parameter value for the first verification symbol does not exceed the defined deviation threshold, determining that the calibration information is accurate for the first arm portion and for at least one additional arm portion that precedes the first arm portion in the series of arm portions.

The embodiment B15 comprises the computer system of one of the embodiments B1 to B14, wherein when the computer system is in communication with a conveyor belt used for robot operation, the control circuit is configured to do so in response to a determination that the at least one deviation parameter value exceeds the defined deviation threshold value is to cause the conveyor belt to stop.

The embodiment B16 comprises the computer system of one of the embodiments B1 to B15, wherein the reference image is a first reference image associated with the first pose of the robot arm, the verification image being a first verification image associated with the first pose, the The group of reference image coordinates is a first group of reference image coordinates associated with the first pose, and the group of verification image coordinates is a first group of verification image coordinates associated with the first pose. In this embodiment, the control circuit is configured to: determine a second group of reference image coordinates, the second group of reference image coordinates being corresponding coordinates at which the group of verification symbols appear in a second reference image, the second reference image being generated by the camera when there is the robot arm is in a second pose; After the second reference image has been generated, outputting a further movement command for controlling the robot arm in order to move the robot arm into the second pose; Receiving a second verification image which also represents the group of verification symbols and is generated by the camera when the robot arm has been moved into the second pose as a result of the further movement command; and determining a second group of verification image coordinates, the second group of verification image coordinates being corresponding coordinates at which the group of verification symbols appear in the second verification image. In this embodiment, the group of corresponding deviation parameter values is further based on the corresponding deviation amounts between the second group of reference image coordinates and the second group of verification image coordinates.

While various embodiments have been described above, it should be understood that they are presented as illustrations and examples of the present invention, and not as limitations. It will be understood by those skilled in the art that various changes in form and detail can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the present invention should not be limited by any of the exemplary embodiments discussed above, but should be defined only in accordance with the appended claims and their equivalents. It should also be understood that any feature of any embodiment discussed herein and any reference cited herein can be used in combination with the features of any other embodiment. All patents and publications discussed herein are incorporated herein by reference in their entirety.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 16/732832 [0001]
• US 16/525004 [0001]
• US 16/369630 [0001]
• US 62/916798 [0001]
• US 16/295940 [0008, 0041]

Non-patent literature cited

• "Method and Control System for Verifying and Updating Camera Calibration for Robot Control", filed January 2, 2020 [0001]
• "Method and Control System for Verifying and Updating Camera Calibration for Robot Control", filed on July 29, 2019 [0001]
• "Method and Control System for Verifying and Updating Camera Calibration for Robot Control", filed on March 29, 2019 [0001]

Claims (16)

Computer system comprising: a communication interface configured to communicate with: (i) a camera with a camera field of view and (ii) a robot with a robotic arm with multiple arm parts movably attached to each other and with a group of verification symbols attached to corresponding arm parts of the multiple Arm parts are arranged; and a control circuit configured when the robot arm is in the camera's field of view to: Issuing a movement command for controlling movement of the robot arm to perform a robot operation, the movement command based on calibration information; Determining a group of reference image coordinates, the group of reference image coordinates being corresponding coordinates at which the group of verification symbols appear in a reference image, the reference image being an image for representing the group of verification symbols and being generated by the camera during a first time period when the robot arm is in a first pose; Issuing, during a second time period after the first time period, an additional movement command for controlling the robot arm to move into the first pose; Receiving a verification image which is an additional image for representing the group of verification symbols and which is generated by the camera when the robot arm has been moved into the first pose as a result of the additional movement command; Determining a group of verification image coordinates, the group of verification image coordinates being corresponding coordinates at which the group of verification symbols appear in the verification image; Determining a group of corresponding deviation parameter values based on corresponding deviation amounts between the group of reference image coordinates and the group of verification image coordinates, the group of corresponding deviation parameter values being connected to the group of verification symbols, Determining whether at least one deviation parameter value of the group of corresponding deviation parameter values exceeds a defined deviation threshold value, and Executing at least one of the following in response to a determination that at least one deviation parameter value of the group of corresponding deviation parameter values exceeds the defined deviation threshold value: issuing a notification that at least one of the group of corresponding deviation parameter values exceeds the defined deviation threshold value, or executing a calibration process for determining updated ones Calibration information. Computer system according to Claim 1 wherein the first pose is associated with a first additional movement command output during the first time period, the first additional movement command having one or more actuation parameter values for controlling the robot arm so that it moves into the first pose, the reference image being generated by the camera, when the robot arm is in the first pose as a result of the first additional movement command, and wherein the additional movement command issued during the second time period is a second additional movement command and also includes the one or more actuation parameter values. Computer system according to Claim 2 wherein each verification symbol of the group of verification symbols has a circular shape, and wherein the one or more actuation parameter values of the first additional movement command and the second additional movement command cause the group of verification symbols to be positioned such that the group of verification symbols in the reference image and in appears in the verification image without eccentricity or with corresponding amounts of eccentricity that are smaller than a defined eccentricity threshold value. Computer system according to Claim 1 wherein the control circuit is configured to determine one or more actuation parameter values for the additional movement command that causes each verification symbol of the group of verification symbols to be moved so that it directly faces the camera; or wherein the control circuit is configured to determine one or more actuation parameter values for the additional movement command that causes each verification symbol of the group of verification symbols to be moved so that it is directly facing the camera, and which one or the multiple actuation parameter values cause the set of verification symbols to be tangent to one or more imaginary spheres that are concave with respect to the camera. Computer system according to Claim 1 wherein if at least one verification symbol of the group of verification symbols is shaped as a circular ring, the control circuit is configured to identify the at least one verification symbol in the reference image and in the verification image by identifying the circular ring, or where, if at least one verification symbol of the group of verification symbols is shaped as a circular ring, the control circuit is configured to identify the at least one verification symbol in the reference image and in the verification image by identifying the circular ring, and if the group of verification symbols as corresponding circular rings with different corresponding sizes is formed, the control circuit is configured to determine the at least one verification symbol based on a size of a corresponding circular ring that forms the at least one verification symbol et to identify. Computer system according to Claim 1 wherein the control circuit is configured to identify at least one verification symbol of the group of verification symbols in the reference image based on a defined model that describes a geometry of the robot arm, or wherein the control circuit is configured to identify the at least one verification symbol of the group of verification symbols in the reference image based on a defined model that describes a geometry of the robot arm, and wherein the control circuit is configured, based on the model, to assign a region within the reference image in which the appearance of the at least one verification symbol is expected determine and search for the at least one verification symbol within the region of the reference image. Computer system according to Claim 1 , wherein the control circuit is configured to determine, based on a comparison between the group of corresponding deviation parameter values, a type of calibration error that causes the at least one deviation parameter value to exceed the defined deviation threshold value, the type of calibration error indicating whether a loss of accuracy in the Calibration information is representative of a change in the robot, or whether the loss of accuracy is representative of a change in the camera. Computer system according to Claim 7 , wherein the control circuit is configured to determine whether the group of corresponding deviation parameter values completely exceeds the defined deviation threshold value, and whether the group of corresponding deviation parameters differs from one another by more than a defined uniformity threshold value, and in response to determining that the group of corresponding deviation parameter values completely exceeds the defined deviation threshold value and does not differ from one another by more than the defined uniformity threshold value, to determine that the type of calibration error is a calibration error that is representative of a change in the camera. Computer system according to Claim 8 wherein the control circuit is further configured, in response to a determination, that one or more of the group of corresponding deviation parameter values do not exceed the defined deviation threshold value or that the group of corresponding deviation parameter values differ from one another by more than the defined uniformity threshold value, that the type of calibration error is a calibration error that is representative of a change in the robot. Computer system according to Claim 1 wherein the calibration information is associated with a first calibration process and the calibration process for generating the updated calibration information is a second calibration process following the first calibration process, wherein the control circuit is configured to output the additional movement command and a camera command to receive the verification image in response to output a defined trigger condition, the defined trigger condition comprising at least one of: a defined time period that has elapsed since the first calibration process, or an event that was not planned by the computer system and that results in a change to the robot or the camera. Computer system according to Claim 1 wherein, when the plurality of robot arm parts are arranged as a series of arm parts from a base of the robot to a robot end effector, the control circuit is configured to: Determining whether a deviation parameter value for a first verification symbol of the group of verification symbols exceeds the defined deviation threshold value; Identifying, from the plurality of arm parts, a first arm part on which the first verification symbol is arranged; and in response to a determination that the value of the deviation parameter for the first verification symbol does not exceed the defined deviation threshold, determining that the calibration information is accurate for the first arm portion and for at least one additional arm portion that precedes the first arm portion in the series of arm portions are. Computer system according to Claim 1 wherein when the computer system is in communication with a conveyor belt used for robot operation, in response to a determination that the at least one deviation parameter value exceeds the defined deviation threshold, the control circuit is configured to cause the conveyor belt to stop. Computer system according to Claim 1 , wherein the reference image is a first reference image associated with the first pose of the robot arm, the verification image is a first verification image associated with the first pose, the group of reference image coordinates being a first group of reference image coordinates that corresponds to the first pose, and the set of verification image coordinates is a first set of verification image coordinates associated with the first pose, wherein the control circuit is configured to: determine a second set of reference image coordinates, the second set of reference image coordinates being corresponding coordinates the set of verification symbols appearing in a second reference image, the second reference image being generated by the camera when the robot arm is in a second pose; Outputting, after the generation of the second reference image, a further movement command for controlling the robot arm in order to move the robot arm into the second pose; Receiving a second verification image which also represents the group of verification symbols and is generated by the camera when the robot arm has been moved into the second pose as a result of the further movement command; Determining a second group of verification image coordinates, the second group of verification image coordinates being corresponding coordinates at which the group of verification symbols appear in the second verification image, the group of corresponding deviation parameter values further being based on corresponding deviation amounts between the second group of reference image coordinates and the second group of Verification image coordinates based. Non-transitory computer readable medium having instructions stored thereon which, when executed by a control circuit of a computer system, cause the control circuit to issue a movement command based on calibration information, the computer system configured to communicate with: (i) a camera having a camera field of view and (ii ) a robot having a robot arm which comprises a plurality of arm parts which are movably attached to each other and which comprises a group of verification symbols which are arranged on corresponding arm parts of the plurality of arm parts, and wherein the movement command is for controlling the movement of the robot arm around one To perform robot operations; Determining a group of reference image coordinates, the group of reference image coordinates being corresponding coordinates at which the group of verification symbols appear in a reference image, the reference image being an image for representing the group of verification symbols and being generated by the camera during a first time period when the robot arm is in a first pose; Issuing, during a second time period after the first time period, an additional movement command for controlling the robot arm to move into the first pose; Receiving a verification image which is an additional image for representing the group of verification symbols and which is generated by the camera when the robot arm has been moved into the first pose as a result of the additional movement command; Determining a group of verification image coordinates, the group of verification image coordinates being corresponding coordinates at which the group of verification symbols appear in the verification image; Determining a group of corresponding deviation parameter values based on corresponding deviation amounts between the group of reference image coordinates and the group of Verification image coordinates are based, wherein the group of corresponding deviation parameter values is associated with the group of verification symbols, determining whether at least one deviation parameter value of the group of corresponding deviation parameter values exceeds a defined deviation threshold value, and performing at least one of the following in response to a determination that at least one of the Deviation parameter values of the group of corresponding deviation parameter values exceeds the defined deviation threshold value: outputting a notification that at least one of the group of corresponding deviation parameter values exceeds the defined deviation threshold value, or executing a calibration process to determine updated calibration information. Non-transitory computer readable medium after Claim 14 , wherein the first pose is associated with a first additional movement command output during the first time period, and the first additional movement command has one or more actuation parameter values for controlling the robot arm to move in the first pose, the additional movement command occurring during the second time period is output, is a second additional movement command and also includes the one or more actuation parameter values; or wherein the commands, when executed by the control circuit, further cause the control circuit to determine, based on a comparison between the group of corresponding deviation parameter values, a type of calibration error that causes the at least one deviation parameter value to exceed the defined deviation threshold value, the type of calibration error indicates whether a loss of accuracy in the calibration information is representative of a change in the robot or whether the loss of accuracy is representative of a change in the camera. A method carried out by a computer system comprising: Issuing, by the computer system, a movement command based on calibration information, the computer system being configured to communicate with: (i) a camera having a camera field of view and (ii) a robot having a robotic arm having a plurality of arm parts movably attached to one another, and having a group of verification symbols disposed on respective arm parts of the plurality of arm parts, and wherein the movement command is for controlling movement of the robot arm to perform a robot operation; Determining, by the computer system, a group of reference image coordinates, the group of reference image coordinates being corresponding coordinates at which the group of verification symbols appear in a reference image, the reference image being an image for representing the group of verification symbols and of the camera during a first Time period is generated when the robot arm is in a first pose; Issuing, by the computer system, during a second time period after the first time period, an additional move command to control the robotic arm to move into the first pose; Receiving a verification image by the computer system, the verification image being an additional image to represent the group of verification symbols and generated by the camera when the robot arm has been moved to the first pose as a result of the additional movement command; Determining, by the computer system, a group of verification image coordinates, the group of verification image coordinates being corresponding coordinates at which the group of verification symbols appear in the verification image; Determining, by the computer system, a group of corresponding deviation parameter values based on corresponding deviation amounts between the group of reference image coordinates and the group of verification image coordinates, the group of corresponding deviation parameter values being connected to the group of verification symbols, Determining, by the computer system, that at least one deviation parameter from the group of corresponding deviation parameter values exceeds a defined deviation threshold value, and Executing at least one of the following, by the computer system in response to a determination that at least one of the deviation parameter values of the group of corresponding deviation parameter values exceeds the defined deviation threshold value: issuing a notification that at least one of the group of corresponding deviation parameter values exceeds the defined deviation threshold value, or executing a calibration process to determine updated calibration information.

Images