WO2022061673A1

WO2022061673A1 - Calibration method and device for robot

Info

Publication number: WO2022061673A1
Application number: PCT/CN2020/117538
Authority: WO
Inventors: 华韬; 李�浩; 席宝时; 吴剑强
Original assignee: 西门子（中国）有限公司
Priority date: 2020-09-24
Filing date: 2020-09-24
Publication date: 2022-03-31
Also published as: CN116157837A

Abstract

A calibration method for a robot, comprising executing a camera calibration process, wherein executing a camera calibration process comprises the following steps: capturing an image of an environment and forming a three-dimensional virtual object of the environment (301); placing a calibration object (403) in a target area of the environment on the basis of the captured image of the environment and the three-dimensional virtual object of the environment (302); using a first camera (102) to capture an image of the calibration object (403) by means of relative movement between the first camera (102) associated with a robot (101) and the calibration object (403) (303), wherein the first camera (102) is a 2D camera; and determining, on the basis of the image of the calibration object (403) captured by the first camera (102), a parameter of the first camera (102) used for calibration (304). According to the calibration method, a user (107) can be guided to perform camera calibration by means of an AR technology, such that operation is intuitive and convenient, operation complexity is reduced, and accuracy is improved. The present invention further relates to a calibration device for a robot.

Description

Calibration method and device for robot

technical field

The present disclosure relates to the field of machine vision, and more particularly, to a calibration method, apparatus, computing device, computer-readable storage medium, and program product for a robot.

Background technique

With the development of robotics, more and more robotic workstations are used in industrial applications, such as automatic loading and unloading, welding, stamping, spraying, and various other processes. Robots can be flexibly combined with different equipment to meet demanding production process requirements. It can easily realize multi-machine linkage automatic production line and digital factory layout, save manpower to the greatest extent and improve enterprise efficiency.

Although the application of robots in industry has many advantages, the high technical and professional requirements for integration, operation and maintenance in robotic applications limit its use. In particular, calibration including vision system calibration (camera calibration), TCP (tool center point) calibration, and hand-eye calibration has become a major problem for users using robots.

SUMMARY OF THE INVENTION

Robot calibration is one of the key technologies in the process of robot operation. However, for ordinary users, the existing calibration methods require users to have considerable professional knowledge and spend a lot of time and energy to calibrate the robot to determine the operation of the robot. Whether it is ideal or not, this increases the user's operating threshold. For example, the existing calibration methods require a lot of manual operations, the calibration efficiency is low, and the operation process is complicated, and is highly dependent on the operator's subjective judgment and experience.

A first embodiment of the present disclosure proposes a calibration method for a robot, the calibration method includes performing a camera calibration process, wherein performing the camera calibration process includes the steps of capturing an image of an environment and forming the environment based on the captured image of the environment and the three-dimensional virtual object of the environment, placing a calibration object in a target area in the environment; through the first camera associated with the robot and the calibration relative movement between objects to capture an image of the calibration object using the first camera, wherein the first camera is a 2D camera; and determining based on the image of the calibration object captured by the first camera Parameters of the first camera used for calibration.

In this embodiment, the augmented reality (AR) technology can be used to guide the user to perform camera calibration, so that the operation is intuitive and convenient, which advantageously reduces the operation complexity and improves the accuracy, thereby effectively lowering the user's operation threshold, making Users without professional knowledge can also easily achieve camera calibration.

A second embodiment of the present disclosure provides a calibration device for a robot, the calibration device includes a camera calibration unit, the camera calibration unit includes an environment capture module configured to capture an image of an environment and form the environment the three-dimensional virtual object; the placement module is configured to place the calibration object in the target area in the environment based on the captured image of the environment and the three-dimensional virtual object of the environment; the first camera capture module is configured to capture an image of the calibration object using the first camera through relative movement between the first camera associated with the robot and the calibration object, wherein the first camera is a 2D camera; parameter determination module , configured to determine parameters of the first camera for calibration based on the image of the calibration object captured by the first camera.

A third embodiment of the present disclosure provides a computing device comprising: a processor; and a memory for storing computer-executable instructions that, when executed, cause the processor to The method described in the first embodiment is performed.

A fourth embodiment of the present disclosure proposes a computer-readable storage medium having computer-executable instructions stored thereon for executing the steps described in the first embodiment. method described.

A fifth embodiment of the present disclosure proposes a computer program product tangibly stored on a computer-readable storage medium and comprising computer-executable instructions that, when executed, cause At least one processor executes the method described in the first embodiment.

Description of drawings

The features, advantages and other aspects of various embodiments of the present disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description, In the attached picture:

FIG. 1 illustrates an exemplary scenario in which embodiments of the present disclosure may be applied.

FIG. 2 illustrates another exemplary scenario in which embodiments of the present disclosure may be applied.

FIG. 3 shows a flowchart of a calibration method for a robot according to an embodiment of the present disclosure.

4 illustrates an exemplary arrangement of a camera calibration process that may be used with a robot, according to embodiments of the present disclosure.

FIG. 5 shows another flowchart of a calibration method for a robot according to an embodiment of the present disclosure.

FIG. 6 illustrates an exemplary calibration artifact for TCP calibration according to embodiments of the present disclosure.

7 illustrates an exemplary arrangement of a TCP calibration process that may be used with a robot, according to embodiments of the present disclosure.

FIG. 8 shows another flowchart of a calibration method for a robot according to an embodiment of the present disclosure.

9 illustrates an exemplary arrangement of a hand-eye calibration process that may be used in a robot, according to embodiments of the present disclosure.

10 shows a block diagram of an exemplary calibration apparatus for a robot according to an embodiment of the present disclosure.

11 shows a block diagram of an exemplary computing device for robotic calibration according to an embodiment of the present disclosure.

detailed description

Various exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. Although the example methods, apparatuses described below include software and/or firmware executing on hardware among other components, it should be noted that these examples are merely illustrative and should not be regarded as limiting. For example, it is contemplated that any or all hardware, software and firmware components may be implemented exclusively in hardware, exclusively in software, or in any combination of hardware and software. Accordingly, while exemplary methods and apparatus have been described below, those skilled in the art will readily appreciate that the examples provided are not intended to limit the manner in which these methods and apparatus may be implemented.

Additionally, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of methods and systems in accordance with various embodiments of the present disclosure. It should be noted that the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented using dedicated hardware-based systems that perform the specified functions or operations , or can be implemented using a combination of dedicated hardware and computer instructions. In the different drawings, the same reference numbers refer to the same or similar elements.

As used herein, the terms "including", "including" and similar terms are open-ended terms, ie, "including/including but not limited to," meaning that other content may also be included. The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment" and so on.

This paper involves various coordinate systems of the robot system, such as the base coordinate system of the robot, the tool coordinate system, and the camera coordinate system. The base coordinate system of the robot is based on the robot installation base and is used to describe the coordinate system of the movement of the robot body. The tool coordinate system TCS is a coordinate system established with the tool center point (TCP) as the origin. Before the tool is not assembled, the default TCP is the center point of the robot end flange. After the tool is assembled, the robot TCP moves to the end of the tool. The camera coordinate system is based on the camera and is used to describe the coordinate system of the object movement.

FIG. 1 illustrates an exemplary scenario 100 in which embodiments of the present disclosure may be applied. The scene 100 includes a robot 101 and an associated first camera 102 , which is affixed to the robot 101 . For example, the robot 101 may be an industrial-oriented multi-joint manipulator or a multi-degree-of-freedom robotic device. The robot 101 includes a movable end 103 on which the first camera 102 can be fixed and move together with the movement of the end 103 of the robot 101 . That is, in this scene 100, since the end of the robot 101 and the first camera 102 are fixed together, it is called eye in hand. The end 103 of the robot 101 may also have a tool portion 104 for assembling (eg, suction, insertion, etc.) tools or components (eg, welding guns, nozzles, bolts, etc.) for processing the target workpiece. The scene 100 is also arranged with a target object 105 . The target object 105 may be an actual object to be processed (eg, a target workpiece) or a calibration object (eg, a calibration plate, etc.). The first camera 102 is a 2D camera, and a second camera 106 is also arranged in the scene 100 . The second camera 106 is a 3D camera, which may be a binocular camera, a structured light camera, or any camera capable of returning depth information. In the scene 100, a second camera 106 may be disposed relatively above the robot 101 to capture and track the movement of the robot 101 (eg, its tip, etc.) and the like. In the scenario 100, the user 107 may carry (eg, hand-held, worn on the head, etc.) an AR device 108, which may include, but is not limited to, a smartphone, tablet, teach pendant, or headset equipment. The AR device 108 can take pictures of the real environment or objects in the environment from different positions and different angles within the field of view 109 of the AR device 108 to form a three-dimensional virtual object through a plurality of cameras arranged on or outside the AR device 108 . The AR device 108 may have a display to display in the virtual environment corresponding three-dimensional virtual objects of the real environment or objects in the environment.

FIG. 2 illustrates an exemplary scenario 200 in which embodiments of the present disclosure may be applied. Scene 200 is similar to scene 100 except that the associated first camera 102 is fixed outside the robot 101 . That is, in this scene 200, the first camera 102 is located outside the robot 101 and does not move with the movement of the tip 103 of the robot 101, referred to as eye to hand.

3 shows a flowchart of a calibration method 300 for a robot according to an embodiment of the present disclosure, and FIG. 4 shows an exemplary arrangement 400 of a camera calibration process that may be used for a robot according to an embodiment of the present disclosure. The method 300 may be applied to the exemplary scenario 100 shown in FIG. 1 (eyes in hands) and the exemplary scenario 200 shown in FIG. 2 (eyes out of hands). The method 300 is described below in conjunction with FIGS. 3 and 4 .

Referring to Figure 3, method 300 includes steps 301-304 for performing a camera calibration process. Additionally, the method 300 may further include steps for performing a TCP calibration process (see FIG. 5 ) and for performing a hand-eye calibration process (see FIG. 8 ).

In the process of image measurement and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a space object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are camera parameters (for example, , the camera's intrinsic and extrinsic parameters, distortion parameters). These parameters are usually obtained through experiments and calculations, and this process of solving parameters is called camera calibration. By determining camera parameters through camera calibration, lens distortion can be corrected, a corrected image can be generated, and a three-dimensional scene can be reconstructed from the obtained image.

The method 300 begins at step 301 by capturing an image of the environment and forming a three-dimensional virtual object of the environment.

In some embodiments, step 301 may include: photographing the environment from different positions and angles, thereby obtaining at least two images for each scene; The depth information of each pixel, and the three-dimensional virtual object of the environment is formed based on the depth information and the two-dimensional information contained in the image. For example, an AR device held by the user 107 may be equipped with or connected to multiple cameras that capture images of the real environment from different locations and at different angles. Taking two cameras as an example, when the two cameras take pictures of the same scene from different positions in the real environment, due to the different shooting angles of the two cameras, the different positions of the pixels in the two images of the same scene can be used. Triangulate the depth of each pixel in a two-dimensional image. Therefore, although the image captured by a single camera is two-dimensional, the three-dimensional information required to form a three-dimensional virtual object of a real environment can be obtained by supplementing the depth information obtained by triangulation on this basis.

Next, the method 300 proceeds to step 302 . In step 302, a calibration object is placed in a target area in the environment based on the captured image of the environment and the three-dimensional virtual objects of the environment. For example, the target area can be a specific area in the workbench.

In some embodiments, step 302 may include: detecting visual markers disposed in the environment from captured images of the environment; determining a target area based on the detected visual markers; capturing an image of the calibration object and forming the calibration object The 3D virtual object of the calibration object is superimposed on the 3D virtual object of the environment to be displayed as an augmented reality image; and the calibration object is moved so that the 3D virtual object of the calibration object is located in the 3D virtual object of the target area. For example, a target area in the environment can be defined by a number of identifiable visual markers, and by detecting the visual markers, the target area can be quickly determined. A plurality of visual markers may be arranged to overlay within the field of view of the first camera to be captured by the first camera. Similarly, the AR device 108 held by the user 107 can capture images of the calibration object through multiple cameras and form a three-dimensional virtual object of the calibration object. After the 3D virtual object of the calibration object is superimposed on the 3D virtual object of the environment to be displayed as an augmented reality image (eg, real-time tracking display), the user 107 or the robot 101 can be guided to move the calibration by, for example, voice, text, or image. object. For example, when the 3D virtual object of the calibration object is located at least partially outside the 3D virtual object of the target area, the AR device 108 may prompt the user 107 or guide the robot 101 to continue to move the calibration object until the 3D virtual object of the calibration object is located in 3D of the target area in virtual objects.

Referring to FIG. 4 , an exemplary arrangement 400 includes a plurality of visual markers 402 (eg, four illustrated) that define a target area 401 . The visual marker 402 may have a characteristic color (eg, a different color from the rest of the environment) and/or a characteristic pattern (eg, a specific graphic code (barcode, QR code, etc.), etc.). For example, the AR device 108 may capture an image of the environment through a camera and identify the visual marker 402 from the image, thereby determining the target area 401 defined by the visual marker 402 . The example arrangement 400 also includes a calibration object 403 (eg, a calibration plate) that has been placed in the target area 401. The calibration object 403 may have a reference pattern attached to the surface for camera calibration, the reference pattern including, but not limited to, a checkerboard pattern, a circular array pattern, a non-circular array pattern, and the like.

Next, the method 300 proceeds to step 303 . In step 303, the first camera is used to capture an image of the calibration object through relative movement between the first camera associated with the robot and the calibration object, wherein the first camera is a 2D camera.

In some embodiments, when the first camera is fixed outside the robot, step 303 may include: obtaining a plurality of actual positions for placing the calibration object in the target area, the plurality of actual positions corresponding to the three-dimensional virtual objects in the environment moving the calibration object so that the three-dimensional virtual objects of the calibration object are respectively located at the plurality of virtual positions; and using the first camera to capture images of the calibration object at the plurality of actual positions, respectively. For example, when the first camera 102 is fixed outside the robot 101 (ie, the eyes are outside the hand), the calibration object 403 needs to be placed at a suitable position in the target area 401 so that the corners on the calibration object 403 cover the first camera 102, and the correlation between the camera parameters and the different poses of the calibration object 403 will affect the determination of camera-specific parameters. In one example, based on factors such as whether the corners of the calibration object 403 are covered in the field of view of the first camera 102 , the correlation between camera parameters, and different poses, etc., the calculation method for placing the calibration object 403 in the target area 401 can be calculated. Multiple actual positions, and the user 107 or the robot 101 is designated by the AR device 108 to place the calibration object 403 at the calculated multiple actual positions. Similar to step 302 , the calibration object 403 may be placed at multiple actual locations by determining whether the three-dimensional virtual object of the calibration object 403 is placed at multiple virtual locations corresponding to multiple actual locations, and the first camera 102 may be used to place the calibration object 403 at multiple actual locations, respectively. Images of calibration object 403 are captured at multiple actual locations.

In some embodiments, when the first camera is fixed on the robot, step 303 may be performed by moving the first camera around the calibration object to a plurality of actual positions, and using the first camera to capture images of the calibration object at the plurality of actual positions respectively .

Next, the method 300 proceeds to step 304 . In step 304, parameters of the first camera for calibration are determined based on the image of the calibration object captured by the first camera. As mentioned above, the parameters of the first camera may include at least one of intrinsic parameters, extrinsic parameters, and distortion parameters. In one example, the classical Zhang Zhengyou calibration method (see "A Flexible New Technique for Camera Calibration", IEEE TRANSACTIONS ANALYSIS AND MACHINE INTELLIGENCE, Volume 22, Issue 11, 2000) can be used to determine the parameters of the first camera, using The number of checkerboard grid points is p x q. Each time the calibration board is placed, the corresponding camera parameters will change. Detect the feature points in the captured image of the calibration board (for example, the corner points of the checkerboard pattern) to form p x q an equation. When the calibration plate is placed n times, n x p x q equations are formed to estimate the internal parameters and all external parameters in the ideal case of no distortion, and then the least squares method is used to estimate the distortion coefficient under the actual radial distortion. Large likelihood method, optimized estimation to improve estimation accuracy. In other examples, the parameters of the first camera may also be determined according to any applicable existing camera calibration method, which will not be described in detail.

Compared with the traditional camera calibration process, the camera calibration process according to the method 300 can provide guidance to the user based on the AR technology during the camera calibration, so that the operation is intuitive and convenient, and the operation complexity is advantageously reduced and the accuracy is improved, so as to be effective. The user's operating threshold is lowered, so that users without professional knowledge can easily calibrate the camera.

FIG. 5 shows a flowchart of a calibration method 500 for a robot according to an embodiment of the present disclosure, FIG. 6 shows an exemplary calibration workpiece 600 for TCP calibration according to an embodiment of the present disclosure, and FIG. 7 shows An exemplary arrangement 700 of a TCP calibration process that may be used with a robot in accordance with embodiments of the present disclosure is presented. The method 500 may be applied to the exemplary scenario 100 shown in FIG. 1 (eyes in hands) and the exemplary scenario 200 shown in FIG. 2 (eyes out of hands). The method 500 is described below in conjunction with FIGS. 5 , 6 and 7 .

Referring to Figure 5, method 500 includes steps 501-504 for performing a TCP calibration process. Method 500 may be included in method 300 or performed independently of method 300 .

As previously mentioned, the tool portion 104 on the end 103 of the robot 101 can be equipped with tools to handle the target workpiece. In order to describe the pose of the tool in space, a coordinate system is bound (defined) on the tool, that is, the tool coordinate system TCS, and the origin of the tool coordinate system TCS is TCP. By default the TCP is located on the end 103 of the robot 101 , eg the center point of the flange of the tool part 104 , before the tool is not assembled. For example, the coordinates of the center point of the robot end flange can be obtained from the robot controller. Alternatively, the coordinates of the center point of the flange at the end of the robot can be obtained from the teach pendant, where the teach pendant is a handheld device for manual manipulation, programming, parameter configuration and monitoring of the robot. After the tool is assembled, the robot TCP (eg, on the tool end) needs to be calibrated and the origin of the tool coordinate system TCS is set from the default TCP to the robot TCP.

The method 500 begins at step 501 by using a second camera to separately capture a plurality of images of a first workpiece placed in a plurality of poses, wherein the plurality of poses have different spatial inclination angles, and the first workpiece is mounted on the robot. On the end, the second camera is a 3D camera and is fixed outside the robot. For example, the first workpiece may be, for example, a calibration workpiece 600 as shown in FIGS. 6 and 7 for simulating a tool in actual use. A side view 601 and a top view 602 of a calibration workpiece 600 are shown in FIG. 6 , respectively, having a top end 611 and an end 612 . The calibration workpiece 600 may be mounted to the tool portion 104 of the distal end 103 of the robot 101 through the tip 612 . Referring to FIG. 7, an exemplary arrangement 700 includes a second camera 106 and a calibration workpiece 600 placed in a number of different poses (eg, four

poses

701, 702, 703, 704 in FIG. 7), each pose having Different spatial inclination angles.

Next, method 500 proceeds to step 502 . In step 502, based on the captured images, a plurality of workpiece end coordinates of the end of the first workpiece in the 3D camera coordinate system of the second camera under the plurality of poses are determined. As mentioned before, before assembling the tool, the default TCP is the center point of the robot end flange, while after assembling the tool/workpiece, the robot TCP (eg, on the end of the tool/workpiece) needs to be calibrated. In this step, the position of the workpiece end in the 3D camera coordinate system is determined based on the 3D camera. In some embodiments, the number of poses may be at least four.

In some embodiments, step 502 may include: detecting marker points arranged on the first workpiece from the captured plurality of images to obtain a plurality of marker point coordinates of the marker points in a 3D camera coordinate system; and based on the plurality of marker points Mark the coordinates of the point, and determine the coordinates of the ends of the first workpiece in the 3D camera coordinate system. As shown in FIG. 6 , the calibration workpiece 600 may have one or more marking points 613 (eg, four in FIG. 6 ), which may be used, for example, to enable the calibration workpiece 600 or at least a portion thereof (eg, including the marking points) 613 and end 612 ) are overlaid in the field of view of the second camera 106 . After identifying the marker point 613 in the captured image, a plurality of marker point coordinates in the 3D camera coordinate system under different poses can be obtained. By calibrating the rigid relationship between the marker 613 on the workpiece 600 and the end 612 (eg, the marker is rigidly connected to the robot TCP, so the distance from each other is determined), the end 612 in the 3D camera coordinate system can be obtained The corresponding multiple workpiece end coordinates in .

In some embodiments, the marker points may have characteristic colors and/or characteristic patterns. For example, marker points 613 may have a characteristic color (eg, a different color than the rest of the calibration workpiece 600) and/or a characteristic pattern (eg, a specific graphic code (barcode, QR code, etc.), etc.) for identification/detection.

Next, the method 500 proceeds to step 503 . In step 503, a plurality of robot end coordinates of the end of the robot in the base coordinate system of the robot under the plurality of poses are obtained. For example, a plurality of robot end coordinates in the base coordinate system of the robot under different poses can be obtained from the robot controller or the teach pendant.

Next, method 500 proceeds to step 504 . In step 504, based on multiple workpiece end coordinates and multiple robot end coordinates, determine the coordinates of the robot TCP to be calibrated in the robot's base coordinate system under one posture of the robot, and determine the robot TCP and the robot TCP located on the end of the robot The relative relationship between the default TCP. An exemplary TCP calibration process is described below in conjunction with FIG. 7 .

Referring to FIG. 7 , the robot 101 assumes 4 different poses, the second camera 106 recognizes the marker point 613 on the calibration workpiece 600 , obtains the coordinates of the marker point 613 in the 3D camera coordinate system, and deduces the end of the calibration workpiece 600 612 the coordinates in the 3D camera coordinate system, while obtaining the coordinates of the center point of the end flange of the robot 101 from, for example, the robot controller. The following coordinates can be obtained:

In the first pose 701: the coordinates of the robot end in the robot base coordinate system (x1, y1, z1), and the coordinates of the workpiece end in the 3D camera coordinate system (mx1, my1, mz1) are calibrated.

In the second pose 702: the coordinates of the robot end in the robot base coordinate system (x2, y2, z2), and the coordinates of the workpiece end in the 3D camera coordinate system (mx2, my2, mz2) are calibrated.

In the third pose 703: the coordinates of the robot end in the robot base coordinate system (x3, y3, z3), and the coordinates of the workpiece end in the 3D camera coordinate system (mx3, my3, mz3) are calibrated.

In the fourth pose 704: the coordinates of the robot end in the robot base coordinate system (x4, y4, z4), and the coordinates of the workpiece end in the 3D camera coordinate system (mx3, my3, mz3) are calibrated.

Perform mathematical translation transformation on the above coordinates, so that the coordinates of the end of the calibration workpiece in the 3D camera coordinate system are the same. The following coordinates can be obtained:

In the second pose 702: the coordinates of the robot end in the robot base coordinate system (x2-mx2+mx1, y2-my2+my1, z2-mz2+mz1), and the coordinates of the workpiece end in the 3D camera coordinate system (mx1, my1,mz1).

In the third pose 703: the coordinates of the robot end in the robot base coordinate system (x3-mx3+mx1, y3-my3+my1, z3-mz3+mz1), the coordinates of the workpiece end in the 3D camera coordinate system (mx1, my1,mz1).

In the fourth pose 704: the coordinates of the robot end in the robot base coordinate system (x4-mx4+mx1, y4-my4+my1, z4-mz4+mz1), the coordinates of the workpiece end in the 3D camera coordinate system (mx1, my1,mz1).

After the translation transformation, the coordinates (x0, y0, z0) of the robot TCP in the robot base coordinate system are at the center of the sphere, and the default TCP point (the coordinate point of the center of the end flange is obtained from the teach pendant) is located on the spherical surface, which can be determined according to the following equation (1)-(4) to solve the coordinates (x0, y0, z0) of the robot TCP:

(x1-x0) ² +(y1-y0) ² +(z1-z0) ² =R ² (1)

(x2-mx2+mx1-x0) ² +(y2-my2+my1-y0) ² (2)

+(z2-mz2+mz1-z0) ² =R2

(x3-mx3+mx1-x0) ² +(y3-my3+my1, -y0) ² (3)

+(z3-mz3+mz1-z0) ² =R ²

(x4-mx4+mx1-x0) ² +(y4-my4+my1-y0) ² (4)

+(z4-mz4+mz1-z0) ² =R ²

According to the solved (x0, y0, z0) and the robot state (x1, y1, z1, rx1, ry1, rz1) at the first robot pose, the TCP value (x _tcp , y _tcp , z _tcp , rx _tcp , ry _tcp , rz _tcp ) to determine the relative relationship between the robot TCP and the default TCP located on the end of the robot.

The traditional TCP calibration process usually uses a 3, 4 or 5 point method, where the user needs to move the TCP to a reference point (for example, a fixed point placed within the robot's workspace) 3, 4 or 5 times using different poses to make the TCP coincides with the reference point. However, such traditional TCP calibration methods require the manual participation of users, require users to be familiar with operations and master professional knowledge, and have disadvantages such as slow calibration speed and insufficient calibration accuracy. Compared with the traditional TCP calibration process, according to the TCP calibration process of the method 500, the TCP calibration can be automatically performed without manual participation, the operation complexity is effectively reduced, the TCP calibration can be quickly realized, and errors caused by manual participation are avoided at the same time. , which improves the calibration accuracy.

FIG. 8 shows another flowchart of a calibration method 800 for a robot according to an embodiment of the present disclosure, and FIG. 9 shows an exemplary arrangement 900 of a hand-eye calibration process that may be used for a robot according to an embodiment of the present disclosure. The method 800 may be applied to the exemplary scenario 100 shown in FIG. 1 (eyes in hands) and the exemplary scenario 200 shown in FIG. 2 (eyes out of hands). The method 800 is described below in conjunction with FIGS. 8 and 9 .

Referring to Figure 8, method 800 includes steps 801-805 for performing a hand-eye calibration process. Method 800 may be included in method 300 or performed independently of method 300 .

The purpose of hand-eye calibration is to obtain the relationship between the robot coordinate system and the camera coordinate system, and finally transfer the result of visual recognition to the robot coordinate system. As mentioned above, according to the different places where the camera is fixed, the hand-eye calibration can be divided into two forms. If the camera and the robot end are fixed together, it is called eye in hand; It is out of sight.

The method 800 begins at step 801 by capturing a first target image of a second workpiece to be processed by the robot using a first camera, and capturing a second target image of the second workpiece using a second camera. The second workpiece may be, for example, a target workpiece 901 to be processed by the robot 101 as shown in FIG. 9 . A first target image of the target workpiece 901 may be captured by the first camera 102 and a second target image of the target workpiece 901 may be captured by the second camera 106 .

The method 800 then proceeds to step 802 . In step 802, a first transformation relationship between the 2D camera coordinate system of the first camera and the 3D camera coordinate system of the second camera is determined based on the first target image and the second target image. In this step, the transformation relationship between different camera coordinate systems can be determined, that is, the mapping relationship between the coordinates of the points in the first target image and the coordinates of the corresponding points in the second target image can be determined.

In some embodiments, step 802 may include: detecting a plurality of feature points arranged in the second workpiece from the first target image and the second target image, so as to obtain the first position of the plurality of feature points in the 2D camera coordinate system A plurality of feature coordinates and a second plurality of feature coordinates in the 3D camera coordinate system. In some embodiments, the number of feature points may be at least four. For example, the coordinates of the 2D camera coordinate system of the first camera 102 may be based on camera parameters obtained after camera calibration of the first camera 102 (the camera calibration may be, for example, the camera calibration process described herein or any other suitable camera calibration process ), calculated using the 2D information in the target image.

Referring to FIG. 9, a target workpiece 901 may have a plurality of feature points 902 (eg, 4 in the figure) arranged on the target workpiece 901, and the feature points 902 may have, for example, a specific shape, size, mark, etc., so as to be recognized/detected from the captured image Feature points. For example, the target workpiece 901 may be a PCB board, and the feature points 902 may be CAD marked screw holes arranged on the PCB board. The first camera 102 and the second camera 106 can respectively identify the feature points 902 from the captured target image, and can obtain a plurality of feature points 902 in the 2D camera coordinate system of the first camera 102 and the 3D camera coordinate system of the second camera 106 as follows: Coordinates in:

The first feature point 902 is the coordinates of the 3D camera coordinate system (x1, y1, z1), and the coordinates of the 2D camera coordinate system (X1, Y1, Z1).

The second feature point 902, 3D camera coordinate system coordinates (x2, y2, z2), 2D camera coordinate system coordinates (X2, Y2, Z2).

The third feature point 903 is the coordinates of the 3D camera coordinate system (x3, y3, z3), and the coordinates of the 2D camera coordinate system (X3, Y3, Z3).

The fourth feature point 902, 3D camera coordinate system coordinates (x4, y4, z4), 2D camera coordinate system coordinates (X4, Y4, Z4).

The coordinate conversion relationship in the 2D camera and 3D camera coordinate systems satisfies the following equation (5):

According to P _3D =H1·P _2D , where P _3D and P _2D are the coordinates of the same point in different camera coordinate systems, the homography matrix H1 between the two camera coordinate systems can be obtained, that is, to determine the first A first transformation relationship between the 2D camera coordinate system of the camera 102 and the 3D camera coordinate system of the second camera 106 .

The method 800 then proceeds to step 803 . In step 803, the end of the robot is moved to a plurality of positions, the first plurality of tool coordinates of the robot TCP in the base coordinate system of the robot are determined, and the second plurality of tool coordinates of the robot TCP in the 3D camera coordinate system are determined. In this step, the first plurality of tool coordinates of the robot TCP in the robot's base coordinate system may be determined, for example, by capturing an image of the end of the robot by a second camera (eg, using the TCP calibration process described herein or any other applicable TCP calibration process) and a second plurality of tool coordinates in the 3D camera coordinate system. In some embodiments, the number of locations may be at least four.

For example, by moving (eg, translating, or posing in different poses) the end 103 of the robot to multiple positions (eg, four), the base coordinate system of the robot TCP in the robot 101 and the camera of the second camera 106 are respectively determined The coordinates in the coordinate system.

In some embodiments, step 803 may include, for at least one of the plurality of positions, translating to the at least one position in the one pose of the robot, based on the robot TCP and the default The relative relationship between the TCPs determines at least one tool coordinate of the robot TCP in the base coordinate system of the robot. For example, in order to facilitate the calculation, based on the obtained relative relationship between the robot TCP and the default TCP in a certain posture of the robot, the end of the robot can be translated to at least one position in the robot posture, according to the control from the robot. The coordinates of the default TCP in the robot base coordinate system obtained by the controller are used to determine the coordinates of the robot TCP in the robot base coordinate system.

The method 800 then proceeds to step 804 . In step 804, a second transformation relationship between the base coordinate system of the robot and the 3D camera coordinate system is determined using the first plurality of tool coordinates and the second plurality of tool coordinates.

The coordinate transformation relationship between the base coordinate system of the robot 101 and the 3D camera coordinate system satisfies the following equation (6):

P _3D = H2 · P _robot (6)

P _3D and P _robot are the coordinates of the same point in the 3D camera coordinate system and the robot base coordinate system, respectively. From this, the homography matrix H2 between the 3D camera coordinate system and the robot base coordinate system can be obtained, that is, to determine the robot's The second transformation relationship between the base coordinate system and the 3D camera coordinate system.

The method 800 then proceeds to step 805 . In step 805, based on the first transformation relationship and the second transformation relationship, a third transformation relationship between the base coordinate system of the robot and the 2D camera coordinate system is determined.

Based on equations (5) and (6), the coordinate transformation relationship in the robot base coordinate system and the 3D camera coordinate system satisfies the following equation (7):

P _robot = H2 ^-1 · H1 · P _2D (7)

P _robot and P _2D are the coordinates of the same point in the robot base coordinate system and the 2D camera coordinate system, respectively. From this, the homography matrix H2 ^-1 · H1 between the robot base coordinate system and the 2D camera coordinate system can be obtained, namely , and determine the third transformation relationship (hand-eye relationship) between the robot's base coordinate system and the 2D camera coordinate system.

In some embodiments, the at least one location is at or near the at least one feature point. For example, to reduce hand-eye calibration errors, the at least one location may be at or near the at least one identifiable feature point. In some embodiments, AR techniques as previously described herein may be employed to intuitively and conveniently guide the end of the robot to move at or near at least one recognizable feature point, such as inserting the end 612 of the calibration workpiece 600 into the feature point 902 to further reduce the hand-eye calibration error.

Compared with the traditional hand-eye calibration process, according to the hand-eye calibration process of the method 800, the hand-eye calibration can be automatically performed without manual participation, which effectively reduces the operation complexity, and the calculation is simple, so that the hand-eye calibration can be quickly realized, and the manual operation is avoided. The error caused by participation improves the calibration accuracy.

10 shows a block diagram of an exemplary calibration apparatus 1000 for a robot according to an embodiment of the present disclosure. The apparatus 1000 includes a camera calibration unit 1010, a TCP calibration unit 1020, and a hand-eye calibration unit 1030. The device 1000 also includes a communication unit (not shown) to communicate with external other devices (eg, to receive/transmit instructions and data from and/or to external devices).

The camera unit 1010 includes an environment capture module 1011 , a placement module 1012 , a first camera capture module 1013 and a parameter determination module 1014 .

The environment capture module 1011 is configured to capture images of the environment and form three-dimensional virtual objects of the environment. In some embodiments, the environment capture module 1011 can be further configured to: take pictures of the environment from different positions and different angles, so as to obtain at least two images for each scene; The depth information of each pixel in the image; and the three-dimensional virtual object of the environment is formed based on the depth information and the two-dimensional information contained in the image.

The placement module 1012 is configured to place the calibration object in a target area in the environment based on the captured image of the environment and the three-dimensional virtual object of the environment. In some embodiments, the placement module 1012 may be further configured to: detect visual markers disposed in the environment from the captured image of the environment; determine a target area based on the detected visual markers; image and form a 3D virtual object of the calibration object; superimpose the 3D virtual object of the calibration object on the 3D virtual object of the environment for display as an augmented reality image; move the calibration object so that the 3D virtual object of the calibration object is located in the in the 3D virtual object of the target area. In some embodiments, the visual marker has a characteristic color and/or pattern.

The first camera capture module 1013 is configured to capture an image of the calibration object using the first camera through relative movement between the first camera associated with the robot and the calibration object, wherein the first camera is a 2D camera. In some embodiments, when the first camera is fixed outside the robot, the first camera capturing module 1013 may be further configured to obtain a plurality of actual positions for placing the calibration object in the target area, the plurality of actual positions corresponding to moving the calibration object so that the 3D virtual objects of the calibration object are respectively located at the plurality of virtual positions; and using the first camera to capture images of the calibration object at the plurality of actual positions, respectively.

The parameter determination module 1014 is configured to determine parameters of the first camera for calibration based on the images of the calibration object captured by the first camera.

The TCP calibration unit 1020 includes a first workpiece capture module 1021 , a workpiece coordinate determination module 1022 , a robot coordinate acquisition module 1023 and a TCP coordinate determination module 1024 .

The first workpiece capture module 1021 is configured to use a second camera to capture a plurality of images of a first workpiece placed in a plurality of poses, respectively, wherein the first workpiece is mounted on the end of the robot, and the second camera is 3D camera and fixed outside the robot.

The workpiece coordinate determination module 1022 is configured to determine, based on the plurality of captured images, a plurality of workpiece end coordinates of the end of the first workpiece in the 3D camera coordinate system of the second camera under the plurality of poses. In some embodiments, the workpiece coordinate determination module 1022 may be configured to detect marker points arranged on the first workpiece from the captured plurality of images to obtain a plurality of marker point coordinates of the marker points in the 3D camera coordinate system and, based on the coordinates of the plurality of marked points, determining a plurality of workpiece end coordinates of the end of the first workpiece in the 3D camera coordinate system. In some embodiments, the marker points have characteristic colors and/or characteristic patterns.

The robot coordinate obtaining module 1023 is configured to obtain a plurality of robot end coordinates of the end of the robot in the base coordinate system of the robot under a plurality of poses.

The TCP coordinate determination module 1024 is configured to: based on the multiple workpiece end coordinates and the multiple robot end coordinates, determine the coordinates of the robot TCP to be calibrated in the base coordinate system of the robot under one posture of the robot, and determine the robot TCP and the robot TCP The relative relationship between the default TCPs on the end of the robot.

The hand-eye calibration unit 1030 includes a second workpiece capture module 1031 , a first transformation determination module 1032 , a tool coordinate determination module 1033 , a second transformation determination module 1034 and a third transformation determination module 1035 .

The second workpiece capture module 1031 is configured to capture a first target image of a second workpiece to be processed by the robot using the first camera, and to capture a second target image of the second workpiece using the second camera.

The first transformation determination module 1032 is configured to determine a first transformation relationship between the 2D camera coordinate system of the first camera and the 3D camera coordinate system of the second camera based on the first target image and the second target image. In some embodiments, the first transformation determination module 1032 may be further configured to: detect a plurality of feature points arranged in the second workpiece from the first target image and the second target image to obtain the plurality of feature points in 2D a first plurality of feature coordinates in a camera coordinate system and a second plurality of feature coordinates in a 3D camera coordinate system; and determining the first plurality of feature coordinates using the first plurality of feature coordinates and the second plurality of feature coordinates A transformation relationship.

The tool coordinate determination module 1033 is configured to: move the end of the robot to a plurality of positions, determine the first plurality of tool coordinates of the robot TCP in the base coordinate system of the robot, and determine the second plurality of tool coordinates of the robot TCP in the 3D camera coordinate system. tool coordinates. In some embodiments, the tool coordinate determination module 1033 may be further configured to: translate to at least one of the plurality of positions in one pose of the robot to the at least one position, based on the relative relationship between the robot TCP and the default TCP relationship, at least one tool coordinate of the robot TCP in the base coordinate system of the robot is determined. In some embodiments, the at least one location is at or near at least one feature point.

The second transformation determination module 1034 is configured to determine a second transformation relationship between the base coordinate system of the robot and the 3D camera coordinate system using the first plurality of tool coordinates and the second plurality of tool coordinates.

The third transformation determining module 1035 is configured to: determine a third transformation relationship between the base coordinate system of the robot and the 2D camera coordinate system based on the first transformation relationship and the second transformation relationship.

Although the camera calibration unit 1010 , the TCP calibration unit 1020 and the hand-eye calibration unit 1030 are shown as being integrated in the apparatus 1000 in the example of FIG. 10 , it should be understood that the apparatus 1000 may include at least one of these units to separately implement the corresponding Calibration process.

11 illustrates a block diagram of an exemplary computing device 1100 for implementing industrial camera focusing, according to an embodiment of the present disclosure. Computing device 1100 includes processor 1101 and memory 1102 coupled with processor 1101 . The memory 1102 is used to store computer-executable instructions that, when executed, cause the processor 1101 to perform the methods in the above embodiments (eg, any one or more steps of the

aforementioned methods

300 , 500 or 800 ).

Also, alternatively, the above-described method can be implemented by a computer-readable storage medium. The computer-readable storage medium carries computer-readable program instructions for carrying out various embodiments of the present disclosure. A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above. Computer-readable storage media, as used herein, are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

Accordingly, in another embodiment, the present disclosure presents a computer-readable storage medium having computer-executable instructions stored thereon for performing various implementations of the present disclosure method in the example.

In another embodiment, the present disclosure presents a computer program product tangibly stored on a computer-readable storage medium and comprising computer-executable instructions that, when executed, cause At least one processor executes the methods in various embodiments of the present disclosure.

In general, the various example embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor or other computing device. While aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it is to be understood that the blocks, apparatus, systems, techniques, or methods described herein may be taken as non-limiting Examples are implemented in hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof.

Computer-readable program instructions or computer program products for executing various embodiments of the present disclosure can also be stored in the cloud, and when invoked, the user can access the data stored in the cloud for execution through the mobile Internet, fixed network or other network. The computer-readable program instructions of an embodiment of the present disclosure implement the technical solutions disclosed in accordance with various embodiments of the present disclosure.

Although embodiments of the present disclosure have been described with reference to several specific embodiments, it is to be understood that embodiments of the present disclosure are not limited to the specific embodiments disclosed. The embodiments of the present disclosure are intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

A calibration method for a robot, the calibration method comprising performing a camera calibration process, wherein performing the camera calibration process comprises the following steps:

A, capturing an image of an environment and forming a three-dimensional virtual object of said environment,

B, placing a calibration object in a target area in the environment based on a captured image of the environment and a three-dimensional virtual object of the environment,

C, capturing an image of the calibration object using a first camera associated with the robot through relative movement between the calibration object and the calibration object, wherein the first camera is a 2D camera, and

D. Determine parameters of the first camera for calibration based on the image of the calibration object captured by the first camera.
The calibration method according to claim 1, wherein the step A comprises:

A1, photographing the environment from different positions and angles, so as to obtain at least two images for each scene,

A2, the depth information of each pixel in the image is obtained by measuring the triangulation method based on the stereo image, and

A3: Based on the depth information and the two-dimensional information included in the image, a three-dimensional virtual object of the environment is formed.
The calibration method according to claim 1, wherein the step B comprises:

B1, detecting from a captured image of the environment visual markers arranged in the environment,

B2, determining the target area based on the detected visual markers,

B3, capturing the image of the calibration object and forming a three-dimensional virtual object of the calibration object,

B4, superimposing the three-dimensional virtual object of the calibration object on the three-dimensional virtual object of the environment for display as an augmented reality image, and

B5. Move the calibration object so that the three-dimensional virtual object of the calibration object is located in the three-dimensional virtual object of the target area.
The calibration method of claim 3, wherein the visual marker has a characteristic color and/or a characteristic pattern.
The calibration method according to claim 3, wherein when the first camera is fixed outside the robot, the step C comprises:

C1, obtaining a plurality of actual positions for placing the calibration object in the target area, the plurality of actual positions corresponding to a plurality of virtual positions in the three-dimensional virtual object of the environment,

C2, moving the calibration object so that the three-dimensional virtual objects of the calibration object are respectively located at the plurality of virtual positions, and

C3, using the first camera to capture images of the calibration object at the multiple actual positions respectively.
The calibration method according to claim 1, wherein the calibration method further comprises performing a TCP calibration process, wherein performing the TCP calibration process comprises the following steps:

E, using a second camera to capture a plurality of images of the first workpiece placed in a plurality of poses, wherein the plurality of poses have different spatial inclination angles, and the first workpiece is mounted on the on the end of the robot, the second camera is a 3D camera and is fixed outside the robot,

F, based on the multiple captured images, determine multiple workpiece end coordinates of the end of the first workpiece in the 3D camera coordinate system of the second camera under multiple poses,

G, obtaining a plurality of robot end coordinates of the end of the robot in the base coordinate system of the robot under the plurality of poses, and

H, based on the multiple workpiece end coordinates and the multiple robot end coordinates, determine the coordinates of the robot TCP to be calibrated in the base coordinate system of the robot under one posture of the robot, and determine the robot The relative relationship between the TCP and the default TCP located on the end of the robot.
The calibration method according to claim 6, wherein the step F comprises:

F1, detecting a marker point arranged on the first workpiece from the captured images to obtain a plurality of marker point coordinates of the marker point in a 3D camera coordinate system, and

F2: Determine, based on the coordinates of the plurality of marked points, a plurality of workpiece end coordinates of the end of the first workpiece in the 3D camera coordinate system.
The calibration method according to claim 7, wherein the marked points have characteristic colors and/or characteristic patterns.
The calibration method according to claim 6, further comprising performing a hand-eye calibration process, wherein performing the hand-eye calibration process comprises the following steps:

1, using the first camera to capture a first target image of a second workpiece to be processed by the robot, and using the second camera to capture a second target image of the second workpiece,

J, based on the first target image and the second target image, determine a first transformation relationship between the 2D camera coordinate system of the first camera and the 3D camera coordinate system of the second camera,

K, move the end of the robot to a plurality of positions, determine the first plurality of tool coordinates of the robot TCP in the base coordinate system of the robot, and determine the coordinates of the robot TCP in the 3D camera coordinate system the second plurality of tool coordinates,

L, using the first plurality of tool coordinates and the second plurality of tool coordinates to determine a second transformation relationship between the robot's base coordinate system and the 3D camera coordinate system, and

M. Determine a third transformation relationship between the base coordinate system of the robot and the 2D camera coordinate system based on the first transformation relationship and the second transformation relationship.
The calibration method according to claim 9, wherein, the step J comprises:

J1. Detect a plurality of feature points arranged in the second workpiece from the first target image and the second target image, so as to obtain the first position of the plurality of feature points in the 2D camera coordinate system a plurality of feature coordinates and a second plurality of feature coordinates in the 3D camera coordinate system, and

J2, using the first plurality of feature coordinates and the second plurality of feature coordinates to determine the first transformation relationship.
The calibration method according to claim 10, wherein the step K comprises:

For at least one of the plurality of positions, translate to the at least one position in the one pose of the robot, and determine the robot based on the relative relationship between the robot TCP and the default TCP At least one tool coordinate of the TCP in the base coordinate system of the robot.
11. The calibration method of claim 10, wherein the at least one location is at or near the at least one feature point.
A calibration device for a robot, the calibration device comprising a camera calibration unit, the camera calibration unit comprising:

an environment capture module configured to capture images of the environment and form three-dimensional virtual objects of the environment,

a placement module configured to place a calibration object in a target area in the environment based on the captured image of the environment and the three-dimensional virtual object of the environment,

a first camera capture module configured to capture an image of the calibration object using the first camera through relative movement between the first camera associated with the robot and the calibration object, wherein the first camera for 2D cameras, and

A parameter determination module configured to determine parameters of the first camera for calibration based on the image of the calibration object captured by the first camera.
The calibration device of claim 13, wherein the environment capture module is further configured to:

The environment is photographed from different positions and angles, resulting in at least two images for each scene,

Depth information for each pixel in the image is measured by a triangulation method based on a stereo image, and

A three-dimensional virtual object of the environment is formed based on the depth information and two-dimensional information contained in the image.
The calibration device of claim 13, wherein the placement module is further configured to:

detecting from a captured image of the environment visual markers arranged in the environment,

determining the target area based on the detected visual marker,

capturing an image of the calibration object and forming a three-dimensional virtual object of the calibration object,

superimposing the three-dimensional virtual object of the calibration object on the three-dimensional virtual object of the environment for display as an augmented reality image, and

The calibration object is moved such that the three-dimensional virtual object of the calibration object is located in the three-dimensional virtual object of the target area.
14. The calibration device of claim 13, wherein the visual marker has a characteristic color and/or a characteristic pattern.
The calibration device according to claim 13, wherein when the first camera is fixed outside the robot, the first camera capture module is further configured to:

obtaining a plurality of actual positions for placing the calibration object in the target area, the plurality of actual positions corresponding to a plurality of virtual positions in a three-dimensional virtual object of the environment,

moving the calibration object such that three-dimensional virtual objects of the calibration object are located at the plurality of virtual locations, respectively, and

Using the first camera to capture images of the calibration object at the plurality of actual locations, respectively.
The calibration device according to claim 13, further comprising a TCP calibration unit, the TCP calibration unit comprising:

A first workpiece capture module configured to use a second camera to capture a plurality of images of the first workpiece placed in a plurality of poses, respectively, wherein the first workpiece is mounted on the end of the robot, so the second camera is a 3D camera and is fixed outside the robot,

a workpiece coordinate determination module configured to determine, based on the captured images, a plurality of workpiece end coordinates of the end of the first workpiece in the 3D camera coordinate system of the second camera under a plurality of poses,

a robot coordinate obtaining module configured to obtain a plurality of robot end coordinates of the end of the robot in the base coordinate system of the robot under the plurality of poses, and

The TCP coordinate determination module is configured to determine, based on the plurality of workpiece end coordinates and the plurality of robot end coordinates, the coordinates of the robot TCP to be calibrated in the base coordinate system of the robot under one posture of the robot , and determine the relative relationship between the robot TCP and the default TCP located on the end of the robot.
The calibration device according to claim 18, wherein the workpiece coordinate determination module is further configured to:

detecting marker points arranged on the first workpiece from the captured images to obtain marker coordinates of the marker points in a 3D camera coordinate system, and

Based on the plurality of marker point coordinates, a plurality of workpiece end coordinates of the end of the first workpiece in the 3D camera coordinate system are determined.
19. The calibration device according to claim 19, wherein the marking points have characteristic colors and/or characteristic patterns.
The calibration device according to claim 18, further comprising a hand-eye calibration unit, the hand-eye calibration unit comprising:

A second workpiece capture module configured to capture a first target image of a second workpiece to be processed by the robot using the first camera, and to capture a second image of the second workpiece using the second camera target image,

A first transformation determination module configured to determine a first transformation between the 2D camera coordinate system of the first camera and the 3D camera coordinate system of the second camera based on the first target image and the second target image A transformation relationship,

a tool coordinate determination module configured to move the end of the robot to a plurality of positions, determine a first plurality of tool coordinates of the robot TCP in the base coordinate system of the robot, and determine the robot TCP in the the second plurality of tool coordinates in the 3D camera coordinate system,

A second transformation determination module configured to use the first plurality of tool coordinates and the second plurality of tool coordinates to determine a second transformation relationship between the robot's base coordinate system and the 3D camera coordinate system ,as well as

A third transformation determining module is configured to determine a third transformation relationship between the base coordinate system of the robot and the 2D camera coordinate system based on the first transformation relationship and the second transformation relationship.
The calibration device of claim 21, wherein the first transformation determination module is further configured to:

A plurality of feature points arranged in the second workpiece are detected from the first target image and the second target image, so as to obtain a first plurality of feature points of the plurality of feature points in the 2D camera coordinate system feature coordinates and a second plurality of feature coordinates in the 3D camera coordinate system, and

The first transformation relationship is determined using the first plurality of feature coordinates and the second plurality of feature coordinates.
The calibration device of claim 22, wherein the tool coordinate determination module is further configured to:

For at least one of the plurality of positions, translate to the at least one position in the one pose of the robot, and determine the robot based on the relative relationship between the robot TCP and the default TCP At least one tool coordinate of the TCP in the base coordinate system of the robot.
24. The calibration device of claim 23, wherein the at least one location is at or near the at least one feature point.
Computing equipment, the computing equipment includes:

processor; and

a memory for storing computer-executable instructions which, when executed, cause the processor to perform the method of any of claims 1-12.
A computer-readable storage medium having computer-executable instructions stored thereon for performing the method of any of claims 1-12.
A computer program product tangibly stored on a computer-readable storage medium and comprising computer-executable instructions which, when executed, cause at least one processor to perform the execution according to claims 1-12 The method of any of the above.