CN112743548B - Method, system and terminal for unifying hand-eye calibration of two mechanical arms - Google Patents

Abstract

The invention provides a method, a system and a terminal for unifying hand-eye calibration of two mechanical arms, which comprise the following steps: planning a motion path of the mechanical arm in the space according to the preset posture boundary point; controlling the mechanical arm to pass through the set sampling points according to the planned path, and simultaneously controlling the camera to take pictures at each sampling point; respectively calculating the pose relationship between the fixed camera and the corresponding calibration plate in each sampling point and the pose relationship between the camera arranged at the tip of the mechanical arm and the corresponding calibration plate, and obtaining the pose of the calibration plate under a camera coordinate system; and solving the pose relation between the camera and the mechanical arm according to the poses of the mechanical arm in the multiple groups of sampling points and the poses of the calibration plate in the camera coordinate system. The invention can be simultaneously suitable for two calibration modes of eyes on the hand and eyes outside the hand, only the boundary of the gesture needs to be preset manually when the calibration gesture is selected, each calibration point position does not need to be defined manually, and the labor cost is greatly reduced.

Description

A method, system and terminal for unifying hand-eye calibration of two robotic arms

technical field

The invention relates to the field of computer vision camera calibration and robotic arm control, and in particular, to a method, system and terminal for unifying the hand-eye calibration of two robotic arms.

Background technique

Camera calibration is a fundamental problem in the field of computer vision. Camera calibration is divided into camera internal parameter calibration and external parameter calibration. The purpose of internal parameter calibration is to obtain the imaging properties of the camera itself, and the purpose of external parameter calibration is to obtain the attitude and position of the camera. Hand-eye calibration usually refers to the process of solving the pose relationship between the camera and the robotic arm with one or more robotic arms and one or more cameras in a system.

According to the positional relationship between the vision sensor and the industrial robot manipulator, the vision sensor system can be divided into two forms: Eye-in-Hand and Eye-to-Hand. Eyes out of hand means that the vision sensor (industrial camera) is installed in a fixed position relative to the base and working plane of the robot, and does not move with the movement of the robot arm. In industrial production activities, this method is often used to visually locate the operation target and guide the robot to operate within a large range. Since the robot working plane is fixed and the camera installation position is fixed, it is only necessary to obtain the mapping relationship between the image plane and the robot working plane to visually guide the robot. Similarly, eye on hand refers to the vision sensor (industrial camera) mounted on the robot. As the robot moves, the position and attitude relationship between the camera and the robotic arm can be obtained by calibrating, and the object coordinates in the camera coordinate system can be converted into the robotic arm coordinate system to guide the robotic arm to complete a series of tasks.

The existing camera calibration technology is generally only suitable for specific scenarios and specific installation methods, and is not general enough to adapt to all hand-eye calibration situations, and the flexibility is relatively poor.

After searching, the Chinese invention patent CN110103217A discloses a hand-eye calibration method for an industrial robot, and proposes a general hand-eye calibration process for a fixed camera. Solve the relationship between the camera and the robotic arm by the least squares method. Specifically, set up multiple calibration data collection points in the working plane area of the industrial robot and record their coordinates in the industrial robot base calibration system, and then drive the end of the operating arm of the industrial robot to drive the calibration plate around the center of the end of the manipulator arm. The axis rotates, and in this process, the camera is controlled to collect images of the calibration board at different positions, and the calibration points are extracted from them. , so as to obtain the mapping between the industrial robot coordinate system and the camera image coordinate system, so as to realize the calibration.

The above-mentioned patents are only applicable to specific scenarios and specific installation methods, and can only calibrate the robot system with eyes outside the hands, but cannot calibrate the systems with eyes on the hands. It cannot meet the requirements for some robot systems that may need to perform both eye-on-hand and eye-out-of-hand calibrations at the same time. For the prior art, it is necessary to use two different calibration methods to complete the task. Moreover, the patent does not have a modular decoupling design for data acquisition and calculation, which is not flexible enough and has poor ductility when the equipment or algorithm is updated.

SUMMARY OF THE INVENTION

Aiming at the defects in the prior art, the purpose of the present invention is to provide a method and system for unifying the hand-eye calibration of two mechanical arms. Calibration.

One aspect of the present invention provides a method for unifying the hand-eye calibration of two robotic arms, including:

According to the predetermined attitude boundary points, plan the path of the robot arm moving in space;

Control the robotic arm to pass through the set sampling points according to the planned path, and control the camera to take pictures at each sampling point;

Calculate the pose relationship between the camera and the calibration board in each sampling point, including: calculating the pose relationship between the fixed camera and the corresponding calibration board in each sampling point, and the camera installed on the tip of the robotic arm and the calibration board. Corresponding to the pose relationship between the calibration boards, and obtain the pose of the calibration board in the camera coordinate system;

According to the pose of the manipulator in the multiple sets of sampling points and the pose of the calibration board in the camera coordinate system, the pose relationship between the camera and the manipulator is solved. Here, when the camera is "outside the hand" In the installation mode, the pose relationship between the camera and the base of the robot arm needs to be solved; when the camera is installed in the "eye on the hand" mode, the pose relationship between the camera and the tip of the robot arm needs to be solved.

Optionally, planning the path of the robotic arm moving in space according to the predetermined attitude boundary points, including:

A specified number of points are uniformly sampled in a predetermined boundary by an interpolation algorithm;

Calculated by distance in angular space to generate the shortest path through these points in sequence.

Optionally, the separately calculating the pose relationship between the camera and the calibration board in each sampling point, including:

Identify the position of the feature in the image captured by the camera based on the feature on the calibration board;

According to the position of the feature, the pose relationship between the camera and the calibration plate is solved.

Optionally, according to the pose of the robotic arm in the multiple sets of sampling points and the pose of the calibration board in the camera coordinate system, solve the pose relationship between the camera and the robotic arm, including:

For each sampling point, each sampling point here contains a set of data, which is the data obtained by the robotic arm in a certain sampling attitude. These data are: first, obtain the pose relationship between the corresponding robotic arm tip and the robotic arm base , and then obtain the pose relationship between the camera and the calibration board, and obtain matrix B and matrix A, respectively, where:

For the case where the eye is outside the hand: the camera is fixed relative to the base of the manipulator, the calibration board is fixed at the tip of the manipulator, the matrix A is defined as the external parameter matrix from the calibration board to the camera, and the matrix x is the external parameter from the camera to the base of the manipulator matrix, matrix y is the external parameter matrix from the calibration board to the tip of the manipulator, matrix B is the external parameter matrix from the tip of the manipulator to the base of the manipulator;

In all the collected data, x and y will not change and are constant unknown quantities. Here x is the pose relationship between the camera and the robot arm base to be solved; for each pose, Ax and yB represent From the calibration board to the external parameters of the manipulator base, x and y are solved by using the relationship of multiple groups of Ax=yB;

For the case where the eye is on the hand: the camera is fixed on the tip of the manipulator, and the calibration board and the base of the manipulator are relatively fixed; define matrix A as the external parameter matrix from the calibration board to the camera, and matrix x as the external parameter matrix from the camera to the tip of the manipulator , the matrix y is the external parameter matrix from the calibration board to the base of the manipulator, the matrix B is the external parameter matrix from the base of the manipulator to the tip of the manipulator, and the others are the same as the case where the eye is outside the hand.

Optionally, the method further includes: self-checking the calibration result.

Specifically, the self-checking of the calibration results includes:

Regardless of whether the eye is on the hand or the eye is outside the hand, for each collected data point, the calculated y' based on x is obtained by calculating AxB ^-1 ;

Perform error analysis between the y' calculated in multiple groups and the y finally calculated above.

A second aspect of the present invention provides a system for unifying the hand-eye calibration of two robotic arms, including:

A path planning module, which plans the path of the robotic arm moving in space according to the predetermined attitude boundary points;

a data acquisition module, which controls the robotic arm to pass through the set sampling points according to the path planned by the path planning module, and simultaneously controls the camera to take pictures at each sampling point;

A calibration calculation module, which calculates the pose relationship between the camera and the calibration board in each sampling point according to the data of the data acquisition module, including: calculating the distance between the fixed camera and the corresponding calibration board in each sampling point and the pose relationship between the camera installed on the tip of the robotic arm and the corresponding calibration board, and obtain the pose of the calibration board in the camera coordinate system;

A pose relationship calculation module, which solves the pose relationship between the camera and the robotic arm according to the pose of the robotic arm in the multiple sets of sampling points of the calibration calculation module and the pose of the calibration board in the camera coordinate system .

Optionally, the data acquisition module includes:

The robotic arm control module, which interacts with the robotic arm, sends action commands to the robotic arm and reads the operating status of the robotic arm;

The camera control module, which interacts with the camera, controls the work of the camera, sets camera parameters, sends shooting instructions, and reads the shooting results of the camera;

A data storage module, which organizes and stores the robot arm data and camera data obtained by the robot arm control module and the camera control module, wherein the arrangement refers to the sequence of the robot arm attitude information in the order of sampling points , the camera information is named and aggregated according to the timestamp.

Optionally, the calibration calculation module includes:

a data reading module, which reads the collected raw data for calibration from the storage device, the raw data refers to the picture sampled by the camera and the posture of the robotic arm in each sampling point;

PNP module, which calculates the pose between the calibration board and the camera according to the data read by the data reading module;

Hand-eye calibration calculation module, this module uses a variety of optional algorithms to calculate hand-eye calibration according to the pose obtained by the PNP module, and calculates the attitude relationship between the camera and the robotic arm and between the camera and the calibration board.

According to a third aspect of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the program, the processor can be used to execute the unified two A method for hand-eye calibration of a robotic arm.

Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

The above-mentioned method, system and terminal for unifying the hand-eye calibration of two manipulators of the present invention use the same framework to unify the two calibration methods of eye-on-hand and eye-out-of-hand, and different systems can be calibrated by the present invention.

The above-mentioned method, system, and terminal for unifying the hand-eye calibration of two manipulators of the present invention only need to manually predetermine the boundary of the posture when selecting the calibration posture, and do not need to manually define each calibration point, which greatly reduces labor costs.

The above-mentioned method, system, and terminal for unifying the hand-eye calibration of two manipulators of the present invention decouple the calibration acquisition and calculation parts through modular design, so that the hardware and the solution algorithm can be updated more conveniently during the use process. , which can be easily implemented in the system.

Description of drawings

Other features, objects and advantages of the present invention will become more apparent by reading the detailed description of non-limiting embodiments with reference to the following drawings:

1 is a flow chart of a method for unifying the hand-eye calibration of two robotic arms according to an embodiment of the present invention;

FIG. 2 is a block diagram of a system module for unifying the hand-eye calibration of two robotic arms according to an embodiment of the present invention;

3 is a flow chart of a method for unifying the hand-eye calibration of two robotic arms according to a preferred embodiment of the present invention;

4 is a schematic diagram of the positional relationship of the calibration plate fixed on the tip of the robot arm according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of the positional relationship between the calibration plate and the base of the robot arm according to an embodiment of the present invention.

Detailed ways

The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

FIG. 1 is a flow chart of a method for unifying the hand-eye calibration of two robotic arms according to an embodiment of the present invention; as shown in FIG. 1 , the method in this embodiment can be implemented according to the following steps:

S100, plan the path of the robotic arm moving in space according to the predetermined attitude boundary point;

S200, control the robotic arm to pass through the set sampling points according to the planned path, and control the camera to take pictures at each sampling point in the corresponding installation mode;

S300, respectively calculate the pose relationship between the camera and the calibration board in each sampling point, including: calculating the pose relationship between the fixed camera and the corresponding calibration board in each sampling point, and the position and orientation relationship installed on the tip of the robotic arm The pose relationship between the camera and the corresponding calibration board, and the pose of the calibration board in the corresponding camera coordinate system is obtained;

S400, according to the pose of the manipulator in the multiple sets of sampling points and the pose of the calibration board in the camera coordinate system, the pose relationship between the camera and the manipulator in the corresponding installation mode is solved.

For the above cameras, "eyes on hands" are cameras installed at the tip of the robotic arm, and "eyes outside hands" are fixed cameras.

The method for unifying the two types of hand-eye calibration of the robotic arm in the above-mentioned embodiment of the present invention can be applied to both the eye-on-hand and the eye-outside calibration methods. When selecting the calibration posture, only the boundary of the manual predetermined posture is required, and no Manually define each calibration point, which greatly reduces labor costs.

Of course, in other preselected embodiments, after the pose relationship is obtained in the above S400, the result may be further calibrated to perform self-checking.

As a preference of the above embodiment, planning the path of the robotic arm moving in space according to the predetermined attitude boundary points, including: uniformly sampling a specified number of points in the predetermined boundary through interpolation sampling; according to the distance in the angular space Compute, generate the shortest path through these points in turn. The interpolation algorithm here includes, but is not limited to, sampling at equal intervals in the pose space.

As a preference of the above embodiment, calculating the pose relationship between the camera and the calibration board in each sampling point respectively includes: identifying the position of the feature in the image captured by the camera based on the feature on the calibration board; position to solve the pose relationship between the camera and the calibration board.

FIG. 4 is a schematic diagram of the positional relationship of the calibration plate fixed on the tip of the robot arm according to an embodiment of the present invention; FIG. 5 is a schematic diagram of the positional relationship between the calibration plate and the robot arm base according to an embodiment of the present invention. Referring to FIG. 4 and FIG. 5 , in the above embodiment, the pose between the corresponding camera and the robotic arm is obtained according to the pose of the robotic arm in multiple sets of sampling points and the pose of the calibration board in the camera coordinate system. In the specific operation, for each sampling point, each sampling point contains a set of data, which is the data obtained by the robotic arm in a certain sampling attitude. These data are: First, obtain the corresponding tip of the robotic arm and the base of the robotic arm Then the pose relationship between the camera and the calibration board is obtained, and the matrix B and matrix A are obtained respectively, where:

For the case where the eye is outside the hand: the camera is fixed relative to the base of the manipulator, the calibration board is fixed at the tip of the manipulator, the matrix A is defined as the external parameter matrix from the calibration board to the camera, and the matrix x is the external parameter from the camera to the base of the manipulator matrix, matrix y is the external parameter matrix from the calibration board to the tip of the manipulator, matrix B is the external parameter matrix from the tip of the manipulator to the base of the manipulator;

In all the collected data, x and y will not change and are constant unknown quantities. Here x is the pose relationship between the camera and the robot arm base to be solved; for each pose, Ax and yB represent From the calibration board to the external parameters of the manipulator base, x and y are solved by using the relationship of multiple groups of Ax=yB;

For the case where the eye is on the hand: the camera is fixed on the tip of the manipulator, and the calibration board and the base of the manipulator are relatively fixed; define matrix A as the external parameter matrix from the calibration board to the camera, and matrix x as the external parameter matrix from the camera to the tip of the manipulator , the matrix y is the external parameter matrix from the calibration board to the base of the manipulator, the matrix B is the external parameter matrix from the base of the manipulator to the tip of the manipulator, and the others are the same as the case where the eye is outside the hand.

2 is a block diagram of a system module for unifying the hand-eye calibration of two robotic arms according to an embodiment of the present invention; with reference to FIG. 3 , the system for unifying the hand-eye calibration of two robotic arms in this embodiment includes:

A path planning module, which plans the path of the robotic arm moving in space according to the predetermined attitude boundary points;

Data acquisition module, this module controls the path planned by the robotic arm according to the path planning module, passes through the set sampling points, and controls the camera to take pictures at each sampling point; here, the data acquisition module will be used when the robotic arm passes through each sampling point. Obtain the data from the camera and the real-time attitude information of the robotic arm respectively;

The calibration calculation module, which calculates the pose relationship between the camera and the calibration board in each sampling point according to the data of the data acquisition module, including: calculating the position between the fixed camera and the corresponding calibration board in each sampling point pose relationship, and the pose relationship between the camera installed on the tip of the robotic arm and the corresponding calibration board, and obtain the pose of the calibration board in the camera coordinate system;

The pose relationship calculation module, which solves the pose relationship between the camera and the robotic arm according to the pose of the robotic arm in the multiple sets of sampling points of the calibration calculation module and the pose of the calibration board in the camera coordinate system.

The system for unifying the hand-eye calibration of two robotic arms in the above-mentioned embodiments of the present invention can be applied to two calibration methods of eye-on-hand and eye-out-of-hand at the same time. There are two calibration methods outside the hand, and different systems can be calibrated. When selecting the calibration posture, only the boundary of the manual predetermined posture is required, and each calibration point does not need to be manually defined, which greatly reduces the labor cost.

On the basis of the above embodiment, as a preferred way, the data acquisition module includes: a robotic arm control module, a camera control module and a data storage module, wherein the robotic arm control module interacts with the robotic arm, sends action instructions to the robotic arm and reads Get the operating status of the manipulator; the camera control module interacts with the camera, controls the work of the camera, sets the camera parameters, sends shooting instructions, and reads the camera shooting results; the data storage module obtains the data from the manipulator control module and the camera control module. The robot arm data and camera data are organized and stored. Such module division decouples various parts of the system, and it is easy to update and upgrade any module, especially when the robot arm or camera needs to be replaced, it can be quickly applied to the system only by modifying the corresponding module.

On the basis of the above embodiment, as a preferred way, the calibration calculation module includes a data reading module, a PNP module and a hand-eye calibration calculation module, wherein: the data reading module reads the collected raw data for calibration from the storage device ; The PNP module calculates the pose between the calibration board and the camera according to the data read by the data reading module; the hand-eye calibration calculation module uses a variety of optional algorithms to calculate the hand-eye calibration according to the pose obtained by the PNP module. Obtain the pose relationship between the camera and the robotic arm and between the camera and the calibration board. Similarly, the calculation part is further decoupled, and a more appropriate solution method can be selected for different scenarios to achieve better results. At the same time, when there is an updated algorithm with better effect, it can be easily applied to this system.

In this embodiment, PNP reads the data according to the data reading module, wherein the PNP algorithm includes but is not limited to EPNP, Iterative and other algorithms for solving PNP problems, according to the calibration feature points in the camera coordinate system and the calibration features in the picture The correspondence between the points calculates the pose between the calibration board and the camera. Here, the calibration feature points depend on the calibration board, including but not limited to the corners of the checkerboard calibration board or the center of the dots on the dot calibration board.

In this embodiment, the hand-eye calibration calculation module uses a variety of optional algorithms for calculating the AxyB problem according to the pose obtained by the PNP module, including but not limited to the methods already preset in the module. Any algorithm for solving the AxyB problem is be usable.

Of course, in other preferred embodiments, the above-mentioned system for unifying the hand-eye calibration of the two robotic arms may also include a self-checking module for performing self-checking on the obtained calibration results.

FIG. 3 is a flowchart of a method for unifying the hand-eye calibration of two manipulators according to a preferred embodiment of the present invention. As shown in FIG. 3 , a specific application example is provided in this specific embodiment, which is to perform eye-on-hand and eye-out-of-hand calibration for a system composed of a six-axis robotic arm and two cameras. Specifically, this embodiment provides a method for hand-eye calibration of a six-axis robotic arm. The method realizes the hand-eye calibration of the eye-on-hand and the eye-off-hand of the robotic arm through the steps of path planning, data acquisition, and calibration calculation. . As shown in Figure 3, the method for unifying the hand-eye calibration of the two robotic arms includes the following steps:

Step 1: Install the equipment to be used for hand-eye calibration (manipulator, camera, calibration plate, device for fixing, etc.);

Step 2: Set the attitude boundary points required for data collection. Here, 8 points in the three-dimensional space are set to form a three-dimensional space area approximate to the platform; of course, other methods can also be used to set the three-dimensional space area in other embodiments. ;

Step 3: The path planning module automatically selects an optional number of sampling points to form a motion path according to the predetermined attitude boundary points, and uniformly samples 100 points in the boundary space through the interpolation algorithm, and calculates and generates a traversing 100 points. The shortest path; of course, other numbers of sampling points can also be selected in other embodiments;

Step 4: Control the robotic arm to pass through each sampling point one by one, and control the camera to take pictures at each sampling point and store them;

Step 5: After the data collection is completed, the PNP module is used to calculate the pose relationship between the fixed camera in each sampling point, the camera installed on the tip of the robotic arm, and the corresponding calibration board;

Step 6: Use the hand-eye calibration calculation module to solve the relationship between the camera and the robotic arm according to the pose of the robotic arm in multiple sets of sampling points (read from the robotic arm) and the pose of the calibration board in the camera coordinate system (calculated in step 5). pose relationship;

Step 7: Use the self-check module to do a preliminary self-check on the calibration results after the calculation is over.

In the above embodiment, step 3 may include the following steps:

Step 3.1: uniformly sample the specified number of points in the boundary reserved in step 2 by the interpolation algorithm;

Step 3.2: Calculate and generate the shortest path passing through these points in sequence according to the distance in the angle space;

In the above embodiment, step 5 may include the following steps:

Step 5.1: Identify the position of the feature in the image captured by the camera based on the feature on the calibration board (including but not limited to the checkerboard calibration board, the dot calibration board, AprilTags, etc.);

Step 5.2: Solve the pose relationship between the camera and the calibration board according to one or more optional PNPs (including but not limited to EPNP, DLS, IPPE and other related algorithms) solving algorithm;

In the above embodiment, step 6 may include the following steps:

Step 6.1: For each group of sampling points, first read the pose relationship between the tip of the corresponding robotic arm and the base of the robotic arm;

Step 6.2: Then read the pose relationship between the camera and the calibration board;

Step 6.3: For the case where the eyes are outside the hand: the camera is fixed relative to the base of the robotic arm, and the calibration plate is fixed at the tip of the robotic arm (including but not limited to using a gripping or fixing mechanism) as shown in Figure 4, define matrix A is the extrinsic parameter matrix from the calibration board to the camera, the matrix x is the extrinsic parameter matrix from the camera to the base of the manipulator, the matrix y is the extrinsic parameter matrix from the calibration board to the tip of the manipulator, and the matrix B is the matrix from the tip of the manipulator to the base of the manipulator. extrinsic matrix. Among them, the A and B matrices are obtained from step 6.2 and step 6.1, respectively, and are here as known quantities in the formula; x and y will not change in all the collected data, and are constant unknown quantities, where x It is the pose relationship between the camera and the robotic arm base that we want to solve. For each pose, Ax and yB represent the external parameters from the calibration board to the base of the manipulator. Using multiple sets of Ax=yB relationships, x and y can be solved (including but not limited to using Kronecker product, quaternion, etc. algorithm). For the case where the eye is on the hand, it is similar: the camera is fixed on the tip of the robotic arm, and the calibration board and the base of the robotic arm are relatively fixed; as shown in Figure 5, the matrix A is defined as the external parameter matrix from the calibration board to the camera, and the matrix x is the camera to the camera. The external parameter matrix of the tip of the manipulator, the matrix y is the external parameter matrix from the calibration board to the base of the manipulator, and the matrix B is the external parameter matrix from the base of the manipulator to the tip of the manipulator. Among them, the A and B matrices are obtained from step 6.2 and step 6.1, respectively, and are here as known quantities in the formula; x and y will not change in all the collected data, and are constant unknown quantities, where x It is the pose relationship between the camera and the robotic arm base that we want to solve. For each pose, Ax and yB represent the external parameters from the calibration board to the base of the manipulator. Using multiple sets of Ax=yB relationships, x and y can be solved (including but not limited to using Kronecker product, quaternion, etc. algorithm).

In the above embodiment, step 7 may include the following steps:

Step 7.1: Whether the eye is on the hand or the eye is outside the hand, for each collected data point, we can calculate y' based on x by calculating AxB ^-1 ;

Step 7.2: Perform error analysis between the y' calculated in step 7.1 and the y finally calculated in step 6 (including but not limited to statistics after conversion into Euler angles or quaternions). Here, the purpose of error analysis is to measure the difference between the pose represented by y and the pose represented by y', and the measurement method includes but is not limited to calculating the second-order general between y and y'.

The above embodiments of the present invention use the same frame to unify the two calibration methods of eye on hand and eye on hand, solve the problem that the prior art is only applicable to specific scenarios and specific installation methods are not general enough, and can adapt to all hand-eye calibration methods. Happening. The present invention has good flexibility and good ductility when the equipment or algorithm is updated through the modular decoupling design of data acquisition and calculation, and can flexibly replace different components in the calibration process to cope with different scenarios.

In another embodiment of the present invention, a terminal is also provided, including a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor can be used to execute any of the above implementations when the processor executes the program In the example, the method of unifying the hand-eye calibration of the two manipulators.

It should be noted that the steps in the method provided by the present invention can be implemented by using the corresponding modules, devices, units, etc. in the system, and those skilled in the art can refer to the technical solutions of the system to implement the method. The step flow, that is, the embodiment in the system can be understood as a preferred example for implementing the method, and details are not described here.

Those skilled in the art know that, in addition to implementing the system provided by the present invention and its respective devices in the form of pure computer-readable program codes, the system and its respective devices provided by the present invention can be completely implemented by logically programming the method steps. Switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers are used to achieve the same function. Therefore, the system and its various devices provided by the present invention can be regarded as a kind of hardware components, and the devices for realizing various functions included in the system can also be regarded as structures in the hardware components; The means for implementing various functions can be regarded as either a software module implementing a method or a structure within a hardware component.

Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and those skilled in the art can make various variations or modifications within the scope of the claims, which do not affect the essential content of the present invention. The above-mentioned preferred features can be used in any combination as long as they do not conflict with each other.

Claims (5)

1. a method for unifying two kinds of mechanical arm hand-eye calibration, is characterized in that, comprises: According to the predetermined attitude boundary points, plan the path of the robot arm moving in space; Control the robotic arm to pass through the set sampling points according to the planned path, and control the camera to take pictures at each sampling point; Calculate the pose relationship between the camera and the calibration board in each sampling point, including: calculating the pose relationship between the fixed camera and the corresponding calibration board in each sampling point, and the camera installed on the tip of the robotic arm and the calibration board. Corresponds to the pose relationship between the calibration boards, and obtains the pose of the calibration board in the camera coordinate system; where: The pose relationship between the camera and the calibration board in each sampling point is calculated separately, including: Identify the position of the feature in the image captured by the camera based on the feature on the calibration board; According to the position of the feature, solve the pose relationship between the camera and the calibration board; According to the pose of the manipulator in the multiple sets of sampling points and the pose of the calibration board in the camera coordinate system, the pose relationship between the camera and the manipulator under the corresponding camera installation mode is solved, including: For each sampling point, each sampling point here contains a set of data, which is the data obtained by the robotic arm in a certain sampling attitude. These data are: first, obtain the pose relationship between the corresponding robotic arm tip and the robotic arm base , and then obtain the pose relationship between the camera and the calibration board, and obtain matrix B and matrix A, respectively, where: For the case where the eye is outside the hand: the camera is fixed relative to the base of the manipulator, the calibration board is fixed at the tip of the manipulator, the matrix A is defined as the external parameter matrix from the calibration board to the camera, and the matrix x is the external parameter from the camera to the base of the manipulator matrix, matrix y is the external parameter matrix from the calibration board to the tip of the manipulator, matrix B is the external parameter matrix from the tip of the manipulator to the base of the manipulator; In all the collected data, x and y will not change and are constant unknown quantities. Here x is the pose relationship between the camera and the robot arm base to be solved; for each pose, Ax and yB represent From the calibration board to the external parameters of the manipulator base, x and y are solved by using the relationship of multiple groups of Ax=yB; For the case where the eye is on the hand: the camera is fixed on the tip of the manipulator, and the calibration board and the base of the manipulator are relatively fixed; define matrix A as the external parameter matrix from the calibration board to the camera, and matrix x as the external parameter matrix from the camera to the tip of the manipulator , the matrix y is the external parameter matrix from the calibration board to the base of the manipulator, the matrix B is the external parameter matrix from the base of the manipulator to the tip of the manipulator, and the others are the same as the case where the eye is outside the hand; According to the pose of the manipulator in the multiple sets of sampling points and the pose of the calibration board in the camera coordinate system, the pose relationship between the camera and the manipulator in the corresponding situation is solved. Here, when the camera is "eye in hand" When the installation method is "outside", the pose relationship between the camera and the base of the robot arm needs to be solved; when the camera is installed with the "eye on the hand", the pose relationship between the camera and the tip of the robot arm needs to be solved. relation; Self-check of calibration results; where: Whether the eye is on the hand or the eye is outside the hand, for each sampling point, y' calculated based on x is obtained by calculating AxB ^-1 ; A is the external parameter matrix from the calibration board to the camera, and x is the base from the camera to the robotic arm The external parameter matrix of the seat, B is the external parameter matrix from the base of the manipulator to the tip of the manipulator; Perform error analysis between the calculated y' of multiple groups and the final calculated external parameter matrix y from the calibration plate to the tip of the manipulator. 2. The method for unifying the hand-eye calibration of two kinds of mechanical arms according to claim 1, wherein, according to the predetermined attitude boundary point, planning the path of the robotic arm moving in space, comprising: A specified number of points are uniformly sampled in a predetermined boundary by an interpolation algorithm; Calculated by distance in angular space to generate the shortest path through these points in sequence. 3. A system for unifying two kinds of manipulator hand-eye calibration for implementing the method for unifying two kinds of manipulator hand-eye calibration methods described in any one of claims 1-2, is characterized in that, comprising: A path planning module, which plans the path of the robotic arm moving in space according to the predetermined attitude boundary points; a data acquisition module, which controls the robotic arm to pass the set sampling point according to the path planned by the path planning module, and simultaneously controls the camera in the corresponding installation mode to take pictures at each sampling point; A calibration calculation module, which calculates the pose relationship between the camera and the calibration board in each sampling point according to the data of the data acquisition module, including: calculating the distance between the fixed camera and the corresponding calibration board in each sampling point and the pose relationship between the camera installed on the tip of the robotic arm and the corresponding calibration board, and obtain the pose of the calibration board in the camera coordinate system in the corresponding installation mode; The pose relationship calculation module, which solves the relationship between the camera and the robotic arm in the corresponding installation mode according to the pose of the robotic arm in the multiple sets of sampling points of the calibration calculation module and the pose of the calibration board in the camera coordinate system. pose relationship between Self-checking module for self-checking the obtained calibration results; The calibration calculation module includes: a data reading module, which reads the collected raw data for calibration from the storage device, the raw data refers to the picture sampled by the camera and the posture of the robotic arm in each sampling point; PNP module, this module calculates the pose between the calibration board and the camera through the PNP algorithm according to the data read by the data reading module and according to the correspondence between the calibration feature points in the camera coordinate system and the calibration feature points in the picture; The hand-eye calibration calculation module, which calculates the hand-eye calibration according to the pose obtained by the PNP module, and calculates the attitude relationship between the camera and the robotic arm and between the camera and the calibration board. 4. The system for unifying two kinds of mechanical arm hand-eye calibration according to claim 3, is characterized in that, described data acquisition module, comprises: The robotic arm control module, which interacts with the robotic arm, sends action commands to the robotic arm and reads the operating status of the robotic arm; The camera control module, which interacts with the camera, controls the work of the camera, sets camera parameters, sends shooting instructions, and reads the results of camera shooting; A data storage module, which organizes and stores the robot arm data and camera data obtained by the robot arm control module and the camera control module, wherein the arrangement refers to the sequence of the robot arm attitude information in the order of sampling points , the camera information is named and aggregated according to the timestamp. 5. A terminal comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor can be used to execute any one of claims 1-2 when the processor executes the program the method described.

Images