CN115049744A - Robot hand-eye coordinate conversion method and device, computer equipment and storage medium - Google Patents

Abstract

The present application relates to a robot hand-eye coordinate conversion method, device, computer equipment, storage medium and computer program product. The method includes: collecting images of the end effector of the robot on which the calibration plate is installed by using a scanner of the robot, so as to obtain the calibration plate images of the end effector in different poses; , perform detection according to at least two kinds of graphics, and obtain at least two groups of different mark point areas; based on the distance between the mark point areas in each group, determine whether the regional center of the mark point area in each group is valid; The regional center and each valid regional center are at the center marker points of different poses to generate marker point sequences of different poses; through the marker point sequences of different poses, the coordinate system transformation relationship between the robot and the scanner is calibrated. By adopting the method, the coordinate conversion accuracy of the scanner and the robot can be improved.

Description

Robot hand-eye coordinate conversion method, device, computer equipment and storage medium

technical field

The present application relates to the field of robotics, and in particular, to a method, device, computer equipment, storage medium and computer program product for converting robot hand-eye coordinates.

Background technique

With the development of artificial intelligence technology, robots have been widely used in many industries. Among them, in the field of industrial application, the robot has a visual perception system. Using the three-dimensional information obtained by the visual perception system, the robot can control the end effector to perform mechanical processing and installation. In simple terms, the 3D perception system is equivalent to the human eye, and the end effector is equivalent to the human hand, which completes the preset action tasks through the cooperation between the hand and the eye.

In order to ensure that the robot can accurately move the space object to the target position, it is necessary to determine the conversion relationship between the coordinate system of the vision system and the coordinate system of the manipulator. The traditional conversion relationship determination method has poor accuracy, and the obtained result will be different from the actual value. larger deviation.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide a robot hand-eye coordinate conversion method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy in response to the above technical problems.

In a first aspect, the present application provides a robot hand-eye coordinate conversion method. The method includes:

Using the scanner of the robot, image acquisition is performed on the end effector of the robot on which the calibration plate is installed, so as to obtain the calibration plate images of the end effector in different poses;

Detecting each marker point in the calibration plate images of different poses according to at least two kinds of graphics to obtain at least two groups of different marker point regions;

Based on the distance between the marker point areas in each group, determine whether the area center of the marker point area in each group is valid;

According to each valid said area center and each valid said area center at the center marker point of said different poses, generate a sequence of marker points of different poses;

The coordinate system conversion relationship between the end effector and the scanner is calibrated through the sequence of marker points of different poses.

In one of the embodiments, the respective marker points in the calibration board images of different poses are detected according to at least two kinds of graphics, and at least two groups of different marker point regions are obtained, including:

For each marker point in the calibration plate image, edge extraction and detection are performed according to at least two different graphics, to obtain at least two different image contours;

Screen the image contours of the contour length intervals in each group;

Based on the screened image contours in each group, the at least two groups of different marker point regions are obtained.

In one embodiment, the image contours include circular contours and polygonal contours, and the at least two groups of different marker point regions are obtained based on the screened image contours in each group, including:

Calculate the similarity based on the contour area and contour length of the screened circular contour, and obtain the similarity of the circular contour;

Based on the similarity of the circular contour, from the found circular contour, select the circular mark point area corresponding to each mark point;

Fitting the polygonal outlines in each group that has been found to obtain a polygonal fitting image;

Using the quadrilateral outline in the polygon fitting image as the quadrilateral mark point area corresponding to each mark point;

The determining whether the area center of the marker point area in each group is valid includes:

Based on the distance between the circular mark point area and the quadrilateral mark point area of each mark point, it is determined whether the area center of the circular mark point area is valid.

In one of the embodiments, determining whether the area centers of the marker point areas in each group are valid based on the distances between the marker point areas in each group includes:

Overlap detection is performed on the marker point regions in each group respectively, and the overlapping marker point regions in each group are eliminated to obtain the marker point regions after the elimination of each group;

Compare the distance between the marked point regions after each group is eliminated with the adjacent threshold distance of the marked point, and obtain the comparison results of multiple adjacent points;

Based on the comparison results of each of the adjacent points, it is respectively determined whether the area centers in the marked point areas of the respective groups are valid or not.

In one embodiment, the overlapping detection is performed on the marker point regions in each group respectively, and the overlapping marker point regions in each group are eliminated to obtain the marker point regions after the elimination of each group, including:

Make a combined comparison of the marked point areas in each group;

Calculate the overlap detection distance between the marker point regions of the combined comparison;

When the overlapping detection distance satisfies the contour detection threshold, calculate the area side length and area area of each marker point area in the combined comparison;

Based on the area side lengths and area areas of the respective marker point regions in the combined comparison, the overlapping marker point regions in each group are eliminated to obtain the eliminated marker point regions in each group.

In one of the embodiments, according to each valid area center and each valid area center at the center marker points of the different poses, generating a sequence of marker points of different poses, including:

Perform mean calculation based on the center of the effective area with the same pose to obtain the center of gravity of the center of the region with different poses;

Based on the distance between the center of gravity of the same pose and the center of the effective area, find out the center marker point of each pose from the center of each area;

Determining the center mark points of each of the regional centers in different poses as the polar origin of each of the poses;

Based on the center of each region and the polar coordinate origin under each of the poses, the position of the center of each of the regions in the polar coordinate system of each of the poses is obtained;

According to the angles of the centers of the regions in the polar coordinate system, the positions of the centers of the regions in the polar coordinate system of the poses are sorted respectively, so as to obtain marker point sequences of different poses.

In one of the embodiments, the coordinate system transformation relationship includes a rotation transformation relationship and a translation transformation relationship, and the coordinates between the robot and the scanner are calibrated through the marker point sequences of different poses Department conversion relationship, including:

Calibrate the transformation relationship between the rotation of the robot and the scanner through the rotation vectors corresponding to the sequence of landmarks in different pose transformation processes;

Obtain the first translation vector through the origin translation information of the sequence of landmarks in different pose conversion processes;

The second translation vector is obtained by calculating the marker point sphere center obtained by fitting the marker point sequence of the different poses, and the marker point sequence of the preset pose;

The first translation vector and the second translation vector are combined to obtain the translation conversion relationship.

In a second aspect, the present application also provides a robot hand-eye coordinate conversion device. The device includes:

The image acquisition module is used to collect images of the end effector of the robot on which the calibration plate is installed through the scanner of the robot, so as to obtain the calibration plate images of the end effector in different poses;

an edge detection module, configured to detect each marker point in the calibration plate images of different poses according to at least two kinds of graphics to obtain at least two groups of different marker point regions;

an effective marker point determination module, for determining whether the regional center of the marker point area in each group is valid based on the distance between the marker point areas in each group;

a marker point sequence generation module, configured to generate marker point sequences of different poses according to each valid said area center and the center marker points of each valid said area center in said different poses;

The hand-eye calibration module is used for calibrating the coordinate system conversion relationship between the robot and the scanner through the marker point sequences of different poses.

In a third aspect, the present application also provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the processor implements the steps of transforming the robot hand-eye coordinates in any of the foregoing embodiments.

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, and when the computer program is executed by the processor, implements the steps of transforming the robot hand-eye coordinates in any of the foregoing embodiments.

In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program, which, when executed by the processor, implements the steps of transforming the robot's hand-eye coordinates in any of the foregoing embodiments.

The above-mentioned robot hand-eye coordinate conversion method, device, computer equipment, storage medium and computer program product are used to detect each marker point in the calibration plate images of different poses according to at least two kinds of graphics, and detect through the geometric features of the image. based on the distance between the mark point areas in each group, determine whether the mark points in the calibration plate image are valid, and then according to each valid area center and each valid said area center Describe the center marker points of different poses, and generate marker point sequences of different poses, so as to determine the position of each marker point in the calibration board image through the marker point sequence, so as to convert the robot hand-eye calibration calculation into three-dimensional space between point pairs The translation and rotation relationship calculation can be based on the PnP algorithm to calculate the corresponding translation and rotation relationship, without the need to calculate based on the change relationship between the poses, the calculation error is easy to control, and it is convenient for application developers to debug.

Description of drawings

Fig. 1 is the application environment diagram of the robot hand-eye coordinate conversion method in one embodiment;

2 is a schematic flowchart of a robot hand-eye coordinate conversion method in one embodiment;

Fig. 3 is the structural representation of the sign board in one embodiment;

4 is a schematic structural diagram of a marker point in another embodiment;

5 is a structural block diagram of a robot hand-eye coordinate conversion device in one embodiment;

FIG. 6 is a diagram of the internal structure of a computer device in one embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

The robot hand-eye coordinate conversion method provided in the embodiment of the present application can be applied to the application environment shown in FIG. 1 . The terminal 102 communicates with the server 104 through the network. The data storage system may store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network server. The terminal 102 collects images of the end effector of the robot on which the calibration plate is installed through the scanner of the robot, and obtains the calibration plate images of the end effector in different poses; Each of the marker points in , is detected according to at least two kinds of graphics, and at least two groups of different marker point areas are obtained; based on the distance between the marker point areas in each group, the regional center of the marker point area in each group is determined. Whether it is valid; according to each of the area centers and the center marker points of each of the area centers in the different poses, generate a sequence of marker points of different poses; through the sequence of marker points of the different poses, demarcate the end The coordinate system conversion relationship between the actuator and the scanner.

Wherein, the terminal 102 can be, but is not limited to, various robots, personal computers, notebook computers, smart phones, tablet computers, IoT devices and portable wearable devices, and the IoT devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle equipment, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, or the like. The server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

In one embodiment, as shown in FIG. 2 , a method for converting hand-eye coordinates of a robot is provided, and the method is applied to the terminal 102 in FIG. 1 as an example for description, including the following steps:

Step 202 , by using the scanner of the robot, image acquisition is performed on the end effector of the robot on which the calibration plate is installed, and images of the calibration plate in different poses of the end effector are obtained.

The robot includes the robot body, the scanner and the end effector of the robot; the coordinate system displacement relationship between the robot body and the scanner is fixed, and the scanner includes a depth map and a grayscale camera to determine the marker points and their pixel coordinates; and the robot body At least one axis of the robot is installed with the end effector of the robot, the manipulator controlled by the end effector is installed with a sign board for hand-eye calibration, and the sign board is affixed with sign points. The marking board is a target made of lightweight aluminum alloy. There are coded marking points on the target. White paper should be pasted on the target first, and then marking points should be pasted on it to improve the identification accuracy of long-distance marking points. point.

In one embodiment, the calibration board is shown in FIG. 3 , the edge position of the marking board is annularly arranged with the position marking points of the annular belt, and a corresponding nearby marking point is set near the position marking point of a certain annular belt as a marking position, and the marking board The center position is provided with a center position marker point. In another embodiment, there are at least 9 ring-belt position markers, and a corresponding nearby marker point is set near a ring-belt position marker point as a marker position, and each adjacent ring-belt position marker point and the center position The angular interval of the line connecting the marker points is at least 15 degrees, which serves as a candidate marker point in the calibration plate image and is the marker point for image detection. The angular interval is less than 5 degrees, and the marker bit is not a marker point for image detection. A marker point is a graph composed of multiple features, as shown in Figure 4.

In one embodiment, the working range of a certain robotic arm corresponding to the end effector of the robot is determined, and within this working range, the end effector carrying the sign plate is controlled to cover the working range as much as possible to perform multiple movements, and The image acquisition of the multiple movements of the end effector is performed by the scanner to reduce the error of the subsequent calculation of the pose estimation. Keep the angle between the tool coordinate system and the robot body coordinate system unchanged.

In one embodiment, the image acquisition of the multiple movements of the end effector by the scanner includes: in the process of calculating the coordinate transformation relationship of translation, keeping the origin of the tool coordinate system different during each movement, and The coordinate system of the robot tool is rotated, and after the movement is completed, the left and right two sets of images are collected through the left and right two images of the binocular camera of the scanner, and then the left and right two sets of calibration plate image sequences with the end effector in different poses are obtained; when When the left and right two sets of calibration plate image sequences and the internal and external parameters in the binocular camera are calculated according to the trigonometry, the coordinate positions of each marker point under different poses are obtained.

It should be understood that the image acquisition of the multiple movements of the end effector by the scanner involves two processes, one of which is to calculate the coordinate transformation relationship of rotation, and the other process is to calculate the coordinate transformation relationship of translation. The two processes are complementary, and the two processes are not limited to the above.

After image acquisition by the scanner, images of the calibration plate with different poses are obtained. In the calibration board images of various poses, there are also differences in the shape, coordinate position and other landmark features of the same marker point. After detecting an image based on these differences, the image detection results under different poses are also different.

In step 204, each marker point in the calibration plate images of different poses is detected according to at least two kinds of graphics, and at least two groups of different marker point regions are obtained.

Each marker point in the calibration plate image has its own pixel coordinate position area, and there are corresponding marker point features in the pixel coordinate position area of each marker point. Multiple marker point regions corresponding to each marker point in the calibration plate image. The multiple marker point areas corresponding to each marker point will be divided into corresponding different groups of marker point areas according to the different graphics to be detected. There are differences in image attributes such as the position, side length, and area of point regions, and it can be judged by at least one image attribute whether each marker point region in the same group should be eliminated.

After some of the marker point regions in the same group are eliminated, the corresponding relationship between the marker point regions of different groups is determined based on the positions of the marker point regions, and each marker point corresponding to the marker point regions of different groups is determined based on the correspondence relationship. area to determine whether the area center of the marker point area in each group is valid.

Step 206 , based on the distances between the marker point regions in each group, determine whether the region centers of the marker point regions in each group are valid.

For the distance between the marker point areas in each group, it may be the distance between the marker point areas in the same group and the distance between the marker point areas in different groups. When the distance between the marker point areas in the same group is less than the threshold for non-maximum value suppression, the terminal determines that this part of the marker point area in the same group is a non-maximum value calculation in the adjacent point area and the repeated point area Area, perform non-maximum calculation on the non-maximum calculation area, and remove too adjacent points or duplicate points.

For the distance between the marker point areas in different groups, the terminal calculates the distance between the corresponding marker point areas in different groups, compares the distance with the corresponding threshold, and determines the marker point in the corresponding group of a certain graph Whether the zone's zone center is valid. For example, the marker area in the ellipse group has an elliptical outline, and the marker area in the quadrilateral group has a quadrilateral outline. When the distance between the ellipse group and the corresponding marker area in the quadrilateral group is less than the corresponding threshold, the ellipse group The area centers of each mark point area in the group are valid respectively, while the area centers of the corresponding mark point areas in the quadrilateral group are invalid.

After obtaining the center of each area, determine whether the area center in the sign board image of each pose matches the number of marker points attached to the calibration board. If so, determine the center of the corresponding pose based on the area center of each pose Mark points; if they do not match, abnormal processing of the number of coding points will be performed until the center of the area in the image of the marking board of each pose matches the number of marking points pasted on the calibration board.

Among them, the abnormal processing of the number of regional centers includes: if the number of regional centers of a certain pose is less than the number of mark points attached to the calibration board, adjust the pose and rescan until the abnormality disappears; if the number of regional centers is greater than the calibration board The number of marked points pasted, the point with the largest distance between the area center and the average pixel coordinate is removed in turn, until the abnormality disappears.

Step 208: Generate marker point sequences of different poses according to each valid area center and the center marker points of each valid area center in different poses.

The center marker point is one of the center of each area under a certain pose, and the center marker point is used to represent the center position marker point set on the marker board. Since the poses to which the calibration plate images belong are different, the center of gravity of each area center in the same calibration plate image is not the center marker point, but the center of the area closest to the center of gravity of each area center is the center marker point.

For the calculation process of the center marker points of different poses, it includes: the terminal averages the position data of the center of each region under a certain pose, and obtains the average value of the region center under the pose, and the average value of the regional center is the average value of the region center under a certain pose The position of the center of gravity of the marker point; calculate the distance between the center of gravity of the pose and the center of each area, obtain the calculation result between the center of each area and the center of gravity, and select each area under the pose based on the calculation result One of the centers, using the selected area center as the center marker point; wherein, the calculation result represents the distance between the center of gravity position and the center marker point under the pose, which is smaller than the center of gravity position under the pose and any area center the distance between.

After obtaining the center marker point, the distance between each valid area center is determined based on the center marker point, which is equivalent to the distance between the center marker points, so that the corresponding relationship between each area center and the corresponding marker point is established. The ordering of the marker points in the calibration plate image is determined, constituting the marker point sequence. Each marker point in the marker point sequence can be further coded to facilitate better identification and calling of the marker point.

In one embodiment, according to each valid area center and the center marker points of each valid area center in different poses, generating a sequence of marker points of different poses, including: based on the position of the same center of gravity of the pose and the difference between the center of the valid region Calculate the mean value of the distance between them to obtain the center of gravity position of the regional center of different poses; based on the distance between the center of gravity position of the same pose and the effective regional center, find the center mark point of each pose from the center of each region; The center mark points of the center of each area in different poses are determined as the polar coordinate origin under each pose; based on the center of each region and the polar coordinate origin under each pose, the polar coordinates of each region center in the polar coordinate system of each pose are obtained. Position; according to the angle of each regional center in the polar coordinate system, the positions of each regional center in the polar coordinate system of each pose are sorted, and the sequence of landmark points of different poses is obtained.

After the polar coordinate origin is determined, the center of the area with the smallest adjacent angle is calculated as the location of the starting marker point, the angle corresponding to the starting marker point is set as the starting angle, and the polar coordinate system is constructed based on the polar coordinate origin and the starting marker point. , although the angle of each area center in the polar coordinate system is different, after rotation and inclination, because the number of each marker point and its order in the polar coordinate system are determined, and each marker point is based on the starting marker point and the The polar coordinate origin, according to the angle of each area center in the polar coordinate system, can still know the relative position of each marker point in the marker point sequence, that is, the marker point sequence in the polar coordinate system has the characteristics of anti-rotation and tilt.

In one embodiment, the central marker points of the center of each region in different poses are respectively determined as the polar coordinate origins under each pose, including: averaging the region centers with the same poses respectively to obtain the region centers of each pose Based on the distance interval between the center of gravity of the regional center of the same pose and the regional center, screen each regional center to obtain the center marker point of each pose; determine the center marker point of each pose as each The polar coordinate origin in the pose.

In one embodiment, according to the angle of each area center in the polar coordinate system, the positions of each area center in the polar coordinate system of each pose are sorted respectively, and the sequence of marker points of different poses is obtained, including: calculating the center marker The initial polar coordinate angle between the point and the center of each area, and then sort according to the initial polar coordinate angle to obtain the initial mark point sequence; calculate the angle difference between two adjacent points in the initial mark point sequence, and determine the angle difference The position with the smallest modulo length is the target starting point; according to the position of the target starting point in the initial marker point sequence, determine the target angle between the target starting point and the center of each area in the polar coordinate system, and rearrange the centers of each area based on the target angle , get the target polar coordinate sequence.

In step 210, the coordinate system transformation relationship between the robot and the scanner is calibrated through the sequence of landmark points of different poses.

In one embodiment, the coordinate system transformation relationship between the robot and the scanner is calibrated through the sequence of landmark points of different poses, including:

Perform averaging processing based on the position coordinates of each marker point in the sequence of marker points with the same pose, to obtain the mean value of the three-dimensional reconstructed coordinates of each pose, and calculate the mean value of the three-dimensional reconstructed coordinates of each pose. According to the PNP algorithm, the rotation value and the first a translation value;

Perform sphere fitting on the mean values of the three-dimensional reconstructed coordinates of each pose to obtain a fitting sphere with marker points, and calculate based on the center of the sphere of the marker point fitting sphere and the position mean value corresponding to the sequence of marker points of the preset pose to obtain the second translation value;

The rotational coordinate system conversion relationship between the robot and the scanner is calibrated by the rotation value, and the translation coordinate system conversion relationship between the robot and the scanner is calibrated by the first translation value and the second translation value.

Therefore, after obtaining the sequence of landmark points with different poses, the coordinate system conversion relationship between the calibration robot and the scanner can be accurately calculated based on the PNP correlation algorithm.

In one embodiment, the coordinate system conversion relationship includes a rotational conversion relationship and a translational conversion relationship, and the coordinate system conversion relationship between the robot and the scanner is calibrated through the marker point sequence of different poses, including: using the marker point sequence in the The position information of the movement during the transformation of different poses, the transformation relationship between the rotation of the robot and the scanner is calibrated, and the first translation vector is calculated; The mark point sequence is calculated to obtain a second translation vector; the first translation vector and the second translation vector are combined to calibrate the translation conversion relationship.

In one embodiment, the process of calculating the first translation vector specifically includes: on the premise that the rotation value of the tool coordinate system relative to the robot coordinate system is kept unchanged, changing the position of the end effector multiple times, and recording the position at each change. The coordinate value of the origin of the tool coordinate system and the mean value of the three-dimensional reconstructed coordinates of the target marker point obtained by the scanner, and the PNP algorithm is used to obtain the rotation value and the first translation vector between the robot coordinate system and the scanner coordinate system by using these two sets of coordinate values.

The step of calculating the second translation vector includes: in the process of calculating the first translation vector, recording the mean value of the three-dimensional reconstructed coordinates of the target marker point in the final state, and then maintaining the position of the origin of the tool coordinate system at this time. The coordinate system of the secondary rotation tool changes its posture, and the scanner obtains the mean value of the three-dimensional reconstructed coordinates of the marker points in each posture. The mean value obtained under the above-mentioned multiple postures is distributed on a spherical surface, and the coordinates of the center of the sphere are fitted, and the above recorded marks are combined. The mean value of the three-dimensional reconstruction coordinates of the point is obtained, and the second translation vector is obtained.

Finally, the calculated first translation vector and the second translation vector are combined to obtain the translation conversion relationship. For example: first, after collecting 8 groups of landmark point sequences with different poses, the rotation value conversion relationship and the first translation vector can be obtained by using PNP calculation. Next, at least 7 groups of marker point sequences are fitted to obtain the fitted sphere center, the coordinates of the sphere center and the average value corresponding to the first group of marker point sequences whose rotation number is 0, to obtain the second translation vector. Finally, the inversion of the second translation vector is added to the first translation vector calculated by the above PNP to obtain the translation value conversion relationship between the robot and the three-dimensional scanner.

In the above-mentioned robot hand-eye coordinate conversion method, each marker point in the calibration board images with different poses is detected according to at least two kinds of graphics, and the marker point area is screened out through this image geometric feature detection method; The distance between the marked point regions is determined by determining whether the marking points in the calibration board image are valid, and then according to the center of each valid region and the center marker points of each valid region center in different poses, the sequence of marker points of different poses is generated. , to determine the position of each marker point in the calibration plate image through the sequence of marker points, so as to convert the robot hand-eye calibration calculation into the translation and rotation relationship calculation between point pairs in three-dimensional space, and the corresponding translation and rotation can be calculated based on the PnP algorithm. There is no need to calculate based on the changing relationship between poses and poses, the calculation error is easy to control, and it is convenient for application developers to debug.

In one embodiment, in step 204, each marker point in the calibration plate images of different poses is detected according to at least two kinds of patterns, and at least two groups of different marker point regions are obtained, including: For the marker points, edge extraction and detection are performed according to at least two different graphics respectively, so as to obtain at least two sets of different image contours. Screen the image contours for the contour length intervals in each group. Based on the screened image contours in each group, at least two groups of different marker point regions are obtained.

Specifically, for each marker point in the calibration plate image, the edge canny operator is extracted and detected according to at least two different graphs, to obtain at least two sets of different image contours, and the shorter ones are removed from the at least two different image contours. and the longer useless contours to obtain the selected image contours of each group, and the selected image contours of each group are calculated accordingly to obtain at least two groups of different marker point regions.

In one embodiment, the image contours include circular contours and polygonal contours, and based on the screened image contours in each group, at least two groups of different marker point regions are obtained, including: contour areas and contours based on the screened circular contours Calculate the similarity of the length to obtain the similarity of the circular contour; based on the similarity of the circular contour, from the circular contours found, select the corresponding circular mark point area of each mark point; The polygonal outline is fitted to obtain a polygonal fitting image; the quadrilateral outline in the polygonal fitting image is used as the quadrilateral mark point area corresponding to each mark point.

Wherein, after calculating the similarity of the circular contour, select the circular mark point area whose similarity is less than the similarity threshold, and obtain the circular mark point area corresponding to each mark point; wherein, the circular similarity degree=(4.0*PI *Contour area)/(Contour perimeter*Contour perimeter+1e-7), the similarity threshold can be 0.8.

Correspondingly, step 206, judging whether the area center of the mark point area in each group is valid, including: based on the distance between the circular mark point area of each mark point and the quadrilateral mark point area, determine the circular mark point area. Whether the regional center is valid. Among them, since the marker points are circular, the area center of the circular marker point area may be valid, while the area center of the quadrilateral marker point area is invalid, so as to improve the corresponding accuracy.

In one embodiment, step 206 , based on the distances between the marker point regions in each group, determine whether the region centers of the marker point regions in each group are valid, including: performing overlap detection on the marker point regions in each group respectively. , remove the overlapping marker point areas in each group, and obtain the marker point area after the removal of each group; compare the distance between the marker point areas after the removal of each group with the adjacent threshold distance of the marker points, and obtain the comparison results of multiple adjacent points; Based on the comparison results of each adjacent point, it is judged whether the area center in the marked point area of each group is valid or not.

In the process of performing overlapping detection on the marker point regions in each group, the terminal selects the marker point regions in the same group based on the distances between the marker point regions in each group, and calculates the corresponding corresponding points of the selected marker point regions in the same group. The overlap detection attribute of , based on the corresponding overlap detection attributes of the marker point regions, each marker point region in the same group is detected, and the overlapping marker point regions in each group are eliminated based on the detection results.

In one embodiment, overlapping detection is performed on the marker point regions in each group respectively, and the overlapping marker point regions in each group are eliminated to obtain the marker point regions after the elimination of each group, including: Carry out combined comparison; calculate the overlapping detection distance between the marked point regions obtained by comparison; when the overlapping detection distance meets the contour detection threshold, calculate the regional side length and regional area of each marked point region for combined comparison; The area side length and area area of each mark point area, and the overlapping mark point area in each group is eliminated to obtain the mark point area after each group is eliminated.

The overlapping detection distance is the distance between any marker point regions in the same group of marker point regions, and the distance is the center distance between the marker point regions obtained by arbitrarily combining the same group of marker point regions.

For example: after detecting according to the quadrilateral, obtain a plurality of quadrilateral mark point areas in the quadrilateral group, respectively carry out a combined comparison on any two of the quadrilateral mark point areas, and calculate the two mark point areas obtained by the combined comparison When the overlapping detection distance is smaller than the corresponding contour detection threshold, contour detection is performed. In the process of contour detection, calculate the respective area side length and area area of the two marker point areas that are combined and compared, and calculate the ratio between the product of the area side length and the area area. When the ratio is in the valid range, the combined Compare the areas of the two quadrilaterals in the middle, select the one with the larger area to be eliminated, and keep the smaller one to obtain the mark point area after the quadrilateral group is eliminated.

Therefore, the present invention filters the marker points through the geometric feature relationship of the two-dimensional image, can accurately identify and encode the marker points in the case of the XYZ axis rotation and translation, and combine the sub-pixel processing to improve the reconstruction of the binocular 3D scanner. Precision, the robot hand-eye calibration calculation uses the PnP algorithm to calculate the point pair, the calculation process is simple, the error is easy to control and optimize, and it is convenient for application developers to debug. Compared with the traditional hand-eye calibration method, the traditional method needs to solve the AX=XB equation based on the change relationship of the pose. Since the difference between the upper and lower poses is used for the solution, the correct solution of the equation is highly dependent on the accuracy of the measurement data. When there is a slight problem with the measurement data, the obtained result will have a large deviation from the actual true value. The calculation principle of the present invention is relatively intuitive and can better find problems in actual use. Even if there are robot walking errors, three-dimensional scanner accuracy errors, spherical center fitting errors, and errors in calculating rotation and translation during the measurement process, it is still Can guarantee high accuracy.

It should be understood that, although the steps in the flowcharts involved in the above embodiments are sequentially displayed according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and the steps may be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but may be executed at different times. The order of execution of these steps or stages is also not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the steps or stages in the other steps.

Based on the same inventive concept, an embodiment of the present application further provides a robot hand-eye coordinate conversion device for implementing the above-mentioned robot hand-eye coordinate conversion method. The solution to the problem provided by the device is similar to the solution described in the above method, so the specific limitations in one or more embodiments of the robot hand-eye coordinate conversion device provided below can refer to the above for the robot hand-eye coordinate conversion The limitation of the method is not repeated here.

In one embodiment, as shown in FIG. 5, a robot hand-eye coordinate conversion device is provided, including: an image acquisition module 502, an edge detection module 504, a valid marker point determination module 506, a marker point sequence generation module 508, and a hand-eye calibration module Module 510, wherein:

The image acquisition module 502 is configured to perform image acquisition on the end effector of the robot on which the calibration plate is installed through the scanner of the robot, to obtain the calibration plate images of the end effector in different poses;

The edge detection module 504 is configured to detect each marker point in the calibration plate images of different poses according to at least two kinds of graphics, and obtain at least two groups of different marker point regions;

The valid marker point determination module 506 is used to determine whether the regional center of the marker point area in each group is valid based on the distance between the marker point areas in each group;

A marker point sequence generation module 508, configured to generate marker point sequences of different poses according to each valid said area center and each valid said area center at the center marker point of said different poses;

The hand-eye calibration module 510 is used for calibrating the coordinate system transformation relationship between the robot and the scanner through the marker point sequences of different poses.

In one embodiment, the edge detection module 504 includes:

a contour detection unit, configured to perform edge extraction and detection on each marker point in the calibration plate image according to at least two different graphics, to obtain at least two sets of different image contours;

a length screening unit for screening the image contours of the contour length intervals in each group;

A marker point region generating unit, configured to obtain the at least two different groups of marker point regions based on the screened image contours in each group.

In one embodiment, the marker point area generating unit includes:

The similarity calculation unit is used to calculate the similarity based on the contour area and contour length of the screened circular contour, and obtain the circular contour similarity;

a circular area screening unit, for selecting the circular mark point area corresponding to each mark point from the circular contour that has been found based on the similarity of the circular contour;

a polygon fitting unit, used for fitting the polygonal outlines in the found groups to obtain a polygonal fitting image;

The quadrilateral area screening unit is used to fit the quadrilateral outline in the polygonal image as the quadrilateral mark point area corresponding to each mark point;

The effective marker point determination module 506 includes:

A center validity judgment unit, configured to judge whether the area center of the circular mark point area is valid based on the distance between the circular mark point area and the quadrilateral mark point area of each mark point.

In one embodiment, the valid marker point determination module 506 includes:

The overlapping area detection unit is used to respectively perform overlapping detection on the marker point areas in each group, remove the overlapping marker point areas in each group, and obtain the marker point area after each group is removed;

The adjacent point comparison unit is used to compare the distance between the marked point regions after each group is eliminated and the adjacent threshold distance of the marked point to obtain a plurality of adjacent point comparison results;

A center validity judging unit, configured to judge, based on the comparison results of each of the adjacent points, whether the area centers in the marked point areas of the groups after the elimination are valid or not.

In one embodiment, the overlapping area detection unit includes:

The area combination comparison subunit is used to combine and compare the mark point areas in each group respectively;

The first detection subunit is used to calculate the overlapping detection distance between the marked point regions of the combined comparison;

The second detection subunit is used to calculate the area side length and area area of each marker point area in the combined comparison respectively when the overlapping detection distance satisfies the contour detection threshold;

The overlapping area culling subunit is used for culling the overlapping marker point areas in each group based on the area side length and area area of each marker point area compared by the combination to obtain the culled marker point area of each group.

In one embodiment, the marker point sequence generation module 508 includes:

The center of gravity position calculation unit is used to perform mean calculation based on the effective area center with the same pose, and obtain the center of gravity position of the center of the area with different poses;

a center marker point determination unit, which is used to find out the center marker points of each pose from the center of each of the regions based on the distance between the center of gravity of the same pose and the center of the effective area;

The origin generating unit is used to determine the center mark points of the center of each of the regions in different poses as the polar coordinate origin under each of the poses;

a polar coordinate system construction unit, configured to obtain the position of each of the regional centers in the polar coordinate system of each of the poses based on the center of each region and the polar coordinate origin under each of the poses;

The marker point sequence generating unit is configured to sort the positions of the centers of the regions in the polar coordinate system of the poses according to the angles of the centers of the regions in the polar coordinate system, to obtain The sequence of landmark points in different poses.

In one embodiment, the coordinate system transformation relationship includes a rotation transformation relationship and a translation transformation relationship, and the hand-eye calibration module 510 includes:

a rotation relationship calibration unit, used for calibrating the rotation transformation relationship between the robot and the scanner through the rotation vectors corresponding to the sequence of landmarks in different pose transformation processes;

a first vector calculation unit, used for obtaining the first translation vector through the origin translation information of the sequence of landmarks in different pose conversion processes;

The second vector calculation unit is used for calculating the marker point sphere center obtained by fitting the marker point sequence of the different poses, and the marker point sequence of the preset pose to obtain the second translation vector;

A translation relationship calibration unit, configured to combine the first translation vector and the second translation vector to obtain the translation conversion relationship.

Each module in the above-mentioned robot hand-eye coordinate conversion device may be implemented in whole or in part by software, hardware and combinations thereof. The above modules may be embedded in or independent of the processor in the computer device in the form of hardware, or may be stored in the memory of the computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, and the computer device may be a terminal, and its internal structure diagram may be as shown in FIG. 6 . The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input device. Wherein, the processor, the memory and the input/output interface are connected through the system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The input/output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with an external terminal, and the wireless communication can be realized by WIFI, mobile cellular network, NFC (Near Field Communication) or other technologies. When the computer program is executed by the processor, a method for converting hand-eye coordinates of a robot is realized. The display unit of the computer equipment is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device, the display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a display screen The touch layer covered on the top may also be keys, trackballs or touchpads provided on the casing of the computer equipment, and may also be an external keyboard, touchpad or mouse.

Those skilled in the art can understand that the structure shown in FIG. 6 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is also provided, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the foregoing method embodiments when the processor executes the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, implements the steps in the foregoing method embodiments.

In one embodiment, a computer program product is provided, including a computer program, which implements the steps in each of the foregoing method embodiments when the computer program is executed by a processor.

It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all It is the information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to a memory, a database or other media used in the various embodiments provided in this application may include at least one of a non-volatile memory and a volatile memory. Non-volatile memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetic variable memory (Magnetoresistive Random Memory) Access Memory, MRAM), Ferroelectric Random Access Memory (FRAM), Phase Change Memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory may include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration and not limitation, the RAM may be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM). The database involved in the various embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a blockchain-based distributed database, etc., but is not limited thereto. The processors involved in the various embodiments provided in this application may be general-purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, data processing logic devices based on quantum computing, etc., and are not limited to this.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present application. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the present application should be determined by the appended claims.

Claims (10)

1. A robot hand-eye coordinate transformation method, characterized in that the method comprises:

acquiring images of an end effector of the robot, which is provided with a calibration plate, by a scanner of the robot to obtain calibration plate images of the end effector at different poses;

detecting each marker point in the calibration plate images with different poses according to at least two graphs to obtain at least two groups of different marker point areas;

determining whether the region center of the landmark regions in each group is valid based on the distance between the landmark regions in each group;

generating marker point sequences with different poses according to the effective region centers and the central marker points of the effective region centers in the different poses;

and calibrating the coordinate system conversion relation between the robot and the scanner through the marker point sequences with different poses.

2. The method according to claim 1, wherein the detecting each marker point in the calibration plate images of different poses according to at least two graphs to obtain at least two groups of different marker point regions comprises:

performing edge extraction detection on each mark point in the calibration board image according to at least two different patterns respectively to obtain at least two groups of different image profiles;

screening the image contour of the contour length interval in each group;

and obtaining the at least two groups of different mark point areas based on the screened image outlines in the groups.

3. The method of claim 2, wherein the image contours comprise a circular contour and a polygonal contour, and the deriving the at least two different sets of landmark regions based on the image contours in the screened sets comprises:

similarity calculation is carried out on the basis of the screened outline area and the outline length of the circular outline, and the similarity of the circular outline is obtained;

selecting circular mark point areas corresponding to the mark points from the searched circular contour based on the similarity of the circular contour;

fitting the searched polygon outlines in each group to obtain a polygon fitting image;

fitting the polygon to a quadrilateral contour in the image to serve as a quadrilateral marking point area corresponding to each marking point;

the determining whether the area centers of the landmark areas in the respective groups are valid includes:

and judging whether the area center of the circular mark point area is effective or not based on the distance between the circular mark point area and the quadrilateral mark point area of each mark point.

4. The method of claim 1, wherein determining whether the region centers of the landmark regions in each group are valid based on the distances between the landmark regions in each group comprises:

respectively carrying out overlapping detection on the mark point areas in each group, and removing the overlapping mark point areas in each group to obtain the removed mark point areas of each group;

comparing the distance between each group of removed mark point areas with the adjacent threshold distance of the mark points to obtain a plurality of adjacent point comparison results;

and respectively judging whether the area center in each group of the removed mark point areas is effective or not based on the comparison result of each adjacent point.

5. The method according to claim 4, wherein the performing overlap detection on the marker point regions in each group respectively, and removing the overlapped marker point regions in each group to obtain each group of removed marker point regions comprises:

respectively carrying out combination comparison on the mark point areas in each group;

calculating the overlapping detection distance between the marker point areas of the combined comparison;

when the overlapping detection distance meets a contour detection threshold, respectively calculating the region side length and the region area of each mark point region for the combined comparison;

and based on the side length and the area of each mark point area which is combined and compared, eliminating the overlapped mark point areas in each group to obtain the mark point areas after the elimination of each group.

6. The method according to claim 1, wherein the generating of marker point sequences of different poses according to each valid region center and the center marker point of each valid region center in the different poses comprises:

carrying out averaging calculation based on the effective area centers with the same pose to obtain the gravity center positions of the area centers with different poses;

based on the distance between the gravity center position and the effective area center with the same pose, finding out the center mark point of each pose from the area center;

respectively determining the center mark points of the centers of the areas in different poses as the origin of polar coordinates under the poses;

obtaining the position of each region center in the polar coordinate system of each pose based on each region center and the polar coordinate origin of each pose;

and according to the angle of each region center in the polar coordinate system, sequencing the positions of each region center in the polar coordinate system of each pose respectively to obtain a marker point sequence of different poses.

7. The method according to any one of claims 1 to 6, wherein the coordinate system transformation relationship comprises a rotation transformation relationship and a translation transformation relationship, and the calibrating the coordinate system transformation relationship between the robot and the scanner through the marker point sequences of different poses comprises:

calibrating the rotation conversion relation between the robot and the scanner through the corresponding rotation vectors of the mark point sequence in the conversion process of different poses;

obtaining a first translational vector through the origin translation information of the mark point sequence in the conversion process of different poses;

calculating through the mark point spherical center obtained by fitting the mark point sequences with different poses and the mark point sequence with a preset pose to obtain a second translation vector;

and combining the first translation vector and the second translation vector to obtain the translation conversion relation.

8. A robot hand-eye coordinate conversion apparatus, characterized by comprising:

the image acquisition module is used for acquiring images of the end effector of the robot, which is provided with the calibration plate, through a scanner of the robot to obtain calibration plate images of the end effector at different poses;

the edge detection module is used for detecting each mark point in the calibration plate images with different poses according to at least two graphs to obtain at least two groups of different mark point areas;

the effective mark point judging module is used for judging whether the area center of the mark point areas in each group is effective or not based on the distance between the mark point areas in each group;

the mark point sequence generating module is used for generating mark point sequences with different poses according to the effective region centers and the center mark points of the effective region centers in the different poses;

and the hand-eye calibration module is used for calibrating the coordinate system conversion relation between the robot and the scanner through the mark point sequences with different poses.

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.

Images