CN112476489B - Flexible mechanical arm synchronous measurement method and system based on natural characteristics - Google Patents
Flexible mechanical arm synchronous measurement method and system based on natural characteristics Download PDFInfo
- Publication number
- CN112476489B CN112476489B CN202011267484.8A CN202011267484A CN112476489B CN 112476489 B CN112476489 B CN 112476489B CN 202011267484 A CN202011267484 A CN 202011267484A CN 112476489 B CN112476489 B CN 112476489B
- Authority
- CN
- China
- Prior art keywords
- mechanical arm
- flexible mechanical
- arm
- pose
- flexible
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
- B25J19/02—Sensing devices
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J19/00—Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/26—Measuring arrangements characterised by the use of optical techniques for measuring angles or tapers; for testing the alignment of axes
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Manipulator (AREA)
Abstract
The invention provides a flexible mechanical arm synchronous measurement method based on natural characteristics, which can measure the arm shape of a flexible mechanical arm by using the natural characteristics such as rectangles and concentric circles at the positions of a frame, a joint shaft hole and the like on the flexible mechanical arm, so that the rotation angle of each joint of the flexible mechanical arm is calculated, and the space arm shape of the flexible mechanical arm is reconstructed by combining the calculation of a hand-eye camera on the pose of a calibration plate without adding markers such as two-dimensional codes, marker balls and the like or measuring devices such as joint encoders on the flexible mechanical arm. And meanwhile, a set of flexible mechanical arm vision measurement system is provided for carrying out the measurement of the shape and the joint angle of the flexible mechanical arm and the measurement of the target pose, and the vision measurement system can be calibrated, the algorithm measurement precision can be verified and the task under the typical working condition can be simulated before the task is carried out.
Description
Technical Field
The invention belongs to the technical field of robots, and relates to a flexible mechanical arm vision measurement method and system for measuring the shape and the target pose of a flexible mechanical arm.
Background
The system has the advantages of being small in operation space, multiple in barriers and the like in application scenes such as disaster rescue, environment detection, equipment overhaul and maintenance. The traditional robot is difficult to complete tasks in the scenes, and the rope-driven flexible mechanical arm which is flexible in movement and fine in size can play an important role. The flexible mechanical arm is mainly driven by a rope, however, due to the existence of stretching and contracting in the rope driving process, frictional resistance with a contact part and the like, the kinematic and kinetic equations of the flexible mechanical arm are complex and difficult to accurately reflect the actual characteristics of the flexible mechanical arm, so that the flexible mechanical arm is often difficult to accurately reach an expected given position in the task executing process.
Therefore, the positions and postures of all parts of the flexible mechanical arm are obtained by external sensing modes such as vision, so that vision closed-loop feedback of the motion process is realized, the deviation caused by rope errors and inaccurate model calculation is corrected, and the motion of the flexible mechanical arm is more accurate. The measurement method adopted at present mainly is to install typical characteristics such as two-dimensional codes on the arm lever of the flexible mechanical arm, and a marker ball can be added for the arm lever measurement of two-dimensional plane motion, but the methods all need to additionally install typical characteristics for auxiliary measurement on the flexible mechanical arm, the installation precision is difficult to guarantee, the motion performance of the flexible mechanical arm is influenced to a certain extent, and the difficulty in additionally installing the typical characteristics in some working occasions is large. Therefore, it is necessary to research a visual measurement method of natural characteristics of the flexible robot arm itself so as to measure the arm shape of the flexible robot arm easily and conveniently. In addition, flexible robotic arms typically work in unknown environments, constantly open for deployment, docking, and the like. For an unknown unstructured environment, before the work task is executed, the target needs to be detected and positioned to obtain the pose of the target in the environment, so that the flexible mechanical arm is controlled to reach the expected target pose and the task is carried out, and therefore, an effective measurement method and effective measurement equipment are also important.
Disclosure of Invention
The invention aims to provide a flexible mechanical arm synchronous measuring method based on natural characteristics aiming at the problems of poor motion precision, high difficulty in visual measurement of the arm shape of a flexible mechanical arm and complex operation of the conventional flexible mechanical arm, and the arm shape of the flexible mechanical arm can be measured by using the natural characteristics of rectangles, concentric circles and the like at the positions of a frame, a joint shaft hole and the like of the flexible mechanical arm. And meanwhile, a set of flexible mechanical arm vision measurement system is provided for carrying out the measurement of the shape and the joint angle of the flexible mechanical arm and the measurement of the target pose, and the vision measurement system can be calibrated, the algorithm measurement precision can be verified and the task under the typical working condition can be simulated before the task is carried out.
The technical scheme of the invention is that on one hand, the method for synchronously measuring the flexible mechanical arm based on natural characteristics comprises the following steps: A. respectively shooting a flexible mechanical arm image and a target calibration plate image through at least one global camera and at least one hand-eye camera; B. detecting concentric circle features in the flexible mechanical arm image, and setting an ROI (region of interest) region to search for rectangular features according to the positions of the concentric circle features so as to determine a target rectangular outline; C. combining the natural characteristic judgment with a flexible mechanical arm kinematic algorithm, calculating the position range of each arm rod and joint in the flexible mechanical arm through a positive kinematic algorithm of the flexible mechanical arm, and attributing the natural characteristic extracted from the flexible mechanical arm image to each determined arm rod and joint of the flexible mechanical arm; D. solving the pose of the rectangular feature in a camera coordinate system by utilizing a PnP algorithm, and calculating the Cartesian space poses of each joint and arm lever of the flexible mechanical arm according to the relationship between the pre-calibrated target rectangular feature and the joints and the arm levers of the flexible mechanical arm, so as to reconstruct the space arm shape of the flexible mechanical arm and further solve the terminal pose of the flexible mechanical arm; E. and for the calibration plate image shot by the hand-eye camera, identifying and calculating the pose of the calibration plate relative to the camera by using a calibration plate pose calculation algorithm, and using the pose calculation algorithm to correct the kinematic model of the flexible mechanical arm.
Further, the method further comprises: before the step B is executed, for the flexible mechanical arm image shot by the global camera, reducing the environmental noise through a median filtering algorithm, and enhancing the contrast of the characteristic edge and the environment through a high-pass filtering algorithm.
Further, the step B includes: b1, extracting edge features in the flexible mechanical arm image, and detecting a closed contour in the flexible mechanical arm image; b2, setting detection conditions according to the number of pixels occupied by the concentric circle features on the image plane, and screening out closed contours within a preset size range; b3, fitting an ellipse through a closed contour ellipse fitting function; b4, if the difference between the elliptical area of the fitted closed contour and the total area actually occupied by the contour is larger than a preset threshold value, rejecting the contour; b5, for each fitted ellipse, if other ellipses do not exist, so that the center positions of the ellipses are close to each other, eliminating the outline corresponding to the ellipse; b6, if the center positions of the two ellipses are close, if the ratio of the major axis to the minor axis of the two ellipses is too different or the rotation angles of the two ellipses are too different, rejecting the corresponding outline of the ellipse; b7, marking the removed concentric circle features as concentric circle features corresponding to the joint axis holes, and then setting an ROI (region of interest) near the concentric circle features as a search area of the arm lever edge rectangular features; b8, extracting straight line segments in the ROI area by using an LSD straight line detection algorithm; b9, eliminating straight line segments exceeding a preset length range; b10, removing straight line segments without parallel lines; b11, merging the straight line segments of which the deflection angles are smaller than a preset angle and the distances between the straight line segments are smaller than a preset distance into one straight line segment; b12, for the remaining straight-line segments, taking four straight-line segments as a group, and solving the number of the combination by using a permutation and combination algorithm; b13, judging whether the four straight line segments combined into one group can be combined into a rectangle according to the parallel and vertical characteristics of the side lines of the rectangle; and B14, marking the straight line segment group capable of forming the rectangle as the rectangular outline searched in the ROI area.
Further, the step C includes: c1, solving the pixel coordinates of the flexible mechanical arm tail end in the image plane, wherein, the pixel coordinates are obtained through pre-processingFirstly, the relative pose of the tail end of the known flexible mechanical arm and a camera is calibratedcamTendThen through camera internal reference TintCalculating the coordinate of the tail end of the flexible mechanical arm in a pixel plane as
C2, solving the pose of each arm rod of the flexible mechanical arm relative to the tail end of the flexible mechanical arm in Cartesian space by using the kinematics of the flexible mechanical armendTlinki(ii) a C3, solving the position of each arm lever of the flexible mechanical arm on the pixel plane; and C4, matching the pixel plane position distance and the detected rectangular outline with the obtained flexible mechanical arm lever.
Further, the step D includes: d1, calculating the pixel coordinates of four rectangular corner points of the rectangular characteristic contour of the paired arm rods of the flexible mechanical arm; d2, solving the pose of the rectangular feature contour edge in the camera coordinate system by utilizing a PnP algorithm; d3, calculating the Cartesian space poses of the joints and the arm levers of the flexible mechanical arm through the pre-calibrated target rectangular characteristics and the position relations among the joints and the arm levers of the flexible mechanical arm to reconstruct the space arm shape of the flexible mechanical arm, and further solving the terminal pose of the flexible mechanical arm.
Further, the step E includes: and D, if the position of the calibration plate is unknown, the pose of the calibration plate relative to the base can be solved by utilizing the pose of the tail end of the flexible mechanical arm relative to the base of the flexible mechanical arm calculated in the step D.
Or further, the step E comprises: and D, if the pose of the calibration plate is known, reversely solving the terminal pose of the flexible mechanical arm through the solved pose of the camera relative to the calibration plate, and combining the terminal pose of the flexible mechanical arm solved in the step D to correct the kinematic model of the flexible mechanical arm.
The invention also relates to a computer-readable storage medium, on which computer program instructions are stored, which, when executed by a processor, implement the above-mentioned method.
The technical scheme of the invention also relates to a flexible mechanical arm synchronous measurement system based on natural characteristics, the flexible mechanical arm comprises a plurality of groups of joints and arm rods, wherein the joints and the arm rods of each group comprise at least one mechanical part with natural characteristics, the natural characteristics comprise concentric circle characteristics and rectangular characteristics, and the synchronous measurement system comprises: a base for supporting the flexible robotic arm; a support disposed about said flexible robotic arm; the hand-eye camera is arranged at the tail end of the flexible mechanical arm; a calibration plate supported by the support, the calibration plate being positioned such that the calibration plate is captured by the hand-eye camera driven by the plurality of kinematic poses of the flexible robotic arm; a plurality of global cameras supported by the support for capturing images of the flexible robotic arm; a computer device comprising a memory and a processor, the processor implementing the above method when executing a computer program stored in the memory.
Compared with the prior art, the invention has the following characteristics:
1. the invention provides a flexible mechanical arm synchronous measurement method based on natural characteristics, which is characterized in that a global camera is used for shooting arm-shaped photos of a flexible mechanical arm, and the poses of the natural characteristics such as rectangular edges and bearing seat holes on arm rods and joints of the flexible mechanical arm are detected and solved, so that the rotation angle of each joint of the flexible mechanical arm is deduced; the method comprises the steps of shooting a picture of a target of a calibration plate by using a hand-eye camera, resolving the pose of the target relative to the camera, resolving the pose of the target in a Cartesian space when the pose of the target is unknown through a visual detection algorithm, wherein when the pose is known, the pose resolving result can be used for deducing the pose of the tail end of a flexible mechanical arm, and the pose resolving result can be used with a global camera visual detection result or can be used as a backup for each other according to different use conditions.
2. For the problem that the comparison between the natural features and the surrounding environment is not obvious enough compared with typical features such as two-dimensional codes and mark points, a filtering algorithm with a mixed filtering mode such as median filtering and high-pass filtering is added in the measuring method, so that the noise can be greatly reduced, the comparison between the edge and the surrounding environment is increased, and the edge of a target can be conveniently extracted.
3. For the problem of excessive detected interference targets, according to the position rule of the arm rod and the joint, in the measuring method, detecting conditions are set according to the relative positions of the rectangular edge on the arm rod and the concentric circle on the joint, and target features are extracted by matching with the detecting conditions of the length-diameter ratio, the size, the position and the like of the rectangular features and the concentric circle features.
4. The method comprises the steps of combining characteristic judgment and flexible mechanical arm kinematics, calculating the approximate positions of each arm section and joint through the positive kinematics of the flexible mechanical arm, and judging and extracting the attribution problem of the characteristic by taking the approximate positions as criteria.
Drawings
Fig. 1 is a schematic perspective view of a measurement system according to the present invention.
Fig. 2 shows the natural features for visual measurement on a flexible robotic arm according to the present invention.
Fig. 3 is a general flow chart of a measurement method according to the present invention.
Fig. 4 is a flow chart of concentric circle and edge rectangle outline detection in the measurement method according to the present invention.
Fig. 5 is a flowchart of the determination of the correspondence between the flexible manipulator arm segment and the rectangular feature of the image plane in the measurement method according to the present invention.
Detailed Description
The conception, the specific embodiment and the technical effects of the present invention will be clearly and completely described in the following with reference to the embodiments and the accompanying drawings to fully understand the objects, the embodiments and the effects of the present invention.
It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature or indirectly fixed or connected to the other feature. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term "and/or" includes any combination of one or more of the associated listed items.
It will be understood that, although the terms first, second, third, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element of the same type from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of the present disclosure. The use of any and all examples, or exemplary language ("e.g.," such as "or the like") provided herein, is intended merely to better illuminate embodiments of the invention and does not pose a limitation on the scope of the invention unless otherwise claimed.
Referring to fig. 1, a measurement object according to the present invention is a rope-driven flexible robot arm 10. The flexible robotic arm 10 comprises a plurality of serially connected articulated arm segments, each articulated arm segment comprising a combination of a universal joint and a substantially rectangular arm bar. Wherein the joints and arms of each set comprise at least one mechanical part having natural features including concentric circular features and rectangular features.
According to the synchronous measurement system, a plurality of functions such as flexible mechanical arm joint angle measurement, flexible mechanical arm tail end pose measurement, vision measurement system calibration, target pose measurement, vision task simulation and the like are realized. The system is simple to operate, and the performance and the motion rule of the flexible mechanical arm cannot be influenced.
Referring to fig. 1, in some embodiments, a synchronized measurement system according to the present invention may include a calibration plate 20, a hand-eye camera 30, a plurality of global cameras (including a first global camera 40 and a second global camera 50), a stand 60, and a base 70. Wherein the base 70 is used for supporting the flexible mechanical arm; a support 60 is disposed around the flexible robotic arm; the hand-eye camera 30 is arranged at the tail end of the flexible mechanical arm; the stand 60 supports the calibration board 20 and a plurality of global cameras.
In one embodiment of the synchronized measurement system, the pose of the natural features on the moving member of the flexible robotic arm in the cameras may be obtained by two or more global cameras (e.g., first global camera 40, second global camera 50). And the relative pose relationship between the rotation center and the natural characteristic is combined, so that the pose of each rotation center in the camera can be obtained. Since adjacent moving members (such as adjacent arms) are connected by using a two-degree-of-freedom unit (such as a joint part based on the universal coupling principle), the attitude of the rotation center coordinate system of the latter moving member with respect to the rotation center coordinate system of the former moving member is caused by angular rotation in two directions. The relative poses of two adjacent rotation centers are combined to obtain the rotation angles in the two directions, and the rotation angles of all joints are obtained in the same way.
In one embodiment of the synchronous measurement system, the pose of the tail end of the flexible mechanical arm can be obtained by measuring a two-dimensional code arranged at the tail end of the flexible mechanical arm through a global camera, and the pose of the tail end of the flexible mechanical arm can be obtained by observing an identifier at a known pose through a hand-eye camera. The two flexible mechanical arm tail end pose measuring methods can be used simultaneously under partial working conditions, and the measuring precision of the flexible mechanical arm tail end pose can be improved through methods such as judgment or fusion. The pose of the target in the hand-eye camera can be dynamically and accurately measured in a short distance by using the hand-eye camera, the pose of the tail end of the flexible mechanical arm in the base can be measured by using the two global cameras, and the pose of the target in the base can be obtained by combining the relative pose relationship between the hand-eye camera and the tail end of the flexible mechanical arm.
Referring to fig. 1 and 2, in one embodiment of the synchronous measurement system, the camera is used for measuring the characteristics of the rectangular frame and the concentric circle of the bearing seat on each moving member of the flexible mechanical arm, so that the pose of each characteristic in the camera can be obtainedcamTtag. By calibrating the pose of each feature and the rotation center of the adjacent motion member in advance (assuming that the coordinate system of the rotation center is numbered as K)tagTrotThen, the pose of each rotation center in the camera can be obtained as follows:
camTrot＝camTtag·tagTrot
according to the position and posture of the second global camera relative to the first global camera which is calibrated in advancecamTcam2And then, the measured poses of all the rotation centers are converted into the first global camera to be represented, and the poses of the rotation centers of the motion members measured in the second global camera are represented by the following formula:
camTrot＝camTcam2·cam2Ttag·tagTrot
the relative position and posture matrixes are homogeneous transformation matrixes of 4 multiplied by 4, and because the adjacent moving members are connected by adopting double free units, after the position and posture of the rotation center coordinate systems of the two adjacent moving members in the camera are obtained, the relative position and posture of the two adjacent rotation center coordinate systems can be obtainedrotiTrot(i+2)Therefore, the method for solving the rotation angles in two directions between two adjacent moving components comprises the following steps:
θi+1＝atan(rotiTrot(i+2)(1,3)/(-rotiTrot(i+2)(2,3)))
θi+2＝atan(rotiTrot(i+2)(3,1)/rotiTrot(i+2)(3,2))
aiming at the measurement of the pose of the target, the vision measurement system mainly measures the pose of the target in the hand-eye camera by means of the hand-eye cameracamTobjAnd the relative pose of the hand-eye camera calibrated in advance and the tail end of the flexible mechanical arm is utilizedendTcamAnd combining the pose of the tail end of the mechanical arm in the base coordinate system obtained by the global camerabaseTendAnd fusing the information of the hand-eye camera and the global camera, so as to obtain the pose of the target in the flexible mechanical arm base coordinate system, as shown in the following formula:
baseTobj＝baseTend·endTcam·camTobj
wherein it is flexiblePose of mechanical arm end in base coordinate systembaseTendThe pose of the rotation center of the root in the camera can be obtained by means of two-dimensional codes of the tail end and the root and combining the relative poses of the respective rotation centers and the two-dimensional codescam1TrotBPose with end centre of rotation in cameracam2TrotEAnd combining the relative pose relationship between the two cameras to obtainbaseTendAs follows:
baseTend＝(cam1Tbase)-1·cam1Tcam2·cam2Tend＝(cam1TrotB)-1·cam1Tcam2·cam2TrotE。
in one embodiment of the synchronous measurement system, the hand-eye camera 30 is used for observing the calibration plate 20 and solving the pose of the calibration plate relative to the camera coordinate system, the calibration plate 20 can be used for internal reference calibration of the hand-eye cameras 30 of the first global camera 40 and the second global camera 50, can be used as an observation object of the hand-eye camera 30 to test a target recognition and pose resolving algorithm of the flexible mechanical arm, can also be placed at a position with a known pose, and can be used for solving the end pose of the flexible mechanical arm by observing the pose through the hand-eye camera; the mount 60 may be used to mount the first global camera 40, the second global camera 50, the calibration board 20. The support is formed by the aluminium alloy equipment, and first global camera 40, the global camera 50 of second, calibration plate 20 mounted position are adjustable from top to bottom, left and right sides, can satisfy different task demands, and first global camera 40, the global camera 50 of second can be installed in support 60 different positions in order to observe flexible mechanical arm.
The synchronous measurement method implemented based on the synchronous measurement system detects and calculates the poses of the arm rods and joints of the flexible mechanical arm with natural characteristics such as rectangles, ellipses, concentric circles and the like, so that the rotation angles of all joints of the flexible mechanical arm are calculated, the space arm shape of the flexible mechanical arm is reconstructed by combining the calculation of the pose of the calibration plate by the hand-eye camera, and markers such as two-dimensional codes and marker balls or measuring devices such as joint encoders do not need to be added on the flexible mechanical arm.
Referring to fig. 3, in an embodiment, the method for synchronous measurement of a flexible robot arm based on natural characteristics according to the present invention includes the following steps S10 to S60:
and S10, respectively taking a picture of the flexible mechanical arm and a picture of the target calibration plate by the global camera and the hand-eye camera.
And S20, performing image filtering processing, and aiming at the problem that the contrast between natural features and the surrounding environment is small for flexible mechanical arm photos shot by the global camera, reducing the environmental noise by using a median filtering algorithm, and enhancing the contrast between feature edges and the environment by using a high-pass filtering algorithm, so that the edges of targets can be conveniently extracted.
And S30, detecting the rectangular outline of the target. The method comprises the steps of setting detection conditions to detect the rectangular edge of the arm section of the flexible mechanical arm, wherein the main process comprises the steps of firstly detecting concentric circle features 11 in an image, and then setting an ROI (region of interest) region to search for the rectangular features 12 according to the positions of the concentric circle features 11. Referring to fig. 4, the step S30 includes a concentric circle detecting step S31 and a rectangular outline detecting step S32. Specifically, step S31 may further include the following sub-steps: (1) extracting the edge characteristics of the image processed in the step S20, and detecting a closed contour in the image; (2) setting detection conditions according to the number of pixels occupied by the concentric circle features obtained by early-stage tests and theoretical calculation in an image plane, and excluding too large and too small outlines; (3) fitting an ellipse by using a closed contour ellipse fitting function; (4) if the difference between the fitted elliptical area and the actual total occupied area of the contour is large (the difference can be judged by judging whether the contour area is in the range of Smin and Smax), the contour is rejected, because the contour corresponding to the joint shaft hole on the image plane is smooth and complete, and if the difference between the fitted elliptical area and the contour area is large, the detected contour is irregular in shape; (5) for any fitted ellipse, if other ellipses do not exist and the center positions of the two ellipses are relatively close (for example, the difference between the center position of the fitted ellipse and the center position of the detected ellipse reaches more than 10 pixels), the outline corresponding to the ellipse is removed, because the ellipse does not meet the concentric circle condition; (6) if the central positions of the two ellipses are close, but the ratio of the major axis to the minor axis of the two ellipses is too different (for example, the ratio of the outline area to the elliptical area is not between 0.85 and 1), or the rotation angles of the two ellipses are too different (for example, the difference of the major axis to the minor axis of the ellipses close to the central positions is more than 0.2 and the rotation angle is more than 30 °), then the two ellipses are rejected because the ratio of the major axis to the minor axis of the concentric circles in the image plane and the rotation angle are approximately equal; (7) theoretically, the residual concentric circle features are the concentric circle features corresponding to the joint shaft holes, and because the joints are connected with the arm rods in series and the connected joints and the arm rods are closer in three-dimensional space positions, an ROI (region of interest) is arranged near the concentric circle features in an image plane and is used as a search region of the arm rod edge rectangular features. Because the operation speed of the rectangular search algorithm is greatly influenced by the number of detected straight line segments and the detection area, the search area can be reduced by setting the ROI, so that the number of the detected straight line segments is reduced, and the detection speed can be greatly improved. Specifically, step S32 may further include the following sub-steps: (8) extracting straight line segments in the ROI area by using an LSD (least squares) straight line detection algorithm; (9) eliminating too long and too short straight line segments to reduce the number of straight line segments; (10) eliminating straight line segments which are not approximately parallel; (11) searching any remaining straight line segment for a straight line segment close to the deflection angle of the straight line segment, judging the distance between the straight line segments, and combining the straight line segments into a straight line segment if the distance is close; (12) for the remaining straight-line segments, taking four straight-line segments as a group, and solving the possible number of combinations by using a permutation and combination algorithm; (13) judging whether the four straight line segments combined into one group in the previous step can be combined into a rectangle or not, wherein the judgment mode is to judge whether the four straight line segments are approximately parallel in pairs or not, whether the distance between the parallel lines is larger than the other two straight line segments and is smaller than 1.5 times of the other two straight line segments or not, and whether the two parallel lines are approximately vertical or not; (14) and stopping searching if a group of straight line segments exist in the last step to form a rectangle, wherein the rectangle formed by the group of straight line segments is the outline of the rectangle searched in the ROI area.
S40, the pixel space profile is paired with a cartesian space flexible manipulator. Since it is difficult to determine which segment of the flexible robot the rectangular outlines of the plurality of objects extracted from the pixel plane belong to in step S30, the feature determination is combined with the kinematics of the flexible robot in step S40, the approximate positions of the segments and joints of the flexible robot are estimated from the positive kinematics of the flexible robot, and the determination is performed using the estimated approximate positions as criteria to extract the rectangular outlines of the plurality of objectsAttribution to a feature. Referring to FIG. 5, the following sub-steps S41-S46 may be specifically included. S41, solving the pixel coordinates of the tail end of the flexible mechanical arm in the image plane, wherein the relative pose of the tail end of the flexible mechanical arm and the camera is known through pre-calibration and is set ascamTendThe camera internal parameter is TintThe coordinate of the end at the pixel plane is
S42, solving the pose of each arm segment of the flexible mechanical arm relative to the tail end of the flexible mechanical arm in Cartesian space by using the kinematics of the flexible mechanical armendTlinki. S43, solving the position of each arm segment of the flexible mechanical arm on the pixel plane, and knowing the position of the tail end on the pixel plane and the position of each segment of the arm rod relative to the tail end, the position of each arm segment on the pixel plane can be obtained. S44, comparing the rectangular contour detected in step S30 with the position of the arm at the pixel plane obtained in the previous step, and if there are a contour and an arm that are relatively close to each other (for example, the position of a certain arm in space is smaller than a threshold value from the center of the detected rectangle), the two correspond to the same flexible mechanical arm, so step S45 is executed to pair the pixel space contour with the cartesian space flexible mechanical arm; otherwise, step S46 is executed to determine that the corresponding relationship between the segment of flexible mechanical arm and the rectangular contour cannot be detected in the pixel space.
And S50, resolving the characteristic pose, and reconstructing the arm shape of the flexible mechanical arm. For the successfully matched rectangular contour in the step S40, pixel coordinates of four corner points of the contour are solved, the actual size of the arm lever in a three-dimensional space is known, the position of the rectangular edge in a camera coordinate system can be solved by utilizing a PnP algorithm, and the Cartesian space positions of each joint and arm lever of the flexible mechanical arm are calculated through the relationship between the pre-calibrated target feature and the joints and the arm lever of the flexible mechanical arm, so that the spatial arm shape of the flexible mechanical arm is reconstructed, and the terminal position of the flexible mechanical arm is further solved.
And S60, identifying and calculating the position of the camera relative to the calibration plate by using a calibration plate position calculation algorithm for the calibration plate photo shot by the hand-eye camera. If the position of the calibration plate is unknown, the pose of the calibration plate relative to the base of the flexible mechanical arm can be solved by utilizing the pose of the tail end of the flexible mechanical arm relative to the base of the flexible mechanical arm calculated in the step S50, if the pose of the calibration plate is known, the pose of the tail end of the flexible mechanical arm can be solved through the solved pose of the camera relative to the calibration plate, and the pose of the tail end of the flexible mechanical arm is combined with the pose of the tail end solved in the step S50, so that the kinematics model of the flexible mechanical arm is corrected, the arm shape reconstruction accuracy of the flexible mechanical arm is improved, and the comprehensive closed-loop control of the pose of the tail end of the flexible mechanical arm and the arm shape is realized.
It should be recognized that the method steps in embodiments of the present invention may be embodied or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The method may use standard programming techniques. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.
Further, the operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. The invention may also include the computer itself when programmed according to the methods and techniques described herein.
A computer program can be applied to input data to perform the functions described herein to transform the input data to generate output data that is stored to non-volatile memory. The output information may also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including particular visual depictions of physical and tangible objects produced on a display.
The above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiment, and any modifications, equivalent substitutions, improvements, etc. within the spirit and principle of the present invention should be included in the protection scope of the present invention as long as the technical effects of the present invention are achieved by the same means. The invention is capable of other modifications and variations in its technical solution and/or its implementation, within the scope of protection of the invention.
Claims (8)
1. A flexible mechanical arm synchronous measurement method based on natural characteristics is characterized by comprising the following steps:
A. respectively shooting a flexible mechanical arm image and a target calibration plate image through at least one global camera and at least one hand-eye camera;
B. detecting concentric circle features in the flexible mechanical arm image, and setting an ROI (region of interest) region to search for rectangular features according to the positions of the concentric circle features so as to determine a target rectangular outline;
C. combining the natural characteristic judgment with a flexible mechanical arm kinematic algorithm, calculating the position range of each arm rod and joint in the flexible mechanical arm through a positive kinematic algorithm of the flexible mechanical arm, so as to attribute the natural characteristic extracted from the flexible mechanical arm image to each determined arm rod and joint in the flexible mechanical arm, wherein the natural characteristic comprises a concentric circle characteristic and a rectangular characteristic;
D. solving the pose of the rectangular feature in a camera coordinate system by utilizing a PnP algorithm, and calculating the Cartesian space poses of each joint and arm lever of the flexible mechanical arm according to the relationship between the pre-calibrated rectangular feature and the joints and the arm levers of the flexible mechanical arm, so as to reconstruct the space arm shape of the flexible mechanical arm and further solve the terminal pose of the flexible mechanical arm;
E. and for the target calibration plate image shot by the hand-eye camera, identifying and calculating the pose of the target calibration plate relative to the camera by using a target calibration plate pose calculation algorithm, and using the pose calculation algorithm to correct the kinematic model of the flexible mechanical arm.
2. The method of claim 1, wherein the method further comprises:
before the step B is executed, for the flexible mechanical arm image shot by the global camera, reducing the environmental noise through a median filtering algorithm, and enhancing the contrast of the characteristic edge and the environment through a high-pass filtering algorithm.
3. The method according to claim 1 or 2, wherein said step B comprises:
b1, extracting edge features in the flexible mechanical arm image, and detecting a closed contour in the flexible mechanical arm image;
b2, setting detection conditions according to the number of pixels occupied by the concentric circle features on the image plane, and screening out closed contours within a preset size range;
b3, fitting an ellipse through a closed contour ellipse fitting function;
b4, if the difference between the elliptical area of the fitted closed contour and the total area actually occupied by the contour is larger than a preset threshold value, rejecting the contour;
b5, for each fitted ellipse, if other ellipses do not exist so that the difference of the central positions of the two ellipses reaches more than 10 pixels, rejecting the outline corresponding to the ellipse;
b6, if the center positions of the two ellipses are close to each other, if the difference between the ratio of the major axis to the minor axis of the two ellipses reaches more than 0.2 or the difference between the rotation angles of the two ellipses reaches more than 30 degrees, rejecting the corresponding outline of the ellipse;
and B7, marking the concentric circle features left after the elimination as concentric circle features corresponding to the joint axis holes, and then setting an ROI (region of interest) near the concentric circle features to serve as a search region of the arm lever edge rectangular features.
4. The method of claim 3, wherein step B comprises:
b8, extracting straight line segments in the ROI area by using an LSD straight line detection algorithm;
b9, eliminating straight line segments exceeding a preset length range;
b10, removing straight line segments without parallel lines;
b11, merging the straight line segments of which the deflection angles are smaller than a preset angle and the distances between the straight line segments are smaller than a preset distance into one straight line segment;
b12, for the remaining straight-line segments, taking four straight-line segments as a group, and solving the number of the combination by using a permutation and combination algorithm;
b13, judging whether the four straight line segments combined into one group can be combined into a rectangle according to the parallel and vertical characteristics of the side lines of the rectangle;
and B14, marking the straight line segment group capable of forming the rectangle as the rectangular outline searched in the ROI area.
5. The method of claim 1, wherein the step D comprises:
d1, calculating the pixel coordinates of four rectangular corner points of the rectangular characteristic contour of the paired arm rods of the flexible mechanical arm;
d2, solving the pose of the rectangular feature contour edge in the camera coordinate system by utilizing a PnP algorithm;
d3, calculating the Cartesian space poses of the joints and the arm levers of the flexible mechanical arm through the pre-calibrated rectangular features and the position relations among the joints and the arm levers of the flexible mechanical arm to reconstruct the space arm shape of the flexible mechanical arm, and further solving the terminal pose of the flexible mechanical arm.
6. The method of claim 1, wherein the step E comprises:
and D, reversely solving the terminal pose of the flexible mechanical arm through the solved pose of the camera relative to the target calibration plate, and combining the terminal pose of the flexible mechanical arm solved in the step D to correct the kinematic model of the flexible mechanical arm.
7. A computer readable storage medium having stored thereon program instructions which, when executed by a processor, implement the method according to any one of claims 1 to 6.
8. A flexible mechanical arm synchronous measurement system based on natural features, the flexible mechanical arm comprises a plurality of groups of joints and arm rods, wherein the joints and the arm rods in each group contain at least one mechanical part with natural features, the natural features comprise concentric circle features (11) and rectangular features (12), the synchronous measurement system is characterized by comprising:
a base (70) for supporting the flexible robotic arm;
a support (60) disposed about said flexible robotic arm;
a hand-eye camera (30) disposed at a distal end of the flexible robotic arm;
a target calibration plate supported by the support (60) and positioned such that the target calibration plate is captured by a hand-eye camera (30) driven by a plurality of kinematic postures of the flexible robot arm;
a plurality of global cameras supported by said support (60) for capturing images of the flexible robotic arm;
computer means for implementing the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011267484.8A CN112476489B (en) | 2020-11-13 | 2020-11-13 | Flexible mechanical arm synchronous measurement method and system based on natural characteristics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011267484.8A CN112476489B (en) | 2020-11-13 | 2020-11-13 | Flexible mechanical arm synchronous measurement method and system based on natural characteristics |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112476489A CN112476489A (en) | 2021-03-12 |
| CN112476489B true CN112476489B (en) | 2021-10-22 |
Family
ID=74930090
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011267484.8A Active CN112476489B (en) | 2020-11-13 | 2020-11-13 | Flexible mechanical arm synchronous measurement method and system based on natural characteristics |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112476489B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113043332B (en) * | 2021-04-01 | 2022-05-03 | 哈尔滨工业大学（深圳） | Arm shape measuring system and method of rope-driven flexible robot |
| CN113172659B (en) * | 2021-04-22 | 2023-06-09 | 哈尔滨工业大学（深圳） | Flexible robot arm shape measuring method and system based on equivalent center point identification |
| CN113297952B (en) * | 2021-05-21 | 2022-06-24 | 哈尔滨工业大学（深圳） | Measuring method and system for rope-driven flexible robot in complex environment |
| CN116276993B (en) * | 2023-03-03 | 2023-12-05 | 中国人民解放军军事科学院国防科技创新研究院 | Air bag type soft mechanical arm hand-eye calibration and measurement method based on cooperative mark |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015168177A1 (en) * | 2014-04-28 | 2015-11-05 | Massachusetts Institute Of Technology | Multi-link modular continuum robotic endoscope system |
| JP2017091079A (en) * | 2015-11-06 | 2017-05-25 | ファナック株式会社 | Image processing device and method for extracting image of object to be detected from input data |
| CN108340370A (en) * | 2017-01-24 | 2018-07-31 | 发那科株式会社 | The robot system of press device is controlled with power |
| CN108453727A (en) * | 2018-01-11 | 2018-08-28 | 中国人民解放军63920部队 | Mechanical arm tail end position and attitude error bearing calibration based on oval feature and system |
| CN109108942A (en) * | 2018-09-11 | 2019-01-01 | 武汉科技大学 | The mechanical arm motion control method and system of the real-time teaching of view-based access control model and adaptive DMPS |
| CN110293549A (en) * | 2018-03-21 | 2019-10-01 | 北京猎户星空科技有限公司 | Mechanical arm control method, device and neural network model training method, device |
| CN110695993A (en) * | 2019-09-27 | 2020-01-17 | 哈尔滨工业大学（深圳） | Synchronous measurement method, system and device for flexible mechanical arm |
| CN111015670A (en) * | 2019-12-28 | 2020-04-17 | 飞天诚信科技股份有限公司 | Mechanical arm and method for positioning, identifying and processing parts by using same |
| CN111238367A (en) * | 2020-01-13 | 2020-06-05 | 清华大学深圳国际研究生院 | Three-dimensional arm shape measuring method and device for rope-driven mechanical arm |
| WO2020115476A2 (en) * | 2018-12-05 | 2020-06-11 | University Of Plymouth | Robotic arm |
| CN111515943A (en) * | 2019-02-01 | 2020-08-11 | 发那科株式会社 | Three-dimensional shape measurement system and three-dimensional shape measurement method |
| CN111680685A (en) * | 2020-04-14 | 2020-09-18 | 上海高仙自动化科技发展有限公司 | Image-based positioning method and device, electronic equipment and storage medium |
| CN111783248A (en) * | 2020-07-01 | 2020-10-16 | 南京航空航天大学 | Industrial robot dynamics modeling and dynamics parameter identification method |
| EP3726645A1 (en) * | 2019-04-18 | 2020-10-21 | Aselsan Elektronik Sanayi ve Ticaret Anonim Sirketi | A configurable automatic test infrastructure |
-
2020
- 2020-11-13 CN CN202011267484.8A patent/CN112476489B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015168177A1 (en) * | 2014-04-28 | 2015-11-05 | Massachusetts Institute Of Technology | Multi-link modular continuum robotic endoscope system |
| JP2017091079A (en) * | 2015-11-06 | 2017-05-25 | ファナック株式会社 | Image processing device and method for extracting image of object to be detected from input data |
| CN108340370A (en) * | 2017-01-24 | 2018-07-31 | 发那科株式会社 | The robot system of press device is controlled with power |
| CN108453727A (en) * | 2018-01-11 | 2018-08-28 | 中国人民解放军63920部队 | Mechanical arm tail end position and attitude error bearing calibration based on oval feature and system |
| CN110293549A (en) * | 2018-03-21 | 2019-10-01 | 北京猎户星空科技有限公司 | Mechanical arm control method, device and neural network model training method, device |
| CN109108942A (en) * | 2018-09-11 | 2019-01-01 | 武汉科技大学 | The mechanical arm motion control method and system of the real-time teaching of view-based access control model and adaptive DMPS |
| WO2020115476A2 (en) * | 2018-12-05 | 2020-06-11 | University Of Plymouth | Robotic arm |
| CN111515943A (en) * | 2019-02-01 | 2020-08-11 | 发那科株式会社 | Three-dimensional shape measurement system and three-dimensional shape measurement method |
| EP3726645A1 (en) * | 2019-04-18 | 2020-10-21 | Aselsan Elektronik Sanayi ve Ticaret Anonim Sirketi | A configurable automatic test infrastructure |
| CN110695993A (en) * | 2019-09-27 | 2020-01-17 | 哈尔滨工业大学（深圳） | Synchronous measurement method, system and device for flexible mechanical arm |
| CN111015670A (en) * | 2019-12-28 | 2020-04-17 | 飞天诚信科技股份有限公司 | Mechanical arm and method for positioning, identifying and processing parts by using same |
| CN111238367A (en) * | 2020-01-13 | 2020-06-05 | 清华大学深圳国际研究生院 | Three-dimensional arm shape measuring method and device for rope-driven mechanical arm |
| CN111680685A (en) * | 2020-04-14 | 2020-09-18 | 上海高仙自动化科技发展有限公司 | Image-based positioning method and device, electronic equipment and storage medium |
| CN111783248A (en) * | 2020-07-01 | 2020-10-16 | 南京航空航天大学 | Industrial robot dynamics modeling and dynamics parameter identification method |
Non-Patent Citations (2)
| Title |
|---|
| Improved Mechanical Design and Simplified Motion Planning of Hybrid Active and Passive Cable-Driven Segmented Manipulator with Coupled Motion;Tianliang Liu;《Improved Mechanical Design and Simplified Motion Planning of Hybrid Active and Passive Cable-Driven Segmented Manipulator with Coupled Motion》;IEEE Xplore;20200127;第5978-5983页 * |
| 基于视觉的绳驱柔性机械臂形状测量与目标定位方法;严盼辉;《基于视觉的绳驱柔性机械臂形状测量与目标定位方法》;中国优秀博硕士学位论文全文数据库(硕士)信息科技辑;20200215;第I138-1890页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112476489A (en) | 2021-03-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112476489B (en) | Flexible mechanical arm synchronous measurement method and system based on natural characteristics | |
| JP6966582B2 (en) | Systems and methods for automatic hand-eye calibration of vision systems for robot motion | |
| CN107871328B (en) | Machine vision system and calibration method implemented by machine vision system | |
| JP6770605B2 (en) | Vision system for training the assembly system by virtual assembly of the object | |
| US9043146B2 (en) | Systems and methods for tracking location of movable target object | |
| JP6573354B2 (en) | Image processing apparatus, image processing method, and program | |
| JP2023052266A (en) | System and method for combining machine vision coordinate spaces in guided assembly environment | |
| JP6271953B2 (en) | Image processing apparatus and image processing method | |
| JP5671281B2 (en) | Position / orientation measuring apparatus, control method and program for position / orientation measuring apparatus | |
| US9355453B2 (en) | Three-dimensional measurement apparatus, model generation apparatus, processing method thereof, and non-transitory computer-readable storage medium | |
| JP6324025B2 (en) | Information processing apparatus and information processing method | |
| KR102314092B1 (en) | Calibration apparatus and the method for robot | |
| JPWO2015045834A1 (en) | Marker image processing system | |
| CN113172659B (en) | Flexible robot arm shape measuring method and system based on equivalent center point identification | |
| US20220230348A1 (en) | Method and apparatus for determining a three-dimensional position and pose of a fiducial marker | |
| JP7353757B2 (en) | Methods for measuring artifacts | |
| JP6626338B2 (en) | Information processing apparatus, control method for information processing apparatus, and program | |
| JP2022183032A (en) | Hand-eye calibration of camera-guided devices | |
| JP5976089B2 (en) | Position / orientation measuring apparatus, position / orientation measuring method, and program | |
| CN113043332B (en) | Arm shape measuring system and method of rope-driven flexible robot | |
| KR20130075712A (en) | A laser-vision sensor and calibration method thereof | |
| Abad et al. | Camera calibration using two concentric circles | |
| Bianchi et al. | Camera-marker alignment framework and comparison with hand-eye calibration for augmented reality applications | |
| CN112781521A (en) | Software operator shape recognition method based on visual markers | |
| Yu et al. | Vision-based method of kinematic calibration and image tracking of position and posture for 3-RPS parallel robot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |