CN211890823U

CN211890823U - Four-degree-of-freedom mechanical arm vision servo control system based on RealSense camera

Info

Publication number: CN211890823U
Application number: CN201922394781.8U
Authority: CN
Inventors: 陈彦杰; 赖宁斌; 梁嘉诚; 何炳蔚; 林立雄
Original assignee: Fuzhou University
Current assignee: Fuzhou University
Priority date: 2019-12-27
Filing date: 2019-12-27
Publication date: 2020-11-10
Anticipated expiration: 2029-12-27

Abstract

The utility model relates to a four-degree-of-freedom mechanical arm vision servo control system based on a RealSense camera, which comprises an upper computer and a mechanical device; the mechanical device comprises a four-degree-of-freedom mechanical arm, a mechanical arm support, a RealSense D435i camera and a camera support; the four-degree-of-freedom mechanical arm is fixed on a mechanical arm support through bolts, the RealSense D435i camera is connected with the camera support through screws, the camera support is fixed on the mechanical arm support through bolts, and the camera support mounting holes and the four-degree-of-freedom mechanical arm mounting holes are uniformly mounted at the positions of the mechanical arm support mounting holes. The utility model discloses can be fast, stable snatch specific object, it is simple to equip the mechanism, low in production cost snatchs degree of automation height.

Description

Four-degree-of-freedom mechanical arm vision servo control system based on RealSense camera

Technical Field

The utility model relates to a mechanical arm snatchs control technical field, concretely relates to four degree of freedom arm vision servo control systems based on RealSense camera.

Background

As robot research and technology development are continuously progressing, machine vision has been widely used in the field of robots. The mechanical arm is an important component in the field of robots, in the industrial field, the mechanical arm can only grab objects with known positions and directions according to a preset teaching route, and later along with the development of machine vision, the mechanical arm has the capability of sensing the external environment like human beings, so that the mechanical arm can not only realize simple grabbing tasks, but also can intelligently execute the grabbing tasks according to the requirements of users. The advantage of make full use of arm and the special long combination of vision sensor, can accurately distinguish and discern the target object, accurate completion is to the snatching of target object, has important realistic meaning and research value. Therefore, with the rapid development of robotics, computer technologies and imaging equipment, the robot visual servo attracts the attention of students and engineers in various fields, and has important application value in the fields of industrial sorting, medical assistance, automatic assembly and the like.

Disclosure of Invention

In view of this, the present invention provides a four-degree-of-freedom mechanical arm vision servo control system based on a RealSense camera, which is convenient, fast and stable to use.

In order to achieve the above purpose, the utility model adopts the following technical scheme:

a four-degree-of-freedom mechanical arm vision servo control system based on a RealSense camera comprises an upper computer and a mechanical device; the mechanical device comprises a four-degree-of-freedom mechanical arm, a mechanical arm support, a RealSense D435i camera and a camera support; the four-degree-of-freedom mechanical arm is fixed on a mechanical arm support through bolts, the RealSense D435i camera is connected with the camera support through screws, the camera support is fixed on the mechanical arm support through bolts, and the camera support mounting holes and the four-degree-of-freedom mechanical arm mounting holes are uniformly mounted at the positions of the mechanical arm support mounting holes.

Further, the RealSense D435i camera is mounted in an Eye To Hand mode in the first view angle.

Further, the human-computer interface diagram of the upper computer comprises a camera opening button, a camera closing button, a three-dimensional pose estimation button, an RGB image display window and a target feature identification extraction display window.

Compared with the prior art, the utility model following beneficial effect has:

the utility model discloses make full use of arm advantage and visual sensor specialty combination, can accurately distinguish and discern the target object, can be quick, stable snatch specific object, equipment mechanism is simple, low in production cost, can extensively apply to scenes such as the in disorder part arrangement of industry and letter sorting, unmanned aerial vehicle arm operation, have extensive market prospect.

Drawings

Fig. 1 is a control flow chart of the present invention;

fig. 2 is a human-computer interface diagram of the robot vision servo operation upper computer in an embodiment of the present invention;

fig. 3 is a schematic diagram of selecting a virtual coordinate system and feature points of a target object according to an embodiment of the present invention;

fig. 4 is a structural diagram of a RealSense camera and a four-degree-of-freedom robot according to an embodiment of the present invention;

fig. 5 is a diagram illustrating the change of the joint angle of the four-degree-of-freedom mechanical arm according to an embodiment of the present invention;

fig. 6 is a diagram illustrating a change in the position of the end of the four-degree-of-freedom robot arm according to an embodiment of the present invention.

Detailed Description

The present invention will be further explained with reference to the drawings and the embodiments.

Referring to fig. 1, as shown in fig. 4, the present embodiment provides a system of a four-dof robot vision servo control method based on a RealSense camera, including an upper connection and a mechanical device; the mechanical device comprises a bolt 1, a mechanical arm base 2, a camera connecting piece 3, a RealSense D435i camera 4, a four-degree-of-freedom mechanical arm 5 and a target object 6; the RealSense D435i camera 4 is connected with the camera connecting piece 3 through a screw (not shown in the figure), and is connected and fixed with the camera connecting piece 3 and the four-degree-of-freedom mechanical arm 5 through a bolt 1 on the mechanical arm base 2; when the RealSense D435i camera 4 identifies the characteristics of the target object 6, the pose information of the target object 6 in the base coordinate system of the four-degree-of-freedom mechanical arm 5 is calculated, then the Euler angle of the tail end attitude of the mechanical arm 5 is calculated, the rotation angle of each joint of the mechanical arm 5 is calculated according to the inverse kinematics of the robot, the rotation angle deviation required by each joint when the mechanical arm 5 reaches the characteristic expected point is further calculated according to the initial joint angle of the mechanical arm 5, the deviation is subjected to linear fitting and is converted into a control signal to be transmitted to the controller of the mechanical arm 5, and the mechanical arm 5 is driven to complete a specific target grabbing task.

Preferably, as shown in fig. 1, the utility model provides a four-degree-of-freedom mechanical arm vision servo control method based on RealSense camera, including the following steps:

step S1, acquiring the image and depth information of the target object according to the RealSense camera;

step S2, identifying the position information and the posture information of the target object in the camera coordinate system according to the obtained image and the depth information, and converting the position information and the posture information into the posture information in the mechanical arm base coordinate system according to the camera external parameters;

step S3, according to the position and the attitude information under the basic coordinate system of the mechanical arm, the Euler angle of the attitude of the tail end of the mechanical arm is solved, and the rotation angle of each joint of the mechanical arm is solved according to the inverse kinematics of the robot;

and step S4, according to the initial joint angle of the mechanical arm, calculating the rotation angle deviation required by each joint when the mechanical arm reaches the characteristic expected point, performing linear fitting on the rotation angle deviation, converting the rotation angle deviation into a control signal, transmitting the control signal to the mechanical arm controller, and driving the mechanical arm to complete the task of grabbing the target object.

Referring to fig. 2, in the present embodiment, an upper computer is provided, and a robot arm visual servo operation is performed on a human-computer interface diagram of the upper computer. As shown in fig. 2, included in the interface are: the method comprises the steps of opening a camera button, closing the camera button, estimating a three-dimensional pose, displaying an RGB image and identifying, extracting and displaying a target feature. Before a user operates an interface, firstly, a camera RealSense D435i is inserted into a USB3.0 port of a personal computer; when the method is operated, only a camera button is clicked to open, the RGB image of the camera is displayed on the window, and whether a target object is in the camera view field or not is observed; and then, only by clicking the three-dimensional pose estimation button, the program background can perform a series of actions of identifying a target object, extracting and extracting, fitting a contour, extracting position information, extracting posture information, converting a pose relation coordinate system, performing gesture relation robot inverse kinematics, driving a mechanical arm to grab a target and the like.

In this embodiment, the object with specific color characteristics is a green cylindrical bottle body of the RGB image display window in fig. 2. In the process of identifying the target object by the OpenCV algorithm, threshold segmentation is carried out on the color of the target object by using an inRange function, edge detection is carried out on a Canny function, then a required contour is searched by using a findContours function, according to the specific color of the target object, when contour features are extracted, the contour with the largest area in an image is the target contour, the contour of the feature of the target object is extracted, and the blue contour in a target object feature identification extraction display window is shown. When contour features are extracted, contour information is lost, contour fitting is carried out by utilizing an OpenCV function minAreaRect, a red rectangle is shown in a target feature identification extraction display window, the center point of the rectangle is extracted to be used as the center of a target, the three-dimensional coordinate of the center point is used as the position information of the target, and a red circle is shown in the target feature identification extraction display window.

Referring to fig. 3, in the present embodiment, an EPnP algorithm based on the solvepp function in OpenCV is used to estimate the targetThe object attitude is based on the precondition that the EPnP algorithm is applied to: camera reference matrix K, n (n)

4) 3D reference point coordinates in the world coordinate system and 2D reference point coordinates corresponding to the n 3D points and projected on the image;

obtaining the internal parameter of the RealSense camera by directly reading the internal parameter of the camera for the internal parameter matrix K of the camera;

for the 3D reference point coordinates in the n (n is more than or equal to 4) world coordinate systems, the EPnP algorithm represents the 3D coordinates in the world coordinate systems as the weighted sum of a group of virtual control points, and the virtual coordinate systems can be established in the coordinate system of the target object, so that the virtual three-dimensional coordinates of the characteristic points can be obtained. As shown in FIG. 4, the target is a bottle, and a virtual coordinate system of the target is established on the cylindrical part of the bottle

Selecting four

vertexes

1, 2, 3 and 4 of a bottle body rectangle and a cylindrical bottle body central point 5 as characteristic points, and obtaining virtual three-dimensional coordinates of the characteristic points in a virtual coordinate system according to the geometric dimension of the bottle body;

and extracting pixel coordinates corresponding to the feature points from the 2D reference point coordinates projected on the image corresponding to the n 3D points by using a RealSense camera image vision technology. As shown in fig. 3, after contour fitting is performed in OpenCV by using a minarefect function, pixel coordinates of five feature points are returned, three-dimensional coordinates and two-dimensional coordinates of the feature points need to be in one-to-one correspondence in an EPnP algorithm, a point which is farthest from an X axis of a pixel coordinate system is a first feature point in a sequence of the feature point coordinates returned by the minarefect function, then a second feature point to a fourth feature point are sequentially arranged in a clockwise sequence, and a fifth feature point is a rectangular center;

according to the above, the attitude rotation vector of the camera coordinate system under the virtual world coordinate system is obtained by applying the EPnP algorithm, the rotation vector is changed into a rotation matrix by using the Rodrigues function, and the matrix is inverted to obtain the attitude matrix of the target object under the camera coordinate system. Therefore, the pose of the target object in the camera coordinate system is obtained, and the pose of the target object in the mechanical arm base coordinate system is obtained according to the pose relation matrix of the camera in the mechanical arm base coordinate system.

In this embodiment, the joint angle of the robot arm is as shown in FIG. 5

And (3) linear fitting:

the end of the robot arm is finally driven to the desired point position of the target object by performing a rotational movement, as shown in fig. 6, from a point in space

Move to the desired point

And further grabbing to complete the visual servo task.

The above is only the preferred embodiment of the present invention, and all the equivalent changes and modifications made according to the claims of the present invention should be covered by the present invention.

Claims

1. A four-degree-of-freedom mechanical arm vision servo control system based on a RealSense camera is characterized by comprising an upper computer and a mechanical device; the mechanical device comprises a four-degree-of-freedom mechanical arm, a mechanical arm support, a RealSense D435i camera and a camera support; the four-degree-of-freedom mechanical arm is fixed on a mechanical arm support through bolts, the RealSense D435i camera is connected with the camera support through screws, the camera support is fixed on the mechanical arm support through bolts, and the camera support mounting holes and the four-degree-of-freedom mechanical arm mounting holes are uniformly mounted at the positions of the mechanical arm support mounting holes.

2. The RealSense camera-based four-degree-of-freedom robotic arm visual servo control system of claim 1, wherein: the RealSense D435i camera mounting mode is an Eye To Hand mode of a first view angle.

3. The RealSense camera-based four-degree-of-freedom robotic arm visual servo control system of claim 1, wherein: the human-computer interface diagram of the upper computer comprises a camera opening button, a camera closing button, a three-dimensional pose estimation button, an RGB image display window and a target feature identification extraction display window.