CN107363831A - The teleoperation robot control system and method for view-based access control model - Google Patents

Abstract

The present invention relates to a vision-based remote operation robot control system and method. The control system includes a master station and a slave station. The master station includes a straight bracket, a cross bracket, a depth camera and a master station controller. The master station The station controller is connected with the depth camera, and determines the pose increment of the cross support and the angle between the straight support and the vertical direction according to the color image and the corresponding depth image; the slave station includes a mechanical arm, and the mechanical arm The end of the arm is connected to the mechanical gripper and the robotic arm controller; the robotic arm controller is connected to the main station controller through the network; the robotic arm controller is connected to the mechanical arm and the mechanical gripper respectively, according to the The pose increment of the cross bracket controls the target pose of the mechanical arm, and controls the opening and closing of the mechanical gripper according to the included angle, so that the operator can perform the operation tasks naturally, while simplifying the equipment structure and improving Recognition of complex actions of the operator and improvement of control precision.

Description

Vision-based teleoperation robot control system and method

technical field

The invention relates to the technical field of remote operation robot control, in particular to a vision-based remote operation robot control system and method.

Background technique

Although machine robotics has made great strides in recent years, current robots are still unable to independently perform complex and dangerous operational tasks, such as disposing of nuclear waste, dismantling explosives, underwater and space exploration, etc.

At present, robot teleoperation technology can not only isolate humans from dangerous operating environments, but also utilize human ability and experience in dealing with problems, so it has strong practical value and broad application prospects.

However, the existing teleoperation methods are generally divided into two types: contact type and non-contact type. Among them, the contact teleoperation method usually uses exoskeleton equipment, data gloves, inertial measurement and other methods to obtain the motion information of the operator, and then uses the motion information to control the robot to realize the teleoperation of the robot. Because these methods need to wear the sensor device on the operator's body, the operation movement will be limited and unnatural. The non-contact teleoperation method usually adopts a method based on visual measurement of the operator's action behavior. Therefore, the operator will move naturally during the operation, which is conducive to the operator to complete more complex teleoperation tasks. The accuracy requirement is higher.

In vision-based teleoperation systems, methods that do not use specific markers are most natural to the operator. However, this type of method has high requirements on the recognition algorithm, and currently it can only complete some simple operation tasks, such as simple clipping and so on. In contrast, the vision method using specific markers has the characteristics of simple algorithm, and high detection accuracy can be obtained on the basis of designing simple and effective markers and fast and efficient detection algorithms.

Contents of the invention

In order to solve the above-mentioned problems in the prior art, that is, in order to solve the problems of the complex structure of the external equipment required by the touch-type remote operation method, the unnaturalness of the operator when performing the operation task, and the low degree of recognition of the operator's complex actions, the present invention provides a A vision-based teleoperation robot control system and method.

To achieve the above object, the present invention provides the following scheme:

A vision-based teleoperation robot control system, the control system includes a master station and a slave station, the master station is connected to the slave station through a network; wherein,

The master station includes:

The cross support and the cross support are hand-held devices, and the cross support and the cross support are respectively driven to move by the hand movement of the operator;

Depth camera, used to collect color images and corresponding depth images when the cross support and the straight support move;

The main station controller is connected with the depth camera, and is used to determine the pose increment of the cross bracket and the angle between the straight bracket and the vertical direction according to the color image and the corresponding depth image;

The slave station includes a robotic arm, a mechanical gripper connected to the end of the robotic arm, and a robotic arm controller; the robotic arm controller is connected to the master station controller through a network, and receives information from the cross support. Pose increment and the angle between the straight bracket and the vertical direction; the robotic arm controller is respectively connected with the mechanical arm and the mechanical gripper for controlling the mechanical arm according to the posture increment of the cross bracket The target pose of the arm, and the opening and closing of the mechanical gripper is controlled according to the included angle.

Optionally, a sphere is provided at the three ends of the cross bracket, and a sphere is provided at the joint of the cross bracket; a sphere is respectively provided at both ends of the straight bracket, and the six The spheres are all different colors.

Optionally, the master station also includes:

Six classifiers are respectively connected with the depth camera and the main station controller, and are used to classify and recognize the colors of the six spheres in the color image;

The master station controller is also used to determine the positions of the centers of the six spheres using the center of gravity method, and determine the three-dimensional positions of the six spheres in the current frame in the camera coordinate system according to the positions of the centers of the spheres and the depth image .

Optionally, the master station also includes:

Kalman filter, connected with the master station controller, for predicting the position of each spheroid in the next frame according to the three-dimensional space position of each spheroid in the camera coordinate system in the current frame;

The master station controller is also used to perform local detection and determine the three-dimensional space position of the spheres according to the predicted positions of the spheres in the next frame and the color image collected by the depth camera.

Optionally, the master station further includes an average value filter connected to the controller for filtering the pose increment of the cross support, and sending the filtered pose increment to the The robotic arm controller described above.

Optionally, the slave station also includes a network camera, which is used to collect images of the movement of the mechanical arm and the mechanical gripper and the working scene;

The master station also includes a display connected to the master station controller; the master station controller is also used to receive the images of the motion and working scene of the mechanical arm and the mechanical gripper collected by the network camera, and send them to The display performs display.

According to the embodiments of the present invention, the present invention discloses the following technical effects:

The vision-based teleoperation robot control system of the present invention collects the color images and corresponding depth images that drive the movement of the cross support and the straight support when the operator's hand moves by setting a depth camera, and determines the pose of the cross support through the master station controller Increment and the angle between the straight bracket and the vertical direction; so that the robot arm controller can control the target pose of the robot arm according to the pose increment of the cross bracket, and control the opening and closing of the mechanical gripper according to the angle, so that the operation The operator can perform operating tasks naturally, while simplifying the equipment structure, improving the recognition of the operator's complex movements, and improving the control accuracy.

To achieve the above object, the present invention provides the following scheme:

A vision-based teleoperation robot control method, the control method comprising:

At the main station, the color image and the corresponding depth image that drive the movement of the cross bracket and the straight bracket when the operator's hand moves are collected by the depth camera;

Determine the pose increment of the cross bracket and the angle between the straight bracket and the vertical direction through the master station controller according to the color image and the corresponding depth image;

At the slave station, the pose increment of the cross support and the angle between the straight support and the vertical direction are received by the controller of the mechanical arm, and the target position of the mechanical arm is controlled according to the pose increment of the cross support. posture, and control the opening and closing of the mechanical gripper according to the included angle.

Optionally, a sphere is provided at the three ends of the cross bracket, and a sphere is provided at the joint of the cross bracket; a sphere is respectively provided at both ends of the straight bracket, and the six The spheres are all different colors.

Optionally, the control method also includes:

Carry out classification and identification according to the colors of the six spheres in the color image through six classifiers; determine the positions of the centers of the six spheres by the center of gravity method through the master station controller, and according to the positions of the centers of the spheres and the The depth image determines the three-dimensional positions of the six spheres in the camera coordinate system in the current frame.

Optionally, the control method also includes:

Predict the position of each sphere in the next frame according to the three-dimensional space position of each sphere in the camera coordinate system in the current frame through the Kalman filter; The color image collected by the depth camera is used for local detection to determine the three-dimensional space position of the sphere.

According to the embodiments of the present invention, the present invention discloses the following technical effects:

The vision-based teleoperation robot control method of the present invention uses a depth camera to collect the color images and corresponding depth images that drive the movement of the cross bracket and the straight bracket when the operator's hand moves, and determines the pose increase of the cross bracket through the master station controller. The amount and the angle between the straight bracket and the vertical direction; then the controller of the robotic arm can control the target pose of the robotic arm according to the pose increment of the cross bracket, and control the opening and closing of the mechanical gripper according to the included angle, so that the operation The operator can perform operating tasks naturally, while simplifying the equipment structure, improving the recognition of the operator's complex movements, and improving the control accuracy.

Description of drawings

Fig. 1 is a schematic structural view of the vision-based teleoperated robot control system of the present invention;

Fig. 2 is the method for calculating pose of the embodiment of the present invention;

Fig. 3 is the flow chart of stereoscopic vision software of the embodiment of the present invention;

Fig. 4 is the first color subtable figure of the color table of the embodiment of the present invention;

Fig. 5 is the second color subtable figure of the color table of the embodiment of the present invention;

Fig. 6 is a diagram of the third color subtable of the color table according to the embodiment of the present invention.

Symbol Description:

Operator - 1, straight support - 2, cross support - 3, depth camera - 4, control computer - 5, network - 6, robotic arm controller - 7, mechanical gripper - 8, mechanical arm - 9, network camera —10.

detailed description

Preferred embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention, and are not intended to limit the protection scope of the present invention.

The invention provides a vision-based teleoperation robot control system and method. By setting a depth camera, the color image and the corresponding depth image driven by the movement of the cross bracket and the straight bracket when the operator's hand moves are collected, and the main station controller Determine the pose increment of the cross bracket and the angle between the straight bracket and the vertical direction; so that the robot arm controller can control the target pose of the robot arm according to the pose increment of the cross bracket, and control the position of the mechanical gripper according to the included angle. Open and close, so that the operator can perform the operation tasks naturally, while simplifying the equipment structure, improving the recognition of the operator's complex movements, and improving the control accuracy.

As shown in FIG. 1 , the vision-based teleoperation robot control system of the present invention includes a master station and a slave station, and the master station is connected to the slave station through a network 6 .

Wherein, the master station includes a straight frame 2, a cross frame 3, a depth camera 4 and a master station controller. The straight frame 2 and the cross frame 3 are handheld devices, which are respectively driven by the hand movement of the operator 1. The movement of the cross support 3 and the in-line support 2; the depth camera 4 is used to collect color images and corresponding depth images when the cross support 3 and the in-line support 2 move; the master station controller and the depth The camera 4 is connected to determine the pose increment of the cross support 3 and the angle between the straight support 2 and the vertical direction according to the color image and the corresponding depth image.

The slave station includes a robot arm 9, a robot gripper 8 connected to the end of the robot arm 9, and a robot arm controller 7; wherein, the robot arm controller 7 communicates with the master station controller through a network 6 connection, for receiving the pose increment of the cross support 3 and the angle between the straight support 2 and the vertical direction; the mechanical arm controller 7 is connected with the mechanical arm 9 and the mechanical gripper 8 respectively, according to the The pose increment of the cross support 3 controls the target pose of the mechanical arm 9, and controls the opening and closing of the mechanical gripper 8 according to the included angle. Wherein, the target pose of the robotic arm 9 is the initial pose of the robotic arm 9 plus a filtered pose increment. Specifically, when the angle between the inline bracket 2 and the vertical direction is 0 (that is, the inline bracket 2 is in a vertical posture), the mechanical gripper 8 is closed; when the angle between the inline bracket 2 and the vertical direction When the angle is 90° (that is, the straight bracket 2 is in a horizontal state), the mechanical gripper 8 is opened.

Preferably, the three ends on the cross bracket 3 are respectively provided with a sphere, and the joint of the cross bracket 3 is provided with a sphere; the two ends of the inline bracket 2 are respectively provided with a sphere, and The six spheres are all different colors. For example, the colors of the four spheres on the cross bracket 3 are respectively: the color of the sphere at the joint is red, and the colors of the spheres at the three ends are blue, green and yellow; Spheres can be colored purple and black; however, they are not limited to.

Further, the master station also includes six classifiers, which are respectively connected to the depth camera 4 and the master station controller, for classifying and identifying according to the colors of the six spheres in the color image;

The master station controller is also used to determine the positions of the centers of the six spheres using the center of gravity method, and determine the three-dimensional positions of the six spheres in the current frame in the camera coordinate system according to the positions of the centers of the spheres and the depth image .

As shown in Figure 2, the coordinates of the red sphere are (x _r , y _r , z _r ), the coordinates of the blue sphere are (x _b , y _b , z _b ), and the coordinates of the green sphere are (x _g , y _g , z _g ), the coordinates of the yellow sphere are (x _y , y _y , z _y ), where x, y, and z respectively represent the three-dimensional directions in the camera coordinate system.

Vector from red ball to green ball:

(x ₁ ,y ₁ ,z ₁ )=(x _g -x _r ,y _g -y _r ,z _g -z _r )-------formula (1);

Vector from red ball to yellow ball:

(x ₂ ,y ₂ ,z ₃ )=(x _y -x _r ,y _y -y _r ,z _y -z _r )-------formula (2);

The vector from the red ball to the green ball is the cross product of the above two vectors:

(x ₃ ,y ₃ ,z ₃ )=(y ₁ z ₂ -y ₂ z ₁ ,x ₂ z ₁ -x ₁ z ₂ ,x ₁ y ₂ -x ₂ y ₁ )--Formula (3);

Normalize the above three vectors to obtain unit vectors i, j, k, and through i, j, k, the roll angle R, tilt angle P, and yaw of the cross support 3 relative to the camera coordinate system in the camera coordinate system can be obtained Angle Y. Combining the position of the red ball on the cross support 3, the roll angle R, the tilt angle P, and the yaw angle Y, the pose of the cross support = (x _r , y _r , z _r , R, P, Y)--- --Formula (4).

Further, the master station further includes a memory storing a color table, each of the classifiers is connected to the memory, retrieves the color table, and determines the position of the color ball through a table look-up method. Wherein, the color table represents the classification results of 6 classifiers for 256×256×256 colors. Specifically, the color table includes 3 color sub-tables, and the size of each color sub-table is 256×256.

As shown in Figures 4 to 6, the embodiment of the present invention uses the YCbCr color space, and numbers the colors of each sphere, and the numbers are non-zero integers, such as 1 for blue, 2 for green, 3 for yellow, and 4 for red The size of the three color subtables is 256 * 256; when a color (Y, Cb, Cr) needs to be classified, at first check the first table (as shown in Figure 4) according to the Cb and Cr values of the color, If the corresponding value is 0, it means that the color is the background; if it is non-zero, continue to look at the second table (as shown in Figure 5) and the third table (as shown in Figure 6), if the Y component of the color A value between the corresponding values in the second and third tables indicates that it is the color of the sphere, otherwise it is the background; for example, for a color pair (x ₂ , y ₂ , z), in the first table (x ₂ , y ₂ ) The value of the position is 3, which means it may be yellow. Continue to look at the second and third tables, and get the values of the corresponding positions 84 and 229. If 84≤z≤229, it means the color is yellow, otherwise the color is the background.

In addition, the master station also includes a Kalman filter (not shown in the figure), which is connected to the controller of the master station and is used to predict the position of each sphere in the following three-dimensional space according to the three-dimensional position of each sphere in the camera coordinate system in the current frame The position in one frame; the master station controller is also used to perform local detection to determine the three-dimensional space position of the sphere according to the predicted position of each sphere in the next frame and the color image collected by the depth camera.

Take the detection of the sphere on the cross bracket as an example: as shown in Figure 3, the first step: the master station controller receives the color image obtained by the depth camera, and performs a process on the color image for the four spheres on the cross bracket Global detection until the three-dimensional space position of each sphere in the camera coordinate system in the current frame is determined, and the positions of the 4 spheres in the current frame are sent to the Kalman filter after the detection is successful; the Kalman filter predicts each The position of the sphere in the next frame is sent to the master station controller, so that the master station controller can perform local detection in a small range near the predicted position; if the detection is successful, the state of the Kalman filter is updated, otherwise If the detection fails, it will be transferred to the global detection. Similarly, the same process is used for the detection of the two balls on the straight bracket, so it will not be repeated here.

Specifically, the state equation of the Kalman filter is:

Among them, X(k) is the state of the system at time k, A is the system parameter, Z(k) is the measured value of the system at time k, H is the observation matrix of the system, W(k) represents the noise of the adjustment control process, V( k) represents the noise of the measurement process. In this embodiment, both the noise of the adjustment control process and the noise of the measurement process are Gaussian white noise.

Assuming that the state X(k) only includes the position and velocity in the x direction at time k, to include the y direction and z direction, you only need to expand them into X(k), then the setting method of X(k), A and H is :

X(k)=[x(k)v(k)] ^T ---------- formula (6);

Z(k)=[z(k)] ^T ---------------- formula (8);

H=[1 0]---------------------- formula (9);

Take the spatial positions x, y, and z of the six color balls on the cross bracket as observation values, thereby determining the observation matrix, the spatial positions x, y, z of the six balls and the velocities v _x , v _y , v _z in the corresponding directions As a state variable, to determine the state equation of the Kalman filter.

Further, the master station also includes an average value filter connected to the controller for filtering the pose increment of the cross support, and sending the filtered pose increment to the The robotic arm controller 7 is used to control the target pose of the robotic arm 9 .

In addition, in order to enable the operator to accurately control the movements of the mechanical arm 9 and the mechanical gripper 8 to complete the operation task, the slave station also includes a network camera 10 for collecting the motion and The image of the working scene; the master station also includes a display connected to the master station controller; the master station controller is also used to receive the movement of the mechanical arm and the mechanical gripper collected by the network camera and the image of the work scene image and send it to the monitor for display. In this embodiment, the master station controller and display are integrated in one control computer 5 .

The working process of the vision-based teleoperation robot control system of the present invention (as shown in FIG. 1 ): at the master station, the operator 1 holds the straight bracket 2 and the cross bracket 3 with both hands respectively and moves, and the cross bracket 3 controls the movement of the mechanical arm 9. The end pose, the one-shaped support 2 controls the opening and closing of the mechanical gripper 8; the depth camera 4 simultaneously obtains the color image and the corresponding depth image of the one-shaped support 2 and the cross support 3 during movement, so it can be determined that a point in the scene is in the For the three-dimensional position in the camera coordinate system, the spatial positions of the six spheres on the straight bracket 2 and the cross bracket 3 held by the operator 1 can be obtained through the method of stereo vision; Calculate the pose of the cross bracket and the angle between the straight bracket and the vertical direction by solving the spatial position; when the straight bracket is in a vertical posture, the mechanical gripper 8 is closed, and when it is in a horizontal state, the mechanical gripper 8 is opened; The pose of the bracket is used to represent the pose of the operator’s hand; when the system is started, the control computer first obtains the initial pose of the hand of the operator 1 in this way, and then continuously compares the subsequent obtained pose with the initial pose. difference, to obtain the pose increment of the master station operator 1 hand (that is, the pose increment of the cross bracket); the pose increment is obtained after a mean value filter; The angle between the straight bracket and the vertical direction and the pose increment are transmitted to the mechanical arm controller 7 of the slave station, and the mechanical arm controller 7 transmits the control signal to the mechanical arm and the mechanical gripper to control the mechanical arm 9 The movement of the robot and the opening and closing of the mechanical gripper 8; at the same time, the network camera 10 of the slave station collects images of the movement of the robot arm and the scene where the robot arm is located, and transmits the collected images back to the master station and displays them on the master station. displayed on the station monitor.

The vision-based teleoperation robot control system of the present invention extracts the posture and state of the operator's hand through a depth camera, a straight bracket and a cross bracket, thereby realizing teleoperation with the characteristics of natural operation and low cost; Kalman filter and The color table can reduce the amount of calculation during color detection and meet the real-time requirements of the system; using the network camera to send the image of the operation site back to the main station can intuitively display the state of the robot arm in the operation scene, which is conducive to enhancing the sense of presence and complete relatively complex tasks.

In addition, the present invention also provides a vision-based teleoperation robot control method. Specifically, the vision-based teleoperation robot control method of the present invention includes:

At the main station, the color image and the corresponding depth image that drive the movement of the cross bracket and the straight bracket when the operator's hand moves are collected by the depth camera;

Determine the pose increment of the cross bracket and the angle between the straight bracket and the vertical direction through the master station controller according to the color image and the corresponding depth image;

At the slave station, the pose increment of the cross support and the angle between the straight support and the vertical direction are received by the controller of the mechanical arm, and the target position of the mechanical arm is controlled according to the pose increment of the cross support. posture, and control the opening and closing of the mechanical gripper according to the included angle.

Wherein, the determination of the pose increment of the cross bracket and the angle between the straight bracket and the vertical direction by the master station controller according to the color image and the corresponding depth image specifically includes:

Step 101: The operator holds the cross bracket and the straight bracket in the main station to move, and the master station controller calculates the initial pose of the cross bracket and the angle between the straight bracket and the vertical direction through the method of visual measurement;

Step 102: The operator holds the cross bracket and the straight bracket at the master station and continues to move. The controller of the master station continuously makes a difference between the operator's hand posture obtained later and the initial pose to obtain the pose increment of the cross bracket, and then uses A mean filter filters the pose increments.

At the slave station, the robotic arm controller controls the target pose of the robotic arm according to the pose increment of the cross bracket, and controls the opening and closing of the mechanical gripper according to the included angle. Wherein, the target pose of the robotic arm is the initial pose of the robotic arm plus a filtered pose increment. Specifically, when the angle between the inline bracket and the vertical direction is 0 (that is, the inline bracket is in a vertical posture), the mechanical gripper is closed; when the angle between the inline bracket and the vertical direction is 90° ( That is, when the inline bracket is in a horizontal state), the mechanical gripper opens.

Optionally, a sphere is provided at the three ends of the cross bracket, and a sphere is provided at the joint of the cross bracket; a sphere is respectively provided at both ends of the straight bracket, and the six The spheres are all different colors. For example, the colors of the four spheres on the cross bracket 3 are respectively: the color of the sphere at the joint is red, and the colors of the spheres at the three ends are blue, green and yellow; Spheres can be colored purple and black; however, they are not limited to.

The vision-based teleoperation robot control method of the present invention also includes: classifying and identifying the six spheres in the color image through six classifiers; determining the centers of the six spheres through the master station controller position, and determine the three-dimensional space positions of the six spheres in the camera coordinate system according to the positions of the centers of the spheres and the depth images.

As shown in Figure 2, the coordinates of the red sphere are (x _r , y _r , z _r ), the coordinates of the blue sphere are (x _b , y _b , z _b ), and the coordinates of the green sphere are (x _g , y _g ,z _g ), the coordinates of the yellow sphere are (x _y ,y _y ,z _y ).

Vector from red ball to green ball:

(x ₁ ,y ₁ ,z ₁ )=(x _g -x _r ,y _g -y _r ,z _g -z _r )-------formula (1);

Vector from red ball to yellow ball:

(x ₂ ,y ₂ ,z ₃ )=(x _y -x _r ,y _y -y _r ,z _y -z _r )-------formula (2);

The vector from the red ball to the green ball is the cross product of the above two vectors:

(x ₃ ,y ₃ ,z ₃ )=(y ₁ z ₂ -y ₂ z ₁ ,x ₂ z ₁ -x ₁ z ₂ ,x ₁ y ₂ -x ₂ y ₁ )------Formula( 3);

Normalize the above three vectors to obtain unit vectors i, j, k, and through i, j, k, the roll angle R, tilt angle P, and yaw of the cross support 3 relative to the camera coordinate system in the camera coordinate system can be obtained Angle Y. Combining the position of the red ball on the cross support 3, the roll angle R, the tilt angle P, and the yaw angle Y, the pose of the cross support = (x _r , y _r , z _r , R, P, Y)--- - Formula (4).

Furthermore, the vision-based teleoperated robot control method of the present invention also includes calling a stored color table, and each classifier determines the position of the color ball through a table look-up method. The color table represents the classification results of 6 classifiers for 256×256×256 colors. Specifically, the color table includes 3 color sub-tables, and the size of each color sub-table is 256×256.

As shown in Figures 4 to 6, the embodiment of the present invention uses the YCbCr color space, and numbers the colors of each sphere, and the numbers are non-zero integers, such as 1 for blue, 2 for green, 3 for yellow, and 4 for red The size of the three color subtables is 256 * 256; when a color (Y, Cb, Cr) needs to be classified, at first check the first table (as shown in Figure 4) according to the Cb and Cr values of the color, If the corresponding value is 0, it means that the color is the background; if it is non-zero, continue to look at the second table (as shown in Figure 5) and the third table (as shown in Figure 6), if the Y component of the color A value between the corresponding values in the second and third tables indicates that it is the color of the sphere, otherwise it is the background; for example, for a color pair (x ₂ , y ₂ , z), in the first table (x ₂ , y ₂ ) The value of the position is 3, which means it may be yellow. Continue to look at the second and third tables, and get the values of the corresponding positions 84 and 229. If 84≤z≤229, it means the color is yellow, otherwise the color is the background.

Further, the vision-based teleoperation robot control method of the present invention also includes: predicting the position of each sphere in the next frame through the Kalman filter according to the three-dimensional space position of each sphere under the camera coordinate system in the current frame; The station controller performs local detection according to the predicted position of each sphere in the next frame and the color image collected by the depth camera to determine the three-dimensional space position of the sphere.

Take the detection of the sphere on the cross bracket as an example: as shown in Figure 3, the first step: use the color image obtained by the depth camera to globally detect the four spheres on the cross bracket until it is determined that the sphere is in the current position. position in the frame, and send the positions of the four balls in the current frame to the Kalman filter after successful detection; the Kalman filter predicts the position of each ball in the next frame, so that the master station controls The detector can perform local detection in a small area near the predicted position; if the detection is successful, the state of the Kalman filter will be updated, otherwise the detection will fail and transfer to the global detection, so as to realize the rapid positioning of the sphere and improve the detection speed. In the same way, the detection of the two balls on the straight bracket adopts the same process.

Wherein, the state equation of the Kalman filter is:

Among them, X(k) is the state of the system at time k, A is the system parameter, Z(k) is the measured value of the system at time k, H is the observation matrix of the system, W(k) represents the noise of the adjustment control process, V( k) represents the noise of the measurement process. In this embodiment, both the noise of the adjustment control process and the noise of the measurement process are Gaussian white noise.

Assuming that the state X(k) only includes the position and velocity in the x direction at time k, to include the y direction and z direction, you only need to expand them into X(k), then the setting method of X(k), A and H is :

X(k)=[x(k)v(k)] ^T ---------- formula (6);

Z(k)=[z(k)] ^T ---------------- formula (8);

H=[1 0]---------------------- formula (9);

The space coordinates x, y, z of the 6 color balls on the cross support are used as observation values to determine the observation matrix, the spatial positions x, y, z of the 6 balls and the velocities v _x , v _y , v _z in the corresponding directions As a state variable, to determine the state equation of the Kalman filter.

In addition, in order to allow the operator to accurately control the movements of the robotic arm 9 and the robotic gripper 8 to complete the operation task, the vision-based teleoperation robot control method of the present invention collects the information of the robotic arm and the robotic gripper through the network camera at the slave station. The image of the movement and working scene will be sent back to the master station controller of the master station through the network, and then sent to the monitor for display through the master station controller, which is convenient for the operator to observe. Preferably, the master station controller and the display are integrated on one control computer.

Compared with the prior art, the vision-based teleoperation robot control method of the present invention has the same beneficial effects as the above-mentioned vision-based teleoperation robot control system, which will not be repeated here.

So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings, but those skilled in the art will easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.

Claims (10)

1. A vision-based teleoperated robot control system, characterized in that, the control system comprises a master station and a slave station, and the master station is connected with the slave station through a network; wherein, The master station includes: The cross support and the cross support are hand-held devices, and the cross support and the cross support are respectively driven to move by the hand movement of the operator; Depth camera, used to collect color images and corresponding depth images when the cross support and the straight support move; The main station controller is connected with the depth camera, and is used to determine the pose increment of the cross bracket and the angle between the straight bracket and the vertical direction according to the color image and the corresponding depth image; The slave station includes a robotic arm, a mechanical gripper connected to the end of the robotic arm, and a robotic arm controller; the robotic arm controller is connected to the master station controller through a network for receiving the cross The pose increment of the bracket and the angle between the straight bracket and the vertical direction; the controller of the mechanical arm is connected with the mechanical arm and the mechanical gripper respectively, and is used to control the cross bracket according to the pose increment of the cross bracket. The target pose of the mechanical arm is controlled, and the opening and closing of the mechanical gripper is controlled according to the included angle. 2. The vision-based teleoperated robot control system according to claim 1, wherein a sphere is provided at the three ends of the cross support respectively, and a sphere is provided at the junction of the cross support ; Two ends of the straight bracket are respectively provided with a sphere, and the colors of the six spheres are all different. 3. The vision-based teleoperated robot control system according to claim 2, wherein the master station also includes: Six classifiers are respectively connected with the depth camera and the main station controller, and are used to classify and recognize the colors of the six spheres in the color image; The master station controller is also used to determine the positions of the centers of the six spheres using the center of gravity method, and determine the three-dimensional positions of the six spheres in the current frame in the camera coordinate system according to the positions of the centers of the spheres and the depth image . 4. The vision-based teleoperated robot control system according to claim 3, wherein the master station also includes: Kalman filter, connected with the master station controller, for predicting the position of each spheroid in the next frame according to the three-dimensional space position of each spheroid in the camera coordinate system in the current frame; The master station controller is also used to perform local detection and determine the three-dimensional space position of the spheres according to the predicted positions of the spheres in the next frame and the color image collected by the depth camera. 5. The teleoperated robot control system based on vision according to claim 1, wherein the master station also includes a mean value filter connected with the controller for increasing the pose of the cross support. The amount is filtered, and the filtered pose increment is sent to the controller of the robotic arm through the network. 6. The vision-based teleoperation robot control system according to any one of claims 1-5, wherein the slave station also includes a network camera for collecting motion and work of the mechanical arm and the mechanical gripper an image of the scene; The master station also includes a display connected to the master station controller; the master station controller is also used to receive the images of the motion and working scene of the mechanical arm and the mechanical gripper collected by the network camera, and send them to The display performs display. 7. A vision-based teleoperated robot control method, characterized in that the control method comprises: At the main station, the color image and the corresponding depth image that drive the movement of the cross bracket and the straight bracket when the operator's hand moves are collected by the depth camera; Determine the pose increment of the cross bracket and the angle between the straight bracket and the vertical direction through the master station controller according to the color image and the corresponding depth image; At the slave station, the pose increment of the cross support and the angle between the straight support and the vertical direction are received by the controller of the mechanical arm, and the target position of the mechanical arm is controlled according to the pose increment of the cross support. posture, and control the opening and closing of the mechanical gripper according to the included angle. 8. The vision-based teleoperated robot control method according to claim 7, wherein a sphere is respectively arranged at the three ends of the cross support, and a sphere is provided at the joint of the cross support ; Two ends of the straight bracket are respectively provided with a sphere, and the colors of the six spheres are all different. 9. The vision-based teleoperated robot control method according to claim 8, wherein the control method further comprises: Carry out classification and identification according to the colors of the six spheres in the color image through six classifiers; determine the positions of the centers of the six spheres by the center of gravity method through the master station controller, and according to the positions of the centers of the spheres and the The depth image determines the three-dimensional positions of the six spheres in the camera coordinate system in the current frame. 10. The vision-based teleoperated robot control method according to claim 9, wherein the control method further comprises: Predict the position of each sphere in the next frame according to the three-dimensional space position of each sphere in the camera coordinate system in the current frame through the Kalman filter; The color image collected by the depth camera is used for local detection to determine the three-dimensional space position of the sphere.