WO2020221311A1 - 基于可穿戴设备的移动机器人控制系统及控制方法 - Google Patents

基于可穿戴设备的移动机器人控制系统及控制方法 Download PDF

Info

Publication number
WO2020221311A1
WO2020221311A1 PCT/CN2020/087846 CN2020087846W WO2020221311A1 WO 2020221311 A1 WO2020221311 A1 WO 2020221311A1 CN 2020087846 W CN2020087846 W CN 2020087846W WO 2020221311 A1 WO2020221311 A1 WO 2020221311A1
Authority
WO
WIPO (PCT)
Prior art keywords
hand
control
robot
virtual
degree
Prior art date
Application number
PCT/CN2020/087846
Other languages
English (en)
French (fr)
Inventor
纪鹏
马凤英
曹茂永
王斌鹏
李敏
Original Assignee
齐鲁工业大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN201910363168.1A external-priority patent/CN110039545B/zh
Priority claimed from CN201910363155.4A external-priority patent/CN109955254B/zh
Application filed by 齐鲁工业大学 filed Critical 齐鲁工业大学
Priority to KR1020207030337A priority Critical patent/KR102379245B1/ko
Publication of WO2020221311A1 publication Critical patent/WO2020221311A1/zh

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1602—Programme controls characterised by the control system, structure, architecture
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/006—Controls for manipulators by means of a wireless system for controlling one or several manipulators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
    • B25J9/1671—Programme controls characterised by programming, planning systems for manipulators characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00—Programme-controlled manipulators
    • B25J9/16—Programme controls
    • B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/01—Head-up displays
    • G02B27/017—Head mounted
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/002—Specific input/output arrangements not covered by G06F3/01 - G06F3/16
    • G06F3/005—Input arrangements through a video camera
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures

Definitions

  • the present disclosure relates to the technical field related to remote control of mobile robots, and in particular to a mobile robot control system and control method based on wearable devices.
  • a mobile reconnaissance robot is usually composed of a mobile robot body and a vehicle-mounted reconnaissance system. It can perform various combat tasks such as battlefield approach reconnaissance and surveillance, stealth raids, fixed-point clearance, nuclear, biological and chemical processing, and anti-terrorism and EOD.
  • the traditional vehicle-mounted reconnaissance system is generally composed of a camera and a two-degree-of-freedom gimbal, and its control method generally realizes the pitch control of the gimbal through the angle information of the pitch and yaw angle of the joystick.
  • the reconnaissance system is generally composed of a multi-degree-of-freedom manipulator and a reconnaissance camera, where the reconnaissance camera is fixedly connected to the end of the multi-degree-of-freedom manipulator.
  • the end pose of the robot refers to the position and posture of the end effector of the robot in the specified coordinate system.
  • the end effector of the mobile reconnaissance robot is a camera.
  • the end pose of the reconnaissance robot is determined by the end pose of the multi-degree-of-freedom manipulator, and there are many freedoms.
  • the end pose control of the robotic arm usually uses a button or a joystick combined with a button control method. The operator needs to memorize the correspondence between each button and each joint of the vehicle-mounted multi-freedom manipulator. Therefore, this operation method is very complicated and Not intuitive.
  • gestures to control the end pose of a vehicle-mounted multi-free reconnaissance system.
  • a more common gesture control method is to use data gloves or inertial elements. Its advantages are high recognition rate and good stability, but the disadvantage is that it cannot control the end position of the vehicle-mounted multi-degree-of-freedom reconnaissance system and can only control the attitude.
  • input devices are expensive and inconvenient to wear.
  • the other gesture control method is based on vision. This control method can be divided into the control method based on image classification and the control method based on image processing. The former generally analyzes the types of gestures through visual sensors combined with pattern recognition methods.
  • the disadvantage is that it cannot quickly and accurately realize the continuous control of the end-position of the vehicle-mounted multi-degree of freedom reconnaissance system;
  • the latter generally analyzes the motion trajectory of gestures through visual sensors combined with image processing methods, and then realizes the position control of the end of the vehicle multi-degree-of-freedom reconnaissance system based on the position information of the trajectory.
  • Its disadvantage is that it cannot realize the end attitude of the vehicle multi-degree of freedom reconnaissance system. control.
  • the traditional remote control system of a mobile robot is usually implemented by a control box or control box with a joystick and buttons.
  • the buttons of the control box are more complicated.
  • the operator needs to memorize the corresponding relationship between each button and the mobile robot and the vehicle-mounted manipulator.
  • the control method is very unintuitive.
  • Mobile robots and vehicle-mounted reconnaissance systems can’t get rid of their dependence on joysticks, and joysticks need the support of a control box and related hardware devices. Therefore, the controllers of traditional mobile reconnaissance robots are generally larger in size. The problem is that it is not convenient to carry and transport.
  • Gesture is one of the more natural means of human communication, especially for special forces, sign language is a necessary means to communicate with teammates and convey instructions. Especially when it is inconvenient to use voice to communicate, gestures are almost the only means of communication and instructions between special forces.
  • human-computer interaction remote control based on human gestures mainly adopts the method of wearing data gloves or inertial elements. Its advantages are high recognition rate and good stability, but the disadvantages are expensive input devices and inconvenient wearing. Therefore, for fully armed soldiers, how to improve the portability and intuitiveness of the man-machine interactive teleoperation control system of ground-armed reconnaissance robots is a very urgent need.
  • the present disclosure proposes a mobile robot control system and control method based on wearable devices, and provides a mobile robot control system and control method based on wearable devices for mobile robots equipped with multi-degree-of-freedom manipulators. Wearing and detaching realize the continuous control of the end position and posture of the multi-degree-of-freedom manipulator to solve the complicated control methods in the control of the existing mobile reconnaissance robot vehicle multi-degree-of-freedom reconnaissance system and the inability to intuitively control the end position of the vehicle multi-degree-of-freedom reconnaissance system The question of posture.
  • the first aspect of the present disclosure provides a mobile robot control system based on a wearable device, including a master-end wearable teleoperation control device and a slave-end robot.
  • the master-end wearable teleoperation control device and the slave-end robot are wireless
  • the master-end wearable teleoperation control device is worn on the operator and is used to send control instructions and receive data collected by the slave robot;
  • the main-end wearable teleoperation control device includes a wearable binocular camera device, a head-mounted virtual display, a teleoperation controller and a main-end wireless communication device.
  • the teleoperation controller is respectively connected with the wearable binocular camera device and the head-mounted virtual
  • the display is connected to the master-end wireless communication device, the wearable binocular camera device is used to collect the image of the operator's gesture, the head-mounted virtual display is used to display the image taken by the slave robot and the robot arm of the slave robot Virtual model and virtual model of operator gestures.
  • the operator’s head is equipped with a wearable binocular camera device and a head-mounted virtual display, which can realize dual-view image collection.
  • the head-mounted virtual display setting can simultaneously realize the virtual model and the collected surveillance images, which can make the operator feel physically present.
  • the sense of environment can realize the intuitive control of the remote slave robot.
  • the setting of the wearable device frees the operator's hands and reduces the operator's burden.
  • a second aspect of the present disclosure provides a teleoperation control method for the end pose of a robot based on the above-mentioned mobile robot control system, which includes the following steps:
  • Step 101 Set the traction hand type and the release hand type
  • Step 102 Construct a virtual robot arm and a virtual gesture model, and display the virtual robot arm and the virtual gesture model on the front end of the visual body of the head-mounted virtual display;
  • Step 103 Collect dual-view images of the binocular camera
  • Step 104 Use a gesture detection algorithm to detect and determine whether there is an operator's gesture in the dual-view image, if yes, go to step 105, otherwise go to step 103;
  • Step 105 Use a hand shape recognition algorithm to perform hand shape recognition on the gesture, and determine whether a traction hand shape appears, if yes, go to step 106, otherwise go to step 103;
  • Step 106 Process the captured dual-view image and calculate the pose P H of the traction gesture in the coordinate system of the wearable binocular camera device, and convert the pose P H to the position in the screen coordinate system of the head-mounted virtual display.
  • Pose description P V which uses the transformed pose P V to drive the virtual gesture model in the visual body of the head-mounted virtual display;
  • Step 107 Determine whether the difference between the pose P V of the virtual gesture model and the end pose P_M of the virtual manipulator N6 is less than a preset threshold, if yes, go to step 108, otherwise go to step 103;
  • Step 108 Make the pose of the multi-degree-of-freedom manipulator follow the change of the operator's traction hand pose
  • Step 109 It is judged whether there is a detachable hand type. If it is, the position of the multi-degree-of-freedom manipulator stops following the change of the operator's pulling hand position, and step 103 is executed; otherwise, step 108 is executed.
  • the teleoperation control method for the end pose of the robot provided in the second aspect of the present disclosure, when the traction hand type is detected, the driving relationship between the operator's gesture pose and the end pose of the multi-degree-of-freedom manipulator is established to realize the multi-degree-of-freedom machine Continuous control of the posture of the end of the arm; when the disengagement hand is detected, the disengagement is performed so that the posture of the multi-degree-of-freedom manipulator stops following the change of the operator's traction hand posture.
  • the following process and the detaching process of the end of the virtual manipulator and the virtual gesture model are displayed in the head-mounted virtual display, making the control process more intuitive.
  • the third aspect of the present disclosure provides a control method based on the above-mentioned mobile robot control system, which separately collects the actions of the operator’s left and right hands, and controls the movement of the mobile robot body through the actions of one hand.
  • the motion control of the on-board multi-degree-of-freedom manipulator of the mobile robot includes the following steps:
  • Step 201 Collect images within the shooting range of the wearable device of the operator;
  • Step 202 Determine whether there is a hand area in the collected image, if not, go to step 201; otherwise, perform preprocessing on the collected image to obtain a hand piece;
  • Step 203 Determine whether the obtained hand piece is a left-hand piece or a right-hand piece by using the left-hand and right-hand discrimination algorithm, so as to determine whether the movement is the left hand or the right hand;
  • Step 204 Control the movement of the vehicle body of the mobile robot by the movement of one of the hands, and control the movement of the on-board multi-degree-of-freedom manipulator of the mobile robot by the movement of the other hand, and then perform step 201.
  • the control method proposed in the third aspect of the present disclosure uses different hands of the operator to control different action parts of the slave robot, and respectively control the movement of the vehicle body on the road and the action of the end of the multi-degree-of-freedom manipulator. Separate control of the left and right hands can make the slave robot execute the command action more accurately, and reduce the error rate of the slave robot.
  • a fourth aspect of the present disclosure provides a control method based on the above-mentioned mobile robot control system.
  • the method for controlling wide-range movement of a multi-degree-of-freedom manipulator on-board a robot includes the following steps:
  • Step 301 Set the engaging hand type and the corresponding gesture action.
  • the engaging hand type can be set to the end of the vehicle-mounted multi-degree-of-freedom manipulator waiting for the next command at the current position;
  • Step 302 Collect images within the shooting range of the wearable device of the operator;
  • Step 303 Determine whether there is a hand area in the collected image, if not, go to step 302; otherwise, preprocess the collected image to obtain a hand piece, and go to the next step;
  • Step 304 Use a hand shape recognition algorithm to recognize hand shape on the preprocessed hand piece to obtain hand shape information
  • Step 305 Determine whether the obtained hand shape information is a connecting hand type. If it is, the end of the vehicle-mounted multi-degree-of-freedom manipulator continuously executes the corresponding control instructions of the previous hand shape and the second hand shape of the connecting hand shape, and executes it Step 302; otherwise, go to the next step.
  • Step 306 Perform a corresponding action according to the corresponding hand shape, and perform step 302.
  • the control method of the fourth aspect of the present disclosure can realize incremental continuous and precise control of the end position of the reconnaissance system by setting the connecting hand type, and the control is more in line with human operating habits.
  • a wearable binocular camera device and a head-mounted virtual display are set on the operator's head, which can realize dual-view image collection, and the head-mounted virtual display setting can simultaneously realize the virtual model and the collected surveillance images , Can make the operator have the immersive feeling, can realize the intuitive control of the remote slave robot, and liberate the operator's hands through the setting of the wearable device, and reduce the operator's burden.
  • the control process is the pose of the operator's gesture—the pose of the virtual gesture model—the end pose of the virtual manipulator—the end pose of the multi-degree-of-freedom manipulator.
  • the control method proposed in the third aspect of the present disclosure uses different hands of the operator to control different action parts of the slave robot during control. Separate control of the left and right hands can make the slave robot perform command actions more accurately, and set different gesture types for the control of different parts of the slave robot, which are gesture recognition and motion track recognition, and again distinguish the control is movement
  • the robot body is also a vehicle-mounted multi-degree-of-freedom manipulator, which reduces the misoperation rate of the slave robot.
  • the operator's two hand movements are different types of movements, which avoids causing confusion for the operator.
  • the control logic is simple, easy to remember and easy to operate.
  • the control method proposed in the fourth aspect of the present disclosure changes the area within the operator’s ear-hook camera field of view into a virtual touch screen area by setting the articulation hand type, freeing the operator from the physical controller.
  • Dependence Compared with only using gesture types to achieve discrete control of robot actions, the present disclosure can achieve incremental continuous precise control of the end position of the reconnaissance system, and the control is more in line with human operating habits.
  • the present disclosure can achieve precise control of the detection direction of the mobile reconnaissance robot vehicle-mounted multi-degree-of-freedom manipulator.
  • FIG. 1 is a schematic diagram of virtual wear in Embodiment 2 of the present disclosure
  • FIG. 2 is a schematic diagram of virtual detachment of Embodiment 2 of the present disclosure
  • Figure 4 is a schematic structural diagram of a system according to one or more embodiments.
  • Fig. 5 is a block diagram of a wearable remote operation control device at the master end in Embodiment 1 of the present disclosure
  • FIG. 6 is a block diagram of the slave robot in Embodiment 1 of the present disclosure.
  • FIG. 7 is a flowchart of the method of Embodiment 3 of the present disclosure.
  • FIG. 8 is a flowchart of the method of Embodiment 4 of the present disclosure.
  • FIG. 9 is a schematic diagram of a gesture used by an operator to control the movement of the mobile robot body in Embodiment 4 of the present disclosure.
  • FIG. 10 is a schematic diagram of a gesture of the operator controlling the end of the vehicle-mounted multi-degree-of-freedom manipulator to turn left in Embodiment 4 of the present disclosure
  • FIG. 11 is a schematic diagram of a gesture of the operator controlling the end of the vehicle-mounted multi-degree-of-freedom manipulator to turn right in Embodiment 4 of the disclosure;
  • FIG. 12 is a schematic diagram of a gesture of an operator controlling the end of the vehicle-mounted multi-degree-of-freedom manipulator to move up in Embodiment 4 of the present disclosure
  • FIG. 13 is a schematic diagram of a gesture of an operator controlling the end of the vehicle-mounted multi-degree-of-freedom manipulator to move down in Embodiment 4 of the present disclosure
  • FIG. 14 is a schematic diagram of a gesture of the operator controlling the end of the vehicle-mounted multi-degree-of-freedom manipulator to tilt up in Embodiment 4 of the present disclosure
  • FIG. 15 is a schematic diagram of the gesture of the operator controlling the end of the vehicle-mounted multi-degree-of-freedom manipulator to bend down in Embodiment 4 of the present disclosure
  • N1 mobile robot body, N2, multi-degree-of-freedom manipulator, N3, surveillance camera, N4, video glasses, N5, binocular camera, N6, virtual manipulator;
  • main-end wearable remote operation control device 101, remote operation controller, 102, left wearable visual equipment, 103, right wearable visual equipment, 104, head-mounted virtual display, 105, wireless audio prompt equipment, 106.
  • Wireless data transmission equipment 107.
  • Wireless image transmission equipment 107.
  • Slave robot 201, vehicle controller, 202, mobile robot body, 203, linkage mechanism, 204, weapon device, 205, laser ranging sensor, 206, hand and eye monitoring camera, 207, reconnaissance camera, 208, laser Radar, 209, slave wireless data transmission equipment, 210, slave wireless image transmission equipment, 211, motor driver, 212, robotic arm driver, 213, car body drive motor unit, 214, robotic arm drive motor unit.
  • Robots can be divided into many categories according to different end effectors.
  • the end effector is fixed on the end of the robot arm to perform corresponding tasks.
  • End effectors such as dexterous hands and grippers, cameras, etc., detect the end effectors of the robot
  • this embodiment takes a surveillance robot as an example for description, but the continuous control method of the end pose of the robot in the present disclosure is not limited to the surveillance robot, but is applicable to the control of all robots.
  • a mobile robot control system includes a master-end wearable teleoperation control device and a slave-end robot.
  • the master-end wearable The teleoperation control device and the slave robot communicate wirelessly, and the master-end wearable teleoperation control device is worn on the operator and is used to send control instructions and receive data collected by the slave robot;
  • the main-end wearable teleoperation control device includes a wearable binocular camera device, a head-mounted virtual display, a teleoperation controller and a main-end wireless communication device.
  • the teleoperation controller is respectively connected with the wearable binocular camera device and the head-mounted virtual
  • the display is connected to the master-end wireless communication device, the wearable binocular camera device is used to collect the image of the operator's gesture, the head-mounted virtual display is used to display the image taken by the slave robot and the robot arm of the slave robot Virtual model and virtual model of operator gestures. Setting as a binocular camera device can realize the collection of dual-view images.
  • the remote operation controller may be a wearable computer that can collect in real time dual-view images of gestures taken by the wearable binocular camera device, and calculate the pose information of the operator's gesture based on the dual-view images of the gesture, and Gesture pose information displays a virtual gesture model in real time on the front end of the perspective view body of the video glasses;
  • the wearable binocular camera device may be a binocular camera N5, and the binocular camera N5 is used to collect dual-view images of the operator's gesture.
  • the operator uses the gesture pose within the field of view of the binocular camera N5 to control the end pose of the vehicle-mounted multi-free reconnaissance system.
  • the head-mounted virtual display can be video glasses N4, used to display the reconnaissance images taken from the end robot reconnaissance camera N3, and the virtual model of the multi-degree-of-freedom manipulator N2 and the virtual model of the operator’s gestures, where the reconnaissance image can be located
  • the rear end of the perspective view body of the video glasses, the virtual model of the multi-degree-of-freedom manipulator N2 and the virtual model of the operator's gestures are located at the front end of the perspective view body of the video glasses; this embodiment adopts the perspective view body display, which can Use other visual bodies.
  • the perspective view volume is the view volume through the perspective projection.
  • the perspective projection view volume is similar to a pyramid whose top and bottom are cut, that is, the prism. Its characteristics are: near large and far small.
  • the slave robot includes a mobile robot body N1, a multi-degree-of-freedom manipulator N2, a surveillance camera N3, a slave-end wireless communication device, and a vehicle-mounted controller, which is connected to the mobile robot body N1, a multi-degree-of-freedom manipulator N2, and a surveillance camera.
  • the camera N3 is connected to the slave wireless communication device.
  • the reconnaissance camera N3 is installed at the end of the multi-degree-of-freedom manipulator N2 for collecting reconnaissance data.
  • the mobile robot body N1 also includes a vehicle body drive motor unit and a motor driver, which are respectively connected to the vehicle controller and the drive motor unit.
  • the mobile robot body N1 receives the control of the wearable teleoperation control device of the master terminal through the vehicle-mounted controller to move the position.
  • the vehicle-mounted controller sends the control command to the motor driver, and the motor driver controls the corresponding motor of the driving motor group to realize the movement of the robot position from the end.
  • the multi-degree-of-freedom manipulator N2 receives the control of the main-end wearable teleoperation control device and executes corresponding actions.
  • the multi-degree-of-freedom manipulator N2 includes a linkage mechanism, a mechanical arm driver and a mechanical arm drive motor group.
  • the on-board controller sends the control command to the robot arm driver, and the robot arm driver drives the corresponding motor of the robot arm drive motor group to realize the movement of the link mechanism angle and position, thereby changing the joint angle information of each joint of the multi-degree-of-freedom robot arm N2 .
  • the virtual model of the robot arm of the slave robot is the virtual model of the multi-degree-of-freedom robot arm N2.
  • the virtual model of the multi-degree-of-freedom manipulator N2 may be a virtual manipulator N6 drawn according to the D-H parameters of the multi-degree-of-freedom manipulator N2.
  • the operator uses the gesture pose within the field of view of the binocular camera N5 to control the end pose of the vehicle-mounted multi-free reconnaissance system.
  • a robot remote control system based on a wearable device includes a master-end wearable teleoperation control device 100 and a slave-end robot 200 that are wirelessly connected, the master-end wearable teleoperation control
  • the device 100 is worn on the operator and is used to send control instructions and receive data collected from the robot 200;
  • the master-end wearable teleoperation control device 100 includes a wearable binocular camera device, a head-mounted virtual display 104, a master-end wireless communication device and a teleoperation controller 101, the wearable binocular camera device, a head-mounted virtual display 104 and the master-end wireless communication device are respectively connected to the remote operation controller; the wearable binocular camera device is worn on the operator’s head position to collect the operator’s actions, and the remote operation controller 101 generates control instructions according to the corresponding actions. Send to the slave robot 200.
  • the wearable binocular camera device includes a left wearable vision device 102 and a right wearable vision device 103, which are worn on the left and right sides of the operator's head, and can shoot The image in front of the operator is used to collect the motion information of the operator's hand.
  • the hand motion information may include position information and hand shape information of the hand in the image.
  • the left wearable vision device and the right wearable vision device may specifically be ear-hook cameras.
  • the head-mounted virtual display 104 can display pictures taken by the surveillance camera carried by the slave robot 200; the teleoperation controller receives the picture information taken by the slave robot 200, and controls the head-mounted virtual display 104 to display the pictures taken on-site.
  • the virtual display 104 may specifically be video glasses.
  • the master-end wireless communication device realizes wireless transmission through the wireless transmission module, which can be divided into a wireless data transmission device 106 for transmitting data and a wireless image transmission device 107 for transmitting image and video data, which realizes the master-end wearable remote operation control device
  • the information transmission between 100 and the slave robot 200 is specifically used to send control instructions to the slave robot 200, receive sensor data sent back from the slave robot 200, and receive image data sent back from the slave robot 200.
  • the wireless transmission module may include an image transmission station for transmitting image data and a data transmission station for transmitting control commands, such as a 5.8GHz wireless image transmission station and a 433MHz wireless data transmission station. If the remote control distance is short, a WIFI communication module can be used to realize image transmission and control command transmission at the same time.
  • the master-end wearable teleoperation control device 100 may also include a wireless audio prompt device 105, which is connected to the teleoperation controller 101 and is used to prompt the operator of the control instruction to be executed.
  • the slave robot 200 may be specifically a ground-armed reconnaissance robot for performing reconnaissance tasks, including a mobile robot body and a vehicle-mounted multi-degree-of-freedom manipulator.
  • the mobile robot car body may include a mobile robot body 202, a car body drive motor group 213, a motor driver 211, a surveillance camera 207, and a lidar 208, a wireless communication device from the end and a vehicle controller 201.
  • the slave-end wireless communication device includes a slave-end wireless data transmission device 209 and a slave-end wireless image transmission device 210 for storing and transmitting data and images, respectively. It can move under the control of the master-end wearable teleoperation control device 100, which is used to replace the operator to enter the dangerous area to perform combat tasks.
  • the motor driver 211, the vehicle body drive motor group 213, and the mobile robot body 202 are connected in sequence.
  • the motor driver 211 is used to control the vehicle body drive motor group 213 according to the control instructions sent by the master.
  • the vehicle body drive motor group 213 is connected to move The robot body 202 realizes the movement from the end robot 200.
  • the vehicle body driving motor group 213 includes at least a left motor and a right motor.
  • the left motor and the right motor can rotate in the same direction, and can control the robot to move forward and backward.
  • the left motor and the right motor can rotate in opposite directions, and can control the robot to turn left or right.
  • the lidar 208 is used to measure the obstacle information around the ground-armed reconnaissance robot at the slave end.
  • the lidar 208 is connected to the onboard controller 201.
  • the onboard controller 201 receives the measured obstacle information and transmits the obstacle information to the master terminal.
  • the remote operation controller 101 can display obstacle information on the head-mounted virtual display 104 of the master terminal.
  • the structure of the slave wireless communication device and the master wireless communication device can be the same, and the same wireless transmission module can be selected.
  • the reconnaissance camera 207 is used to photograph battlefield environment information and can be directly set on the vehicle body.
  • the reconnaissance camera 207 is connected to the vehicle controller 201 and is used to transmit the collected environmental images to the remote operation controller of the master terminal.
  • the vehicle-mounted multi-degree-of-freedom manipulator includes a link mechanism 203, a manipulator drive motor group 214, a manipulator driver 212, a laser ranging sensor 205, a hand-eye monitoring camera 206 and a weapon device 204.
  • the end of the link mechanism 203 is fixed with a hand-eye monitoring camera 206.
  • the link mechanism 203, the robotic arm drive motor group 214 and the robotic arm driver 212 are sequentially connected.
  • the link mechanism 203 is composed of at least two links.
  • the robotic arm driver 212 receives the control information sent by the master and controls it according to the control information.
  • the manipulator drives the motor group 214 to work, thereby driving the linkage mechanism 203 to move to the position where the operator wants to move, and the hand-eye monitoring camera 207 provided at the end of the linkage mechanism 203 captures the image information of the target of interest.
  • the laser ranging sensor 205 and the weapon device 204 are used for reconnaissance and strike missions with the robotic arm driver 212, and both can be set at the end of the linkage mechanism 203; the laser ranging sensor 205 is used to measure the distance information of hitting the target.
  • the settings of the surveillance camera 207 and the hand-eye monitoring camera 207 are used to collect different images.
  • the surveillance camera 207 collects environmental data, and realizes the collection of environmental images through the path by the movement of the slave robot 200.
  • the hand-eye surveillance camera 207 is used for control by the operator For image acquisition of key areas or regions of interest, the setting of two cameras realizes the image acquisition of the robot's work site without blind spots.
  • the vehicle-mounted controller 201 can control and collect the data of the laser radar 208, the laser ranging sensor 205, the reconnaissance camera 207 and the hand-eye monitoring camera 207 and send it wirelessly to the master remote operation device, and can also receive the master through the slave wireless communication device.
  • the motor driver 211 or the robot arm driver 212 controls the corresponding vehicle body driving motor group 213 or the robot arm driving motor group 214 according to the control instructions.
  • This embodiment provides a remote operation control method for the end pose of a robot based on the mobile robot control system described in Example 1, as shown in Figures 1 to 3, specifically the end pose remote operation control of a multi-degree-of-freedom manipulator
  • the method can realize continuous control of the position and posture of the end of the robotic arm through the movement of gestures, including the following steps:
  • Step 101 Set the traction hand type and the release hand type
  • the traction hand type means that when the operator is detected to be the hand type, the pose of the virtual gesture model is kept coincident with the end pose of the virtual manipulator in the video glasses, and the operator can drive the video glasses through the pose of the gesture
  • the position and posture (ie pose) of the virtual gesture model in N4 the virtual gesture model can perform real-time continuous control of the end pose of the virtual manipulator N6.
  • the virtual gesture model no longer follows the operator's gesture movement, and the operator's gesture cannot perform real-time continuous control of the virtual manipulator N6.
  • the traction hand type and the release hand type can be any hand type, and can be set according to the needs.
  • the traction hand type can be a hand type representing a Cartesian coordinate system.
  • the ring finger and little finger of the hand type are in a curved state.
  • the thumb, index finger, and middle finger are in a straight state, and the three fingers are perpendicular to each other to form a Cartesian coordinate system;
  • the detachable hand can be a one-handed fist hand.
  • steps of initializing and establishing a wireless connection may also be included:
  • Step 102 Construct a virtual robotic arm and a virtual gesture model and display them on the front end of the visual body of the head-mounted virtual display;
  • step 102 the method of constructing a virtual mechanical arm and displaying it on the front end of the visual body of the head-mounted virtual display is specifically as follows:
  • the action of the multi-degree-of-freedom manipulator is controlled by the on-board controller.
  • the manipulator driver drives the corresponding motor of the manipulator drive motor group to realize the movement of the angle and position of the linkage mechanism, thereby changing the joints of each joint of the multi-degree-of-freedom manipulator N2 Angle information.
  • the joint angle information of each joint of the multi-degree-of-freedom manipulator can be directly read by the on-board controller.
  • the teleoperation controller calculates the D-H parameters of the multi-degree-of-freedom manipulator according to the collected joint angle information
  • the angle of each joint of the virtual manipulator N6 is controlled by the received joint angle information, the base coordinate system of the virtual manipulator N6 is described by the screen coordinate system of the video glasses N4, and the end coordinate system of the virtual manipulator N6 is denoted as (O M -X M -Y M -Z M) , a virtual terminal end of the robot arm N6 posture denoted by P M, including position information and posture information;
  • the construction method of the virtual gesture model can be specifically as follows:
  • the surveillance environment information of the slave robot can also be displayed in the video glasses N4.
  • the surveillance images collected by the surveillance camera N3 can be displayed on the vision of the video glasses N4
  • the body may also include the step of displaying the image taken by the slave robot on the head-mounted virtual display, which is specifically as follows: collect the reconnaissance image from the slave robot; the teleoperation controller receives the reconnaissance image and displays it in the head-mounted virtual display in real time. The rear end of the viewing body.
  • Step 103 Collect dual-view images of the binocular camera N5; collect the hand shape information of the operator through the binocular camera N5.
  • the dual-view image includes images of left and right views.
  • Step 104 Use a gesture detection algorithm to detect and determine whether there is an operator's gesture in the dual-view image. If yes, proceed to the next step; otherwise, proceed to step 103; as long as the operator's gesture appears in the dual-view image, then Go to step 105.
  • the gesture detection algorithm may specifically be a gesture detection algorithm based on a skin color threshold.
  • Step 105 Use a hand type recognition algorithm to perform hand type recognition on the gesture, and determine whether there is a traction hand type, if yes, go to the next step, otherwise go to step 103; the hand type recognition algorithm is specifically a hand type recognition algorithm based on deep learning.
  • the multi-degree-of-freedom manipulator N2 When the traction hand shape is detected in the dual dual-view images, the multi-degree-of-freedom manipulator N2 must be controlled by the operator's hand traction. If there is no traction hand shape, perform step 3 again to collect the operator's hand shape information through the binocular camera N5.
  • Step 106 Process the captured dual-view image and calculate the pose P H of the traction gesture in the coordinate system of the wearable binocular camera device, and convert the pose P H to the position in the screen coordinate system of the head-mounted virtual display.
  • Pose description P V which uses the transformed pose P V to drive the virtual gesture model in the visual body of the head-mounted virtual display;
  • the DeepPrior++ algorithm can be used to solve the pose P H of the traction gesture in the coordinate system of the wearable binocular camera device.
  • the DeepPrior++ algorithm can realize the estimation of the gesture pose under stereo vision.
  • Solving the pose P H of the traction gesture in the coordinate system of the wearable binocular camera device can also adopt the following steps:
  • the traction gesture P H includes position information and posture information.
  • the solution of the position information is directly realized by using the gesture detection results in the left and right views and the parallax principle;
  • the posture information of the traction gesture P H is realized using a method based on regression learning:
  • the posture information of the traction gesture P H can be implemented using the method based on regression learning as follows:
  • the hand-held three-axis attitude sensor can be used to rotate around the three axes of the three-axis attitude sensor in front of the dual-view camera and collect the dual-view gesture detection result image corresponding to each output data of the attitude sensor.
  • Two frames of gesture images and one frame of gesture data acquired at the same time are used as input samples and output samples, respectively.
  • the collected dual-view gesture images and corresponding posture data are used as input sample training set and output sample set respectively.
  • the posture information of the traction gesture can be solved directly through the dual-view gesture image.
  • Step 106 may firstly establish the correspondence between the operator's traction gesture and the virtual gesture model, and convert the pose P H into the pose P V through the correspondence.
  • the specific corresponding relationship can be a proportional relationship.
  • the position information of the pose P H of the operator's traction gesture in the coordinate system of the wearable binocular camera device and the position information of the pose P V are in a proportional relationship.
  • the pose of the pose P H The information is also proportional to the posture information of the pose P V.
  • the pose P H of the pulling gesture is described in the coordinate system of the binocular camera N5.
  • the palm of the pulling gesture can be specified as the origin, and the origin coordinate system of the palm of the pulling gesture is (O H -X H -Y H- Z H ), the direction pointed by the middle finger of the traction gesture is the X-axis direction, the direction pointed by the thumb is the Y-axis direction, and the direction pointed by the middle finger is the Z-axis direction.
  • the position information of the pose P H is determined by the origin of the palm of the traction gesture The offset description of the O H relative to the origin of the N5 coordinate system of the binocular camera.
  • the posture information of the pose P H is based on the rotation of the X H axis, Y H axis and Z H axis of the coordinate system of the traction gesture to each axis of the Binocular camera N5 coordinate system description.
  • the pose P V of the virtual traction gesture is described in the screen coordinate system of the video glasses N4, and the palm of the virtual traction gesture can be specified as the origin, and the origin coordinate system of the palm of the virtual traction gesture is denoted as ( OV ⁇ X V -Y V -Z V ), the direction of the middle finger of the virtual traction gesture is the X-axis direction, the direction of the thumb is the Y-axis direction, the direction of the middle finger is the Z-axis direction, and the position information of the pose P V It is described by the offset of the origin O V at the palm of the virtual traction gesture relative to the origin of the screen coordinate system of the video glasses N4.
  • the posture information of the pose P V is determined by the coordinate system X V axis, Y V axis and Z V axis of the virtual traction gesture.
  • the pose transformation using P V driver model wearing a virtual display gesture virtual view volume of the virtual model of the gesture begins to follow the gesture of moving the mobile operator.
  • the driving method specifically includes: a gesture-dimensional virtual model wearing a virtual display is loaded into the location information which is required for real-time rendering the view volume is directly assigned by the position information of the position and orientation of P V, which visual
  • the pose information required for real-time rendering in the volume is directly assigned by the pose information of the pose P V.
  • Step 107 Judge whether the difference between the pose P V of the virtual gesture model and the end pose P V of the virtual robot arm N6 is less than a preset threshold, if yes, proceed to the next step; otherwise, proceed to step 3;
  • Step 107 is the process of realizing the wearing. Specifically, the distance between the pose P V of the virtual gesture model and the end pose P M of the virtual manipulator N6 is quickly approached. Through step 6, the operator moves the traction gesture, so that the virtual gesture model also moves until the pose P V of the virtual gesture model approaches the end pose P M of the virtual robot arm N6.
  • Step 107 is a specific implementation process:
  • the operator observes a relative relationship between a perspective view volume of the virtual video glasses N4 traction gesture pose P V virtual terminal N6 manipulator pose P M, by constantly moving the traction bits gesture P H is used to make the difference between the pose P V of the virtual traction gesture in the perspective view volume of the video glasses N4 and the end pose P M of the virtual manipulator N6 continuously decrease, and the difference between the two poses is described by the following formula :
  • the image at this time can be considered a virtual machine
  • the teleoperation controller is implemented by executing steps 103-107 multiple times.
  • the wearing process is completed, the multi-degree-of-freedom manipulator N2 can be pulled.
  • Step 108 Make the pose of the multi-degree-of-freedom manipulator follow the change of the operator's traction hand pose
  • step 108 The steps of step 108 are specifically:
  • the corresponding joint angle values are converted into control instructions and transmitted to the slave robot, so that the joint angles of the joints of the multi-degree-of-freedom manipulator are equal to the joint angles of the virtual manipulator.
  • the teleoperation controller converts the joint angles of the virtual manipulator N6 into control instructions and sends them to the slave robot N1 through the wireless communication channel.
  • the joint angles of N6 are the same; so that the pose of the multi-degree-of-freedom manipulator N2 follows the change of the gesture pose of the operator.
  • the position of the virtual robot arm can be adjusted to the posture, and the position and posture change of the virtual robot arm N6 can be displayed in the video glasses N4 in real time.
  • the step 108 further includes: redrawing the virtual robot arm N6 in the viewing volume according to the calculated joint angle values of the virtual robot arm N6.
  • the joint angle values of the virtual manipulator N6 are redrawn in the perspective view of the video glasses N4, so that the end pose of the virtual manipulator N6 is always the same as the virtual traction gesture. The end pose remains the same.
  • Step 109 It is judged whether there is a detachable hand type. If it is, the position of the multi-degree-of-freedom manipulator stops following the change of the operator's pulling hand position, and step 103 is executed; otherwise, step 108 is executed.
  • the detachment gesture can be set to a left-hand fist state. If the operator's gesture becomes a detachment gesture, the end pose of the virtual robotic arm N6 is no longer controlled by the operator. It is visually believed that at this time the end of the virtual robotic arm N6 has virtually detached from the operator's gesture.
  • the releasing gesture can be any gesture, one-handed gesture, or two-handed gesture. At this point, the traction process is over, and other commands can be completed or executed.
  • This embodiment provides a control method based on the robot control system described in embodiment 1, which separately collects the actions of the operator’s left and right hands, and controls the movement of the mobile robot body through the actions of one hand.
  • the motion controls the motion of the on-board multi-degree-of-freedom manipulator arm of the mobile robot.
  • the remote operation controller can collect the images taken by the wearable binocular camera device and analyze whether there are the operator's left and right hands and their hand type and position coordinates in the image. When detecting the operator's left and right hands, it can be based on the type and position of the hand
  • the coordinates send corresponding control commands to the slave robot 200 through the wireless communication device to control the movement of the slave robot 200 and the movement of the vehicle-mounted multi-degree-of-freedom manipulator.
  • the name of the control command can also be prompted by wireless audio before the control command is issued
  • the device feeds back to the operator. In addition, it can also process sensor data and monitoring images sent back from the end robot 200 received through the wireless communication device, and display them on the head-mounted virtual display 104.
  • Step 201 Collect images within the shooting range of the wearable device of the operator;
  • the wearable binocular camera device can specifically be a wearable camera, which is set on the head of the manipulator to collect images around the manipulator.
  • the manipulator needs to place the corresponding hand on the controller according to the control to be performed. Do the corresponding actions within the camera range.
  • the left and right cameras can be set up, and the left and right cameras collect the left image and the right image respectively.
  • the image stitching method can be used to cut off the overlapping parts of the two images and stitch them into a wide field of view image. Image.
  • Step 202 Determine whether there is a hand area in the collected image, if not, go to step 201; otherwise, perform preprocessing on the collected image to obtain a hand piece;
  • the method of judging whether there is a hand in the collected image can use a gesture detection algorithm, and the gesture detection algorithm can specifically use a gesture detection algorithm based on skin color.
  • the specific method of preprocessing the collected image to obtain the hand piece is: if the presence of the hand is detected, the gesture segmentation algorithm is used to eliminate the background in the area containing the hand, and the scale normalization is further used to include the hand.
  • the image of the part is normalized to the same size of the hand piece.
  • Step 203 Determine whether the obtained hand piece is a left-hand piece or a right-hand piece by using the left-hand and right-hand discrimination algorithm, so as to determine whether the movement is the left hand or the right hand;
  • the method for judging whether it is left-handed or right-handed by the left-handed discrimination algorithm can be specifically as follows:
  • the left-handed discrimination algorithm judges that it is a binary classification problem.
  • a classifier such as convolution Neural network
  • Step 204 Control the movement of the vehicle body of the mobile robot by the movement of one of the hands, and control the movement of the on-board multi-degree-of-freedom manipulator of the mobile robot by the movement of the other hand, and then perform step 201.
  • gestures and finger movement trajectories can be used to control which part of the slave robot 200 moves.
  • This embodiment is set to control the movement of the mobile robot body through gestures, and the finger movement trajectory is set. Control the movement of the vehicle-mounted multi-degree-of-freedom robotic arm.
  • step 204 one of the hands is used to control the movement of the mobile robot body.
  • the specific steps are as follows:
  • Step 2041 Set the correspondence between the motion control instructions of the slave robot 200 and the hand shape information;
  • the hand shape information is the gesture information made by the operator, which may include fists, scissors hands, OK gestures, etc., and the motion control instructions include forward, Go back, turn left, turn right, turn around.
  • the specific correspondence can be set according to specific needs. Generate the corresponding correspondence table.
  • Step 2042 When the recognized hand piece is a hand set to control the movement of the mobile robot body, use hand shape recognition to calculate the hand piece for recognition to obtain hand shape information;
  • different hands of the operator are used to control different action parts of the slave robot 200. Separate control of the left and right hands can make the slave robot 200 perform command actions more accurately. Firstly, it is judged whether it is the left hand or the right hand.
  • the hand distinction is used to distinguish whether to control the mobile robot body or the vehicle-mounted multi-degree-of-freedom manipulator.
  • the recognition of hand gestures and the recognition of motion trajectories again distinguish between the control of the mobile robot body and the vehicle-mounted multi-degree-of-freedom manipulator, which reduces the misoperation rate.
  • the operator's two hand movements are different types of movements, which avoids causing confusion for the operator.
  • the control logic is simple, easy to remember and easy to operate.
  • any one of the hands can be set to control the movement of the mobile robot body, and in this embodiment, the left hand can be selected.
  • the left-hand control of the movement of the mobile robot body is set, the movement of the vehicle-mounted multi-degree-of-freedom manipulator is controlled by the right hand.
  • gestures and finger movement tracks can be used to control which part of the slave robot 200 moves can be set. In this embodiment, gesture control is set.
  • Step 2043 Generate a motion control instruction of the slave robot 200 according to the corresponding relationship between the motion control instruction of the slave robot 200 and the hand shape information and the hand shape information obtained by recognition, and send the motion control instruction to the slave robot 200, and the slave robot 200 according to The control instruction executes the corresponding action.
  • Step 2043 also includes the following steps: setting the motion name corresponding to the motion control instruction, and after generating the motion control instruction of the slave robot 200, sending the motion name corresponding to the motion control instruction to the wireless audio prompt device, and the wireless audio prompt device broadcasts Actions to be performed by the slave robot 200.
  • the controller can determine whether the action to be performed is correct according to the broadcast.
  • step 204 the movement of the on-board multi-degree-of-freedom manipulator of the mobile robot is controlled by the movement of the other hand.
  • the specific steps are:
  • Step 204-1 When the recognized hand piece is a hand set to control the movement of the on-board multi-degree-of-freedom manipulator of the mobile robot, use a fingertip positioning algorithm to analyze the motion trajectory of any fingertip in the image;
  • Step 204-2 Generate a position tracking instruction according to the motion trajectory, and send the position tracking instruction to the slave robot 200;
  • Step 204-3 The slave robot 200 generates the position coordinates of a specific action according to the position tracking instruction, and the end of the link mechanism 203 sequentially passes through the position coordinates to track the motion trajectory of the operator's fingertip.
  • the fingertip positioning algorithm is used to analyze the trajectory of any fingertip in the image, and the fingertip positioning algorithm based on contour curvature and the fingertip positioning algorithm based on convex hull analysis can be used.
  • the position coordinates can be set with the base of the link mechanism 203 as the origin.
  • This embodiment proposes another control method based on the wearable device-based robot remote control system described in Embodiment 1.
  • the method differs from the method in Embodiment 3 in that there is no need to distinguish between left and right hands for control.
  • This embodiment uses different settings
  • the gesture controls the movement of the mobile robot body and the on-board multi-degree-of-freedom manipulator control. It is possible to implement actions outside the imaging range of the wearable binocular camera device of the operator.
  • the vehicle-mounted multi-degree-of-freedom manipulator when the vehicle-mounted multi-degree-of-freedom manipulator is controlled by recognizing the movement trajectory of the finger, the movement trajectory of the operator's hand needs to be completely within the display camera range, and the movement range of the operator needs to be in the wearable binocular camera Within the camera range of the device, the motion range of the vehicle-mounted multi-degree-of-freedom manipulator is restricted.
  • This embodiment can realize the movement of the vehicle-mounted multi-degree-of-freedom manipulator in a wider range than that of the third embodiment.
  • the mobile robot will keep moving forward until the operator’s gesture becomes a stop hand pattern, then the mobile robot stops moving forward. That is to say, when the mobile robot car body is moving forward, backward, turning left and turning right, it uses the stop hand to stop it.
  • the movement of the robot body is a keeping movement, until a stop signal appears, otherwise it continues to move.
  • the stop hand type may not be set.
  • the pitch angle, upward movement distance, downward movement distance, left movement distance and right movement distance of the end of the robotic arm are realized by following the pitch hand type or the end traction hand type. , Once the pitching hand type or the end pulling hand type changes to the engaging hand type, the gesture exceeds the camera range, etc., the end of the robotic arm will naturally stop. Don't add a stop hand to control the end of the robotic arm.
  • a method for controlling wide-range movement of a multi-degree-of-freedom manipulator on-board robot which includes the following steps:
  • Step 301 Set the engaging hand type and the corresponding gesture action.
  • the engaging hand type can be set to the end of the vehicle-mounted multi-degree-of-freedom manipulator waiting for the next command at the current position ;
  • Step 302 Collect images within the shooting range of the wearable device of the operator;
  • Step 303 Determine whether there is a hand area in the collected image, if not, go to step 302; otherwise, preprocess the collected image to obtain a hand piece, and go to the next step;
  • Step 304 Use a hand shape recognition algorithm to recognize hand shape on the preprocessed hand piece to obtain hand shape information
  • Step 305 Determine whether the obtained hand shape information is a connecting hand type. If it is, the end of the vehicle-mounted multi-degree-of-freedom manipulator continuously executes the corresponding control instructions of the previous hand shape and the second hand shape of the connecting hand shape, and executes it Step 302; otherwise, go to the next step.
  • Step 306 Perform a corresponding action according to the corresponding hand shape, and perform step 302.
  • step 301 the following steps are also included:
  • the teleoperation controller and the slave robot 200 perform initialization operations.
  • 3002 Establish a wireless communication channel between the teleoperation controller and the slave robot 200.
  • the slave robot 200 collects the reconnaissance image of the robot camera, and then sends it to the teleoperation controller through the wireless communication channel.
  • the teleoperation controller receives the reconnaissance image through the wireless communication device and displays the reconnaissance image in the video worn by the operator in real time. Glasses on.
  • different hand shapes are set corresponding to different actions of the slave robot 200, and different hand shapes are set corresponding to the corresponding actions of the slave robot 200 for control.
  • the operator can set according to needs.
  • the actions of the slave robot 200 include the end action of the vehicle-mounted multi-degree-of-freedom manipulator arm and the movement of the mobile robot body.
  • the movement of the mobile robot body movement includes stopping, forwarding, retreating, turning left, turning right, etc., this embodiment sets The corresponding relationship between the corresponding hand type and its corresponding gestures and control instructions can be shown in Figure 9.
  • the empty-hand type H1 corresponds to no control instructions, and the slave robot 200 is stationary; the forward hand type H2 corresponds to the forward instruction, and the mobile robot car
  • the body start motor driver 211 moves forward; in the same way, the left hand type H3, the right hand type H4, the back hand type H5, and the stop hand type H6 correspond to the left, right, back, and stop actions of the mobile robot body respectively.
  • a corresponding correspondence table can be established, and the operator can change the correspondence table between gestures and control instructions according to his own habits.
  • step 303 and step 304 may be the same as the method described in Embodiment 2.
  • step 305 the end of the on-vehicle multi-degree-of-freedom manipulator continuously executes the corresponding control instructions of the previous hand type of the engaging hand type and the subsequent hand type of the engaging hand type specifically as follows:
  • the end of the vehicle-mounted multi-degree-of-freedom manipulator will stop at the current position after performing the action corresponding to the previous hand type
  • Step 302 to step 304 are executed, the next hand shape of the engaging hand is detected, and the end of the on-board multi-degree-of-freedom manipulator moves from the current position to perform the action corresponding to the next hand of the engaging hand.
  • the empty-handed H1 is set as an example of the connection gesture.
  • the movement of the end traction hand H8 combined with this hand corresponds to the upward, downward, left, and right movement of the end of the vehicle-mounted multi-degree-of-freedom manipulator.
  • the device 201 sequentially sends the control instructions for moving the end of the multi-degree-of-freedom manipulator arm up to a distance of K2*U1, stopping at the current position, and moving up from the current position by a distance of K2*U1.
  • K2 is the displacement coefficient, which is used to adjust the proportional relationship between the vertical movement distance of the end tractor H8 and the vertical movement distance of the end position of the multi-degree-of-freedom manipulator.
  • L1 is greater than the preset threshold, then the operator poses the empty-handed H1 and moves the empty-handed L1 back to control In the field of view of the ear-hook camera, then once again put out the end pulling hand H8 and move it to the left again L2, L1 and L2 are the moving distances, then the operator’s end pulling hand H8 is in the operator’s ear-hook camera
  • the leftward movement angle of the multi-degree-of-freedom robotic arm is
  • the teleoperation controller sends to the vehicle controller 201 the deflection angle of the end of the multi-degree-of-freedom manipulator arm Stop at the current position, deflection angle to the left from the current position Distance control instructions.
  • K1 is the deflection coefficient, which is used to adjust the ratio between the left-right movement distance of the end tractor H8 and the left-right deflection angle of the vehicle-mounted multi-degree-of-freedom manipulator.
  • This embodiment proposes a method for controlling wide-range movement of a multi-degree-of-freedom manipulator arm on a robot vehicle.
  • the area within the field of view of the operator’s ear-hook camera is turned into a virtual touch screen area, which frees the operator from physical control.
  • the present invention can achieve incremental continuous precise control of the end position of the reconnaissance system, and the control is more in line with human operating habits.
  • corresponding actions are performed according to the corresponding hand shape.
  • the hand shape can be set according to personal habits or agreed hand shape, and the corresponding relationship between the hand shape and the corresponding action is set.
  • the actions mainly include forward, backward, left turn, right turn and stop. This embodiment can be specifically as follows:
  • the teleoperation controller does not issue a control instruction to the slave robot 200, and then continues to perform step 302;
  • the teleoperation controller sends a stop control command through the wireless communication device to stop the mobile reconnaissance robot from moving, and then execute step 302;
  • the actions of the multi-degree-of-freedom manipulator of the slave robot mainly include deflection at a certain angle and movement up and down, left and right, and this embodiment can be specifically as follows:
  • the teleoperation controller sends out the pitch angle of the end-of-vehicle multi-degree-of-freedom manipulator through the wireless communication device and the pitch angle of the pitch-hand H7 Maintain consistent control instructions until the operator poses other hand shapes, and then go to step 302; this invention can realize the detection direction of the end of the multi-degree-of-freedom manipulator by measuring the rotation angle of a specific gesture type such as the pitch hand H7 in the image ( Relative to the precise control of the horizontal plane, the reconnaissance direction of the camera is equal to the pitch angle of the pitch hand.
  • the hand type of the operator is the end traction hand type H8, as shown in Figure 10, Figure 11, Figure 12 and Figure 13, the displacement distance and displacement direction of the hand shape will be further detected.
  • the preset threshold is exceeded, when the end-towing hand H8 moves left and right, the teleoperation controller sends out through the wireless communication device to enable the end of the multi-degree-of-freedom manipulator to deflect to the left or right, respectively or
  • the control command, K1 deflection coefficient is used to adjust the proportional relationship between the left and right movement distance of the end traction hand H8 and the left and right deflection angle of the vehicle-mounted multi-degree-of-freedom manipulator.
  • L and R are the leftward movement distance of the end traction hand H8 and Moving distance to the right, r is the radius of rotation of the end of the vehicle-mounted multi-degree-of-freedom manipulator around its base.
  • the remote operation controller sends a control through the wireless communication device that can make the end position of the multi-degree-of-freedom manipulator move up or down by K2*U or K2*D respectively.
  • K2 is the displacement coefficient, which is used to adjust the proportional relationship between the up and down movement distance of the end drag hand H8 and the up and down movement distance of the end position of the multi-degree-of-freedom manipulator.
  • U and D are the up and right movement of the end drag H8 respectively Moving distance
  • the teleoperation controller sends a control command through the wireless communication device to stop the vehicle-mounted multi-degree-of-freedom reconnaissance system from moving up and down, and step 302 is executed; when the end-towing hand H8 When it becomes a connecting hand type, step 305 is executed.
  • the hand shape in this embodiment is only an example hand shape, and the specific hand shape can be set according to needs.
  • An electronic device includes a memory, a processor, and computer instructions stored on the memory and running on the processor. When the computer instructions are executed by the processor, the steps described in the method in Embodiment 2, 3, or 4 are completed.
  • a computer-readable storage medium for storing computer instructions, which when executed by a processor, complete the steps described in the method in Embodiment 2, 3, or 4.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Multimedia (AREA)
  • Automation & Control Theory (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Manipulator (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

本公开提出了基于可穿戴设备的移动机器人控制系统及控制方法，第一种控制方法流程为操作员手势的位姿—虚拟手势模型的位姿—虚拟机械臂末端位姿—多自由度机械臂末端位姿，通过建立操作者手势位姿和多自由度机械臂末端位姿的驱动关系，实现对多自由度机械臂末端位姿的连续控制。第二种方法在进行控制时采用操控者不同的手控制从端机器人的不同动作部位，通过左右手分别控制使得从端机器人执行命令动作的准确度更高；第三种方法通过设置衔接手型，使操控员摆脱了对物理控制器的依赖；相比于仅使用手势类型实现对机器人动作的离散控制，实现对侦察系统末端位置的增量式连续精确控制。

Description

基于可穿戴设备的移动机器人控制系统及控制方法 技术领域
本公开涉及于移动机器人的远程控制相关技术领域，具体的说，是涉及基于可穿戴设备的移动机器人控制系统及控制方法。
背景技术
本部分的陈述仅仅是提供了与本公开相关的背景技术信息，并不必然构成在先技术。
移动侦察机器人通常是由移动机器人车体和车载侦察系统组成，它可以执行战场抵近侦察监视、潜行突袭、定点清剿、核生化处理及反恐排爆等多种作战任务。传统的车载侦察系统一般是由摄像头和两自由度的云台组成，其控制方式一般通过摇杆的俯仰角和偏航角的角度信息实现对云台的俯仰控制。对于搭载多自由度侦察系统的移动侦察机器人，其侦察系统一般是由多自由度机械臂和侦察摄像头组成，其中侦察摄像头固定连接在多自由度机械臂末端。机器人末端位姿是指机器人末端执行器在指定坐标系中的位置和姿态，移动侦察机器人的末端执行器为摄像头，侦察机器人末端位姿由多自由度机械臂的末端位姿决定，而多自由度机械臂的末端位姿控制通常使用按钮或摇杆结合按钮的控制方式，操作者需要记忆每个按钮与车载多自由机械臂每个关节的对应关系，因此这种操作方式复杂度很高且不直观。
近些年来，出现了使用手势控制车载多自由侦察系统末端位姿的方式。一种较为常见的手势控制方式是使用佩戴数据手套或者惯性元件，其优点是识别率高，稳定性好，缺点是不能实现对车载多自由度侦察系统末端位置的控制只能实现对姿态的控制并且输入设备昂贵，穿戴很不方便。另一种手势控制方式是基于视觉的控制方式，这种控制方式又可分为基于图像分类的控制方式和基于图像处理的控制方式，前者一般是通过视觉传感器结合模式识别方法分析手势的种类，然后根据手势的类型信息实现对车载多自由度侦察系统末端位姿的运动控制，如上移、下移等，其缺点是无法快速准确地实现对车载多自由度侦察系统末端位姿的连续控制；后者一般是通过视觉传感器结合图像处理方法分析手势的运动轨迹，然后根据轨迹的位置信息实现对车载多自由度侦察系统末端的位置控制，其缺点是无法实现对车载多自由度侦察系统末端姿态的控制。
另外，传统的移动机器人的远程控制系统通常采用带有摇杆和按键的控制箱或控制盒来实现。对于搭载多自由度机械臂的移动机器人，控制箱的按键更为繁杂，操控者需要记忆每个按键与移动机器人及车载机械臂的对应关系，控制方式非常不直观。移动机器人及车载侦察系统都摆脱不了对摇杆的依赖，而摇杆又需要控制箱及相关的硬件设备的支持，因此传统的移动侦察机器人的控制器的体积都普遍较大，由此带来的问题是不方便携带和运输。
手势是人类沟通手段中较为自然的一种，尤其是对于特种兵来说，手语是与队友沟通和传达指令的必需手段。尤其是对于不方便采用语音进行沟通时，手势几乎是特种兵之间沟通和传达指令的唯一手段。目前基于人体手势的人机交互远程控制主要采用佩戴数据手套或者惯性元件的方式，其优点是识别率高，稳定性好，缺点是输入设备昂贵，穿戴很不方便。因 此，对于全副武装的士兵来说，如何提高地面武装侦察机器人的人机交互遥操作控制系统便携性和操控直观性是非常迫切的需求。
近年来也出现了采用触屏平板电脑的控制方式，这种控制方式通过采用不同的触屏手势代替了摇杆和按钮，虽然这很大程度上减小了控制器体积，但是仍然会占用操控者的双手。尤其是对于战场上全副武装的士兵来说，带有一定重量和一定体积的实体控制器是一种不小的负担，并且不利于全副武装的士兵在战斗状态(手持武器)和操控状态(手持控制器)之间的快速转换。
发明内容
本公开为了解决上述问题，提出了基于可穿戴设备的移动机器人控制系统及控制方法，针对搭载多自由度机械臂的移动机器人提供了基于可穿戴设备的移动机器人控制系统及控制方法，通过在自由穿戴和脱卸实现对多自由度机械臂末端位置和姿态的连续控制，以解决现有移动侦察机器人车载多自由度侦察系统控制中存在控制方式复杂以及不能直观地控制车载多自由度侦察系统末端位姿的问题。
为了实现上述目的，本公开采用如下技术方案：
本公开的第一方面提供了一种基于可穿戴设备的移动机器人控制系统，包括主端可穿戴遥操作控制装置和从端机器人，所述主端可穿戴遥操作控制装置和从端机器人通过无线通信，所述主端可穿戴遥操作控制装置穿戴在操作员身上，用于发送控制指令和接收从端机器人采集的数据；
主端可穿戴遥操作控制装置包括可穿戴双目摄像装置、头戴虚拟显示器、遥操作控制器和主端无线通信设备，所述遥操作控制器分别与可穿戴双目摄像装置、头戴虚拟显示器和主端无线通信设备连接，可穿戴双目摄像装置用于采集操作员手势的图像，所述头戴虚拟显示器用于显示从端机器人拍摄的图像以及用于显示从端机器人的机械臂的虚拟模型和操作员手势的虚拟模型。
在操作者的头部设置可穿戴双目摄像装置和头戴虚拟显示器，可以实现双视角图像的采集，头戴虚拟显示器设置可以同时实现虚拟模型和采集的侦查图像，能够使得操作者有身临其境的感觉，能实现远程从端机器人的直观控制，通过可穿戴装置的设置解放了操作员的双手，减轻了操作员的负担。
本公开的第二方面提供了基于上述的一种移动机器人控制系统的机器人末端位姿的遥操作控制方法，包括如下步骤：
步骤101、设置牵引手型和脱卸手型；
步骤102、构建虚拟机械臂和虚拟手势模型，并将拟机械臂和虚拟手势模型显示在头戴虚拟显示器视景体的前端；
步骤103、采集双目摄像头的双视角图像；
步骤104、采用手势检测算法检测，判断双视角图像中是否有操作员的手势存在，如果是，则执行步骤105，否则执行步骤103；
步骤105、采用手型识别算法对手势进行手型识别，判断是否出现了牵引手型，如果是，执行步骤106，否则执行步骤103；
步骤106、对拍摄的双视角图像进行处理并求解牵引手势在可穿戴双目摄像装置坐标系中的位姿P H，将位姿P H转换为在头戴虚拟显示器的屏幕坐标系中的位姿描述P V，采用转化后的位姿P V驱动头戴虚拟显示器视景体中的虚拟手势模型；
步骤107、判断虚拟手势模型的位姿P V与虚拟机械臂N6末端位姿P_M的差是否小于预设阈值，如果是，执行步骤108，否则执行步骤103；
步骤108、使得多自由度机械臂的位姿跟随操作员的牵引手型位姿变化；
步骤109、判断是否出现脱卸手型，如果是，多自由度机械臂的位姿停止跟随操作员的牵引手型位姿变化，并执行步骤103；否则，执行步骤108。
本公开第二方面设置的机器人末端位姿的遥操作控制方法，当检测到牵引手型，通过建立操作者手势位姿和多自由度机械臂末端位姿的驱动关系，实现对多自由度机械臂末端位姿的连续控制；当检测到脱卸手型，进行脱卸使得多自由度机械臂的位姿停止跟随操作员的牵引手型位姿变化。同时在头戴虚拟显示器中显示虚拟机械臂末端和虚拟手势模型的跟随过程和脱卸过程，使得控制过程更直观。通过设置对应的手势启动和停止对从端机器人的多自由度机械臂的控制，控制方法简单而且可靠。
本公开第三方面提供了基于上述一种移动机器人控制系统的控制方法，分别采集操控者的左手和右手的动作，通过一只手的动作控制移动机器人车体的移动，通过另一只手的动作控制移动机器人的车载多自由度机械臂的动作，包括如下步骤：
步骤201：采集操控者可穿戴设备可拍摄范围内的图像；
步骤202：判断采集的图像中是否有手部区域，如果没有，执行步骤201；否则，对采集的图像进行预处理，获得手部裁片；
步骤203：利用左右手判别算法判断获得的手部裁片是左手裁片还是右手裁片，从而确定做动作的是左手还是右手；
步骤204：通过其中一只手的动作控制移动机器人车体的移动，通过另一只手的动作控制移动机器人的车载多自由度机械臂的动作，然后，执行步骤201。
本公开第三方面提出的控制方法在进行控制时采用操控者不同的手控制从端机器人的不同动作部位,分别控制车体在路面上的移动和多自由度机械臂末端的动作。通过左右手分别控制可以使得从端机器人执行命令动作的准确度更高，减小了从端机器人的误动作率。
本公开第四方面提供了基于上述一种移动机器人控制系统的控制方法，机器人车载多自由度机械臂宽范围移动的控制方法包括如下步骤：
步骤301：设定衔接手型和对应的手势动作，设定不同的手型对应从端机器人不同的动作，衔接手型可以设定为车载多自由度机械臂末端在当前位置等待下一指令；
步骤302：采集操控者可穿戴设备可拍摄范围内的图像；
步骤303：判断采集的图像中是否有手部区域，如果没有，执行步骤302；否则，对采集的图像进行预处理，获得手部裁片，并执行下一步；
步骤304：采用手型识别算法对预处理后的手部裁片进行手型识别，得到手型信息；
步骤305：判断获得的手型信息是否为衔接手型，如果是，车载多自由度机械臂末端连续执行衔接手型前一手型和衔接手型后一手型的对应的控制指令的动作，并执行步骤302；否则,执行下一步。
步骤306：按照相应的手型执行相应的动作，并执行步骤302。
本公开第四方面的控制方法，通过设置衔接手型，可以实现对侦察系统末端位置的增量式连续精确控制，而且控制更符合人的操作习惯。
与现有技术相比，本公开的有益效果为：
(1)本公开第一方面在操作者的头部设置可穿戴双目摄像装置和头戴虚拟显示器，可以实现双视角图像的采集，头戴虚拟显示器设置可以同时实现虚拟模型和采集的侦查图像，能够使得操作者有身临其境的感觉，能实现远程从端机器人的直观控制，通过可穿戴装置的设置解放了操作员的双手，减轻了操作员的负担。
(2)本公开第一方面提出的系统，通过设置可穿戴的设备，包括可穿戴双目摄像装置和头戴虚拟显示器，提高了操控的直观性，解放机器人操控者的双手，并且装置结构简单，便于佩戴，便于操控者在操作机器人的同时可以同时手持武器或者进行其他手部动作。
(3)本公开第二方面提出的控制方法，控制流程为操作员手势的位姿—虚拟手势模型的位姿—虚拟机械臂末端位姿—多自由度机械臂末端位姿，通过建立操作者手势位姿和多自由度机械臂末端位姿的驱动关系，实现对多自由度机械臂末端位姿的连续控制。同时在头戴虚拟显示器中显示虚拟机械臂末端和虚拟手势模型的跟随过程，使得控制过程更直观。通过设置对应的手势启动和停止对从端机器人的多自由度机械臂的控制，控制方法简单而且可靠。
(4)本公开第三方面提出的控制方法，在进行控制时采用操控者不同的手控制从端机器人的不同动作部位。通过左右手分别控制可以使得从端机器人执行命令动作的准确度更高，并且对从端机器人不同部位的控制设置不同的手势类型，分别为手势的识别和运动轨迹的识别，再次区分控制的是移动机器人车体还是车载多自由度机械臂，减小了从端机器人的误动作率。同时操控者两个手动作是不同类型的动作，避免引起操控者的混乱，控制逻辑简单，容易记忆，方便操作。
(5)本公开第四方面提出的控制方法，通过设置衔接手型，将操控员耳挂式摄像视野范围内的区域变成了一个虚拟触屏区域，使操控员摆脱了对物理控制器的依赖；相比于仅使用手势类型实现对机器人动作的离散控制，本公开可以实现对侦察系统末端位置的增量式连续精确控制，而且控制更符合人的操作习惯。
(6)本公开通过测量特定手势在图像内的旋转角度可以实现对移动侦察机器人车载多自由度机械臂侦察方向的精确控制。
附图说明
构成本申请的一部分的说明书附图用来提供对本申请的进一步理解，本申请的示意性实施例及其说明用于解释本申请，并不构成对本申请的限定。
图1是本公开实施例2的虚拟穿戴的示意图；
图2是本公开实施例2的虚拟脱卸的示意图；
图3是本公开实施例2的控制方法的流程图；
图4是根据一个或多个实施方式的系统的结构示意图；
图5是本公开实施例1主端可穿戴遥操作控制装置的框图；
图6是本公开实施例1的从端机器人的框图；
图7是本公开实施例3的方法流程图；
图8是本公开实施例4的方法流程图；
图9是本公开实施例4操控员控制移动机器人车体运动的手势示意图；
图10是本公开实施例4操控员控制车载多自由度机械臂末端左转的手势示意图；
图11是本公开实施例4操控员控制车载多自由度机械臂末端右转的手势示意图；
图12是本公开实施例4操控员控制车载多自由度机械臂末端上移的手势示意图；
图13是本公开实施例4操控员控制车载多自由度机械臂末端下移的手势示意图；
图14是本公开实施例4操控员控制车载多自由度机械臂末端上仰的手势示意图；
图15是本公开实施例4操控员控制车载多自由度机械臂末端下俯的手势示意图；
其中：N1、移动机器人本体，N2、多自由度机械臂，N3、侦查摄像头，N4、视频眼镜，N5、双目摄像头，N6、虚拟机械臂；
其中：100、主端可穿戴遥操作控制装置，101、遥操作控制器，102、左可穿戴视觉设备，103、右可穿戴视觉设备，104、头戴虚拟显示器，105、无线音频提示设备，106、无线数传设备，107、无线图传设备；
200、从端机器人，201、车载控制器，202、移动机器人本体，203、连杆机构，204、武器装置，205、激光测距传感器，206、手眼监控摄像头，207、侦查摄像头，208、激光雷达，209、从端无线数传设备，210、从端无线图传设备，211、电机驱动器，212、机械臂驱动器，213、车体驱动电机组，214、机械臂驱动电机组。
具体实施方式：
下面结合附图与实施例对本公开作进一步说明。
应该指出，以下详细说明都是示例性的，旨在对本申请提供进一步的说明。除非另有指明，本文使用的所有技术和科学术语具有与本申请所属技术领域的普通技术人员通常理解的相同含义。
需要注意的是，这里所使用的术语仅是为了描述具体实施方式，而非意图限制根据本申请的示例性实施方式。如在这里所使用的，除非上下文另外明确指出，否则单数形式也意图包括复数形式，此外，还应当理解的是，当在本说明书中使用术语“包含”和/或“包括”时，其指明存在特征、步骤、操作、器件、组件和/或它们的组合。需要说明的是，在不冲突的情况 下，本公开中的实施例及实施例中的特征可以相互组合。下面将结合附图对实施例进行详细描述。
机器人根据不同的末端执行器可以分为很多类，末端执行器固定在机器人机械臂的末端用于执行相应的任务，末端执行器如灵巧手和夹持器、摄像机等，侦查机器人的末端执行器为侦查摄像机，本实施例以侦查机器人为例进行说明，但本公开的机器人末端位姿的连续控制方法并不仅限于侦查机器人，而是适用于所有的机器人的控制。
实施例1
在一个或多个实施方式中公开的技术方案中，如图1和图2所示，一种移动机器人控制系统，包括主端可穿戴遥操作控制装置和从端机器人，所述主端可穿戴遥操作控制装置和从端机器人通过无线通信，所述主端可穿戴遥操作控制装置穿戴在操作员身上，用于发送控制指令和接收从端机器人采集的数据；
主端可穿戴遥操作控制装置包括可穿戴双目摄像装置、头戴虚拟显示器、遥操作控制器和主端无线通信设备，所述遥操作控制器分别与可穿戴双目摄像装置、头戴虚拟显示器和主端无线通信设备连接，可穿戴双目摄像装置用于采集操作员手势的图像，所述头戴虚拟显示器用于显示从端机器人拍摄的图像以及用于显示从端机器人的机械臂的虚拟模型和操作员手势的虚拟模型。设置为双目摄像装置可以实现采集双视角图像。
遥操作控制器可以为穿戴式计算机，所述穿戴式计算机可实时采集可穿戴双目摄像装置拍摄的手势的双视角图像，并根据手势的双视角图像计算操作员手势的位姿信息，并根据手势位姿信息在视频眼镜的透视视景体前端实时显示一个虚拟手势模型；
可穿戴双目摄像装置可以为双目摄像头N5，所述双目摄像头N5用于采集操作员手势的双视角图像。操作员在双目摄像头N5视野范围内使用手势位姿来实现对车载多自由侦察系统末端位姿的控制。
头戴虚拟显示器可以为视频眼镜N4，用于显示从端机器人侦察摄像头N3拍摄的侦查图像，以及用于显示多自由度机械臂N2的虚拟模型和操作员手势的虚拟模型，其中侦查图像可以位于视频眼镜的透视视景体的后端，多自由度机械臂N2的虚拟模型和操作员手势的虚拟模型位于视频眼镜的透视视景体的前端；本实施例采用了透视视景体显示，可以采用其他视景体。透视视景体即通过透视投影的视景体，透视投影的视景体类似于一个顶部和底部都被进行切割过的棱锥,即棱台，它的特点是:近大远小。
从端机器人包括移动机器人本体N1、多自由度机械臂N2、侦查摄像头N3、从端无线通信设备和车载控制器，所述车载控制器分别与移动机器人本体N1、多自由度机械臂N2、侦查摄像头N3和从端无线通信设备连接。侦察摄像头N3安装在多自由度机械臂N2末端用于采集侦查数据，移动机器人本体N1还包括车体驱动电机组和电机驱动器，所述电机驱动器分别与车载控制器和驱动电机组连接。移动机器人本体N1通过车载控制器接收主端可穿戴遥操作控制装置的控制进行位置上的移动。车载控制器将控制命令发送至电机驱动器，电机驱动器控制驱动电机组的相应电机，实现从端机器人位置的移动。
多自由度机械臂N2接收主端可穿戴遥操作控制装置的控制执行相应的动作，所述多自由度机械臂N2包括连杆机构、机械臂驱动器和机械臂驱动电机组。车载控制器将控制命令发送至机械臂驱动器，机械臂驱动器驱动机械臂驱动电机组的相应电机，实现连杆机构角度和位置的移动，从而改变多自由度机械臂N2的各关节的关节角信息。
从端机器人的机械臂的虚拟模型为多自由度机械臂N2的虚拟模型。所述多自由度机械臂N2的虚拟模型可以为按照多自由度机械臂N2的D-H参数绘制的虚拟机械臂N6。
操作员在双目摄像头N5视野范围内使用手势位姿来实现对车载多自由侦察系统末端位姿的控制。
进一步地，如图4所示，一种基于可穿戴设备的机器人远程控制系统，包括通过无线连接的主端可穿戴遥操作控制装置100和从端机器人200，所述主端可穿戴遥操作控制装置100穿戴在操控者身上，用于发送控制指令和接收从端机器人200采集的数据；
所述主端可穿戴遥操作控制装置100包括可穿戴双目摄像装置、头戴虚拟显示器104、主端无线通信设备和遥操作控制器101，所述可穿戴双目摄像装置、头戴虚拟显示器104和主端无线通信设备分别与遥操作控制器连接；可穿戴双目摄像装置穿戴在操控者头部位置，用于采集操控者的动作，遥操作控制器101根据相应的动作生成控制指令并发送至从端机器人200。
如图5所示，可穿戴双目摄像装置至少设置一个，可穿戴双目摄像装置包括左可穿戴视觉设备102和右可穿戴视觉设备103，分别穿戴于操控员头部左右两侧，可以拍摄操控员前方的图像，用于采集操控者的手部的动作信息。手部的动作信息可以包括手在图像中的位置信息和手型信息。左可穿戴视觉设备和右可穿戴视觉设备可以具体为耳挂式摄像头。
所述头戴虚拟显示器104可以显示从端机器人200搭载的监控摄像头拍摄的画面；遥操作控制器接收从端机器人200的拍摄的画面信息，控制头戴虚拟显示器104显示现场拍摄的画面，头戴虚拟显示器104可以具体为视频眼镜。
主端无线通信设备通过无线传输模块实现无线传输，可以分为用于传输数据的无线数传设备106和用于传输图像视频数据的无线图传设备107，实现端主端可穿戴遥操作控制装置100和从端机器人200之间的信息传输，具体的用于发送控制指令给从端机器人200、接收从端机器人200发送回的传感器数据以及接收从端机器人200发送回的图像数据等。所述无线传输模块可以包括用于传送图像数据的图传电台和用于传送控制指令的数传电台，如5.8GHz无线图传电台和433MHz无线数传电台。遥控距离较短也可以直接采用一个WIFI通信模块同时实现图像传输和控制指令传输。
所述主端可穿戴遥操作控制装置100，还可以包括无线音频提示设备105，无线音频提示设备105与遥操作控制器101连接，用于提示操控员将要执行的控制指令。
从端机器人200可以具体为地面武装侦察机器人，用于执行侦察任务，包括移动机器人车体和车载多自由度机械臂。
进一步地，如图6所示，所述移动机器人车体可以包括移动机器人本体202、车体驱动电机组213、电机驱动器211、侦查摄像头207和激光雷达208，从端无线通信设备和车载控制器201，从端无线通信设备包括分别用于储输数据和图像的从端无线数传设备209和从端无线图传设备210。可以接收主端可穿戴遥操作控制装置100的控制下进行移动，用于代替操控员进入危险区域执行作战任务。所述电机驱动器211、车体驱动电机组213和移动机器人本体202依次连接，所述电机驱动器211用于根据主端发送的控制指令控制车体驱动电机组213，车体驱动电机组213连接移动机器人本体202实现从端机器人200移动，所述车体驱动电机组213至少包括左侧电机和右侧电机，所述左侧电机和右侧电机可同向转动，可控制机器人实现前进和后退，所述左侧电机和右侧电机可异向转动，可控制机器人实现左转弯或右转弯。
激光雷达208用于测量所述从端地面武装侦察机器人周围的障碍物信息，激光雷达208与车载控制器201连接，车载控制器201接收测量的障碍物信息，并将障碍物信息传输至主端的遥操作控制器101，可以将障碍物信息在主端的头戴虚拟显示器104上显示。从端无线通信设备和主端无线通信设备的结构可以相同，可以选用相同的无线传输模块。所述侦查摄像头207用于拍摄战场环境信息，可以直接设置在车体上，所述侦查摄像头207与车载控制器201连接，用于将采集的环境图像传输至主端的遥操作控制器。
车载多自由度机械臂包括连杆机构203、机械臂驱动电机组214、机械臂驱动器212、激光测距传感器205、手眼监控摄像头206和武器装置204，连杆机构203的末端固定设置手眼监控摄像头206，连杆机构203、机械臂驱动电机组214和机械臂驱动器212依次连接，连杆机构203至少由两段连杆组成，机械臂驱动器212接收到主端发送的控制信息，根据控制信息控制机械臂驱动电机组214工作，从而驱动连杆机构203动作，移动到操控者想要移动的位置，通过设置在连杆机构203末端的手眼监控摄像头207拍摄感兴趣目标的图像信息。
激光测距传感器205和武器装置204分别与机械臂驱动器212用于侦察和打击任务，均可以设置在连杆机构203的末端；所述激光测距传感器205用于测量击打目标的距离信息。
侦查摄像头207和手眼监控摄像头207的设置用于采集不同的图像，侦查摄像头207采集环境数据，通过从端机器人200的移动实现通过路径的环境图像采集，手眼监控摄像头207用于根据操作人的控制进行重点区域或感兴趣区域的图像采集，两个摄像头的设置实现了机器人工作现场图像的无死角采集。
车载控制器201可以控制采集激光雷达208、激光测距传感器205、侦查摄像头207和手眼监控摄像头207的数据并通过无线发送给主端遥操作装置，也可通过所述从端无线通信设备接收主端遥操作装置发送的控制指令，并根据控制指令通过所述电机驱动器211或所述机械臂驱动器212控制相应的所述车体驱动电机组213或所述机械臂驱动电机组214。
实施例2
本实施例提供基于实施例1所述一种移动机器人控制系统的机器人末端位姿的遥操作控制方法，如图1-图3所示，具体的是多自由度机械臂末端位姿遥操作控制方法，能够通过手 势的运动实现对机械臂末端位置和姿态的连续控制，包括如下步骤：
步骤101、设置牵引手型和脱卸手型；
所述牵引手型是指当检测到操作员为此手型时，使得虚拟手势模型的位姿与视频眼镜中的虚拟机械臂末端位姿保持重合，操作员可通过手势的位姿驱动视频眼镜N4中的虚拟手势模型的位置和姿态(即位姿)，则虚拟手势模型可以对虚拟机械臂N6末端位姿进行实时的连续控制。
当手势变为脱卸手势时，则虚拟手势模型不再跟随操作员的手势移动，操作员手势也就不能对虚拟机械臂N6进行实时连续控制。
牵引手型和脱卸手型可以为任意手型，可以根据需要自行设置，本实施例设置牵引手型可以为表示笛卡尔坐标系的手型，该手型中的无名指和小拇指为弯曲状态，大拇指、食指和中指为伸直状态，且三根手指互相垂直构成笛卡尔坐标系；脱卸手型可以为单手握拳手型。
在步骤101之前还可以包括初始化和建立无线连接的步骤：
将遥操作控制器和从端机器人初始化；
建立遥操作控制器和从端机器人N1之间的无线通信通道；
步骤102：构建虚拟机械臂和虚拟手势模型并显示在头戴虚拟显示器视景体的前端；
所述步骤102构建虚拟机械臂并显示在头戴虚拟显示器视景体的前端的方法具体为：
1021)、读取从端机器人的多自由度机械臂的各关节的关节角信息；
多自由度机械臂的动作是由车载控制器控制,机械臂驱动器驱动机械臂驱动电机组的相应电机，实现连杆机构角度和位置的移动，从而改变多自由度机械臂N2的各关节的关节角信息。多自由度机械臂的各关节的关节角信息可以由车载控制器直接读取。
1022)、遥操作控制器根据采集的关节角信息计算多自由度机械臂的D-H参数；
1023)、根据多自由度机械臂的D-H参数构建虚拟机械臂，并将虚拟机械臂显示在头戴虚拟显示器视景体的前端。
所述虚拟机械臂N6的各关节的角度由接收到的关节角信息控制，虚拟机械臂N6的基坐标系由视频眼镜N4的屏幕坐标系描述，虚拟机械臂N6的末端坐标系记为(O M-X M-Y M-Z M)，虚拟机械臂N6的末端的位姿由P M表示，包括位置信息和姿态信息；
虚拟手势模型的构建方法可以具体为：
(1)使用3D建模软件离线建立牵引手型的三维虚拟手势模型；
(2)实时将该三维虚拟手势模型加载并渲染到头戴虚拟显示器视景体的前端，其在视景体中的位置和姿态由操作者的牵引手型的位置和姿态驱动。
为便于操作者的操作具有目的性和准确性，在视频眼镜N4中还可以显示从端机器人所处的侦查环境信息，具体的可以将侦查摄像头N3采集的侦查图像显示在视频眼镜N4的视景体中，还可以包括在头戴虚拟显示器上显示从端机器人拍摄的图像的步骤，具体如下：采集从端机器人端的侦察图像；遥操作控制器接收侦察图像并其实时显示在头戴虚拟显示器的视景体后端。
步骤103、采集双目摄像头N5的双视角图像；通过双目摄像头N5采集操作员的手型信息。双视角图像包括左右两个视角的图像。
步骤104、采用手势检测算法检测，判断双视角图像中是否有操作员的手势存在，如果是，则执行下一步，否则执行步骤103；只要在双视角图像中出现了操作员的手势，此时执行步骤105。
手势检测算法可以具体为基于肤色阈值的手势检测算法。
步骤105、采用手型识别算法对手势进行手型识别，判断是否出现了牵引手型，如果是，执行下一步，否则执行步骤103；手型识别算法具体为基于深度学习的手型识别算法。
当检测到双双视角图像中监测到牵引手型，此时要实现通过操作员的手型牵引控制多自由度机械臂N2。如果没出现牵引手型，重新通过执行步骤3双目摄像头N5采集操作员的手型信息。
步骤106、对拍摄的双视角图像进行处理并求解牵引手势在可穿戴双目摄像装置坐标系中的位姿P H，将位姿P H转换为在头戴虚拟显示器的屏幕坐标系中的位姿描述P V，采用转化后的位姿P V驱动头戴虚拟显示器视景体中的虚拟手势模型；
求解牵引手势在可穿戴双目摄像装置坐标系中的位姿P H可以采用DeepPrior++算法，DeepPrior++算法可以实现立体视觉下对手势位姿的估计。
求解牵引手势在可穿戴双目摄像装置坐标系中的位姿P H还可以采用以下步骤：
(1061)牵引手势位姿P H，包含位置信息和姿态信息，其中位置信息的求解直接使用左右视图中手势检测结果和视差原理来实现；
(1062)牵引手势位姿P H的姿态信息使用基于回归学习的方法实现：
牵引手势位姿P H的姿态信息使用基于回归学习的方法实现具体可以为：
(1062.1)首先采集双视角手势图像与对应姿态数据集。可以采用手持三轴姿态传感器在双视角摄像头前分别绕三轴姿态传感器的三个轴做旋转运动并采集姿态传感器每一次输出数据对应的双视角手势检测结果图像。同一时刻获取的两帧手势图像和一帧姿态数据作分别为输入样本和输出样本。采集到的双视角手势图像和对应姿态数据分别作为输入样本训练集和输出样本集。
(1062.2)使用回归学习方法拟合双视角手势图像与姿态数据的映射关系。
(1062.3)经过以上两步就可以直接通过双视角的手势图像求解牵引手势的姿态信息。
步骤106可以首先是建立操作员的牵引手势与虚拟手势模型的对应关系，通过对应关系将位姿P H转换为位姿P V。具体的对应关系可以为正比例关系，操作员的牵引手势在可穿戴双目摄像装置坐标系中的位姿P H的位置信息和位姿P V的位置信息成正比例关系，位姿P H的姿态信息和位姿P V的姿态信息也成正比例关系。
所述牵引手势的位姿P H是在双目摄像头N5的坐标系中描述的，可以规定牵引手势的手心为原点，牵引手势手心处的原点坐标系为(O H-X H-Y H-Z H)，牵引手势的中指所指方向为X轴方向，大拇指所指方向为Y轴方向，中指所指方向为Z轴方向，其中位姿P H的位置 信息由牵引手势手心处的原点O H相对双目摄像头N5坐标系原点的偏移描述，位姿P H的姿态信息由牵引手势的坐标系X H轴、Y H轴和Z H轴对于双目摄像头N5坐标系各轴的旋转描述。
所述虚拟牵引手势的位姿P V是在视频眼镜N4的屏幕坐标系中描述的，可以规定虚拟牵引手势的手心为原点，所述虚拟牵引手势手心处的原点坐标系记为(O V-X V-Y V-Z V)，虚拟牵引手势的中指所指方向为X轴方向，大拇指所指方向为Y轴方向，中指所指方向为Z轴方向，其中位姿P V的位置信息由虚拟牵引手势手心处的原点O V相对视频眼镜N4的屏幕坐标系原点的偏移描述，位姿P V的姿态信息由虚拟牵引手势的坐标系X V轴、Y V轴和Z V轴对于视频眼镜N4的屏幕坐标系各轴的旋转描述。
其次，采用转化后的位姿P V驱动头戴虚拟显示器视景体中的虚拟手势模型，虚拟手势模型就开始跟随操作员的手势移动进行移动。
所述驱动方法具体为：维虚拟手势模型加载到头戴虚拟显示器中后，其在视景体中的实时绘制时所需的位置信息由位姿P V的位置信息直接赋值，其在视景体中的实时绘制时所需的姿态信息由位姿P V的姿态信息直接赋值。
由于虚拟手势模型的位姿由位姿P V的位置信息和姿态信息实时直接赋值，因此虚拟手势模型在视景体中的位姿与位姿P V完全一致，因此可理解为虚拟手势模型的位姿由位姿P V驱动。
步骤107、判断虚拟手势模型的位姿P V与虚拟机械臂N6末端位姿P V的差是否小于预设阈值，如果是，执行下一步；否则执行步骤3；
步骤107是实现穿戴的过程，具体的是实现虚拟手势模型的位姿P V与虚拟机械臂N6末端位姿P M的距离快接近。通过步骤6操作员移动牵引手势，从而虚拟手势模型也跟随移动，直到虚拟手势模型的位姿P V接近虚拟机械臂N6末端位姿P M。
步骤107具体的实现过程为：操作员观察视频眼镜N4的透视视景体中虚拟牵引手势位姿P V与虚拟机械臂N6末端位姿P M之间的相对关系，通过不断移动牵引手势的位姿P H来使视频眼镜N4的透视视景体中的虚拟牵引手势的位姿P V与虚拟机械臂N6末端位姿P M的差不断减小，所述两位姿之差由下式描述：
d＝|P V-P M|
当虚拟牵引手势的位姿P V与虚拟机械臂N6末端位姿P M的差d小于预设阈值时，则认为虚拟机械臂N6末端已经与虚拟牵引手势重合，可以形象的认为此时虚拟机械臂N6末端虚拟穿戴在了虚拟牵引手势上了。此过程中，遥操作控制器通过多次执行步骤103-107实现。当穿戴过程完成，可以牵引多自由度机械臂N2。
步骤108、使得多自由度机械臂的位姿跟随操作员的牵引手型位姿变化；
所述步骤108的步骤具体为：
使得虚拟机械臂末端位姿P M的值与虚拟手势模型的位姿P V相等，求解虚拟机械臂N6对应的各关节角值；具体的通过机器人逆运动学求解算法实时求解当虚拟机械臂末端位姿P M的值与虚拟手势模型的位姿P V相等时，虚拟机械臂N6对应的各关节角度值。
根据求解的虚拟机械臂对应的各关节角度值转换为控制指令传输至从端机器人，使得多自由度机械臂各关节的关节角度与虚拟机械臂的各关节角度值相等。
具体的可以为：遥操作控制器将虚拟机械臂N6各关节角转换为控制指令并通过无线通信通道发送给从端机器人N1，从端机器人N1的车载控制器读取接收到的控制指令之后，将控制指令转换为电机驱动指令，然后通过机械臂驱动器控制多自由度机械臂N2的机械臂驱动电机组的各关节电机开始转动，使多自由度机械臂N2各关节的关节角与虚拟机械臂N6的各关节角相同；从而使得多自由度机械臂N2的位姿跟随操作者手势位姿的变化。
为使得操作员的操控更加直观可以调整虚拟机械臂的位置为姿态，可以使得视频眼镜N4中实时显示虚拟机械臂N6的位姿变化。
所述步骤108还包括：根据求解的虚拟机械臂N6对应的各关节角度值在视景体中对虚拟机械臂N6进行重绘。根据机器人逆运动学算法得到的虚拟机械臂N6的各关节角度值在视频眼镜N4的透视视景体中对虚拟机械臂N6进行重绘，使虚拟机械臂N6末端位姿始终与虚拟牵引手势的末端位姿保持相同。
步骤109、判断是否出现脱卸手型，如果是，多自由度机械臂的位姿停止跟随操作员的牵引手型位姿变化，并执行步骤103；否则，执行步骤108。牵引过程中实时判断操作员手势是否变为脱卸手势，本实施例脱卸手势可以设置为左手握拳状态，如果操作员手势变为脱卸手势则虚拟机械臂N6末端位姿不再受操作员控制，可以形象的认为此时虚拟机械臂N6末端已经从操作员的手势上虚拟脱卸下来了。脱卸手势可以为任意手势，可以为单手手势，也可以为双手手势。至此，牵引过程结束，可结束也可执行其他命令。
实施例3
本实施例提供一种基于实施例1所述机器人控制系统的控制方法，分别采集操控者的左手和右手的动作，通过一只手的动作控制移动机器人车体的移动，通过另一只手的动作控制移动机器人的车载多自由度机械臂的动作。
遥操作控制器可以采集可穿戴双目摄像装置拍摄的图像并分析在图像中是否有操控员的左右手及其手型种类和位置坐标，在检测到操控员的左右手时能根据手型种类和位置坐标通过无线通信设备发送相应的控制指令给从端机器人200，以控制从端机器人200的运动及车载多自由度机械臂的运动，还可以在发出控制指令前将控制指令的名称通过无线音频提示设备反馈给操控员，除此之外，还可以处理通过无线通信设备接收到的从端机器人200发送回的传感器数据和监控图像，并显示在头戴虚拟显示器104上。
具体的，如图7所示，可以包括如下步骤：
步骤201：采集操控者可穿戴设备可拍摄范围内的图像；
可穿戴双目摄像装置具体的可以是可穿戴的摄像头，设置在操控者的头部可以采集操控者周围的图像，当设置了一个摄像头时，需要操控者根据要进行的控制将相应的手在摄像范围内做相应的动作。为便于操控者的自由操作，可以在左右都设置摄像头，左右摄像头分别 采集左侧图像和右侧图像，可以采用图像拼接方法将两幅图像重叠部分剪除并拼接成一副宽视野图像，即为采集的图像。
步骤202：判断采集的图像中是否有手部区域，如果没有，执行步骤201；否则，对采集的图像进行预处理，获得手部裁片；
判断采集的图像中是否有手部的方法可以采用手势检测算法，手势检测算法具体的可以采用基于肤色的手势检测算法。
对采集的图像进行预处理，获得手部裁片的具体方法为：如果检测到手部的存在，则使用手势分割算法将包含手部的区域中的背景剔除，进一步采用尺度归一化将包含手部的图像归一化为相同尺寸的手部裁片。
步骤203：利用左右手判别算法判断获得的手部裁片是左手裁片还是右手裁片，从而确定做动作的是左手还是右手；
采用左右手判别算法判断是左手裁片还是右手裁片的方法可以具体为：
左右手判别算法判断是二分类问题,首先准备包含左手图像的训练样本集(如其标签为0)和包含右手图像的训练样本集(如其标签为1)，选用一种分类器(如可以为卷积神经网络)对带有标签的两个图像集进行学习，学习方法可采用误差反传算法，学习完成后，可得到一个左右手二分类分类器，使用此分类器可对获得的手部裁片进行左右手类型判别。
步骤204：通过其中一只手的动作控制移动机器人车体的移动，通过另一只手的动作控制移动机器人的车载多自由度机械臂的动作，然后，执行步骤201。
具体的是采用手势和手指的移动轨迹是控制从端机器人200的哪个部分的移动可以是设定的，本实施例设定了通过手势控制移动机器人车体的移动，设定了手指的移动轨迹控制车载多自由度机械臂的动作。
步骤204中通过其中一只手的动作控制移动机器人车体的移动，具体步骤为：
步骤2041、设定从端机器人200运动控制指令与手型信息的对应关系；手型信息为操控者摆出的手势信息，可以包括拳头、剪刀手、OK手势等等，运动控制指令包括前进、后退、左转、右转、掉头。具体的对应关系可以根据具体的需要设置。生成相应的对应关系表。
步骤2042、当识别的手部裁片为设定为控制移动机器人车体的移动的一只手，采用手型识别算对手部裁片进行识别，得到手型信息；
在进行控制时采用操控者不同的手控制从端机器人200的不同动作部位。通过左右手分别控制可以使得从端机器人200执行命令动作的准确度更高，首先判断是左手还是右手，通过手的区分来区分是控制移动机器人车体还是车载多自由度机械臂，通过两个手的手势的识别和运动轨迹的识别再次区分控制移动机器人车体还是车载多自由度机械臂，减小了误动作率。同时操控者两个手动作是不同类型的动作，避免引起操控者的混乱，控制逻辑简单，容易记忆，方便操作。
可以设定其中任意一只手控制移动机器人车体的移动，本实施例可以选择为左手。当设定了左手控制移动机器人车体的移动，则车载多自由度机械臂的动作就由右手控制。具体的 是采用手势和手指的移动轨迹是控制从端机器人200的哪个部分的移动可以是设定的，本实施例设定了通过手势控制。
步骤2043、根据从端机器人200运动控制指令与手型信息的对应关系以及识别获得的手型信息生成从端机器人200运动控制指令，将运动控制指令发送至从端机器人200，从端机器人200根据控制指令执行相应的动作。
步骤2043还包括如下步骤：设定对应运动控制指令的运动名称，当生成从端机器人200运动控制指令后，将对应运动控制指令的运动名称发送至无线音频提示设备，所述无线音频提示设备播报从端机器人200将要执行的动作。操控者根据播报可以确定将要执行的动作是否正确。
步骤204中通过另一只手的动作控制移动机器人的车载多自由度机械臂的动作，具体步骤为：
步骤204-1、当识别的手部裁片为设定为控制移动机器人的车载多自由度机械臂的动作的一只手，采用指尖定位算法分析任意指尖在图像中的运动轨迹；
步骤204-2、根据运动轨迹生成位置跟踪指令，并将位置跟踪指令发送至从端机器人200；
步骤204-3、从端机器人200根据位置跟踪指令生成具体动作的位置坐标，连杆机构203末端依次经过位置坐标实现操控者指尖运动轨迹的跟踪。
采用指尖定位算法分析任意指尖在图像中的运动轨迹,可以采用基于轮廓曲率的指尖定位算法和基于凸包分析的指尖定位算法。
位置坐标可以为多个，当设置的位置坐标密度越大动作轨迹与操控者的手指动作轨迹重合度越高。位置坐标的设置可以以连杆机构203的基座作为原点。
实施例4
本实施例提出了另一种基于实施例1所述一种基于可穿戴设备的机器人远程控制系统的控制方法，与实施例3的方法不同在于，不用区分左右手进行控制，本实施例通过设置不同的手势控制移动机器人车体的运动和车载多自由度机械臂控制。可以实现操控者的可穿戴双目摄像装置的摄像范围外的动作的执行。
实施例3中的方法在通过识别手指的移动轨迹进行车载多自由度机械臂控制时，操控者手部的移动轨迹需要完整在呈现摄像范围内，操控者的动作幅度需要在可穿戴双目摄像装置的摄像范围内，车载多自由度机械臂的动作幅度就受到了限制。本实施例可以实现比实施例3更宽范围内的车载多自由度机械臂的移动。
对于移动机器人的车体控制，只要操作员的手势出现前进手型，则移动机器人就一直前进，直到操作员手势变为停止手型，则移动机器人停止前进。也就是说移动机器人车体在前进、后退、左转和右转过程中，使用停止手型使其停止。机器人车体的运动是属于保持运动，直到出现停止信号，否则持续运动。
对于机械臂末端的控制，可以不设置停止手型，机械臂末端的俯仰角度、上移距离、下移距离、左移距离和右移距离都是通过跟随俯仰手型或末端牵引手型实现的，一旦俯仰手型 或末端牵引手型变为衔接手型、手势超出摄像头范围等则机械臂末端就自然停止了，不要专门加一个停止手型来对机械臂末端进行控制。
在一个或多个实施方式中公开的技术方案中，如图8所示，提供一种机器人车载多自由度机械臂宽范围移动的控制方法，包括如下步骤：
步骤301：设定衔接手型和对应的手势动作，设定不同的手型对应从端机器人200不同的动作，衔接手型可以设定为车载多自由度机械臂末端在当前位置等待下一指令；
步骤302：采集操控者可穿戴设备可拍摄范围内的图像；
步骤303：判断采集的图像中是否有手部区域，如果没有，执行步骤302；否则，对采集的图像进行预处理，获得手部裁片，并执行下一步；
步骤304：采用手型识别算法对预处理后的手部裁片进行手型识别，得到手型信息；
步骤305：判断获得的手型信息是否为衔接手型，如果是，车载多自由度机械臂末端连续执行衔接手型前一手型和衔接手型后一手型的对应的控制指令的动作，并执行步骤302；否则,执行下一步。
步骤306：按照相应的手型执行相应的动作，并执行步骤302。
步骤301之前还包括如下步骤：
3001、遥操作控制器和从端机器人200进行初始化操作。
3002、建立遥操作控制器和从端机器人200之间的无线通信通道。
3003、从端机器人200采集机器人摄像头的侦察图像，然后通过无线通信通道发送给遥操作控制器，遥操作控制器的通过无线通信设备接收到侦察图像后将侦察图像实时显示在操控员佩戴的视频眼镜上。
所述步骤301中设定不同的手型对应从端机器人200不同的动作，对应从端机器人200的相应动作设定不同的手型进行控制，操控员可以根据需要自行设定。从端机器人200的动作包括车载多自由度机械臂末端动作和移动机器人车体移动，所述移动机器人车体移动的移动包括停止、前进、后退、左转、右转等等，本实施例设置的对应手型及其对应的手势及控制指令的对应关系可以如图9所示，空手型H1对应不发送控制指令，从端机器人200静止在原地；前进手型H2对应前进指令，移动机器人车体启动电机驱动器211向前方移动；同理，左转手型H3、右转手型H4、后退手型H5、停止手型H6分别对应移动机器人车体左转、右转、后退和停止一切动作。可以建立相应的对应表，操控者可以根据自己的习惯更改手势与控制指令的对应表。
所述步骤303和步骤304的方法可以与实施例2中记载的方法相同。
步骤305中，车载多自由度机械臂末端连续执行衔接手型前一手型和衔接手型后一手型的对应的控制指令的动作具体为：
车载多自由度机械臂末端执行前一手型对应的动作后，停止在当前位置；
执行步骤302-步骤304，检测到衔接手型的下一手型，车载多自由度机械臂末端从当前位置动作执行衔接手型的下一手型对应的动作。
本实施例以设定空手型H1作为衔接手势进行举例说明。可以使用末端牵引手型H8对车载多自由度机械臂末端位置进行连续增量式左右偏转控制或上下位移控制。首先定义末端牵引手型H8结合在此手型下的移动对应车载多自由度机械臂末端的向上、向下、向左、向右移动。
当操控者单次控制需要向上移动的距离超过摄像范围，为实现连续增量式的向上移动，可以设定末端牵引手型衔接手型的前一手型和后一手型为端牵引手型H8并向上移动，如图12所示，操控员使用末端牵引手型H8先向上移动U1，U1大于预设阈值，然后操控员摆出空手型H1，并将空手型H1重新移动到操控者耳挂式摄像头的视野范围内，然后再次摆出末端牵引手型H8并再次向上移动U2,U1和U2为移动的距离，移动的距离大于预设阈值即操控者的滑动超出摄像范围，则操控员的末端牵引手型H8在操控者耳挂式摄像头视野内的移动总距离为U＝U1+U2，则多自由度机械臂末端向上移动距离为K2*(U1+U2),遥操作控制器向车载控制器201依次发送多自由度机械臂末端向上移动距离为K2*U1、停止在当前位置、从当前位置向上移动K2*U1距离的控制指令。其中K2为位移系数，用于调节末端牵引手型H8上下移动距离与多自由度机械臂末端位置上下移动距离的比例关系。
当操控者单次控制需要向下移动的距离超过摄像范围，为实现连续增量式的向下移动，可以设定末端牵引手型衔接手型的前一手型和后一手型为端牵引手型H8并向下移动，如图13所示，仅仅是移动方向与图12相反，此处不再赘述。
当操控者单次控制需要向左移动的距离超过摄像范围，为实现连续增量式的向左偏转，可以设定末端牵引手型衔接手型的前一手型和后一手型为端牵引手型H8并向左移动，如图10所示，操控员使用末端牵引手型H8先向在移动L1，L1大于预设阈值，然后操控员摆出空手型H1，并将空手型L1重新移动到操控者耳挂式摄像头的视野范围内，然后再次摆出末端牵引手型H8并再次向左移动L2,L1和L2为移动的距离，则操控员的末端牵引手型H8在操控者耳挂式摄像头视野内的移动总距离为L＝L1+L2，则多自由度机械臂末端向左移动角度为
Figure PCTCN2020087846-appb-000001
遥操作控制器向车载控制器201依次发送多自由度机械臂末端向左偏转角度为
Figure PCTCN2020087846-appb-000002
停止在当前位置、从当前位置向左偏转角度
Figure PCTCN2020087846-appb-000003
距离的控制指令。其中K1为偏转系数，用于调节末端牵引手型H8左右移动距离与车载多自由度机械臂末端左右偏转角度的比例关系。
当操控者单次控制需要向右移动的距离超过摄像范围，为实现连续增量式的向右偏转，可以设定末端牵引手型衔接手型的前一手型和后一手型为端牵引手型H8并向右移动，如图11所示，仅仅是移动方向与图10相反，此处不再赘述。
本实施例提出的一种机器人车载多自由度机械臂宽范围移动的控制方法，将操控员耳挂式摄像视野范围内的区域变成了一个虚拟触屏区域，使操控员摆脱了对物理控制器的依赖；相比于仅使用手势类型实现对机器人动作的离散控制，本发明可以实现对侦察系统末端位置的增量式连续精确控制，而且控制更符合人类的操作习惯。
所述步骤306中，按照相应的手型执行相应的动作，手型是可以根据个人习惯或者约定的手型进行设定的，并设定手型与对应动作的对应关系，从端机器人车体的动作主要有前进、后退、左转、右转和停止，本实施例可以具体如下：
(3061)如果操控员的手型为空手型H1，则遥操作控制器不发出控制指令给从端机器人200，然后继续执行步骤302；
(3062)如果操控员的手型为停止手型H6，则遥操作控制器通过无线通信设备发出停止控制指令使移动侦察机器人停止运动，然后执行步骤302；
(3063)如果操控员的手型为前进手型H2、后退手型H3、左转手型H4或右转手型H5，则遥操作控制器通过无线通信设备发出前进指令、后退指令、左转指令和右转指令，直到操控员摆出停止手型H6，执行步骤302；
从端机器人多自由度机械臂的动作主要有偏转一定的角度和上下左右移动，本实施例可以具体如下：
(3064)如图14和图15所示，如果操控员的手型为控制车载多自由度机械臂末端俯仰角的俯仰手型H7，则首先计算俯仰手型H7在平面内的上仰角度α或下俯角度β，上仰角度α或下俯角度β是相对于水平面的角度。然后根据俯仰手型H7的俯仰角度计算车载多自由度机械臂末端的俯仰角，最后遥操作控制器通过无线通信设备发出能使车载多自由度机械臂末端俯仰角与俯仰手型H7的俯仰角度保持一致的控制指令，直到操控员摆出其他手型，然后执行步骤302；本通过测量特定手势类型如俯仰手型H7在图像内的旋转角度可以实现对多自由度机械臂末端的侦察方向(相对于水平面)的精确控制，摄像头的侦察方向等于俯仰手型的俯仰角度。
(3065)如果操控员的手型为末端牵引手型H8时，如图10、图11、图12和图13所示，则进一步检测手型的位移距离和位移方向，当操控者位移距离未超过预设阈值时，当末端牵引手型H8是左右移动时，则遥操作控制器通过无线通信设备发出能使多自由度机械臂末端向左或向右偏转角度分别为
Figure PCTCN2020087846-appb-000004
或
Figure PCTCN2020087846-appb-000005
的控制指令，其中为K1偏转系数，用于调节末端牵引手型H8左右移动距离与车载多自由度机械臂末端左右偏转角度的比例关系，L和R为末端牵引手型H8向左移动距离和向右移动距离，r为车载多自由度机械臂末端绕其基座的旋转半径，当末端牵引手型H8停止移动或者移出摄像头视野，遥操作控制器通过无线通信设备发出能使多自由度机械臂末端停止左右偏转的控制指令，然后继续执行步骤302；当末端牵引手型H8变为衔接手型时，执行步骤305。
(3066)当末端牵引手型H8是上下移动时，则遥操作控制器通过无线通信设备发出能使多自由度机械臂末端位置向上或向下移动距离分别为K2*U或K2*D的控制指令，其中K2为位移系数，用于调节末端牵引手型H8上下移动距离与多自由度机械臂末端位置上下移动距离的比例关系，U和D分别为末端牵引手型H8向上移动距离和向右移动距离，当末端牵引手型H8停止移动、移出摄像头视野，遥操作控制器通过无线通信设备发出能使车载多自由度侦察系统停止上下移动的控制指令，执行步骤302；当末端牵引手型H8变为衔接手型时， 执行步骤305。
本实施例的手型仅仅是示例手型，具体的手型，可以根据需要自行设定。
实施例5
一种电子设备，包括存储器和处理器以及存储在存储器上并在处理器上运行的计算机指令，所述计算机指令被处理器运行时，完成实施例2、3或4中方法所述步骤。
实施例6
一种计算机可读存储介质，用于存储计算机指令，所述计算机指令被处理器执行时，完成实施例2、3或4中方法所述步骤。
以上所述仅为本申请的优选实施例而已，并不用于限制本申请，对于本领域的技术人员来说，本申请可以有各种更改和变化。凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。
上述虽然结合附图对本公开的具体实施方式进行了描述，但并非对本公开保护范围的限制，所属领域技术人员应该明白，在本公开的技术方案的基础上，本领域技术人员不需要付出创造性劳动即可做出的各种修改或变形仍在本公开的保护范围以内。

Claims (14)

  1. 基于可穿戴设备的移动机器人控制系统，其特征是：包括主端可穿戴遥操作控制装置和从端机器人，所述主端可穿戴遥操作控制装置和从端机器人通过无线通信，所述主端可穿戴遥操作控制装置穿戴在操作员身上，用于发送控制指令和接收从端机器人采集的数据；
    主端可穿戴遥操作控制装置包括可穿戴双目摄像装置、头戴虚拟显示器、遥操作控制器和主端无线通信设备，所述遥操作控制器分别与可穿戴双目摄像装置、头戴虚拟显示器和主端无线通信设备连接，可穿戴双目摄像装置用于采集操作员手势的图像，所述头戴虚拟显示器用于显示从端机器人拍摄的图像以及用于显示从端机器人的机械臂的虚拟模型和操作员手势的虚拟模型。
  2. 如权利要求1所述基于可穿戴设备的移动机器人控制系统，其特征是：从端机器人包括移动机器人本体、多自由度机械臂、侦查摄像头、无线通信设备和车载控制器，所述车载控制器分别与移动机器人本体N1、多自由度机械臂N2、侦查摄像头N3、无线通信设备连接；移动机器人本体接收主端可穿戴遥操作控制装置的控制进行位置上的移动，车载多自由度机械臂接收主端可穿戴遥操作控制装置的控制执行相应的动作，所述从端机器人的机械臂的虚拟模型为多自由度机械臂的虚拟模型。
  3. 基于权利要求1-2所述控制系统的控制方法，其特征是，包括如下步骤：
    步骤101、设置牵引手型和脱卸手型；
    步骤102、构建虚拟机械臂和虚拟手势模型并显示在头戴虚拟显示器视景体的前端；
    步骤103、采集双目摄像头的双视角图像；
    步骤104、采用手势检测算法检测，判断双视角图像中是否有操作员的手势存在，如果是，则执行下一步，否则执行步骤103；
    步骤105、采用手型识别算法对手势进行手型识别，判断是否出现了牵引手型，如果是，执行下一步，否则执行步骤103；
    步骤106、对拍摄的双视角图像进行处理并求解牵引手势在可穿戴双目摄像装置坐标系中的位姿P H，将位姿P H转换为在头戴虚拟显示器的屏幕坐标系中的位姿描述P V，采用转化后的位姿P V驱动头戴虚拟显示器视景体中的虚拟手势模型；
    步骤107、判断虚拟手势模型的位姿P V与虚拟机械臂N6末端位姿P V的差是否小于预设阈值，如果是，执行下一步；否则执行步骤103；
    步骤108、使得多自由度机械臂的位姿跟随操作员的牵引手型位姿变化；
    步骤109、判断是否出现脱卸手型，如果是，多自由度机械臂的位姿停止跟随操作员的牵引手型位姿变化，并执行步骤103；否则，执行步骤108。
  4. 如权利要求3所述控制方法，其特征是：所述步骤108使得多自由度机械臂的位姿跟随操作员的牵引手型位姿变化，步骤具体为：
    使得虚拟机械臂末端位姿P M的值与虚拟手势模型的位姿P V相等，求解虚拟机械臂对应的各关节角值；
    根据求解的虚拟机械臂对应的各关节角度值转换为控制指令传输至从端机器人，使得多自由度机械臂各关节的关节角度与虚拟机械臂的各关节角度值相等；
    或/和
    所述步骤108还包括：根据求解的虚拟机械臂N6对应的各关节角度值在视景体中对虚拟机械臂N6进行重绘。
  5. 如权利要求3所述控制方法，其特征是：牵引手势在可穿戴双目摄像装置坐标系中的位姿P H的位置信息和位姿P V的位置信息成正比例关系，位姿P H的姿态信息和位姿P V的姿态信息也成正比例关系。
  6. 如权利要求3所述控制方法，其特征是：所述步骤102构建多虚拟机械臂并显示在头戴虚拟显示器视景体的前端的方法具体为：
    读取从端机器人的多自由度机械臂的各关节的关节角信息；
    遥操作控制器根据采集的关节角信息计算多自由度机械臂的D-H参数；
    根据多自由度机械臂的D-H参数构建虚拟机械臂，并将虚拟机械臂显示在头戴虚拟显示器视景体的前端。
  7. 如权利要求3所述控制方法，其特征是：步骤103之前还包括，在头戴虚拟显示器上显示从端机器人拍摄的图像的步骤：
    采集从端机器人端的侦察图像；
    遥操作控制器接收侦察图像并其实时显示在头戴虚拟显示器的视景体后端。
  8. 基于权利要求1-2任一项所述控制系统的控制方法，其特征是，分别采集操控者的左手和右手的动作，通过一只手的动作控制移动机器人车体的移动，通过另一只手的动作控制移动机器人的车载多自由度机械臂的动作，包括如下步骤：
    步骤201：采集操控者可穿戴设备可拍摄范围内的图像；
    步骤202：判断采集的图像中是否有手部区域，如果没有，执行步骤201；否则，对采集的图像进行预处理，获得手部裁片；
    步骤203：利用左右手判别算法判断获得的手部裁片是左手裁片还是右手裁片，从而确定做动作的是左手还是右手；
    步骤204：通过其中一只手的动作控制移动机器人车体的移动，通过另一只手的动作控制移动机器人的车载多自由度机械臂的动作，然后，执行步骤201。
  9. 如权利要求8所述控制方法，其特征是：步骤204中通过其中一只手的动作控制移动机器人车体的移动，具体步骤为：
    设定从端机器人运动控制指令与手型信息的对应关系；
    当识别的手部裁片为设定为控制移动机器人车体的移动的一只手，采用手型识别算法对手部裁片进行识别，得到手型信息；
    根据从端机器人运动控制指令与手型信息的对应关系以及识别获得的手型信息生成从端 机器人运动控制指令，将运动控制指令发送至从端机器人，从端机器人根据控制指令执行相应的动作。
  10. 如权利要求8所述控制方法，其特征是：步骤204中通过另一只手的动作控制移动机器人的车载多自由度机械臂的动作，具体步骤为：
    当识别的手部裁片为设定为控制移动机器人的车载多自由度机械臂的动作的一只手，采用指尖定位算法分析任意指尖在图像中的运动轨迹；
    根据运动轨迹生成位置跟踪指令，并将位置跟踪指令发送至从端机器人；
    从端机器人根据位置跟踪指令生成具体动作的位置坐标，连杆机构末端依次经过位置坐标实现操控者指尖运动轨迹的跟踪。
  11. 基于权利要求1-2任一项所述控制系统的控制方法，其特征是，机器人车载多自由度机械臂宽范围移动的控制方法包括如下步骤：
    步骤301：设定衔接手型和对应的手势动作，设定不同的手型对应从端机器人不同的动作，衔接手型可以设定为车载多自由度机械臂末端在当前位置等待下一指令；
    步骤302：采集操控者可穿戴设备可拍摄范围内的图像；
    步骤303：判断采集的图像中是否有手部区域，如果没有，执行步骤302；否则，对采集的图像进行预处理，获得手部裁片，并执行下一步；
    步骤304：采用手型识别算法对预处理后的手部裁片进行手型识别，得到手型信息；
    步骤305：判断获得的手型信息是否为衔接手型，如果是，车载多自由度机械臂末端连续执行衔接手型前一手型和衔接手型后一手型的对应的控制指令的动作，并执行步骤302；否则,执行下一步；
    步骤306：按照相应的手型执行相应的动作，并执行步骤302。
  12. 如权利要求11所述控制方法，其特征是：步骤305中，车载多自由度机械臂末端连续执行衔接手型前一手型和衔接手型后一手型的对应的控制指令的动作，具体为：
    车载多自由度机械臂末端执行前一手型对应的动作后，停止在当前位置；
    执行步骤302-步骤304，检测到的衔接手型的下一手型，车载多自由度机械臂末端从当前位置动作执行衔接手型的下一手型对应的动作。
  13. 一种电子设备，其特征是，包括存储器和处理器以及存储在存储器上并在处理器上运行的计算机指令，所述计算机指令被处理器运行时，完成权利要求3-12任一项方法所述步骤。
  14. 一种计算机可读存储介质，其特征是，用于存储计算机指令，所述计算机指令被处理器执行时，完成权利要求3-12任一项方法所述步骤。
PCT/CN2020/087846 2019-04-30 2020-04-29 基于可穿戴设备的移动机器人控制系统及控制方法 WO2020221311A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
KR1020207030337A KR102379245B1 (ko) 2019-04-30 2020-04-29 웨어러블 디바이스 기반의 이동 로봇 제어 시스템 및 제어 방법

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN201910363168.1A CN110039545B (zh) 2019-04-30 2019-04-30 一种基于可穿戴设备的机器人远程控制系统及控制方法
CN201910363155.4A CN109955254B (zh) 2019-04-30 2019-04-30 移动机器人控制系统及机器人末端位姿的遥操作控制方法
CN201910363155.4 2019-04-30
CN201910363168.1 2019-04-30

Publications (1)

Publication Number Publication Date
WO2020221311A1 true WO2020221311A1 (zh) 2020-11-05

Family

ID=73028793

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/087846 WO2020221311A1 (zh) 2019-04-30 2020-04-29 基于可穿戴设备的移动机器人控制系统及控制方法

Country Status (2)

Country Link
KR (1) KR102379245B1 (zh)
WO (1) WO2020221311A1 (zh)

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113099204A (zh) * 2021-04-13 2021-07-09 北京航空航天大学青岛研究院 一种基于vr头戴显示设备的远程实景增强现实方法
CN113218249A (zh) * 2021-05-30 2021-08-06 中国人民解放军火箭军工程大学 跟随式遥操作战车及控制方法
CN113229941A (zh) * 2021-03-08 2021-08-10 上海交通大学 基于增强现实的介入机器人无接触遥操系统及标定方法
CN113741785A (zh) * 2021-08-27 2021-12-03 深圳Tcl新技术有限公司 指令确定方法、装置、存储介质及电子设备
CN113768630A (zh) * 2021-08-06 2021-12-10 武汉中科医疗科技工业技术研究院有限公司 主手夹持机构、主手控制台、手术机器人及主从对齐方法
CN114378823A (zh) * 2022-01-20 2022-04-22 深圳市优必选科技股份有限公司 一种机器人动作控制方法、装置、可读存储介质及机器人
CN114578720A (zh) * 2020-12-01 2022-06-03 合肥欣奕华智能机器股份有限公司 控制方法及控制系统
CN114643576A (zh) * 2020-12-17 2022-06-21 中国科学院沈阳自动化研究所 一种基于虚拟力引导的人机协同目标抓取方法
CN114683272A (zh) * 2020-12-31 2022-07-01 国网智能科技股份有限公司 变电站巡检机器人的增稳控制方法、控制器及机器人
CN114770583A (zh) * 2022-04-29 2022-07-22 大连工业大学 一种基于vr的智能装配系统
CN115157261A (zh) * 2022-07-27 2022-10-11 清华大学深圳国际研究生院 基于混合现实的柔性机械臂遥操作人机交互装置及方法
US20230037237A1 (en) * 2021-07-19 2023-02-02 Colorado School Of Mines Gesture-controlled robotic feedback
CN116052500A (zh) * 2023-01-31 2023-05-02 苏州安全精灵智能科技有限公司 口罩防护体验方法、电子设备、系统及可读存储介质

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102522142B1 (ko) * 2021-07-05 2023-04-17 주식회사 피앤씨솔루션 양손 제스처를 이용해 조작 신호를 입력하는 착용형 증강현실 장치 및 양손 제스처를 이용한 착용형 증강현실 장치의 조작 방법
KR102532351B1 (ko) * 2021-08-05 2023-05-15 서울대학교병원 헤드셋 기반의 비접촉 손동작 인식 기술을 활용한 수술 로봇 제어 시스템
KR102549631B1 (ko) * 2022-07-21 2023-07-03 주식회사 포탈301 신체 일부의 자세 및 장치의 기울기를 이용한 실시간 작업 장치 및 카메라 제어 방법 및 장치
KR102525661B1 (ko) * 2023-01-18 2023-04-24 박장준 작업 장치의 원격 제어를 위한 실시간 훈련 방법 및 장치

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103279191A (zh) * 2013-06-18 2013-09-04 北京科技大学 一种基于手势识别技术的3d虚拟交互方法及系统
CN103398702A (zh) * 2013-08-05 2013-11-20 青岛海通机器人系统有限公司 一种移动机器人远程操控装置及其操控技术
CN104057450A (zh) * 2014-06-20 2014-09-24 哈尔滨工业大学深圳研究生院 一种针对服务机器人的高维操作臂遥操作方法
JP2016107379A (ja) * 2014-12-08 2016-06-20 ファナック株式会社 拡張現実対応ディスプレイを備えたロボットシステム
CN109955254A (zh) * 2019-04-30 2019-07-02 齐鲁工业大学 移动机器人控制系统及机器人末端位姿的遥操作控制方法
CN110039545A (zh) * 2019-04-30 2019-07-23 齐鲁工业大学 一种基于可穿戴设备的机器人远程控制系统及控制方法

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011110621A (ja) * 2009-11-24 2011-06-09 Toyota Industries Corp ロボットの教示データを作成する方法およびロボット教示システム
CN107921639B (zh) * 2015-08-25 2021-09-21 川崎重工业株式会社 多个机器人系统间的信息共享系统及信息共享方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103279191A (zh) * 2013-06-18 2013-09-04 北京科技大学 一种基于手势识别技术的3d虚拟交互方法及系统
CN103398702A (zh) * 2013-08-05 2013-11-20 青岛海通机器人系统有限公司 一种移动机器人远程操控装置及其操控技术
CN104057450A (zh) * 2014-06-20 2014-09-24 哈尔滨工业大学深圳研究生院 一种针对服务机器人的高维操作臂遥操作方法
JP2016107379A (ja) * 2014-12-08 2016-06-20 ファナック株式会社 拡張現実対応ディスプレイを備えたロボットシステム
CN109955254A (zh) * 2019-04-30 2019-07-02 齐鲁工业大学 移动机器人控制系统及机器人末端位姿的遥操作控制方法
CN110039545A (zh) * 2019-04-30 2019-07-23 齐鲁工业大学 一种基于可穿戴设备的机器人远程控制系统及控制方法

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114578720B (zh) * 2020-12-01 2023-11-07 合肥欣奕华智能机器股份有限公司 控制方法及控制系统
CN114578720A (zh) * 2020-12-01 2022-06-03 合肥欣奕华智能机器股份有限公司 控制方法及控制系统
CN114643576A (zh) * 2020-12-17 2022-06-21 中国科学院沈阳自动化研究所 一种基于虚拟力引导的人机协同目标抓取方法
CN114643576B (zh) * 2020-12-17 2023-06-20 中国科学院沈阳自动化研究所 一种基于虚拟力引导的人机协同目标抓取方法
CN114683272B (zh) * 2020-12-31 2023-09-12 国网智能科技股份有限公司 变电站巡检机器人的增稳控制方法、控制器及机器人
CN114683272A (zh) * 2020-12-31 2022-07-01 国网智能科技股份有限公司 变电站巡检机器人的增稳控制方法、控制器及机器人
CN113229941A (zh) * 2021-03-08 2021-08-10 上海交通大学 基于增强现实的介入机器人无接触遥操系统及标定方法
CN113099204B (zh) * 2021-04-13 2022-12-13 北京航空航天大学青岛研究院 一种基于vr头戴显示设备的远程实景增强现实方法
CN113099204A (zh) * 2021-04-13 2021-07-09 北京航空航天大学青岛研究院 一种基于vr头戴显示设备的远程实景增强现实方法
CN113218249A (zh) * 2021-05-30 2021-08-06 中国人民解放军火箭军工程大学 跟随式遥操作战车及控制方法
CN113218249B (zh) * 2021-05-30 2023-09-26 中国人民解放军火箭军工程大学 跟随式遥操作战车及控制方法
US20230037237A1 (en) * 2021-07-19 2023-02-02 Colorado School Of Mines Gesture-controlled robotic feedback
CN113768630A (zh) * 2021-08-06 2021-12-10 武汉中科医疗科技工业技术研究院有限公司 主手夹持机构、主手控制台、手术机器人及主从对齐方法
CN113741785A (zh) * 2021-08-27 2021-12-03 深圳Tcl新技术有限公司 指令确定方法、装置、存储介质及电子设备
CN114378823A (zh) * 2022-01-20 2022-04-22 深圳市优必选科技股份有限公司 一种机器人动作控制方法、装置、可读存储介质及机器人
CN114378823B (zh) * 2022-01-20 2023-12-15 深圳市优必选科技股份有限公司 一种机器人动作控制方法、装置、可读存储介质及机器人
CN114770583A (zh) * 2022-04-29 2022-07-22 大连工业大学 一种基于vr的智能装配系统
CN115157261A (zh) * 2022-07-27 2022-10-11 清华大学深圳国际研究生院 基于混合现实的柔性机械臂遥操作人机交互装置及方法
CN116052500A (zh) * 2023-01-31 2023-05-02 苏州安全精灵智能科技有限公司 口罩防护体验方法、电子设备、系统及可读存储介质

Also Published As

Publication number Publication date
KR20200140834A (ko) 2020-12-16
KR102379245B1 (ko) 2022-03-28

Similar Documents

Publication Publication Date Title
WO2020221311A1 (zh) 基于可穿戴设备的移动机器人控制系统及控制方法
CN109955254B (zh) 移动机器人控制系统及机器人末端位姿的遥操作控制方法
CN110039545B (zh) 一种基于可穿戴设备的机器人远程控制系统及控制方法
WO2023056670A1 (zh) 复杂光照条件下基于视触融合的机械臂自主移动抓取方法
Krupke et al. Comparison of multimodal heading and pointing gestures for co-located mixed reality human-robot interaction
CN109164829B (zh) 一种基于力反馈装置和vr感知的飞行机械臂系统及控制方法
CN114080583B (zh) 视觉教导和重复移动操纵系统
US20200055195A1 (en) Systems and Methods for Remotely Controlling a Robotic Device
KR101762638B1 (ko) 최소 침습 수술 시스템에서 손 제스처 제어를 위한 방법 및 장치
KR101789064B1 (ko) 원격조종 최소 침습 종속 수술 기구의 손 제어를 위한 방법 및 시스템
KR101785360B1 (ko) 최소 침습 수술 시스템에서 손 존재 검출을 위한 방법 및 시스템
US8155787B2 (en) Intelligent interface device for grasping of an object by a manipulating robot and method of implementing this device
CN109983510A (zh) 机器人控制系统、机械控制系统、机器人控制方法、机械控制方法和记录介质
CN114728417A (zh) 由远程操作员触发的机器人自主对象学习
CN106313049A (zh) 一种仿人机械臂体感控制系统及控制方法
CN111459277B (zh) 基于混合现实的机械臂遥操作系统及交互界面构建方法
CN113183133B (zh) 面向多自由度机器人的手势交互方法、系统、装置及介质
CN113021357A (zh) 一种便于移动的主从水下双臂机器人
CN112828916A (zh) 冗余机械臂遥操作组合交互装置及冗余机械臂遥操作系统
CN108062102A (zh) 一种手势控制具有辅助避障功能的移动机器人遥操作系统
US20240149458A1 (en) Robot remote operation control device, robot remote operation control system, robot remote operation control method, and program
CN115958575A (zh) 类人灵巧操作移动机器人
CN112959342B (zh) 基于操作员意图识别的飞行机械臂抓取作业遥操作方法
Bai et al. Kinect-based hand tracking for first-person-perspective robotic arm teleoperation
KR101956900B1 (ko) 최소 침습 수술 시스템에서 손 존재 검출을 위한 방법 및 시스템

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 20207030337

Country of ref document: KR

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20798282

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20798282

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20798282

Country of ref document: EP

Kind code of ref document: A1