CN109129523B - Mobile robot real-time remote control system based on human-computer interaction - Google Patents
Mobile robot real-time remote control system based on human-computer interaction Download PDFInfo
- Publication number
- CN109129523B CN109129523B CN201811004647.6A CN201811004647A CN109129523B CN 109129523 B CN109129523 B CN 109129523B CN 201811004647 A CN201811004647 A CN 201811004647A CN 109129523 B CN109129523 B CN 109129523B
- Authority
- CN
- China
- Prior art keywords
- sensor
- axis
- main control
- head
- manipulator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/006—Controls for manipulators by means of a wireless system for controlling one or several manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/003—Controls for manipulators by means of an audio-responsive input
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
- B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
Abstract
The invention relates to a real-time remote control system of a mobile robot based on human-computer interaction. The local main control terminal and the far-end executor carry out information bidirectional transmission in real time through a communication system. In the local main control terminal, a head attitude sensor collects head attitude information of an operator and sends the head attitude information to a remote main control computer of a remote actuator for processing, an attitude control signal of a two-degree-of-freedom mechanical arm with a camera mounted at the top end is generated, the action attitude of the mechanical arm is consistent with the head action attitude of the operator, the camera transmits the collected image information back to the local main control terminal, and the image information is displayed on a head-mounted VR display after being processed; the sensor is arranged on the far-end actuator with the manipulator, collects and processes far-end field environment information and state information of the actuator, and transmits the far-end field environment information and the state information of the actuator back to the local main control terminal for displaying, so that an operator can conveniently control the far-end actuator through the control handle.
Description
Technical Field
The invention belongs to the technical field of remote unmanned electronic control, and relates to a mobile robot real-time remote control system based on human-computer interaction.
Background
At present, with the rapid development of artificial intelligence technology, the control of various mechanical devices is being changed from remote teleoperation control to autonomous machine operation control, the technologies of big data, cloud computing, data mining, machine learning and the like are rapidly developed, and intelligent manufacturing is promoted to be a national strategy and becomes a global hotspot. However, in many cases, the autonomous execution by using a machine has a great defect, manual remote participation is still needed, the remote teleoperation technology still occupies an irreplaceable position, and the remote teleoperation technology cannot be used in some high-risk environment operation scenes such as earthquake relief work, flood fighting and emergency rescue, high-altitude object hanging operation, deep well excavation operation, deep sea fishing operation and the like and some entertainment occasions such as aquarium sightseeing, shallow sea shooting and the like. At present, the prior art has the defects of complex implementation method, difficult maintenance, higher cost and certain defects.
Disclosure of Invention
Aiming at the defects of the prior art, the embedded real-time remote unmanned operation control system is easy to realize by hardware, low in cost, strong in embeddability and good in real-time performance, is suitable for being applied to various environments, enables an operator to monitor the condition of a remote working site in real time through image, sound and sensor data information at a local main control terminal through real-time bidirectional information transmission between the local main control machine and a remote main control machine, makes judgment, and utilizes a control handle in a hand to accurately control the motion of a four-wheel drive platform and the motion of a manipulator at the remote site in real time, so that the remote unmanned operation with more convenience, accuracy, visualization, interaction and low cost is realized.
The invention is realized by the following steps:
a real-time remote control system of a mobile robot based on human-computer interaction comprises a local main control terminal, a communication system and a far-end actuator, wherein the communication system is respectively connected with the local main control terminal and the far-end actuator so as to realize real-time bidirectional communication between the local main control terminal and the far-end actuator; the communication system comprises wired communication and wireless communication; the local main control terminal comprises a local main control computer, a head-wearing VR display, a head posture sensor, an earphone, an external sound card and a control handle, wherein the head-wearing VR display, the head posture sensor, the earphone, the external sound card and the control handle are respectively connected with the local main control computer; the far-end actuator comprises a far-end main control computer, a video and audio acquisition mechanism, a motion mechanism, a manipulator actuating mechanism and a site environment and device state sensing mechanism, the video and audio acquisition mechanism, the movement mechanism, the manipulator execution mechanism and the site environment and device state sensing mechanism are respectively connected with the remote main control computer, the video and audio acquisition mechanism comprises a camera, a camera attitude control board card, a two-degree-of-freedom mechanical arm, a sound pickup and an external sound card, the motion mechanism comprises a motion control board card, an inertial sensor and a four-wheel drive platform, the manipulator execution mechanism comprises a manipulator action control board card and a manipulator, the manipulator comprises a mechanical arm and a mechanical claw, and the field environment and device state sensing mechanism comprises a field environment information sensor, a device state information sensor and a data acquisition and processing board card; the head attitude sensor is arranged on a helmet worn by an operator, accurate attitude information of the head of the operator is obtained by fusing multi-sensor information and is sent to the far-end actuator through the communication system, the far-end main control computer transmits the attitude information to the camera attitude control board card for processing, attitude control signals of the two-degree-of-freedom mechanical arm are generated, and the swing attitude of the two-degree-of-freedom mechanical arm is consistent with the head attitude of the operator; the control handle is provided with a motion mechanism for controlling the far-end actuator and a control rocker and a knob of a manipulator actuating mechanism, a control signal given by the control handle is sent to the far-end main control machine through the local main control machine through the communication system and is respectively sent to the motion control board card and the manipulator action control board card, the control rocker gives a motion control signal of the motion mechanism, the knob gives a control signal of a manipulator swinging angle and a manipulator opening and closing angle of the manipulator actuating mechanism, the motion control board card of the motion mechanism receives the motion control signal given by the control rocker and converts the motion control signal into corresponding four paths of PWM waves to control the rotating speed and the forward and reverse rotation of four direct current motors of the four-wheel drive platform, and meanwhile, an inertial sensor connected to the motion control board card collects and feeds back the linear acceleration and the turning angle speed information of the motion of the four-wheel drive platform, further realizing closed-loop control on the forward movement, the backward movement and the steering of the four-wheel drive platform; after receiving the angle control signal given by the knob, a manipulator action control plate of the manipulator actuating mechanism converts the angle control signal into three corresponding paths of PWM waves, controls three steering engines of a manipulator and a gripper of the manipulator and further controls the swinging angle of the manipulator and the opening and closing angle of the gripper; after the data acquisition and processing board card processes the field environment information acquired by the field environment information sensor and the equipment state information data acquired by the device state sensor, the data acquisition and processing board card is sent to the local main control computer by the remote main control computer through the communication system and displayed in the head-mounted VR display; the camera is fixed on the top end of the two-degree-of-freedom mechanical arm, a field picture shot by the camera is compressed by the far-end main control computer and then transmitted to the local main control computer for decompression, the compressed field picture is displayed by the head-mounted VR display, the bottom end of the two-degree-of-freedom mechanical arm and the far-end main control computer are fixed on a damping platform, the damping platform is installed on the four-wheel drive platform, the far-end main control computer collects field audio data through the sound pickup and the external sound card, sends the collected field audio data to the local main control computer through a communication system, and the collected field audio data is played through the earphone and the external sound card.
Preferably, the communication system implements wireless communication through a pair of wireless bridges, and implements wired communication through the power carrier module and the power carrier line.
Preferably, the head posture sensor is fixed at the top end and both sides of the helmet.
Preferably, the camera is a drive-free high-definition infrared monocular camera capable of night vision.
Preferably, the data acquisition and processing board card is externally connected with a plurality of sensors, including an environment temperature and humidity sensor, an air pressure height sensor, an illumination intensity sensor, an oil tank liquid level sensor, a motor load current sensor and a vibration sensor.
Compared with the prior art, the invention has the following beneficial effects:
the invention discloses a real-time remote control system of a mobile robot based on human-computer interaction. The head attitude sensor of the local main control terminal collects head pitching and horizontal rotation information of an operator, the head pitching and horizontal rotation information is received by the local main control computer and is sent to the remote actuator through the communication system, a camera attitude control board card of the video and audio acquisition mechanism generates a control signal of the two-degree-of-freedom mechanical arm, the action attitude of the two-degree-of-freedom mechanical arm is consistent with the action attitude of the head of the operator, a camera which is mounted at the top end of the two-degree-of-freedom mechanical arm and follows the two-degree-of-freedom mechanical arm shoots working site pictures in real time, image information is sent to the local main control computer of the local main control terminal through the communication system by the remote main control computer of the remote actuator, and the image information is displayed on. The video and audio acquisition mechanism of the far-end actuator acquires sound information of a working site through the pickup and the external sound card, the sound information is sent to the local main control computer through the far-end main control computer through the communication system, and the audio information is played by the earphone connected with the local main control computer and the external sound card, so that the function of acquiring audio of the far-end working site can be realized. The data acquisition and processing board card connected with the remote main control computer can be connected with various sensors according to actual needs to realize real-time detection of on-site environment information and self working state information of the equipment, and the sensor information is processed and then sent back to the local main control computer through the remote main control computer through a communication system to be displayed on an interface. Meanwhile, local operators can accurately control the motion of the four-wheel drive platform and the motion of the manipulator on the far-end site in real time by using a control handle in a hand according to the video, audio and sensing information of the far-end site transmitted back in real time. According to the invention, the AR follow-up platform with multiple information acquisition functions is arranged on the remote unmanned operation equipment, and the remote unmanned operation which is more convenient, more accurate, visual, interactive, low-cost and low-delay is realized by acquiring various information such as on-site images, sounds, equipment states and environmental parameters and displaying the information on the head-mounted VR display of the local main control terminal, so that an operator can safely, accurately and efficiently carry out the remote unmanned operation.
Drawings
FIG. 1 is a schematic structural diagram of a real-time remote control system of a mobile robot based on human-computer interaction according to the present invention;
FIG. 2 is a wiring diagram of the real-time remote control system of the present invention;
FIG. 3 is a position of the head position sensor of the present invention on a helmet;
fig. 4 is a block diagram of the motion mechanism control of the present invention.
Detailed Description
Exemplary embodiments, features and performance aspects of the present invention will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
As shown in fig. 1, a mobile robot real-time remote control system based on human-computer interaction includes a local main control terminal 1, a communication system 2 and a remote actuator 3, where the communication system 2 is connected to the local main control terminal 1 and the remote actuator 3 respectively, so as to implement real-time bidirectional communication between the local main control terminal 1 and the remote actuator 3; the communication system 2 comprises wired communication and wireless communication, wireless communication is realized through a pair of wireless bridges, and wired communication is realized through a power carrier module and a power carrier line; the local main control terminal 1 comprises a local main control computer 11, a head-wearing VR display 13, a head attitude sensor 12, an earphone and external sound card 14 and a control handle 15, the head-wearing VR display 13, the head-wearing VR display 12, the earphone and external sound card 14 and the control handle 15 are respectively connected with the local main control computer 11, a far-end actuator 3 comprises a far-end main control computer 31, a video and audio acquisition mechanism, a movement mechanism, a manipulator execution mechanism and a site environment and device state sensing mechanism, the video and audio acquisition mechanism, the movement mechanism, the manipulator execution mechanism and the site environment and device state sensing mechanism are respectively connected with the far-end main control computer 31, the video and audio acquisition mechanism comprises a camera 32, a two-degree-of-freedom mechanical arm 33, a camera attitude control board 34, a sound pickup and external sound card 35, the movement mechanism comprises a movement control board 38, an inertial sensor 37 and a four-wheel drive platform 36, the manipulator execution mechanism comprises a manipulator action control board card 40 and a manipulator 43, the manipulator 43 comprises a mechanical arm and a mechanical claw, and the site environment and device state sensing mechanism comprises a site environment information sensor 41, a device state information sensor 44 and a data acquisition and processing board card 39; the head attitude sensor 12 is mounted on a helmet worn by an operator, the accurate attitude information of the head of the operator obtained by information fusion of multiple sensors is sent to the remote actuator 3 through the communication system 2, the remote master control machine 31 transmits the information to the camera attitude control board 34 for processing, an attitude control signal of the two-degree-of-freedom mechanical arm 33 is generated, and the swing attitude of the two-degree-of-freedom mechanical arm 33 is consistent with the head attitude of the operator; the control handle 15 is provided with a control rocker and a knob for controlling the motion mechanism of the remote actuator 3 and the manipulator actuator, a control signal given by the control handle is sent to the remote master control machine 31 through the local master control machine 11 via the communication system 2 and is respectively sent to the motion control board card 38 and the manipulator action control board card 40, the control rocker gives a motion control signal of the motion mechanism, the knob gives a control signal of the swing angle of the manipulator actuator and the opening and closing angle of the mechanical claw, the motion control board card 38 of the motion mechanism receives the motion signal given by the control handle 15 and converts the motion signal into corresponding four-way PWM waves to control the rotating speed and the forward and reverse rotation of the four direct current motors of the four-wheel drive platform 36, and meanwhile, an inertial sensor 37 connected to the motion control board card 38 collects and feeds back the linear acceleration and the turning angle speed information of the motion of the four-wheel drive platform 36, further realizing closed-loop control of forward, backward and steering of the four-wheel drive platform 36; after receiving an angle control signal given by the control handle 15, a manipulator action control board card 40 of the manipulator actuating mechanism converts the angle control signal into three corresponding paths of PWM waves, controls a manipulator 43 of the manipulator and three steering engines of the mechanical jaws, and further controls the swinging angle of the manipulator and the opening and closing angle of the mechanical jaws; after the data acquisition and processing board 39 processes the field environment information acquired by the field environment information sensor 41 and the device state information data acquired by the device state information sensor 44, the processed data is sent to the local main control computer 11 by the remote main control computer 31 through the communication system 2, and is displayed in the head-mounted VR display 13; the camera 32 is fixed on the top end of the two-degree-of-freedom mechanical arm 33, a field picture shot by the camera 32 is compressed by the far-end main control computer 31 and then transmitted to the local main control computer 11 for decompression, and is displayed by the head-worn VR display 13, the bottom end of the two-degree-of-freedom mechanical arm 33 and the far-end main control computer 31 are fixed on the four-wheel drive platform 36, the far-end main control computer 31 collects field audio data through a sound pickup and an external sound card 35, the audio data are sent to the local main control computer 11 through the communication system 2, and the audio data are played by an earphone and the external sound card 14.
Preferably, the camera 32 is a night-vision drive-free high-definition infrared monocular camera, so that the robot can work normally at night or in the absence of light.
Preferably, the data acquisition and processing board card 39 is externally connected with various sensors, including an environment temperature and humidity sensor, an air pressure height sensor, an illumination intensity sensor, an oil tank liquid level sensor motor load current sensor, a vibration sensor and the like, and is used for acquiring field environment information and equipment working state information, each sensor can be flexibly embedded into each equipment of the remote actuator, and the sensors can be selectively used according to specific field working requirements and have flexibility.
Preferably, the local main control computer and the remote main control computer of the invention operate a human-computer interaction software system which is automatically developed by using a QT platform under a Linux system. The hardware part of the remote control system has good anti-seismic property and dustproof and explosion-proof property, all the fixing screws are reinforced by adopting a high-strength anaerobic type thread locking agent, and all the components are sealed by adopting epoxy resin.
In this embodiment, as shown in fig. 2, the local master control machine 11 of the local master control terminal 1 and the remote master control machine 31 of the remote actuator 3 both use a 3 rd generation enhanced raspberry group, and through bidirectional information transmission between the local master control machine 11 and the remote master control machine 31, an operator can monitor the status of a remote work site in real time through image, sound and sensor data information at the local master control terminal 1, make a judgment, and perform real-time and accurate control on the motion of the four-wheel drive platform and the motion of the manipulator at the remote site by using a control handle in a hand, thereby implementing more convenient, accurate visualization, interactivity, low cost, and low-delay remote unmanned operation. The local main control computer 11 and the remote main control computer 31 are both provided with 100Mb bandwidth network cable interfaces, and according to different application scenarios of the system, the communication system 2 may adopt a wired or wireless communication mode, wherein the wireless communication mode is realized by a pair of 5.8GHz wireless network bridges, the wired communication mode is realized by a power carrier module and a power carrier cable, and in both communication modes, the communication system is connected with the local main control computer 11 and the remote main control computer 31 through the network cable. The local main control computer 11 is externally connected with a head-wearing VR display 13 through an HDMI interface, is externally connected with an external sound card and an earphone through a 3.5mm audio interface, and is respectively externally connected with an external posture sensor 12 and a control handle 15 through two USB interfaces. The local main control computer 11 and the remote main control computer 31 participate in data processing and transmission; the head-mounted VR display 13 is responsible for displaying images and data; the earphone and external sound card 14 plays the field audio data collected by the remote main control computer 31 through the sound pickup and external sound card 35; the head attitude sensor 12 is installed at different positions of a helmet worn by an operator, accurate head attitude information of the operator is obtained by adopting a multi-sensor information fusion technology, the acquired head attitude information of the operator is transmitted to the local main control computer 11 through a USB-to-TTL serial port line CH340 and is stored in a data sending buffer zone of the local main control computer 11, and the head attitude information is sent to the camera attitude control board card 34 of the video and audio acquisition mechanism after being sent to the far-end main control computer 31 through the communication system 2. The control handle 15 is provided with a remote end actuator movement mechanism and a control rocker and a knob of a manipulator execution mechanism, the voltage of a potentiometer end in the rocker and the knob is sampled through an ADC (analog to digital converter), and is converted into original digital quantity required by the movement mechanism and the manipulator execution mechanism, furthermore, a local operator can give control signals of forward movement, backward movement, left rotation, right rotation and the like of the remote end movement mechanism through two rockers on the control handle, give control signals of a mechanical arm swinging angle and a mechanical claw opening and closing angle of the remote end manipulator execution mechanism through rotating the knobs of three potentiometers on the control handle 15, and the control signals given by the control handle 15 are also transmitted to the local main control computer 11 through a USB (universal serial bus) to TTL (transistor-transistor logic) serial port line CH340 and are stored in a data buffer zone sent by the local main control computer 11.
In the 3 parts of far-end executor, camera attitude control integrated circuit board 34, motion control integrated circuit board 38 and manipulator action control integrated circuit board 40 in the far-end executor all adopt embedded STM32 development board, all communicate through USB commentaries on classics TTL serial port line CH340 between each integrated circuit board and far-end main control computer 31, the adapter directly is connected through the USB interface between external sound card 35 and far-end main control computer 31, data acquisition and processing integrated circuit board 39 utilizes the serial ports of far-end raspberry group self-band to carry out the upload of data.
In the far-end actuator video/audio acquisition mechanism, the camera attitude control board 34 receives the head attitude sensor data in the data buffer area sent by the local master control machine 11, and converts the data into a control signal of the two-degree-of-freedom mechanical arm motor to control the rotation of the two-degree-of-freedom mechanical arm in the horizontal and pitching directions, so that the action of the two-degree-of-freedom mechanical arm 33 can simulate the action of the head of an operator in real time. The camera 32 adopts a CSI interface camera matched with the raspberry pi, is directly connected with a remote-end raspberry pi CSI interface through a 15-pin CSI flat cable, and directly performs video acquisition through the remote-end raspberry pi; the adapter directly links to each other with the USB interface of external sound card and far-end raspberry group, directly carries out the collection of audio frequency through the far-end raspberry group.
In the far-end actuator movement mechanism, an ADRC controller is adopted, after receiving a movement signal given by a control handle 15 in a data buffer area sent by a local main control computer 11, a movement control board card 38 quantizes and correspondingly processes the movement signal, converts the movement signal into corresponding four paths of PWM waves, controls the rotating speed and the forward and reverse rotation of four direct current motors of a four-wheel drive platform 36, and simultaneously, an inertial sensor 37 connected to the movement control board card 38 acquires and feeds back linear acceleration and turning angle speed information of the movement of the four-wheel drive platform 36, thereby realizing accurate closed-loop control of the forward, backward and steering of the four-wheel drive platform 36.
In the manipulator execution mechanism of the remote actuator, after receiving a given action signal of a control handle in a data buffer area sent by the local main control machine 11, the manipulator action control board 40 also quantizes and correspondingly processes the action signal, converts the action signal into corresponding three paths of PWM waves, controls the three steering engines of the manipulator 43 and the manipulator claw, and further realizes the control of the swing angle of the manipulator and the opening and closing angle of the manipulator claw.
In the far-end actuator field environment and device state sensing mechanism, the data acquisition and processing board card 39 processes the acquired environment information and equipment state information data, and then the processed data is transmitted to the data transmission buffer area of the far-end main control computer 31 through the serial port, and is transmitted to the local main control computer 11 through the communication system, and is displayed in the head-mounted VR display 13.
In the communication system 2, the local main control computer 11 and the remote main control computer 31 are both provided with 100Mb bandwidth network cable interfaces, and a TCP/IP protocol is adopted to realize data bidirectional transmission between the data sending and receiving buffer of the local main control computer and the data sending and receiving buffer of the remote main control computer. In addition, according to different application scenes of the system, the communication system can selectively adopt wired or wireless communication modes, for example, the communication system is used on the land and can adopt a 5.8GHz wireless network bridge for communication, if the communication system is used underwater, a pair of power carrier modules are adopted for communication through a zero-buoyancy power carrier line, and particularly which communication mode is adopted needs to be selected according to the actual situation, so that the flexibility is strong.
As shown in fig. 3, the head posture sensors 12 are fixed at the top and both sides of the helmet, and the accurate acquisition of the head posture of the operator is realized by three head posture sensors through data fusion and a kalman filter algorithm. Wherein, the No. 2 head attitude sensor at the top end of the helmet is used as a main sensor, and the No. 1 and No. 3 head attitude sensors at the two ends are used as auxiliary sensors. Each head attitude sensor consists of an embedded STM32 development board and an inertial navigation module GY-901 which communicate with each other in an IIC synchronous communication mode, wherein the GY-901 module comprises three-axis gyroscope sensors, three-axis accelerometer sensors and three-axis magnetometer sensors. The directions of the three axes of the three-axis gyro sensor of the main sensor and the auxiliary sensor are set as follows: in the No. 2 main sensor, the Z axis is vertically directed upwards, the X axis is horizontally directed right ahead, the Y axis is perpendicular to the X axis and the Z axis, the Y axis is horizontally directed to the left end, the Z axis is used for measuring the horizontal rotation angle of the head, and the Y axis is used for measuring the pitching angle of the head; in the auxiliary sensor No. 1, the Z axis is vertically directed downwards, the Y axis is horizontally directed backwards, the X axis is perpendicular to the Z axis and the Y axis, the X axis is horizontally directed to the right end, the Z axis is utilized to obtain an angle value of horizontal rotation of the head with a sign opposite to that measured by the Z axis of the main sensor, and the X axis is utilized to obtain a head pitch angle value with a sign opposite to that measured by the Y axis of the main sensor; in the No. 3 auxiliary sensor, the Z axis is vertically directed downwards, the Y axis is horizontally directed forwards, the X axis is perpendicular to the Z axis and the Y axis, the X axis is horizontally directed to the left end, the Z axis is utilized to obtain an angle value of horizontal rotation of the head with a sign opposite to that measured by the Z axis of the main sensor, and the X axis is utilized to obtain a head pitch angle value with a sign same as that measured by the Y axis of the main sensor; in the three head attitude sensors, quaternion attitude calculation is carried out by utilizing output values of a gyroscope to obtain Euler angles in three directions, the magnetometer is matched with the accelerometer to obtain another group of Euler angles in the three directions, effective angles capable of representing head attitude are extracted, and then complementary fusion processing is carried out on the obtained two groups of head attitude data. The STM32 development board of the auxiliary sensor respectively collects and filters corresponding sub-module data and complementarily fuses head attitude angles obtained in different calculation modes, IIC communication is utilized to collect data to the STM32 development board of the main sensor and fuse the data, then the processed head attitude angle data are sent to a data sending buffer area of a local main control computer through a CH340 serial port line, and the data are sent to a far-end actuator 3 through a communication system 2.
As shown in fig. 4, for a control block diagram of a system motion mechanism, in the design of a control algorithm of the system motion mechanism, an ADRC controller is adopted by a controller, and for two second-order subsystems of linear speed control and turning angle control of a four-wheel drive platform, three-order Extended State Observers (ESO) are respectively adopted to estimate variables other than two state quantities and total disturbance and compensate in the controller, so as to achieve the purposes of fast response of the system and automatic suppression of disturbance.
Two rocker potentiometers which are independent of each other are arranged on a control handle 15 of the local main control terminal, one of the rocker potentiometers is responsible for giving a linear motion speed signal of the four-wheel drive platform, the other rocker potentiometer is responsible for giving a left-right steering angle signal of the four-wheel drive platform, and the two control signals are original digital signals obtained by sampling the voltage at the ends of the potentiometers by using an ADC (analog-to-digital converter).
After receiving the original control signals of the four-wheel drive platform given by the two rockers, the local main control computer sends the original control signals to the remote main control computer 31 through the communication system 2, and then the original control signals are transmitted to the motion control board card through a CH340 serial port line.
In the design of the control algorithm of the system movement mechanism, the design process of the controller is as follows:
quantizing and amplitude limiting the received original control signal in a motion control board card, and converting the signal into a motion speed given signal u of a four-wheel drive platform controller1 *(t) and a steering angle given signal u2 *(t), hereinafter simply referred to as speed given and angle given. The output quantity of the system, namely the motion turning angle a and the linear velocity u of the four-wheel drive platform, is obtained by adopting inertial navigation sensors arranged on the four-wheel drive platform through attitude calculation and data processing, and is referred to as the current turning angle and the current linear velocity in the following.
Further, for the four-wheel drive platform linear speed control section, u is given for the speed1 *(t) arranging the transition u11(t) and extracting its differential signal u12(t), taking the speed and acceleration of the linear motion of the four-wheel drive platform as two state variables x of the linear motion1And x2Expanding the interference amount suffered by the four-wheel drive platform during linear motion into a third state variable x3Estimating estimated values z of three states of linear motion of the four-wheel drive platform according to the linear velocity input signal U1(t) of the four-wheel drive platform and the current linear velocity signal U11、z12And z13Wherein the estimated value z11、z12As feedback values of the current linear velocity and acceleration, and u11(t) and u12(t) respectively making difference to obtain error signals e of linear velocity and acceleration11(t) and e12(t) and then by the pair e11(t) and e12(t) obtaining an error feedback law by nonlinear combination, and further obtaining an error feedback control quantity U01(t) subsequently using the disturbance estimate z13To U01And (t) compensating to obtain the final control quantity acted on the four-wheel drive platform when the platform moves linearly. The process principle of controlling the turning angle of the four-wheel drive platform is the same as that of controlling the linear motion speed. In addition, the control signal fusion link in the figure ensures that the speed control and the turning control are not influenced mutually, namely when the speed control and the turning control are not influenced mutuallyThe turning angle is set to be zero, and when the movement speed is set to be not zero, the four-wheel drive platform independently performs linear movement with controllable speed; when the movement speed is set to be zero and the turning angle is set to be not zero, the four-wheel drive platform independently performs angle-controllable in-situ rotation movement; when the given movement speed is not zero and the given turning angle is not zero, the four-wheel drive platform makes large-radian turning movement.
To sum up, when the real-time remote control system of the mobile robot based on human-computer interaction works specifically, the head attitude sensor of the local main control terminal collects head pitching and horizontal rotation information of an operator, the head pitching and horizontal rotation information is received by the local main control computer and is sent to the remote actuator through the communication system, a camera attitude control board card of the video and audio acquisition mechanism generates a control signal of the two-degree-of-freedom mechanical arm, the action attitude of the two-degree-of-freedom mechanical arm is consistent with the action attitude of the head of the operator, a camera which is mounted at the top end of the two-degree-of-freedom mechanical arm and follows the two-degree-of-freedom mechanical arm shoots a work site picture in real time, image information is sent to the local main control computer of the local main control terminal through the communication system by the remote main control computer of the remote actuator, and is displayed. The video and audio acquisition mechanism of the far-end actuator acquires sound information of a working site through the pickup and the external sound card, the sound information is sent to the local main control computer through the far-end main control computer through the communication system, and the audio information is played by the earphone connected with the local main control computer and the external sound card, so that the function of acquiring audio of the far-end working site can be realized. The data acquisition and processing board card connected with the remote main control computer can be connected with various sensors according to actual needs to detect field environment information and self working state information of the equipment, and the sensor information is processed and then sent back to the local main control computer through the remote main control computer through a communication system to be displayed on an interface. Meanwhile, local operators can accurately control the motion of the four-wheel drive platform and the motion of the manipulator on the far-end site in real time by using a control handle in a hand according to the video, audio and sensing information of the far-end site transmitted back in real time. According to the invention, the AR follow-up platform with multiple information acquisition functions is arranged on the remote unmanned operation equipment, and the remote unmanned operation which is more convenient, more accurate, visual, interactive, low-cost and low-delay is realized by acquiring various information such as on-site images, sounds, equipment states and environmental parameters and displaying the information on the head-mounted VR display of the local main control terminal, so that an operator can safely, accurately and efficiently carry out the remote unmanned operation.
Finally, it should be noted that: the above-mentioned embodiments are only used for illustrating the technical solution of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (3)
1. The utility model provides a real-time remote control system of mobile robot based on human-computer interaction which characterized in that:
the system comprises a local main control terminal, a communication system and a remote actuator;
the local main control terminal comprises a local main control computer, a head-wearing VR display, a head posture sensor, an earphone, an external sound card and a control handle, wherein the head-wearing VR display, the head posture sensor, the earphone, the external sound card and the control handle are respectively connected with the local main control computer;
the far-end actuator comprises a far-end main control computer, a video and audio acquisition mechanism, a motion mechanism, a manipulator actuating mechanism and a site environment and device state sensing mechanism, the video and audio acquisition mechanism, the movement mechanism, the manipulator execution mechanism and the site environment and device state sensing mechanism are respectively connected with the remote main control computer, the video and audio acquisition mechanism comprises a camera, a camera attitude control board card, a two-degree-of-freedom mechanical arm, a sound pickup and an external sound card, the motion mechanism comprises a motion control board card, an inertial sensor and a four-wheel drive platform, the manipulator execution mechanism comprises a manipulator action control board card and a manipulator, the manipulator comprises a mechanical arm and a mechanical claw, and the field environment and device state sensing mechanism comprises a field environment information sensor, a device state information sensor and a data acquisition and processing board card;
the local main control computer and the remote main control computer run in a man-machine interaction software system which is developed by a QT platform under a Linux system, and the communication system comprises wired communication and wireless communication and realizes real-time bidirectional communication between the local main control terminal and the remote actuator;
the head attitude sensor comprises a main sensor and auxiliary sensors which are respectively arranged at the top end and two sides of a helmet worn by an operator, accurate head attitude information of the operator obtained by fusing multi-sensor information is sent to the far-end executor through the communication system, and is transmitted to the camera attitude control board card by the far-end main control computer for processing, so that attitude control signals of the two-degree-of-freedom mechanical arm are generated, and the swing attitude of the two-degree-of-freedom mechanical arm is consistent with the head attitude of the operator;
the main sensor and the auxiliary sensor both comprise a three-axis gyroscope sensor, and the directions of three axes of the three-axis gyroscope sensor in the main sensor and the auxiliary sensor are set as follows: in the main sensor, a Z axis points upwards vertically, an X axis points right ahead horizontally, a Y axis is perpendicular to the X axis and the Z axis and points to the left end horizontally, the Z axis is used for measuring the horizontal rotation angle of the head, and the Y axis is used for measuring the pitching angle of the head; in the first auxiliary sensor, a Z axis is vertically directed downwards, a Y axis is horizontally directed backwards, an X axis is perpendicular to the Z axis and the Y axis, the X axis is horizontally directed to the right end, an angle value of horizontal rotation of the head with a sign opposite to that measured by the Z axis of the main sensor is obtained by utilizing the Z axis, and a head pitch angle value with a sign opposite to that measured by the Y axis of the main sensor is obtained by utilizing the X axis; in the second auxiliary sensor, the Z axis is vertically directed downwards, the Y axis is horizontally directed forwards, the X axis is perpendicular to the Z axis and the Y axis, the X axis is horizontally directed to the left end, the Z axis is used for obtaining an angle value of horizontal rotation of the head with a sign opposite to that measured by the Z axis of the main sensor, and the X axis is used for obtaining a head pitch angle value with a sign same as that measured by the Y axis of the main sensor; in the three head attitude sensors, quaternion attitude calculation is carried out by utilizing output values of a gyroscope to obtain Euler angles in three directions, the magnetometer is matched with the accelerometer to obtain another group of Euler angles in the three directions, effective angles capable of representing head attitude are extracted, and then complementary fusion processing is carried out on the obtained two groups of head attitude data;
the control handle is provided with a motion mechanism for controlling the far-end actuator and a control rocker and a knob of a manipulator actuating mechanism, a control signal given by the control handle is sent to the far-end main control machine through the local main control machine through the communication system and is respectively sent to the motion control board card and the manipulator action control board card, the control rocker gives a motion control signal of the motion mechanism, the knob gives a control signal of a manipulator swinging angle and a manipulator opening and closing angle of the manipulator actuating mechanism, the motion control board card of the motion mechanism receives the motion control signal given by the control rocker and converts the motion control signal into corresponding four paths of PWM waves to control the rotating speed and the forward and reverse rotation of four direct current motors of the four-wheel drive platform, and meanwhile, an inertial sensor connected to the motion control board card collects and feeds back the linear acceleration and the turning angle speed information of the motion of the four-wheel drive platform, further realizing closed-loop control on the forward movement, the backward movement and the steering of the four-wheel drive platform; after receiving the angle control signal given by the knob, a manipulator action control plate of the manipulator actuating mechanism converts the angle control signal into three corresponding paths of PWM waves, controls three steering engines of a manipulator and a gripper of the manipulator and further controls the swinging angle of the manipulator and the opening and closing angle of the gripper;
after the data acquisition and processing board card processes the field environment information acquired by the field environment information sensor and the equipment state information data acquired by the device state information sensor, the data acquisition and processing board card is sent to the local main control computer by the remote main control computer through the communication system and displayed in the head-mounted VR display;
the camera is fixed at the top end of the two-degree-of-freedom mechanical arm, a field picture shot by the camera is compressed by the far-end main control computer and then transmitted to the local main control computer for decompression and displayed by the head-mounted VR display, the bottom end of the two-degree-of-freedom mechanical arm and the far-end main control computer are fixed on a damping platform, the damping platform is installed on the four-wheel drive platform, the far-end main control computer collects field audio data through the sound pickup and the external sound card, sends the field audio data to the local main control computer through a communication system, and plays the field audio data through the earphone and the external sound card;
the communication system realizes wireless communication through a pair of wireless bridges and realizes wired communication through a power carrier module and a power carrier line;
the data acquisition and processing board card is externally connected with various sensors, including an environment temperature and humidity sensor, an air pressure height sensor, an illumination intensity sensor, an oil tank liquid level sensor, a motor load current sensor and a vibration sensor.
2. The real-time remote control system for the mobile robot based on the human-computer interaction as claimed in claim 1, wherein: the head posture sensor is fixed at the top end and two sides of the helmet.
3. The real-time remote control system for the mobile robot based on the human-computer interaction as claimed in claim 1, wherein: the camera is a drive-free high-definition infrared monocular camera capable of night vision.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811004647.6A CN109129523B (en) | 2018-08-30 | 2018-08-30 | Mobile robot real-time remote control system based on human-computer interaction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811004647.6A CN109129523B (en) | 2018-08-30 | 2018-08-30 | Mobile robot real-time remote control system based on human-computer interaction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109129523A CN109129523A (en) | 2019-01-04 |
| CN109129523B true CN109129523B (en) | 2021-07-06 |
Family
ID=64829567
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811004647.6A Active CN109129523B (en) | 2018-08-30 | 2018-08-30 | Mobile robot real-time remote control system based on human-computer interaction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109129523B (en) |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109895123A (en) * | 2019-01-18 | 2019-06-18 | 弗徕威智能机器人科技（上海）有限公司 | A kind of tactile and form two-way synchronization method |
| CN110164439A (en) * | 2019-05-29 | 2019-08-23 | 姚英泽 | Intellect service robot |
| CN111163304B (en) * | 2019-12-27 | 2021-10-08 | 扬州哈工科创机器人研究院有限公司 | Method and device for fusing film and matrix mechanical unit |
| CN111252218A (en) * | 2020-01-20 | 2020-06-09 | 江苏科技大学 | A monitoring sampling underwater robot for ocean or inland river lake |
| CN111369854A (en) * | 2020-03-20 | 2020-07-03 | 广西生态工程职业技术学院 | Vr virtual reality laboratory operating system and method |
| CN111918064A (en) * | 2020-08-11 | 2020-11-10 | 北京科技大学 | Wireless communication and remote control system under dangerous environment |
| CN112614248A (en) * | 2020-12-21 | 2021-04-06 | 三一重机有限公司 | Engineering machinery remote control system |
| CN112603546A (en) * | 2020-12-24 | 2021-04-06 | 哈尔滨思哲睿智能医疗设备有限公司 | Remote operation system based on laparoscopic operation robot and control method |
| CN113612928B (en) * | 2021-10-09 | 2022-04-12 | 深圳市越疆科技有限公司 | Teleoperation system, teleoperation method and chip |
| CN114102618B (en) * | 2021-11-16 | 2023-10-10 | 中电信数智科技有限公司 | Operation and maintenance robot, machine room operation and maintenance method and system based on operation and maintenance robot |
| CN114770583A (en) * | 2022-04-29 | 2022-07-22 | 大连工业大学 | Intelligence assembly system based on VR |
| CN114918938A (en) * | 2022-05-18 | 2022-08-19 | 哈尔滨工业大学 | Nuclear facility retirement double-arm robot and control system |
| CN117381822B (en) * | 2023-12-11 | 2024-02-20 | 北京云中未来科技有限公司 | Manipulator control device and method for rope disorder detection |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102681542A (en) * | 2012-03-07 | 2012-09-19 | 陶重犇 | Experimental platform for indoor multipurpose mobile robot |
| CN106393049A (en) * | 2015-08-12 | 2017-02-15 | 哈尔滨理工大学 | Robot used for high risk operation |
| CN106937531A (en) * | 2014-06-14 | 2017-07-07 | 奇跃公司 | Method and system for producing virtual and augmented reality |
| JP2018165982A (en) * | 2017-01-16 | 2018-10-25 | エヌ・シー・アール・コーポレイションＮｃｒ Ｃｏｒｐｏｒａｔｉｏｎ | Maintenance management method by virtual reality and system of the same |
| KR101915780B1 (en) * | 2016-12-29 | 2018-11-06 | 유한책임회사 매드제너레이터 | Vr-robot synchronize system and method for providing feedback using robot |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6636826B1 (en) * | 1998-12-17 | 2003-10-21 | Nec Tokin Corporation | Orientation angle detector |
| JP2010117260A (en) * | 2008-11-13 | 2010-05-27 | Epson Toyocom Corp | Correction parameter preparation method of attitude detector, device for correction parameter preparation of attitude detector, and attitude detector |
| CN108363415A (en) * | 2018-03-29 | 2018-08-03 | 燕山大学 | A kind of vision remote control servomechanism and method applied to underwater robot |
-
2018
- 2018-08-30 CN CN201811004647.6A patent/CN109129523B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102681542A (en) * | 2012-03-07 | 2012-09-19 | 陶重犇 | Experimental platform for indoor multipurpose mobile robot |
| CN106937531A (en) * | 2014-06-14 | 2017-07-07 | 奇跃公司 | Method and system for producing virtual and augmented reality |
| CN106393049A (en) * | 2015-08-12 | 2017-02-15 | 哈尔滨理工大学 | Robot used for high risk operation |
| KR101915780B1 (en) * | 2016-12-29 | 2018-11-06 | 유한책임회사 매드제너레이터 | Vr-robot synchronize system and method for providing feedback using robot |
| JP2018165982A (en) * | 2017-01-16 | 2018-10-25 | エヌ・シー・アール・コーポレイションＮｃｒ Ｃｏｒｐｏｒａｔｉｏｎ | Maintenance management method by virtual reality and system of the same |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109129523A (en) | 2019-01-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109129523B (en) | Mobile robot real-time remote control system based on human-computer interaction | |
| CN109311639B (en) | Remote control device for cranes, construction machines and/or pallet trucks | |
| WO2016165316A1 (en) | Underwater body device of underwater robot and method for autonomous obstacle avoidance | |
| CN107968915B (en) | Real-time control system and method for underwater robot camera pan-tilt | |
| CN201845454U (en) | 6-DOF (six degrees of freedom) motion simulator | |
| CN104571124B (en) | Three-degree-of-freedom attitude simulation device for underwater vehicle | |
| JP2017524958A (en) | Gimbal and gimbal simulation system | |
| CN103147789A (en) | System and method for controlling underground coal mine rescue robot | |
| CN112634318B (en) | Teleoperation system and method for underwater maintenance robot | |
| CN102679114B (en) | Hoisting chamber | |
| CN105150206B (en) | A kind of six sufficient both arms combined mobile robot system | |
| WO2022166264A1 (en) | Simulation training system, method and apparatus for work machine, and electronic device | |
| CN110977981A (en) | Robot virtual reality synchronization system and synchronization method | |
| CN110794710A (en) | Underwater robot simulation system | |
| CN107471216A (en) | VR body man-controlled mobile robots under hazardous environment | |
| CN113031462A (en) | Port machine inspection route planning system and method for unmanned aerial vehicle | |
| CN202429357U (en) | Two-DOF (Degree of Freedom) cabin control system | |
| CN112783204B (en) | Real-time tracking system and method for ground maneuvering target of four-rotor unmanned aerial vehicle | |
| Omerdic et al. | Remote presence: Long endurance robotic systems for routine inspection of offshore subsea oil & gas installations and marine renewable energy devices | |
| CN102673795B (en) | Two-degree-of-freedom pod control system | |
| CN115284247A (en) | Live working robot master-slave teleoperation method and system based on heterogeneous master hand | |
| CN207067783U (en) | A kind of underwater robot | |
| CN115076561A (en) | Tele-immersion type binocular holder follow-up system and method applied to engineering machinery | |
| Fernando et al. | Effectiveness of Spatial Coherent Remote Drive Experience with a Telexistence Backhoe for Construction Sites. | |
| CN113840008A (en) | Coal mining machine immersion type remote control device, coal mining machine remote control system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |