RU2794556C1 - Method for remote control of mobile robot with delays in the information transmission channel - Google Patents

Abstract

FIELD: robotics.

SUBSTANCE: method for remote control of a mobile robot with delays in the information transmission channel. To control a mobile robot, operator commands are transmitted via a communication channel from the control panel to the mobile robot, telemetry information is transmitted from the robot to the control panel, the current state of the robot is compared with an earlier one that is relevant for the moment corresponding to the telemetry data presented on the console monitor to the operator, when they issued the command, the operator's commands are adjusted, taking into account the correction for the delay in a certain way.

EFFECT: increased efficiency of remote control.

1 cl, 3 dwg

Description

The field of technology to which the invention belongs

The invention relates to the field of robotics, namely to remote control systems for ground mobile robots.

For ground-based mobile robotic systems, the use of the remote control mode is predominantly characteristic. The applied control systems generally consist of a control panel, an onboard control system of a mobile robot and a data transmission channel connecting them, wired or wireless.

During the control process, the human operator constantly generates and sends commands to the robot drives using the remote control, based on the information received from the robot, including the video image from the television cameras installed on it. The presence of time delays in the transmission of information significantly reduces the efficiency of tasks. When the delays are constant in time and small enough (various experimental data determine the threshold in the range of 200-400 milliseconds), the operator can adapt to them, predicting the result of executing his commands. With an increase in delays, as well as their inconsistency, the operator, excessively adjusting the speed and direction of the robot, begins to issue erroneous commands. This reduces the efficiency of control, and in some cases may cause loss of stability of the control system.

Time delays that occur in the control system are of a different nature and can be divided into several groups:

- delays in the transmission and execution of control commands caused by the time required for processing data received from the joystick, transmitting information between the electronic nodes of the console, transmitting information via a communication channel, receiving and processing the received commands by the on-board control system. Typical values are 50 to 100 milliseconds;

- video transmission delays, determined by the time required for receiving a video image by a television camera of a mobile robot, processing it, overlaying technical information, digital compression, transmission over a communication channel, pre-processing on the remote control and displaying on a monitor. Typical values are between 200 and 500 milliseconds depending on camera type and video format;

- delays due to the use of signal repeaters, networks with mesh topology, etc.;

- delays due to the use of specialized modules as part of the system, for example, blocks of cryptographic information protection;

- delays caused by packet loss between the mobile robot and the control panel, requiring retransmission of data.

The last three groups can have a significant impact on the total delay time in the control system, not only increasing it up to a few seconds, but also making it variable in time.

State of the art

There is a fairly large number of works related to control problems under the action of time delays.

Known method and control system described in the publication (Toward safe and stable time-delayed mobile robot teleoperation through sampling-based path planning / Nieto J. [et al.] // Robotica. - 2012. - Vol. 30. - No. 3. - Pp.351 - 361. - DOI: 10.1017/S0263574711000695). The proposed control scheme is based on a probabilistic path planner, which, in combination with a variable sampling time prediction module, allows the operator to control the movement of a mobile robot without collision with obstacles in the presence of non-constant time delays.

A significant drawback of the technical solution is the possibility of its use only in a structured environment, subject to the use of rangefinders that provide information on the distance to obstacle objects. The inclusion of rangefinders in the control system complicates and increases the cost of its implementation.

For mobile robots operating in a real environment, the main obstacles that pose a threat are elevation changes, places with unsuitable surfaces for movement, and other terrain features. The detection of such obstacles using only rangefinders is very difficult, and the use of intelligent technical vision tools further complicates the control system and is not always possible.

Also, the disadvantages in this case include an increased level of system autonomy. The operator only continuously sets the target for the scheduler, and the system controls the movement independently. In some cases, when the task is not so much to avoid obstacles as to position the robot relative to surrounding objects, for example, for the subsequent operation of manipulators, the proposed method can make control less predictable for a human operator.

A method of controlling a mobile robot is known under a patent (Patent US 11173605 B2. Method of controlling mobile robot, apparatus for supporting the method, and delivery system using mobile robot: No. 16/199,990: Appl. 11/26/2018: publ. 11/16/2021 / Jin Hyo Kim; Applicant, Patentee Dogugonggan Co., Ltd., Gangwon-do (KR)). The method involves calculating the control actions on the drives of a mobile robot as the sum of commands received from the operator's direct control system and the commands of the autonomous control module included in the overall system and working in parallel with the operator, taken with weight coefficients determined depending on the current values of delay, risk collision with obstacles and the complexity of the environment.

The main disadvantages of the method are the same as those of the solution described above: the use of information about the distance to obstacles in front of the robot and a partial increase in the level of autonomy, which can lead to operator disorientation.

The use of an autonomous control module also significantly complicates and increases the cost of implementing the control system.

Known method and control system described in the publication (Slawinski E. Teleoperation of mobile robots with time-varying delay / Slawinski E., Mut V.A., Postigo J.F. // IEEE Transactions on Robotics. - 2007. - Vol.23. -No. 5. - Pp.1071-1082. - DOI: 10.1109/TRO.2007.906249). The method consists in continuously compensating the influence of non-constant time delays on the commands for controlling the linear and angular velocities of the mobile robot in the process of moving in an environment with obstacles towards a predetermined goal. The method is based on the use of information about the current state of the robot and the environment and control commands generated by the operator, who observes time-delayed environmental data on the console monitor. The method uses an operator model and a virtual force that depends on the distance to obstacles.

The main disadvantage of the technical solution is the use of a previously known target point of movement and the coordinates of the robot relative to it, which significantly reduces the possibility of its practical application.

The second disadvantage is associated with the use of rangefinders and information about the distance to obstacles in the path of the robot.

The closest analogue chosen for the prototype is the control method known from the patent (Patent US 2015120048 A1. Control synchronization for high-latency teleoperation: No. 14/062,632: Appl. 24.10.2013: publ. 30.04.2015 / Matthew D. Summer , Paul M. Bosscher, Michael J. Summer, Miguel Ortega-Morales; Applicant, Patentee Harris Corporation, Melbourne, FL (US)), and consisting in the following. The human operator, using the control panel, enters a command, as a result of which the mobile robot must change its state - the orientation and position in space of the robot itself or the links of the manipulator installed on it. The command is transmitted to the robot via a communication channel with time delays. In the opposite direction, from the robot to the control panel, telemetry data is transmitted. At the moment of receiving the command on board the robot, the current state is compared with the earlier one, stored in memory and relevant for the moment, which corresponds to the telemetry data presented on the console monitor to the operator when he entered the command. Based on the results of this comparison and the state of the robot that should have been reached with no delay, the command is adjusted to account for the delay correction. The corrected command is executed by the robot drives. To determine which state should be selected from the memory for comparison, the method provides for several different options, including the assessment of the state of the communication channel, the signal level and the number of errors, the transmission, along with operator commands and telemetry, of special metadata packets that mark the transmitted information and associate it with corresponding states of the robot.

The method can provide compensation for the effect of time delays in the communication channel on the results of the execution of the operator's commands and the control of the mobile robot as a whole, however, it has a number of significant drawbacks.

The technical solution does not describe a method for determining, based on the operator's commands, a local target - the position and orientation of the mobile robot, which should have been achieved by the robot in the absence of delays in the channel. Also, the solution does not describe how to correct operator commands, including setting the linear and angular velocities necessary to achieve a local goal from the current state.

The method proposed in the technical solution for storing its state in the robot’s memory, including position and orientation, including those described using geographic coordinates, leads to the need to equip mobile robots with geopositioning systems or other navigation systems that can accurately determine the position in the surrounding space. The use of such systems complicates and increases the cost of implementation.

Disclosure of the essence of the invention

The problem to which the invention is directed is to increase the efficiency of remote control by eliminating the negative aspects inherent in known control methods with compensation of time delays, namely:

- eliminating the need to use information about the obstacles in front of the mobile robot and the corresponding sensors, including rangefinders and vision systems;

- eliminating the need to use information about the position of the mobile robot and the target point of movement in the coordinate system associated with the surrounding space, and the corresponding systems for navigation and positioning in space;

- exclusion of the influence of elements that cause disorientation of the human operator and reduce the predictability of control.

The problem is solved by applying a remote control method, which consists in introducing a delay compensation unit into the software of the onboard system of the mobile robot, which, receiving commands from the control panel, corrects them based on data on previously received operator commands and the current state of the robot.

Only the angular velocity, which determines the direction of motion, is subject to such an adjustment. Linear speed does not affect the quality of control so much: the operator in most cases quickly adapts and is able to assess the position of the robot in the surrounding space, taking into account the delay, even if it changes in time. Line speed adjustments by the system can interfere with this adaptation and make control less predictable for the human operator. On the other hand, the absence of the need to adjust the linear speed of the mobile robot makes it possible to simplify the control method and system by eliminating additional calculations.

During the control process, the video camera continuously captures information about the state of the environment and the position of the mobile robot in it. The control system transmits this information to the console. In the compensation block, the instantaneous value of the total effective time delay between the moment of fixing the information and the receipt by the on-board system of the operator's commands generated on the basis of this information is determined.

At each moment of time on board, based on the data from the sensors of the drives, the values of the linear and angular velocities of the mobile robot are stored in the RAM of the onboard control system. Using the local coordinate system associated with the current state of the robot (position and orientation), the delay time and data on the speeds acting during it, the state of the robot at the time of fixing the video image is determined by numerical integration.

Using the operator's commands received from the console and the calculated position and orientation of the robot, the coordinates of the local target to which the operator wanted to bring the robot are determined.

With the help of the operator model, the task of the angular velocity is calculated, which ensures the movement of the robot from the earlier, delayed in time

state, to the point of the local goal. A correction is calculated to compensate for the individual characteristics of the operator, as the difference between the commands of the human operator and the model.

Using the same model, the task of the angular velocity is calculated, which ensures the movement of the robot from the current state to the point of the local target. The calculated value is corrected using the calculated correction. The obtained corrected value of the angular velocity and the value of the linear velocity specified by the operator are used to calculate tasks for the robot drives.

The method does not use any additional systems and sensors, except for the rotation speed sensors built into the drives, with the help of which the robot speeds are calculated. Correcting with a correction that takes into account the difference in control of the model and the person, the angular speed values calculated using the operator model, as well as the use of the uncorrected line speed value, make the control more predictable for the human operator.

Brief description of the drawings

The essence of the invention is illustrated by figures 1-3.

Figure 1 shows a control system with a delay compensation unit. Noted:

1 - mobile robot operator;

2 - control panel;

3 - information output block;

4 - control interface;

5 - data transmission channel;

6 - video transmission delay h _v ,

7 - delay transmission of commands h _c ;

8 - onboard control system of the robot;

9 - block of sensors and video cameras;

10 - delay compensation block;

11 - drive control system;

12 - environment.

где

- линейная скорость,

- угловая. Вектор состояния передают в блок компенсации 10.The video image obtained using video cameras 9 is transmitted via communication channel 5 to the control panel 4. In the same block 9, the state vector s is formed, represented by the speed values of the mobile robot

Where

- linear speed,

- angular. The state vector is transmitted to the compensation block 10.

представленный командами оператора:

где

- задание линейной скорости,

- угловой.From the control panel 4 to the compensation unit 10 comes the control vector

represented by operator commands:

Where

- line speed setting,

- corner.

где

- задание линейной скорости,

- угловой, передают из блока компенсации 10 в систему управления приводами 11.Corrected control vector

Where

- line speed setting,

- angular, transferred from the compensation unit 10 to the drive control system 11.

Figure 2 shows the time axis, delays in the control system and marked:

- текущий момент времени, в котором происходит вычисление заданий для приводов мобильного робота;

- the current moment of time at which tasks are calculated for the drives of the mobile robot;

- момент времени, в который оператор на основе полученных данных сформировал вектор управления

- the point in time at which the operator, based on the received data, formed the control vector

- момент времени, который оператор видел на мониторе пульта, когда формировал вектор управления;

- the moment of time that the operator saw on the console monitor when he formed the control vector;

- задержка передачи видеоизображения;

- video transmission delay;

- задержка передачи команд.

- command transmission delay.

Figure 3 shows the positions of the robot and the target in the local coordinate system:

- вектор текущего состояния робота, принятого за начальное, содержащий координаты робота и угол его поворота;

- the vector of the current state of the robot, taken as the initial one, containing the coordinates of the robot and the angle of its rotation;

- вектор более раннего состояния, актуального для момента времени t₃, также содержащий координаты и угол поворота;

- the vector of an earlier state, relevant for the time t ₃ , also containing the coordinates and the angle of rotation;

- вектор, определяющий локальную цель;

- a vector that defines a local goal;

- линейные скорости робота в моменты времени t₁ и t₃;

- linear speeds of the robot at times t ₁ and t ₃ ;

- угловые скорости робота в моменты времени t₁ и t₃;

- angular velocities of the robot at times t ₁ and t ₃ ;

- угол поворота робота в момент времени t₃;

- the angle of rotation of the robot at time t ₃ ;

- курсовой угол на локальную цель в момент времени t₃;

- heading angle to the local target at time t ₃ ;

- угловая ошибка в моменты времени t₁ и t₃.

- angular error at times t ₁ and t ₃ .

Implementation of the invention

The control method is as follows.

как разницу между текущим значением таймера и принятым в посылке (фигура 2).First of all, the instantaneous value of the effective delay is determined. To do this, in the remote control process (figure 1), the data transmitted by the onboard control system 8 and containing a video image from a television camera is marked with metadata packets, including the values of the internal timer, which fix the time of sending. The control panel 2, when transmitting operator commands, marks the parcels with metadata received with a video image displayed on the monitor at the time the commands are generated. On board the robot, using the information obtained in the metadata about the moment of sending the video image, the instantaneous value of the total delay is determined

as the difference between the current value of the timer and the one received in the package (figure 2).

In some cases, marking data containing video is significantly more difficult, for example, when using analog video cameras and corresponding specialized transceivers. In this case, other approaches can be used to determine the delay time, including using the sum of a predetermined video data transmission time and the command transmission time calculated using the synchronized console and robot clocks.

For further calculations, a coordinate system is used that is tied to the state of the mobile robot at time t ₁ (figure 3).

определяемые на основе данных, получаемых с датчиков в составе приводов. На основе этих данных и информации о суммарном времени задержки с помощью численного интегрирования восстанавливают состояние робота в момент t₃.During the movement of a mobile robot, the values of linear and angular velocities are constantly stored in the RAM on board.

determined on the basis of data received from sensors in the actuators. On the basis of these data and information about the total delay time using numerical integration restore the state of the robot at time t ₃ .

Any method of numerical integration can be applied, including the direct Euler method. Despite the fact that the estimate obtained is approximate, a relatively short integration interval allows us to consider the accuracy sufficient for its further application.

Using the commands sent by the operator and the calculated state of the robot at the moment t ₃ , the coordinates of the local target are determined - the point to which the operator wanted to direct the mobile robot:

- координаты искомой точки,

- коэффициенты, определяющие интервал предсказания угла и расстояния соответственно, зависящие от индивидуальных особенностей оператора. Коэффициенты в общем случае могут быть определены эмпирически для каждого оператора мобильного робота.Where

- coordinates of the desired point,

- coefficients that determine the interval for predicting the angle and distance, respectively, depending on the individual characteristics of the operator. The coefficients in the general case can be determined empirically for each mobile robot operator.

Next, the angular error is calculated at time t ₃ , which determines the difference between the heading angle to the calculated local target and the rotation angle of the robot:

нулевой, угловая ошибка в этот момент равна курсовому углу, и ее вычисляют с на основе координат цели:Since in the local system used, the rotation angle of the robot at the moment of time

zero, the angular error at this moment is equal to the heading angle, and it is calculated from the coordinates of the target:

To calculate the angular velocity setting that ensures the movement of the robot to a local target, an operator model is used, represented by a modified control law described in (Closed loop steering of unicycle like vehicles via Lyapunov techniques / Aicardi M. [et al.] // IEEE robotics & automation magazine. - 1995. - Vol. 2. - No. 1. - Pp.27-35. - DOI: 10.1109/100.388294), from which the member responsible for the final orientation of the robot is excluded:

- угловая ошибка,

- эмпирически подбираемые коэффициенты, учитывающие индивидуальные особенности оператора.Where

- angular error,

- empirically selected coefficients, taking into account the individual characteristics of the operator.

Using the operator model for the current state of the robot (at time t ₁ ) and the restored state (at time t ₁ ), the tasks of angular velocities are determined:

A correction is calculated that determines the difference between the commands generated by the human operator and the model for the time t ₃ :

обеспечивающее движение робота из текущего состояния в точку, в которую его хотел направить оператор:With the help of the calculated correction, the value is compensated

ensuring the movement of the robot from the current state to the point where the operator wanted to send it:

при расчете движения в момент

позволяет:Using the moment correction

when calculating the movement at the moment

allows:

на качество управления и избавиться от необходимости их определения для каждого оператора, используя усредненные значения;- significantly reduce the influence of coefficients

on the quality of control and get rid of the need to determine them for each operator, using averaged values;

- increase the predictability of control by reducing the deviation of the final trajectory of the robot, controlled by the system that implements the proposed method, from the expected by the operator. This makes it possible to further reduce excessive course corrections by the operator and thereby improve the quality of control.

The corrected control vector is formed from the linear speed specified by the operator and the calculated angular speed:

These values are passed on to the control system to calculate the tasks for the robot drives.

To test the proposed method, a computer model was created and a number of experiments were carried out. The model is based on a software package based on the Unity platform. During the experiments, the operator controlled the mobile robot model with a joystick, moving along a predetermined reference trajectory displayed on the screen. Video from the virtual camera was transmitted directly or with a specified delay. Commands were also transmitted to the robot model with a given delay. The robot's movement trajectories in the virtual environment were recorded for subsequent analysis. To evaluate the control efficiency, we used the task execution time and the similarity of the recorded trajectory with the reference one. Fréchet distance was used as a similarity measure (A comparative analysis of trajectory similarity measures / Tao Y. [et al.] // GIScience & Remote Sensing. - 2021. - Vol.58. - No. 5. - Pp.643-669 .- DOI: 10.1080/15481603.2021.1908927).

The simulation results showed that in the absence of a delay, the use of the proposed method does not interfere with the operator and does not have a noticeable effect on the results of the task.

With delays within half a second, the task execution time differs little, since the operator is able to quickly adapt to the delay and, within certain limits, predict the result of command execution. At the same time, the trajectories obtained using the proposed method are smoother compared to control without delay compensation and have, on average, 20% lower Fréchet distance values.

Delays of one and two seconds significantly affect the quality of control and lead to the appearance of characteristic "loops" in the motion trajectories. The application of the method allows to significantly improve the controlled indicators: to reduce the task execution time to 20% and the trajectory error to 30%.

Management at delays of five seconds or more is less dependent on the use of the method. This is because the operator intuitively switches to the start-stop strategy: he sends a command and waits for the result of its execution to be displayed on the monitor before sending the next one. This approach significantly increases the time, and a noticeable number of straight sections appear in the trajectories.

Thus, computer simulation has shown that the proposed method provides an increase in the efficiency of remote control of a mobile robot in the presence of time delays in the control channels.

Claims (1)

A method for remote control of a mobile robot in the presence of delays in the information transmission channel, which consists in transmitting an operator command over the communication channel from the control panel to the mobile robot, as a result of which the mobile robot must change its state - orientation and position in space, transmission from the robot to the remote control control of telemetric information, comparison of the current state of the robot with an earlier one relevant for the moment, which corresponds to the telemetry data presented on the monitor of the operator’s console when he formed the command, correction of the operator’s command taking into account the correction for the delay, characterized in that only the angular velocity is subject to correction mobile robot; in the control process determine the instantaneous value of the total effective time delay between the current and earlier state; to determine an earlier state using a local coordinate system associated with the current state; the earlier state is calculated from the current one by numerical integration using the data stored in the memory about the linear and angular velocities of the robot acting during the delay time; the coordinates of the local target - the point to which the operator wanted to bring the robot, is determined using the earlier state, the operator's commands and the delay time; the task of the angular velocity, which ensures the movement of the robot from the current state to the local target, is calculated using the operator model; the angular velocity task calculated using the operator's model is compensated by the correction, which is calculated as the difference between the tasks of the angular velocity by the operator and the model, providing movement from an earlier state to a local target; the corrected command is generated from the linear speed given by the operator and the calculated angular speed.

Images