JP6729991B2 - Remote control communication system, relay method and program therefor - Google Patents

Description

The present invention relates to a remote control device and a communication system that transmit and receive signals via a communication path.

With the rapid spread and progress of the Internet, optical communication networks, mobile networks, etc. in recent years, the technology of the Internet of Things (IoT) is drawing attention. With the spread of IoT services, the number of devices connected to the network is expected to increase further in the future. In the network equipment of telecommunications carriers and service providers, equipment expansion and functional expansion are required more and more in the future in preparation for the explosive increase in the number of connection terminals and network bandwidth demand.

By the way, in the era of IoT service popularization, it is expected that there is an increasing need for remote control of IoT devices such as moving objects (robots, drones, automobiles, etc.) and robot arms via a communication network. In particular, in remote control of flying objects, automobiles, surgical arms, etc., deterioration of communication quality may affect human life. Therefore, even when the number of IoT devices connected to the network explosively increases, the deterioration of communication quality may occur. A method of preventing or compensating for deterioration of communication quality is important.

In the remote control of the IoT device, delay is one of the factors that greatly affect the accuracy and comfort of the operation. As an example, consider the flow when trying to remotely control the robot arm while visually confirming the situation at a remote location with a camera. The video information captured by the camera is transmitted to the operator, and the operator viewing the video information inputs an operation command to the remote controller (control stick, handle, button, etc.), and the operation command information is sent from the remote controller to the robot arm. In the series of operations in which the movable part (motor, etc.) of the robot arm operates according to the received operation command information and the resulting change in the camera field of view is also transmitted to the operator as image information, the remote controller If there is a large communication delay with the operation target device, there will be a gap between the image viewed by the operator and the phenomenon that is actually occurring at a remote location, and an appropriate operation command will be issued at the appropriate timing. Since it cannot be transmitted from the operator to the robot arm, it is difficult to perform accurate control, and the operator feels awkward.

The methods disclosed in Non-Patent Documents 1 and 2 are examples of methods for compensating for delays in remote operation of robots and the like. Non-Patent Documents 1 and 2 are capable of synchronizing the movement of the manipulator on the operating side with the manipulator on the operating side while experiencing the haptic information of the manipulator on the operated side in a remote place at the operator's hand. In a remote control system, which is supposed to be used for, a communication delay compensator equipped with a communication disturbance observer is provided inside the remote controller, and a disturbance and reaction force estimation observer is provided inside the robot and is corrected by calculation. It is a technology that compensates for communication delay by transmitting force information to the operator.

Further, in Non-Patent Document 3, a remote controller or a robot is provided with a delay measuring unit, a predicting unit, an image generating unit, and the like, and based on movement information of the robot, an imaging viewpoint of an imaging device at a first time point when a subject image is captured By predicting the viewpoint shift between the imaging viewpoint of the imaging device at the second time point different from the first time point, even if a delay is included between the time when the image is captured and the time when the image is displayed, This is a technique for displaying an image with a small deviation from the image captured by the robot at the current position.

Hayata Sakai, 2 others, "A Method of Scaling Bilateral Control System with Time Delay in Terms of Modal Space", Industrial Technology (ICIT), 2016 IEEE International Conference, 2016 Shohei Hyodo, et al., “A method of arranging controlled object model in communication disturbance observer considering unstable communication network”, IEEJ Transactions on IEEJ Transactions on Industry Application, Vol.135, No.3, pp.192-198, 2015 Tetsuo Shintoku, “Remote Manipulation System Using Environmental Model”, 1991 Mechanical Research News, ISSN 0286-2271, Institute of Mechanical Engineering, Institute of Mechanical Engineering

All of the above-mentioned Non-Patent Documents 1 to 3 need to have a high-performance simulator inside or in the vicinity of the device on the operating side or the device on the operated side, which leads to downsizing, economy, and power saving as an IoT device There was a problem.

Further, in all of the technologies of Non-Patent Documents 1 to 3, the delay itself in the communication network is treated as an uncontrollable disturbance factor, and the accuracy of delay compensation is provided inside or near the operating side or operated side device. There is a problem that it depends largely on the performance of the simulator.

The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an economical IoT device remote control service and to improve the accuracy of remote control and the quality of comfort. Especially.

In order to achieve the above object, the invention of the present application connects a master device for remote operation and a slave device that is an operation target of remote operation via a network, and the slave device receives an operation received from the master device. in remote control communication system the master device that outputs the sensor information received from the slave device to the operator while operating on the basis of the command information, the on communication path connecting between said master device and said slave device A relay node that relays the operation command information and the sensor information is arranged, and the relay node includes a first delay acquisition unit that measures or estimates a communication delay between itself and the master device; Second delay acquisition means for measuring or estimating the communication delay between the two, first interception means for intercepting the operation command information transmitted from the master device to the slave device, and the second interception means transmitted from the slave device to the master device. Second interception means for intercepting sensor information, the delay information obtained by the first delay acquisition means and the second delay acquisition means, the operation command information obtained by the first interception means, and the second interception information. A simulator for predicting a state in the immediate future of the slave device based on the sensor information acquired by the means; and a notification means for notifying the master device of the prediction information calculated by the simulator, To do.

Further, according to the present invention, a master device for performing a remote operation and a slave device that is an operation target of the remote operation are connected via a network, and the slave device operates based on operation command information received from the master device. Along with the remote control communication system in which the master device outputs the sensor information received from the slave device to the operator, the operation command information and the sensor are provided on a communication path connecting the master device and the slave device. A relay node for relaying information is arranged, and the relay node measures a communication delay between itself and the master device and a first delay acquisition means for measuring or estimating a communication delay between itself and the slave device. Second delay acquisition means for measuring or estimating, first interception means for intercepting operation command information transmitted from the master device to the slave device, and intercepting sensor information transmitted from the slave device to the master device. Second interception means, each delay information obtained by the first delay obtaining means and the second delay obtaining means, operation command information obtained by the first intercepting means, and sensor information obtained by the second intercepting means Based on the above, a simulator for predicting a state of the slave device in the immediate future and a notification means for transmitting control command information corrected by the simulator result to the slave device are provided.

According to the present invention, since the computational resources of the simulator provided in the relay node can be shared by a large number of users, it is not necessary to mount an expensive simulator on the IoT device or remote controller, and economical IoT device remote operation is possible. Service becomes feasible.

In addition, the relay node has a communication path length between the slave device and the slave device inevitably becomes shorter than the communication path length between the master device and the slave device. Can be obtained, and the delay time can be substantially shortened. As a result, the gap between the sensor information currently displayed to the operator and the reality can be shown to the operator more accurately, so that improvement in accuracy of remote operation and improvement in quality of experience can be expected.

Overall configuration diagram of a remote control communication system according to the first embodiment Functional block diagram of the remote control communication system according to the first embodiment. The flowchart explaining operation|movement of the relay node which concerns on 1st Embodiment. Sequence diagram of conventional remote control communication system Sequence diagram of the remote control communication system according to the first embodiment Display example of corrected sensor information Another display example of corrected sensor information Overall configuration diagram of a remote control communication system according to a second embodiment Functional block diagram of a remote control communication system according to a third embodiment The flowchart explaining operation|movement of the relay node which concerns on 3rd Embodiment. Sequence diagram of the remote control communication system according to the third embodiment

[First Embodiment]
A remote control communication system according to the first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is an overall configuration diagram of a remote control communication system.

As shown in FIG. 1, the remote control communication system according to the present embodiment is configured by connecting a master device 10 as an operating device and a slave device 20 as an operated device via a network. In the present invention, a relay node 100 having a delay compensation mechanism is arranged on the communication path connecting the master device 10 and the slave device 20.

Examples of the master device 10 include a general-purpose computer and a device mounted as a dedicated device. Examples of the slave device 20 include an IoT, a robot, a drone, an automobile, a robot arm, and a remote surgery arm.

The master device 10 includes a command input unit 11 that inputs an operation command from the operator 1 and transmits the operation command information to the slave device 20, while the slave device 20 uses the operation command information received from the master device 10. An operation unit 21 that operates based on the above is provided.

The command input unit 11 is a device for the operator 1 to input an operation command necessary for operating the slave device 20, and for example, operates a keyboard, a mouse, a joystick, a control stick, a handle, a button, a touch panel, a foot switch, and the like. The person 1 may directly perform the operation, or a part or all of the body or the like of the operator 1 is detected by a camera, infrared rays, or the like, and the detection result is used as the operation instruction of the operator 1. It may be one, or any combination thereof.

The operation unit 21 is not only a device that performs a mechanical operation such as an actuator, a motor, or a solenoid, but also a device that performs an optical operation such as an LED or an LCD, or a device that performs an acoustic operation such as a speaker buzzer. , Which operates based on operation command information from the master device 10.

Further, the slave device 20 includes a sensor 22 that detects various states such as the surrounding environment and the operating state of the slave device 20 and transmits them to the master device 10, while the master device 10 receives the sensor information received from the slave device 20 from the operator. 1 is provided with an output unit 12 for presentation.

The sensor 22 detects various kinds of state information such as the operating state of the slave device 20 itself and the surrounding environment of the slave device 20. For example, the sensor 22 acquires optical information of a camera (visual sensor), a microphone, etc. For acquiring acoustic information of the device (hearing sensor), those for acquiring mechanical information such as micro switch, potentiometer, pressure sensor (tactile sensor or force sensor), any combination of these, etc. Various ones can be used. Further, the sensor 22 may be integrated with the operation unit 21, or both may be arranged separately.

The output unit 12 is a device for presenting the sensor information from the sensor 22 to the operator 1, and is configured by a device corresponding to the sensor information. For example, when the sensor information is image information, a display device such as an LCD is used. Further, for example, when the sensor information is voice information, an acoustic device such as a speaker is used. Further, for example, when the sensor information is mechanical information, it is conceivable that the mechanical information is converted into visual or sound information and the information is output by a display device or an acoustic device. Further, when the sensor information is mechanical information, it may be a mechanical drive device for reproducing a mechanical operation. The output unit 12 may arbitrarily combine the various output devices described above according to the sensor information. The output unit 12 may be integrated with the command input unit 11 or may be arranged separately.

FIG. 1 illustrates a case where the slave device 20 is a robot. Various actuators of the robot correspond to the operation unit 21, and a camera provided in the robot corresponds to the sensor 22. 1 illustrates a case where the master device 10 is a computer, a controller such as a joystick connected to the computer corresponds to the command input unit 11, and an LCD connected to the computer corresponds to the output unit 12. To do.

The master device 10 and the slave device 20 can communicate with each other via a network. The architecture of the transmission medium/topology of the network is not limited, and any wired network, wireless network, or any combination thereof can be used. The master device 10 and the slave device 20 each include a communication interface (not shown) for accessing the network.

On the communication path connecting the master device 10 and the slave device 20, a communication bearer 1, which is a communication line for transmitting operation command information from the master device 10 to the slave device 20, and the slave device 20 to the master device 10. A communication bearer 2, which is a communication line for transmitting sensor information, is formed in the. The network can provide different QoS services to the communication bearer 1 and the communication bearer 2.

The relay node 100 according to the present invention is arranged on the communication path connecting the master device 10 and the slave device 20. The relay node 100 can be shared by not only one master device 10/slave device 20 set but also a plurality of master device 10/slave device 20 sets. Various implementations can be considered for the relay node 100 depending on the form of the network. The relay node 100 is, for example, a router in the middle of a communication path, an edge computing server, a mobile edge computing platform, a PON (Passive Optical Network) station side device (OLT: Optical Line Terminal) in an access network, or a concentrating switch in an access network. , Base stations in mobile networks, facilities in central stations in core metro networks, servers on the Internet, etc.

Next, the configuration of the remote control communication system according to the present embodiment will be described with reference to FIG. FIG. 2 is a functional block diagram of the remote control communication system.

As shown in FIG. 2, the master device 10 includes the command input unit 11 and the output unit 12 described above, the transmission unit 13 that transmits the operation command information input by the command input unit 11 to the slave device 20, and the slave device 20. The receiving unit 14 receives the sensor information from the sensor unit and passes the sensor information to the output unit 12.

The slave device 20 receives the operation command information from the master device 10 and the transmission unit 23 that transmits the sensor information acquired by the sensor 22 to the master device 10 to the operation unit 21. The receiver 24 is provided.

The relay node 100 relays the operation command information received from the master device 10 to the slave device 20, a receiving unit 101/transmitting unit 102, a mirroring unit 103 that mirrors the operation command information to intercept the operation command information, A delay measuring unit 104 that measures or estimates a communication delay (including both uplink and downlink) between the relay node 100 and the master device 10, and a receiver that relays the sensor information received from the slave device 20 to the master device 10. 111/transmitting unit 112, a mirroring unit 113 for mirroring the sensor information in order to intercept the sensor information, and a communication delay (including both up and down) between the relay node 100 and the slave device 20 or And a delay measuring unit 104 for estimating. It should be noted that "up" means the direction from the slave device 20 to the master device 10, and "down" means the direction from the master device 10 to the slave device 20.

The delay measuring method by the delay measuring units 104 and 114 will be described. The master device 10, the slave device 20, and the relay node 100 each include means (not shown) for time synchronization, and it is premised that they are time synchronized with each other. Then, the transmission unit 13 of the master device 10 and the transmission unit 23 of the slave device 20 add a time stamp to the information to be transmitted. Various conventionally known methods can be used for the time synchronization method.

The delay measuring unit 104 calculates the delay time (downlink) required for communication from the master device 10 to the relay node 100 based on the difference between the time stamp attached to the operation command information and the current time. In addition, the delay measuring unit 104 measures the round trip time required for communication between the relay node 100 and the master device 10, and subtracts the delay time (downstream) described above from this round trip time, so that The delay time (uplink) required for communication to the master device 10 is calculated.

Similarly, the delay measuring unit 114 calculates the delay time (uplink) required for communication from the slave device 20 to the relay node 100 based on the difference between the time stamp attached to the sensor information and the current time. Further, the delay measuring unit 114 measures the round trip time required for communication between the relay node 100 and the slave device 20 and subtracts the above-mentioned delay time (upstream) from this round trip time, so that the delay node The delay time (downlink) required for communication with the slave device 20 is calculated.

As shown in FIG. 2, the relay node 100 is further based on the delay information measured by the delay measuring units 104 and 114, the operation command information acquired by the mirroring unit 103, and the sensor information acquired by the mirroring unit 113. The slave unit 20 includes a simulator unit 120 that predicts a state in the immediate future, and a combining unit 131 that combines the simulation result of the simulator unit 120 with the sensor information and sends the sensor information to the master device 10.

The simulator unit 120 calculates the state of the slave device 20 around the time when the sensor information that has recently arrived at the relay node 100 reaches the operator 1 based on the following information. (A) Communication delay between the slave device 20 and the relay node 100 (both upstream/downstream), (b) Communication delay between the relay node 100 and the master device 10 (both upstream/downstream), accumulated slave Sensor information from the device 20, (d) Control command information from the master device 10 to the slave device 20 that was intercepted immediately before.

The synthesizing unit 131 is one implementation of a notifying unit that notifies the master device 10 of the prediction information calculated by the simulator unit 120, and is estimated at the time when it reaches the operator 1 based on the calculation result of the simulator unit 120. The state information of the slave device 20 is processed/corrected with respect to the latest sensor information and transmitted to the master device 10. For example, when the sensor information is image information, the composition unit 131 processes/corrects a part of the image information based on the prediction information calculated by the simulator unit 120.

The operations of the simulator unit 120 and the synthesizing unit 131 will be described with reference to the flowchart of FIG. As shown in FIG. 3, when the sensor information, delay information, and operation command information necessary for the simulation are gathered (step S1), the simulator unit 120 is estimated at the time (future) scheduled to reach the operator 1. The status information of the slave device 20 is calculated (step S2). Note that sensor information may be received during this process. Next, the simulator unit 120 performs a calculation for adding a correction based on the calculation result to the sensor information that was most recently intercepted (step S3). The combining unit 131 corrects the sensor information based on the calculation result in step S3 to create replacement sensor information, and passes the replacement sensor information to the transmitting unit 102 for the master device 10 (step S4).

Next, a remote operation sequence by the remote operation communication system according to the present embodiment will be described with reference to FIGS. 4 and 5. Here, the slave device 20 exemplifies a device including a camera 22 that captures image information of a surrounding environment as a sensor to acquire image information, and a movable unit 21 such as a robot arm as an operation unit. Further, the master device 10 exemplifies a device including a control stick 11 as an operation input unit and a monitor 12 for displaying image information as an output unit.

First, as a comparison target, a sequence in a conventional remote control communication system having no delay compensation mechanism of the present invention will be described with reference to FIG. It should be noted that FIG. 4 shows only the sequence necessary for explaining the present invention.

As shown in FIG. 4, the image information captured by the camera 22 is sent to the master device 10 via the relay node 100, displayed on the monitor 12, and visually perceived by the operator 1 (steps S11 to S12). The operator 1 determines the next operation to be applied to the slave device 20, and operates the control stick 11 with a command from the brain to the hand (steps S13 to S14). As a result, the operation command information is transmitted from the control stick 11 to the slave device 20 via the relay node 100 (step S15). The slave device 20 receives the operation command information and determines/executes the operation direction/direction/rotation angle/distance/power of the movable part 21 (step S16). The movable part 21 operates to change the field of view of the camera (step S17), and the changed image information is sent from the camera 22 to the master device 10 via the relay node 100 (steps S18 to S19).

Here, it takes time to confirm the feedback due to a communication delay between the master device 10 and the slave device 20. Therefore, since the operator 1 cannot know the actual state of the slave device 20 even by looking at the image information, there is a problem that the accuracy and comfort of the operation cannot be maintained.

Next, a sequence in the remote control communication system according to the present embodiment will be described with reference to FIG. It should be noted that FIG. 5 shows only the sequence necessary for explaining the present invention.

As shown in FIG. 5, the image information captured by the camera 22 is sent to the master device 10 via the relay node 100, displayed on the monitor 12, and visually perceived by the operator 1 (steps S21 to S22). The operator 1 determines the next operation to be applied to the slave device 20, and operates the control stick 11 with a command from the brain to the hand (steps S23 to S24). As a result, the operation command information is transmitted from the control stick 11 to the slave device 20 via the relay node 100 (step S25). The above is the same as the conventional one.

Here, the relay node 100 receives the image information from the slave device 20 as needed until the operation command information is received (steps S26 and S27). As described above, the relay node 100 calculates the state of the slave device 20 when the image information most recently received reaches the operator, and the slave device 20 estimated at the time when it reaches the operator 1 based on the calculation result. The state information of is combined with the latest image information and transmitted to the master device 10 (steps S28 to S30). The image information is displayed on the monitor 12 of the master device 10 and visually perceived by the operator 1 (step S31).

6 and 7 show examples of image information corrected and processed by the synthesizing unit 131 and displayed on the monitor 12 according to the present embodiment. In the example of FIG. 6, the latest image information to be corrected/processed by the combining unit 131 and the simulation image are combined, and the object (at the position P0 according to the latest image information to be corrected/processed according to the object ( In FIG. 6, the vehicle is displayed, and the object is also displayed at the actual (real-time) object position P1 predicted as a result of considering the delay by the simulator.

In the example of FIG. 7, information summarizing the simulation result is displayed at the actual (real-time) position P2 predicted as a result of the delay being taken into consideration by the simulator in the latest image information to be corrected/processed in the combining unit 131. For example, it is a combination of the center of gravity, contour, edge, vertex, etc. (annular figure in FIG. 7) of the object.

As described above, in the remote control communication system according to the present embodiment, since the calculation resources of the simulator provided in the relay node 100 can be shared by many users, it is necessary to mount an expensive simulator on the IoT device or the remote controller. And the economical IoT device remote control service can be realized.

Further, in the relay node 100, the communication path length between the slave device 20 and the slave device 20 is necessarily shorter than the communication path length between the master device 10 and the slave device 20, and the slave node is closer to the real time than the master device 10. The sensor information on the device 20 side can be acquired, and the delay time can be substantially shortened. As a result, the gap between the sensor information currently displayed to the operator and the reality can be shown to the operator more accurately, so that improvement in accuracy of remote operation and improvement in quality of experience can be expected.

[Second Embodiment]
A remote control communication system according to a second embodiment of the present invention will be described with reference to the drawings. FIG. 8 is an overall configuration diagram of the remote control communication system. The same elements as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted. Only the differences from the first embodiment will be described. Further, FIG. 8 illustrates an example in which a camera is used as the sensor 22 as in FIG. 1.

In the present embodiment, the summarized information about the gap between the sensor information (camera image) that reaches the operator 1 and the reality is transmitted to the operator as soon as possible via the communication bearer of lower delay/high quality. Generally, the communication bearer used for the control signal is expected to have lower delay and higher quality than the communication bearer for video streaming. Therefore, the communication bearer 1 of the operation command information is used in this embodiment. And transmits the summary information.

The relay node 100 relays the sensor information (camera image) so as to be streamed to the master device 10 via the communication bearer 2. On the other hand, the relay node 100 calculates the prediction information in the simulator unit 120, and transmits the calculated prediction information to the master device 10 via the communication bearer 1. As a result, the delay can be made smaller.

[Third Embodiment]
A remote control communication system according to a third embodiment of the present invention will be described with reference to the drawings. FIG. 9 is a functional block diagram of the remote control communication system. The same elements as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted. Only the differences from the first embodiment will be described.

In the present embodiment, unlike the first embodiment in which the calculation result by the simulator unit 120 is reflected in the sensor information, as shown in FIG. 9, the combining unit that reflects the calculation result by the simulator unit 120 in the operation command information. 132 is provided. The combining unit 132 corrects and processes the operation command information based on the prediction information calculated by the simulator unit 120.

The operations of the simulator unit 120 and the synthesizing unit 132 will be described with reference to the flowchart of FIG. As shown in FIG. 10, the simulator unit 120 has received the sensor information, delay information, and operation command information necessary for the simulation, and when the operation command signal is received, the operation command signal most recently intercepted is the past. At what time, the sensor information issued is received and the time is calculated back (steps S41, S42). Next, the simulator unit 120 estimates the operation intention of the operator 1 for the slave device 20 (step S43), and operates from the difference between the sensor information at the past time calculated in step S32 and the latest sensor information. The state change of the slave device 20 due to a disturbance factor (wind, external force, etc.) not depending on the command signal is grasped (step S44). Then, the synthesizing unit 132, for the operation command information that was most recently intercepted, of the state change that occurred during the time lag due to the delay that the operator 1 could not know and the operator's intention estimated in step S43. A new operation command signal for eliminating the discrepancy is generated (step S45). Next, the combining unit 132 transmits the corrected replacement operation command information to the slave device 20 (step S46).

Next, a sequence in the remote control communication system according to the present embodiment will be described with reference to FIG. It should be noted that FIG. 11 shows only the sequence necessary for explaining the present invention.

Here, the slave device 20 exemplifies a device including a camera 22 that captures image information of a surrounding environment as a sensor to acquire image information, and a movable unit 21 such as a robot arm as an operation unit. Further, the master device 10 exemplifies a device including a control stick 11 as an operation input unit and a monitor 12 for displaying image information as an output unit.

As shown in FIG. 11, the image information captured by the camera 22 is sent to the master device 10 via the relay node 100, displayed on the monitor 12, and visually perceived by the operator 1 (steps S51 to S52). The operator 1 determines the next operation to be applied to the slave device 20, and operates the control stick 11 with a command from the brain to the hand (steps S53 to S54). As a result, the operation command information is transmitted from the control stick 11 to the slave device 20 via the relay node 100 (step S55).

Here, the relay node 100 receives image information from the slave device 20 as needed until the operation command information is received (steps S56 and S57).

If the operator 1 is in the vicinity of the slave device 20, the relay node 100 determines that the operation command that should have been originally executed at the time of the step S51 is the step S53, and that the gap due to the communication delay is generated. In consideration of the fact that the situation is different from that in step S51 in the latest image information (step S57), the operation instruction information is corrected and an operation instruction signal is transmitted to the slave device 20 (step S58). , S59).

Accordingly, it is possible to suppress the disturbance of the operation caused by the disturbance factor that the operator 1 cannot know, which occurs during the time lag between the time of step S51 and the time of step S55 caused by the delay.

A specific example of this sequence will be described. It is assumed that the operator observes the image emitted at the time of step S51 at the time of step 53, and the operator tries to give a command to the slave device (drone) to “turn rightward”.

The relay node 100 can know the intention that the operator wants to turn right with respect to the drone at the time of step S55. At the same time, when the latest image information (step S56) was acquired, the difference was compared with the image at the time of step S51. The drone had already started to move to the right due to the influence of strong winds, etc. without making a sharp turn to the right. Let's say that you also understand things.

In view of the above information, the relay node 100 rewrites the operation command information “Turn right swiftly” that is intercepted at the time of step S55 as “Turn right swiftly” and transmits the operation command information. Cancel the transmission.

Although the embodiment of the present invention has been described in detail above, the present invention is not limited to this. For example, in the above-described embodiment, the image information is mainly illustrated as the sensor information, but the present invention can be applied to other sensor information.

DESCRIPTION OF SYMBOLS 1... Operator 10... Master device 11... Command input part 12... Output part 13... Transmitting part 14... Receiving part 20... Slave device 21... Operating part 23... Transmitting part 24... Receiving part 22... Sensor 100... Relay node 101, 111... Receiving section 102, 112... Transmitting section 103, 113... Mirroring section 104, 114... Delay measuring section 120... Simulator section 131, 132... Combining section

Claims (8)

A master device for performing remote operation and a slave device that is a remote operation target are connected via a network, and the slave device operates based on operation instruction information received from the master device, and the master device is In the remote control communication system that outputs the sensor information received from the slave device to the operator,
A relay node for relaying the operation command information and the sensor information is arranged on a communication path connecting the master device and the slave device,
The relay node is
First delay acquisition means for measuring or estimating a communication delay between itself and the master device;
Second delay acquisition means for measuring or estimating a communication delay between itself and the slave device;
First interception means for intercepting operation command information transmitted from the master device to the slave device,
Second interception means for intercepting sensor information transmitted from the slave device to the master device;
The slave based on the respective delay information acquired by the first delay acquisition means and the second delay acquisition means, the operation command information acquired by the first interception means, and the sensor information acquired by the second interception means. A simulator that predicts the state of the device in the immediate future,
A remote control communication system, comprising: a notification unit that notifies the master device of the prediction information calculated by the simulator. The remote control communication system according to claim 1, wherein the notifying unit notifies the master device of the prediction information by combining a simulation result with the sensor information. The remote control communication system according to claim 1, wherein the notifying unit notifies the master device of the prediction information by synthesizing information summarizing a simulation result with the sensor information. The remote controller according to claim 1, wherein the notification unit transmits the prediction information to the master device via a communication bearer having a lower delay different from a communication bearer used for transmitting and receiving the sensor information. Operation communication system. A master device for performing remote operation and a slave device that is a remote operation target are connected via a network, and the slave device operates based on operation instruction information received from the master device, and the master device is In the remote control communication system that outputs the sensor information received from the slave device to the operator,
A relay node for relaying the operation command information and the sensor information is arranged on a communication path connecting the master device and the slave device,
The relay node is
First delay acquisition means for measuring or estimating a communication delay between itself and the master device;
Second delay acquisition means for measuring or estimating a communication delay between itself and the slave device;
First interception means for intercepting operation command information transmitted from the master device to the slave device,
Second interception means for intercepting sensor information transmitted from the slave device to the master device;
The slave based on the respective delay information acquired by the first delay acquisition means and the second delay acquisition means, the operation command information acquired by the first interception means, and the sensor information acquired by the second interception means. A simulator that predicts the state of the device in the immediate future,
A remote control communication system comprising: a notification unit that transmits control command information corrected based on the simulator result to the slave device. A master device for performing remote operation and a slave device that is a remote operation target are connected via a network, and the slave device operates based on operation instruction information received from the master device, and the master device is In the remote control communication system that outputs the sensor information received from the slave device to the operator,
A relay node for relaying the operation command information and the sensor information is arranged on a communication path connecting the master device and the slave device,
The first delay acquisition means of the relay node measures or estimates the communication delay between itself and the master device,
The second delay acquisition means of the relay node measures or estimates a communication delay between itself and the slave device,
The first interception means of the relay node intercepts operation command information transmitted from the master device to the slave device,
A second interception means of the relay node intercepts sensor information transmitted from the slave device to the master device,
The simulator of the relay node, each delay information acquired by the first delay acquisition means and the second delay acquisition means, operation command information acquired by the first interception means, and the sensor acquired by the second interception means Predicts the state of the slave device in the immediate future based on the information,
The notification method of the relay node notifies the master device of the prediction information calculated by the simulator. A master device for performing remote operation and a slave device that is a remote operation target are connected via a network, and the slave device operates based on operation instruction information received from the master device, and the master device is In the remote control communication system that outputs the sensor information received from the slave device to the operator,
A relay node for relaying the operation command information and the sensor information is arranged on a communication path connecting the master device and the slave device,
The first delay acquisition means of the relay node measures or estimates the communication delay between itself and the master device,
The second delay acquisition means of the relay node measures or estimates a communication delay between itself and the slave device,
The first interception means of the relay node intercepts operation command information transmitted from the master device to the slave device,
A second interception means of the relay node intercepts sensor information transmitted from the slave device to the master device,
The simulator of the relay node, each delay information acquired by the first delay acquisition means and the second delay acquisition means, operation command information acquired by the first interception means, and the sensor acquired by the second interception means Predicts the state of the slave device in the immediate future based on the information,
A relay method in a remote control communication system, wherein the notification means of the relay node transmits control command information corrected by the simulator result to the slave device. A program that causes a computer to function as each unit according to claim 1.

Images