JP2022164028A - Remote operation control system, remote operation control method, and program - Google Patents

Abstract

To provide a remote operation control system, a remote operation control method, and a program for enabling an operator to bodily sense an operation to the same remote operation object by another operator.SOLUTION: According to an embodiment, a remote operation control system includes a control information generation part for generating control information to operate a remote operation object on the basis of first operation information showing an operation inputted by a first operator through a first controller for receiving an operation to the remote operation object and second operation information showing an operation inputted by a second operator through a second controller for receiving the operation to the remote operation object, and a feedback information generation part for generating first feedback information for feeding back operation contents of the remote operation object to the first controller and generating second feedback information to feeding back the operation contents of the remote operation object to the second controller on the basis of the control information.SELECTED DRAWING: Figure 4

Description

The present invention relates to a remote operation control system, a remote operation control method, and a program.

2. Description of the Related Art In recent years, there has been known a technique in which an operator remotely controls a remote control target such as a robot. For example, there is known a system in which a plurality of operators take turns remotely operating the same robot by switching the remote control authority (see Patent Document 1).

JP 2019-005819 A

However, in the conventional technology, it was not possible for a plurality of operators to operate the same remote-operated object and to experience the operation of the same remote-operated object by other operators.

Aspects of the present invention have been made in consideration of such circumstances, and provide a remote operation control system, a remote operation control method, and a program that allow a user to experience the operation of the same remote operation target by another operator. intended to provide

A remote operation control system, a remote operation control method, and a program according to the present invention employ the following configurations.
(1): A remote operation control system according to one aspect of the present invention is a remote operation control system in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world, in which an operation on the remote operation target is performed. First operation information indicating an operation input by a first operator via a first controller that accepts the operation, and indicating an operation input by a second operator via a second controller that accepts an operation for the remote operation target. an information acquisition unit that acquires second operation information; and control that generates control information for operating the remote operation target based on the first operation information and the second operation information acquired by the information acquisition unit. an information generating unit for generating, based on the control information, first feedback information for feeding back details of the action of the remote-operated target to the first controller, and feeding back the details of the action of the remote-operated target to the second controller; and a feedback information generation unit that generates second feedback information.

(2): In the aspect of (1) above, the feedback information generation unit determines the details of the operation by the first operator based on the action of the remote-operated object based on the operation input by the first operator. , generating first feedback information to be transmitted to the second operator via the second controller; (2) generating second feedback information for transmitting the operation content of the operator to the first operator via the first controller;

(3): Image data to be displayed on the first display unit for the first operator or the second display unit for the second operator in the aspect (1) or (2) above. and a comparison image generation unit that generates a comparison image showing a comparison result between the operation input by the first operator and the operation input by the second operator.

(4): In any one of the aspects (1) to (3) above, the control information generation unit generates the content of the remote operation by the first operator and the content of the remote operation by the second operator. The control information is generated based on the result of synthesizing the .

(5): In the aspect of (4) above, the control information generation unit, when synthesizing the details of the remote operation by the first operator and the details of the remote operation by the second operator, generates the first The control information is generated by prioritizing the operation of the second operator over that of the operator.

(6): In any one of the aspects (1) to (5) above, the feedback information generating unit, based on the detection result of a sensor mounted on the remote operation target, causes the remote operation target to: generating third feedback information for transmitting the obtained sensation or touch to the first operator via the first controller; It generates first feedback information for transmitting a sensation or feeling obtained on the operation target side to the second operator via the second controller.

(7): A remote operation control method according to another aspect of the present invention is a remote operation control in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world, wherein the remote operation first operation information indicating an operation input by a first operator through a first controller that accepts an operation on an object; and input by a second operator through a second controller that accepts an operation on the remotely operated object. obtaining second operation information indicating an operation; generating control information for operating the remote operation target based on the obtained first operation information and the second operation information; and generating control information for operating the remote operation target based on the control information to generate first feedback information for feeding back details of the action of the remote-operated target to the first controller, and to generate second feedback information for feeding back the details of the action of the remote-operated target to the second controller. control method.

(8): A program according to another aspect of the present invention provides a remote operation control in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world, by causing a computer to perform an operation on the remote operation target. first operation information indicating an operation input by a first operator via a first controller that accepts; obtaining second operation information; generating control information for operating the remote operation target based on the obtained first operation information and the second operation information; The program generates first feedback information for feeding back details of an operation of a remote-controlled object to the first controller, and generates second feedback information for feeding back details of the operation of the remote-controlled object to the second controller.

According to the aspects (1) to (8) above, it is possible to experience the operation of the same remote-operated object by another operator.
According to the above (2), a motion of a robot or the like remotely controlled by one operator can be transmitted to the other operator.
According to the above (3), one operator can confirm the erroneous operation corrected by the other operator, the difference between his operation and the operation by the other operator, and the like in the image.
According to (4) above, the remote-controlled object can be operated according to the remote control by two or more operators, and for example, it is possible for a skilled worker to guide the work of a newcomer in real time.
According to (5) above, the remote operation by the other operator can be reflected in the action of the remote operation target rather than the remote operation by the other operator, and the remote operation by one operator can be reflected by the other operator. Can be modified for remote operation.
According to (6) above, it is possible to convey to the operator the sensations and sensations obtained when the operator actually operates the operation object instead of the remote operation using the remote operation target.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows the outline|summary of the remote operation control system which concerns on this embodiment. It is a functional block diagram of a configuration installed in a work space where a first operator works. It is a functional block diagram of a robot. It is a functional block diagram of a robot remote control device. It is a figure which shows an example of a comparison image. FIG. 4 is a sequence diagram showing an example of processing in the robot remote control system; 4 is a flowchart showing an example of processing by a robot remote control device; FIG. 10 is a sequence diagram showing another example of processing in the robot remote control system; 9 is a flowchart showing another example of processing by the robot remote control device;

Hereinafter, embodiments of a remote operation control system, a remote operation control method, and a program according to the present invention will be described with reference to the drawings. In addition, in the drawings used for the following description, the scale of each member is appropriately changed so that each member has a recognizable size.

<Overview>
FIG. 1 is a diagram showing an outline of a remote operation control system 1 according to this embodiment. The remote operation control system 1 includes, for example, a first HMD (head mounted display) 10, a first controller 20, a first environment sensor 30, a first terminal device 40, a second HMD 50, a second controller 60, a first A second environment sensor 70 , a second terminal device 80 , a robot 100 , a third environment sensor 200 , and a robot remote control device 300 are provided.

For example, the first HMD 10, the first controller 20, and the first environment sensor 30 are connected to the first terminal device 40 via short-range wireless communication. The first terminal device 40 is connected to the robot 100, the third environment sensor 200, and the robot remote control device 300 via the network NW. Also, the second HMD 50, the second controller 60, and the second environment sensor 70 are connected to the second terminal device 80, for example, via short-range wireless communication. The second terminal device 80 is connected to the robot 100, the third environment sensor 200, and the robot remote control device 300 via the network NW. The network NW includes, for example, the Internet, a WAN (Wide Area Network), a LAN (Local 1 Area Network), a provider device, a radio base station, and the like.

The configuration is not limited to this, and the first terminal device 40 and the second terminal device 80 may be omitted. In this case, first HMD 10, first controller 20, first environment sensor 30, second HMD 50, second controller 60, and second environment sensor 70 are connected to robot 100, third environment sensor 200, and robot remote control via network NW. It may be directly connected to the operation control device 300 .

The robot 100, the third environment sensor 200, and the robot remote control device 300 are connected via a wireless or wired communication network, for example. Without being limited to this, the robot 100, the third environment sensor 200, and the robot remote control device 300 may be directly connected by wire without going through a communication network.

The first HMD 10 and the first controller 20 are devices worn by, for example, a newcomer A (an example of a first operator), and the second HMD 50 and the second controller 60 are worn by, for example, an expert B (an example of a second operator). It is a device that The first controller 20 and the second controller 60 are, for example, tactile data gloves worn on the hands of the operator. The first controller 20 and the second controller 60 detect orientation, movement of each finger, hand movement, arm movement, etc. by the sensors provided therein, and transmit information indicating the detection results (hereinafter referred to as operation information) to It is transmitted to the robot remote control device 300 . The operation information includes information indicating the operation by the newcomer A, information indicating the operation by the expert B, and the like.

For example, a first environment sensor 30 and a first terminal device 40 are installed in the work space where the newcomer A works. The first environment sensor 30 is, for example, a dedicated device that includes a communication function, an RBG camera, and a depth sensor. The first environment sensor 30 is installed at a position where the newcomer A's work can be photographed and detected. The first terminal device 40 is, for example, a tablet terminal or a personal computer having a communication function. For example, the first environment sensor 30 may be attached to the first terminal device 40 , and the first environment sensor 30 may have the functions of the first terminal device 40 .

For example, a second environment sensor 70 and a second terminal device 80 are installed in the work space where the expert B works. The second environment sensor 70 is, for example, a dedicated device that includes a communication function, an RBG camera, and a depth sensor. The second environment sensor 70 is installed at a position where the operation of the expert B can be photographed and detected. The second terminal device 80 is, for example, a tablet terminal or a personal computer having a communication function. For example, the second environment sensor 70 may be attached to the second terminal device 80 , and the second environment sensor 70 may have the functions of the second terminal device 80 .

The third environment sensor 200 is, for example, a dedicated device having a communication function, an RBG camera, and a depth sensor, and is installed at a position where it can photograph and detect the work of the robot 100 . Note that the third environment sensor 200 may be provided in the robot 100 or may be attached to the robot 100 .

For example, the newcomer A remotely operates the robot 100 by moving the hand or fingers wearing the first controller 20 while watching the image displayed on the first HMD 10 . Further, the expert B remotely operates the robot 100 by moving the hand or fingers wearing the second controller 60 while watching the image displayed on the second HMD 50 . In this embodiment, the newcomer A is a worker who remotely operates the robot 100 to create an object such as a handicraft that requires skill to create. The expert B remote-controls the robot 100 during, before, and after the operation of the newcomer A, so that the newcomer A can experience the example of the expert B, and is instructed to correct the work by the newcomer A. Be a leader. In the remote operation, the newcomer A and the expert B cannot directly view the motion of the robot 100, but can indirectly view the image of the robot 100 through the first HMD 10 and the second HMD 50. FIG.

The robot 100 is an example of a remote control target that is a physical object. The remote operation target is not limited to this, and may be a virtual object in the virtual world. For example, a robot or an avatar, which is a virtual object that appears in a virtual world such as two-dimensional or three-dimensional animation, may be a remote control target. An example in which the remote control target is the robot 100 will be described below. In the case where the remote control target is a virtual object, the process for operating the robot 100 can be read as the process for operating the virtual object in the virtual world. In other words, the robot remote control device 300 has a configuration such as an image processing unit for operating a virtual object in the virtual world, and the first controller 20 and the second controller 60 have a configuration for operating the virtual object in the virtual world. may be provided.

The robot remote control device 300 feeds back the motion of the robot 100 remotely controlled by the newcomer A or the expert B to the first controller 20 or the second controller 60 . Thereby, the first controller 20 reproduces the motion of the robot 100 remotely controlled by the expert B, and the second controller 60 reproduces the motion of the robot 100 remotely controlled by the newcomer A. For example, in a scene in which expert B instructs newcomer A to use remote control to transform an object to be manipulated, the robot 100 moves the robot 100 to the object to be manipulated according to expert B's remote control. When the arm is attached, an action that puts more force on the object to be manipulated is executed. By feeding back the operation performed by the robot 100 to the first controller 20 and the second controller 60, the content of the remote operation by the expert B is fed back to the newcomer A, and the work of the newcomer A is performed by the expert B. It is possible to give the newcomer A a sensation and feel as if he were directly instructing the newcomer A with his hand.

By doing so, the expert B can experience the operation of the newcomer A through the second controller 60, and the expert B can input an operation to correct the operation of the newcomer A to the second controller 60. can be done. Therefore, the newcomer A can experience the operation of the expert B through the first controller 20, and remotely operate the robot 100 while receiving guidance from the expert B while being away from the expert B and the robot 100. can do.

Although the details will be described later, the robot remote control device 300 may feed back to the newcomer A and the expert B the senses and sensations obtained on the robot 100 side.

FIG. 2 is a functional block diagram of the configuration installed in the work space where the first operator works. It should be noted that each component installed in the work space where the expert B works also has the same function, and detailed description thereof will be omitted.

<HMD>
The first HMD 10 includes, for example, a display unit 11, a line-of-sight detection unit 12, a sensor 13, a control unit 14, a communication unit 15, and a sound pickup unit 16. The first HMD 10 displays images for remotely operating the robot 100 based on the image data received from the robot remote control device 300 . For example, the first HMD 10 displays a state image of the robot 100. FIG. The state image of the robot 100 is an image showing the motion of the robot 100, and includes, for example, an image of the hand of the robot 100 captured.

The display unit 11 is, for example, an LCD or an organic EL display device. The line-of-sight detection unit 12 detects movement of the newcomer A's line of sight. The line-of-sight detection unit 12 may include a needle detector for detecting the newcomer A's line-of-sight information. The sensor 13 is, for example, an acceleration sensor, a gyroscope, or the like. The sensor 13 detects the movement and tilt of the newcomer A's head.

For example, the control unit 14 causes the display unit 11 to display, for example, a state image of the robot based on the image data transmitted by the robot remote control device 300 .

For example, the control unit 14 generates information (hereinafter referred to as line-of-sight information) indicating the line-of-sight movement of the newcomer A based on the detection result of the line-of-sight detection unit 12 . The control unit 14 also generates information (hereinafter referred to as head movement information) indicating the movement and tilt of the head of the newcomer A based on the detection result of the sensor 13 . Line-of-sight information and head motion information are information included in operator sensor values, which will be described later. The control unit 14 causes the generated line-of-sight information and head motion information to be transmitted to the robot remote control device 300 via the first terminal device 40 .

The communication unit 15 is, for example, a wireless module for using Bluetooth (registered trademark), and has an antenna, a transmitting/receiving device, and the like. The communication unit 15 communicates with the first terminal device 40 via short-range wireless communication under the control of the control unit 14 . Not limited to this, the communication unit 15 may include a communication interface such as a NIC and communicate with the first terminal device 40, the robot 100, and the robot remote control device 300 via the network NW.

The sound pickup unit 16 is, for example, a microphone, and outputs a picked-up sound signal to the control unit 14 . The control unit 14 uses the communication unit 15 to transmit the acoustic signal to the robot remote control device 300 . When voice information is received from the robot remote control device 300 , the control unit 14 causes the speaker 17 to output the voice information. For example, when the robot remote control device 300 acquires the audio signal of the conversation of the newcomer A via the first HMD 10 , it provides the audio signal of the conversation of the newcomer A to the expert B via the second HMD 50 . For example, when the robot remote control device 300 acquires an audio signal of expert B's conversation via the second HMD 50, the robot remote control device 300 provides the audio signal of expert B's conversation to newcomer A via the first HMD 10. .

<Controller>
The first controller 20 comprises, for example, a sensor 21, a control section 22, a communication section 23, and a feedback means 24.

The sensor 21 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic force sensor, or the like. The control unit 22 tracks the movement of each finger based on detection results detected by multiple types of sensors included in the sensor 21, for example. Based on the detection result of the sensor 21, the control unit 22 also provides information indicating the posture and position of the operator's arm, such as orientation, movement of each finger, and movement of the hand (hereinafter referred to as operator arm information). ). The operator arm information includes, for example, hand position/orientation information, finger angle information, elbow position/orientation information, and information on the entire human arm such as movement tracking information. The operator arm information is an example of operation information. The control unit 22 transmits the acquired operator arm information to the robot remote control device 300 via the first terminal device 40 .

The communication unit 23 is a wireless module for using Bluetooth (registered trademark) or the like, and has an antenna, a transmitting/receiving device, and the like. The communication unit 23 communicates with the first terminal device 40 via short-range wireless communication under the control of the control unit 22 . Not limited to this, the communication unit 23 may include a communication interface such as a NIC and communicate with the first terminal device 40, the robot 100, and the robot remote control device 300 via the network NW.

The feedback means 24 feeds back feedback information (pressure, temperature, etc.) from the end effector of the robot 100 to the operator under the control of the control section 22 .

The feedback means 24 includes, for example, means for reproducing sensations and tactile sensations obtained on the robot 100 side and means for reproducing motions of the robot 100 . Means for reproducing the sensations and sensations obtained on the robot 100 side are, for example, means for reproducing the sensations and sensations when the robot hand is in contact with an object or the like while the robot 100 is working. The means for reproducing the motion of the robot 100 is, for example, means for reproducing the direction and speed of movement of the robot arm of the robot 100 .

For example, the feedback means 24 may include means for applying vibration (not shown), means for applying air pressure (not shown), means for restraining hand movement (not shown), means for feeling temperature (not shown), hardness and softness. A means (not shown) or the like for making the user feel a sense of urgency is provided. The feedback means 24 reproduces the sensations and sensations obtained on the robot 100 side and the motions of the robot 100 that are fed back from the robot 100 under the control of the control unit 22 . By doing so, the first controller 20 can transmit the remote operation of the newcomer A or the expert B to the newcomer A.

The controller 22 controls the feedback means 24 based on feedback information received from the robot remote control device 300 . Details of the feedback information will be described later.

<Environmental sensor>
The first environment sensor 30 includes, for example, a photographing device 31, a sensor 32, an object position detection section 33, a control section 34, and a communication section 35. Note that the second environment sensor 70 and the third environment sensor 200 also have similar functions, and detailed description thereof will be omitted.

The imaging device 31 is, for example, an RGB camera, and the sensor 32 is, for example, a depth sensor. Note that the imaging device 31 and the sensor 32 may be distance sensors. In the first environment sensor 30, the positional relationship between the imaging device 31 and the sensor 32 is known.

The object position detection unit 33 detects the three-dimensional position, size and shape of the target object in the captured image based on the image captured by the imaging device 31 and the detection result detected by the sensor 32 by a known method. do. For example, the object position detection unit 33 refers to a pattern matching model or the like stored in the object position detection unit 33, and performs image processing (edge detection, binarization processing, feature amount extraction, image enhancement processing, image extraction, pattern matching processing, etc.) to estimate the position of the object. Note that the object position detection unit 33 detects the position of each object when a plurality of objects are detected from the captured image.

The control unit 34 transmits information indicating the position of the object detected by the object position detection unit 33, the image captured by the imaging device 31, and the sensor value detected by the sensor 32 via the first terminal device 40. Send to the robot remote control device 300 . The information indicating the position of the object, the image captured by the imaging device 31, and the sensor values detected by the sensor 32 are included in the environmental sensor values described later.

The communication unit 35 is a wireless module for using Bluetooth (registered trademark) or the like, and has an antenna, a transmitting/receiving device, and the like. The communication unit 35 communicates with the first terminal device 40 via near field communication. Not limited to this, the communication unit 35 may include a communication interface such as a NIC and communicate with the first terminal device 40, the robot 100, and the robot remote control device 300 via the network NW.

The first environment sensor 30 may be a motion capture device, and may detect the position information of the object by motion capture. may be attached. In this case, the GPS receiver may transmit position information to the robot remote control device 300 .

<Example of robot configuration>
FIG. 3 is a functional block diagram of the robot 100. As shown in FIG. The robot 100 includes, for example, a communication unit 110, a sound pickup unit 120, a sensor 130, a driving unit 140, a robot arm 150, a power supply 160, a storage unit 170, and a control unit 180.

The communication unit 110 includes, for example, a communication interface such as NIC. The communication unit 110 is connected to the first terminal device 40, the second terminal device 80, the third environment sensor 200, and the robot remote control device 300 via the network NW under the control of the control unit 180. FIG. The communication unit 110 is a wireless module for using Bluetooth (registered trademark) or the like, and has an antenna, a transmitting/receiving device, and the like. 300.

The sound pickup unit 120 is, for example, a microphone array including a plurality of microphones. The sound pickup unit 120 outputs the picked-up sound signal to the control unit 180 . The sound pickup unit 120 may have a speech recognition processing function. In this case, the sound pickup unit 120 outputs the speech recognition result to the control unit 180 .

The sensor 130 is, for example, an acceleration sensor, a gyroscope sensor, a magnetic force sensor, each joint encoder, or the like. For example, the sensors 130 are attached to each joint, head, etc. of the robot 100 . Sensor 130 outputs information indicating the detected result to control unit 180 .

The driving section 140 drives each section (arms, fingers, legs, head, torso, waist, etc.) of the robot 100 under the control of the control section 180 . The drive unit 140 drives the robot arm 150, for example. The robot arm 150 includes, for example, actuators, gears, artificial muscles, and the like. The robot arm 150 also has a hand sensor 151 . The hand sensor 151 includes, for example, means for sensing temperature (not shown), means for sensing hardness and softness (not shown), and the like.

A power supply 160 supplies power to each part of the robot 100 . Power source 160 may comprise, for example, a rechargeable battery or charging circuit.

The storage unit 170 is realized by, for example, RAM (Random Access Memory), ROM (Read Only Memory), HDD (Hard Disk Drive), and the like. The storage unit 170 stores programs executed by the processor, and various information such as operation parameters for operating the robot 100 .

The control unit 180 performs control to operate the robot 100 by remote control and control to operate the robot 100 without using remote control. In the former case, the controller 180 operates the robot 100 according to control information generated by the robot remote control device 300 .

In the latter case, the control unit 180 performs speech recognition processing (speech segment detection, sound source separation, sound source localization, noise suppression, sound source identification, etc.) on the acoustic signal collected by the sound collection unit 120 . When the voice recognition result includes an action instruction for the robot 100, the control unit 180 controls the action of the robot 100 based on the action instruction by voice. Based on the information stored in the storage unit 170, the control unit 180 performs image processing (edge detection, binarization processing, feature amount extraction, image enhancement processing, Image extraction, pattern matching processing, etc.) may be performed, and the motion of the robot 100 may be controlled based on the results of voice recognition and image processing.

<Configuration example of robot remote control device 300>
FIG. 4 is a functional block diagram of the robot remote control device 300. As shown in FIG. The robot remote control device 300 includes, for example, a communication unit 310, an input unit 320, an output unit 330, a storage unit 340, and a processing unit 350.

The communication unit 310 includes, for example, a communication interface such as NIC. The communication unit 310 communicates with the first terminal device 40, the second terminal device 80, the robot 100, and the third environment sensor 200 via the network NW under the control of the processing unit 350. FIG.

The input unit 320 includes, for example, some or all of various keys, buttons, dial switches, a mouse, and the like. The output unit 330 includes, for example, an LCD or organic EL display device, a speaker, a projector, and the like. The storage unit 340 is realized by RAM, ROM, HDD, and the like, for example. The storage unit 340 stores programs executed by the processor, robot information 341, feedback parameter information 342, and the like. Details of each information will be described later.

The processing unit 350 includes, for example, an information acquisition unit 351, an intention estimation unit 352, a robot operation determination unit 353, a control information generation unit 354, a feedback information generation unit 355, a robot state image generation unit 356, a comparison image A generation unit 357 and a communication control unit 358 are provided. Some or all of these functional units are realized by executing a program (software) by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). Some or all of these components are implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), and FPGA (Field-Programmable Gate Array). It may be realized by cooperation of software and hardware. The program may be stored in advance in the storage unit 340 of the robot remote control device 300, or may be stored in a detachable storage medium such as a DVD or CD-ROM, and the storage medium is installed in the drive device. It may be installed in the storage unit 340 by

For example, the information acquisition unit 351 acquires first operation information from the first terminal device 40 via the network NW. The first operation information is information indicating an operation input by the newcomer A via the first controller 20 that receives an operation on the robot 100 to be remotely operated. Also, the information acquisition unit 351 acquires the second operation information from the second terminal device 80 via the network NW. The second operation information is information indicating an operation input by the expert B via the second controller 60 that receives an operation on the robot 100 to be remotely operated. The first operation information and the second operation information include, for example, operator arm information.

The information acquisition unit 351 also acquires the first operator sensor value information from the first terminal device 40 via the network NW. The first operator sensor value information is information indicating the detection result of a sensor detecting a state change of the newcomer A who is the first operator. Contains information indicating the detection result of The information acquisition unit 351 also acquires the second operator sensor value information from the second terminal device 80 via the network NW. The second operator sensor value information is information indicating the result of detection by a sensor of the state change of the expert B who is the second operator. It contains information indicating the detection result of the sensor. The first operator sensor value information and the second operator sensor value information include line-of-sight information and head motion information detected by the first HMD 10 or the second HMD 50, for example.

Also, the information acquisition unit 351 acquires the first environmental sensor value information from the first terminal device 40 via the network NW. The first environment sensor value information is information indicating the detection result of a sensor detecting a change in the environment around the robot 100 remotely operated by the newcomer A, who is the first operator. 30 includes information indicating the detection results of the sensors mounted on the device. Also, the information acquisition unit 351 acquires the second environmental sensor value information from the second terminal device 80 via the network NW. The second environment sensor value information is information indicating the detection result of a sensor detecting a change in the environment around the robot 100 remotely controlled by the expert B who is the second operator. It includes information indicating the detection result of the sensor mounted on the sensor 70 . The information acquisition unit 351 also acquires third environment sensor value information from the third environment sensor 200 via the network NW. The third environment sensor value information is information indicating the result of detection by sensors of changes in the environment around the robot 100 remotely controlled by the first operator or the second operator. It includes information indicating the detection result of the sensor mounted on the sensor 200 . The first environmental sensor value information, the second environmental sensor value information, and the third environmental sensor value information include, for example, object position information, an image captured by the imaging device 31, and sensor values detected by the sensor 32. be

The intention estimation unit 352 estimates the operator's action intention based on the information acquired by the information acquisition unit 351 . For example, the intention estimation unit 352 estimates the operator's motion intention using at least one of line-of-sight information, operator arm information, and head motion information. Note that the intention estimation unit 352 may also estimate the intention using the environmental sensor value.

Further, the intention estimating unit 352 inputs the operator sensor value and line-of-sight information acquired by the information acquiring unit 351 to the learned model, thereby estimating the intention of the action that the operator is about to perform. good. By generating control information based on such an estimation result and controlling the robot 100 based on the generated control information, the robot 100 can be controlled even when the operator does not perform a remote operation for accurate positioning. can also perform accurate registration. Therefore, it is possible to cause the robot 100 to perform the work with high accuracy. Note that the learned model is, for example, data obtained by inputting operator sensor values and line-of-sight information in advance and learning the work that the operator is about to perform as teacher data.

The robot operation determining unit 353 determines the motion intention estimated by the intention estimating unit 352, the detection result detected by the sensor 130 of the robot 100, the image processing result of the image captured by the imaging device of the third environment sensor 200, and the like. Based on this, the operation method for the operation object operated by the robot 100 is determined. The operation method for the operation object includes various operation methods such as a grasping method for grasping the operation object, a movement method for moving the operation object, a deformation method for deforming the operation object, and the like.

The control information generation unit 354 refers to the storage unit 340 and generates control information for operating the robot 100 to be remotely operated based on the first operation information or the second operation information acquired by the information acquisition unit 351. . For example, the control information generation unit 354 refers to the robot information 341, acquires operation parameters according to the operation by the operator, and generates control information for causing the robot 100 to operate based on the acquired operation parameters. The robot information 341 is, for example, information in which motion parameters corresponding to operation amounts are associated with each other.

The control information generation section 354 may generate control information for operating the robot 100 based on the operation method determined by the robot operation determination section 353 . By doing so, it is possible to estimate the operator's intention, and to cause the robot 100 to perform an action closer to the operator's intention based on the estimation result.

Further, the control information generator 354 may generate control information for operating the robot 100 based on the first operator sensor information and the first environment sensor value information in addition to the first operation information. Further, the control information generator 354 may generate control information for operating the robot 100 based on the second operator sensor information and the second environment sensor value information in addition to the second operation information.

Furthermore, the control information generation unit 354 controls the robot 100 based on the result of synthesizing the details of the remote operation by the newcomer A who is the first operator and the details of the remote operation by the expert B who is the second operator. You may generate|occur|produce the control information which operates. For example, the control information generation unit 354 generates control information based on the total value of the operation parameter according to the first operation information and the operation parameter according to the second operation information. The control information generator 354 may generate the control information based on a value obtained by subtracting the operation parameter according to the second operation information from the operation parameter according to the first operation information. Note that the method of synthesizing the details of the remote operation by the newcomer A, who is the first operator, and the details of the remote operation by the expert B, who is the second operator, is not limited to this, and various existing methods can be adopted. be able to. By doing so, the robot 100 can be operated according to remote control by two or more operators. Therefore, it becomes possible for the expert B to guide the work of the newcomer A in real time.

Further, when synthesizing the content of the remote operation by the first operator and the content of the remote operation by the second operator, the control information generation unit 354 determines that the operation of the second operator is higher than that of the first operator. You may generate|occur|produce the control information which gave priority to. For example, when the operation by the first operator and the operation by the second operator are performed simultaneously, the control information generation unit 354 performs the first weighting on the operation parameter based on the operation by the first operator. , performing a second weighting greater than the first weighting on the operating parameter based on the operation by the second operator to generate control information. The first weighting may be 0, and the operation by the first operator may not be reflected in the control information. Further, when the operation direction by the first operator and the operation direction by the second operator are different, the control information generation unit 354 may generate control information so as to operate in the operation direction by the second operator. By doing so, the remote operation by the second operator can be reflected in the motion of the robot 100 rather than the remote operation by the first operator, and the remote operation by the first operator can be corrected to the remote operation by the second operator. can do.

Further, when synthesizing the details of the remote operation by the first operator and the details of the remote operation by the second operator, the control information generation unit 354 combines the timing of the remote operation by the first operator and the remote operation by the second operator. You may make it perform operation timing continuously. For example, the control information generation unit 354 may generate control information that causes the second operator to perform the action corresponding to the remote operation immediately after the first operator has performed the action corresponding to the remote operation. By doing so, it is possible to correct the remote operation by the second operator immediately after the remote operation by the first operator.

The feedback information generation unit 355 transmits the operation details of the first operator to the second operator via the second controller 60 based on the motion of the robot 100 based on the operation input by the newcomer A who is the first operator. generate second feedback information for communication to the . Further, the feedback information generation unit 355 transmits the details of the operation by the second operator via the first controller 20 based on the action of the remote-operated object based on the operation input by the expert B who is the second operator. A first feedback information is generated for communication to the first operator. By doing so, the motion of the robot or the like remotely controlled by one operator can be transmitted to the other operator.

For example, the feedback information generation unit 355 refers to the feedback parameter information 342 in the storage unit 340, and based on the control information generated by the control information generation unit 354, determines the operation content of the robot 100 to be remotely operated by the first controller. generating first feedback information to be fed back to 20; Further, the feedback information generation unit 355 refers to the feedback parameter information 342 in the storage unit 340, and based on the control information generated by the control information generation unit 354, the operation content of the robot 100 to be remotely operated is transmitted to the second controller. generating second feedback information to be fed back to 60; The feedback information generator 355 refers to the feedback parameter information 342, acquires feedback parameters corresponding to the control information, and generates feedback information based on the acquired feedback parameters. The feedback parameter information 342 is, for example, information that associates feedback parameters corresponding to operation parameters and manipulated variables of the robot 100 .

For example, when the robot 100 is remotely operated by the first operator, the feedback information generation unit 355 generates the remote operation information in order to feed back the contents of the remote operation by the first operator to the motion of the robot 100 to the second operator. Second feedback information is generated for causing the second controller 60 to reproduce the motion of the robot 100 that has been detected. On the other hand, when the robot 100 is remotely operated by the second operator, the feedback information generating unit 355 generates the remote operation information in order to feed back the contents of the remote operation by the second operator to the first operator. First feedback information for causing the first controller 20 to reproduce the motion of the robot 100 is generated.

As described above, when the control information generator 354 generates the control information based on the synthesis result of the contents of the remote operation by the first operator and the contents of the remote operation by the second operator, the feedback information generator 355 is first feedback information for causing the first controller 20 to reproduce the motion of the remotely-operated robot 100 in order to feed back the motion of the robot 100 based on the synthesized result to the first operator and the second operator; , may generate second feedback information for causing the second controller 60 to reproduce the motion of the remotely-operated robot 100 .

In addition, as described above, when synthesizing the content of the remote operation by the first operator and the content of the remote operation by the second operator, the control information generation unit 354 determines that the second operator When the control information giving priority to the remote operation by the operator is generated, the feedback information generating unit 355 generates the first operator's feedback of the motion of the robot 100 preferentially to the first operator over the second operator. 1 feedback information may be generated. For example, the feedback information generation unit 355 generates first feedback information and second feedback information in which the content of the feedback to the first operator is emphasized by comparing the content of the feedback to the second operator. Emphasizing the content of feedback includes, for example, applying vibration, applying air pressure with stronger pressure, restricting hand movement, and the like.

Further, the feedback information generation unit 355 transmits sensations and sensations obtained on the robot 100 side to the first operator via the first controller 20 based on the detection results detected by the sensor 130 and the hand sensor 151 of the robot 100 . generate third feedback information for communication to the Further, the feedback information generation unit 355 transmits sensations and sensations obtained on the robot 100 side to the second operator via the second controller 60 based on the detection results detected by the sensor 130 and the hand sensor 151 of the robot 100 . generate fourth feedback information for transmission to the By doing so, it is possible to convey to the operator the sensations and tactile sensations obtained when the operator actually operates the remote operation target instead of the remote operation using the robot 100 .

The robot state image generation unit 356 performs image processing (edge detection, binarization, feature quantity extraction, image enhancement, image extraction, clustering processing, etc.) on the image captured by the imaging device 31 of the third environment sensor 200, for example. I do.

In addition, the robot state image generation unit 356 generates the image of the robot 100 based on the information indicating the result of the above-described image processing (hereinafter referred to as image processing result information), the control information generated by the control information generation unit 354, and the like. The position and motion of the hand are estimated, the motion of the operator's hand is estimated, and a robot state image to be displayed on the first HMD 10 and the second HMD 50 is created based on the estimation result.

The robot state image may include system state information indicating the state of the system, such as information about the process that the robot remote control device 300 is about to perform and error information.

The comparison image generation unit 357 generates image data of a comparison image showing a comparison result between the operation input by the first operator and the operation input by the second operator. The image data of the comparison image is displayed on the first display unit for the first operator (for example, the display unit of the first HMD 10) or the second display unit for the second operator (for example, the display unit of the second HMD 50). This is the image data to be displayed on the . The comparative image may be, for example, an image displaying the difference between the remote operation by the first operator and the remote operation by the second operator, or an image emphasizing the remote operation by the second operator rather than the remote operation by the first operator. including images. By doing so, the first operator can check the erroneous operation corrected by the second operator, the difference between his operation and the operation by the second operator, and the like on the image.

The communication control unit 358 controls the communication unit 310 to communicate with the first terminal device 40, the second terminal device 80, the robot 100, and the third environment sensor 200 via the network NW.

<Example of comparison image>
FIG. 5 is a diagram showing an example of a comparison image. The illustrated comparison image is an example of a comparison image that is generated when the newcomer A and the expert B perform a remote operation for transforming the manipulation object 500, for example. The illustrated comparison image shows the left-hand robot arm 150L of the robot 100, and omits the right-hand robot arm of the robot 100 and other images.

For example, a newcomer A remotely controls the robot 100 to make a teacup using a potter's wheel. The expert B remotely operates the robot 100 to support the newcomer A in making hot water only. Assume that the manipulation object 500 is, for example, clay placed on a potter's wheel. The work performed by the newcomer A and the expert B is not limited to this, and includes various work such as lathe processing, plate painting, and glass work.

The robot arm image LA1 is an image showing the state of the robot arm 150L when remotely operated by the newcomer A at the first timing. The robot arm image LA2 is an image showing the state of the robot arm 150L when remotely operated by the newcomer A at the second timing. The robot arm image LB3 is an image showing the state of the robot arm 150L when remotely operated by the expert B at the third timing. The first timing is the most past timing, the second timing is the next timing after the first timing, and the third timing is the next timing after the second timing.

From the illustrated comparative image, it can be seen that the expert B corrects the operation of the newcomer A at the first timing and the second timing to the position at the third timing. By displaying this comparative image on the first HMD 10, the newcomer A can easily recognize the difference between the remote control by himself and the remote control by the expert B. Further, by displaying this comparative image on the second HMD 50, the expert B can share the comparative image with the newcomer A and verbally communicate points of work and points to be corrected.

In the drawing, an example is shown in which two arm images are displayed in the comparison image when remote-controlled by newcomer A, but the number of arm images is not limited to this, and may be one. Moreover, although an example in which one arm image is displayed when being remotely operated by the expert B is shown in the comparison image, the present invention is not limited to this, and two or more arm images may be displayed.

<Sequence diagram (Part 1)>
FIG. 6 is a sequence diagram showing an example of processing in the robot remote control system 1. As shown in FIG. Here, an example in which the second operator performs the remote operation after the first operator performs the remote operation, and an example in which the robot 100 is controlled based on the operation information will be described.

First, the first controller 20 generates first operation information based on the operation input by the first operator (step S1), and transmits the first operation information to the robot remote control device via the first terminal device 40. 300 (step S2). The robot remote control device 300 generates control information based on the received first operation information (step S3) and transmits it to the robot 100 (step S4). The robot 100 operates the robot arm 150 and the like based on the received control information (step S5). On the other hand, the robot remote control device 300 generates second feedback information based on the control information generated in step S3 (step S6), and transmits the generated second feedback information to the second terminal device 80 via the second terminal device 80. 2 to the controller 60 (step S7). The second controller 60 reproduces the motion of the robot 100 remotely controlled by the first operator in step S1 based on the received feedback information (step S8).

Next, the second controller 60 generates second operation information based on the operation input by the second operator (step S11), and transmits the second operation information via the second terminal device 80 to the robot remote control device. 300 (step S12). The robot remote control device 300 generates control information based on the received second operation information (step S13) and transmits it to the robot 100 (step S14). The robot 100 operates the robot arm 150 and the like based on the received control information (step S15). On the other hand, the robot remote control device 300 generates first feedback information based on the control information generated in step S13 (step S16), and transmits the generated first feedback information to the first terminal device 40 via the first terminal device 40. 1 controller 20 (step S17). The first controller 20 reproduces the motion of the robot 100 remotely controlled by the second operator in step S11 based on the received feedback information (step S18).

<Flowchart (Part 1)>
FIG. 7 is a flowchart showing an example of processing by the robot remote control device 300. As shown in FIG. Here, an example in which the second operator performs the remote operation after the first operator performs the remote operation, and an example in which the robot 100 is controlled based on the operation information will be described. Also, processing by the intention estimation unit 352 and the robot operation determination unit 353 and processing by the robot state image generation unit 356 are omitted.

First, the information acquisition unit 351 determines whether or not first operation information (or second operation information) has been acquired (step S101). When the first operation information (or the second operation information) is acquired, the control information generation unit 354 generates control information for operating the robot 100 based on the acquired first operation information (or the second operation information). (Step S102), the generated control information is transmitted to the robot 100 (Step S103).

Next, feedback information generation section 355 generates second feedback information when control information is generated based on the first operation information, and generates second feedback information when control information is generated based on the second operation information. 1 feedback information is generated (step S104). The communication control unit 358 then transmits the second feedback information to the second controller 60 and the first feedback information to the first controller 20 (step S105).

Next, when the control information is generated based on the first operation information and the control information is generated based on the second operation information before that, the comparison image generation unit 357 generates the first operation information. Image data of a first comparison image showing a comparison result between the operation input by the operator and the operation input by the second operator is generated (step S106). On the other hand, when the control information is generated based on the second operation information and the control information is generated based on the first operation information before that, the comparison image generation unit 357 generates the second operator's image data of a second comparison image showing a comparison result between the operation input by the first operator and the operation input by the first operator. The communication control unit 358 then transmits the image data of the first comparison image to the first HMD 10, and transmits the image data of the second comparison image to the second HMD 50 (step S107).

<Sequence diagram (Part 2)>
FIG. 8 is a sequence diagram showing another example of processing in the robot remote control system 1. As shown in FIG. Here, an example will be described in which the remote operation by the first operator and the remote operation by the second operator are performed at the same timing, and the robot 100 is controlled based on the operation information. The same timing is not limited to simultaneous, and includes those with a time difference of several seconds.

First, the first controller 20 generates first operation information based on the operation input by the first operator (step S21), and transmits the first operation information via the first terminal device 40 to the robot remote control device. 300 (step S22). At the same timing, the second controller 60 generates the second operation information based on the operation input by the second operator (step S23), and transmits the second operation information via the second terminal device 80 to the remote robot. It is transmitted to the control device 300 (step S24).

Next, based on the received first operation information and second operation information, the robot remote operation control device 300 generates control information that combines the content of the remote operation by the first operator and the content of the remote operation by the second operator. is generated (step S25) and transmitted to the robot 100 (step S26). The robot 100 operates the robot arm 150 and the like based on the received control information (step S27).

Next, the robot remote control device 300 generates second feedback information based on the control information generated in step S25 (step S28), and transmits the generated second feedback information to the second terminal device 80 via the second terminal device 80. 2 to the controller 60 (step S29). The second controller 60 reproduces the motion of the robot 100 remotely controlled by the first operator in step S21 based on the received feedback information (step S30).

Further, the robot remote control device 300 generates first feedback information based on the control information generated in step S25 (step S31), and transmits the generated first feedback information to the first terminal device 40 via the first terminal device 40. 1 controller 20 (step S32). The first controller 20 reproduces the motion of the robot 100 remotely controlled by the second operator in step S23 based on the received feedback information (step S33).

<Flowchart (Part 2)>
FIG. 9 is a flowchart showing another example of processing by the robot remote control device 300. FIG. Here, an example will be described in which the remote operation by the first operator and the remote operation by the second operator are performed at the same timing, and the robot 100 is controlled based on the operation information. Also, processing by the intention estimation unit 352 and the robot operation determination unit 353 and processing by the robot state image generation unit 356 are omitted.

First, the information acquisition unit 351 determines whether or not the first operation information and the second operation information have been acquired (step S201). When the first operation information and the second operation information are acquired, the control information generation unit 354 generates the content of the remote operation by the first operator and the second operation based on the acquired first operation information and the second operation information. Control information is generated by synthesizing the details of the remote operation by the operator (step S202), and the generated control information is transmitted to the robot 100 (step S203).

Next, feedback information generator 355 generates first feedback information and second feedback information based on the control information generated in step S202 (step S204). The communication control unit 358 then transmits the first feedback information to the first controller 20 and the second feedback information to the second controller 60 (step S205).

Next, the comparison image generator 357 generates image data of a comparison image based on the first operation information and the second operation information acquired in step S201 (step S206). The communication control unit 358 then transmits the generated image data of the comparison image to the first HMD 10 and the second HMD 50 (step S207).

According to the above-described embodiment, in a remote operation control system in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world, the first controller 20 receives an operation for the remote operation target. Acquire first operation information indicating an operation input by one operator and second operation information indicating an operation input by a second operator via a second controller 60 that receives an operation for a remote operation target. Based on the information acquisition unit 351, the first operation information and the second operation information acquired by the information acquisition unit 351, a control information generation unit 354 that generates control information for operating the remote operation target, and based on the control information: a feedback information generation unit 355 that generates first feedback information for feeding back the details of the operation of the remote-controlled object to the first controller 20 and generates second feedback information for feeding back the details of the operation of the remote-controlled object to the second controller 60; , it is possible to experience the operation of the same remote-operated object by another operator.

The embodiment described above can be expressed as follows.
a storage device storing a program;
a hardware processor;
By the hardware processor executing the program stored in the storage device,
In a remote operation control system in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world,
First operation information indicating an operation input by the first operator via the first controller 20 that receives an operation on the remotely operated target, and input by the second operator via the second controller 60 that accepts an operation on the remotely operated target: obtaining second operation information indicating the input operation;
generating control information for operating a remote operation target based on the acquired first operation information and second operation information;
Based on the control information, generate first feedback information that feeds back the action content of the remote-controlled target to the first controller 20, and generate second feedback information that feeds back the action content of the remote-controlled target to the second controller 60;
A remote operation control system configured to:

As described above, the mode for carrying out the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be made without departing from the scope of the present invention. can be added.

For example, part of the robot remote control device 300 may be mounted on the robot 100 . For example, the robot 100 may perform the functions of the processing unit 350 and the storage unit 340 .

1... Remote operation control system 10... First HMD
20... First controller 30... First environment sensor 40... First terminal device 50... Second HMD
60... Second controller 70... Second environment sensor 80... Second terminal device 100... Robot 200... Third environment sensor 300... Robot remote control device 351... Information acquisition unit 352... Intention estimation unit 353... Robot operation determination unit 354 ... control information generation unit 355 ... feedback information generation unit 356 ... robot state image generation unit 357 ... comparison image generation unit 358 ... communication control unit

Claims (8)

In a remote operation control system in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world,
first operation information indicating an operation input by a first operator through a first controller that accepts an operation on the remote operation target; and by a second operator through a second controller that accepts an operation on the remote operation target an information acquisition unit that acquires second operation information indicating an input operation;
a control information generation unit that generates control information for operating the remote operation target based on the first operation information and the second operation information acquired by the information acquisition unit;
Based on the control information, first feedback information is generated for feeding back the operation content of the remote-controlled object to the first controller, and second feedback information is generated for feeding back the operation content of the remote-controlled object to the second controller. a feedback information generator that generates;
A remote operation control system. The feedback information generation unit
A second controller for transmitting the operation content of the first operator to the second operator via the second controller based on the action of the remote-operated object based on the operation input by the first operator. 1 generate feedback information;
a second operator for transmitting the operation content of the second operator to the first operator via the first controller based on the action of the remote-operated object based on the operation input by the second operator; 2 generate feedback information;
The remote operation control system according to claim 1. image data to be displayed on a first display unit for the first operator or a second display unit for the second operator, wherein the operation input by the first operator and the second A comparison image generation unit that generates a comparison image showing a comparison result with the operation input by the operator,
3. The remote operation control system according to claim 1 or 2. The control information generation unit generates the control information based on a result of synthesizing the content of the remote operation by the first operator and the content of the remote operation by the second operator.
The remote operation control system according to any one of claims 1 to 3. The control information generation unit is configured to, when synthesizing the contents of the remote operation by the first operator and the contents of the remote operation by the second operator, the operation of the second operator compared to the first operator. generating the control information prioritizing
The remote operation control system according to claim 4. The feedback information generation unit
third feedback information for transmitting a sensation or feeling obtained on the side of the remote-operated target to the first operator via the first controller, based on a detection result by a sensor mounted on the remote-operated target; to generate
Fourth feedback information for transmitting a sensation or feeling obtained on the side of the remotely operated object to the second operator via the second controller based on the detection result by the sensor mounted on the remote operated object. to generate
The remote operation control system according to any one of claims 1 to 5. the computer
In remote operation control in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world,
first operation information indicating an operation input by a first operator through a first controller that accepts an operation on the remote operation target; and by a second operator through a second controller that accepts an operation on the remote operation target Acquiring second operation information indicating the input operation,
generating control information for operating the remote control target based on the acquired first operation information and the second operation information;
Based on the control information, first feedback information is generated for feeding back the operation content of the remote-controlled object to the first controller, and second feedback information is generated for feeding back the operation content of the remote-controlled object to the second controller. generate,
Remote operation control method. In remote operation control in which an operator remotely operates a remote operation target, which is a real object or a virtual object in a virtual world,
to the computer,
first operation information indicating an operation input by a first operator through a first controller that accepts an operation on the remote operation target; and by a second operator through a second controller that accepts an operation on the remote operation target obtaining second operation information indicating the input operation;
generating control information for operating the remote control target based on the acquired first operation information and the second operation information;
Based on the control information, first feedback information is generated for feeding back the operation content of the remote-controlled object to the first controller, and second feedback information is generated for feeding back the operation content of the remote-controlled object to the second controller. generate,
program.

Images