JP2023009642A - Remote operation system, operation device and remote operation support method - Google Patents

Abstract

To provide a remote operation system which makes a remote operator to easily move at the time of remote operation, enables remote operation for a long time, and facilitates learning of an operation method of an operation device.SOLUTION: A remote operation system includes a robot device having a first moving mechanism, and an operation device for instructing control of operation of the robot device from an operation site. The operation device has a display device, and a second moving mechanism. The operation device receives a first image imaged by the robot device, displays the first image on the display device, and transmits an instruction signal including an instruction to move the robot device to the robot device when detecting operation of moving the operation device by an operator. The robot device receives the instruction signal, and controls the first moving mechanism so as to move the robot device corresponding to movement of the operator on the basis of the instruction signal.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to a remote control system, an operator, and a remote control support method.

Conventionally, an inspection support system that supports inspection work at a remote location is known as a system that supports work at a remote location (see Patent Document 1). This inspection support system includes a field device including an unmanned mobile body and a remote device including a virtual reality device. The unmanned mobile body controls a drive mechanism, a virtual reality camera that captures images of the inspection site, a microphone that collects the sound of the inspection site, an arm, and a drive mechanism to move the unmanned mobile body and control the arm. and a control unit that operates the equipment to be inspected. In addition, the virtual reality device includes a display processing unit that displays an image of the inspection site according to the orientation of the inspector so that the inspector can see it on the display unit based on the information of the image obtained by the virtual reality camera, and a microphone. a sound output processing unit for outputting the sound of the inspection site from a speaker based on the obtained sound information.

Japanese Patent Application Laid-Open No. 2020-149349

In the system of Patent Literature 1, when the virtual reality device is smart glasses, the display unit is an augmented reality display that superimposes an image of an area corresponding to the orientation of the inspector on the outside scenery and allows the inspector to visually recognize the image. Therefore, it may be difficult for the inspector to move around the inspection site due to the reduced visibility of the outside scenery depending on the state of the displayed image of the inspection site. It is also conceivable to employ an HMD (Head Mount Display) as a virtual reality device from the viewpoint of enhancing the stereoscopic effect and sense of distance by displaying visual information, which is an image of the inspection site. However, since the HMD blocks the operator's entire field of view, it is more difficult to move at the inspection site, and the HMD is heavy, so it is difficult for the inspector to work for a long time.

Further, for example, a controller is used for remote control of an unmanned mobile unit placed at a remote inspection site. However, a certain level of skill is required to operate a controller for remotely controlling an unmanned mobile object, and high training costs are required. In addition, when an input operation or the like is required during remote operation, it is necessary to temporarily remove the HMD worn by the inspector and use an input device such as a PC that is different from the controller. Therefore, the work efficiency of the input work is insufficient.

The present disclosure provides a remote control system, a controller, and a remote control method that allow a remote operator to easily move during remote control, enable remote control for a long period of time, and easily learn how to operate the controller. I will provide a.

One aspect of the present disclosure is a remote control system that includes a robot device that includes a first movement mechanism, and an operator that instructs control of the operation of the robot device from an operation site, wherein the operator: A display device and a second movement mechanism, wherein the operator receives a first image captured by the robot apparatus, causes the display device to display the first image, and is operated by an operator. When an operation to move the manipulator is detected, an instruction signal including an instruction to move the robot apparatus is transmitted to the robot apparatus, and the robot apparatus receives the instruction signal and based on the instruction signal. and a remote control system that controls the first moving mechanism so that the robot device moves in accordance with the movement of the operating device.

One aspect of the present disclosure is an operation device that instructs control of the operation of a robot device that includes a first movement mechanism from an operation site, and includes a processor, a communication device, a display device, and a second movement mechanism. , wherein the processor receives a first image captured by the robot apparatus via the communication device, causes the display device to display the first image, and operates the operation device by an operator. The operator transmits an instruction signal including an instruction to move the robot apparatus via the communication device when an operation to move is detected.

One aspect of the present disclosure is a remote operation support method for instructing control of an operation of a robot device having a first movement mechanism from an operation site, the method receiving a first image captured by the robot device, Displaying the first image on a display device, and transmitting an instruction signal including an instruction to move the robot device to the robot device when an operator's operation to move the operation device including the display device is detected. , is a remote control support method.

According to the present disclosure, it is easy for a remote operator to move during remote operation, long-term remote operation is possible, and it is easy to learn how to operate the operation device.

1 is a block diagram showing a configuration example of a remote control system according to an embodiment of the present disclosure; FIG. Schematic diagram showing an example of the appearance of an object placed at a remote site and a remote-controlled robot Schematic diagram showing an example of the external appearance of the operation device and the operator placed at the operation site A diagram for explaining an example of detection of an input operation based on input from an input sensor. A diagram showing a display example of information on an object existing outside the display range and within the imaging range. A diagram showing an example of an instruction of the direction of the remote camera based on a swipe operation on the operation device. A diagram showing an example of an instruction to move the remote control robot based on a swipe operation on the operation device. A diagram showing an example of an alert display when there are obstacles around the remote control robot. Schematic diagram showing an example of a form in which a plurality of operating devices are linked Schematic diagram showing an example of a form in which a plurality of operating devices are linked Schematic diagram showing an example of a form in which a plurality of operating devices are linked Flowchart showing an operation example of the remote control system Schematic diagram showing the configuration of a remote control robot in a modified example

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art. It should be noted that the accompanying drawings and the following description are provided to allow those skilled in the art to fully understand the present disclosure and are not intended to limit the claimed subject matter thereby.

For example, the term "unit" or "apparatus" as used in the embodiments is not limited to a physical configuration that is mechanically implemented simply by hardware, but also includes a configuration that implements the functions of the configuration by software such as a program. Also, the function of one configuration may be implemented by two or more physical configurations, or the functions of two or more configurations may be implemented by, for example, one physical configuration.

(Configuration of remote control system)
FIG. 1 is a block diagram showing a configuration example of a remote control system 5 according to an embodiment of the present disclosure. FIG. 2 is a schematic diagram showing an appearance example of the object 100 and the remote control robot 10 placed at the remote site LN1. FIG. 3 is a schematic diagram showing an appearance example of the operation device 30 and the operator 200 arranged at the operation site LN2.

In this embodiment, in FIG. 2, the left-right direction of the remote-controlled robot 10 facing the object 100 is defined as the x1-axis direction, and the direction in which the remote-controlled robot 10 approaches or moves away from the object is defined as the front-back direction in FIG. The vertical direction perpendicular to the horizontal direction is taken as the up-down direction and taken as the z1-axis direction. In FIG. 3, the left-right direction of the operator 30 facing the operator 200 is defined as the x2-axis direction, the direction in which the operator 200 approaches or moves away from the operator 30 is defined as the front-rear direction, and the y2-axis direction is defined as the direction perpendicular to the horizontal direction. Let the vertical direction be the z2-axis direction with the up-down direction. Therefore, the x1y1 plane and the x2y2 plane are along the horizontal direction.

The remote control system 5 includes one or more remote control robots 10 and one or more controllers 30 . The remote control robot 10 and the manipulator 30 are arranged at different locations, for example. As shown in FIG. 2, the remote-controlled robot 10 is placed, for example, at a remote site LN1. A remote control robot 10 and an object 100 are arranged at the remote site LN1. A target object 100 is an object to be manipulated by the remote control robot 10 . In FIG. 2, a plurality of objects 100 are placed on the shelf. Further, as shown in FIG. 3, the operation device 30 is arranged, for example, at the operation site LN2 (eg, own room (home room)). An operator 30 is arranged and an operator 200 is located at the operation site LN2. An operator 200 is a person who operates the operation device 30 .

As shown in FIG. 1, the remote control robot 10 includes a processor 11, a power supply 12, a memory 13, a communication device 14, a microphone 15 (microphone), an environment sensor 16, a remote camera 17, and a platform mechanism. 18 , an arm mechanism 19 (arm mechanism), and a moving mechanism 20 .

The processor 11 may include an MPU (Micro processing Unit), a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphical Processing Unit). The processor 11 may be configured by various integrated circuits (eg, LSI (Large Scale Integration), FPGA (Field Programmable Gate Array)). The processor 11 implements various functions by executing programs held in the memory 13 . The processor 11 comprehensively controls each part of the remote control robot 10 and performs various processes.

The power supply 12 supplies necessary power to each part of the remote control robot 10 via the processor 11 .

The memory 13 includes a primary storage device (eg, RAM (Random Access Memory) or ROM (Read Only Memory)). The memory 13 may include a secondary storage device (for example, HDD (Hard Disk Drive) or SSD (Solid State Drive)) or a tertiary storage device (for example, optical disk or SD card). Also, the memory 13 may be an external storage medium. The memory 13 stores various data, information, programs, and the like.

The communication device 14 communicates various data or information. The communication device 14 communicates according to a wired or wireless communication method. The communication method is WAN (Wide Area Network), LAN (Local Area Network), or cellular communication for mobile phones (for example, LTE, 5G), or short-range communication (for example, infrared communication or Bluetooth (registered trademark) communication) or Power line communication or the like may be used.

The microphone 15 picks up sounds around the remote control robot 10 . The sounds here broadly include environmental sounds, voices, music, or other sounds of the remote site LN1. The microphone 15 is installed anywhere on the remote control robot 10 . The microphones 15 correspond to the ears of the remote control robot 10, for example, and are arranged at both ends of the upper end of the remote camera 17 in the x1-axis direction.

The environment sensor 16 is a sensor that detects arbitrary information in the environment in which the remote control robot 10 is placed. The environment sensor 16 may be a depth camera, a stereo camera, a three-dimensional LiDAR, or the like capable of acquiring distance information in the surrounding environment of the remote control robot 10 .

Based on the information detected by the environment sensor 16, the processor 11 generates a three-dimensional model of the remote site LN1 (a model of the surrounding environment of the remote control robot 10) (also referred to simply as a surrounding environment model), and stores it in the memory 13. can be stored. The peripheral environment model of the remote control robot 10 can be used for avoiding obstacles when the remote control robot 10 moves. The surrounding environment model may be generated while the remote control robot 10 moves and changes the current position sequentially. For example, the processor 11 simultaneously performs self-location estimation and environment map generation (map within the remote site LN1) according to the SLAM (Simultaneous Localization And Mapping) algorithm. Thereby, the processor 11 generates an environment map even in an unknown environment.

The environment sensor 16 acquires distance information of the surrounding environment of the remote control robot 10 periodically (for example, about 1 Hz to 30 Hz). The processor 11 measures the amount of difference between the surrounding environment model based on the acquired distance information and the previously acquired surrounding environment model at the current position (current self-estimated position) of the remote control robot 10, and calculates the difference amount. is greater than or equal to a predetermined amount, the surrounding environment model is updated. This makes it easier for the remote control robot 10 to adapt to changes in the surrounding environment. In addition, the remote control robot 10 suppresses updating of the surrounding environment model when the difference amount is less than a predetermined amount, thereby suppressing updating when there is no significant change in the surrounding environment, thereby reducing the processing load. The environment sensor 16 is installed, for example, on a pedestal 21 (see FIG. 2).

The remote camera 17 takes an image of the surroundings of the remote control robot 10 to obtain an image. The angle of view of the remote camera 17 may be wide or narrow. An image captured by the remote camera 17 includes, for example, an object 100 as a subject. The remote camera 17 corresponds to the eyes of the remote control robot 10 and is rotatably supported by the platform mechanism 18 (see FIG. 2).

In addition, the processor 11 performs image recognition at a predetermined timing (for example, all the time) on the image captured by the remote camera 17 to recognize an object in the image. By sufficiently shortening the predetermined timing interval, the processor 11 can continue to recognize the target object 100 and other objects to be manipulated by the remote control robot 10 . Therefore, the remote control robot 10 will be described below assuming that the object 100 and other objects are always recognized.

The camera platform mechanism 18 allows the orientation of the remote camera 17 with respect to the body section 22 of the remote control robot 10 to be adjustable, and connects the body section 22 and the remote camera 17 . Under the control of the processor 11 , the camera platform mechanism 18 receives power from the power supply 12 and drives to adjust the orientation of the remote camera 17 with respect to the body 22 of the remote control robot 10 . The camera platform mechanism 18 is rotatable about two axes, the pitch axis and the yaw axis, among the roll axis, the pitch axis, and the yaw axis, for example. note that. The camera platform mechanism 18 may be rotatable about three axes, a roll axis, a pitch axis, and a yaw axis.

The arm mechanism 19 includes mechanisms from the proximal end connected to the body portion 22 to the distal end corresponding to the hand portion 19h. The arm mechanism 19 connects one or more (two in FIG. 2) arms 19a, hands 19h, arms 19a to each other, arms 19a and hands 19h, or fingers within hands 19h. and a joint. The arm mechanism 19 is driven by receiving power from the power supply 12 under the control of the processor 11 to perform arbitrary operations on the object 100 . The hand part 19h can, for example, operate to grasp the target object 100 existing at the remote site LN1, release the grasped object, or move the grasped object. The arm mechanism 19 of the remote controlled robot 10 has, for example, 6 or 7 degrees of freedom. As the number of joints increases, the arm mechanism 19 can move more smoothly. The arm mechanism 19 corresponds to the arm and hand of the remote control robot 10 . It should be noted that the movement of the hand portion 19h is described as changing the position of the hand portion 19h without driving the moving mechanism 20. FIG.

The moving mechanism 20 receives power from the power source 12 and drives according to the control of the processor 11 to arbitrarily move the remote control robot 10 . The moving mechanism 20 is configured by, for example, a mecanum wheel, an omni wheel, or wheels. When the moving mechanism 20 is a mecanum wheel, the remote control robot 10 can move in all directions and can turn around the seven axes of the mecanum wheel. When the movement mechanism 20 is an omniwheel, the remote control robot 10 can move in all directions and can turn around the central axis of the omniwheel. When the moving mechanism 20 is a wheel, the remote control robot 10 can move in the front-rear direction (the y1-axis direction in FIG. 2) and can turn around the central axis of the wheel.

For example, when the movement mechanism 20 is a four-wheel omni wheel, the processor 11 moves in the front-rear direction, moves in the left-right direction (the x1-axis direction in FIG. 2), and rotates clockwise or counterclockwise along the x1y1 plane. command the forward and reverse speed of each omniwheel. The moving mechanism 20 corresponds to the legs of the remote control robot 10 .

As shown in FIG. 1, the operation device 30 includes a processor 31, a power source 32, a memory 33, a communication device 34, a speaker 35, an environment sensor 36, a display device 37, an input sensor 38, a deformation mechanism 39, and a moving mechanism 40 .

The processor 31 may include an MPU, CPU, DSP, GPU, or the like. The processor 11 may be composed of various integrated circuits (eg, LSI, FPGA). The processor 31 implements various functions by executing programs held in the memory 33 . The processor 31 comprehensively controls each part of the operation device 30 and performs various processes.

The power source 32 is composed of a rechargeable secondary battery or the like. The power supply 32 can monitor the remaining battery level and charge control by the processor 31 and supplies necessary power to each part of the operation device 30 .

Memory 33 includes primary storage (eg, RAM or ROM). The memory 33 may include a secondary storage device (eg, HDD or SSD) or a tertiary storage device (eg, optical disc or SD card). Also, the memory 33 may be an external storage medium. The memory 33 stores various data, information, programs, and the like.

The communication device 34 communicates various data or information. The communication device 34 communicates according to a wired or wireless communication method. The communication method may be WAN, LAN, or cellular communication for mobile phones (eg LTE, 5G), short-range communication (eg infrared communication or Bluetooth® communication), power line communication, or the like.

The speaker 35 outputs sounds related to various sounds. The speaker 35 outputs ambient sound at the remote site LN1 acquired from the remote control robot 10 via the communication device 34, for example. This allows the operator 200 to audibly grasp the situation of the remote site LN1. Moreover, the speaker 35 may output an alert sound at the time of an alert, which will be described later.

The environment sensor 36 is a sensor that detects arbitrary information in the environment of the operation site LN2 where the operation device 30 is arranged. The environment sensor 36 may be a depth camera, a stereo camera, or a 3D LiDAR capable of acquiring distance information in the surrounding environment of the manipulator 30 .

Based on the information detected by the environment sensor 36 , the processor 31 generates a three-dimensional model of the operation site LN 2 (a model of the environment around the operator 30 ) (also referred to simply as a model of the surrounding environment), and stores the model in the memory 33 . You can The generated peripheral environment model of the operating device 30 can be used to avoid obstacles when the operating device 30 moves. The surrounding environment model of the operation device 30 may be generated while the operation device 30 moves and changes the current position sequentially. For example, the processor 31 simultaneously performs self-position estimation and generation of an environment map (map inside the operation site LN2) according to the SLAM algorithm. Thereby, the processor 31 generates an environment map even in an unknown environment.

In addition, the processor 31 may identify the position and orientation of the operator 200 (an example of a person) located around the operating device 30 based on the peripheral environment model of the operating device 30 . In this case, the processor 31 uses a posture estimation model, which is also expressed as “human body detection,” to detect previously learned joint points of the operator 200 and connect the joint points, thereby estimating the posture of the operator 200 . may be detected. The posture of the operator 200 here includes arbitrary gestures made by the operator 200 . The gestures here may include gripping (grabbing), releasing (releasing) the object 100, moving, and other gestures. Information on the joint points of the operator 200 learned in advance may be held in the memory 13 .

Further, the processor 31 may perform control to maintain the position of the operator 200 relative to the self-estimated position (the position of the operation device 30) within a predetermined distance based on the input detected by the input sensor 38B, which will be described later. As a result, the operation device 30 can perform “automatic follow-up control” for controlling the operation device 30 to follow the movement of the operator 200 .

The display device 37 is a liquid crystal display, an organic EL (Electro Luminescence) display, a rear projection display, or another display device. The display device 37 may be a touch panel that also serves as an operation device. The display device 37 can display various data or information (for example, alert information). Further, the display device 37 may be a projector that projects an image to be displayed on the rear side of the operation device 30 (the positive side in the Y1 axis direction). In this case, the display device 37 may project an image onto a wall surface, a floor surface, or the like on the back side of the display device 37 . This embodiment mainly exemplifies that the display device 37 is a touch panel. In addition, in the basic posture of the operation device 30, the display surface of the display device 37 is arranged along the xz plane, that is, arranged along the vertical direction (see FIG. 3).

A plurality of types of the input sensor 38 are prepared according to different uses, and include an input sensor 38A and an input sensor 38B. The input sensor 38A is installed on the outer frame of the display device 37 (outside the frame at the peripheral edge) (see FIG. 3). The input sensor 38A is, for example, a strain sensor (for example, a metal strain sensor) or a pressure sensor. The input sensor 38A detects the force applied by the operator to the manipulator 30, and detects, for example, the magnitude, direction, or other information regarding this force. The detection result by the input sensor 38A is used, for example, for processing related to movement or change in attitude of the remote control robot 10. FIG.

The input sensor 38B is installed on the inner frame of the display device 37 (inside the frame at the peripheral edge) (see FIG. 3). Input sensor 38B includes, for example, a camera or an infrared sensor. The input sensor 38B scans three-dimensional hand movements or finger poses on the front surface of the display device 37 . Input sensor 38B may cooperate with processor 31 to operate as a gesture sensor. In this case, like LeapMotion (registered trademark), a stereo camera as the input sensor 38B captures the reflected light of the light emitted from an infrared LED (Light Emitting Diode) reflected by the hand 200h of the operator 200, and the processor 31 detects poses (gestures) based on hand positions, postures, or a combination thereof.

The deformation mechanism 39 deforms the form of the operation device 30 . For example, the deformation mechanism 39 changes the positional relationship (attitude) of the display device 37 with respect to the pedestal 52 of the operation device 30 . The pedestal 52 is arranged along the horizontal direction. The deformation mechanism 39 changes the angle (orientation) of the display device 37 with respect to a predetermined direction (eg, vertical direction). In the present embodiment, the mode in which the display device 37 is arranged in a direction perpendicular to the horizontal direction (vertical direction) is also referred to as “vertical placement”. Moreover, the configuration in which the display device 37 is arranged horizontally is also referred to as “horizontal placement”.

The moving mechanism 40 receives power from the power supply 12 and drives according to the control of the processor 11 to arbitrarily move the operating device 30 . The moving mechanism 40 is configured by, for example, a mecanum wheel, an omni wheel, or wheels. When the moving mechanism 40 is a mecanum wheel, the operating device 30 can move in all directions and can be turned around the seven axes of the mecanum wheel. When the moving mechanism 40 is an omniwheel, the operating device 30 can move in all directions and can turn around the central axis of the omniwheel. When the moving mechanism 40 is a wheel, the operation device 30 can move in the front-rear direction (the y2-axis direction in FIG. 3) and can turn around the central axis of the wheel.

For example, when the movement mechanism 40 is a four-wheeled omni wheel, the processor 31 moves in the front-rear direction, moves in the left-right direction (the x2-axis direction in FIG. 3), and rotates clockwise or counterclockwise along the x2y2 plane. command the forward and reverse speed of each omniwheel.

The moving mechanism 40 is driven by the control of the processor 31 and automatically moves the operating device 30 , triggered by an operation in which the operating device 30 is pushed or pulled by the operator 200 . Further, the moving mechanism 40 may be manually movable by operating the operating device 30 as described above and rotating wheels or the like.

Next, detection of an input operation based on the input of the input sensor 38A will be described.
FIG. 4 is a diagram for explaining an example of detection of an input operation based on the input of the input sensor 38A.

The operation device 30 includes an operation body portion 51 , a base 52 and a support column 53 . The operation main body portion 51 is a part that receives an operation by the operator 200 and includes the display device 37 . A moving mechanism 40 is attached to the base 52 . The support column 53 connects the operation main body portion 51 and the pedestal 52 . One or a plurality of support columns 53 may be provided. When a plurality of support columns 53 are provided, the support columns 53 can receive the load of the operation main body 51 in a distributed manner, thereby improving the mechanical strength of the operation device 30 . In FIG. 4, two support columns 53A and 53B are provided as the support column 53. As shown in FIG. In addition, illustration of the support column 53 in the operation device 30 is omitted except in FIG. 4 .

Each support column 53 may be made of a metal material that is elastically deformed slightly, such as aluminum. In FIG. 4, on each of the two support columns 53, a strain gauge as the input sensor 38A is adhered to at least two surfaces, ie, the front or rear surface and the right or left side.

While the operator 200 grips a portion of the operation main body 51 (for example, at least one of the right end and the left end), the operator 200 applies a force (operating force ) is applied (pressed), a compressive force or a tensile force is applied to the support column 53 according to the magnitude (operating amount) and direction (operating direction) of the operating force. The strain gauge adhered to the support column 53 expands and contracts according to this compressive force or tensile force, thereby increasing or decreasing the resistance value of the strain gauge. In this case, the current flowing through the strain gauge increases or decreases the voltage flowing through the resistor according to the change in resistance of the strain gauge. The processor 31 can detect the operating force of the operator 200 by monitoring this voltage change in the strain gauge. FIG. 4 shows an operation position of the operator's 200 left hand 200h1 on the operation device 30 and an operation position of the operator's 200 right hand 200h2 on the operation device 30. FIG.

For example, when the strain gauge detects the same forward force (an example of an input) on the support columns 53A and 53B, the processor 31 detects it as a forward operation (an example of the input operation) on the operating device 30. . The processor 31 detects the same rearward force on the support column 53A and the support column 53B by the strain gauge as a rearward operation on the operation device 30 . When the strain gauge detects the same rightward force on the support column 53A and the support column 53B, the processor 31 detects that the operation device 30 is operated in the rightward direction. When the strain gauge detects the same leftward force on the support column 53A and the support column 53B, the processor 31 detects that the operation device 30 is operated in the left direction.

Further, for example, when the strain gauge detects a force in the forward direction on the support column 53A and a force in the rearward direction on the support column 53B, the processor 31 detects left turning (counterclockwise) operation on the operation device 30. To detect. When the strain gauge detects a rearward force on the support column 53A and a forward force on the support column 53B, the processor 31 detects that the operation device 30 is turned to the right (clockwise). Processor 31 detects an operation to move operating device 30 based on each of these input operations.

In this manner, the processor 31 detects the magnitude and direction of the operating force (forward/backward/left/right movement and left/right turning) by the operator 200 on the operating device 30 . Then, the processor 31 causes the remote control robot 10 to move at a moving speed (including turning speed) and moving direction (including turning direction) according to the intention of the operator 200 based on the magnitude and direction of the operation force. A movement instruction signal instructing the remote control robot 10 to move is output via the communication device 34 so as to move. That is, the processor 31 detects the moving speed and moving direction of the operator 30 and outputs a movement instruction signal for moving the remote control robot 10 in the corresponding speed and direction.

Note that such an input operation is an example, and other input operations may be performed. For example, in the above description, the operation device 30 is operated with both left and right hands of the operator 200, but the operation device 30 can be operated with either the left or right hand of the operator 200 to instruct the remote control robot 10 to move or the like. You may

Also, the processor 31 may detect the operation of the operator 200 from the movement of the moving mechanism 40 or the like. In this case, only when it is detected that the operator 200 is in contact with the operation device 30, the movement instruction may be issued based on the movement of the movement mechanism 40 or the like. Thereby, the operating device 30 can prevent an erroneous instruction. Further, when it is not detected that the operator 200 is in contact with a predetermined portion of the operation device 30 (for example, the outer frame of the display device 37), regardless of whether or not the operation device 30 is moved, the operation is performed. The processor 31 may suppress (for example, lock) the driving of the moving mechanism 40 so that the movement of the device 30 is restricted. In addition, as the contents of the restriction on the movement of the moving mechanism 40, for example, the processor 31 may lock the moving mechanism 40 so that it cannot be moved, or apply a brake so as to apply a load to the movement of the moving mechanism 40. Conceivable. As a result, the operator 30 can prevent the operator 200 from accidentally moving the operator 30 and the remote-controlled robot 10, so that the remote-controlled robot 10 can be operated safely. Further, when the operator 200 is not in contact with a predetermined portion of the operation device 30, the processor 31 makes the operation device 30 movable, but regardless of whether the operation device 30 is moved, the movement instruction signal may be controlled so as not to output This makes it possible to move only the controller 30 without moving the remote control robot 10, for example, when the controller 30 is located at a position that is difficult for the operator 200 to operate.

Next, monitoring of the remote site LN1 by the remote control system 5 will be described.

In the remote control system 5, the remote control robot 10 acquires various information on the remote site LN1. Specifically, the remote camera 17 captures the image of the imaging target. This image includes, for example, the object 100 . Also, the microphone 15 picks up sounds (environmental sounds) around the remote control robot 10 to obtain sound information of the environmental sounds. Communication device 14 transmits the resulting image and sound information to manipulator 30 . The operator 30 reproduces the state of the remote site LN1 at the operation site LN2. Specifically, communication device 34 receives image and sound information from remote controlled robot 10 . A display device 37 displays the received image. The speaker 35 outputs (sounds) the environmental sound of the remote site LN1 based on the received sound information.

As a result, the operator 200 sees the image displayed on the display surface of the display device 37 of the operation device 30 with the naked eye, and the image is captured by the remote camera 17 of the remote control robot 10 as if looking through a window. It is possible to monitor (monitor) the remote site LN1 reflected in the captured image. In addition, the operator 200 can also confirm the environmental sound of the remote site LN1.

In the remote control system 5, the processor 31 of the operation device 30 detects the distance d1 (not shown) between the operation device 30 and the operator 200 by a distance measuring sensor or the like as the input sensor 38A, and performs remote control based on this distance d1. Control of the imaging range by the remote camera 17 of the robot 10 may be instructed. In this case, for example, the communication device 34 of the operator 30 transmits distance information indicating the distance d1 between the operator 30 and the operator 200 to the remote control robot 10, and the communication device 14 of the remote control robot 10 transmits this distance You may receive information. Processor 11 may then determine the imaging range of remote camera 17 based on the distance information. For example, as the distance between the operator 30 and the operator 200 increases, the angle of view of the remote camera 17 may be narrowed to narrow the imaging range, and as the distance decreases, the angle of view may be widened to widen the imaging range. By doing so, the image of the remote location displayed on the display device 37 changes in the same manner as when the display device 37 is a real window, so that the operation device 30 gives the operator 200 a sense of realism. It can provide an operating environment. Alternatively, the processor 31 may calculate the imaging range based on the distance, transmit the imaging range information to the remote control robot 10, and the communication device 14 may receive the imaging range information. Processor 11 may then determine the imaging range of remote camera 17 to be this imaging range.

In controlling the imaging range, the processor 11 may drive the moving mechanism 20 and control the distance d2 (not shown) between the object 100 and the remote control robot 10 at the remote site LN1. In this case, the processor 11 shortens the distance d2 between the object 100 and the remote control robot 10 as the distance d1 between the manipulator 30 and the operator 200 becomes shorter. The longer the distance, the longer the distance d2 between the object 100 and the remote control robot 10 may be.

In addition, the processor 11 may control the zoom magnification of the remote camera 17 in controlling the imaging range. In this case, the processor 11 increases the zoom magnification of the remote camera 17 as the distance d1 between the operation device 30 and the operator 200 becomes shorter, and increases the zoom magnification of the remote camera 17 as the distance d1 between the operation device 30 and the operator 200 becomes longer. may be reduced.

Thereby, the operation device 30 can display the entire image of the target object 100 when the operator 200 leaves the operation device 30 . Therefore, the operator 200 can look down on the object 100 and check the whole. Further, the operation device 30 can display the details of the target object 100 when the operator 200 approaches the operation device 30 . Therefore, the operator 200 can confirm the details while approaching the object 100 . Further, the operation device 30 does not require manual operation of the remote camera 17 of the remote control robot 10 .

In the remote control system 5, the remote control robot 10 may have an optional camera (not shown) in addition to the input sensor 38B. An optional camera may be installed on the pedestal 21 of the remote control robot 10 . The optional camera captures an image including the floor around the remote control robot 10 at the remote site LN1. The imaging direction of the optional camera may be the same as or different from the imaging direction of the camera as the input sensor 38B. Also, the operation device 30 may include a projector 41 (see FIG. 9) that projects an image. The projector 41 may be installed on the pedestal 52 .

For example, the communication device 14 transmits an image including the captured floor surface to the operation device 30 . Communication device 34 receives this image from remote controlled robot 10 . The projector 41 may project this image onto the floor around the operation device 30 .

As a result, the operator 30 can reproduce the state of the remote site LN1 at the operation site LN2 including not only the imaging range of the camera as the input sensor 38A but also the floor outside the imaging range of this camera. For example, the controller 30 can additionally project an image of the feet of the remote-controlled robot 10, or can project the movement of the remote-controlled robot 10 as an animation. Therefore, the remote control system 5 can provide the operator 200 with a feeling as if the operator 200 were at the remote site LN1 and a feeling as if the operator 200 were moving within the remote site LN1, thereby improving the sense of reality. .

In the remote control system 5 , the imaging range of the remote camera 17 may be wider than the image range of the image displayed by the display device 37 . That is, a part of the imaging range of the remote camera 17 may be used as the image range (display range) displayed on the display device 37 . The processor 11 may always perform image recognition processing on images captured in this imaging range. Therefore, the remote control robot 10 can recognize what kind of object exists outside the display range of the display device 37 and within the imaging range of the remote camera 17 . Further, even if this object is a moving object, the remote control robot 10 can always track it by image recognition.

FIG. 5 is a diagram showing a display example of information on an object existing outside the display range and within the imaging range. For example, the processor 31 may cause the display device 37 to display a balloon object BO1 when a tool used for assembling a predetermined part is detected outside the display range and within the imaging range. In this case, the processor 31 may instruct the detection direction of the tool in the vicinity of the frame (peripheral end) of the display device 37 to display the balloon object BO1 superimposed on the image captured by the remote camera 17. . The balloon object BO1 describes, for example, tool identification information (for example, tool name), and guides the operator 200 to the presence of the tool and the direction in which the tool is present with respect to the remote control robot 10 as a base point. Accordingly, the operator 200 can make the remote control robot 10 grasp the tool by confirming the guide information and making a gesture of grasping the tool in consideration of the direction in which the tool is located.

In the remote control system 5, if the display device 37 is a touch panel, the processor 31 may detect a swipe operation via the touch panel. The processor 31 drives the platform mechanism 18 of the remote control robot 10 based on the operation amount of the swipe operation (the amount of movement of the finger of the operator 200 on the touch panel), the operation speed (swipe speed), and the operation direction. you can In this case, the processor 31 may generate a camera platform instruction signal for instructing control of the camera platform mechanism 18 based on, for example, the operation amount, the operation speed, and the operation direction of the swipe operation. The communication device 34 then transmits a camera platform instruction signal to the remote control robot 10 . In the remote control robot 10, the communication device 14 receives the camera platform instruction signal, the processor 11 drives the camera platform mechanism 18 based on the camera platform instruction signal, and controls the rotation amount and rotation speed of the camera platform mechanism 18. , and the direction of rotation may be determined, and the orientation of the remote camera 17 may be changed by rotating the camera platform mechanism 18 .

FIG. 6 is a diagram showing an example of an indication of the direction of the remote camera 17 based on a swipe operation on the operation device 30. As shown in FIG. In FIG. 6, the operation device 30 detects a rightward swipe operation via the touch panel. The remote control robot 10 drives the pan head mechanism 18 according to this swipe operation, rotates the direction of the remote camera 17 clockwise by 90 degrees with respect to the body portion 22 , and changes the imaging direction of the remote camera 17 .

Accordingly, the operator 200 can change the orientation of the remote camera 17 connected to the platform mechanism 18 to the direction the operator 200 wants to see by performing a swipe operation in the direction the operator 200 wants to see. The person 200 can check the state of the remote site LN1 in the direction he/she wants to see.

Next, movement of the remote control robot 10 will be described.

When the operator 200 presses the outer frame of the display device 37 of the controller 30 with a hand 200h (for example, both hands or one hand) to move the controller 30 or change the posture of the controller 30, the remote control robot 10 follows the operation device 30 to move or change its posture.

As described above, the outer frame of the display device 37 of the operation device 30 is provided with the input sensor 38B such as a metal strain sensor. This input sensor 38B can detect the magnitude and direction of the force (operating force) applied by the operator 200 . Based on the magnitude and direction of the detected operating force, the processor 31 may instruct the remote controlled robot 10 to move by a movement amount and a movement direction corresponding to the operating force. In this case, specifically, the processor 31 generates a movement instruction signal for instructing the movement of the remote control robot 10 based on the detected operating force (magnitude and direction of the operating force), and the communication device 34 A movement instruction signal may be sent to the remote controlled robot 10 . Then, the communication device 14 may receive the movement instruction signal from the operation device 30 and drive the movement mechanism 40 based on the movement instruction signal.

As a result, since the remote control robot 10 moves along the direction (the direction of the operation force) pushed by the operator 200, the viewpoint and field of view of the remote control robot 10 (that is, the position and imaging range of the remote camera 17) are changed. can be changed.

Further, the processor 31 may instruct the remote control robot 10 to move in the direction based on the swipe operation based on the swipe operation on the touch panel as the display device 37 . In this case, specifically, the processor 31 may generate a movement instruction signal for moving the remote controlled robot 10 based on the swiping operation, and the communication device 34 may transmit this movement instruction signal to the remote controlled robot 10. . The communication device 14 may then receive the movement instruction signal from the operator 30, and the processor 11 may drive the movement mechanism 40 based on the movement instruction signal. In this case, similarly to the camera platform instruction signal, the processor 31 may generate the movement instruction signal based on the operation amount, operation speed, and operation direction of the swipe operation. Then, the processor 11 may determine the movement amount, movement speed, and movement direction of the remote control robot 10 based on the operation amount, operation speed, and operation direction of the swipe operation, and move the remote control robot 10 .

FIG. 7 is a diagram showing an example of an instruction for movement of the remote control robot 10 based on a swipe operation on the operation device 30. As shown in FIG. In FIG. 7, the operation device 30 detects a rightward swipe operation via the touch panel. The remote control robot 10 drives the movement mechanism 20 according to this swipe operation and moves rightward.

As a result, the remote control robot 10 moves along the swipe direction, so the viewpoint and field of view of the remote control robot 10 (that is, the position and imaging range of the remote camera 17) can be changed.

Also, the remote site LN1 and the operation site LN2 may differ in size (scale) (vertical and horizontal height) of the work space. The processor 31 calculates the ratio of the scale of the work space of the remote site LN1 to the scale of the work space of the operation site LN2 (also referred to simply as the scale ratio) and the amount of movement of the operator 30 at the operation site LN2 (the amount of movement of the operator 30 The amount of movement of the remote control robot 10 at the remote site LN1 is calculated based on the operation force corresponding to ), and the remote control robot 10 may be commanded to move by this amount of movement. Note that the scale ratio information may be stored in the memory 33 in advance, or the scale ratio information may be input via a touch panel, for example. Note that this scale ratio does not necessarily have to be the same ratio in all directions. For example, when the front-rear direction width of the remote site LN1 is wider than the left-right direction width, there is little room for moving the operation device 30 in the left-right direction at the remote site LN1. In this case, the processor 31 may set the scale ratio such that a small movement translates into a large movement at the operating site in the left-to-right direction rather than in the front-to-rear direction. Further, when the operator 200 directly sets the scale ratio via a touch panel or the like, the scale ratio does not necessarily have to be a value corresponding to the ratio between the remote site LN1 and the operation site LN2. That is, this scale ratio indicates the ratio of the amount of movement between the remote site LN1 and the operation site LN2, and whether or not the ratio matches or resembles the actual ratio depends on the preference of the operator 200. may be changed by

For example, when the scale ratio is 1 or more, the remote site LN1 is wider than the operation site LN2, so the processor 31 determines Adjustments may be made to increase the amount of movement of the remote control robot 10 . Therefore, the processor 31 may set the movement amount command value to the remote control robot 10 larger than when the scale ratio is not considered.

For example, when the scale ratio is less than 1, the remote site LN1 is narrower than the operation site LN2. Adjustments may be made so that the amount of movement of the remote control robot 10 is reduced. Therefore, the processor 31 may set the movement amount command value to the remote control robot 10 smaller than when the scale ratio is not considered.

In consideration of safety when the remote control robot 10 moves, the upper limit of the movement speed of the movement mechanism 20 of the remote control robot 10 is set to a predetermined value lower than the maximum speed at which the movement mechanism 20 can move. may be Therefore, the processor 31 may limit the upper limit of the movement speed of the operating device 30 based on the upper limit of the movement speed of the remote control robot 10 . As a result, the moving speed of the operating device 30 is set so as not to exceed the upper limit value, so that the remote control system 5 can improve the safety when the operating device 30 is moved. Further, the operator 200 can recognize that the upper limit of the movement speed is low by actually visually confirming the movement of the operation device 30, and the movement speed of the remote control robot 10 is limited. can recognize In this case, the operating device 30 may brake the moving mechanism 40 to convey to the operator 200 that the speed is limited as a sense of resistance. For example, the processor 31 may control the movement mechanism 40 to suppress the movement of the manipulator 30 when the movement speed of the remote control robot 10 corresponding to the movement speed of the manipulator 30 reaches the upper limit.

In addition, when the movement speed of the remote-controlled robot 10 is restricted for safety, the operator 200 can check the image from the remote-controlled robot 10 on the display device 37 so that the remote-controlled robot 10 moves slowly. can feel. On the other hand, the processor 31 of the operation device 30 calculates the expected arrival time to the target position based on the operation by the operator 200, and causes the display device 37 to display information about this time (for example, the remaining travel time). good too. This allows the operator 200 to know the waiting time until the movement of the remote control robot 10 to the target position is completed, and the operator 200 performs another operation on the operator 30 during that time. Thus, it is possible to prevent the remote control robot 10 from starting a wrong movement.

Further, in the remote control robot 10, when an obstacle (eg, a chair or a cardboard) is detected by the environment sensor 16 (eg, LiDAR) in the movement direction of the remote control robot 10 intended by the operator 200, the processor 11 moves. Driving of the mechanism 20 may be suppressed, or an alert regarding the obstacle may be displayed on the operation device 30 . As suppression of driving, stopping, reduction of speed, etc. can be considered. In this case, specifically, the communication device 14 transmits obstacle detection information indicating that an obstacle has been detected to the operating device 30 . The obstacle detection information may include information about obstacles (obstacle names, characteristics, etc.). Acquisition of information such as names and characteristics of obstacles can be realized by, for example, using a known object recognition technique or reading markers or the like attached to each obstacle. In operation device 30 , communication device 34 receives obstacle detection information from operation device 30 . The processor 31 causes the display device 37 to display the obstacle detection information as an alert. At least one of the suppression of driving of the moving mechanism 20 and the display of the alert may be performed. Alternatively, the alert may be an alert to the effect that the remote control robot 10 cannot be moved in the direction in which the obstacle exists.

As a result, the remote-controlled robot 10 can, for example, invalidate the movement command (movement instruction signal) of the remote-controlled robot 10 from the operator 30 to avoid colliding with an obstacle during movement. Therefore, the remote control system 5 can improve safety at the remote site LN1. Also, the operator 200 cannot directly visually confirm the remote site LN1, but can recognize that an obstacle is approaching by confirming the obstacle detection information. Also, by checking the obstacle detection information, the operator 200 can grasp the information about the obstacle, and can grasp the degree of obstruction of the obstacle at the remote site LN1.

FIG. 8 is a diagram showing an example of an alert display when an obstacle exists around the remote-controlled robot 10. As shown in FIG. In FIG. 8, the display device 37 displays obstacle detection information on the balloon object BO2. Here, as the obstacle detection information, "obstacle present" indicating that an obstacle exists, and "movable" indicating that the remote control robot 10 cannot move due to the obstacle are displayed.

Note that the remote control robot 10 also detects obstacles outside the range displayed on the display device 37 of the operation device 30 (that is, outside the space corresponding to the imaging range and image range), and transmits obstacle detection information to the operation device 30. may be sent to As a result, the remote control robot 10 can notify the operator 200 of the existence of an obstacle that cannot be observed by the operator 200, so that the operator 200 can more accurately take countermeasures against the obstacle.

Further, in the operation device 30, when an obstacle is detected by the environment sensor 36 (for example, LiDAR) in the movement direction of the operation device 30 corresponding to the movement direction of the remote control robot 10 intended by the operator 200, the processor 31 , the driving of the moving mechanism 40 may be suppressed and an alert regarding the obstacle may be displayed on the display device 37 . As suppression of driving, stopping, reduction of speed, etc. can be considered. In this case, the processor 31 causes the display device 37 to display obstacle detection information indicating that an obstacle has been detected. The obstacle detection information may include information about obstacles (obstacle names, characteristics, etc.).

As a result, the operation device 30 can, for example, invalidate a movement command (movement instruction signal) for moving the operation device 30 to avoid colliding with an obstacle due to movement. Therefore, the remote control system 5 can improve safety at the operation site LN2. In addition, the operator 200 confirms the obstacle detection information displayed on the screen while confirming the state of the remote site LN1 via the display device 37, so that the operation device 30 becomes an obstacle at the operation site LN2. You can recognize that you are getting closer. Also, by checking the obstacle detection information, the operator 200 can grasp the information about the obstacle, and can grasp the degree of hindrance of the obstacle at the operation site LN2.

Note that the processor 31 of the operation device 30 generates a movement instruction signal that instructs the movement of the operation device 30 in the same manner as the movement instruction signal that instructs the movement of the remote control robot 10 based on the operation on the operation device 30, and moves the movement instruction signal. The operator 30 may be moved based on the instruction signal. In this case, the method of determining the movement amount, movement speed, movement direction, etc. of the operation device 30 based on the input operation may be the same as in the case of the remote control robot 10 . Therefore, in the remote control system 5, the remote control robot 10 and the operation device 30 may move together. In this case, the operation to move the remote control robot 10 is also the operation to move the operating device 30 . Therefore, it can be said that the remote control robot 10 is driven by the movement mechanism 20 and moves corresponding to the movement of the operation device 30 .

Next, posture control of the remote control robot 10 based on gestures of the operator 200 will be described.

As described above, in the operation device 30, the processor 31 detects the gesture performed by the hand 200h of the operator 200 based on the input detected by the input sensor 38B. The processor 31 directs the posture of the arm mechanism 19 of the remote controlled robot 10 so that the remote controlled robot 10 assumes the posture corresponding to the detected gesture. In this case, specifically, the communication device 34 transmits to the remote control robot 10 an attitude instruction signal that instructs the attitude of the arm mechanism 19 of the remote control robot 10 . The communication device 14 receives attitude instruction information from the manipulator 30 . The processor 11 calculates each joint angle forming the arm mechanism 19 based on the posture instruction signal. If the characteristics of the arm mechanism 19 of the remote control robot 10 are known, the hand 19h can be obtained from the three-dimensional coordinates by solving inverse kinematics (IK) from the three-dimensional coordinates of the hand 19h. It is known that each joint angle of the arm mechanism 19 can be calculated to reach the position shown. In the present embodiment, this method is used to calculate each joint angle using the three-dimensional coordinates of the hand 19h of the remote control robot 10 corresponding to the hand 200h of the operator 200. FIG. The three-dimensional coordinates of the hand portion 19h may be included in the posture-indicating information, or may be calculated from other information included in the posture-indicating information (eg, the three-dimensional coordinates of the hand 200h), and the processor 11 drives the arm mechanism 19 and controls each joint of the arm mechanism 19 to have the calculated joint angle, thereby providing a posture corresponding to the gesture of the operator 200 .

As a result, the remote control robot 10 can assume the same posture as the operator's 200 gesture. Therefore, for example, when the operator 200 makes a gesture of grabbing the target object 100 displayed on the display device 37, the remote control robot 10 can take a posture of grabbing the target object 100 that actually exists at the remote site LN1. Other gestures can be similarly implemented. Note that since the positions and number of joints of the operator 200 and the positions and number of joints of the arm mechanism 19 do not completely match, the gestures of the operator 200 and the posture of the remote control robot 10 are somewhat different. It becomes the operation to do.

Further, the operating speed of the arm mechanism 19 (including the hands) of the remote control robot 10 is a predetermined value whose upper limit value is lower than the maximum operating speed of the arm mechanism 19 in consideration of safety. is sometimes set to In this case, the hand 19h of the arm mechanism 19 of the remote control robot 10 can move behind the movement of the operator's 200 hand 200h. Therefore, the processor 11 of the remote control robot 10 calculates the planned trajectory of the movement of the arm mechanism 19 of the remote control robot 10 to the target position, and causes the controller 30 to display the calculated planned trajectory of the arm mechanism 19. You can direct In this case, the communication device 14 transmits the calculated planned trajectory of the arm mechanism 19 to the operation device 30 . The communication device 34 receives the planned trajectory of the arm mechanism 19 from the manipulator 30 . The processor 11 may cause the display device 37 to display information indicating the planned trajectory of the arm mechanism 19 . When the imaging range of the remote camera 17 includes the arm mechanism 19 of the remote control robot 10 , the arm mechanism 19 is reflected in the image displayed on the display device 37 . In this case, the display device 37 may display a planned path along which the arm mechanism 19 will move in the future, superimposed on the arm mechanism 19 in the image. As a result, the operator 200 can grasp in advance the movement trajectory of the arm mechanism 19 whose movement has not been completed, and can wait for the completion of movement of the arm mechanism 19 with peace of mind.

The processor 11 may also calculate the estimated movement time required for the movement of the arm mechanism 19 to the target position, and cause the operation device 30 to display the calculated estimated movement time. In this case, the communication device 14 transmits the calculated estimated travel time to the operation device 30 . The communication device 34 receives the estimated travel time from the operation device 30 . The processor 11 may cause the display device 37 to display information indicating the estimated movement time required for movement of the arm mechanism 19 to the target position. Thereby, the operator 200 can grasp in advance how much time it takes to move the arm mechanism 19 in order to perform the posture corresponding to the gesture, and can grasp that the arm mechanism 19 is not defective.

In addition, in the remote control robot 10, when an obstacle is detected by the environment sensor 16 (for example, LiDAR) in the moving direction of the arm mechanism 19 of the remote control robot 10 in response to the gesture of the operator 200, the processor 11 The driving of the unit mechanism 19 may be suppressed, and an alert regarding the obstacle may be displayed on the operating device 30 . As suppression of driving, stopping, reduction of speed, etc. can be considered. In this case, specifically, the communication device 14 transmits obstacle detection information indicating that an obstacle has been detected to the operating device 30 . The obstacle detection information may include information about obstacles (obstacle names, characteristics, etc.). In operation device 30 , communication device 34 receives obstacle detection information from operation device 30 . The processor 31 causes the display device 37 to display the obstacle detection information as an alert. At least one of suppressing the driving of the arm mechanism 19 and displaying an alert may be performed.

If the remote control robot 10 can avoid the obstacle without changing the final position of the hand 19h by moving each joint angle of the arm mechanism 19, the remote control robot 10 automatically avoids the obstacle. You may This process can be achieved by considering the coordinates of the obstacles when solving the inverse kinematics. In this case, the processor 31 may display an alert indicating that the obstacle has been automatically avoided.

As a result, the remote control robot 10 can, for example, invalidate the posture instruction signal (posture command) from the operation device 30 to avoid colliding with an obstacle due to the movement of the arm mechanism 19 . Therefore, the remote control system 5 can improve safety at the remote site LN1. Although the operator 200 cannot directly visually confirm the remote site LN1, by confirming the obstacle detection information, at least part of the arm mechanism 19 (for example, the hand 19h) is approaching the obstacle. I can recognize that. Also, by checking the obstacle detection information, the operator 200 can grasp the information about the obstacle, and can grasp the degree of obstruction of the obstacle at the remote site LN1.
The remote control robot 10 may also detect obstacles outside the range displayed on the display device 37 of the operating device 30 and transmit obstacle detection information to the operating device 30 . As a result, the remote control robot 10 can notify the operator 200 of the existence of an obstacle that cannot be observed by the operator 200, so that the operator 200 can more accurately take countermeasures against the obstacle.

In addition, when the remote control robot 10 has one arm mechanism 19, that is, in the case of a single arm, when the operator 200 makes a gesture with one hand, the other hand becomes free and does not make a gesture. . In this case, the outer frame of the display device 37 may be grasped with the free other hand. In this case, the input sensor 38A detects the operating force (pressing force) applied to the operating device 30 by the operator's 200 other hand. The processor 31 detects an input by the input sensor 38B, that is, detects a gesture by the operator 200 only during the period during which the input is detected by the input sensor 38A, that is, during the period during which the operating force is detected. may

Thereby, the processor 31 scans gestures made by one hand only while the other hand is holding the operation device 30 (that is, while the other hand is in contact with the outer frame of the operation device 30). , and instruct the remote control robot 10 to take a posture based on the gesture. As a result, it is possible to prevent the operation of the operator 200 from being unintentionally recognized by the operating device 30 and causing the remote control robot 10 to operate. Therefore, the remote control system 5 can safely remotely control the remote control robot 10 as in the case of using a deadman's switch of an industrial robot.

In addition, the processor 31 instructs the remote control robot 10 to remotely operate the remote control robot 10 by gesture input in the vicinity of the operation device 30 depending on whether the operator 200 is holding the operation device 30 at less than two locations. It may be determined whether to instruct (drive the arm mechanism 19) or to instruct the control of movement by the moving mechanism 20. FIG. Here, when there is only one operator 200, the fact that the operator 200 is gripping the operating device 30 at two locations means that the operator 200 is holding the operating device 30 with both hands. As shown, this action corresponds to an operation to move the operating device 30 . In this case, the processor 31 instructs movement control. On the other hand, the processor 31 can estimate that there is no intention to move the operator 200 when the number of locations where the operator 200 is in contact with the operation device 30 is one or zero. In this case, the processor 31 determines that the operator 200 intends to perform remote operation by gesture input, and instructs remote operation. As a result, the operator 200 can switch between movement control of the remote control robot 10 and gesture input to the remote control robot 10 with a simple operation. When the operator 200 touches the operation device 30 at 0 locations, the processor 31 determines that the operator 200 has no intention of moving or performing remote control, and instructs movement control. It may be controlled so that neither the remote control instruction nor the remote control instruction is performed.

In addition, the processor 31 detects not only the detected gestures, but also fine movements (positions, postures, and movements) of the operator's 200 hand 200h (including fingers). 19 can be instructed to be reflected. A specific operation can be realized by the calculation of inverse kinematics (IK) as described above. For example, assume that the operator 200 makes a gesture of grasping the object 100 with the hand 200h, moves the object 100 while grasping it with the hand 200h, releases the hand 200h, and arranges the object 100 at a predetermined position. . The processor 31 cooperates with the input sensor 38B to detect a series of movements (pick-and-place movements) of the hand 200h. The processor 31 generates an attitude instruction signal based on this detection result, and transmits the attitude instruction signal to the remote control robot 10 via the communication device 34 . The remote-controlled robot 10 can follow (trace) and implement this pick-and-place operation in accordance with the attitude instruction signal from the manipulator 30 . Specifically, the processor 31 periodically acquires the coordinates of the hand 19h corresponding to the hand 200h that grips the object 100, and calculates each joint angle of the arm mechanism 19 from the acquired coordinates by inverse kinematics (IK ) is calculated and reflected to realize a series of movements corresponding to the gesture. Note that the remote control robot 10 may perform automatic control so that the movement of the arm mechanism 19 at the operation site LN2 becomes smooth.

Note that when the arm mechanism 19 operates following the movement of the hand 200 h detected by the operation device 30 , the movement of the arm mechanism 19 is included in the imaging range of the remote camera 17 . Therefore, the result of the remote control robot 10 following the movement of the operator 200 is reflected in the image displayed by the display device 37 . Therefore, the operator 200 can confirm from the display contents of the display device 37 whether or not the movement of the operator 200 has been properly followed.

Note that the processor 31 instructs control of the movement of the arm mechanism 19 of the remote control robot 10 based on the spatial movement, posture, and gesture of the hand 200h of the operator 200 on the display surface side of the display device 37. was exemplified, but it is not limited to this. For example, if the display device 37 is a touch panel, the touch panel may detect a 5-finger touch or a 10-finger touch, and the processor 31 may recognize various gestures based on the detection results of the touch panel. Even in this case, the manipulator 30 may instruct movement control of the arm mechanism 19 of the remote control robot 10 based on the recognized gesture.

As a scale ratio, a scale ratio of the amount of movement may be set between the remote site LN1 and the operation site LN2. That is, the scale ratio may be the ratio of the amount of movement in the space of the remote site LN1 and the amount of movement in the space of the operation site LN2. In this case, the gesture may also have that scale ratio applied. That is, the processor 31 may take this scale ratio into account when converting the 3D coordinates at the remote site LN1 to the 3D coordinates at the operation site LN2. As a result, the operation device 30 can reduce the sense of incongruity of the operation (for example, gesture) due to the difference in the scale ratio with respect to the amount of movement. Also, the processor 31 may be able to set the scale ratio of the gesture independently of the scale ratio of the amount of movement. For example, the processor 31 sets the gesture scale ratio so that large movements of the remote site LN1 are converted into small movements of the operation site LN2, which is useful when fine manipulations are required. Moreover, if the processor 31 sets the scale ratio of the gesture so that a small movement of the remote site LN1 is converted into a large movement of the operation site LN2, it is possible to realize a large operation exceeding the movable range of the human body in a short time.

Next, the form of one operating device 30 will be described.

The deformation mechanism 39 is capable of deforming the form of the operation device 30 . The operation device 30 can be transformed into at least a vertical configuration in which the display surface of the display device 37 is arranged along the z-axis direction and a horizontal configuration in which the display surface of the display device 37 is arranged along the xy plane. be. The deformation mechanism 39 can adjust the form of the operation device 30 so that the display surface of the display device 37 is at a predetermined angle with respect to the vertical direction between the vertical position and the horizontal position. good too.

The processor 31 releases the electromagnetic brake of the deformation mechanism 39 only when the input sensors 38A and 38B are powered off or disabled and the moving mechanism 40 is in a stopped state. can be controlled as follows. In other words, if this condition is not met, the electromagnetic brake is not released and the shape of the operating device 30 cannot be changed by the deformation mechanism 39. If this condition is met, the electromagnetic brake is released and the deformation mechanism 39 is disabled. It is possible to change the form of the operation device 30 by Therefore, the deformation mechanism 39 can be manually deformed only when this condition is satisfied. As a result, the operation device 30 can be prevented from malfunctioning, such as the operation device 30 unintentionally starting to move when the operation device 30 is deformed.

The operation device 30 may also include an input sensor 38C that detects the tilt angle (orientation) of the display surface of the display device 37 with respect to the vertical direction (see FIG. 1). The operating device 30 indicates that it is in a vertical orientation when the tilt angle is 0 degrees, and indicates that it is in a horizontal orientation when the tilt angle is 90 degrees. Input sensor 38C may be, for example, a proximity sensor or a gyro sensor. For example, at least one of the proximity sensors is installed in the joint portion of the deformation mechanism 39, and detects the state of approach of a metal bracket that constitutes the outer frame of the display device 37, thereby detecting the tilt angle. good. The proximity sensor may detect the tilt angle according to the inductive current method. It should be noted that the input sensor 38C may detect the tilt angle by other methods.

Next, a form in which a plurality of manipulators 30 cooperate will be described.

In this embodiment, a plurality of operation devices 30 can work together to take various forms. For example, the processor 31 of each operating device 30 acquires the position of the operator 200 with respect to each operating device 30 based on the information detected by the environment sensor 36, and the positional relationship between each operating device 30 and the operator 200 is The moving mechanism 40 of each operating device 30 may be driven to move so as to achieve a predetermined positional relationship. Information on the predetermined positional relationship may be held in the memory 33 of each operating device 30, for example. Thereby, each operating device 30 can be arranged by securing a proper position with respect to the operator 200 autonomously. The fact that the positional relationship between the operation device 30 and the operator 200 can be set to a predetermined positional relationship is the same even when there is one operation device 30 (single unit).

For example, at the operation site LN2, the plurality of manipulators 30 may be vertically arranged, and the plurality of manipulators 30 may be spaced apart from each other. In this case, a plurality of remote control robots 10 may also be arranged at the remote site LN1. In this case, as described above, the position of each operating device 30 may be determined by moving each operating device 30 according to the operating force of the operator 200 on each operating device 30, or the position of each operating device 30 may be determined in advance. The relationship has been determined and may be held in memory 33 . Further, the plurality of remote control robots 10 at the remote site LN1 may also be arranged with a distance corresponding to the distance between the plurality of manipulators 30 maintained. In this case, the scale ratio between the remote site LN1 and the operation site LN2 may be considered to determine the distances between the plurality of operators 30 and arrange the plurality of operators 30 .

As a result, the remote operation system 5 allows the plurality of operators 30 to be arranged at a distance, and according to the positional relationship of the plurality of operators 30 at the operation site LN2, the plurality of remote control devices at the remote site LN1. A positional relationship of the robot 10 can be defined. Therefore, the images captured at the positions at the remote site LN1 corresponding to the plurality of manipulators 30 are displayed on the display devices 37 of the plurality of manipulators 30 . Therefore, the operator 200 can confirm the state at the corresponding position of the remote site LN1 by confirming the display of the display devices 37 of the plurality of operation devices 30 .

Moreover, FIG. 9 is a schematic diagram showing an example of a form in which a plurality of operation devices 30 (30A, 30B) are linked. As shown in FIG. 9, at the operation site LN2, two operators 30 out of the plurality of operators 30 are arranged vertically, and the positional relationship between the two operators 30 is in the x2y2 plane. It may be arranged vertically. Further, at the remote site LN1, the same number of remote control robots 10 as the number of manipulators 30 may be arranged. Two remote control robots 10 out of the plurality of remote control robots 10 may be arranged in the same positional relationship as the two manipulators 30 at the operation site LN2. In this case, the two remote cameras 17 of the two remote control robots 10 capture images with the direction perpendicular to the x1y1 plane as the imaging direction.

As a result, the display devices 37 of the two operation devices 30 are arranged in directions different by 90 degrees with respect to the position of the operator 200 , and cover the field of view in front of the operator 200 . The two display devices 37 display images captured at positions on the remote site LN1 corresponding to the plurality of operation devices 30 . Therefore, the remote control system 5 can transmit the situation of the remote site LN1 to the operator 200 with a more realistic feeling.

In FIG. 9, there are shelves in the front display device 37A and the right display device 37B when viewed from the operator 200, and the object 100 is placed on each shelf. This state is the same at the remote site LN1, and the object 100 placed on the shelf exists in front of the remote-controlled robot 10 corresponding to the manipulator 30A, and the remote-controlled robot 10 corresponding to the manipulator 30B. An object 100 placed on a shelf is shown on the right side.

Further, in FIG. 9, the projectors 41 are provided on the operation devices 30A and 30B, respectively, and project images onto the floor surface from two directions. In FIG. 9, an image is projected onto the projection range AR1. As a result, the controller 30 can supplementally project the image of the feet of the remote control robot 10 as described above, thereby improving the sense of reality. Furthermore, by projecting images onto the floor surface from two directions, it is possible to suppress the occurrence of shadows due to projection. 9, illustration of the environment sensor 36 installed on the pedestal 52 is omitted.

Moreover, FIG. 10 is a schematic diagram showing an example of a form in which a plurality of operation devices 30 (30A, 30B) are linked. As shown in FIG. 10, at the operation site LN2, a plurality of operation devices 30 are arranged vertically, and the plurality of operation devices 30 are arranged in parallel in any direction on the x2y2 plane without being separated from each other. may be In this case, the display surfaces of the respective display devices 37 of the plurality of manipulators 30 may be arranged on the same plane. In other words, the form is such that one display device 37 is enlarged by a plurality of display devices 37 . A plurality of remote control robots 10 may also be arranged at the remote site LN1. The plurality of remote control robots 10 may be arranged in the same positional relationship as the plurality of manipulators 30 at the operation site LN2. In this case, the two remote cameras 17 of the two remote control robots 10 capture images with parallel directions on the x1y1 plane.

As a result, the remote control system 5 can display the work environment of the remote control robot 10 at the remote site LN1 over a wide range by expanding it like one screen on a plurality of display devices 37. FIG. Therefore, the remote control system 5 can reproduce the situation of the remote site LN1 at the operation site LN2 with high visibility.

Also, FIG. 11 is a schematic diagram showing an example of a form in which a plurality of operation devices 30 (30A, 30B) are linked. As shown in FIG. 11, at the operation site LN2, one of two operation devices 30 out of the plurality of operation devices 30 is arranged vertically and the other is arranged horizontally. 30 may be arranged adjacently. Further, at the remote site LN1, only one remote control robot 10 or a plurality of remote control robots 10 may be arranged. In this case, the display device 37A of the manipulator 30 placed vertically and the display device 37B of the manipulator 30 placed horizontally may display the same image captured by one remote camera 17. , may display different images captured by different remote cameras 17 . Also, the display device 37B may display a predetermined operation screen for performing an operation instead of the image captured by the remote camera 17 .

Thereby, the remote control system 5 uses the two controllers 30 in cooperation, and presents a three-dimensional working environment such as using the display device 37A as a shelf and using the display device 37B as a desk, for example. can. In this case, by using both the display devices 37A and 37B for operation, the remote control system 5 reproduces the situation at the remote site (remote site LN1) three-dimensionally and performs intuitive operations. can be done. In this case, by aligning the coordinate spaces of the display devices 37A and 37B, the remote control system 5 can remotely perform complicated operations such as moving an article between the shelf and the desk. Further, the remote control system 5 may use one of the display devices 37 for display confirmation and the other display device 37 for operation. For example, the operator 200 uses the display device 37A for display confirmation and uses the touch panel as the display device 37B for operation, thereby confirming the status of the shelf with the display device 37A and displaying the desk with the display device 37B. Since the above operation can be performed, an improvement in work efficiency can be expected. In addition, the remote control system 5 uses the display device 37 that is not for operation only for display, thereby preventing erroneous operations, such as a gesture performed with the intention of operating on the desk being reflected in the operation of the shelf. can be done.

Thus, a plurality of remote control robots 10 may be provided. Each of the plurality of manipulators 30 and each of the plurality of remote control robots 10 may be arranged in corresponding positions and orientations. Corresponding position and orientation information may be held in at least one of the memories 13 , 33 .

As a result, even when there are a plurality of manipulators 30 and remote control robots 10, the operator 200 can confirm the positional relationships and orientations of the plurality of remote control robots 10 by checking the positional relationships and orientations of the plurality of manipulators 300. It can be grasped intuitively. For example, the positional relationship of each remote control robot 10 at the remote site LN1 is reflected as a similar positional relationship of each operator 30 at the operation site LN2.

FIG. 12 is a flowchart showing an operation example of the remote control system 5. As shown in FIG.

Before starting the operation of the remote control system 5, preparation processing is performed as a preparation phase. In the preliminary preparation phase, when each of the remote control robot 10 and the operation device 30 is activated, a surrounding environment model is generated autonomously. Specifically, in the remote control robot 10, the environment sensor 16 detects information (for example, distance information) regarding any environment at the remote site LN1. The processor 11 generates a peripheral environment model of the remote site LN1 based on the information detected by the environment sensor 16, and stores it in the memory 13. FIG. Similarly, in the operation device 30, the environment sensor 36 detects information (for example, distance information) regarding any environment at the operation site LN2. The processor 31 generates a surrounding environment model of the operation site LN2 based on the information detected by the environment sensor 36, and stores it in the memory 33. FIG. The generation of such a surrounding environment model may be repeated at a predetermined timing, and the surrounding environment models of the remote site LN1 and the operation site LN2 may be updated. In addition, the inclination angle of the display device 37 is adjusted in advance so that the operation device 30 has a predetermined form.

First, the processor 31 of the operating device 30 acquires joint angle information of the deformation mechanism 39 (S11). The joint angle of the deformation mechanism 39 corresponds to the tilt angle of the display device 37 . Here, the processor 31 acquires the tilt angle of the display device 37 of the operation device 30 detected by the input sensor 38C.

Processor 31 generates a coordinate transformation matrix based on the tilt angle of display device 37 (S12). This coordinate transformation matrix is used when measuring the amount of movement and orientation information. For example, the amount of movement of the remote control robot 10 or the operation device 30 based on the operating force on the operation device 30, or the amount of movement for taking a posture based on the gesture by the operator 200 is the reference form of the display device 37 (vertical orientation). form). Therefore, when the tilt of the display device 37 is different from the tilt of the reference form, the operation force on the operation device 30 and the direction of the gesture by the operator 200 are resolved into two directions according to the tilt angle. occurs. The processor 31 uses the coordinate transformation matrix to calculate the movement amount of the remote control robot 10 or the operation device 30 based on the operation force on the operation device 30 or the movement amount for performing the posture based on the gesture by the operator 200. This will eliminate this error.

The processor 31 determines whether or not the input sensor 38A has detected an operating force (an example of information) on the operation device 30 (S13).

When the input sensor 38A detects the operating force on the operating device 30, the processor 31 calculates the amount of movement of the operating device 30 based on the operating force (magnitude of the operating force) on the operating device 30 based on the coordinate transformation matrix. Calculate (S14). In this case, the processor 31 may calculate the moving direction of the operating device 30 based on the direction in which the operating force is applied to the operating device 30 . The processor 31 controls driving of the moving mechanism 40 to move the operating device 30 based on the calculated moving amount of the operating device 30 or based on the moving amount and moving direction of the operating device 30 (S15). The communication device 34 transmits a movement instruction signal including information on the calculated movement amount to the remote control robot 10 (S16). The movement instruction signal may include information on the direction of movement as well as the amount of movement.

In the remote control robot 10 , the communication device 14 receives movement instruction signals from the operator 30 . The processor 11 controls driving of the moving mechanism 20 based on the received movement instruction signal to move the remote control robot 10 (S17).

On the other hand, in step S13, if the input sensor 38A does not detect an operating force on the operation device 30 (No in S13), the processor 31 detects whether a gesture (an example of information) is detected based on the input of the input sensor 38B. It is determined whether or not (S18). A gesture is defined by the posture of at least one of the operator's hand 200h and arm. When the gesture by the operator 200 is detected, the processor 31 calculates posture information of the hand 200h and arm of the operator 200 indicating the detected gesture based on the coordinate conversion information (S19). The communication device 34 transmits an attitude instruction signal including the measured attitude information to the remote control robot 10 (S20).

In the remote controlled robot 10 , the communication device 14 receives attitude instruction signals from the manipulator 30 . Processor 11 controls the driving of arm mechanism 19 based on the received posture instruction signal (S21). In this case, the processor 11 controls the operation device 30 to take the same posture as the various gestures performed by the operator 200 . That is, the processor 11 causes the positions and postures of the hands 200h and arms of the remote control robot 10 to follow the positions and postures of the hands 200h and arms of the operator 200 .

(Modification)
FIG. 13 is a schematic diagram showing the configuration of an operating device 30A in a modified example. The operation device 30A is installed, for example, suspended from the ceiling. A moving mechanism 40A of the manipulator 30A is composed of a three-axis horizontal articulated robot arm. Specifically, the moving mechanism 40A includes a first axis J1, a second axis J2 and a third axis J3. In the moving mechanism 40A, the first axis J1, the second axis J2, and the third axis J3 are arranged in this order from the side closer to the ceiling. Note that the operating device 30A has the same configuration as the operating device 30 except for the configuration of the moving mechanism 40A.

As with the operating device 30, when the operating device 30A receives an operation based on physical contact such as being pushed or pulled by the operator 200, the rotation angle of each axis of the operating device 30A changes. The operation device 30A detects the rotation angles of each of the first axis J1, the second axis J2 and the third axis J3, for example, using an angle sensor (not shown). This rotation angle is an example of an input detected by an input sensor. The processor 31 instructs movement control of the remote control robot 10 based on the detected rotation angle of each axis. It should be noted that the instruction method for controlling the movement of the remote-controlled robot 10 differs depending on the mode of the remote-controlled robot 10 (eg, ceiling-mounted type, ground-mounted type).

For example, the remote control robot 10 is a remote control robot 10A that is installed, for example, suspended from the ceiling like the operation device 30A, and the movement mechanism 20A of the remote control robot 10A is a three-axis horizontal articulated robot arm. Assume it is configured. Further, similarly to the operation device 30A, the moving mechanism 20A includes a first axis J11, a second axis J12 and a third axis J13, and from the side near the ceiling, the first axis J11, the second axis J12 and the third axis Assume that they are arranged in the order of J13.

In this case, the communication device 34 of the manipulator 30A transmits an angle instruction signal including information on the detected rotation angle of each axis to the remote control robot 10A. In remote controlled robot 10A, communication device 14 receives the angle indicating signal. Processor 11 controls driving of moving mechanism 20A based on the received angle instruction signal. That is, the processor 11 controls the first axis J1, the second axis J2 and the third axis J3 of the movement mechanism 40A of the controller 30A and the first axis J11, the second axis J12 and the third axis J11 of the movement mechanism 20A of the remote control robot 10A. By setting the three axes J13 and J13 to the same rotation angle, the remote control robot 10A and the operation device 30A can be moved in the same manner.

Also, for example, let us assume that the remote control robot 10 is a remote control robot 10 that moves on the ground using a moving mechanism 20 as in the above embodiment. In this case, the processor 31 of the manipulator 30A, for example, solves forward kinematics (FK) based on the detected rotation angles of the respective axes to determine the post-movement position P ( Px, Py, Pz) are calculated. Here, x, y, and z represent components in three orthogonal directions (x component, y component, and z component) in a three-dimensional space. The communication device 34 transmits to the remote controlled robot 10 a position indication signal including the calculated position information of the remote controlled robot 10 after movement. In remote controlled robot 10, communication device 14 receives position indication signals. Processor 11 controls driving of moving mechanism 20 based on the received position indication signal. As a result, the remote control robot 10 can reach the post-movement position of the movement mechanism 20 calculated by the operation device 30A.

Further, the controller 30A may instruct the remote-controlled robot 10 not to provide the position information after the remote-controlled robot 10 has moved, but to provide the speed information of the remote-controlled robot 10 to the remote-controlled robot 10 . In this case, the processor 31 sequentially calculates the position of the remote-controlled robot 10 after movement, calculates the difference between the calculated positions, and based on this difference, the speed V (Vx, Vy, Vz) of the remote-controlled robot 10 ) are calculated sequentially. The communication device 34 sequentially transmits to the remote-controlled robot 10 speed instruction signals including information on the calculated speed V of the remote-controlled robot 10 . In the remote controlled robot 10, the communication device 14 sequentially receives speed indication signals. Processor 11 controls driving of moving mechanism 20 based on the received speed instruction signal. Therefore, the remote control robot 10 can reach the post-movement position of the movement mechanism 20 calculated by the operation device 30A by continuing the movement of the movement mechanism 20 at the instructed speed.

In this way, the remote control system 5 changes the rotation angles of the first axis J1, the second axis J2 and the third axis J3 of the movement mechanism 40A of the controller 30A to the post-movement position of the movement mechanism 20A of the remote control robot 10A. Alternatively, by converting to speed, the remote control robot 10A and the manipulator 30A can be moved in the same manner.

According to the remote control system 5 of the present embodiment, the operator 200 can easily move during remote control, can perform remote control for a long time, and can easily learn how to operate the operator 30. Easy. Further, the remote control system 5 can appropriately present visual information such as the environment of the remote site LN1 and the work to the operator 200 in robot remote control (telepresence). Therefore, the operator 200 can easily operate remotely. For example, even if the remote site LN1 is a factory and it is difficult for the operator 200 as a remote operator to go to the factory, the remote operation system 5 can be remotely operated from the operator 200's home or the like. Being able to remotely control the control robot 10 is very beneficial.

It should be noted that, in the present embodiment, the operating device 30 is mainly used to calculate the information (for example, the amount of movement) regarding the specific motion of the remote control robot 10 based on the input operation to the operating device 30. It is not limited to this. For example, the operation device 30 transmits information of the input operation to the operation device 30 to the remote control robot 10, and the processor 11 of the remote control robot 10 transmits the information of the remote control robot 10 based on the input operation to the operation device 30. Information about the motion may be calculated.

In this embodiment, the movement of the hand portion 19h of the remote control robot 10 is performed without driving the movement mechanism 20, but the movement of the movement mechanism 20 may be included. In this case, the direction and amount of driving the moving mechanism 20 can be calculated by regarding the moving mechanism 20 as one of the joints. As a result, for example, when operating an object 100 that cannot be reached at the current position of the remote control robot 10, the input of the movement of the remote control robot 10 itself can be omitted. In this case, the processor 31 may also drive the moving mechanism 40 of the operation device 30 in synchronization with the driving of the moving mechanism 20 of the remote control robot 10 . In this case, the operating device 30 may drive the moving mechanism 40 after confirming (recognizing) that there is no obstacle or operator 200 that may collide with the moving mechanism 40 .
In this embodiment, the remote site LN1 and the operation site LN2 are not only separated from each other (remote sites) but also physically isolated, so that the operator 200 can sense the remote site LN1 with all five senses. Although an example in which direct recognition is not possible has been described, the remote site LN1 and the operation site LN2 may not be isolated. Specifically, the remote site LN1 and the operation site LN2 may be remote sites LN1 and operation sites LN2 that are separated from each other in one large space such as a factory. In this case, the remote site LN1 may be visible from the operation site LN2. However, even in such a case, the operator 200 can recognize the remote site LN1 from the viewpoint of the remote-controlled robot 10 via the operation device 30, so that the remote-controlled robot 10 can be operated more intuitively. .

As described above, the remote control system 5 of the above-described embodiment includes the remote control robot 10 (an example of the robot device) equipped with the movement mechanism 20 (an example of the first movement mechanism), and the remote control robot 10 from the operation site LN2. and an operation device 30 for instructing control of the operation of. The operation device 30 includes a display device 37 and a moving mechanism 40 (an example of a second moving mechanism). The operation device 30 receives the first image captured by the remote control robot 10, displays the first image on the display device 37, and when an operation of moving the operation device 30 by the operator 200 is detected, An instruction signal including an instruction to move the remote controlled robot 10 is sent to the remote controlled robot 10 . The remote control robot 10 receives the instruction signal, and controls the movement mechanism 20 based on the instruction signal so that the remote control robot 10 moves in accordance with the movement of the operation device 30 .

For example, controller 30 may comprise processor 31 , communication device 34 , display device 37 and input sensor 38 . Communication device 34 may receive the first image captured by remote controlled robot 10 . Display device 37 may display the first image. The processor 31 detects an input operation to the operation device 30 by the operator 200 based on the input detected by the input sensor 38, and generates an instruction signal instructing control of the operation of the remote control robot 10 based on the input operation. You can The communication device 34 may send instruction signals to the remote controlled robot 10 . The remote controlled robot 10 may receive the instruction signal and act based on the instruction signal.

As a result, the remote control system 5 can operate the controller 30 while checking the state of the remote location. The operator 200 can confirm the display of the display device 37 having a screen larger than the area around the eyes, unlike smart glasses, HMD, or the like. In other words, the remote control system 5 can prevent the field of view of the operator 200 from being restricted. Therefore, the operator 200 can easily move, for example, at the operation site LN2 (for example, his/her own room). In addition, since the remote control system 5 can eliminate the need for the operator 200 to wear the operating device 30 on the head, the operator's burden due to the weight of the operating device 30 can be reduced. Therefore, the operator 200 can perform remote operation for a long time using the operation device 30 . Further, since the operator 200 can intuitively operate the operation device 30 using the input sensor 38, it is not necessary to learn a complicated operation method of the controller. Further, since the operation device 30 includes both the input sensor 38 and the display device 37 and can detect an input operation based on the input of the input sensor 38, it is unnecessary to prepare an operation device other than the display device. .

Further, for example, when the operator 200 moves the operating device 30 by hand, the remote control robot 10 moves to a desired viewpoint. The operator 200 sees the image displayed on the display surface of the display device 37 of the operation device 30 with the naked eye, and is reflected in the image captured by the remote control robot 10 as if looking through a window. Remote site LN1 can be monitored. Further, the operating device 30 may acquire the environmental sound of the remote site LN1 picked up by the remote control robot 10 from the remote control robot 10 and output the sound. In this case, the operator 200 can also confirm the environmental sound of the remote site LN1.

Also, the input sensor 38 may detect the posture of at least one of the hand and arm of the operator 200 in the vicinity of the operation device 30 as the input operation. In other words, the processor 31 may detect the posture of at least one of the hand and arm of the operator 200 as the input operation based on the input detected by the input sensor 38 . The processor 31 may generate an instruction signal that instructs the posture of the arm mechanism 19 included in the remote control robot 10 based on the input operation. In this case, if there are less than two contact points on the peripheral edge of the display device 37 detected by the input sensor 38, the processor 31 generates the remote control robot 10 based on the input operation in the vicinity of the operation device 30. may output an instruction signal that instructs the posture of the arm mechanism 19 provided in the . The remote control robot 10 may drive the arm mechanism 19 based on the command signal. In addition, when the input sensor 38 does not detect contact with the peripheral edge of the display device 37, the processor 31 detects whether or not the posture of at least one of the hand and the arm in the vicinity of the operation device 30 is detected. , the output of the instruction signal for instructing the posture of the arm mechanism 19 may be suppressed.

As a result, the remote control system 5 can cause the remote control robot 10 to perform actions (for example, gripping and releasing the object 100 ) corresponding to predetermined postures (gestures) of the operator 200 . Therefore, for example, when the operator 200 makes a gesture of gripping the target object 100 displayed on the display device 37 of the operation device 30, the remote control robot 10 at the remote location actually grips the target object 100. can be made In addition, by outputting an instruction signal based on detection of contact with a contact point on the peripheral edge of the display device 37 and not outputting an instruction signal when this contact is not detected, remote operation based on gestures can be performed. can be safely carried out.

Also, the posture of operator 200 may include the position of at least one of the hand and arm of operator 200 with respect to operator 30 . The posture of the remote controlled robot 10 may include the position of the arm mechanism 19 with respect to the imaging range of the remote camera 17 (an example of a camera) provided on the remote controlled robot 10 .

As a result, the remote control system 5 makes the position of at least one of the hand and arm of the operator 200 with respect to the operation device 30 one gesture, so that the remote control robot 10 can perform an action (for example, movement) corresponding to the gesture. can be implemented. Therefore, for example, when the operator 200 makes a gesture to move the object 100 displayed on the display device 37 of the operation device 30, the remote control robot 10 at the remote location actually moves the object 100. can be made

Also, the input sensor 38 may be installed at the peripheral edge (for example, the outer frame) of the display device 37 to detect input based on physical contact. The processor 31 may detect an operation for moving the remote controlled robot 10 based on an input operation based on the input, and generate an instruction signal for moving the remote controlled robot 10 . The remote-controlled robot 10 may drive the moving mechanism 20 (an example of the first moving mechanism) included in the remote-controlled robot 10 based on the instruction signal.

As a result, the remote control system 5 can instruct the remote control robot 10 to move, for example, when the operator 200 pushes or pulls the operation device 30 as a trigger.

Further, when the input sensor does not detect an input based on contact with the peripheral edge of the display device 37, regardless of whether or not the operation unit 30 is moved, an instruction signal including an instruction to move the remote control robot 10 is remotely transmitted. It does not have to be sent to the control robot 10 .

As a result, the remote control system 5 can suppress the movement of the remote control robot 10 when the operator 200 does not expressly intend to move the remote control robot 10, thereby improving safety at the remote site LN1. .

Further, when the input sensor does not detect an input based on contact with the peripheral edge of the display device 37 , the movement mechanism 40 may be controlled to suppress movement of the operation device 30 by the operator 200 .

Thereby, the remote control system 5 can suppress the movement of the operation device 30 when the operator 200 has no explicit intention to move the operation device 30, and can improve the safety at the operation site LN2.

Also, the instruction signal may include a signal for instructing control of at least one of the movement amount, movement direction, and movement speed of the remote control robot 10 . The remote control robot 10 may control at least one of the movement amount, movement direction, and movement speed of the remote control robot 10 based on the instruction signal.

As a result, the remote control system 5 can increase the amount of movement by pressing the operating device 30 strongly, and can increase the acceleration and the moving speed by temporarily pressing the operating device 30 strongly. Further, the remote control system 5 can move the remote control robot 10 in a predetermined direction by pushing or pulling the operator 30 in this direction. Therefore, the operator 200 can realize more intuitive remote control.

Further, the operation device 30 may be arranged at the operation site LN2. The remote controlled robot 10 may be located at the remote site LN1. The amount of movement of the operator 30 may be converted into the amount of movement of the remote control robot 10 based on the scale ratio indicating the ratio of the amount of movement at the remote site LN1 to the amount of movement in the space of the operation site LN2. . Also, the scale ratio may be the ratio of the space size of the operation site LN2 and the space size of the remote site LN1.

As a result, the remote control system 5 considers the amount of movement at the operation site LN2 and the amount of movement at the remote site LN1, and at the remote site LN1 based on the operation of the operator 200 on the operation device 30 at the operation site LN2. of the remote control robot 10 can be determined. Therefore, for example, when the amount of movement at the remote site LN1 is smaller than that at the operation site LN2, the remote control system 5 can prevent the movement of the remote control robot 10 from becoming excessive. On the other hand, when the amount of movement at the remote site LN1 is large, it is possible to prevent the movement of the remote control robot 10 from becoming too small.

In addition, the remote control system 5 considers the size of the space between the operation site LN2 and the remote site LN1, and the remote control robot at the remote site LN1 based on the operation of the operator 200 on the operation device 30 at the operation site LN2. Ten behaviors can be determined. Therefore, for example, when the remote site LN1 is narrow with respect to the operation site LN2, the remote control system 5 can prevent the remote control robot 10 from moving excessively. It is possible to prevent the movement of the control robot 10 from becoming too small.

Further, the operation device 30 may include a moving mechanism 40 (an example of a second moving mechanism). The processor 31 may drive the movement mechanism 40 based on the input operation to move the manipulator 30 in conjunction with movement of the remote control robot 10 .

Accordingly, in the remote control system 5, the movement of the remote control robot 10 corresponds to the movement of the operator 30 at the remote site LN1 and the operation site LN2. easier to understand.

Further, the processor 31 may acquire the upper limit value of the movement speed of the remote controlled robot 10 and determine the upper limit value of the movement speed of the operator 30 based on the upper limit value of the movement speed of the remote controlled robot 10 . Further, an upper limit may be set for the movement speed of the remote control robot 10 . When the moving speed of the remote control robot 10 corresponding to the moving speed of the manipulator 30 reaches the upper limit, the manipulator 30 may control the movement mechanism 20 to suppress the movement of the manipulator 30 .

As a result, the remote control system 5 can improve the safety of the operation device 30 during movement in accordance with the upper limit of the movement speed of the remote control robot 10, which takes safety into consideration.

The remote control robot 10 may also include an environment sensor 16 (an example of a first environment sensor) that detects the surroundings of the operation device 30 . The environment sensor 16 may detect the situation around the operation device 30 in a wider range than the space corresponding to the range of the first image. When the environmental sensor 16 detects the first obstacle, the remote control robot 10 may transmit obstacle detection information indicating that the first obstacle has been detected to the operator 300 . Communication device 34 may receive obstacle detection information from remote controlled robot 10 . The processor 31 may cause the display device 37 to display that the first obstacle has been detected.

As a result, the operator 200 can check the display on the display device 37, for example, even if there is an obstacle outside the range of the first image around the remote-controlled robot 10 while the remote-controlled robot 10 is moving. It is possible to recognize objects, to take measures such as stopping the remote control robot 10, and to improve safety at the remote site LN1.

The remote control robot 10 also includes the environment sensor 16 described above. When the environment sensor 16 detects the first obstacle, the remote control robot 10 outputs obstacle detection information indicating that the first obstacle has been detected to the operator. 30. Communication device 34 may receive obstacle detection information from remote controlled robot 10 . The processor 31 may generate an instruction signal for restraining the movement of the remote controlled robot 10 based on the reception of the obstacle detection information. The remote control robot 10 may suppress driving of the moving mechanism 20 based on the instruction signal.

As a result, the remote control system 5 can recognize, for example, that there is an obstacle in the vicinity of the remote control robot 10 while the remote control robot 10 is moving, and the remote control system 5 can automatically perform remote control without additional operations by the operator 200. Movement of the robot 10 can be stopped. Therefore, the remote control system 5 can improve safety at the remote site LN1.

Further, the operation device 30 may include an environment sensor 36 (an example of a second environment sensor) that detects the circumstances around the operation device 30 . For example, when the environment sensor 36 detects a second obstacle in the direction in which the operator 200 moves the operating device 30 , the processor 31 may suppress driving of the moving mechanism 40 .

As a result, the remote control system 5, for example, when the operator 200 is concentrating on the operation of the operation device 30 and the confirmation of the display of the display device 37, the obstacle in the direction of moving the operation device 30 in the operation site LN2 is detected. Even if the user does not notice the object, the movement of the operating device 30 can be automatically stopped. Therefore, the remote control system 5 can ensure the safety of the operator 200 around the operation device 30 .

Further, the operation device 30 may be arranged at the operation site LN2. The operation device 30 may include an environment sensor 36 (an example of a second environment sensor) that detects the circumstances around the operation device 30 . The processor 31 may move the operator 30 within a predetermined distance from the operator 200 when the environment sensor 36 detects the position of the operator 200 at the operation site LN2.

As a result, even if the operator 200 moves around the operation site LN2, the operation device 30 follows the movement of the operator 200, so that the operator 200 can easily confirm the display and perform the operation using the operation device 30. .

Also, the imaging range when one image is captured by the remote control robot 10 may be wider than the display range of the first image displayed by the display device 37 . When the processor 31 recognizes an object existing inside the imaging range and outside the display range, the processor 31 may cause the display device 37 to display information about the object superimposed on the first image.

Accordingly, even if an object (for example, a tool) useful to the operator 200 exists outside the display range that can be confirmed by the display device 37, the operator 200 can recognize its existence.

Further, the operation device 30 may include a deformation mechanism 39 that supports the display surface of the display device 37 so that the angle of the display surface of the display device 37 with respect to the vertical direction (z2 direction) can be changed. Processor 31 may generate the indication signal based on the angle of the display surface of display device 37 relative to the vertical direction.

As a result, the remote control system 5 can freely adjust the angle of the display surface of the display device 37 to realize a desired information display form. Also, movement information and posture information calculated according to the angle of the display surface can be distributed in a plurality of directions. Even in this case, the remote control system 5 generates an instruction signal according to the angle of the display surface and issues an operation instruction, so that the accuracy of the operation control of the remote control robot 10 based on the detected input operation is reduced. can be suppressed.

The display device 37 is a touch panel that also serves as an operation device, and may be arranged along the horizontal direction.

As a result, the remote control system 5 can freely adjust the angle of the display surface of the display device 37 and can be operated by the touch panel, so that a desired operation mode can be realized.

Various embodiments have been described above with reference to the drawings, but it goes without saying that the present disclosure is not limited to such examples. It is obvious that a person skilled in the art can conceive of various modifications or modifications within the scope described in the claims, and these also belong to the technical scope of the present disclosure. Understood. Also, the components in the above embodiments may be combined arbitrarily without departing from the gist of the disclosure. In particular, in the above-described embodiment, the processor 31 of the operation device 30 performs the main processing, but which device in the remote control system 5 performs which processing may be arbitrary. For example, the various instruction information may be generated by the processor 11 of the remote control robot 10 based on the information acquired and transferred by the operation device 30 instead of being generated by the processor 31 .

The execution order of each process such as actions, procedures, steps, and stages in the devices, systems, programs, and methods shown in the claims, the specification, and the drawings is , etc., and unless the output of the previous process is used in the subsequent process, it can be implemented in any order. Regarding the operation flow in the claims, the specification, and the drawings, even if the description is made using "first," "next," etc. for convenience, it means that it is essential to carry out in this order. is not.

The present disclosure provides a remote control system, a controller, and a remote control method that allow a remote operator to easily move during remote control, enable remote control for a long period of time, and easily learn how to operate the controller. etc. is useful.

5 Remote control system 10 Remote control robot 11 Processor 12 Power supply 13 Memory 14 Communication device 15 Microphone 16 Environment sensor 18 Platform mechanism 19 Arm mechanism 19a Arm 19h Hand 20 Moving mechanism 21 Pedestal 22 Body 30 Manipulator 31 Processor 32 Power supply 33 memory 34 communication device 35 speaker 36 environment sensor 37, 37A, 37B display device 38, 38A, 38B, 38C input sensor 39 deformation mechanism 40 movement mechanism 41 projector 51 operation main unit 52 pedestal 53, 53A, 53B support column 100 object 200 operator AR1 projection range

Claims (18)

A remote control system comprising: a robot device having a first movement mechanism;
The manipulator comprises a display device and a second movement mechanism,
The operating device is
receiving a first image captured by the robotic device and causing the display device to display the first image;
transmitting an instruction signal including an instruction to move the robot device to the robot device when an operator's operation to move the controller is detected;
The robotic device is
receiving the instruction signal;
controlling the first movement mechanism so that the robot device moves in accordance with the movement of the operation device, based on the instruction signal;
Remote control system. the instruction signal includes a signal for instructing control of at least one of a movement amount, a movement direction, and a movement speed of the robot device;
The robot device controls at least one of a movement amount, a movement direction, and a movement speed of the robot device based on the instruction signal.
The remote control system according to claim 1. The operator is arranged at the operation site,
The robotic device is placed at a remote site,
The amount of movement of the operator is converted into the amount of movement of the robot device based on a scale ratio indicating the ratio of the amount of movement at the remote site to the amount of movement in the space at the operation site.
The remote control system according to claim 2. 4. The remote control system according to claim 3, wherein the scale ratio is the ratio of the size of the space at the operation site and the space at the remote site. An upper limit is set for the movement speed of the robot device,
When the moving speed of the robot device corresponding to the moving speed of the manipulator reaches the upper limit, the manipulator controls the second movement mechanism to suppress the movement of the manipulator.
The remote control system according to claim 2. An input sensor is installed at the peripheral edge of the display device,
the input sensor detects input based on physical contact;
the operating device detects an operation to move the operating device based on an input operation based on the input;
The remote control system according to claim 1. When the input sensor does not detect an input based on contact with the peripheral edge of the display device, the operation device provides an instruction including an instruction to move the robot device regardless of whether or not the operation device is moved. sending no signal to said robotic device;
The remote control system according to claim 6. When the input sensor does not detect an input based on contact with the peripheral edge of the display device, the operating device controls the second movement mechanism to suppress movement of the operating device by the operator. do,
The remote control system according to claim 6. The input sensor further detects a posture of at least one of the operator's hand and arm in the vicinity of the operator as the input operation,
When the number of contact points on the peripheral edge of the display device detected by the input sensor is less than two, the manipulator is an arm included in the robot device generated based on an input operation in the vicinity of the manipulator. outputting the instruction signal for instructing the attitude of the unit mechanism;
The robot device drives the arm mechanism based on the instruction signal.
The remote control system according to claim 6. When the input sensor does not detect contact with the peripheral edge of the display device, the operating device detects whether or not the posture of at least one of the hand and the arm in the vicinity of the operating device is detected. first, suppressing the output of the instruction signal that instructs the posture of the arm mechanism;
The remote control system according to claim 9. The robotic device is
a first environment sensor that detects a situation around the robot device in a range wider than the space corresponding to the range of the first image;
when a first obstacle is detected by the first environmental sensor, transmitting obstacle detection information indicating that the first obstacle has been detected to the operating device;
When the obstacle detection information is received, the operation device causes the display device to display that the first obstacle has been detected.
The remote control system according to any one of claims 1-10. The robotic device is
a first environment sensor that detects a situation around the robot device in a range wider than the space corresponding to the range of the first image;
when a first obstacle is detected by the first environmental sensor, transmitting obstacle detection information indicating that the first obstacle has been detected to the operating device;
The operating device is
receiving the obstacle detection information from the robot device;
generating the instruction signal for suppressing the movement of the robot device based on the reception of the obstacle detection information;
The robot apparatus suppresses driving of the first movement mechanism based on the instruction signal.
The remote control system according to any one of claims 1-10. The operating device includes a second environment sensor that detects a situation around the operating device,
The operating device suppresses driving of the second moving mechanism when the second environment sensor detects a second obstacle in the direction in which the operator moves the operating device.
The remote control system according to claim 11 or 12. The operator is arranged at the operation site,
The operating device includes a second environment sensor that detects a situation around the operating device,
12. Any one of claims 1 to 11, wherein the operating device moves the operating device within a predetermined distance from the operator when the position of the operator at the operation site is detected by the second environment sensor. The remote control system described in . an imaging range when the first image is captured by the robot device is wider than a display range of the first image displayed by the display device;
When an object existing inside the imaging range and outside the display range is recognized, the operation device causes the display device to display information about the object superimposed on the first image.
The remote control system according to any one of claims 1-14. The operation device includes a deformation mechanism that supports the angle of the display surface of the display device with respect to the vertical direction so that it can be changed,
The operation device generates the instruction signal based on the angle of the display surface of the display device with respect to the vertical direction.
The remote control system according to any one of claims 1-15. An operation device for instructing control of the operation of a robot device having a first movement mechanism from an operation site,
comprising a processor, a communication device, a display device, and a second movement mechanism;
The processor
receiving a first image captured by the robotic device via the communication device;
causing the display device to display the first image;
transmitting an instruction signal including an instruction to move the robot apparatus via the communication device when an operation by an operator to move the operation device is detected;
operator. A remote operation support method for instructing control of the operation of a robot device having a first movement mechanism from an operation site,
receiving a first image captured by the robotic device;
causing a display device to display the first image;
transmitting an instruction signal including an instruction to move the robot apparatus to the robot apparatus when an operation by an operator to move the operation device including the display device is detected;
Remote control support method.

Images