WO2024089890A1

WO2024089890A1 - Remote operation system and remote operation method

Info

Publication number: WO2024089890A1
Application number: PCT/JP2022/040482
Authority: WO
Inventors: 正樹春名; 茂明田頭; 正樹荻野
Original assignee: 三菱電機株式会社
Priority date: 2022-10-28
Filing date: 2022-10-28
Publication date: 2024-05-02

Abstract

A remote operation system (100) according to the present disclosure comprises: a remote machine (1) in which a camera (2) is installed and which is remotely operated; a video presentation device (4) for presenting to an operator, as a presentation video, a video captured by the camera (2); an operation device (5) that is operated by the operator, receives, in response to an operation by the operator, an input of a target posture of the remote machine (1) as a position in the presentation video, and receives, in response to an operation by the operator, an input of a position movement instruction indicating that the position of the remote machine (1) is to be moved; and an operation calculator (3) for driving the posture of the remote machine (1) on the basis of the target posture received by the operation device (5), and moving the position of the remote machine (1) on the basis of the position movement instruction received by the operation device (5).

Description

Remote control system and remote control method

This disclosure relates to a remote control system and a remote control method.

In a remote control system that operates machinery remotely, for example, a combination of a head-mounted display and an operation interface that detects the operator's gestures is used.

Patent document 1 discloses a technology that uses a fisheye stereo camera to present a wide-field-of-view image to a remote operator without the need to drive the camera.

JP 2021-180415 A

However, the technology described in Patent Document 1 requires the operator to wear a head-mounted display and have stereoscopic perception, and the operator must operate the joystick while sequentially checking the surroundings using stereoscopic perception, which places a high operational burden on the operator.

The present disclosure has been made in consideration of the above, and aims to provide a remote control system that can reduce the operational burden on the operator.

In order to solve the above-mentioned problems and achieve the object, the remote operation system disclosed herein comprises a remote machine equipped with a camera and remotely operated, an image presentation device that presents an image captured by the camera to an operator as a presentation image, and an operation device that is operated by the operator and accepts, through the operator's operation, an input of a target attitude of the remote machine as a position in the presentation image, and accepts, through the operator's operation, an input of a position movement instruction indicating that the position of the remote machine is to be moved. The remote operation system further comprises an operation calculator that drives the attitude of the remote machine based on the target attitude received by the operation device, and moves the position of the remote machine based on the position movement instruction received by the operation device.

The remote control system disclosed herein has the effect of reducing the operational burden on the operator.

FIG. 1 is a diagram showing a configuration example of a remote control system according to a first embodiment; FIG. 1 is a schematic diagram showing a specific example of a remote control system according to a first embodiment; FIG. 1 is a diagram showing an example of a method for setting a target attitude according to the first embodiment; FIG. 1 is a diagram showing an example of an operation using the operation device according to the first embodiment; A flowchart showing an example of an operation in the operational computing unit according to the first embodiment. FIG. 1 is a diagram showing an example of the configuration of a computer system that realizes an operational calculator according to a first embodiment; FIG. 13 is a diagram showing a configuration example of a remote control system according to a second embodiment. FIG. 13 is a diagram showing an example of a screen showing an operation method according to the second embodiment; FIG. 13 is a diagram showing an example of a screen showing an operation method according to the second embodiment; FIG. 13 is a diagram showing an example of a screen showing an operation method according to the second embodiment; FIG. 13 is a diagram showing an example in which a terminal device is used as an image presentation device according to a second embodiment; FIG. 13 is a diagram showing an example of a video display method according to the second embodiment; FIG. 13 is a diagram showing a configuration example of a remote control system according to a third embodiment. FIG. 13 is a diagram showing an example of camera switching in the third embodiment; FIG. 13 is a diagram showing an example of a camera provided inside a hand of a manipulator according to a third embodiment; FIG. 13 is a diagram showing an example of a camera according to a third embodiment for photographing a manipulator from the side. FIG. 13 is a diagram showing an example of video switching in the third embodiment. FIG. 13 is a diagram showing an example of superimposing images captured from the side in the third embodiment; FIG. 13 is a diagram showing an example of a method for detecting pressure and displaying pressure according to the third embodiment; FIG. 13 is a diagram showing another example of a method for detecting pressure and displaying pressure according to the third embodiment; FIG. 13 is a diagram showing a configuration example of a remote control system according to a fourth embodiment. FIG. 13 is a diagram showing an example of a setting screen for a hand angle according to the fourth embodiment; FIG. 13 is a diagram showing an example of a setting screen for a hand angle according to the fourth embodiment; FIG. 13 is a diagram showing a configuration example of a remote control system according to a fifth embodiment.

Below, the remote control system and remote control method according to the embodiment will be described in detail with reference to the drawings.

Embodiment 1.
1 is a diagram showing a configuration example of a remote operation system according to embodiment 1. A remote operation system 100 of this embodiment includes a remote machine 1, a camera 2, an operation calculator 3, an image display device 4, and an operation device 5.

The remote machine 1 is a machine that is remotely operated, and may be, for example, but is not limited to, a combination of a cart and a manipulator, a mobile vehicle, a manipulator, a humanoid robot, etc. In the following, as an example, a case where the remote machine 1 is a combination of a cart and a manipulator will be mainly described.

The remote machine 1 includes an attitude drive mechanism 11 and a position drive mechanism 12. The attitude drive mechanism 11 is a mechanism that can change the attitude of the remote machine 1, and changes the attitude of the remote machine 1 according to a control signal received from the operation calculator 3. Here, the attitude indicates the orientation of the remote machine 1. For example, if the remote machine 1 is a combination of a cart and a manipulator, the target attitude indicates the target orientation of the cart. For example, if the remote machine 1 is a vehicle, the target attitude indicates the target orientation of the vehicle. For example, if the remote machine 1 is a manipulator, the base of the manipulator is rotatable, and the target attitude indicates the target orientation of the manipulator. For example, the attitude drive mechanism 11 includes an actuator that changes the orientation of the wheels of the remote machine 1. The attitude drive mechanism 11 may be a drive mechanism that can change the orientation of the remote machine 1 on the spot, or may be a drive mechanism that can change the orientation by turning. Also, for example, if the remote machine 1 is a combination of a cart and a manipulator, the target attitude may be appropriately allocated to the orientation of the cart and the target orientation of the manipulator.

The position drive mechanism 12 is a mechanism capable of changing the position of the remote machine 1, and changes the position of the remote machine 1 in response to a control signal received from the operation calculator 3. The position drive mechanism 12 includes, for example, an actuator capable of changing at least one of the horizontal and vertical positions of the remote machine 1.

The camera 2 is mounted on the remote machine 1 and captures the surroundings of the remote machine 1. There is no restriction on the number of cameras 2, and one or more may be used. In the following embodiment, an example in which the camera 2 includes two fisheye cameras with different viewpoints will be described, but the camera 2 is not limited to a fisheye camera.

The image presentation device 4 is, for example, a display device such as a monitor, a display, or a smartphone monitor, and presents an image to an operator who remotely operates the remote machine 1. The image presentation device 4 presents, for example, an image captured by the camera 2 to the operator as a presented image. The operation device 5 is a device operated by the operator, such as, but not limited to, a mouse, a keyboard, a touchpad, or a device that performs screen operation by detecting a face or line of sight. For example, as described below, the operation device 5 accepts input of a target attitude of the remote machine 1 as a position in the presented image by the operator's operation, and accepts input of a position movement instruction indicating that the position of the remote machine 1 is to be moved by the operator's operation. The image presentation device 4 and the operation device 5 may also be integrated, and a touch panel, a smartphone, or the like may be used. In the following, as an example, an example in which the operation device 5 is a mouse and the image presentation device 4 is a monitor will be mainly described.

The operation calculator 3 receives the image captured by the camera 2 from the camera 2, and outputs the received image to the image display device 4, thereby displaying the image on the image display device 4. The operation calculator 3 also controls the remote machine 1 according to the operation content accepted by the operation device 5. For example, the operation calculator 3 drives the attitude of the remote machine 1 based on the target attitude accepted by the operation device 5, and moves the position of the remote machine 1 based on the position movement instruction accepted by the operation device 5. The camera 2 and the operation calculator 3 may be directly connected, or the camera 2 and the operation calculator 3 may have a communication unit not shown in the figure, and data may be transmitted and received by communication. The communication between the camera 2 and the operation calculator 3 may be wired communication, wireless communication, or a combination of wired communication and wireless communication. Similarly, the operation calculator 3 and the remote machine 1 may be directly connected, or the operation calculator 3 and the remote machine 1 may have a communication unit not shown in the figure, and data may be transmitted and received by communication. Similarly, the image presentation device 4 and the operation device 5 may be directly connected to the remote machine 1, or the image presentation device 4, the operation device 5, and the operation calculator 3 may be provided with a communication unit (not shown), and data may be transmitted and received through communication. The communication between the image presentation device 4 and the operation device 5 and the remote machine 1 may be wired communication, wireless communication, or a combination of wired communication and wireless communication.

Furthermore, the operation calculator 3 may be provided at a first location where the remote machine 1 is located, or at a second location where the image presentation device 4 and the operation device 5 are located, i.e., on the operator's side. For example, the operation calculator 3 may be mounted on the remote machine 1. Furthermore, the operation calculator 3 may be provided at a location different from both the first location and the second location. For example, the operation calculator 3 may be provided on a cloud server.

The operation calculator 3 includes an input discrimination unit 31, a display information generation unit 32, a target attitude setting unit 33, an attitude drive unit 34, a target attitude determination unit 35, and a position drive unit 36. The input discrimination unit 31 acquires operation information indicating the operation content from the operation device 5, and outputs the operation information to the corresponding functional unit according to the content of the acquired operation information. The display information generation unit 32 generates display data indicating a display screen to be displayed on the image presentation device 4 using the image received from the camera 2 and the operation information received from the input discrimination unit 31, and transmits the generated display data to the image presentation device 4. The display information generation unit 32 also outputs the image received from the camera 2 to the target attitude setting unit 33.

The target attitude setting unit 33 sets a target attitude of the remote machine 1 based on the operation information received from the input discrimination unit 31, and notifies the attitude driving unit 34 and the target attitude determination unit 35 of the set target attitude. When the attitude driving unit 34 is notified of the target attitude by the target attitude setting unit 33, it generates a control signal for driving the attitude of the remote machine 1 based on the target attitude, and transmits the generated control signal to the remote machine 1. Furthermore, when the attitude driving unit 34 is notified by the target attitude determination unit 35 that the target attitude has not been reached, it generates a control signal for driving the attitude of the remote machine 1, and transmits the generated control signal to the remote machine 1.

The target attitude determination unit 35 determines whether the attitude of the remote machine 1 has become the target attitude, and if it determines that it has become the target attitude, it notifies the position drive unit 36 that it has become the target attitude. If the target attitude determination unit 35 determines that it has not become the target attitude, it notifies the attitude drive unit 34 that it has not become the target attitude. When notified by the target attitude determination unit 35 that the target attitude has been achieved, if the operation information received from the input discrimination unit 31 indicates that the position of the remote machine 1 is to be moved, the position drive unit 36 generates a control signal for moving the position of the remote machine 1 and transmits the generated control signal to the remote machine 1.

FIG. 2 is a diagram showing a schematic example of a remote control system 100 according to the present embodiment. In the example shown in FIG. 2, the remote machine 1 includes a manipulator 6 and a dolly 7. A fish-eye camera 2-1 is provided on the front of the dolly 7, and a fish-eye camera 2-2 is provided near the hand 6a of the manipulator 6. In the example shown in FIG. 2, the image display device 4 is a monitor, and the operation device 5 is a mouse. The image display device 4 displays an image captured by the camera 2-1 or an image captured by the camera 2-2. Both cameras 2-1 and 2-2 are examples of the camera 2 shown in FIG. 1. In this embodiment, the camera switching, that is, the switching of whether the image captured by the camera 2-1 or the image captured by the camera 2-2 is displayed on the image display device 4, is performed by the operator. The operator remotely controls the movement of the remote machine 1 between the target attitude and position by operating the operation device 5 while watching the image displayed on the image display device 4.

FIG. 3 is a diagram showing an example of a method for setting a target posture in this embodiment. FIG. 3 is a diagram showing an example of a display screen of the image display device 4. In the example shown in FIG. 3, an image captured by the camera 2-1 is displayed on the image display device 4. This image shows the upper part of the dolly 7 and the hand 6a of the manipulator 6. In the example shown in FIG. 3, a marker 201 indicating the target posture is displayed together with the image captured by the camera 2-1. The marker 201 in the display screen can be changed by the operator operating the operation device 5. For example, if the operation device 5 is a mouse, the position of the marker 201 in the display screen moves in accordance with the movement of the operation device 5, and the position of the marker 201 in the display screen is determined by pressing the mouse, which is the operation device 5, thereby specifying the target posture. Note that in FIG. 3, the dolly 7, the hand 6a, and the marker 201 are respectively given the

symbols

7, 6a, and 201 for the purpose of explanation, but these symbols are not displayed on the display screen. Similarly, in the subsequent figures of the display screen, symbols are given for the purpose of explanation, but these symbols are not displayed on the display screen.

In this embodiment, for example, the remote machine 1 operates while the mouse, which is the operating device 5, is pressed. More specifically, when the position of the marker 201, i.e., the target attitude, is determined by pressing the mouse, which is the operating device 5, the operation calculator 3 controls the attitude of the remote machine 1 so that the cart 7 rotates in a direction corresponding to the target attitude. That is, the operation calculator 3 may display the marker 201 that can be moved by operating the operating device 5 in the presented image, and set the position of the marker 201 at the time when the operating device 5 is pressed by the operator as the target attitude. Then, after starting to drive the attitude of the remote machine 1 by setting the target attitude, the operation calculator 3 may determine that the position movement instruction has been accepted by the operating device 5 if the pressing of the operating device 5 continues at the time when the attitude of the remote machine 1 reaches the target attitude, and may start moving the position of the remote machine 1.

When the mouse, which is the operating device 5, is no longer pressed while the remote machine 1 is operating, the operation calculator 3 stops driving the attitude of the remote machine 1 and stopping the movement of the position of the remote machine 1. Then, when the remote machine 1 is driven to the target attitude, that is, when the orientation of the remote machine 1 corresponds to the target attitude, the operation calculator 3 controls the position of the remote machine 1 so that the remote machine 1 starts moving in the direction of travel in that attitude. While the mouse, which is the operating device 5, is pressed, the operation calculator 3 continues to move the position of the remote machine 1, and when the mouse, which is the operating device 5, is no longer pressed, the operation calculator 3 stops the remote machine 1.

FIG. 4 is a diagram showing an example of an operation using the operation device 5 of this embodiment. Each diagram in FIG. 4 shows an example of a display screen of the image presentation device 4, as in FIG. 3, and a marker 201 indicating the target attitude is displayed on each display screen together with an image captured by the camera 2-1. FIG. 4 shows an example in which a mouse is used as the operation device 5. The first display screen in FIG. 4 shows a state in which a target is set. In detail, the operator moves the marker 201 to a position to be set as the target attitude and presses the mouse, which is the operation device 5, to set the target attitude. In FIG. 4, the vector from the center of the remote machine 1 to the front (front face) of the remote machine 1 is vector 202, and the vector corresponding to the target attitude is vector 203. The operation calculator 3 calculates vector 202 and vector 203 based on the image and the position of the marker 201 in the image, and starts control to rotate the cart 7 of the remote machine 1 counterclockwise so that vector 202 coincides with vector 203.

After setting the target attitude, if the operator continues to press the mouse, which is the operating device 5, the attitude of the carriage 7 of the remote machine 1 will change, as shown in the second display screen of Figure 4, i.e., the orientation of the carriage 7 of the remote machine 1 will rotate counterclockwise, and the marker 201 will be positioned in front of the carriage 7 of the remote machine 1. This ends the attitude drive.

If the mouse, which is the operating device 5, continues to be pressed in the state shown in the second display screen of FIG. 4, the operation calculator 3 starts position driving to move the trolley 7 of the remote machine 1 forward, as shown in the third display screen of FIG. 4. Thereafter, the position driving of the trolley 7 continues until the pressing of the mouse, which is the operating device 5, ends.

In the example shown in FIG. 4, the operator sets the target posture by moving the mouse, which is the operation device 5, to the desired position in the image projected as the display screen and pressing the mouse, and by continuing to press the mouse, the remote machine 1 is driven to the target posture as described above. Then, when the remote machine 1 is in the target posture, if the operator continues to press the mouse, which is the operation device 5, position driving of the remote machine 1 begins. In this embodiment, the operator does not need to wear a head-mounted display, and there is no need for stereoscopic perception. This reduces the operation load on the operator. Also, the operator can change the posture and position of the remote machine 1 simply by moving and pressing the mouse, which is the operation device 5, while looking at the display screen of the image presentation device 4, allowing remote operation with simple operations.

In the above example, the operator continues to press the mouse, which is the operation device 5, to instruct the remote machine 1 to continue operating. That is, in the above example, pressing the mouse after the remote machine 1 has been driven to the target posture means the start of position driving, and then stopping the mouse pressing means the end of position driving. The specific method of specifying the operation of the remote machine 1 is not limited to the above example. For example, the following operation may be performed. In the first display screen of FIG. 4, the operator moves the marker 201 to a position corresponding to the target posture, and then clicks or double-clicks the mouse, which is the operation device 5, to once confirm the target posture. Thereafter, while the posture of the remote machine 1 is being changed, the operator does not press the mouse, and the operation calculator 3 stops the remote machine 1 when the posture of the remote machine 1 becomes the target posture. In this state, the operator may press the mouse, which is the operation device 5, to start position driving, and stop pressing the mouse to end the position driving. Alternatively, when the attitude of the remote machine 1 reaches the target attitude, the operator can click or double-click the mouse, which is the operating device 5, to start driving the position, and click or double-click again to end the driving of the position.

The operation device 5 may be a touchpad, in which case, for example, pressing the touchpad may be used instead of pressing the mouse, and tapping or double tapping the touchpad may be used instead of clicking or double clicking the mouse. The operation device 5 may be a touch panel integrated with the image presentation device 4, in which case the operator determines the target posture by touching a position on the touch panel that corresponds to the target posture on the display screen, and, for example, pressing the touch panel may be used instead of pressing the mouse, and tapping or double tapping the touch panel may be used instead of clicking or double clicking the mouse. The operation device 5 may be a keyboard. When the operation device 5 is a keyboard, for example, specific keys such as arrow keys may be assigned to up, down, left, and right movement, and the marker 201 may be moved using these keys, and pressing specific keys such as the enter key may be treated the same as pressing the mouse.

FIG. 5 is a flowchart showing an example of the operation of the operation calculator 3 of this embodiment. Note that FIG. 5 shows an example in which an operator starts pressing the operation device 5 at a position indicating the target posture, and continues pressing the operation device 5 while the remote machine 1 is operating, thereby remotely operating the remote machine 1. The operation calculator 3 first judges whether or not a target setting has been input (step S1). In detail, the input discrimination unit 31 judges whether or not the operation information received from the operation device 5 is information indicating the setting of the target posture. As described above, the information indicating the setting of the target posture is, for example, information input from the operation device 5 when the operation device 5 is pressed, and includes information indicating the position of the operation device 5 (position on the display screen). In addition, the input discrimination unit 31 notifies the position drive unit 36 of operation information indicating that the input of a movement instruction is continuing while the operation device 5 is being pressed. If there is no target setting input (step S1 No), the operation calculator 3 repeats step S1.

If there is an input for setting a target attitude (step S1 Yes), a target attitude is set (step S2). In detail, the input discrimination unit 31 outputs operation information to the target attitude setting unit 33, and the target attitude setting unit 33 sets a target attitude using the image acquired from the display information generation unit 32 and the operation information indicating the setting of the target attitude acquired from the input discrimination unit 31, and notifies the attitude driving unit 34 and the target attitude determination unit 35 of the set target attitude. For example, the target attitude setting unit 33 obtains the vector 202 illustrated in the first row of FIG. 4 using the image, obtains the vector 203 using the operation information and the image, and calculates the difference between the

vectors

202 and 203 to calculate the direction and angle by which the remote machine 1 is rotated as the target attitude. In this way, the target attitude is calculated as a relative value from the current attitude, that is, the amount of change from the current attitude.

Next, the operation calculator 3 performs posture driving (step S3). In detail, the posture driving unit 34 generates a control signal for driving the posture of the remote machine 1 based on the target posture notified by the target posture setting unit 33, and transmits the generated control signal to the remote machine 1. For example, the posture driving unit 34 generates a control signal to change the orientation at a constant speed so as to approach the target posture received from the target posture setting unit 33.

Next, the operation calculator 3 determines whether the target attitude has been reached (step S4). In detail, the target attitude determination unit 35 determines whether the remote machine 1 has reached the target attitude based on the target attitude notified by the target attitude setting unit 33. As described above, when the target attitude is indicated as a relative value from the current attitude, for example, the target attitude determination unit 35 determines that the remote machine 1 has reached the target attitude when the target attitude is equal to or less than a threshold value. The threshold value is determined in advance, and may be set to, for example, 0, or may be set to a value corresponding to an error range such that the amount of change in the target attitude, i.e., the attitude that changes the remote machine 1, can be considered to be 0.

If it is determined that the target posture has been reached (step S4 Yes), the operation calculator 3 determines whether the input of a movement command is continuing (step S5). In detail, if it is determined that the target posture has been reached, the target posture determination unit 35 notifies the position drive unit 36 of this fact, and upon receiving the notification that the target posture has been reached, the position drive unit 36 determines Yes in step S5 if it has received operation information indicating that movement is continuing from the input discrimination unit 31. If it is determined that the input of a movement command is continuing (step S5 Yes), the operation calculator 3 performs position drive (step S6). In detail, the position drive unit 36 generates a control signal for moving the remote machine 1 forward, and transmits the generated control signal to the remote machine 1.

If it is determined in step S4 that the target posture has not been reached (step S4: No), the operation calculator 3 repeats the process from step S3. If it is determined in step S5 that the input of the movement command is not continuing (step S5: No), the operation calculator 3 ends the process.

Note that FIG. 5 shows an example in which the operator remotely controls the remote machine 1 by starting to press the operation device 5 at a position indicating the target posture and continuing to press while the remote machine 1 is operating. However, when operation is performed using other operation methods, the operation is generally similar to FIG. 5, although it is changed appropriately depending on the operation method. For example, when the operation device 5 is clicked at a position indicating the target posture, and position drive is started by clicking after the remote machine 1 reaches the target posture, in step S5, the operation calculator 3 judges Yes when it receives operation information indicating that a click was made after the remote machine 1 reaches the target posture.

In addition, in the above example, position driving is performed after posture driving, but this is not limited to the above, and operations related to posture driving and position driving may be performed separately. For example, the operator may use the operation device 5 to perform a predetermined operation such as double-clicking or double-tapping to drive the position of the remote machine 1, and posture driving may be performed by specifying a position using the operation device 5 and continuing to press it (long press).

Next, the hardware configuration of the operation calculator 3 of this embodiment will be described. The operation calculator 3 of this embodiment shown in FIG. 1 functions as the operation calculator 3 when a computer system executes a program, which is a computer program describing the processing in the operation calculator 3. FIG. 6 is a diagram showing an example of the configuration of a computer system that realizes the operation calculator 3 of this embodiment. As shown in FIG. 6, this computer system includes a control unit 101, an input unit 102, a memory unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the memory unit 103 form a processing circuit.

6, the control unit 101 is, for example, a processor such as a CPU (Central Processing Unit), and executes a program in which the processing in the operation calculator 3 of this embodiment is described. Note that a part of the control unit 101 may be realized by dedicated hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array). The input unit 102 is a button, a keyboard, a mouse, a touchpad, etc. The storage unit 103 includes various memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory) and a storage device such as a hard disk, and stores the program to be executed by the control unit 101, necessary data obtained in the process of processing, etc. The storage unit 103 is also used as a temporary storage area for the program. As described above, the display unit 104 is, for example, a display. Note that the display unit 104 and the input unit 102 may be integrated and realized by a touch panel, etc. The communication unit 105 is a receiver and a transmitter that perform communication processing. The output unit 106 is a speaker or the like. Note that FIG. 6 is an example, and the configuration of the computer system is not limited to the example of FIG. 6. For example, in this embodiment, the computer system that realizes the operation calculator 3 does not need to include the display unit 104 and the output unit 106.

Here, an example of the operation of the computer system until the program of this embodiment is in a state where it can be executed will be described. In a computer system having the above-mentioned configuration, for example, a computer program is installed in the storage unit 103 from a CD-ROM or DVD-ROM set in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from the storage unit 103 is stored in the main memory area of the storage unit 103. In this state, the control unit 101 executes processing as the operation calculator 3 of this embodiment according to the program stored in the storage unit 103.

In the above explanation, a program describing the processing in the operational calculator 3 is provided on a CD-ROM or DVD-ROM as a recording medium, but this is not limiting. Depending on the configuration of the computer system and the capacity of the program provided, for example, a program provided via a transmission medium such as the Internet may be used.

The operation device 5 shown in FIG. 1 may be the input unit 102 in the computer system that realizes the operation calculator 3 shown in FIG. 1, or may be provided separately from the input unit 102. The image presentation device 4 shown in FIG. 1 may be the display unit 104 in the computer system that realizes the operation calculator 3 shown in FIG. 1, or may be provided separately from the display unit 104.

The input discrimination unit 31, display information generation unit 32, target posture setting unit 33, posture drive unit 34, target posture determination unit 35, and position drive unit 36 shown in FIG. 1 are realized by the control unit 101 shown in FIG. 6 executing a computer program stored in the storage unit 103 shown in FIG. 6. The storage unit 103 shown in FIG. 6 is also used to realize the input discrimination unit 31, display information generation unit 32, target posture setting unit 33, posture drive unit 34, target posture determination unit 35, and position drive unit 36 shown in FIG. 1.

As described above, the remote operation system 100 of this embodiment comprises the remote machine 1, the image presentation device 4 that presents the image from the camera 2 to the operator, the operation device 5, and the operation calculator 3. The operation calculator 3 drives the attitude of the remote machine 1 based on the target attitude specified as a position in the target image by the operation device 5, and moves the position of the remote machine 1 based on instructions to start and end position drive input using the operation device 5. This reduces the operational burden on the operator.

Embodiment 2.
7 is a diagram showing a configuration example of a remote control system according to the second embodiment. A remote control system 100a according to the second embodiment is similar to the remote control system 100 according to the first embodiment, except that it includes an operation calculator 3a instead of the operation calculator 3 and a precision operation device 8a is added. Components having the same functions as those in the first embodiment are given the same reference numerals as those in the first embodiment, and duplicated explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

The precision operation device 8a is a device that allows more precise operation than the operation device 5, and is, for example, a joystick, an operation device that combines a joystick and a dial, or a mouse that allows precision operation. When a joystick is used as the precision operation device 8a and the remote machine 1 has multiple drive axes, the operation of which drive axis is to be performed may be switched by a button on the joystick or an operation on the operation device 5. The precision operation device 8a may also be the same as the operation device 5 in terms of hardware. For example, a mouse that can be switched between a normal mode and a mode that allows precision operation may be set to the normal mode when used as the operation device 5, and to the mode that allows precision operation when used as the precision operation device 8a. For example, when the operation device 5 is a touch panel, a mouse, a keyboard, or the like, the operation calculator 3a may enlarge the image displayed on the image display device 4 in response to the operation of the operation device 5, thereby enabling precision operation by the operator. The operation of enlarging may be an operation used for enlarging a general screen, and for example, when the operation device 5 is a touch panel, a pinch out may be used as the operation of enlarging. The enlargement operation is not limited to this, and a button for enlargement or the like may be displayed on the image display device 4, and the position to be enlarged and displayed may be moved using the operation device 5.

The operation calculator 3a is similar to the operation calculator 3 of the first embodiment, except that an input switch 8b is added and an input discrimination unit 31a is provided instead of the input discrimination unit 31. In FIG. 7, an input switch is provided separately from the operation calculator 3a, but the input switch 8b may be provided within the operation calculator 3a.

The input switch 8b switches the operation target from the operator between the operation device 5 and the precision operation device 8a. For example, the input switch 8b has a function of switching between normal operation (normal operation mode) and precision operation (precision operation mode). In the normal operation mode, the input switch 8b outputs operation information input from the operation device 5 to the input discrimination unit 31a. When the input discrimination unit 31a receives operation information input from the operation device 5 via the input switch 8b, it performs the same operation as in the first embodiment. In the precision operation mode, the input switch 8b outputs operation information input from the precision operation device 8a to the input discrimination unit 31a. The input discrimination unit 31a outputs the operation information to the attitude drive unit 34 or the position drive unit 36 depending on the content of the operation information.

When the operation device 5 and the precision operation device 8a are different devices, the input switch 8b may switch between normal operation and precision operation depending on which of these devices the input is from, or may switch between normal operation and precision operation when a specific operation is performed by the operation device 5. In addition, the image presentation device 4 may be provided with buttons for switching operations, and the input switch 8b may switch between normal operation and precision operation by pressing these buttons by the operation device 5, or may detect the operator's operation by gesture and switch between normal operation and precision operation by the operator's gesture. In addition, when the remote machine 1 has a manipulator 6, the attitude drive unit 34 of the remote machine 1 may have a function as a manipulator drive unit that drives each joint and hand 6a of the manipulator 6. In this case, the attitude drive unit 34 may generate a control signal for the attitude drive unit 34 to control the manipulator 6 in response to the operation by the precision operation device 8a, and transmit the generated control signal to the remote machine 1. Alternatively, the manipulator drive unit may be provided separately from the attitude drive unit 34.

The operation method described in the first embodiment can reduce the operational burden on the operator, but since the operation is performed using an operating device 5 such as a mouse, it may be difficult to precisely operate the remote machine 1. Also, for example, when the remote machine 1 has a manipulator 6 and approaches an object to be operated by the manipulator 6, it may be easier for the operator to perform precise remote operation by using an operation that directly corresponds to the drive shaft of the remote machine 1, such as an operation using a joystick, rather than using the operation method described in the first embodiment. In this embodiment, an input switch 8b that switches between normal operation and precise operation is provided, allowing the operator to more appropriately operate the remote machine 1.

When performing an operation using precision manipulation, a screen showing an operation method corresponding to the precision manipulation device 8a may be displayed on the image presentation device 4. The display screen of one image presentation device 4 may be divided to separate the image captured by the camera 2 and the screen showing the operation method, or multiple image presentation devices 4 may be provided, and the image captured by the camera 2 and the screen showing the operation method may be displayed on different image presentation devices 4.

8 to 10 are diagrams showing an example of a screen showing the operation method of this embodiment. FIG. 8 to FIG. 10 are display examples of the operation method when a joystick is used as the precision operation device 8a in precision operation, and in FIG. 8 to FIG. 10, the operation of the joystick in the Arm Joystick Mode, the Wrist Mode, and the Drive Joystick Mode and the corresponding operation of the remote machine 1 are displayed together with an image showing the operation of the remote machine 1. The display information generating unit 32 receives an input regarding the setting of the mode from the input discrimination unit 31, and controls the image display device 4 to display, for example, one of FIG. 8 to FIG. 10 according to the current mode. This allows the operator to operate the joystick while checking the operation method, and allows the operator to operate the remote machine 1 even if the operator has not memorized the correspondence between the joystick operation and the corresponding operation of the remote machine 1. In addition, it is possible to suppress the operator's operation errors. Note that FIG. 8 to FIG. 10 are examples, and the settable modes, the correspondence between each joystick operation and the operation of the remote machine 1 are not limited to the examples shown in FIG. 8 to FIG. 10. 8 to 10 show an example in which a joystick is used as the precision operation device 8a, but even when a precision operation device 8a other than a joystick is used, the same effect as when a joystick is used can be obtained by displaying an operation method corresponding to the precision operation device 8a. Also, not limited to precision operation, an operation method corresponding to the operation device 5 may be displayed in the same manner for normal operation using the operation device 5.

Also, a portable terminal device such as a smartphone or tablet may be used as the image presentation device 4. FIG. 11 is a diagram showing an example in which a terminal device is used as the image presentation device 4 of this embodiment. When a terminal device is used as the image presentation device 4, an image corresponding to the location where the terminal device is held is displayed on the terminal device. This makes it possible to realize a virtual wide monitor by moving the image presentation device 4. For example, the image displayed on the image presentation device 4, which is a terminal device, may also be able to be enlarged and reduced. In addition, both the monitor and the terminal device as shown in FIG. 2 may be used as the image presentation device 4. For example, an image connected to an image displayed on the image presentation device 4, which is a monitor, is displayed according to the position of the image presentation device 4, which is a terminal device, thereby realizing a virtual screen wider than the image presentation device 4, which is a monitor.

Furthermore, when the image displayed on the image display device 4 is enlarged, the dolly 7 disappears, and the relative positional relationship between the manipulator 6 and the dolly 7 may become unclear. FIG. 12 is a diagram showing an example of a method of displaying an image in this embodiment. The upper part of FIG. 12 shows an example of a display screen when the image captured by the camera 2-1 is enlarged. The upper part of FIG. 12 is, for example, an enlarged upper part of the third display screen of FIG. 4 described in embodiment 1, and although the hand 6a is enlarged, the dolly 7 displayed on the third display screen of FIG. 4 disappears. This makes it difficult for the operator to know the orientation of the dolly 7 and to operate it. For this reason, a three-dimensional figure 308 shown by a dashed line in the lower part of FIG. 12 may be superimposed on the image. That is, the display information generating unit 32 may generate a three-dimensional figure 308 that imitates the current state of the remote machine 1 so that the orientation of the trolley 7 can be known, for example, by using CG (Computer Graphics), generate display data in which the three-dimensional figure 308 is superimposed on the image captured by the camera 2-1, and display the display data on the image display device 4. By displaying the three-dimensional figure 308 that imitates the current state of the remote machine 1, the operator can grasp the orientation of the trolley 7. Note that FIG. 12 is an example, and the display method of the three-dimensional figure 308 is not limited to the example shown in FIG. 12, and the image to be displayed is not limited to the image of the camera 2-1. In addition, instead of the three-dimensional figure 308, a two-dimensional figure, symbol, character, etc. indicating the orientation of the trolley 7 may be displayed, or a symbol, character, etc. indicating the orientation of the trolley 7 may be displayed together with the three-dimensional figure 308.

The operation calculator 3a of this embodiment is realized by a computer system, similar to the operation calculator 3 of embodiment 1. For example, the operation calculator 3a of this embodiment is realized by the computer system illustrated in FIG. 6, similar to the operation calculator 3 of embodiment 1.

Embodiment 3.
13 is a diagram showing a configuration example of a remote control system according to the third embodiment. A remote control system 100b according to the present embodiment is similar to the remote control system 100a according to the second embodiment, except that the remote control system 100b according to the present embodiment includes an operation calculator 3b instead of the operation calculator 3a, and multiple cameras 2 are provided. Components having the same functions as those in the second embodiment are given the same reference numerals as those in the second embodiment, and duplicated explanations are omitted. Below, differences from the second embodiment will be mainly explained.

In the first and second embodiments, one or more cameras 2 are provided, but in the present embodiment, two or more cameras 2 are provided. For example, as shown in FIG. 2, cameras 2-1 and 2-2 are provided as cameras 2.

The operation calculator 3b is the same as the operation calculator 3a of the second embodiment, except that a camera switching unit 37, which is a camera switch, is added. The camera switching unit 37 switches between the multiple cameras 2 as the camera from which the image to be displayed on the image presentation device 4 is obtained. That is, the camera switching unit 37 switches the camera 2 corresponding to the presented image by selecting the presented image to be presented on the image presentation device 4 from the multiple images captured by each of the multiple cameras 2. The camera switching unit 37 notifies the display information generating unit 32 of information indicating the camera 2 from which the image to be displayed on the image presentation device 4 is obtained, that is, information indicating the camera 2 selected as the selected display target. The display information generating unit 32 generates display data using the notified image of the camera 2. The camera switching unit 37 may switch the image to be displayed on the image presentation device 4 by receiving operation information instructing the camera to be switched from the operation device 5 or the precision operation device 8a via the input discrimination unit 31a, or may switch the image to be displayed on the image presentation device 4 by detecting the operator's gesture. Alternatively, a button for switching cameras may be displayed on the image presentation device 4, and the operator may use the operation device 5 or the precision operation device 8a to press the button to switch the camera 2. Alternatively, the image displayed on the image presentation device 4 may be switched according to preset conditions and the image displayed on the image presentation device 4.

14 is a diagram showing an example of switching of the camera 2 in this embodiment. FIG. 14 shows an example in which the camera 2-1 installed on the dolly 7 and the camera 2-2 installed near the hand 6a of the manipulator 6 are used, as described in FIG. 2 of the first embodiment, and the image captured by the camera 2-1 is displayed on the display screen 301, and the image captured by the camera 2-2 is displayed on the display screen 302. The camera switching unit 37 may switch the image displayed as the display screen on the image presentation device 4 according to the operation information input from the operation device 5 or the precision operation device 8a, as described above, or may switch by detecting the gesture of the operator, or may switch according to a preset condition and the image displayed on the image presentation device 4. The preset condition may be, for example, but is not limited to, a condition in which the image of the camera 2-2 is displayed when the distance between the remote machine 1 and the object is equal to or less than a specified distance, and the image of the camera 2-1 is displayed when the distance between the remote machine 1 and the object is greater than a specified distance. The target object is determined in advance by the operator using the operation device 5 or the precision operation device 8a.

In this way, by the camera switching unit 37 switching between cameras 2, for example, the operator can perform remote control while watching the image captured by camera 2-2 until the remote machine 1 approaches the target object to a certain extent, and when the remote machine approaches the target object beyond a certain extent, the operator can perform remote control while watching the image captured by camera 2-1.

Also, a camera provided inside the hand 6a of the manipulator 6 may be used as one of the cameras 2. FIG. 15 is a diagram showing an example of a camera provided inside the hand 6a of the manipulator 6 in this embodiment. In FIG. 15, the manipulator 6 and the object 9 to be grasped by the hand 6a are shown on the left, and an image displayed on the image display device 4 is shown on the right. The display screen 310 shows the screen on which the image captured by the camera 2-2 is displayed, and the display screen 320 shows the screen on which the image captured by the camera 2-3 is displayed. As shown in FIG. 15, the camera 2-3 is provided inside the hand 6a, and when the hand 6a approaches the object 9 to be grasped, the image displayed on the image display device 4 is switched from the image captured by the camera 2-2 to the image captured by the camera 2-3.

By checking the images captured by the cameras 2-3, the operator can perform operations while understanding the state of the grasped object 9 in detail. For example, if either the grasped object 9 or the hand 6a is a deformable structure, the operator can check the degree of deformation of the deformable structure and adjust the gripping force of the hand 6a. The adjustment of the gripping force of the hand 6a may be performed by the operating device 5 or the precision operating device 8a, or by other operating means other than these.

In the above example, the image displayed on the image presentation device 4 was an image captured by one camera 2, but the image presentation device 4 may display images captured by multiple cameras 2 as separate display screens. In this case, the display information generation unit 32 may display the image captured by the selected camera 2 at high illuminance and reduce the illuminance of the image of the unselected camera 2. The display information generation unit 32 may also display the display screen of the image of the selected camera 2 in the center and display the image of the unselected camera 2 at the edge. The display information generation unit 32 may also generate display data so that the size of the image of the unselected camera 2 is smaller than the size of the image of the selected camera 2.

The operation calculator 3b of this embodiment is realized by a computer system, similar to the operation calculator 3 of embodiment 1. For example, the operation calculator 3b of this embodiment is realized by the computer system illustrated in FIG. 6, similar to the operation calculator 3 of embodiment 1. Note that, although the camera switching unit 37 is provided within the operation calculator 3b in FIG. 13, a camera switcher having the functions of the camera switching unit 37 may be provided separately from the operation calculator 3b.

In addition, FIG. 13 shows an example in which a camera switching function is added to the operation calculator 3a of embodiment 2, but the remote control system 100 of embodiment 1 may be equipped with multiple cameras 2, and a camera switching unit 37 may be added to the operation calculator 3 to add a camera switching function.

Furthermore, as one of the multiple cameras 2, a camera 2 that photographs the manipulator 6 from the side may be provided. FIG. 16 is a diagram showing an example of a camera 2 in this embodiment that photographs the manipulator 6 from the side. In the example shown in FIG. 16, camera 2-4, which is one of the multiple cameras 2, photographs the manipulator 6 from the side. A drive mechanism is provided in the camera 2-4, and the drive mechanism drives the camera 2-4 so as to track, for example, the tip of the hand 6a. FIG. 17 is a diagram showing an example of switching of images in this embodiment. In the example shown in FIG. 17, for example, a display screen 304 on which an image photographed by camera 2-4 is displayed and a display screen 302 on which an image photographed by camera 2-2 is displayed are switched by a gesture of the operator.

For example, when the operator looks towards the image presentation device 4 while the display screen 302 is displayed, the display screen displayed on the image presentation device 4 switches to the display screen 304 which displays the image captured by the camera 2-4. This allows the operator to check the manipulator 6 from the side, and to avoid the manipulator 6 colliding with an external mechanism. Note that while FIG. 17 shows an example of switching between the image captured by the camera 2-2 and the image captured by the camera 2-4, it is also possible to switch between the image captured by the camera 2-1 and the image captured by the camera 2-4 in a similar manner.

Also, the image captured by camera 2-4 from the side may be superimposed on the image captured by camera 2-1 or camera 2-2. FIG. 18 is a diagram showing an example of superimposing an image captured from the side in this embodiment. The upper left diagram in FIG. 18 shows an example in which an image obtained by capturing the tip of hand 6a and the object to be grasped 9 by camera 2-2 is displayed as display screen 302. In the upper right diagram in FIG. 18, display screen 304 showing the image from camera 2-4 is displayed on display screen 302. In this way, display screen 304 may be displayed on display screen 302, but if left as is, the entire screen will look unnatural. For this reason, a virtual mirror 307 may be displayed as shown in the lower left diagram in FIG. 18. That is, the display information generating unit 32 generates a composite image showing the mirror 307 and the image displayed on the mirror 307, assuming that the image from the camera 2-4 is reflected in the virtual mirror 307, generates display data by superimposing the composite image on the image captured by the camera 2-1 or the camera 2-2, and displays the display data on the image presenting device 4. This allows the operator to recognize the image from the camera 2-4 as a natural image.

In the example shown in FIG. 10, the camera 2-3 is provided at the tip of the hand 6a, but one or more pressure detection sensors may be provided inside the tip of the hand 6a. FIG. 19 is a diagram showing an example of a pressure detection and pressure display method in this embodiment. When the operator's finger 601 traces the display unit 602, which is the image presentation device 4 or another monitor, the tip of the hand 6a provided with the pressure detection sensor moves and traces the surface of an object such as a grasped object 9. Note that the display unit 602 may have a function as an input means such as a touch panel, or the movement of the finger may be detected and the tip of the hand 6a may be driven in accordance with the detected movement. Pressure information indicating the pressure detected by the pressure detection sensor is input to the display information generation unit 32 in the same way as the image of the camera 2. The display information generation unit 32 may generate display data based on the pressure information so as to change at least one of, for example, the color density, transparency, and composition ratio, and display the display data on the image presentation device 4 or another monitor to visually present the force haptics to the operator. The presentation of the haptic sensation is not limited to being displayed on the image display device 4 or another monitor, but may be achieved by displaying it on the operator's fingertips using projection mapping.

20 is a diagram showing another example of the pressure detection and pressure display method of this embodiment. In the example shown in FIG. 20, when the operation calculator 3c receives an operation instructing to trace the surface of an object such as the grasped object 9, it moves the tip of the hand 6a so as to trace the surface of the object such as the grasped object 9, and outputs pressure information indicating the pressure detected by the pressure detection sensor and movement information indicating the amount of movement of the tip of the hand 6a to the haptic presentation device 607. Alternatively, using a means for detecting the movement of the finger 601, the operation calculator 3c may control the horizontal position of the tip of the hand 6a in accordance with the movement of the finger 601, thereby moving the tip of the hand 6a so as to trace the surface. The haptic presentation device 607 includes, for example, a contact unit 604, a drive unit 605 capable of driving the contact unit 604, and a support unit 606 that supports the operator's finger from above. For example, the haptic feedback device 607 controls the drive unit 605 based on the movement information to move the horizontal position of the contact unit 604, and moves the contact unit 604 up and down based on the pressure information. When the operator brings the finger 601 into contact with the contact unit 604, the finger 601 moves horizontally together with the contact unit 604, and also moves up and down according to the pressure information. This allows the haptic feedback to be presented, and the operator to perceive the surface condition. Note that, without using a pressure detection sensor, the unevenness of the surface of the object may be detected by detecting the vertical position of the tip of the hand 6a, and the operation calculator 3c may move the contact unit 604 up and down according to the detected unevenness.

A membrane may also be provided on the portion where the operator places his/her finger 601. The membrane can provide the operator with a sliding sensation when tracing a surface. The operation calculator 3c can also reproduce and communicate to the operator the unevenness detected by the tip of the hand 6a by moving the membrane up and down in response to pressure information. The unevenness of the surface of the object can also be detected by detecting the vertical position of the tip of the hand 6a, and the operation calculator 3c can move the contact part 604 up and down in response to the detected unevenness.

Embodiment 4.
21 is a diagram showing a configuration example of a remote control system according to the fourth embodiment. A remote control system 100c according to the present embodiment is similar to the remote control system 100b according to the third embodiment, except that the remote control system 100c according to the present embodiment includes an operation calculator 3c instead of the operation calculator 3b, and a remote machine 1a instead of the remote machine 1. Components having the same functions as those in the third embodiment are given the same reference numerals as those in the third embodiment, and duplicated explanations will be omitted. Below, differences from the third embodiment will be mainly explained.

The remote machine 1a is similar to the remote machine 1 of embodiment 3, except that a hand drive mechanism 13 is added to the remote machine 1. The hand drive mechanism 13 drives the hand 6a based on a control signal received from the operation calculator 3c. Note that the remote machines 1 of embodiments 1 to 3 may also be provided with a hand drive mechanism 13, and there are no particular restrictions on the method of operating the hand drive mechanism 13 in embodiments 1 to 3.

The operation calculator 3c is similar to the operation calculator 3b of embodiment 3, except that a hand angle setting unit 38 is added and an input discrimination unit 31c is provided instead of the input discrimination unit 31a. In this embodiment, the operator can also set the hand angle of the hand 6a using the operation device 5, and when the input discrimination unit 31c receives operation information indicating the hand angle of the hand 6a, it outputs the operation information to the hand angle setting unit 38. The hand angle setting unit 38 uses the operation information received from the input discrimination unit 31c to generate a control signal for controlling the hand 6a of the remote machine 1a, and transmits the generated control signal to the remote machine 1a.

22 and 23 are diagrams showing an example of a hand angle setting screen in this embodiment. FIG. 22 and FIG. 23 each show a display screen displayed on the image display device 4. In the example shown in FIG. 22, angle change buttons 401 to 403 are displayed on the right side of the display screen 301 on which the image captured by the camera 2-1 is displayed. The angle change button 401 is a button for setting the hand angle of the hand 6a in the horizontal direction, the angle change button 402 is a button for setting the hand angle of the hand 6a to 45 deg (the angle between the horizontal direction and the hand 6a is 45 deg), and the angle change button 403 is a button for setting the hand angle of the hand 6a in the vertical direction. In the example shown in FIG. 22, the operator presses the buttons displayed on the image display device 4 using the operation device 5 to set the hand angle of the hand 6a. In addition, FIG. 22 shows an example in which the hand end angle of the hand 6a can be set to three types of hand end angles, horizontal, 45 deg, and vertical, but the number of hand end angles that can be set for the hand end angle of the hand 6a is not limited to this example. For example, it may be possible to set it to one of two types of hand end angles, or it may be possible to set the hand end angle to more than three types. In addition, the display and operation for setting the hand end angle are not limited to FIG. 22. For example, an indicator showing a memory along with the hand end angle may be displayed, and the hand end angle of the hand 6a may be set by moving the memory with the operating device 5, or the hand end angle may be set by a gesture, or the hand end angle of the hand 6a may be set by a method other than these.

23 illustrates an operation method different from that of FIG. 22. In the example shown in FIG. 23, a side view of the hand 6a is displayed as the display screen 304, and the operator rotates the displayed hand 6a using the operation device 5 to set the hand end angle of the hand 6a. That is, an image including the hand end of the hand 6a is displayed on the hand end angle setting screen, and the hand end angle is set by the operator changing the angle of the hand end in the image using the operation device 5. In the example shown in FIG. 23, when the hand end angle of the hand 6a is set in the horizontal direction, the operator uses the operation device 5 to move the operation device 5 in the rotation direction 404 while pressing the part of the hand 6a, and ends the pressing when the desired hand end angle is reached. This sets the hand end angle of the hand 6a. Also, for example, as in the first embodiment, the target hand end angle may be set by clicking the target hand end angle with the marker 201. The image of the hand 6a shown in FIG. 23 may be one that has been created in advance by simulating the remote machine 1a, or may be an image captured by a camera 2 that is provided on the manipulator 6 and captures images of the hand 6a from the side. In the latter case, similar to the operation of the posture and position of the remote machine 1 in embodiment 1, the operator may start pressing the operation device 5 after moving the marker 201 to the target hand angle, and continue pressing the operation device 5 until the target hand angle is reached.

23 may be displayed on the image display device 4 together with the screen (image captured by the camera 2) displayed in the third embodiment, or a specific operation by the operation device 5 may be performed to switch to the screen shown in FIG. 23. For example, when the dolly 7 is clicked while the image captured by the camera 2 is displayed on the image display device 4, the mode may be changed to a mode in which the attitude and position of the remote machine 1 are operated as described in the first embodiment, and when the hand 6a is clicked, the hand tip angle setting screen of the hand 6a may be displayed. Similarly, the position of the hand 6a of the manipulator 6 may be set, and when the manipulator 6 is clicked, the setting screen of the position of the hand 6a of the manipulator 6 may be displayed. For example, the operation calculator 3c may grasp the joint angle by attaching an AR marker to each joint of the manipulator 6, or may control the position of the hand 6a by constructing a robot model inside and grasping the joint angle, or may control the position of the hand 6a by other methods. The position of the hand 6a of the manipulator 6 may be set by clicking or continuously pressing the operating device 5 at the position of the marker 201 as described in embodiment 1, or, as illustrated in FIG. 22, a button indicating the position of the hand 6a of the manipulator 6 may be displayed and the position of the hand 6a of the manipulator 6 may be set by the button.

In this manner, in this embodiment, the remote machine 1a has a hand 6a whose end-point angle can be set, and the image display device 4 displays a setting screen for setting the end-point angle of the hand 6a. In this embodiment, the operator can also set the end-point angle of the hand 6a with a simple operation. Furthermore, the position of the manipulator 6 and the end-point angle of the hand 6a may be set using the precision operation device 8a, as described in the second embodiment.

The operation calculator 3c of this embodiment is realized by a computer system, similar to the operation calculator 3 of embodiment 1. For example, the operation calculator 3c of this embodiment is realized by the computer system illustrated in FIG. 6, similar to the operation calculator 3 of embodiment 1.

In addition, while FIG. 21 shows an example in which a hand angle setting function is added to the operation calculator 3b of embodiment 3, a hand angle setting unit 38 may be added to the operation calculator 3a of embodiment 1 or the operation calculator 3b of embodiment 2 to add a hand angle setting function.

Embodiment 5.
24 is a diagram showing a configuration example of a remote control system according to the fifth embodiment. A remote control system 100d according to the fifth embodiment is similar to the remote control system 100c according to the fourth embodiment, except that the remote control system 100d according to the fifth embodiment includes an operation calculator 3d instead of the operation calculator 3c. Components having the same functions as those in the fourth embodiment are given the same reference numerals as those in the fourth embodiment, and duplicated explanations are omitted. Below, differences from the fourth embodiment will be mainly explained.

The operation calculator 3d is similar to the operation calculator 3c of embodiment 4, except that a motion switching unit 39, which is a motion switch, is added, and the input discrimination unit 31d is provided instead of the input discrimination unit 31c. The motion switching unit 39 switches the operation part of the remote machine 1a in conjunction with switching by the camera switching unit 37. For example, the camera switching unit 37 notifies the motion switching unit 39 of information indicating the selected camera 2, and the motion switching unit 39 switches the operation part based on the notified information.

For example, when the camera switching unit 37 selects the camera 2-1 illustrated in FIG. 2, the motion switching unit 39 selects the cart 7 of the remote machine 1a as the operation part, when the camera switching unit 37 selects the camera 2-2 illustrated in FIG. 2, the motion switching unit 39 selects the manipulator 6 or hand 6a of the remote machine 1a as the operation part, and when the camera switching unit 37 selects the camera 2-3 illustrated in FIG. 15, the motion switching unit 39 selects the hand 6a of the remote machine 1a as the operation part. The motion switching unit 39 notifies the input discrimination unit 31d of the selected operation part. The input discrimination unit 31d outputs the input operation information to the function unit corresponding to the operation part. For example, when the operation part is the cart 7, the input discrimination unit 31d outputs operation information to the target attitude setting unit 33 and the attitude driving unit 34, and when the operation part is the hand 6a, the input discrimination unit 31d outputs operation information to the hand angle setting unit 38. When the operation part is the manipulator 6, the input discrimination unit 31d outputs operation information to a manipulator driving unit (not shown), or to a target attitude setting unit 33 and an attitude driving unit 34 that function as a manipulator driving unit.

As a result, the operator can switch the operation part simply by performing a switching operation or gesture of the camera switching unit 37, thereby reducing the operational burden and allowing the operator to perform operations. In addition, the motion switching unit 39 may set a gain for the motion of the remote machine 1 when the precision operation device 8a is operated according to the camera 2 selected by the camera switching unit 37. The gain may be determined in advance for each camera 2. The gain may be set instead of switching the operation part, or both the operation part switching and the gain setting may be performed. The motion switching unit 39 outputs the set gain to the target attitude setting unit 33, the attitude driving unit 34, and the hand angle setting unit 38. In addition, the camera switching unit 37 may set a gain according to the magnification of the enlargement or reduction even when the image displayed on the image presentation device 4 is enlarged or reduced.

The operation calculator 3d of this embodiment is realized by a computer system, similar to the operation calculator 3 of embodiment 1. For example, the operation calculator 3d of this embodiment is realized by the computer system illustrated in FIG. 6, similar to the operation calculator 3 of embodiment 1. In addition, a motion switch having the function of the motion switch unit 39 may be provided separately from the operation calculator 3d.

In addition, while FIG. 24 shows an example in which a motion switching function is added to the operation calculator 3c of embodiment 4, a motion switching function may be added by adding a motion switching unit 39 to the operation calculator 3a of embodiment 2 or the operation calculator 3b of embodiment 3.

The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or the embodiments may be combined with each other. In addition, parts of the configurations may be omitted or modified without departing from the spirit of the invention.

1, 1a remote machine, 2, 2-1 to 2-3 camera, 3, 3a, 3b, 3c, 3d operation calculator, 4 image display device, 5 operation device, 6 manipulator, 6a hand, 7 dolly, 8a precision operation device, 8b input switch, 9 grasped object, 11 attitude drive mechanism, 12 position drive mechanism, 13 hand drive mechanism, 31, 31a, 31c, 31d input discrimination unit, 32 display information information generation unit, 33 target attitude setting unit, 34 attitude driving unit, 35 target attitude determination unit, 36 position driving unit, 37 camera switching unit, 38 hand tip angle setting unit, 39 motion switching unit, 100, 100a, 100b, 100c, 100d remote operation system, 101 control unit, 102 input unit, 103 memory unit, 104 display unit, 105 communication unit, 106 output unit, 107 system bus.

Claims

A remote machine equipped with a camera and remotely operated;
an image presentation device that presents the image captured by the camera to an operator as a presentation image;
an operation device that is operated by the operator, receives an input of a target attitude of the remote machine as a position in the presented image by the operation of the operator, and receives an input of a position movement instruction indicating that the position of the remote machine is to be moved by the operation of the operator;
an operation calculator that drives the attitude of the remote machine based on the target attitude received by the operation device and moves the position of the remote machine based on the position movement instruction received by the operation device;
A remote control system comprising:
The operation calculator includes:
displaying a marker that can be moved by operating the operation device within the presented image, and setting a position of the marker at a time when the operation device is pressed by the operator as the target attitude;
The remote operation system according to claim 1, characterized in that after starting to drive the attitude of the remote machine by setting the target attitude, when it is determined that the attitude of the remote machine has reached the target attitude, if the operating device continues to be pressed at the time when the attitude of the remote machine has reached the target attitude, it is determined that the position movement instruction has been accepted by the operating device, and movement of the position of the remote machine is started.
The remote control system according to claim 2, characterized in that when the pressing of the operating device is stopped, the driving of the attitude of the remote machine and the movement of the position of the remote machine are stopped.
A precision operation device that enables more precise operation than the operation device;
Equipped with
an input switch for switching an operation target from an operator between the operation device and the precision operation device;
4. The remote control system according to claim 1, further comprising:
The camera is a plurality of cameras,
a camera switcher that switches the camera corresponding to the presentation image by selecting the presentation image from among a plurality of images captured by each of the plurality of cameras;
5. The remote control system according to claim 1, further comprising:
The remote machine has a hand capable of setting a hand angle,
The remote control system according to claim 5 , wherein the image presentation device presents a setting screen for setting the tip angle of the hand.
The remote control system of claim 6, characterized in that the setting screen displays an image including the tip of the hand, and the hand angle is set by the operator using the operation device to change the angle of the tip in the image.
The remote control system according to claim 6 or 7, characterized in that the camera includes a camera provided on the inside of the hand.
a motion switch for switching an operating part of the remote machine in conjunction with the switching by the camera switch;
9. The remote control system according to claim 5, further comprising:
A remote operation method in a remote operation system including a remote machine equipped with a camera and remotely operated, an image display device that displays an image captured by the camera as a display image to an operator, an operation device, and an operation calculator, comprising:
the operation device is operated by the operator, and receives an input of a target attitude of the remote machine as a position in the presented image by the operation of the operator, and receives an input of a position movement instruction indicating that the position of the remote machine is to be moved by the operation of the operator;
a step of the operation computing unit driving the attitude of the remote machine based on the target attitude and moving the position of the remote machine based on the position movement instruction;
A remote control method comprising: