JP2015112688A - Robot remote control system and robot remote control method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot remote control system and a robot remote control method capable of visually setting the position and attitude of a tool provided in a robot relative to a target while viewing an image of a camera fixed to the robot.SOLUTION: A robot remote control system includes a three-dimensional marker 30 fixed to a position at which the three-dimensional marker 30 as well as a tool 12 (hand, for example) and a target 8 (workpiece, for example) is within a visual field of a camera 14 during operation of the tool 12. The three-dimensional marker 30 is set such that a shape thereof on a monitor image changes with a change in a position or an attitude of the tool 12 relative to the target 8.

Description

  The present invention relates to a robot remote control system and method.

The remote operation system is a system that includes a robot for performing work on an object and a remote operation device, and remotely operates the robot wirelessly or the like.
Here, the work refers to, for example, gripping, transporting, and setting a workpiece in an environment (a high temperature environment, a low temperature environment, a contaminated environment, etc.) that cannot be approached by humans. The robot is a self-propelled or fixed device including, for example, a robot arm (for example, an articulated arm or a manipulator). The robot further includes a communication device that communicates with the remote control device, a tool (for example, a hand) for performing work, and a camera for monitoring the work state. The remote operation device includes a communication device that communicates with a robot, a display device (monitor) that displays an image of a camera, and an operation device (such as a joystick) that allows a person to operate a robot arm.

The remote operation system described above is disclosed in, for example, Patent Documents 1 to 3.
The “robot system” disclosed in Patent Document 1 has a camera built in a hand, and acquires a wider range of images by rotating the hand by shifting the rotation axis of the hand and the axis of the camera.
Patent Document 2 “Laser Sensor Control Device and Laser Sensor Control Method” adds a laser sensor to a hand and uses this to acquire a three-dimensional image.
Patent Document 3 “Manipulator Control Device and Operation Support Device” improves work efficiency by photographing a work place with a plurality of movable cameras when a robot is remotely operated.

JP 2011-101915 A JP 2011-62786 A Japanese Patent Laid-Open No. 10-249786

  FIG. 1 is a schematic diagram of the above-described conventional remote control system. In this example, the robot 1 has a robot arm 2 (in this example, an articulated arm) fixed at a predetermined position, and a camera 3 and a tool 4 (in this example, a hand) fixed at the tip of the robot arm 2. . The person 5 (operator) operates the operating device 7 (joystick or the like) while viewing the image (monitor image) of the camera 3 on the monitor 6, and holds the object 8 (cylindrical workpiece in this example) with the hand 4. Carry out operations such as transport, installation.

In such a remote operation system, in order to remotely operate the robot 1 and grip the work 8 with the hand 4, the person 5 remotely operates the position and posture of the hand 4 to bring the hand 4 into the position and posture of the work 8. It is necessary to match. The position has three degrees of freedom of x, y, and z, and the posture has three degrees of freedom around the x axis, the y axis, and the z axis.
Further, when the workpiece 8 is first gripped by the hand 4 and the gripped workpiece 8 is set in the work place, the person 5 needs to match the position and posture of the work place 8.

  However, since the monitor image is two-dimensional information, the positional deviation of the monitor 6 in the vertical and horizontal directions can be easily visually recognized, but as shown in FIG. 2, the positional deviation of the monitor 6 in the depth direction is difficult to visually recognize. Further, it is difficult to visually recognize the deviation of the posture between the hand 4 and the work 8.

  Further, FIG. 3A shows a case where an installation mark 9 is provided on the installation surface of the workpiece 8 and the workpiece 8 is installed at the position of the installation mark 9. In this case, even if the posture of the hand 4 changes with respect to the installation surface of the installation mark 9 as shown in FIG. 3B, there is almost no difference in the appearance of the installation mark 9 on the monitor image. It is not known how much the hand 4 is inclined with respect to the installation surface of the mark 9. In particular, it is difficult to match the position and posture of the hand 4 to a flat and wide work place such as the ground or floor.

  The present invention has been developed to solve the above-described problems. That is, an object of the present invention is to provide a remote control system and method for a robot that can visually match the position and posture of a tool provided on a robot with respect to an object while viewing an image of a camera fixed to the robot. It is to provide.

According to the present invention, a tool for performing an operation on an object, a camera capable of simultaneously photographing the tool and the object during the operation, and the position and orientation of the tool and the camera can be moved in a three-dimensional space. A robot having a robot arm, and a work-side wireless device that transmits an image of the camera and receives an operation command for the robot arm;
A remote operation device comprising: a display device for displaying the image; an operation device for operating a robot arm; and an operation-side wireless device for receiving the image and transmitting an operation command for the robot arm;
A solid marker that is fixed to a position that falls within the field of view of the camera together with the tool and the object during the work, and is set so that the shape on the image changes due to a change in the position or posture of the tool with respect to the object; A remote control system for a robot is provided.

  The three-dimensional marker is a cylinder, a circular pipe, a rectangular column, a rectangular pipe, a quadrangular pyramid, or a triangular pyramid that is fixed to the installation surface of the object and extends in a height direction orthogonal to the installation surface.

  The three-dimensional marker has a visually observable pattern provided at regular intervals from the installation surface of the object.

The remote control device has a storage device for storing a selected image selected from the images,
The display device displays the selected image superimposed on the image being worked on.

The robot has a distance sensor for detecting a distance from the tool to an installation surface of an object,
The display device superimposes and displays the detected distance on the image in operation.

According to the present invention, a tool for performing an operation on an object, a camera capable of simultaneously photographing the tool and the object during the operation, and the position and orientation of the tool and the camera can be moved in a three-dimensional space. A robot having a robot arm, and a work-side wireless device that transmits an image of the camera and receives an operation command for the robot arm;
Preparing a remote operation device having a display device for displaying the image, an operation device for operating a robot arm, and an operation side wireless device for receiving the image and transmitting an operation command to the robot arm,
A solid marker whose shape on the image changes due to a change in the position or orientation of the tool relative to the object is fixed at a position that falls within the field of view of the camera together with the tool and the object during the work,
There is provided a remote operation method for a robot, wherein the remote operation device is operated while visually recognizing a change in the shape of the three-dimensional marker on the image.

A storage device stores a selected image selected from the images,
The display device superimposes and displays the selected image on the image being worked on.

The distance sensor detects the distance from the tool to the installation surface of the object,
The display device superimposes and displays the detected distance on the image being worked.

According to the above-described apparatus and method of the present invention, the three-dimensional marker is fixed to a position within the field of view of the camera together with the tool and the object during the work, and the camera image (monitor) is changed by the change in the position or posture of the tool with respect to the object. The shape on the image changes.
Accordingly, a person (operator) who operates the remote control device operates the remote control device while visually recognizing the change in the shape of the three-dimensional marker, and while viewing the camera image, The posture can be adjusted visually.

It is a schematic diagram of the conventional remote control system. It is a schematic diagram of a conventional monitor image. It is another schematic diagram of a conventional remote control system and a monitor image. 1 is an overall configuration diagram of a remote control system according to the present invention. It is explanatory drawing of the monitor image in process by this invention. It is explanatory drawing in case a solid marker is a rectangular pipe. It is explanatory drawing when a solid marker is a quadrangular pyramid.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, the same code | symbol is attached | subjected to the common part in each figure, and the overlapping description is abbreviate | omitted.

FIG. 4 is an overall configuration diagram of a remote control system according to the present invention.
In this figure, the remote control system of the present invention includes a robot 10, a remote control device 20, and a three-dimensional marker 30.

  The robot 10 includes a tool 12, a camera 14, a robot arm 16, and a work side wireless device 18.

The tool 12 is a hand or a tool for performing work. Here, the work refers to, for example, gripping, transporting, and setting a work in an environment (such as a high-temperature environment, a low-temperature environment, and a contaminated environment) that cannot be approached by humans, and refueling work on a work vehicle.
The camera 14 is a digital still camera or video camera that can simultaneously photograph the tool 12 and the object 8 (for example, a workpiece) during work. The tool 12 and the camera 14 are fixed to the tip of the robot arm 16.
The robot arm 16 is an articulated arm or manipulator that can move the position and posture of the tool 12 and the camera 14 in a three-dimensional space.
The work side wireless device 18 is a transmission / reception device that transmits an image of the camera 14 and receives an operation command for the robot arm 16.

  The remote operation device 20 includes a display device 22, an operation device 24, and an operation side wireless device 26.

The display device 22 is a monitor that displays an image (monitor image) of the camera 14.
The operation device 24 is, for example, a joystick, and operates the robot arm 16.
The operation-side wireless device 26 is a transmission / reception device that receives an image of the camera 14 and transmits an operation command to the robot arm 16.

  The three-dimensional marker 30 is fixed to a position (for example, the installation surface 11 of the object 8) that falls within the field of view of the camera 14 together with the tool 12 and the object 8 during work. The three-dimensional marker 30 is set so that the shape on the image (monitor image) of the camera 14 changes due to a change in the position or orientation of the tool 12 with respect to the object 8.

  In this example, the three-dimensional marker 30 is a cylinder that is fixed to the installation surface 11 of the object 8 and extends in the height direction orthogonal to the installation surface 11. However, the present invention is not limited to a cylinder, and may be a circular pipe, a rectangular column, a rectangular pipe, a quadrangular pyramid, a triangular pyramid, or the like.

The three-dimensional marker 30 has a visually observable pattern 32 provided at a certain interval from the installation surface 11 of the object 8. In this example, the pattern 32 is a ring drawn around a cylinder.
The pattern 32 may be a simple color or a three-dimensional shape.
When the three-dimensional marker 30 is cylindrical (for example, a circular pipe or a rectangular pipe), the pattern 32 (for example, a ring) is preferably drawn not only on the outer peripheral surface but also on the inner peripheral surface. Thereby, even when the outer peripheral surface of the three-dimensional marker 30 is not visible and only the inner peripheral surface looks like a mortar by a perspective method, the posture of the three-dimensional marker 30 can be visually recognized.

In FIG. 4, the robot 10 further includes a distance sensor 19 that detects the distance from the tool 12 to the installation surface 11 of the object 8. The display device 22 superimposes and displays the detected distance on the image (monitor image) being worked on.
In this example, the remote control device 20 further includes a storage device 28 that stores a selected image 27 (see FIG. 5) selected from the image (monitor image) of the camera 14. The display device 22 superimposes and displays the selected image 27 stored on the image of the camera 14 in operation.
The selected image 27 is one or a plurality of images arbitrarily selected from images (monitor images) being worked in advance. For example, it is preferable to store a monitor image as the selection image 27 when alignment is particularly necessary during the work (in the middle of assembly, at the time of completion, etc.).
The selected image 27 may be only a part of the monitor image (for example, the tool 12). The selected image 27 is preferably displayed as a color (for example, white) different from the image being worked on or a translucent image.

  The remote operation method of the present invention prepares the robot 10 and the remote operation device 20 described above. The remote operation method of the present invention further prepares the above-described three-dimensional marker 30, fixes the three-dimensional marker 30 at a position within the field of view of the camera 14 together with the tool 12 and the object 8 during work, and the shape of the three-dimensional marker 30. The remote control device 20 is operated while visually recognizing the change on the image.

In the remote operation method of the present invention, the selected image 27 selected from the images is stored in the storage device 28, and the selected image 27 is superimposed on the image being worked on and displayed on the display device 22.
Further, the distance from the tool 12 to the installation surface 11 of the object 8 is detected by the distance sensor 19, and the detected distance is superimposed on the image being worked on by the display device 22 and displayed.

FIG. 5 is an explanatory diagram of a monitor image (image of the camera 14) during work according to the present invention. In this figure, (A) shows a monitor image during work, and (B) shows how the three-dimensional marker 30 is seen.
As shown in FIG. 5A, the installation mark 9 on the monitor image has almost no difference in appearance even when the tool 12 (in this example, the hand 12) is tilted.
On the other hand, the three-dimensional marker 30 (in this example, a cylinder) and the pattern 32 (in this example, a ring) drawn around the cylinder greatly change depending on the inclination of the hand 12, as shown in FIG. The posture of the hand 12 with respect to the work place is easy to visually recognize.
When the three-dimensional marker 30 is a circular pipe, the pattern 32 (for example, a ring) is preferably drawn not only on the outer peripheral surface but also on the inner peripheral surface. Thus, even when the outer peripheral surface of the circular pipe is not visible and only the inner peripheral surface appears to be a mortar shape by perspective, the posture of the circular pipe (that is, the hand 12) can be visually confirmed.
Moreover, the distance to the installation surface 11 can be easily grasped by superimposing and displaying the distance detected by the distance sensor 19 on the image (monitor image) being worked.
Further, by superimposing and displaying the selected image 27 stored on the monitor image being worked on, the tool 12 can be easily moved to the position of the tool 12 when the assembly is completed, for example, by superimposing the tool 12 on the selected image 27. be able to.

  FIG. 6 is an explanatory diagram when the three-dimensional marker 30 is a rectangular pipe.

As shown in FIG. 6A, in this example, the three-dimensional marker 30 is a rectangular pipe having a square cross section, and a rectangular ring pattern 32 is drawn around the rectangular pipe at intervals corresponding to one side of the square. .
The height of the three-dimensional marker 30 is three times the length of one side of the square in this example, but the present invention is not limited to this and may be short or long. In this example, two rectangular ring patterns 32 are provided in the middle of the height direction, but may also be provided at the upper and lower ends.
When the three-dimensional marker 30 is a rectangular pipe, the pattern 32 (for example, a ring) is preferably drawn not only on the outer peripheral surface but also on the inner peripheral surface. As a result, even when the outer peripheral surface of the rectangular pipe is not visible and only the inner peripheral surface looks like a mortar shape by perspective, the posture of the rectangular pipe (that is, the hand 12) can be visually recognized.

  FIG. 6B shows the three-dimensional marker 30 on the monitor image when the three-dimensional marker 30 and the hand 12 face each other. From this monitor image, it can be seen that the three-dimensional marker 30 is almost directly facing the hand 12.

  FIG. 6C shows the three-dimensional marker 30 on the monitor image when the posture of the hand 12 is shifted around the y-axis (see FIG. 4) from (B). From this monitor image, the deviation from (B) is known from the positions of the left and right side surfaces b and the edge a. Also, the deviation from (B) can be seen from the ratio of the c part and the d part.

  FIG. 6D shows the three-dimensional marker 30 on the monitor image when the posture of the hand 12 is shifted around the x-axis (see FIG. 4) from (B). From this monitor image, a difference from (B) is found by the difference between the width e and the width f. The same applies to FIG.

  FIG. 6F shows the three-dimensional marker 30 on the monitor image when the posture of the hand 12 is deviated around the z-axis (see FIG. 4) from (B). From this monitor image, the deviation from (B) can be seen from the inclination with the edge of the monitor.

  FIG. 7 is an explanatory diagram when the three-dimensional marker 30 is a quadrangular pyramid.

As shown in FIG. 7A, in this example, the three-dimensional marker 30 is a quadrangular pyramid with a bottom surface (pyramid shape), and a plurality of circular patterns 32 (in this example) are formed on each side of the square near the bottom surface. 5). A sphere 33 is fixed to the apex of the quadrangular pyramid.
The height of the three-dimensional marker 30 is about twice the length of one side of the square in this example, but the present invention is not limited to this and may be short or long.

  FIG. 7B shows the three-dimensional marker 30 on the monitor image when the three-dimensional marker 30 and the hand 12 face each other. From this monitor image, it can be seen that the three-dimensional marker 30 is almost directly facing the hand 12 from the position of the sphere 33 at the vertex.

  7C and 7D show the three-dimensional marker 30 on the monitor image when the posture of the hand 12 is deviated around the y-axis (see FIG. 4) from (B). The shift of the hand 12 around the y-axis is known from the position of the sphere 33 in the monitor image.

  FIG. 7E shows the three-dimensional marker 30 on the monitor image when the posture of the hand 12 is shifted around the x axis (see FIG. 4) from (B). The shift of the hand 12 around the x-axis is known from the position of the sphere 33 in the monitor image.

  FIG. 7F shows the three-dimensional marker 30 on the monitor image when the posture of the hand 12 is shifted around the z-axis (see FIG. 4) from (B). From this monitor image, the deviation from (B) can be seen from the inclination with the edge of the monitor.

According to the above-described apparatus and method of the present invention, the three-dimensional marker 30 is fixed at a position where it enters the field of view of the camera 14 together with the tool 12 and the object 8 during work, and the position or posture of the tool 12 with respect to the object 8 is determined. The shape on the image (monitor image) of the camera 14 changes due to the change.
Accordingly, the person 5 (operator) who operates the remote operation device 20 operates the remote operation device 20 while visually recognizing the change in the shape of the three-dimensional marker 30 and looks at the image (monitor image) of the camera 14. The position and posture of the tool 12 can be visually matched with the object 8.

  In addition, this invention is not limited to embodiment mentioned above, is shown by description of a claim, and also includes all the changes within the meaning and range equivalent to description of a claim.

1 robot, 2 robot arm (articulated arm), 3 camera,
4 tools (hand), 5 people, 6 monitors,
7 Operation device (joystick), 8 Object (work),
9 Installation mark, 10 Robot, 11 Installation surface,
12 tools (hand or tool), 14 cameras,
16 robot arm, 18 work side wireless device, 19 distance sensor,
20 remote control device, 22 display device (monitor),
24 operation device (joystick), 26 operation side wireless device,
27 selected images, 28 storage devices, 30 solid markers,
32 patterns, 33 spheres

Claims (8)

A tool for performing an operation on an object, a camera capable of simultaneously photographing the tool and the object during the operation, a robot arm capable of moving the position and posture of the tool and the camera in a three-dimensional space, and the camera A robot having a work side wireless device that transmits an image of and receives an operation command for the robot arm;
A remote operation device comprising: a display device for displaying the image; an operation device for operating a robot arm; and an operation-side wireless device for receiving the image and transmitting an operation command for the robot arm;
A solid marker that is fixed to a position that falls within the field of view of the camera together with the tool and the object during the work, and is set so that the shape on the image changes due to a change in the position or posture of the tool with respect to the object; A remote control system for a robot characterized by comprising:   The three-dimensional marker is a cylinder, a circular pipe, a rectangular column, a rectangular pipe, a quadrangular pyramid, or a triangular pyramid that is fixed to an installation surface of an object and extends in a height direction orthogonal to the installation surface. The robot remote control system according to claim 1.   2. The remote control system for a robot according to claim 1, wherein the three-dimensional marker has a visually observable pattern provided at a predetermined interval from an installation surface of the object. The remote control device has a storage device for storing a selected image selected from the images,
The robot remote control system according to claim 1, wherein the display device displays the selected image superimposed on the image being worked. The robot has a distance sensor for detecting a distance from the tool to an installation surface of an object,
The robot remote control system according to claim 1, wherein the display device displays the detected distance superimposed on the image during work. A tool for performing an operation on an object, a camera capable of simultaneously photographing the tool and the object during the operation, a robot arm capable of moving the position and posture of the tool and the camera in a three-dimensional space, and the camera A robot having a work side wireless device that transmits an image of and receives an operation command for the robot arm;
Preparing a remote operation device having a display device for displaying the image, an operation device for operating a robot arm, and an operation side wireless device for receiving the image and transmitting an operation command to the robot arm,
A solid marker whose shape on the image changes due to a change in the position or orientation of the tool relative to the object is fixed at a position that falls within the field of view of the camera together with the tool and the object during the work,
A remote control method for a robot, wherein the remote control device is operated while visually recognizing a change of the shape of the three-dimensional marker on the image. A storage device stores a selected image selected from the images,
The robot remote operation method according to claim 6, wherein the display device superimposes and displays the selected image on the image being worked. The distance sensor detects the distance from the tool to the installation surface of the object,
7. The robot remote control method according to claim 6, wherein the display device displays the detected distance superimposed on the image during work.

Images