JPH0847881A - Method of remotely controlling robot - Google Patents

Abstract

PURPOSE:To provide a method of remotely controlling a robot, which can effectively remote-control the robot. CONSTITUTION:Cameras 8a, 8b are attached to the left and right parts of a grip 5 of a robot 1, for picking up images of a surface to be worked, and thus picked-up images are delivered to a computer 6, and an arbitrary shape is determined as a target 14a from one of the images. Further, a loop-like edge of the target 14a which is continuous is extracted, and the shape thereof is recognized, and the trigonometrical survey of the grip 5 of the positions and the postures of the robot 1 and the target 15 is made from an epipolar line at an arbitrary point on the loop-like shape, and coordinates of another point with respect to the arbitrary point, and the grip 5 of the robot 1 is controlled in accordance with values obtained by the trigonometric survey.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention triangulates the positions and orientations of a robot's hand and a target on a work surface.
The present invention relates to a remote control method for a robot capable of operating a hand by accurately adjusting its position and orientation with respect to a target.

[0002]

2. Description of the Related Art Conventionally, in a robot having six degrees of freedom, such as a manipulator or a robot, for example, a plurality of arms are connected by joints, and the joints of the arms are swung and raised so that the hand of the tip of the arm is moved. , 6 degrees of freedom (x,
Various operations can be performed by freely moving in the (y, z, φ, θ, ψ) directions.

[0003]

The master / slave system using a joystick is the mainstream for remote control of this robot. In this method, the operator operates the joystick while looking at the image by attaching the camera near his or her hand, and moving the hand to the desired position. There is a problem in that the robot cannot follow such movements at such a high speed, and that the posture control and position control of the hands of the robot impose a considerable burden on the operator.

There is also a method of preliminarily storing a moving procedure for operation in the robot in a computer by teaching and moving the robot as taught, but if the operation target is different, teaching must be performed each time. There are problems that must be addressed.

Recently, the work task of this type of robot is
For example, ORU replacement at a space station, various inspections of a nuclear reactor, etc. are required to be used under the condition that a worker cannot directly operate in the working environment. However, it cannot be expected that good work tax will be obtained.

SUMMARY OF THE INVENTION It is therefore an object of the present invention to solve the above problems and provide a remote control method for a robot which enables the remote control robot to perform an effective operation.

[0007]

In order to achieve the above object, the present invention is to attach cameras to the left and right of the hand of a robot, photograph the surface to be worked by the left and right cameras, and capture the image into a computer. Determine an arbitrary shape from one of the images as a target, recognize the shape by extracting the edges of the target that are connected in a loop, and then recognize the corresponding loop shape of the target in the image captured by the other camera. Then, the position / orientation of the robot's hand and target is triangulated from the epipolar line of any point of one loop shape and the coordinates of the other point corresponding to that arbitrary point, and the robot is based on the triangulation value. It controls your minions.

[0008]

According to the above construction, first, the center of both cameras,
Any point of the target in the image and the point of the actual target are on the same plane called the epipolar plane, and the line of intersection between this epipolar plane and the image plane is the epipolar plane. The target point on the top is on the epipolar line and can be found by triangulation with the actual target point, and stereo correspondence can be made by measuring each left and right point, so the position and orientation of the hand and the target can be determined. The robot can be easily controlled based on this.

[0009]

An embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

First, the outline of the present invention will be described with reference to FIG.

In FIG. 1, reference numeral 1 denotes a robot such as a manipulator and a handling device. For example, as shown in the drawing, a first arm 3 is provided on a machine base 2 so as to be swingable and liftable, and the first arm 3 is A second arm 4 is provided so as to be bendable, and a hand 5 for handling is provided on the second arm 4 so as to be rotatable and bendable. The hand 5 is three-dimensional (x, y, z). ) Direction, and can rotate about each axis in the (φ, θ, ψ) direction, and can move in the 6 degrees of freedom (x, y, z, φ, θ, ψ) direction.

Although the details of the control of the arms 3 and 4 and the hand 5 of the robot 1 are omitted, an angle detection sensor is provided in each of the arms 3 and 4 and the hand 5, and each arm 3 and 4.
4 and the coordinates of the hand 5 can be recognized, and a command from the computer 6 can be moved to a commanded position by controlling a driving device (not shown) that drives the robot 1 by the communication signal 7. At the same time, the hand 5 can hold and move any article or device.

On the left and right of the hand 5 of the robot 1, CC
Cameras 8a and 8b such as a D camera are attached, and the front of the hand 5 can be picked up by both cameras 8a and 8b.

The images of the cameras 8a and 8b can be taken into the computer 6 by the communication signal 7.

The computer 6 comprises a main body 9, a display 10 and a keyboard 11, and a mouse 12 is connected to the main body 9.

Now, it is assumed that the hand 5 of the robot 1 handles the article 13 and that the hand 5 holds the projection 14 of the article 13 and grips the article 13.

In the present invention, the cameras 8a and 8b photograph the article 13, the image is taken into the computer 6, one of the images of the cameras 8a and 8b is displayed on the display 10, and the pointer 15 of the mouse 12 is displayed. The position of the protrusion 14 of the article 13 with respect to the hand 5 can be triangulated by aligning with the position of the protrusion 14a of the article 13a displayed on the display 10 and clicking the protrusion 14a as a target at that position. ing.

This processing flow will be described below with reference to FIG.

When the control is started, the left and right images are first captured, then the target designation in the left image (or the right image may be performed) is performed by the operator, and then the designated target planes are connected in a loop. Extract the edges,
Similarly, the target plane in the right image is extracted by stereo correspondence.

By extracting the left and right target surfaces, the difference between the shapes of the two targets is the displacement when stereoscopically viewed by both cameras, and at the same time, the displacement is the coordinate in the real space by trigonometry (triangulation principle). The position / orientation information can be determined.

Therefore, corresponding points in the right image of an arbitrary point on the target surface of the left image are detected. This point is 3
Detect more than points. Similarly, the three points on the target surface of the right image are also searched, the epipolar line of each point is searched, the other point and the point of the actual target surface are stereo-correlated, and the coordinates of each point of the target are calculated. To decide.

Next, the position / orientation of the target with respect to the hand of the robot is calculated, a command is given to grip the target with the hand, and the control ends.

Next, each flow will be described.

A. Acquisition of left and right images Pixel data of left and right images captured by a CCD camera or the like is stored in a computer, and the left image is displayed on the display.

B. Designation of Target in Left Image by Operator First, when the operator selects the protrusion 14a of the projected article 13a as a target, the operator moves the pointer 15 on the screen with the mouse 12 and moves the pointer 15 to the edge of the target. Instead of pointing at, point at the approximate center of the surface of the target and click on that position with the mouse 12.

The pointer of the mouse can click any point (dot) in the image, and the periphery of the clicked point is searched for an edge connected in a loop having the pointed point inside.

(1) Search for one edge: First, as shown in FIG. 3, the image shows the edge EL of the target TL, and the mouse 1 is placed at the approximate center of the target surface ML.
When the pointer 2 is clicked, the edge EL is searched so as to gradually widen the circle around the clip point p pointed at the beginning as shown by the arrow A in the figure.

(2) Tracking of edges connected in a loop: As shown in FIG. 4, the edges E1, E6, Ex other than those forming the loop are included inside and outside the edge EL. It is necessary to distinguish it from the looped edge EL to be detected.

Therefore, from the edge point of the edge detected first, the edge point of another edge is searched for within the range circled in the figure. Then, from the other end point of the newly detected edge, the end point of another edge within a certain range is searched for.

In this way, as shown in the figure, when the edge E1 is searched for the end point of the edge as indicated by the dotted arrow P1, the end point cannot be detected in the circle C1, so it is not judged as a looped edge, Next, another edge E2 is searched. At this time, the edge E2 which started the tracking is searched as shown by the dotted arrow 2 in the figure, and other consecutive edges E2 to E5 are searched by circles C3 to C6. At the edge E5, at the circle C7, the outer edges E6 not related to the loop are connected, but because the circle C6 cannot search for other edges, the circle C7
Return to and search for edge E7. This edge E
Delete 6 so that it is not selected twice. Also this circle C7
Although there is an edge Ex in the vicinity of, the search is not performed because it is out of the range of the circle C7. Circle C in this way
After searching up to 9 and detecting the edge E2 at which the tracking has started, the tracked edge EL can be detected as the shape of the target in a loop.

If the edges to be searched for are exhausted without detecting the looped edges, this process has failed.

(3) Judgment of loop having pointed point inside: In the detected loop, as shown in FIG.
FIG. 5 shows that the point p is pointed inside.
As shown in (b), some are not. In order to determine this, the vector V formed by the end points of the respective edges forming the loop and the designated point p is obtained, and the angle θ formed by the adjacent vectors V is then determined.
Is calculated, and the sum of the angles θ between all the vectors V is 2π (r
When it is ad), it is determined to be a loop having the point inside.

C. Stereo Correlation As shown in FIG. 6A, a loop forming the target EL detected from the left image is stereo-correlated from the right image shown in FIG. 6B to extract the edge ER of the right image. . All the edges other than the edges EL forming the loop detected in the left image are deleted in advance so as not to perform extra stereo correspondence.

After stereo matching, there is an end point of the edge E5 already associated with the right image with respect to the edge E7 which is not yet associated with the edge ER forming the loop detected in the left image. Round C7
Search for edge E7 within the range. If the direction and length of the detected edge E7 satisfy certain conditions,
Correlate this.

Finally, the deviation σ of the parallax at the end points of the left and right stereo-correlated edges EL and ER is obtained.
At this time, as shown in FIGS. 7A and 7B, a set of edges E7 E7 having a deviation σ of 3σ or more is deleted as a correspondence error.

When the stereo correspondence is not successful, the operator corrects the edge.

D. Detection of Target Position / Posture When detecting the target position / posture, consider the case where the left and right cameras have their optical axes in arbitrary directions as shown in FIG. This is for obtaining the position / orientation viewed from the tool coordinate of the robot of the camera when calibrating the camera which will be described later. However, it is assumed that they are placed almost parallel to the optical axis, without considering any arrangement that is physically impossible to obtain a stereo image, or where the left and right cameras are swapped or the top and bottom are upside down.

(1) Selection of corresponding points: position of target surface /
Three points are detected in the three-dimensional space in order to detect the posture.
Therefore, three corresponding points on the edges of the left and right images that are connected in a loop are selected. Usually, feature points such as corners are used, but in the case of a surface having many convex corners, it is often conceivable that the corner points are misaligned. Therefore, it is assumed that all the excess edges except those associated with the edge of the left image have been deleted.

The epipolar line is calculated from FIG. 9 as follows.

First, the target point is P, the world coordinates thereof are X, Y, Z, and the centers of the left and right lenses are Ol and Or, respectively, and the left image plane 20a and the right image plane 20b are set.
The coordinates of the poses of (x1, y1, zl) and (x
r, yr, zr).

Next, d is a line connecting the left and right lens centers Ol and Or, Pl and Pr are target points of the left and right images 20a and 20b, and a straight line POl and a target P connecting the target P and the left lens center Ol. Assuming a straight line POr connecting the center Or of the right lens, these points P, Ol, Or, Pl,
Pr and the lines d, PO1, POr are in the same epipolar plane EP.

Now, the epipolar line at the point Pr at which the straight line POr and the right image surface (broad surface near infinity including the right screen) 20b intersect is defined as Epi, and the epipolar line Epi, the target p of the left image, and the lens center Ol. Straight line PO connecting
Let xe be the intersection with l. Further, the coordinates at the point Pr are (tr, kr, nl), and the normal direction of the optical axis direction is (k
r) and the direction of the epipolar line Epi are (tr).

Now, considering the point Pr on the right image plane 20b, the direction vector tr of the epihola line is obtained by the vector kr in the optical axis direction and the vector nl in the normal direction of the epipolar plane EP.

The normal vector tr and the vector nl are expressed by the following equations 1 to 4.

[0045]

[Equation 1]

[0046]

[Equation 2]

[0047]

(Equation 3)

[0048]

[Equation 4]

The direction vector tr of the epipolar line can be obtained from the normal vector kr and the vector nl, and the epipolar line Epi can be obtained.

The intersection of the epipolar line and the edge is 2
8A, if the selected arbitrary point is on either the left or right edge of the loop, as shown in FIG. 8B, if the intersection is on the left with respect to the epipolar line Epi, then FIG.
Use to select the left, and for right, select the right of the intersection.

(2) Position detection in three-dimensional space: The positions of three points in the three-dimensional space are calculated at the corresponding points of each of the three detected left and right images.

The position vector P to be obtained can be calculated by the following equations 5-8 using c1 and c2 as coefficients.

[0053]

(Equation 5)

[0054]

(Equation 6)

[0055]

(Equation 7)

[0056]

(Equation 8)

From the above equations 1 to 4, the epipolar line Epi
Since d is known and d is known, c1 and c2 can be calculated, and the position vector P can be calculated.

The position vector of the target to be detected next is specified by the tool coordinate system installed near the hand of the robot. Depending on this tool coordinate system and camera attitude,
This is because ~ 8 becomes complicated. The tool coordinate system used here is set at a position where its Z axis is not perpendicular to the optical axis of the camera.

As shown in FIG. 10 (a), the posture of the target surface has three coordinate points P1, P2, P3 (in the figure, P1, P2, P3 selected by the edge E of the target, where the coordinate system on the image plane 20 is x, y. 2 points P1 and P2 on the left edge and 1 point P3 on the right edge
), The direction in the three-dimensional space corresponding to the two points P1 and P2 on the left edge indicates the y-axis direction (from the point P1 with a small y value in the coordinate system on the image to the point P2 with a large y value). If the direction of the normal vector with respect to the plane coordinate in the y-axis direction on the side of the point P3 on the right edge is the Z-axis direction, the posture of the target surface T can be obtained as shown in FIG. 10 (b).

Further, the origin on the target surface is a line segment of a three-dimensional space formed by a point having a large y value in the coordinate system on the image among the two points P1 and P2 on the left edge and a point P3 on the right edge. Place at midpoint o.

D. Calibration In order for the robot to grip the target, the position / orientation data of the target obtained by stereo image measurement is converted to data viewed from the tool coordinate system and given to the robot.

The relationship between the tool coordinate system and the default hand camera coordinate system is as shown in FIG.

From FIG. 11, the (xc, yc, zc) positions of the hand coordinate system and the tool coordinate system are deviated.

Therefore, six parameters (x, y, z, φ, θ, ψ) representing the position / orientation of the lens center of the camera and the tool coordinate system in the three-dimensional space are determined.

E. Setting of hand camera coordinate system Default conversion from robot tool coordinate system to hand camera coordinate system (default camera coordinate system) TC
Set like.

[0066]

[Equation 9]

From the above, the position / orientation of the target can be obtained by simply operating the mouse and clicking the target with the mouse pointer in the image.
Since the coordinates of the target can be easily recognized, this can be easily handled.

In this remote control, a robot is equipped with a computer in which a program for recognizing a position / orientation and a program for controlling the robot are installed, and only image information is transmitted to a computer on the operator side, and only a target is designated there. By doing so, fully automatic remote control is possible, which is ideal for operation in situations where no one can enter the work environment.

[0069]

In summary, according to the present invention, the left and right cameras provided at the hands of the robot take pictures of the target,
Since stereo correspondence can be performed by triangulating any point of the target in the image and the point of the actual target using the epipolar line, it is possible to obtain the position and orientation of the hand and the target. Based on this, it becomes possible to easily control the robot.

[Brief description of drawings]

FIG. 1 is a schematic view showing an embodiment of the present invention.

FIG. 2 is a diagram illustrating a processing flow of the present invention.

FIG. 3 is a diagram illustrating a method of searching for a loop edge from a click point in the present invention.

FIG. 4 is a diagram illustrating an edge tracking method for recognizing a loop edge from a target edge in the present invention.

FIG. 5 is a diagram illustrating a relationship between a clip point and a target edge in the present invention.

FIG. 6 is a diagram illustrating correspondence between left and right edges in the present invention.

FIG. 7 is a diagram illustrating an example in which left and right edges cannot be associated with each other in the present invention.

FIG. 8 is a diagram illustrating selection of corresponding points of left and right points on an epipolar line in the present invention.

FIG. 9 is an explanatory diagram for triangulating the position / orientation of the target using epipolar lines on the epipolar surface in the present invention.

FIG. 10 is a diagram illustrating a posture of a target surface in the present invention.

FIG. 11 is a diagram showing a relationship between a default camera coordinate system and position measurement in the present invention.

[Explanation of symbols]

 1 Robot 5 Hand 6 Computer 8a, 8b Camera 14a Target E Edge Epi Epipolar line

 ─────────────────────────────────────────────────── ─── Continuation of front page (72) Inventor Seiji Koide 3-15-15 Toyosu, Koto-ku, Tokyo Ishikawajima Harima Heavy Industries Co., Ltd. Toji Technical Center

Claims (1)

[Claims] 1. A camera is attached to the left and right of the hand of a robot, the surface to be worked is photographed by the left and right cameras, the image is taken into a computer, an arbitrary shape is determined as a target from one image, and the target is determined. Recognize the shape by extracting the continuous edge of the loop shape, and then recognize the corresponding target loop shape in the image captured by the other camera, and the epipolar line of any point of the one loop shape and its A remote operation method for a robot characterized by triangulating the position and orientation of the robot's hand and target from the coordinates of the other point corresponding to an arbitrary point, and controlling the robot's hand based on the triangulation value. .