JP2003311670A - Positioning control method of robot arm and robot equipment - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To control the position attitude of a robot arm to a work with higher precision in robot equipment for performing a visual feedback control for controlling the position attitude of the robot arm so that the image of the work photographed by a camera reaches the same state with a reference image in real operation, by storing a work photographed in advance as the reference image with the camera fixed on the robot arm. <P>SOLUTION: The equipment fixes a laser displacement sensor 18 to the robot arm 3 in addition to the CCD camera 16, measures the distance between the sensor and the work 2 with the laser displacement sensor 18, and stores the distance as reference distance information, and then, in real operation, controls the robot arm 3 with feedback so that the photographed image with the CCD camera 16 reaches the same state with the reference image, and the distance information measured with the laser displacement sensor 18 becomes same with the reference distance information. Thereby, the distance between the robot arm 3 and the work 2 can be controlled in high precision. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a robot arm position control method and a robot apparatus for feedback-controlling the position and orientation of the robot arm on the basis of data of an image picked up by a visual sensor provided on the robot arm. Is.

[0002]

For example, in the control technology of an assembling robot, when a work is gripped by a robot arm, the work is photographed by a camera attached to the robot arm to control the position and orientation of the robot arm. There is a technology of such visual feedback control (so-called visual servo). In teaching, for example, the robot arm is moved to a position immediately before gripping the work, the work is photographed by the CCD camera attached to the robot arm at that position, and the image is stored in advance as a reference image. At times, the work is always photographed by the CCD camera, and the photographed image and the reference image are compared, and the position and orientation of the robot arm are controlled so that the error between them is zero, that is, the appearance is the same as when teaching. It is a technique to do. According to this, the robot, CC
Installation position parameters of each component such as D camera and C
The position and orientation of the robot arm can be accurately and quickly controlled without strict calibration, that is, internal parameters for converting information from a CD camera into a robot control amount are obtained in advance. Is.

If such a visual feedback control technique is adopted, the position control when the hand grips the work should be accurately performed. However, if it is adopted in the actual assembling work, it will be as follows. The inventors' research has revealed that a problem occurs.

For example, this occurs when the positioning accuracy is required in the work of gripping an object with a robot hand or assembling the gripped object with another object. Due to the nature of image capturing, there is a limit to improvement in accuracy due to the relationship between the number of pixels and resolution. That is, assuming that the optical axis direction of the camera is the Z axis and the imaged screen is the XY plane, the visual servo causes (x, y) in the XY plane obtained from the two-dimensional image data imaged by the camera and around the Z axis. High-precision control can be performed on rotation (Rz), but there is a limit to accuracy on rotation (Rx, Ry) around the X-axis or Y-axis or in the height direction (Z-axis direction). .

As an example, as shown in FIG. 7, CC
When the object P is photographed by the D camera M, X
When the camera M changes from a position indicated by a solid line to a position indicated by a chain double-dashed line (a position inclined by a minute angle in the Rx direction) with respect to a minute change in the pitch angle that is a rotation angle about the axis (Rx). As in the photographed image, the change in the number of CCD pixels is shown in Fig. 8.
If the degree of change from (a) to (b) is small,
Since there is a quantization error, there is a limit to the range that can be captured as a change on the image.

Further, as shown in FIG. 9, for minute changes in the height (Z-axis) direction, the state shown in FIG.
Since it is the state change of (b), it is still difficult to catch it as a change on the image, which causes a problem in accuracy.

The present invention is intended to solve the problem that occurs in such visual feedback control, and an object thereof is a robot arm position control method capable of controlling the position and orientation of the robot arm with respect to an object with higher accuracy. And to provide a robot apparatus.

[0008]

In order to achieve the above object, the invention of claim 1 uses a visual sensor attached to a robot arm to photograph an object such as a work in advance and store it as a reference image. The robot arm is configured to perform visual feedback control for controlling the position and orientation of the robot arm so that the image of the object captured by the visual sensor is in the same state as the reference image. An auxiliary sensor is provided in the visual sensor, the relationship information between the auxiliary sensor and the object is acquired by the auxiliary sensor in the position and orientation of the robot arm when the visual sensor captures the reference image, and the reference information is acquired in advance. As the visual image so that the image of the object captured by the visual sensor is in the same state as the reference image. In addition to back control, feedback control is performed to control the position and orientation of the robot arm so that the relationship information between the auxiliary sensor and the object acquired by the auxiliary sensor is the same as the reference relationship information. It is characterized by.

According to the means of claim 1, when an angle sensor or a distance sensor is selected as the auxiliary sensor,
The angle sensor or the distance sensor measures the relationship between the angle sensor or the distance sensor and the object, that is, the tilt angle of the object with respect to the angle sensor or the distance between the distance sensor and the object, and the reference relationship is used. It is stored as information, and during actual work, the captured image by the visual sensor is in the same state as the reference image, and the relationship information acquired by the angle sensor or the distance sensor is in the same state as the reference relationship information. By performing feedback control of the robot arm, the position and orientation of the robot arm can be accurately controlled.

Further, according to the invention of claim 2, an object such as a work is photographed in advance by a visual sensor attached to the robot arm and the image is stored as a reference image, and the object photographed by the visual sensor is photographed. A robot apparatus for performing visual feedback control for controlling the position and orientation of the robot arm so that the image is in the same state as the reference image, the robot device being attached to the robot arm and acquiring relationship information with the object. An auxiliary sensor to
Storage means for storing, as reference relationship information, relationship information between the auxiliary sensor and the object acquired by the auxiliary sensor in the position and orientation of the robot arm when the visual sensor captured the reference image, and the visual sensor. In addition to performing the visual feedback control so that the image of the object captured by the same state as the reference image, the relationship information between the auxiliary sensor and the object acquired by the auxiliary sensor is the reference relationship. The control means for performing feedback control for controlling the position and orientation of the robot arm is provided so as to be the same as the information. The same effect as that of the first aspect can be obtained by the means of the second aspect.

[0011]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment in which the present invention is applied to a vertical articulated robot apparatus for assembly will be described with reference to FIGS.
Will be described with reference to. FIG. 1 schematically shows the configuration of an assembly system using the robot apparatus 1. In this assembly system, a robot arm 3 grips a work 2 as an object supplied on a work table (not shown) at an arbitrary position and posture, and the work 2 is conveyed to an assembly station by a conveyor (not shown). It is to be attached to the attached body 4. Although not shown, a plurality of feature points (specific edges or holes, affixed markers, etc.) are set on the work 2 for visual recognition described later.

The robot arm 3 constitutes a robot body. This robot arm 3 is composed of a base portion 5, a shoulder portion 6, a lower arm portion 7, an intermediate arm portion 8, an upper arm portion 9, a wrist portion 10, a flange portion 11 and the like, and the connection configuration of these respective portions is as follows. ing. That is, the base portion 5 is fixed to the installation floor surface, and the shoulder portion 6 rotates (turns) around the vertical axis on the base portion 5.
A pair of lower arm portions 7 are connected to the left and right ends of the shoulder portion 6 so as to be vertically rotatable about a horizontal axis. Between the tips of the pair of lower arm portions 7,
A front end portion of the intermediate arm portion 8 is connected so as to be vertically rotatable about a horizontal axis.

A rear end of an upper arm 9 is coaxially rotatably connected to a front end of the intermediate arm 8, and a wrist portion 10 is provided at a left and right end of the upper arm 9.
Is rotatably connected about a horizontal axis, and its wrist portion 1
A flange portion 11 is coaxially rotatably connected to the front end portion of 0. Then, the flange 12 has a hand 12
(Shown in FIG. 4) is removably attached.

Each part 6 to 11 of the above-mentioned robot arm 1
Are driven by servomotors (not shown) with encoders, and these servomotors are controlled by a robot controller 14 forming a part of the controller 13. .

The controller 13 is a robot controller 1
4, a visual recognition device 15, CC as a visual sensor
A D camera 16, a sensor output processing device 17, and a laser displacement sensor 18 as an auxiliary sensor are provided.
The visual recognition device 15 and the CCD camera 16 are for visual feedback control, and the CCD camera 16 is attached to, for example, the wrist unit 10 among the joints of the robot arm 3.

The visual recognition device 15 is mainly composed of a CPU, and is a ROM or RAM storing a program for position / orientation control.
And an interface circuit for inputting an image signal from the CCD camera 16, an image processing circuit, an image memory, and the like. This visual recognition device 15
A control operation signal (position correction amount) is given to the robot controller 14 based on image processing. In this case, in the visual recognition device 15, the registration data of the reference image obtained by the teaching work executed in advance is stored in the image memory. This teaching work is performed by manually operating the robot arm 3 by using a teaching pendant (not shown), for example.
Is moved to a teaching target position above the work 2 (for example, the position and posture immediately before gripping the work 2 by the hand 12 as shown in FIG. 4), the work 2 is photographed by the CCD camera 16 at this position, and the photographed image is obtained. Is registered (stored) in the visual recognition device 15 as a reference image.

FIG. 3 shows registration data of a reference image obtained by photographing the work 2 with the CCD camera 16. The registration data is set with the positional relationship between the reference point and the characteristic point in the field of view of the CCD camera 16 as the characteristic amount. That is, an arbitrary reference point P0 is set on the visual field, and the work 2
The characteristic points on the shape of are defined as characteristic points P1 to P5. Then, on the photographed image, vectors V1 to V5 formed by connecting the reference point P0 and the respective characteristic points P1 to P5
And the data of the angles φ1 to φ5 formed by the adjacent vectors V1 to V5 are calculated, and this feature amount data is used as the registration data of the reference image.

At the time of actual work, the control device 13 as a control means takes an image of the work 2 by the CCD camera 16 at an arbitrary position and posture, processes the taken image data by the visual recognition device 15, and presently processes it. In the same manner, the feature amount (data of V1 to V5, angles φ1 to φ5) on the image is obtained as current data (feedback signal), and the error (deviation) is compared with the registered data of the reference image. The position correction amount is calculated so as to decrease (for example, 0), that is, the image by the CCD camera 16 looks the same as at the time of teaching, and is output to the robot controller 14 as a control operation signal.

On the other hand, the sensor output processing device 17 and the laser displacement sensor 18 are for distance feedback control in this embodiment, and the laser displacement sensor 18 is the same arm as the CCD camera 16 in this embodiment, that is, the wrist unit 1.
It is attached to 0. And the laser displacement sensor 1
Reference numeral 8 measures the relational information with the work 2, here, the distance between the laser displacement sensor 18 and the work 2.

The sensor output processing device 17 is composed mainly of a CPU, a ROM storing a program for position / orientation control,
The configuration includes a RAM, an interface circuit for inputting a measurement signal from the laser displacement sensor 18, a processing circuit for the measurement signal, and the like. The sensor output processing 17 gives a control operation signal to the robot controller 14 based on the signal processing. In this case, the sensor output processing device 17 stores reference distance information as reference relationship information in the RAM in advance. Then, the sensor output processing device 17 compares the distance information (feedback information) measured by the laser displacement sensor 18 with the reference distance information to reduce the error (deviation) thereof (for example, 0), that is, the laser. The position correction amount is calculated and output to the robot controller 14 so that the distance between the displacement sensor 18 and the work 2 becomes the same as during the teaching work.

The above reference distance information is acquired in accordance with the acquisition of the reference image in the teaching work described above. The operation contents when the reference image and the distance information are acquired are shown in the flowchart of FIG. That is, first, the robot arm 3 is moved to the teaching target position shown in FIG. 4 using the teaching pendant (step A1). Then, the work 2 is photographed by the CCD camera 16 at the teaching target position,
The image data is acquired (step A2). Then, the laser displacement sensor 18 measures the distance between the displacement sensor 18 and the workpiece 2, and the sensor output processing device 17 processes the measurement signal output from the laser displacement sensor 18 to obtain the distance data ( Step A3). And
The acquired image data and distance data are stored as a reference image and reference distance information in the visual recognition device 15 and the sensor output processing device 17 serving as storage means (step A4), and the teaching ends.

The robot controller 14 uses the CP
ROM storing RA, mainly U, RA
The servo control unit is provided with M and outputs a control command to the robot arm 3 as a drive signal. This robot controller 14 (servo controller)
Together with the visual recognition device 15 and the sensor output processing device 17 constitute a velocity loop control device as shown in FIG. Note that this speed loop control device is based on software. In this speed loop control device, the CCD camera 16 and the laser displacement sensor 1 are used.
The image data and the measurement data output from 8 are processed by the visual recognition device 15 and the sensor output processing device 17, respectively, and input to the adder 19. On the other hand, adder 1
Reference image and reference distance information are input to 9, and a difference between the reference image and reference distance information and the image information and the measured distance information input from the visual recognition device 15 and the sensor output processing device 17 is obtained as a deviation. The deviation obtained by the adder 19 is input to the amplifier 20, and the amplifier 20
Outputs a speed signal corresponding to the deviation given from the adder 19 to the robot controller 14. The robot controller 14 controls the drive motors of the arms 6 to 11 so that the speed becomes a speed according to the speed signal given from the amplifier 20. By repeating such speed control, the robot arm 3 reaches the teaching target position.

FIG. 6 shows the control contents of the work 2 during the actual work. The actual work is started by photographing the work 2 by the CCD camera 16 (step B1), and then the distance from the work 2 is measured by the laser displacement sensor 18 (step B2). Then, the image captured by the CCD camera 16 is input to the visual recognition device 15, and the visual recognition device 15 acquires the image information I (t) from the imaging data input from the CCD camera 16 and the image information I
(T) is the reference image I (d) stored in the image memory.
And the deviation ΔI (t) is calculated (step A
3). Further, the sensor output processing device 17 processes the detection output of the laser displacement sensor 18 to obtain the distance information S (t), compares the distance information S (t) with the reference distance information S (d), and The deviation ΔS (t) is obtained (step A4).

The visual recognition device 15 and the sensor output processing device 17 inform the robot controller 14 of the deviation ΔI (t) of the image and the deviation ΔS of the distance.
Give (t). The robot controller 14 determines the deviation Δ
It is determined whether the absolute values of I (t) and ΔS (t) are less than or equal to the threshold value, and if the absolute values are greater than the threshold value (step B
5, the robot controller 14 controls the robot arm 3 with the control operation amount (speed) according to the deviations ΔI (t) and ΔS (t) (step B6).

Then, the work 2 is photographed by the CCD camera 16 again, and the distance to the work 2 is measured by the laser displacement sensor 18, and the image information I (t) and the distance information S (t) are used as the reference image I. (D) and the reference distance information S (d) are compared, and the operation of controlling the robot arm 3 at a speed corresponding to the deviations ΔI (t) and ΔS (d) is repeatedly performed. When the absolute values of the deviations ΔI (t) and ΔS (t) become equal to or less than the threshold value by repeating such operations (“YES” in step B5,
That is, when the robot arm 3 reaches the teaching target position with high accuracy, the operation proceeds to the next work, that is, the operation of gripping the work 2 in this embodiment (step B8).

As described above, according to this embodiment, it is possible to accurately control the position in the height (Z-axis) direction, which is difficult to control with high precision only by image information. And this control
The distance information S (t) by the laser displacement sensor 18 is compared with the reference distance information S (d), and the robot arm is arranged so that the distance information S (t) matches (including substantially matches) the reference distance information S (d). Since it is performed by operating 3, the troublesome calibration such as converting the distance measured by the laser displacement sensor 18 into robot coordinates is necessary.
Teaching becomes very simple. As an auxiliary sensor,
An angle sensor may be used, and the inclination angle of the workpiece 2 with respect to the angle sensor may be measured to accurately control the angle around the X axis or the Y axis.

Further, by using the distance sensor and the angle sensor, the position of the robot arm in the Z-axis direction and around the X-axis, Y
The angle around the axis may be controlled. In this case, the image data I (t), Id and the sensor data S
An example of a method for obtaining the arm control amount from (t), Sd will be briefly described. For example, when the translational velocities tx, ty in the XY directions of the arm hand and the rotational velocity ωz about the Z axis are obtained using image data, the well-known optical flow equation is used to obtain the following equation (1). To be

[0028]

[Equation 1]

Further, I (t) and Id are vectors representing image-specific coordinate values detected from the image at the current position and the target position, respectively. For example, the center position of the hole of the work is used as the image feature, and the center positions of the four holes observed from the image at the current position are (x1 (t) y1
(T)), (x2 (t) y2 (t)), (x3 (t) y
3 (t)) and (x4 (t) y4 (t)), and similarly, image feature amounts observed at the target position are (x1y1), (x2
y2), (x3y3), (x4y4),

[Equation 2]

[0030]

[Equation 3]

[0031]

[Equation 4]

Further, the rotational speeds ωx and ωy about the XY axes are calculated using a laser sensor or the like. For example, the inclination angle of the work surface is calculated by the slit light of the laser, and the measured values of the laser sensor corresponding to the inclination angle of the work surface with respect to the X-axis and Y-axis of the laser sensor are respectively S.
If x and Sy,

[Equation 5]

Therefore, the sensor data S (t) and Sd are S
(T) = (Sx (t) Sy (t)), Sd = (SxdS
ydSzd).

After all, the arm control amount R (t) is calculated from the above equations (1), (5) and (6) by R (t) = (txtytz).
ωxωyωz) ^T.

As described above, according to the present invention, the desired control can be performed without explicitly obtaining the parameters for converting the feature amount from the image and the output value from the sensor into the physical momentum for arm control. Can be realized. The above speed calculation method is an example, and there is also a method of calculating the rotation speed around the XY axes from the image feature amount, for example.

The present invention is not limited to the embodiments described above and shown in the drawings, but the following expansions and modifications are possible. The reference distance information is the robot controller 14
May be stored in the memory. The auxiliary sensor is
It is not limited to the distance sensor and the angle sensor. The form of the robot arm is not limited to the vertical articulated type. The invention is not limited to the case of gripping the work 2, but can be widely applied to the case of assembling to the body 4 to be assembled or the case of passing the object to the other party, and the other party may be a work table on which the object is placed. It may be a pallet or the like.

[Brief description of drawings]

FIG. 1 is an overall schematic configuration diagram showing an embodiment of the present invention.

FIG. 2 is a block diagram of a speed loop control device.

FIG. 3 is an explanatory diagram of a storage form of a captured image of a work.

FIG. 4 is an explanatory diagram of teaching work.

FIG. 5 is a flowchart showing the contents at the time of teaching.

FIG. 6 is a flowchart showing the control contents during actual work.

FIG. 7 is a perspective view for explaining a conventional problem part 1

FIG. 8 is a diagram of a captured image in FIG.

FIG. 9 is a perspective view for explaining a conventional problem part 2

FIG. 10 is a diagram of a captured image in FIG.

[Explanation of symbols]

In the figure, 1 is a robot device, 2 is a work (object), 3 is a robot arm, 12 is a hand, 13 is a control device (control means), 14 is a robot controller, 15 is a visual recognition device (storage means), 16 Is a CCD camera (visual sensor), 17 is a sensor output processing device (storage means), and 18 is a laser displacement sensor (auxiliary sensor).

Continued front page    (72) Inventor Takashi Naito             Aichi Prefecture Nagachite Town Aichi District             Ground 1 Toyota Central Research Institute Co., Ltd. (72) Inventor Hideki Nomura             Aichi Prefecture Nagachite Town Aichi District             Ground 1 Toyota Central Research Institute Co., Ltd. F term (reference) 3C007 AS06 BS12 BT08 CT05 CV08                       CW08 KS03 KS05 KS36 KT01                       KT05 KT09 KV11 KX06 LS05                       LT06 LT08 LT12 MT01

Claims (2)

[Claims] 1. An image of an object such as a work is previously photographed by a visual sensor attached to a robot arm and stored as a reference image, and the image of the object photographed by the visual sensor is the same as the reference image. A robot device for performing visual feedback control for controlling the position and orientation of the robot arm so as to be in a state, wherein the robot arm is provided with an auxiliary sensor, and the visual sensor captures the reference image. With the position and orientation of the robot arm, the relationship information between the auxiliary sensor and the object is acquired by the auxiliary sensor and stored as reference relationship information in advance, and the image of the object captured by the visual sensor is the reference image. In addition to performing the visual feedback control so that the same state as A method for controlling a position of a robot arm, wherein feedback control for controlling a position and orientation of the robot arm is performed so that relationship information between the auxiliary sensor and the object is the same as the reference relationship information. 2. An image of an object such as a work is previously photographed by a visual sensor attached to a robot arm and stored as a reference image, and the image of the object photographed by the visual sensor is the same as the reference image. A robot device adapted to perform visual feedback control for controlling the position and orientation of the robot arm so as to be in a state, and an auxiliary sensor attached to the robot arm for acquiring relational information with the object, Storage means for storing relationship information between the auxiliary sensor and the object acquired by the auxiliary sensor in the position and orientation of the robot arm when the sensor took the reference image as reference relationship information, and imaged by the visual sensor The visual feedback control is performed so that the image of the target object is in the same state as the reference image. In addition to that, a control unit that performs feedback control for controlling the position and orientation of the robot arm so that the relationship information between the auxiliary sensor and the object acquired by the auxiliary sensor is the same as the reference relationship information. A robot apparatus characterized by being provided.