JP2001191285A - Robot system and its usage - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a robot system provided with simplified device constitution for all equipment of an image processing function and to provide its usage. SOLUTION: The robot system is provided with a photographing means 20 mounted in a proper position in a robot body 10, an image display means 30, an instructing means 40, and a robot controller 50 with an image processing function. The image display means 30 and the instructing means 40 are integrated together, and an image taken by the photographing means 20 is displayed in the image display means 30 via the robot controller 50 with an image processing function. Using the photographed image, a three-dimensional position of a characteristic point in a workpiece W is measured, for example.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] The present invention relates to a robot system and a method of using the same. More specifically, the present invention relates to a robot system having an image processing function despite its simple configuration and a method of using the same.

[0002]

2. Description of the Related Art The demand for image processing in the application of industrial robots (hereinafter simply referred to as robots) is increasing.
It is predicted that an image processing device will be mounted on nearly half of the robots in the future. Therefore, various proposals have been made regarding mounting an image processing device on a robot.

For example, Japanese Patent Application Laid-Open No. 7-342818 proposes a robot device with a visual sensor that can use a display device (teach pendant) of a robot controller as a display device of an image processing device.

Japanese Unexamined Patent Publication No. Sho 59-95405 discloses that a correspondence between work identification data and a focus position is stored in a memory in advance, and when work identification data is received, a focus is set to a corresponding focus position. A focus adjustment method for an industrial robot with a sensor has been proposed.

In Japanese Patent Application Laid-Open Nos. 4-352204 and 4-352205, an image processing program and a gripping operation program are described in one user program. A method of controlling an automatic gripping device using vision, which transmits a command to a unit via a shared memory, has been proposed.

[0006] JP-A-11-134484 discloses that C
There has been proposed an image processing apparatus which is provided with an image processing unit for performing a sum-of-products operation separately from the PU, and transmits data from a camera to the image processing unit using DMA.

[0007] JP-A-61-279480 discloses that
A method of inputting a known position on a plane by pointing an image captured by two cameras with a light pen or the like, and a method of pointing an image of two cameras installed in different directions by three points. A method for teaching a working point of a robot regarding a method of inputting a dimensional position has been proposed.

Japanese Patent Application Laid-Open No. 8-238341 discloses that a two-dimensional position is pointed on an image by tracing the outline of a workpiece with a mouse or the like on an image picked up by a camera, and then pointing on the image. An offline teaching device for a robot that converts a dimensional position into a three-dimensional position using CAD data has been proposed.

However, in the proposal of Japanese Patent Application Laid-Open No. 7-342818, since the image transmission method and the image processing methods on the robot control device side and the display device side are not known, it is difficult to generalize the system as it is. There's a problem.

In the proposal of Japanese Patent Application Laid-Open No. 59-95405, it is necessary to set work identification data in advance, and since only the focus can be set, there is a problem that the applicable range is narrow.

In the proposals of JP-A-4-352204 and JP-A-4-352205, since the image processing section has a dedicated CPU and memory, there is a problem that the size of the apparatus is increased.

In the proposal of Japanese Patent Application Laid-Open No. 11-134484, since the image processing section is provided separately, there is a problem that the size of the apparatus is increased as described above.

In the proposal of Japanese Patent Application Laid-Open No. 61-279480, there is a problem that the configuration is complicated because a plurality of cameras are required.

In the proposal of Japanese Patent Application Laid-Open No. 8-238341, since the CAD data is used, there is a problem that the apparatus becomes large.

[0015]

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems in the prior art, and provides a robot system having an image processing function and a simplified apparatus configuration, and a method of using the same. It is intended to be.

[0016]

According to a first aspect of the present invention, there is provided a robot system comprising: an imaging unit mounted at an appropriate position on a robot body; an image display unit; and a robot controller having an image processing function. The image picked up by the means is displayed on the image display means via the robot controller with the image processing function.

A second embodiment of the robot system according to the present invention comprises:
An image pickup unit mounted at an appropriate position on the robot body, an image display unit, a teaching unit, and a robot controller with an image processing function are provided, and the image display unit and the teaching unit are integrated, and the image pickup unit A captured image is displayed on the image display means via the robot controller with the image processing function.

In the robot system of the present invention, it is preferable that the arithmetic processing device for controlling the robot body and the arithmetic processing device for performing image processing are also used.

In the robot system of the present invention, it is preferable that the arm control and the image processing are performed in parallel, and in that case, it is more preferable that the priority of the image processing is lowered.

Further, in the robot system according to the present invention, it is preferable that the image display means has a teaching function.

Further, in the robot system according to the present invention, it is preferable that the teaching means is capable of teaching about imaging and image processing.

Further, in the robot system of the present invention, it is preferable that the taught data can be stored.

Further, in the robot system according to the present invention, it is preferable that the teaching unit is configured to be able to operate the robot body when the teaching unit is teaching the imaging and the image processing.

Further, in the robot system according to the present invention, the calibration of the arm encoder can be performed by the mark imaged by the imaging means, or the abnormality of the arm encoder can be detected. Is preferred.

On the other hand, a first mode of use of the robot system according to the present invention comprises a single image pickup means mounted at an appropriate position on the robot body, an image display means capable of specifying a position on a display screen, A method for using a robot system comprising a robot controller with a processing function, comprising: a procedure for setting the robot system to a feature point three-dimensional position measurement mode; , A procedure for designating a feature point on the display image, and a feature point 3 for the designated feature point.
And a procedure for performing a dimensional position measurement process.

A second mode of use of the robot system according to the present invention comprises a single image pickup means mounted at an appropriate position on the robot body, an image display means capable of specifying a position on a display screen, and an image processing function. A method for using a robot system comprising a robot controller with a controller, the procedure for setting the robot system to a curved three-dimensional position measurement mode, and imaging of a workpiece by an imaging unit and displaying the image on an image display unit. It is characterized by comprising a procedure, a procedure of specifying a measurement range on a display image, and a procedure of performing a curve three-dimensional position measurement process on the specified measurement range.

According to a third mode of use of the robot system of the present invention, there is provided a single imaging means mounted at an appropriate position on the robot body, an image display means capable of specifying a position on a display screen, and an image processing function. A method of using a robot system comprising a robot controller with a controller, the procedure of setting the robot system to a tool angle calculation mode,
A procedure of imaging the workpiece by the imaging unit and displaying the image on the image display unit, a procedure of setting a range for specifying a tool angle on the display image, and a procedure of performing a curve three-dimensional position measurement process on the set range. And performing a tool angle calculation process on the curve whose three-dimensional position has been measured.

Here, the curve three-dimensional position measurement processing is as follows.
For example, this is performed using a two-point reference curve three-dimensional position measurement process or a multi-point reference curve three-dimensional position measurement process.

A fourth mode of use of the robot system according to the present invention is as follows: a single imaging means mounted at an appropriate position on the robot body; an image display means capable of specifying a position on a display screen; A method of using a robot system comprising a robot controller with a controller, wherein the robot system is set to a tool path creation mode,
A step of forming marks at predetermined intervals along a tool movement path on a work surface, a step of capturing the mark, and a step of performing a tool path creation process on the captured image. I do.

A fifth mode of use of the robot system according to the present invention is as follows: a single image pickup means mounted at an appropriate position on the robot main body; an image display means capable of specifying a position on a display screen; A method of using a robot system comprising a robot controller with a controller, wherein the robot system is set to a tool path creation mode,
The method is characterized by including a procedure of forming a line on a work surface along a tool movement path, a procedure of imaging the line, and a procedure of performing a tool path creation process on the captured image.

[0031]

Since the robot system of the present invention is configured as described above, the configuration of the robot system is simplified despite having an image processing function.

[0032]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described based on embodiments with reference to the attached drawings, but the present invention is not limited to only such embodiments.

FIG. 1 and FIG. 2 are block diagrams showing a robot system according to an embodiment of the present invention. The robot system S comprises a robot main body 10, an imaging means 20,
The image display unit 30, the teaching unit 40, the robot controller with image processing function 50, and the teach pendant 60 in which the display unit 30 and the teaching unit 40 are integrated are provided as main components.

The robot body 10 is, for example, an articulated robot.

The image pickup means 20 is, for example, an IEEE1394
And a digital CCD camera having a serial communication interface such as USB or the like, and is mounted at an appropriate position on a robot arm (hereinafter simply referred to as an arm). In this case, since the position of the workpiece may change due to the movement of the arm, the arm may have an auto focus and an auto iris function. However, when time is required for the setting, it is preferable that the focus and iris can be set manually from the outside.

The image data picked up by the image pick-up means (hereinafter, also simply referred to as a camera) 20 is divided into a required number of data blocks and then transmitted from the robot controller 50 having the image processing function. The signal is transmitted to the robot controller with image processing function 50 according to the signal.

As shown in FIG. 3, the image display means 30 and the teaching means 40 are, for example, a teach pendant with a liquid crystal display (hereinafter simply referred to as a teach pendant) 61.
It is integrated into.

The image display means 30 includes an image receiving section 31 for receiving an image signal transmitted from the robot controller 50 having an image processing function, and a liquid crystal display 32 for displaying image data received by the image receiving section 31. It is assumed to be provided with. The image display means 3
0 indicates that a point can be specified on the display image by a pointing device or the like. Note that the image receiving unit 31 and the liquid crystal display 32 are the same as the well-known ones conventionally used for the image display device, and therefore detailed description of the configuration is omitted.

The teaching means 40 includes a key input device 41 for inputting teaching data by key operation, and a liquid crystal display 3.
A touch panel interface 42 for inputting teaching data by touching an image displayed by the touch panel 2;
CPU for controlling both of them and sending signals from them to a robot controller 50 with an image processing function
An arithmetic processing unit 43 and a serial interface 44 for exchanging signals with the robot controller 50 having an image processing function are provided. here,
The touch panel interface 42 is preferably of an analog type because it requires fine position designation. In the case where a pointer capable of specifying a position with an accuracy of about a pixel is provided, the pointer may be digital.

The teaching means 40 is capable of teaching both of the teaching of the operation of the arm and the teaching of the imaging conditions and image processing by the changeover switch. That is, switching between the arm teaching mode and the image teaching mode can be performed. However, even in the image teaching mode, since the arm needs to be operated, for example, the position of the imaging unit 20 needs to be changed, the teaching unit 40 is also used in the image teaching mode.
Allows the arm to operate. Such a configuration is realized, for example, by enabling a key or the like used for teaching concerning imaging conditions and image processing in the image teaching mode.

Although the teaching relating to the image processing by the teaching means 40 will be described later, the teaching of the arm itself is the same as in the prior art, so that the description is omitted.

Robot controller 50 with image processing function
Is an arithmetic processing unit 51 composed of a CPU (arithmetic processing device)
And an image receiving unit 52 that receives digital image data from the imaging unit 20, and a storage unit 53 that includes a ROM and a ram.
And a servo controller 54 for controlling a servomotor for driving each axis of the robot, an image display means controller 55, a teaching means connecting section 56, and exchange of signals with external devices such as a printer and an external storage device. An external device connection unit 57 for performing
A data bus for exchanging signals of the respective units.

The CPU constituting the arithmetic processing unit 51 includes:
Since it is necessary to perform image processing without using a processor dedicated to image processing, the CPU is assumed to have a SIMD function. By using a CPU having such a function, 6
Eight pieces of 8-bit data per pixel are taken into the 4-bit register from the image memory of the storage unit 53, and can be binarized by comparing with the reference value. In addition, since SIMD operation capable of simultaneously performing eight product-sum operations can be applied,
An image filtering operation of 3 × 3 pixels, which needs to calculate nine products of the image and the mask data and calculate the total, can be processed at a relatively high speed.

The image receiving section 52 is provided with an interface 52a capable of receiving serial data transmitted from the imaging means 20, and a buffer 52b for temporarily storing the received serial data (FIG. 4). reference). For example, the IEEE 1394 included in the imaging unit 20
It is provided with an interface corresponding to a serial communication interface such as USB and USB, and a FIFO.

The image memory of the ram constituting the storage unit 53 needs to rapidly store the image data (serial data) received by the image receiving unit 52 in accordance with a command from the CPU. For example, synchronous DR
AM.

The image display control unit 55 includes an output image memory 55a for temporarily storing image data read from the image memory of the storage unit 53 in response to a command from the CPU;
A graphic controller 55b for reading and outputting image data stored in the output image memory 55a,
An image transmission unit 55c for receiving a signal from the graphic controller 55b and transmitting the signal to the image display unit 30 is provided. The output image memory 55a,
Graphic controller 55b and image transmission means 55
Since c is the same as a known one conventionally used in an image display device, a detailed description of its configuration is omitted.

The teaching means connection section 56 is a serial interface (teach pendant interface) that can receive teaching data (serial data) from the teaching means, for example. The teaching data input to the robot controller with image processing function 50 via the serial interface is stored in the storage unit 5 according to a command from the CPU.
3 is stored at a predetermined position of the ram.

Servo control unit 54, external device connection unit 57
Is the same as a known one conventionally used for a robot controller, and a detailed description of its configuration will be omitted.

Next, processing in the robot system S having such a configuration will be described.

The format of a command for operating the robot arm and a command for image processing are as follows.
Unify those used for robot control. By doing so, the complexity of creating a statement is eliminated. Then, the created statement is processed as a separate process from the arm control and the image processing. That is, they are processed in parallel. In this case, the priority of the image processing process is set lower than the priority of the arm control process so that the operation of the arm is not affected. Also, by processing the arm control and the image processing as separate processes, when the arm is stopped, the process of the arm control is completed in a short time, so that much time can be allocated to the image processing. it can. Here, these commands can be created by the user.

Some of these commands include a command for arm control and a command for image processing. Such a command is stored at a predetermined position in the ROM of the storage unit. It is stored and read out by the CPU at the time of execution, and the part related to arm control is inserted into a predetermined position of the arm control program that is executed as a separate process, while the part related to image processing is similarly executed as a separate process. Inserted in a predetermined position of the image processing program.

Some of the arm operation instructions and the instructions related to the image processing take a long time to complete the processing. Therefore, a command for instructing the start of the processing, a statement for checking the completion of the processing, a completion of the processing, There are prepared a command sentence to wait for a command, and a command sented to give an instruction and wait until the operation is completed. By preparing such a statement, it is possible to execute a statement for operating the arm during image processing.

Next, the teaching of the image processing by the teaching means 40 will be described.

The changeover switch is switched to the image teaching mode,
An image teaching related menu is displayed on the liquid crystal display 32. For example, the detection area setting menu, the imaging unit 20
, An imaging condition setting menu, a captured image processing condition setting menu, and the like are displayed.

Then, for example, when the imaging condition setting menu of the imaging means 20 is touched, the screen is switched to a screen for setting the iris opening and the focus position. At the same time, a window M is formed on the screen of the liquid crystal display 32, and an image from the imaging unit 20 is displayed. After adjusting the direction and the position of the imaging unit 20 by appropriately moving the arm, the iris opening and the focal position are set. Next, the obtained arm position, iris opening degree, focal position, and the like are stored in the ram of the storage unit 53 of the robot controller with image processing function 50 with a name. By doing so, the name can be called during operation, and the image pickup means 20 and the arm can be automatically set to the above conditions. Thus, by performing such a series of operations on the desired range of the work W, the teaching regarding the imaging unit 20 is completed.

Next, measurement of the three-dimensional position of the characteristic point P of the work W by the robot system S will be described.
Note that this measurement is performed by setting the teaching means 40 to the feature point three-dimensional position measurement mode, and this measurement is performed by changing the position of the feature point P of the work W by the camera 20 mounted on the arm and imaging it a predetermined number of times. This is realized by performing image processing on the obtained image, that is, by performing feature point three-dimensional position measurement processing. Hereinafter, the feature point three-dimensional position measurement processing will be described in detail.

(1) The position and orientation of the camera 20 are adjusted by operating the arm by the teaching means 40 (see FIG. 5).
At the same time, the focus and iris of the camera 20 are also adjusted, and the feature point P of the work W is clearly displayed at the center of the window M of the liquid crystal display 32.

(2) Work W displayed in window M
Is pointed by a pointing device (see FIG. 6).

(3) Image G near Pointing
, Two straight lines are detected, and their intersection is detected. The reason for this is to accurately grasp the position of the point pointed by the pointing device. The detection of a straight line can be easily performed by, for example, performing Hough transform after differentiating an image.

(4) The detected intersections (feature points) are displayed on the screen. If there is a problem with this position, the position of the feature point P is reset and the process returns to (1). If there is no problem in this position, the processing is continued.

(5) An equation of a straight line passing through the principal point and the intersection of the camera 20 is calculated from the current position and orientation of the camera 20 and the position of the intersection, and stored in the ram of the storage unit 53.

(6) Approximately 50 pixels vertically and horizontally around the intersection are stored in the image memory of the storage unit 53.

(7) The arm is moved in a predetermined direction, for example, rightward by about 1 cm.

(8) In this state, the workpiece is picked up again by the camera 20, the image is stored in the image memory, and the normal correlation coefficient between the image near the previous intersection and the vicinity of the previous intersection of the current image is calculated. The position which shows a large value is detected. When there are a plurality of positions showing large values, the position closest to the previous position is adopted.

(9) Two straight lines whose inclinations are close to the previously detected straight line at the position of the current image are detected, and their intersection is calculated.

(10) An equation of a straight line passing through the principal point and the intersection of the camera 20 is calculated from the current position and orientation of the camera 20 and the position of the intersection, and stored in the ram of the storage unit 53.

(11) The intersections of all the straight lines obtained so far are calculated by the least squares method. The obtained intersection is the position of the most probable feature point P at this time.

(12) Approximately 50 pixels vertically and horizontally around the intersection are stored in the image memory of the storage unit 53 again. in this case,
The image data stored in (6) may be overwritten.

(13) The camera 20 is further moved in the predetermined direction, for example, rightward with respect to the characteristic point P on the work W as a center.
Move about cm. In this case, the camera 2
The distance to 0 is the distance between the feature point P calculated to maintain focus and the camera position at (1), and the direction of the camera 20 is such that the position of the feature point P on the screen does not change. Aim at the angle you think.

(14) The processes of (8) to (13) are repeated. However, when the arm reaches the operation limit, (9)
If the intersection cannot be calculated, or if the stop operation is performed by the operator, the process ends.

(15) When the operator performs a continuation process, the camera is returned to the position of (1), and
While moving the camera 20 in a direction opposite to the predetermined direction, for example, in the left direction, the same processing as the processing of (2) to (14) is repeated.

(16) When the operator performs a continuous process operation, the camera 20 is returned to the position of (1), and the camera 20 is moved in a direction perpendicular to the predetermined direction, for example, in the upward direction. ) To (14) are repeated.

(17) When the operator performs the continuation process, the camera 20 is returned to the position (1), and the camera 20 is moved in a direction perpendicular to the predetermined direction, for example, in the downward direction. Processing similar to the processing of 2) to (14) is repeated.

(18) When the processing of (17) is completed, the position of the feature point P is calculated by the processing of (11). If the distance between the straight line obtained by each of the above processes and the calculated feature point P is about 2 mm or more, an error is determined and the fact is displayed on the liquid crystal display 32. And vice versa,
If there is no object of 2 mm or more, the feature point P
Since the measurement of the position has been completed, the position is given a name and stored in the ram of the storage unit 53.

(19) The camera 20 captures an image of the characteristic point P while moving the arm, and the image is displayed on the liquid crystal display 3.
2 and the measured feature point P is displayed on the screen. Thereby, it can be confirmed whether or not the measurement of the feature point P has been correctly performed. Note that this processing may be performed as needed, and therefore need not always be performed.

In the above description, the characteristic point P
Are the intersections of the straight lines, but the feature points P are not limited to the intersections of the straight lines, but can be set as appropriate, for example, the center of a circle.

The method of moving the camera 20 is not limited to the above-mentioned method, but may be any method. However, it is preferable to move on a spherical surface centering on the feature point P, that is, it is preferable to rotate the camera 20,
More preferably, the principal point of the camera 20 is set as the rotation center.

Further, in the above description, only one point, 3
Although the dimensional position was measured, even if the position of the camera 20 is changed, if a plurality of feature points P are focused and the points do not deviate from the field of view of the camera 20, the plurality of points are measured simultaneously. May be. However, in that case, the center for rotating the camera 20 is the average position of the plurality of points.

Further, in the above description, the work W
The operator designates the feature point P only once, but after the operator designates the feature point P, the arm is moved to change the position of the camera 20,
In this state, the same feature point P may be designated again by the operator, and the three-dimensional position of the feature point may be measured from the intersection of two straight lines obtained by these designations.

Furthermore, the above description is at the time of teaching, but when the operating pitch of the arm is long,
The measurement may be performed during the repeat operation. In this case, the feature point P specified by the operator is first measured by the normal feature point three-dimensional position measurement processing, and thereafter, the same processing as described above is automatically performed for the specified operation range of the arm. Is possible.

Further, in the above description, no special consideration is given to the background. However, in the measurement at the time of teaching, the work W and the background may be changed so that the image processing becomes easy. For example, a plate or paper may be placed on the background to make it easier to distinguish the work W, or the work W may be painted with a paint that suppresses reflection to make the work W easier to see.

By the way, the principal point position of the camera 20 moves back and forth to some extent by adjusting the focus. Therefore, in order to avoid a measurement error due to this, a method is adopted in which an image is taken at a specific focus position where the principal point position is known by calibration of the camera 20 in advance, and then the camera 20 is moved to a focused position to take an image. You may. Or use a small aperture, short focal length lens,
The focus adjustment may not be necessary. Alternatively, the adjustment may be made so that the feature point is located on the optical axis of the camera, that is, at the center of the image when the camera is moved in (13). Since the principal point moves along the optical axis, a straight line equation can be obtained with high accuracy in this manner.

In the work by the robot, it may be necessary to move the tool mounted on the arm along the curved edge of the work W. for that reason,
Measurement of the three-dimensional position of the curve is required. Below, curve 3
A three-dimensional position measurement method, that is, a curve three-dimensional position measurement process will be described. The curve three-dimensional position measurement process is performed by setting the teaching means 40 to the curve three-dimensional position measurement mode.

(1) The position and orientation of the camera 20 are adjusted by operating the arm (see FIG. 8), and the focus and iris of the camera 20 are adjusted. It is displayed horizontally at almost the center (see FIG. 9). In this case, the position of the camera 20 is set in consideration of moving the camera 20 from the lower side of the screen to the upper side in the following processing.

(2) Pointing the displayed curve
Specify the measurement range by tracing with the device. In this case, the tracing by the pointing device starts from the relatively horizontal portion of the displayed curve and ends at the relatively horizontal portion (see FIG. 9).

(3) A curve is detected from a pointed image in the vicinity. This detection can be easily performed by, for example, differentiating the image to binarize the image, and further thinning the image.

(4) The detected curve is displayed in the window M of the liquid crystal display 32. If there is a problem such that the displayed curve does not match the captured image, the curve is reset, and if there is no problem with the displayed curve, the process is continued.

(5) The current position and orientation of the camera 20 and the detected curve are stored in the ram of the storage unit 53.

(6) The arm is rotated by about 1 cm in a direction in which the displayed curve moves upward on the screen.

(7) The workpiece W is imaged, and a curve near the curve detected in (3) in the captured image is detected.

(8) The detected curve is displayed in the window M.

(9) The three-dimensional position of the curve is calculated from the previous position and orientation of the camera 20 and the previously detected curve, and the current position and orientation of the camera 20 and the currently detected curve. A three-dimensional center is calculated. This calculation method will be described later.

(10) The arm is further rotated about 1 cm in the same direction as (6). In this case, the center of rotation is the position calculated in (9), and the distance from the three-dimensional center of gravity of the curve to the camera 20 is the same as in (1).
This is to maintain focus on the curve.
The direction of the camera 20 is adjusted so that the position of the center point of the captured image does not change.

(11) The processes of (7) to (10) are repeated. However, if the arm reaches the operating limit or an error such as a curve detection error occurs, the operator determines that the arm may hit an obstacle, or performs a stop operation because the curve is over an obstacle. If so, the process ends.

(12) The three-dimensional positions of the curves are measured from the positions and directions of all the cameras 20 stored so far and the detected curves. The measuring method will be described later.

(13) The workpiece W is imaged while the arm is moved, the image is displayed on the liquid crystal display 32, and the measured curve is displayed on the screen. This makes it possible to confirm whether or not the measurement of the curve has been correctly performed. Note that this processing may be performed as needed, and therefore need not always be performed.

Next, a method of calculating a three-dimensional position of a curve from an image of two points, that is, a three-dimensional position measurement process of a two-point reference curve will be described with reference to FIG. Here, the two points are described as point A and point B.

[0098] (1) pixel on the curve _{C 1} measured at the point A i
For, it calculates the equation of the straight line L _i from the position and orientation of the camera 20.

(2) For all the pixels j _{1 to} j _n in the image of the curve C _{2 taken} at the point B, the equations of the straight lines L _{j1 to} L _jn are calculated from the position and the direction of each camera 20, and then the straight line is calculated. It calculates the distance between the L _i and the straight line _L j1 _{~L jn.}

(3) A straight line L _jm with the minimum distance is selected.

[0102] (4) If the distance of the straight line L _jm is 2mm or more, it indicates that there are no pixels on the curve L ₂ corresponding to the pixel i, the same process for the next pixel i + 1 Do. This process is performed until a straight line B _jm whose distance is within 2 mm is detected. On the other hand, a straight line L _jm
If the distance is within 2 mm, the following processing is performed.

(5) Intersection P _i between straight line L _i and straight line L _jm
Is calculated by the least squares method. This point _Pi is a three-dimensional position corresponding to the pixel i.

[0103] (6) for all other pixels on the curve L ₁ and calculates the three-dimensional position corresponding similarly.

(7) Continuity is checked for all calculated points. For example, if the distance between the point P _i and the point P _{i + 1} is equal to or more than an allowable value (for example, 2 mm), it is determined that an error has occurred.

(8) If an error is determined in (7), the measurement is restarted from the beginning. However, when measuring a curve from two points, the curve detection range is detected in the range traced by the pointing device, so it is unlikely that both ends of the curve detected at the two points are the same point. Ignore errors in the section.

By repeating such processing, 2
The three-dimensional position of the curve is calculated from the image of the point.

Next, a method for calculating a three-dimensional position of a curve from a multi-point image, that is, a multi-point reference curve three-dimensional position measuring process will be described with reference to FIG.

[0108] (1) point workpiece W imaged in _{A 1}
With respect to the pixel C _1i on the curve C ₁ in the image C, the formula of the straight line L _1i is calculated from the position and the direction of the camera 20.

[0109] (2) point _A 2 to A _n all pixels on the curve _C 2 -C _n in the image of the imaged workpiece W in C
About _{21 ~C 2n ... C n1 ~C nn} , linear _L 21 from the position and orientation of the camera 20 _~L 2n _{... L} n1 _~L
calculating a _nn equation, then the straight line _{L 1i} and the straight line _L 21 ~L
_2n ... The distances from L _{n1 to} L _nn are calculated.

[0110] (3) Distance at point _A 2 to A _n is selected a straight line _L 2m _{~L nm} as a minimum.

[0111] (4) pixels the calculated distance corresponds to the pixel _{C 1i} if 2mm above indicates that there is no curve _C 2 -C _n. In this case, the calculation is re-executed as an error, or interpolation is performed from points before and after the point to substitute. Conversely, if the distance is within 2 mm, the pixel C
Since the pixels corresponding to _1i have shown to be present in the curve C ₂ -C _n, the following processing is performed. When this error occurs, the following calculation is performed using only the straight lines L _{2m to} L _{nm of 2} mm or less. However, when the number of straight lines of 2 mm or less is smaller than, for example, n / 2, no calculation is performed. It is to be noted that if the data is at both ends of the curve, it is predicted, so that no error occurs. However, if the data is at an intermediate point, the calculation is performed again from the beginning.

(5) The intersection P _i between the straight line L _1i and the straight lines L _{2m to} L _nm is calculated by the least square method. This point _Pi is a three-dimensional position corresponding to the pixel i.

(6) The distance from the point P _i to the straight line L _{2m to} L _nm is calculated. This distance is an allowable value (for example, 2 mm)
If any of them exceeds the limit, an error will occur. in this case,
The distance is calculated not for all the straight lines L _{2m to} L _nm but only for the straight line described in (4) above. The plurality of straight lines and the points P _i
If the distance to is beyond the allowable value, stop the measurement,
Redo the calculation from the beginning. If the number of straight lines exceeding the allowable value is several, for example, three, the points P _i are excluded except for the straight lines.
Is recalculated.

[0114] (7) performs the same processing for the remaining pixels on the curve _{C 1.}

(8) Check the continuity of all calculated points P _i . For example, if the distance between the points P _i and P _{i + 1} is equal to or more than an allowable value (for example, 2 mm), it is determined that an error has occurred.

(9) If an error is determined in (8), the calculation is restarted from the beginning, or the points before and after that point are substituted by interpolation.

By repeating such processing, the three-dimensional position of the curve is calculated from the multi-point images.

Next, a method of calculating a tool angle, that is, a tool angle calculation process will be described.

To make the robot perform a desired operation,
It is necessary to set not only the position of the tool but also its orientation. That is, it is necessary to set the tool at a predetermined angle.
For example, when the tool is moved along a curve, the angle may be set based on the direction of the center of curvature at each point of the curve and the tangent direction of the curve, and may not be set to a predetermined offset angle from each direction. . However, a second derivative is required to find the center of curvature, and the position of the center of curvature obtained by performing the second derivative is strongly affected by measurement errors. Therefore, the angle of the tool is calculated by the following method. The angle calculation is performed by setting the teaching means 40 to the tool angle calculation mode.

(1) An image of the work W including a curve for setting the tool angle is captured.

(2) Display the captured image in the window M.

(3) A range for setting a tool angle is designated on the displayed image.

(4) The three-dimensional position of the designated curve is measured by the above method.

(5) A point on the curve for which the angle of the tool is to be calculated (hereinafter referred to as a target point) is set, and each point of several tens of pixels before and after the target point (hereinafter referred to as each neighboring point) is set. The tangent direction is calculated.

(6) A difference vector between the target point and each neighboring point is calculated.

(7) An average vector of the obtained difference vectors is calculated.

(8) The tangential component is removed from the obtained average vector.

(9) The tool angle is calculated using the vector from which the tangential direction component has been removed as a substitute vector for the curvature center direction vector. That is, the tool angle is calculated by adding a predetermined offset angle to the angle of the substitute vector.

In this manner, the tool angle can be calculated with less influence of the measurement error. If the curve has an inflection point or a portion close to a straight line, the processing of (6) cannot be performed. Therefore, the tool angle at such a location is determined by the tool angle at the location where the processing before and after the processing could be performed. It is calculated by using and interpolating. This process may be performed subsequent to the curve three-dimensional position measurement process.

Next, a method of creating a tool moving path when the tool is moved along the curved surface of the workpiece W, that is, a tool path creating process will be described.

When moving the tool along the curved surface of the work W, it is necessary to perform three-dimensional measurement of the work W. The measurement of the curved surface is performed by a three-dimensional sensor when there is no pattern or the like on the surface of the workpiece W, but the three-dimensional sensor is expensive. Therefore, the work W
A method of setting the moving path of the tool along the surface in the robot controller with image processing function 50 will be described. This is done by setting the teaching means 40 to the tool path creation mode.

In the first form of the tool path creation processing, a mark is written on the surface of the work W at predetermined intervals along the movement path of the tool, or a sticker with a mark is attached at a predetermined interval on the surface of the work W. Are formed, the three-dimensional positions of these marks are calculated by the above-described method and stored in the ram of the storage unit 53, and the calculated three-dimensional positions are interpolated by an interpolation curve such as a spline curve to move the tool. Is to create. Then, the created route is stored in the RAM of the storage unit 53. Thereby, the movement path of the tool along the surface of the work W is set in the robot controller 50 with the image processing function.

In the second form of the tool path creation processing, a line is written on the surface of the work W along the moving path of the tool, or a line sticker is formed to form a line on the surface of the work W. Is calculated by the method described above, and the line on which the three-dimensional position is calculated is used as the tool movement path. Then, this line is stored in the ram of the storage unit 53 as a moving path of the tool. Thereby, the movement path of the tool along the surface of the work W is set in the robot controller 50 with the image processing function.

Next, calibration of the arm encoder will be described.

This is because the three marks (the positions of which are known in advance) attached to the base part (the part which is fixed and does not move) of the arm are imaged by a camera mounted on the arm, as described above. It can be achieved by processing according to the method.

The first method is to preliminarily set the image pickup position and the movement path of the camera 20 when performing calibration, that is, to perform programming and automatically image three marks to obtain these marks. By measuring the three-dimensional position of the mark, comparing the three-dimensional position of the mark known in advance, and correcting the value of the arm encoder so as to eliminate the difference, calibration of the arm encoder can be performed. In the second method, when the camera cannot be moved to a preset position due to an obstacle or the like, the camera 20 is manually moved to image three marks, and their three-dimensional positions are measured. The arm encoder can be calibrated by comparing the three-dimensional position of the mark and correcting the value of the arm encoder so as to eliminate the difference.

In the calibration, the coordinates of the three measured points are known in advance.
It is necessary to perform coordinate conversion into points, which can be performed using various known coordinate conversion methods. For example, in the coordinate transformation, it is necessary to perform translation by a translation vector and rotation by a rotation matrix. The translation vector can be calculated from the difference between the positions of the centers of gravity of three points, and the rotation matrix is calculated from the center of gravity to each vertex. It can be calculated from the heading unit vector. Then, by processing the position and orientation of the arm at the time of measurement using the obtained translation vector and rotation matrix, the correct position and orientation of the arm are calculated.
Next, the angle of each joint of the arm is calculated from the position and orientation. Thus, an error of the arm encoder is calculated from the current joint angle measured by the arm encoder and the calculated angle. By subtracting this error from the arm encoder, calibration of the arm encoder can be performed. If the error is larger than a predetermined value, the process is executed again as an error.

In this embodiment, the abnormality of the arm encoder can be detected even during the operation. This can be done as follows.

First, an object as clear as possible and easy to detect is registered as a mark. Then, this mark is periodically detected during the operation of the arm, and the positional deviation of the mark on the image is measured. When the deviation becomes a predetermined value or more, it is determined that the mark is abnormal. The mark can be easily detected by a method using a normal correlation coefficient widely used in image processing.

Although the present invention has been described based on the embodiments, the present invention is not limited to only such embodiments, and various modifications can be made. For example, as shown in FIG. 12, a DMA controller may be provided downstream of the image receiving unit 52.

[0141]

As described in detail above, according to the present invention,
An excellent effect that the configuration of the robot system having the image processing function is simplified can be obtained. Further, since the robot controller also has an image processing function, it is possible to obtain an excellent effect that the speed of data transmission and reception is increased and the on-site adjustment work related to the image processing is shortened.

According to the preferred embodiment of the present invention, since the arm control and the image processing are performed in parallel, an excellent effect that the robot controller is effectively used can be obtained. Further, according to another preferred embodiment of the present invention in which the priority of the image processing is lowered, an excellent effect that the arm operation is not hindered even when the arm control and the image processing are performed in parallel. Is obtained.

Further, according to another preferred embodiment of the present invention, an excellent effect that calibration of the arm encoder and abnormality of the arm encoder can be detected can be obtained.

On the other hand, according to the method of using the robot system of the present invention, an excellent effect that the three-dimensional position of the characteristic point of the work and the three-dimensional position of the curve of the work can be measured can be obtained.

Further, according to another method of using the robot system of the present invention, an excellent effect that an angle of a tool can be set and an excellent effect that a movement path of a tool can be created can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram of a robot system according to an embodiment of the present invention.

FIG. 2 is a block diagram of the main part.

FIG. 3 is a block diagram of the image output unit.

FIG. 4 is a block diagram of the image input unit.

FIG. 5 is an explanatory diagram of a method of measuring a three-dimensional position of a feature point of a work by the robot system of the present invention, and shows a state in which the camera captures a feature point of the work.

FIG. 6 is an explanatory diagram of a method of measuring a three-dimensional position of a feature point of a work by the robot system of the present invention, and shows a state in which a captured feature point is displayed in a window.

FIG. 7 is an explanatory diagram of a method of measuring a three-dimensional position of a feature point of a work by the robot system of the present invention, showing feature points calculated from intersections of a large number of straight lines by a least square method.

FIG. 8 is an explanatory diagram of a method of measuring a three-dimensional position of a curve of a work by the robot system of the present invention, and shows a state where the curve is imaged by a camera.

FIG. 9 is an explanatory diagram of a method of measuring a three-dimensional position of a curve of a work by the robot system of the present invention, and shows a state in which an imaged curve is displayed in a window.

FIG. 10 is an explanatory diagram of a method of measuring a three-dimensional position of a curve of a work by the robot system of the present invention, showing a two-point reference curve three-dimensional position measuring process.

FIG. 11 is an explanatory diagram of a method of measuring a three-dimensional position of a curve of a work by the robot system of the present invention, and shows a multi-point reference curve three-dimensional position measuring process.

FIG. 12 is a block diagram showing a main part of another embodiment of the present invention.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Robot main body 20 Imaging means, camera 30 Image display means 31 Image receiving part 32 Liquid crystal display 40 Teaching means 41 Key input device 50 Robot controller with image processing function 51 Arithmetic processing part 52 Image receiving part 53 Storage part 55 Image display means control part 56 teaching means connection unit 60 teach pendant S robot system W work

 ──────────────────────────────────────────────────続 き Continued on front page F term (reference) 3F059 BC00 BC06 BC07 DA02 DB00 DB05 DB08 DB09 FA05 FB12 FB21 FB26 5H269 AB33 BB01 JJ20 QC10 QD02 SA08 9A001 BB04 GG03 GG05 HH19 HH20 HH29 HH34 KK16 KK29 KK32 KK32 KK32

Claims (18)

[Claims] 1. An image pickup device mounted at an appropriate position on a robot body, an image display device, and a robot controller with an image processing function, wherein the image picked up by the image pickup device is the robot controller with the image processing function. The robot system is displayed on the image display means via a display. 2. An image display device, an image display device, a teaching device, and a robot controller with an image processing function, which are mounted at an appropriate position on the robot main body, wherein the image display device and the teaching device are integrated. An image captured by the image capturing means is displayed on the image display means via the robot controller with image processing function. 3. The robot system according to claim 1, wherein an arithmetic processing device for controlling the robot body and an arithmetic processing device for performing image processing are also used. 4. The robot system according to claim 3, wherein the arm control and the image processing are performed in parallel. 5. The robot system according to claim 4, wherein the priority of the image processing is lowered. 6. The robot system according to claim 1, wherein the image display means has a teaching function. 7. The robot system according to claim 2, wherein the teaching means can also teach about imaging and image processing. 8. The robot system according to claim 2, wherein the taught data can be stored. 9. The robot system according to claim 7, wherein when the teaching means is teaching the imaging and the image processing by the teaching means, the robot body is also operable. 10. The robot system according to claim 1, wherein calibration of the arm encoder can be performed by the mark imaged by the imaging means. 11. The robot system according to claim 1, wherein an abnormality of the arm encoder can be detected by a mark imaged by the imaging means. 12. Use of a robot system comprising: a single imaging unit mounted at an appropriate position on a robot body; an image display unit capable of specifying a position on a display screen; and a robot controller with an image processing function. A method for setting the robot system to a feature point three-dimensional position measurement mode, a procedure for capturing an image of a workpiece by an imaging unit and displaying the image on an image display unit, and specifying a feature point on a display image. And a step of performing a feature point three-dimensional position measurement process on a designated feature point. 13. Use of a robot system comprising: a single imaging unit mounted at an appropriate position on a robot body; an image display unit capable of specifying a position on a display screen; and a robot controller with an image processing function. A method of setting the robot system to a curved three-dimensional position measurement mode; a step of imaging a workpiece by an imaging unit and displaying the image on an image display unit; and specifying a measurement range on the display image. A method of using a robot system, comprising: a procedure; and a procedure of performing a curve three-dimensional position measurement process on a specified measurement range. 14. Use of a robot system comprising: a single imaging means mounted at an appropriate position on a robot body; an image display means capable of specifying a position on a display screen; and a robot controller with an image processing function. A method of setting the robot system to a tool angle calculation mode; a step of imaging a workpiece by an imaging unit and displaying the image on an image display unit; and a step of specifying a tool angle on the display image. A robot system, comprising: a setting step; a step of performing a curve three-dimensional position measurement process on a set range; and a step of performing a tool angle calculation process on a curve whose three-dimensional position is measured. How to Use. 15. The method according to claim 13, wherein the curve three-dimensional position measurement processing is performed using a two-point reference curve three-dimensional position measurement processing. 16. The method according to claim 13, wherein the curve three-dimensional position measurement processing is performed using a multi-point reference curve three-dimensional position measurement processing. 17. Use of a robot system comprising: a single imaging unit mounted at an appropriate position on a robot body; an image display unit capable of specifying a position on a display screen; and a robot controller with an image processing function. A method of setting the robot system to a tool path creation mode, a step of forming marks at predetermined intervals on a work surface along a tool movement path, a step of imaging the marks, A procedure for performing a tool path creation process on an image. 18. Use of a robot system comprising: a single imaging unit mounted at an appropriate position on a robot body; an image display unit capable of specifying a position on a display screen; and a robot controller with an image processing function. A method of setting the robot system to a tool path creation mode; a step of forming a line along a tool movement path on a work surface; a step of imaging the line; And a tool path creation process.