JP6611888B2 - Robot device, control method of robot device, program, and recording medium - Google Patents

Description

The present invention is a robot device for controlling the position and orientation of the robot arm by using a visual sensor, the control method of the robot apparatus, a program, and a recording medium.

  Robot devices are widely used in industrial production sites. In this type of robot apparatus, in order to automatically and accurately perform work picking and assembly work, a system configuration is known that combines measurement with a robot and a visual sensor. The visual sensor here is a sensor that can measure the position and orientation of an object two-dimensionally or three-dimensionally.

  This type of visual sensor includes an optical sensor such as a camera, and the position and orientation of an object are two-dimensionally or three-dimensionally determined through image processing such as image recognition for an image captured by the optical sensor. measure. For example, a general monocular industrial camera, a stereo camera in which a plurality of cameras that perform three-dimensional measurement in a specific parallax relationship are arranged, and an optical system that projects sheet-like laser light or pattern light onto an object and a camera are combined. A three-dimensional measurement sensor or the like is used as a visual sensor. In general, expressions such as “vision”, “robot vision”, and “camera” may be used as a term for such a visual sensor for robot use.

  For example, in a robot vision system of a robot apparatus that picks a workpiece, a configuration in which a visual sensor is installed above a workpiece supply area (a conveyor, a tray, a component box, etc.) is known. This visual sensor is arranged at an arrangement position having a fixed and fixed positional relationship with respect to the installation surface of the robot arm independently of the operation of the robot arm, and a relatively wide range (for example, the whole) of the workpiece supply area. As a measurement range.

  In the picking operation, this visual sensor first measures the position and posture of a flat or piled work. Then, the movement of the robot arm is corrected based on the position and orientation information, and a robot hand (gripper, suction pad, etc.) provided at the tip of the robot arm is approached from above the workpiece to obtain a part.

  In order to make such a robot vision system perform the work, the relationship between the coordinate system of the robot and the coordinate system of the visual sensor is often obtained in advance by calibration. In order to obtain the robot motion target value from the measurement value of the visual sensor, it is necessary to coordinate-transform the measurement value obtained in the coordinate system of the visual sensor into a value in the robot coordinate system that serves as a reference for controlling the robot arm. It is.

  As a conventional typical calibration method, there is coordinate alignment by inching. While measuring three or more feature points such as workpieces and markers fixed in space from the visual sensor, the robot coordinates so that the coordinates of representative points such as tool vertices at the tip of the robot arm coincide with the three or more feature points. The operator performs inching operation of the arm and positions it. Then, a calibration value representing the relationship between the coordinate system of the robot and the coordinate system of the visual sensor is obtained from the measured value of the visual sensor and the position and orientation of the robot arm when the robot arm is positioned.

  There is also known a method of performing calibration by attaching a board with a calibration marker to the tip of a robot arm (for example, Patent Document 1 below). In this method, the robot arm is positioned in a plurality of calibration postures, and the measurement value of the marker is acquired from the visual sensor in each calibration posture, and the calibration value is calculated.

JP 2010-172986 A

  The conventional method of coordinate alignment by inching requires a fine adjustment of the position and orientation of the robot arm through the operator's inching operation, and therefore requires a longer work time for coordinate alignment than automated operations. There was a problem of becoming. Moreover, since accuracy depends on the skill level of the operator, there is a problem that the accuracy is not stable.

  Further, the calibration jig as described in Patent Document 1 needs to manufacture the positional relationship between the feature points (markers) of the jig with high accuracy, or to measure with high accuracy after manufacturing. Incurred cost increase. In particular, when using a robot vision system in multiple processes in a factory, it is necessary to use a jig of an appropriate size according to the visual field of measurement of the visual sensor. It had to be prepared and the impact was great.

  Further, in the method of applying a calibration marker to the tip of the robot as in Patent Document 1, there is a possibility that the marker applied to the robot arm may be hidden from the visual sensor depending on the work posture. For example, the marker may be hidden when the measurement direction of the visual sensor and the approach direction of the robot hand are substantially the same. For example, when a workpiece is measured from above with a visual sensor arrangement as described above and picking is performed from above, the robot itself blocks the line of sight of the visual sensor during the picking posture. Cannot be measured.

  Therefore, it is unavoidable to make the robot posture at the time of calibration significantly different from that at the time of actual work so that the marker at the tip of the robot can be seen, and the marker feature point position is far away from the robot tip. It was necessary to design. However, if the robot's posture or the position of the feature point of the marker measured from the visual sensor is greatly changed from the actual work conditions such as picking, the influence of the robot arm operation error and the visual sensor measurement error will increase and the calibration will This causes a problem that the accuracy of the lowering. In general, the absolute position accuracy of the robot is low compared to the repeatability of the robot's repeated position, so the robot should be installed in a posture that is close to the posture of the actual work and covers as wide a range as possible during actual work. It is desirable for accuracy to operate and perform calibration.

  In view of the above problems, an object of the present invention is to provide a robot apparatus capable of calibrating a robot coordinate system and a visual sensor coordinate system with high accuracy even with a low-precision jig without requiring fine adjustment work by an operator. The purpose is to provide. Another object of the present invention is to provide a robot vision system that can perform high-precision calibration without being affected by hiding even if the measurement direction of the visual sensor and the approach direction of the robot hand are substantially the same direction. To do.

To solve the above problems, the present invention, possess a robot arm, a proximal end and a predetermined positional relationship of the robot arm, measures the measurement range including the movable range of the robot arm, the robot arm A first visual sensor that is not supported by the robot arm , a second visual sensor supported by the robot arm , and a control device that controls the position and orientation of the robot arm. The first measurement data obtained by measuring a plurality of characteristic portions provided in the measurement range by the visual sensor, and the robot arm takes a plurality of positions and orientations, and the characteristic portion is detected by the second visual sensor. using the second measurement data obtained by measuring, with a calibration position data corresponding to each of the plurality of position and orientation of the robot arm, the measurement of the first visual sensor And adopting a configuration that calculates the calibration value that relates the command value to be given to the robot arm.

  According to the above configuration, since the reference mark is measured from both the first and second visual sensors, the calibration process can be performed in a short time without the need for strictly fine-adjusting the tip of the robot to the marker. it can. In addition, the calibration value is calculated using the measurement values of the first and second visual sensors, and does not use prior information such as design dimensions related to the coordinate value of the marker. Calibration is possible. In addition, even if the measurement direction of the first visual sensor and the approach direction of the robot hand are almost the same, calibration can be performed without being affected by hiding, so the robot can be operated in a posture close to that during actual work and in a sufficiently wide range. Calibration can be performed with high accuracy.

It is explanatory drawing which showed the structure of the robot apparatus of Example 1 of this invention in the state at the time of picking operation | movement. It is explanatory drawing which showed the structure of the robot apparatus of Example 1 of this invention in the state at the time of calibration. It is explanatory drawing which showed an example of arrangement | positioning of the reference | standard member of this invention. It is explanatory drawing which showed an example from which arrangement | positioning of the reference | standard member of this invention differs. It is the flowchart figure which showed the calibration control procedure in Example 1 of this invention. It is explanatory drawing which showed the relationship of each coordinate system in Example 1 of this invention. It is explanatory drawing which showed the structure of the robot apparatus of Example 2 of this invention in the state at the time of calibration. It is the flowchart figure which showed the calibration control procedure in Example 2 of this invention. It is explanatory drawing which showed the relationship of each coordinate system in Example 2 of this invention. It is explanatory drawing which showed the structure of the robot apparatus of Example 3 of this invention with the state at the time of calibration with the relationship of each coordinate system. It is the flowchart figure which showed the calibration control procedure in Example 3 of this invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described with reference to embodiments shown in the accompanying drawings. The following embodiment is merely an example, and for example, a detailed configuration can be appropriately changed by those skilled in the art without departing from the spirit of the present invention.

  First, mathematical expressions in this specification will be described. The following mathematical expressions are applicable not only to this embodiment but also to embodiments described later.

(Definition of mathematical expression in this specification)
In this specification, a three-dimensional coordinate system is used to express the 6-degree-of-freedom position and orientation of each element, and the relative orientation of the two coordinate systems is represented by a coordinate transformation matrix (homogeneous transformation matrix). In this specification, a coordinate transformation matrix representing a relative position and orientation from an arbitrary coordinate system A to an arbitrary coordinate system B is described in the form of ^A H _B. Also, the relative posture represented by the coordinate transformation matrix may be simply described as “relative posture ^A H _B ”. Here, ^A H _B is a 4 × 4 homogeneous transformation matrix and is defined as follows using the rotation matrix r and the translation vector t.

ここで、ｒは３次元の回転を表す３×３の回転行列であり、ｔは３×１の３次元の並進ベクトルである。座標変換行列の性質として、任意の座標系Ａ，Ｂ，Ｃに対して、下記の関係が成立する。

Here, r is a 3 × 3 rotation matrix representing a three-dimensional rotation, and t is a 3 × 1 three-dimensional translation vector. As a property of the coordinate transformation matrix, the following relationship is established for arbitrary coordinate systems A, B, and C.

すなわち、座標系Ａに対する座標系Ｂの相対姿勢^ＡＨ_Ｂと座標系Ｂに対する座標系Ｃの相対姿勢^ＢＨ_Ｃから、行列の掛け算により座標系Ａに対する座標系Ｃの相対姿勢^ＡＨ_Ｂを求めることができる。

That is, the relative orientation ^B H _C coordinate system C for the relative orientation ^A H _B and the coordinate system B of the coordinate system B with respect to the coordinate system A, obtains the relative position ^A H _B of the coordinate system C with respect to the coordinate system A by matrix multiplication be able to.

In addition, in order to obtain the relative posture ^B H _A of the coordinate system A with respect to the coordinate system B, an inverse matrix may be calculated as follows.

また三次元空間上の任意の点Ｐを座標系Ａから見た座標値の同次ベクトルを^ＡＰと表現する。その座標値を（Ｘ，Ｙ，Ｚ）とすると同次ベクトル^ＡＰは

Also the homogeneous vector coordinate values viewed arbitrary point P on the three-dimensional space from the coordinate system A is expressed as ^{A P.} The coordinate values (X, Y, Z) when the homogeneous vector ^A P is

のように表わされる。
点ＰをＢ座標系から見た座標値^ＢＰに変換するためには以下のように相対姿勢^ＢＨ_Ａ（同次変換行列）を左から掛けることにより、座標変換することができる。

It is expressed as
By multiplying the relative posture ^{B H} A _(homogeneous transformation matrix) from the left as shown below in order to convert the point P in the coordinate value ^{B P} as viewed from the B coordinate system, it is possible to coordinate transformation.

  Next, the hardware configuration of the robot apparatus used in the first embodiment and the calibration procedure will be described. FIG. 1 shows an example of a hardware configuration of a robot apparatus according to this embodiment of the present invention. FIG. 1 shows a configuration of a robot vision system (hereinafter referred to as a robot system) 10 in a picking operation state. FIG. 2 is a system configuration diagram at the time of calibration of the robot system 10 according to the embodiment of the present invention.

  As shown in FIGS. 1 and 2, the robot system 10 includes a robot arm 1, a robot hand 2 fixed to the hand of the robot arm 1, a fixed camera 3 (first visual sensor), a control device 4, and the like. A work placement area 5 and a gantry 9 are provided. At the time of calibration, a hand camera 7 (second visual sensor) and a marker 8 (reference member) are further used (FIG. 2).

  The fixed camera 3 is installed on a mount on a support leg or a pan head arranged on the installation surface of the gantry 9. Or you may arrange | position to the mount etc. which hung from the indoor ceiling which arrange | positions the mount frame 9. FIG. In any case, the fixed camera 3 is fixed and fixed to the installation surface of the gantry 9 that supports the robot arm 1 independently of the operation of the robot arm 1 through these appropriate fixed camera support portions. It arrange | positions at the arrangement | positioning position which has the positional relationship.

  The work placement area 5 constitutes a measurement area in which the work 6 or the marker 8 to be measured by the fixed camera 3 or the hand camera 7 is placed. Naturally, this measurement area includes the movable range of the robot arm 1 and has a fixed and fixed positional relationship with respect to the installation surface of the gantry 9 that supports the robot arm 1.

  The control device 4 is a control means for controlling the operation of the entire robot system 10, and includes a CPU 41 that performs calculation, a memory unit that includes a ROM 42 and a RAM 43, and an interface (I / F) unit 44 that communicates with the outside. ing. These blocks are connected to each other via a bus 45 used for communication inside the control device.

  A memory area for storing calibration posture data and a final calibration value (calibration function) necessary for the measurement of the hand camera 7 generated in the calibration process described later is arranged in the ROM 42 or the RAM 43, for example. For example, the area of the ROM 42 for storing the calibration posture data and the calibration value (calibration function) is preferably composed of a rewritable ROM device such as an EEPROM. Alternatively, the area for storing the calibration attitude data and the calibration value (calibration function) may be arranged in the area of the RAM 43 where the stored contents are backed up even when the main power supply is shut off with an appropriate power supply. With such a configuration, the calibration posture data and the calibration value (calibration function) generated once can be used even if the shut-off of the main power supply is narrowed. This kind of calibration attitude data and calibration value (calibration function) should be stored in a fixed memory over a certain span, so the storage area is a non-volatile memory as described above. Is preferable. For storing these data, an external storage device such as an HDD or an SSD may be used in addition to the non-volatile areas of the ROM 42 and the RAM 43.

  In FIG. 1, the control functions of the CPU 41 are indicated by the units 411 to 414. Although these control units and calculation units are illustrated as if they were hardware blocks, in reality, for example, the CPU 41 is realized by reading and executing a program stored in the ROM 42. These control units and calculation units are a robot control unit 411, a calibration calculation unit 412, a visual sensor control unit 413, and a feature position calculation unit 414.

  In the present embodiment, a case where the robot control unit 411, the calibration calculation unit 412, the visual sensor control unit 413, and the feature position calculation unit 414 are provided inside one control device 4 will be described as an example. However, for each of these blocks, an equivalent function may be realized by combining a plurality of separately provided control devices.

  The robot arm 1 is, for example, a 6-axis vertical articulated robot arm fixed to the gantry 9 and is electrically connected to the interface unit 44 of the control device 4. The robot arm 1 is controlled in position and orientation by the command value of the robot control unit 411, and can move the tip portion with six degrees of freedom. A robot coordinate system R is set at the base end of the robot arm 1 as a coordinate system representing the attachment position and orientation of the robot arm 1 with respect to the gantry 9. The robot control unit 411 communicates with the robot arm 1 and the robot hand 2 via the interface unit 44, and controls these operations.

  In FIG. 1 and FIG. 2, the interface unit 44 is shown as one block for the sake of brevity. However, in practice, the interface unit 44 may be configured by interface units of several different communication methods for communicating with the outside. For example, the interface unit 44 is also used for input / output with the fixed camera 3 and the hand camera 7 described later, but the communication method with these cameras may be different from the communication method used for communication with the robot arm 1. Absent. Further, in FIGS. 1 and 2, a network 50 is connected to the interface unit 44. The CPU 41 can communicate with a server that manages the production process and other robot apparatuses in the plant via the interface unit 44 to the network 50.

  The ROM 42 is used as a program memory for storing a program executed by the CPU 41, for example, a so-called computer-readable recording medium. Such a program memory is not limited to the ROM 42 and can be composed of other storage devices. As a storage device for storing a program executed by the CPU 41, a fixed or removable external storage device such as an HDD or SSD, various removable flash memories or optical disks can be used (all not shown). . In particular, the removable storage device can be used for the purpose of introducing, supplying, or updating the control program of the CPU 41. Alternatively, the control program for the robot system 10 according to the present embodiment can be downloaded via the network 50 connected to the interface unit 44, and can be introduced and supplied (or updated) to the various program memories described above.

  Further, a teaching pendant (not shown) is connected to the interface unit 44 as a teaching device for teaching the operation of the robot arm 1. As is well known, the teaching pendant (TP) has various function keys, a display, etc., and moves the robot arm 1 by manual operation to, for example, sequentially teach a plurality of teaching points, thereby changing the position and orientation of the robot arm 1. Can be programmed. The following calibration process is used to teach the robot arm 1 the position and orientation at which the plurality of markers 8 can be measured by the hand camera 7.

  The interface unit 44 is connected to a display (display device using an LCD panel or the like) (not shown). This display can be used to display an image captured by the fixed camera 3 or the hand camera 7 during a teaching operation using a teaching pendant, for example, in a calibration process described later.

  The robot hand 2 is provided at the tip of the robot arm 1 for gripping the workpiece 6. The robot hand 2 is also connected to the control device 4 via the interface unit 44. The robot hand 2 is controlled by the robot controller 411 of the control device 4 and can perform an operation of gripping and releasing the workpiece 6 by opening and closing the fingers 21. The robot hand 2 includes a hand camera mounting portion 22 for mounting a hand camera 7 described later with good reproducibility. In the present embodiment, the robot hand 2 that grips the workpiece 6 by opening and closing the fingers 21 is illustrated. However, any other end effector that acts on the workpiece 6 may be used. . For example, a suction pad that acquires the workpiece 6 by vacuum suction, an electromagnet tool that acquires the workpiece 6 using magnetic force, or the like may be used instead of the robot hand 2. A hand coordinate system T representing the position and orientation of the robot hand 2 is set in the palm of the robot hand 2.

  The robot arm 1 adjusts the position and orientation of the robot hand 2 by driving each joint based on a command value output from the robot control unit 411 of the control device 4. The robot control unit 411 calculates an angle target value to be taken by each joint of the robot arm 1 with respect to a target value of the relative position and orientation of the hand coordinate system T with respect to the robot coordinate system R (inverse kinematic calculation). . Then, servo control is performed so that the current angle output from an encoder (not shown) provided at each joint of the robot arm 1 matches the angle target value. In addition, the robot control unit 411 can acquire current angle information of each joint from the encoder and calculate the current relative position and orientation of the hand coordinate system T with respect to the robot coordinate system R.

  The work placement area 5 is an area for placing the work 6 during picking. From the viewpoint of supplying the workpiece 6, the workpiece arrangement area may be considered as the above-described workpiece supply area. The work 6 to be supplied is laid flat on the upper surface of the work placement area in different directions as shown in FIG. In the present embodiment, the case of flat placement is illustrated, but the workpieces 6 may be supplied by being partitioned one by one on a dedicated tray, or the workpieces 6 are supplied in a state of being piled up on boxes or trays. May be.

  1 and 2, the fixed camera 3 of this embodiment is composed of a stereo camera capable of measuring a workpiece 6 and a marker 8 three-dimensionally, for example. The shooting distance of the fixed camera 3 and the focal length of the optical system are determined so that the entire range in which the workpiece 6 can exist in the workpiece placement area 5 can be measured.

  The fixed camera 3 is connected to the control device 4 via the interface unit 44, receives a command from the visual sensor control unit 413, photographs the work placement area 5, and transmits image data to the control device 4. The feature position calculation unit 414 performs image processing on the image data captured by the fixed camera 3 to recognize feature parts such as edges and corners of the work 6 and uses the stereo algorithm to detect the position of the work 6 or The position and orientation are calculated three-dimensionally. The camera calibration parameters for associating the two-dimensional coordinate values on the stereo image with the three-dimensional coordinates necessary for performing the three-dimensional measurement from the fixed camera 3 are calibrated in advance by the fixed camera 3 alone. It is assumed that it is stored in

  More specifically, for example, a camera internal parameter for associating a two-dimensional image coordinate and a three-dimensional line-of-sight vector for each camera constituting a stereo camera, and a camera relative position and orientation parameter representing a relative position and orientation between the cameras. Calibrate in advance. Various methods for calibrating a stereo camera are known and will not be described in detail here. For example, a fixed camera coordinate system F representing the position and orientation of the fixed camera 3 is virtually set at one lens principal point position among a plurality of cameras constituting the stereo camera. It is assumed that the measurement value when the measurement is performed from the fixed camera 3 as described above is output in the form of a coordinate value or a coordinate transformation matrix expressed using the fixed camera coordinate system F as a reference.

  The hand camera 7 is a detachable stereo camera that is attached to the hand camera mounting portion 22 of the robot arm 1 when the robot system 10 is calibrated. It is mechanically supported by the robot hand 2 via the hand camera mounting portion 22 and is electrically connected to the interface portion 44 by a wiring (not shown) provided in the body of the robot arm 1 so that it can communicate with the control device 4. It is configured as follows. Similar to the fixed camera 3, the hand camera 7 captures an image according to an imaging command from the visual sensor control unit 413 and transmits image data to the control device 4.

  Regarding the hand camera 7, camera calibration parameters necessary for associating the two-dimensional coordinate values on the stereo image with the three-dimensional coordinates are obtained in advance and stored in the ROM 42 in advance. The feature position calculation unit 414 also performs image processing on the image data measured by the hand camera 7 as in the case of the fixed camera 3, recognizes the feature part of the target object, and determines the position or position and orientation of the feature part. It can be calculated three-dimensionally. For the hand camera 7, for example, a hand camera coordinate system V representative of the position and orientation of the hand camera 7 is virtually set at the lens principal point position of one of a plurality of cameras constituting the stereo camera. Yes. The measured value of the position or position / orientation calculated based on the image data of the hand camera 7 is output as a value based on the hand camera coordinate system V.

  The marker 8 is a reference member that is attached to the workpiece placement area 5 when the robot system 10 is calibrated, and a circular mark 81 is printed in black as a feature that can be measured from the fixed camera 3 and the hand camera 7. In the present embodiment, the printing accuracy and processing accuracy of the marker 8 and the installation accuracy with respect to the workpiece placement area 5 are sufficient if the mark 81 is within the field of view of the fixed camera 3 and the hand camera 7 during calibration, which will be described later. No processing or position adjustment is required. For example, the marker 8 can be mounted by a method in which a female screw portion (not shown) is provided on the upper surface of the work placement area 5 and fixed with screws.

  The marker 8 is arranged so as to cover as much as possible the range in which the features of the workpiece 6 can exist when actually picking the workpiece 6 and the movement range of the hand position that the robot hand 2 can take when picking the workpiece 6 It is desirable to do. The reason is to improve picking accuracy. For example, first, since the fixed camera 3 has an error in camera calibration, there is a systematic error corresponding to the coordinates of the measurement site. Secondly, the robot arm 1 also has a systematic error corresponding to the position and orientation because the manufacturing error of the robot arm 1, the error due to its own weight, the error due to thermal expansion, and the like exist.

  Due to the influence of the above error, the relationship between the robot coordinate system R and the fixed camera coordinate system F cannot be exactly matched. In particular, when the marker 8 is installed and calibrated in a narrow area, the angular deviation of the coordinate system becomes large, and when picking work is performed outside the calibrated area, a relatively large error is generated. Become. Therefore, for the reason described above, it is desirable that the calibration range covers the range that can be taken by the characteristic points of the workpiece 6 and the range that can be taken by the robot arm 1 as much as possible when the workpiece 6 is actually picked. Further, it is desirable that the posture of the robot arm 1 at the time of calibration is as close as possible to the posture at which the workpiece 6 is actually picked.

  FIG. 3 exemplifies design conditions that the arrangement range of the marker 8 should satisfy in relation to the movable range of the robot arm 1 or the robot hand 2. For example, as shown in the figure, when the work placement area 5 is viewed from the top, the minimum polygon surrounding the group of feature points (for example, the center of the circle) of the mark 81 holds the work 6 by the robot hand 2. It is desirable to determine so as to include the gripping center 61 at the time.

  Here, various definitions can be considered for the “grip center” depending on the shape of the robot hand and the workpiece. For example, “when the robot hand 2 acquires the workpiece 6, the robot hand 2 comes into contact with the workpiece 6 by design. It can be defined as “the center of the smallest sphere that encompasses the group”. In the robot system 10 of the present embodiment, the center of symmetry of the two contact surfaces where the finger 21 and the workpiece 6 come into contact during gripping is the gripping center 61.

  In FIG. 3, two arrows perpendicular to the periphery of each gripping center 61 are shown. These are two coordinate axes (in the hand coordinate system T of the robot hand 2 when gripping each workpiece 6 in its posture ( X axis, Y axis).

  As described above, in the present embodiment, the first visual sensor is arranged such that the measurement range includes the work placement area 5. Further, the plurality of markers 8 are arranged so that the polygon connecting the respective characteristic portions includes a range that can be taken by the gripping center 61 of the robot hand 2 that grips the work 6 in the work placement area 5.

  The marker 8 may be temporarily placed on the upper surface of the work placement area 5 as shown in FIGS. 2 and 3 only during calibration, or may be placed on the outer periphery of the work placement area 5 as illustrated in FIG. It may be always installed. Further, for example, when there is no concern about rubbing or scratching by the work 6, the mark 81 may be printed on the upper surface of the work placement area 5 and used as the marker 8.

  When the robot system 10 performs the picking operation, first, the workpiece 6 is measured by the fixed camera 3 to obtain a measurement value representing the position or position / orientation of the characteristic portion of the workpiece 6. Next, based on the measured value, the target position (command value) of the robot arm 1 for obtaining the workpiece 6 by the robot hand 2 is calculated. Based on the calculation result of the target position, the robot control unit 411 controls the robot arm 1 and causes the robot hand 2 to approach the workpiece 6 from above. When this approach operation is completed, the robot control unit 411 controls and closes the finger 21 of the robot hand 2 to grip the workpiece 6. Thereafter, the robot arm 1 and the robot hand 2 are controlled again to transport the workpiece 6 to a workpiece placement position (not shown), and the picking operation for one workpiece 6 is completed. By repeating the above operation until the work 6 is no longer in the work placement area 5, the picking work is realized.

  When the robot hand 2 approaches the workpiece 6 in the workpiece placement area 5, the workpiece 6 cannot be measured from the fixed camera 3 because the workpiece 6 is hidden by the robot arm 1 and the robot hand 2. Therefore, when the picking operation is repeatedly performed, it is necessary to take an image of the fixed camera 3 at a timing when the robot arm 1 exists outside the work placement area 5.

In order to calculate the target value of the robot arm 1 based on the measurement value obtained by measuring the workpiece 6 from the fixed camera 3 in the above-described picking work flow, the measurement value of the fixed camera 3 and the command value of the robot arm 1 are calculated. A calibration function whose relationship is obtained in advance is required. This is because the measurement value measured from the fixed camera 3 is a coordinate value or a coordinate transformation matrix expressed based on the fixed camera coordinate system F, but in order to control the robot arm 1, a command based on the robot coordinate system R is used. Because a value is necessary. In this embodiment, a coordinate transformation matrix ^R H _F representing the relative position and orientation of the robot coordinate system R and the fixed camera coordinate system F is used as a calibration function, and this coordinate transformation matrix ^R H _F is obtained by calibration.

  Hereinafter, the flow of the calibration process in the present embodiment will be described with reference to FIG. FIG. 5 shows a control procedure when the robot system 10 is calibrated. The control procedure of FIG. 5 can be stored in the ROM 42 (or other program memory described above) as a control program of the CPU 41, for example. FIG. 6 shows the relationship among each coordinate system, coordinate transformation matrix, and coordinate values used in the calibration process of this embodiment.

As a rough flow of the calibration process, steps S11 to S13 in the calibration preparation stage are preparatory work steps mainly performed by an operator, and steps S15 to S18 are calibration automatic operations that can be executed by the robot system 10 by an automatic program. It is an execution step. In calibration automatic execution of steps S15 to S18, first, it acquires the measured coordinate values ^F P [i] (the first measurement data) from the fixed camera 3 at step S15. Next, in step S16, the robot arm 1 is controlled using the calibration posture ^R H _T [i] (coordinate transformation matrix corresponding to the calibration posture data) taught in step S13. In step S <b> 17, the coordinate value ^V P [i] (second measurement data) is acquired by measurement from the hand camera 7. Then, in step S18, it calculates a coordinate transformation matrix ^R H _F as the calibration value of the objective (calibration function). That is, the coordinate value ^F P became known in the above step [i], the calibration position ^R H _T [i], and the coordinate value ^V P [i], using the previously acquired coordinate transformation matrix ^T H _V, A coordinate transformation matrix ^R H _F is calculated. The above is a rough flow of calibration.

  In addition, said 1st measurement data are the coordinate data which measured the coordinate value of at least 3 or more points of a characteristic part from the 1st visual sensor (fixed camera 3). The second measurement data is included in the measurement region of the second visual sensor (hand camera 7) for each of a plurality of calibration positions and postures indicated by the calibration posture data in the mark (feature portion) 81. Coordinate data obtained by measuring at least one feature point included in the.

  Hereinafter, each step of calibration in FIG. 5 will be described in more detail. In the following, step numbers in FIG. 5 are abbreviated as “S11” in parentheses (the same applies to the examples described later).

  First, the hand camera 7 is attached to the hand camera attaching part 22 of the robot hand 2 (S11). The robot hand 2 and the hand camera mounting portion 22 are configured so that the hand camera 7 can be mounted on the robot hand 2 (or the robot arm 1) at a predetermined mounting position with good reproducibility. For example, an abutting portion (not shown) is provided on the hand camera mounting portion 22 so that the hand camera 7 can be fixed with a screw (not shown) or the like with good reproducibility.

  Note that the hand camera 7 is always attached to the robot hand 2 when there is no fear of interference with the surroundings without removing the hand camera 7 or when the hand camera 7 is used in another work performed by the robot system 10. May be. For example, when the work 6 is to be inspected using the hand camera 7 after the work 6 is picked and placed at the work placement position, the hand camera 7 is always attached. In that case, it is not necessary to perform the work of installing the hand camera 7 in particular.

  Next, the marker 8 is placed in the work placement area 5 (S12). The requirements for arrangement at the time of installation are, for example, as shown in FIG. The N markers 8 are fixed using a marker fixing screw hole or the like provided in advance on the work placement area 5. As illustrated in FIG. 4, this work is not necessary when the marker 8 is always attached to the work placement area 5.

Next, N calibration postures ^R H _T [i] (N = 1, 2, 3,..., N) are taught to the N markers 8 (S13). Here, the i th calibration posture ^R H _T [i] is the position and posture of the robot arm 1 that can measure the i th marker 8 from the hand camera 7. For example, a robot capable of manually operating the robot arm 1 using a teaching pendant (not shown) with the posture during picking as a reference, and measuring each marker 8 while checking the image of the hand camera 7 on a display (not shown). Teach and register the position and orientation of the arm 1. This teaching need not be performed with high accuracy, and it is sufficient that one marker 8 is included in at least the measurement range of the hand camera 7.

N calibration postures ^R H _T [i] generated as a result of teaching (teaching) in step S13 are stored in the memory (ROM 42 or RAM 43) as calibration posture data. For example, when the robot arm 1 or the fixed camera 3 is replaced and then recalibration is performed, the teaching result in the past may be used as it is.

  Next, the robot arm 1 is retracted from above the work placement area 5 so that the marker 8 can be measured from the fixed camera 3 (S14).

As described above, the preparation stage performed by the operator is completed. Thereafter, the robot system 10 is automatically calibrated by performing a program operation. First, the center coordinates of the mark 81 on each marker 8 are measured from the fixed camera 3 (S15). More specifically, an imaging command is issued from the visual sensor control unit 413 to the fixed camera 3, and stereo image data captured from the fixed camera 3 is acquired. The feature position calculation unit 414 performs a known circle extraction process (when the mark is circular) on each of the acquired stereo image data, and recognizes the image coordinates of the center point of the mark 81. The coordinate values ^{F P [i] (i =} 1,2,3, ..., N) of the three-dimensional mark 81 is calculated as a fixed visual sensor measurement data (first measurement data). That is, correspondence the recognized mark 81 center point between the stereo images, the coordinate values of the three-dimensional marks 81 by the stereo method using the camera calibration parameters read from the ^{ROM42 F P [i] (i} = 1,2,3 , ..., N). Here, N is a positive integer representing the number of markers 8, and for example, N = 5 when there are five markers as shown in FIGS. Further, i is an index (serial number) for distinguishing individual markers. In the loop control in steps S16 and S17 described below, this index i is used as a variable for controlling the loop.

Thereafter, the robot arm 1 is moved to the i-th calibration posture ^R H _T [i] (S16), and the process of measuring the marker 8 from the hand camera (S17) is repeated for the number N of the markers 8. Here, the calibration attitude data storing the N calibration attitudes ^R H _T [i] generated in step S13 is read from the memory and used. Steps S101 and S102 arranged so as to narrow down steps S16 and S17 show a control structure for sequentially incrementing the index i from 1 to N according to a description format in C language, for example. is there. When the actual program is executed, the index i is initialized to 1 at the position of step S101. Further, at the position of step S102, it is determined whether or not steps S16 and S17 have been executed N times. If not executed N times, the index i is incremented and looped to step S16. The process proceeds to S18.

In the i-th calibration posture ^R H _T [i], the process of measuring the i-th marker 8 from the hand camera 7 is the same as the measurement process in the fixed camera 3 described above. However, since the field-of-view range that can be measured by the hand camera 7 is generally narrower than the field-of-view range of the fixed camera 3, in this embodiment, a marker that can be measured at once for each calibration posture ^R H _T [i]. It is assumed that the number of 8 is one. Thus, for each single measurement for the i-th mark 81 on the marker 8, it is measured relative to the hand camera coordinate system V coordinate value ^{V P} [i] (hand visual sensor measurement data) is calculated. By repeating the operation of the robot arm 1 (S16) and the measurement from the hand camera 7 (S17) N times, N sets of data are acquired.

  Finally, a calibration value calculation process is performed using the data acquired in the above process (S18). Hereinafter, the principle of the calibration process in the present embodiment will be described using mathematical expressions. Calculations corresponding to the following formulas are executed by the calibration calculation unit 412. More specifically, the calculation function of the calibration calculation unit 412 is realized by the CPU 41 executing a corresponding calculation control program.

First, the coordinate value ^R P [i] of the mark 81 with respect to the robot coordinate system R is used as the coordinate value ^F P obtained by measuring the marker from the coordinate transformation matrix ^R H _F that is an unknown calibration value and the fixed camera 3 (S15). If it describes using [i], it will become like following Formula.

Also, the coordinate value ^R P [i] is calculated from the calibration posture ^R H _T [i] when the robot arm 1 is operated (S16), and the coordinate value ^V P [ i],

となる。

It becomes.

Here, the coordinate transformation matrix ^T H _V is calibration data representing the relative position and orientation that relate the hand coordinate system T of the robot arm 1 and the hand camera coordinate system V of the hand camera 7. Since this coordinate transformation matrix ^T H _V is a fixed value matrix that does not change depending on the posture of the robot arm 1, it can be obtained in advance. In the present embodiment, it is assumed that the coordinate transformation matrix ^T H _V is obtained in advance and stored in the ROM 42.

As a general method of calculating the coordinate transformation matrix ^T H _V, may be a known hand-eye calibration method while fixing the hand camera 7 to the robot hand 2. Alternatively, the hand camera 7 may be fixed to a dedicated jig and calibrated in a state independent of the robot system 10. In this case, for example, an index with a known coordinate is provided on a dedicated jig and measured from the hand camera 7 so that the position and orientation of the hand camera coordinate system V are based on the abutting surface with respect to the hand camera mounting portion 22. Can be calibrated.

Here, since ^R P [i] is equal in the equations (6) and (7), the following equations are established from both equations.

上式（８）において、左辺の^ＲＨ_Ｆのみが未知の量（６自由度の座標変換行列）であって、その他のパラメータはすべてこれまでのステップにより既知の量である。実際のデータには誤差があるためＮ組全てのデータについて等式が厳密に成立する解は得られない。しかしながら、マーク８１の座標データが同一直線上に無い３点以上あれば、Ｎ個の式を連立して最小二乗法で解くことにより目的の校正パラメータとして座標変換行列^ＲＨ_Ｆを算出することができる。

In the above equation (8), only the left side of ^R H _F is unknown quantities (coordinate transformation matrix 6 degrees of freedom), is a known quantity by step until all other parameters this. Since there is an error in the actual data, a solution in which the equations are strictly valid cannot be obtained for all N sets of data. However, if the coordinate data of the mark 81 is three or more points that are not on the same straight line, the coordinate transformation matrix ^R H _F can be calculated as a target calibration parameter by simultaneously solving N equations using the least square method. it can.

式（９）の最小二乗法を解くには、例えば同時変換行列としての^ＲＨ_Ｆを３つの並進パラメータ（Ｘ，Ｙ，Ｚ）と３つの回転角度パラメータ（θ，φ，ψ）で記述し、式（９）左辺が最小となるようレーベンバーグ・マーカート法などを用い数値演算で求める。また、別の例としては、座標変換行列^ＲＨ_Ｆの回転行列部分を四元数（ｑ１，ｑ２，ｑ３，ｑ４）を用いて表し、四元数ベクトルの長さが１であるという制約条件のもとに、制約条件付きの最小二乗アルゴリズムを用いて解を算出してもよい。

In order to solve the least square method of Equation (9), for example, ^R H _F as a simultaneous conversion matrix is described by three translation parameters (X, Y, Z) and three rotation angle parameters (θ, φ, ψ). Equation (9) is obtained by numerical calculation using the Levenberg-Marquardt method or the like so that the left side is minimized. As another example, a constraint condition that the rotation matrix portion of the coordinate transformation matrix ^R H _F is represented by using a quaternion (q1, q2, q3, q4), and the length of the quaternion vector is 1. The solution may be calculated using a least square algorithm with constraints.

The calibration process described above, the calibration value of the objective, that is, to determine coordinates transformation matrix ^R H _F representing the relative position and orientation of the fixed camera coordinate system F between the robot coordinate system R by numerical calculation. The result calculated by the calibration calculation unit 412 as described above is stored in the ROM 42 or RAM 43 (or other storage device) as a calibration parameter file. When the above-described picking operation is performed, the coordinate value of the fixed camera coordinate system F measured by the fixed camera 3 is converted into the coordinate value of the robot coordinate system R as necessary using the stored calibration parameter file. Can do.

For example, a specific portion of the workpiece 6 is measured by the fixed camera 3 performs coordinate transformation using the coordinate transformation matrix ^R H _F, to generate a command value for controlling the position and orientation of the robot arm 1 and the robot hand 2 it can. Then, the generated command value is transmitted to the robot arm 1 and the robot hand 2 via the robot control unit 411, and for example, a picking operation for grasping and taking out the corresponding part can be performed.

  According to the robot system 10 of the present embodiment, since the mark 81 is measured from both the fixed camera 3 and the hand camera 7, it is not necessary to finely adjust the tip of the robot arm 1 and calibrate in a short time. Is possible. In addition, in the calculation of the calibration value, prior information such as the design dimension relating to the coordinate value of the mark 81 is not used, so that high-precision calibration is possible even if the position accuracy of the marker 8 itself is low.

  Further, even if the fixed camera 3 performs measurement from above and the robot hand 2 approaches the workpiece 6 from above, unlike the calibration method using only the fixed camera 3, the robot hand 2 and the robot arm 1 are used. There is an advantage that it is not affected by concealment. That is, when the mark 81 is calibrated and measured by the hand camera 7, the mark 81 can be reliably measured without being affected by the concealment. Therefore, the robot arm 1 can be operated in a posture that is close to the actual work and in a sufficiently wide range, and the measurement value for the calibration calculation can be acquired. It is possible to reduce the influence of errors and perform highly accurate calibration processing.

  Next, a system configuration at the time of calibration of the robot system 10a according to the second embodiment of the present invention will be described with reference to FIG. Compared to the first embodiment, the second embodiment can reduce the work load for calibration preparation that the operator needs to perform.

  The robot system 10a according to the second embodiment is mainly different from the robot system 10 according to the first embodiment in the following three points.

  First, a hand camera 7a that can be gripped by the robot hand 2 is provided as a hand visual sensor instead of the hand camera 7, and a hand camera stand 71a is provided.

  Second, the reference member measured by the fixed camera 3 and the hand camera 7a at the time of calibration is a marker 8a that can be freely arranged by the operator.

  Third, as a function of the CPU 41 of the control device 4, a calibration posture calculation unit 415 for calculating the calibration posture of the marker 8a arbitrarily arranged by the operator is provided.

  Other components are the same as the robot system 10, and the system configuration at the time of picking work is also the same as the robot system 10. Therefore, the same reference numerals are used for corresponding members, and detailed description thereof is omitted. Shall. Note that some data and programs stored in advance in the ROM 42 are different, which will be described later.

  The hand camera 7 a of this embodiment is a stereo camera having a shape that can be gripped by the robot hand 2. The robot hand 2 can attach and detach the hand camera 7 a by opening and closing the fingers 21. The hand camera 7 a can communicate with the control device 4 using a communication unit (not shown), can capture an image in response to a command from the visual sensor control unit 413, and can transmit image data to the control device 4. . The communication means (not shown) can be configured by a communication cable or the like disposed inside or outside the robot arm 1. In this case, a contact mechanism between the hand camera 7a and the robot hand 2 can be provided with a positioning mechanism such as a positioning pin and an electrical contact for communication in order to grip the hand camera 7a with high accuracy.

  Further, non-contact wireless communication means using electromagnetic waves can be used for inputting / outputting measurement data of the hand camera 7a to / from the control device 4. For example, IEEE802.11 and IEEE802.15 wireless network standards can be used for data transfer by wireless communication. In this case, the wireless network interface of the standard can be provided as a part of the interface unit 44, or can be arranged as an external device connected to the network 50.

  The hand camera stand 71 a is a base for supporting the hand camera 7 a while the robot hand 2 is not wearing the hand camera 7 a, and is provided within the movable range of the robot arm 1. For example, as shown in FIG. 7, a shape having a square hole shape may be a base that can support only the main body portion of the hand camera 7 a while avoiding the lens portion of the hand camera 7 a.

  The robot system 10a can automatically attach the hand camera 7a to the robot hand 2 by controlling the robot arm 1 and the robot hand 2 with the hand camera 7a installed on the hand camera stand 71a. Conversely, the operation of returning the hand camera 7a to the hand camera mount 71a can also be performed automatically. If the position of the hand camera stand 71a is determined at a fixed position with respect to the robot coordinate system (R), the operation of mounting the hand camera 7a as a part of the system program or returning to the hand camera stand 71a is programmed. I can keep it. Further, when the robot system 10a is installed, for example, an operation in which an administrator or an operator uses the teaching pendant (not shown) to attach the hand camera 7a or to return the hand camera 7a to the hand camera stand 71a. Good.

  In the present embodiment, the marker 8a can be installed in any position and orientation on the work placement area 5 by an operator who performs the installation. A circular mark 81a is printed on the marker 8a as in the first embodiment. It is desirable that the marker 8a is not displaced by vibration when the robot arm 1 is operated in a calibration automatic execution step described later. For example, a mechanism for simply fixing the marker 8a to the work placement area 5 using an adhesive or a magnetic force. It may be adopted.

  The calibration posture calculation unit 415 is provided as a control function of the CPU 41. The calibration posture calculation unit 415 can be realized by the CPU 41 executing a program stored in the ROM 42, for example. The calibration posture calculation unit 415 shoots with the fixed camera 3, calculates the calibration posture based on the measurement result of the mark 81 a calculated by the feature position calculation unit 414, and outputs the calibration posture to the robot control unit 411. Details of the processing contents will be described later.

  Next, the flow of calibration using the robot system 10a will be described with reference to the flowchart of FIG. The control procedure of FIG. 8 can be stored in the ROM 42 (or other program memory described above) as a control program for the CPU 41, for example. FIG. 9 shows the relationship among each coordinate system, coordinate transformation matrix, and coordinate values used in the calibration process of this embodiment.

  In the present embodiment, as a general flow, the step of the preparatory work mainly performed by the operator is only step S21, and the automatic calibration execution process of steps S22 to S29 is executed by the robot system 10a by an automatic program. Can do. In the automatic calibration execution processing in steps S22 to S29, the following two points are mainly different from the first embodiment.

  First, the robot hand 2 grips the hand camera 7a by automatic control (S22). Secondly, the calibration posture of the hand camera 7a is automatically calculated (S26) based on the measurement (S24) result from the fixed camera 3 and the calibration initial value (obtained in S25). It is a point that teaching by a person becomes unnecessary.

  Hereinafter, each step of FIG. 8 will be described in detail with an emphasis on the differences from the first embodiment.

  First, the worker installs the marker 8a in the work placement area 5 (S21). As described above, it is desirable to install in as wide a range as possible, but any number of three or more points that are not on the same straight line may be installed at any position in the work placement area 5, and the placement positions may be set in advance. There is no need to design strictly. When the vibration of the work placement area 5 when the robot arm 1 is driven is sufficiently small, the marker 8a may be simply placed in the work placement area 5 or temporarily fixed with an adhesive or magnetic force as described above. May be.

  The preparatory work performed by the operator for the robot system 10a is as described above. From the next step, calibration processing is automatically executed by executing a program stored in the ROM 42 in advance. When the automatic program for calibration is activated, the robot system 10a controls the robot arm 1 and the robot hand 2 to acquire the hand camera 7a supported by the hand camera mount 71a (S22).

  Next, the robot arm 1 is controlled to retract the robot arm 1 to a position where the robot arm 1 and the robot hand 2 do not block the line of sight of the fixed camera 3 for measuring the workpiece placement area 5 from the fixed camera 3 (S23). ). When the hand camera stand 71a is installed at a position where the robot arm 1 and the robot hand 2 do not appear in the fixed camera 3 at the stage of step S22 described above, the robot arm 1 is moved especially at this step. There is no need to let them.

Next, a circular mark 81a is measured on the fixed camera 3 markers 8a, the number N and the coordinate values of the marks ^{81a F P [i] (i} = 1,2,3, ..., N) to get ( S24). After the visual sensor controller 413 acquires the stereo image by performing an imaging command to the fixed camera 3, characterized position calculating unit 414 calculates the number N and the coordinate value ^{F P} [i] of the marks 81a performs image processing. Compared to Example 1, since the number N of the mark 81a has become any worker, not only the feature position calculating unit 414 coordinates ^{F P} [i] In this step, also the number N of the mark 81a It is obtained by image analysis. This step process of calculating a coordinate value ^{F P} [i] in three dimensions by performing a circular extraction processing on the image data are the same as in Example 1, also, obtains the number N of the mark 81a in the process Since it is easy, the details are omitted.

Next, the calibration posture calculation unit 415 acquires the calibration initial value ^R H _F0 by reading it from the ROM 42 (S25). Here, the calibration initial value ^R H _F0 is a provisional value of a coordinate transformation matrix representing the relationship between the robot coordinate system R and the fixed camera coordinate system F, and is stored in advance in the ROM 42 as a setting file. As the calibration initial value ^R H _F0 , for example, a value representing a relative relationship between the robot coordinate system R and the fixed camera coordinate system F when there is no processing or assembly error based on a design value when designing the robot system 10a. Just enter it. As another example, after the fixed camera 3 or the robot arm 1 is exchanged with respect to the robot system 10a that has completed the calibration work, the calibration result at the previous calibration is used as the calibration initial value ^R H _F0. Also good.

That is, the calibration initial value ^R H _F0 is initial value data set in advance with respect to the relationship between the robot coordinate system R and the fixed camera coordinate system F of the first visual sensor (fixed camera 3). In the present embodiment, calibration posture data is calculated based on the first measurement data measured by the first visual sensor (fixed camera 3) and the calibration initial value ^R H _F0 which is initial value data.

Further calibration orientation calculation unit 415, based on the calibration initial value ^R H _F0 acquired coordinate values ^F P [i] and the mark number N of the acquired mark 81a, and at step S25 in step S24, N number of calibration position ^R H _T [i] (i = 1, 2, 3,..., N) is calculated (S26). The N calibration postures ^R H _T [i] are calculated as the position and posture of the robot arm 1 that can photograph the mark 81a from the hand camera 7a.

As a specific method, for example, the measurement visual field of the hand camera 7a with respect to the hand coordinate system T is calculated from the design value, the representative point Q near the center of the measurement visual field is determined, and the coordinate value ^T Q in the hand camera coordinate system V is determined. Information is stored in advance in the ROM 42 (see FIG. 9). Here, have been marked 81a coordinate value ^F P [i] of the obtained in step S24, the use of calibration initial value ^R H _F0 obtained in step S25, the coordinates of the center point P of the mark 81a in the robot coordinate system R the value ^R P [i] can be calculated as follows.

上式（１０）の演算により、ロボット座標系Ｒにおけるマーク８１ａの中心点Ｐの座標値が分かるので、手先に設定された計測視野代表点Ｑの座標値が、マーク８１ａの中心点Ｐに一致するようにロボットアーム１を制御すればよい。任意の姿勢^ＲＨ_Ｔ［ｉ］をロボットアーム１に取らせた場合の、ロボット座標系Ｒに対する計測視野代表点Ｑの座標値^ＲＱ［ｉ］は以下のようになる。

Since the coordinate value of the center point P of the mark 81a in the robot coordinate system R is known by the calculation of the above equation (10), the coordinate value of the measurement visual field representative point Q set at the hand matches the center point P of the mark 81a. The robot arm 1 may be controlled as described above. When the robot arm 1 is in an arbitrary posture ^R H _T [i], the coordinate value ^R Q [i] of the measurement visual field representative point Q with respect to the robot coordinate system R is as follows.

そして、計測視野代表点Ｑがマーク８１ａの中心点Ｐに一致するためには

In order for the measurement visual field representative point Q to coincide with the center point P of the mark 81a,

であればよいから、式（１０）、式（１１）、式（１２）より

From equation (10), equation (11), and equation (12)

を満たすようにＮ個の校正姿勢^ＲＨ_Ｔ［ｉ］を定めればよい。なお、式（１３）の制約だけでは校正姿勢は一意には定まらないので、実際に適用する際には例えば「ロボットハンド２の掌部が鉛直下向きになる」等の制約条件を付して校正姿勢が定まるアルゴリズムとして演算するとよい。その際のロボットアーム１やロボットハンド２の姿勢に関する制約条件は任意に設計し得る事項である。しかしながら、前述のように、実作業（ピッキング）時のロボットアーム１およびロボットハンド２の姿勢から乖離した姿勢にならないように定めることが精度の観点からは望ましい。校正姿勢演算部４１５は上記の演算を行い、Ｎ個の校正姿勢^ＲＨ_Ｔ［ｉ］をロボット制御部４１１に出力する。

N calibration postures ^R H _T [i] may be determined so as to satisfy the above. Note that the calibration posture is not uniquely determined only by the constraint of equation (13), so when actually applied, for example, calibration is performed with a constraint such as “the palm of the robot hand 2 is vertically downward”. It may be calculated as an algorithm that determines the posture. The constraint conditions regarding the posture of the robot arm 1 and the robot hand 2 at that time are matters that can be arbitrarily designed. However, as described above, it is desirable from the viewpoint of accuracy that the posture is not deviated from the postures of the robot arm 1 and the robot hand 2 during actual work (picking). The calibration posture calculation unit 415 performs the above calculation, and outputs N calibration postures ^R H _T [i] to the robot control unit 411.

Thereafter, second measurement data is acquired by the hand camera 7a. That is, the robot controller 411 controls and moves the robot arm 1 based on the calibration posture ^R H _T [i] calculated in step S26 (S27), and the coordinate value ^V P [i of the mark 81a from the hand camera 7a. ] Is repeated N times (S28). Note that steps S201 and S202 arranged so as to narrow down steps S27 and S28 are similar to the first embodiment described above, and the control structure for sequentially incrementing the index i from 1 to N conforms to the description format in C language, for example. It is shown.

At this time, since the calibration posture ^R H _T [i] calculated in step S26 is a value including an error in the calibration initial value ^R H _F0 , the measurement visual field representative point Q exactly matches the mark 81a at each measurement. do not do. However, if the measurement visual field of the hand camera 7a is designed to be sufficiently wide with respect to the error of the calibration initial value ^R H _F0 , the mark 81a is always included in the measurement visual field of the hand camera 7a, and calibration is performed without any problem. be able to.

In step S29, a coordinate transformation matrix ^R H _F is calculated as a target calibration value (calibration function). That is, the mark coordinates ^F P [i] measured from fixed camera 3, a calibration position ^R H _T [i] calculated in step S26, the coordinate value ^V P of the mark 81 measured from the hand camera 7a in step S28 [ i] is used to calculate the coordinate transformation matrix ^R H _F. The calculation process for calculating the coordinate transformation matrix ^R H _F is the same as that in the first embodiment, and a description thereof will be omitted. When the coordinate transformation matrix ^R H _F is calculated using an iterative calculation algorithm that requires an initial value such as the Levenberg-Markert method, the calibration initial value ^R H _F0 obtained in step S25 is again set to a numerical value. It can be used as an initial value for calculation.

By executing the above processing, the coordinate transformation matrix ^R H _F can be calculated as a calibration value representing the relationship between the robot coordinate system R and the fixed camera coordinate system F. That is, the calibration necessary for positioning the robot arm 1 with high accuracy can be performed based on the measurement value of the fixed camera 3.

In the present embodiment, during calibration the calibration position ^R H _T to move [i] the robot arm 1, on the basis of the calibration initial value ^R H _F0 coordinate values obtained by measuring the mark 81a from the fixed camera 3 ^F P [i] It was set as the structure to calculate. With this characteristic configuration, the operator can freely arrange the markers 8 without rigorously designing the markers 8 in advance, and calibrate by automatically operating the robot system 10a without teaching the robot arm 1 after the placement. can do. Accordingly, it is possible to greatly reduce the work load of the operator when starting up the apparatus. Also, even if you want to calibrate multiple robot systems with different workpiece supply areas in the same factory, you can easily use the same marker by changing the arrangement, so prepare calibration jigs individually. There is no need.

  In the present embodiment, the hand camera 7a can be automatically attached and detached by opening and closing the robot hand 2. This characteristic configuration reduces the work load for the operator to attach and detach the hand camera 7a when calibrating the robot system 10a. Further, by supporting the hand camera 7a by the finger 21 of the robot hand 2 that comes into contact with the workpiece during the picking operation, the influence of the processing error of the robot hand 2 can be reduced, and calibration for picking can be performed with high accuracy. . Furthermore, the viewpoint of the hand camera 7a is arranged on the palm of the robot hand 2, and the hand camera 7a is directly opposed to the marker 8a as compared with the case where a camera is provided on the wrist or the like of the robot hand 2. Measurement can be executed and high-precision calibration can be performed.

  Next, Embodiment 3 of the present invention will be described. In the first and second embodiments, the fixed visual sensor, the hand visual sensor are separately provided as separate fixed cameras 3, and the hand cameras 7 and 7a. In contrast, in the third embodiment, the camera is shared by attaching and detaching the fixed visual sensor (stereo camera 3b) to the hand of the robot arm 1 and temporarily using it as the hand visual sensor. . Hereinafter, this embodiment will be described with emphasis on the differences between the first embodiment and the second embodiment.

  FIGS. 10A and 10B show the configuration of the robot system 10b according to the third embodiment, and the relationship between each coordinate system, coordinate values, and coordinate transformation matrix used in the present embodiment. Since the control device 4 is the same as that of the second embodiment, the illustration thereof is omitted.

  As shown in FIG. 10A, as a visual sensor, the robot system 10b requires only one stereo camera 3b. This stereo camera 3b has a stereo camera mounting portion 31b, and is normally fixedly supported in space by a fixed camera support member (not shown) via the stereo camera mounting portion 31b (first arrangement position). The stereo camera 3b is arranged facing the work placement area 5, and can measure the work 6 and the mark 81a. That is, the stereo camera 3b is arranged at an arrangement position having a fixed positional relationship with respect to the installation surface of the gantry 9 that supports the robot arm 1 independently of the operation of the robot arm 1. Similar to the example.

  The stereo camera mounting portion 31b has a positioning mechanism such as an abutment reference surface and positioning pins and a fixing mechanism such as a screw, for example, and is configured to be detachable from the fixed camera support member with good reproducibility.

  In addition, the stereo camera mounting portion 31 b has a shape that can be mounted on the robot hand 2. As shown in FIG. 10B, by supporting the stereo camera 3b at the tip of the robot hand 2, the stereo camera 3b can be temporarily used as a hand camera (7b in the second embodiment) (second embodiment). Positions). In order to position the stereo camera 3b at a predetermined position at the tip of the robot hand 2, for example, a technique such as gripping or screwing with the robot hand 2 can be used.

  The calibration flow in the present embodiment will be described with reference to the flowchart of FIG.

First, the operator installs the marker 8 in the work placement area 5 (S31), and retracts the robot arm 1 from above the work placement area 5 (S32). Next, the first measurement data is acquired by the stereo camera 3b fixed in space as a fixed camera. That is, a mark 81 on the marker 8 is measured by the stereo camera 3b, to measure the coordinates ^F P [i] and the mark number N (S33). The contents of the steps so far can be implemented by the same processing as in the second embodiment.

  Next, the fixed stereo camera 3b is removed from a fixed camera support member (not shown) and replaced with the tip of the robot arm 1 (S34). In subsequent steps S35 to S39, the second measurement data is acquired using the stereo camera 3b as the above-mentioned hand camera 7b, and a calibration operation similar to that in the above-described second embodiment is performed to obtain a target calibration function. Note that the contents of these steps S35 to S39, S301, and S302 are the same as those of steps S25 to S29, S201, and S202 of the second embodiment, and thus detailed description thereof is omitted here.

When the calibration process as described is completed above the calibration value of the objective, i.e., to store the coordinate transformation matrix ^R H _F representing the relative position and orientation of the fixed camera coordinate system F between the robot coordinate system R as calibration parameter file. Further, the stereo camera 3b is attached again to the fixed camera support portion (not shown) via the stereo camera attachment portion 31b. In the subsequent actual work such as picking, the coordinate values of the fixed camera coordinate system F measured by the stereo camera 3b can be converted into the coordinate values of the robot coordinate system R as necessary using the calibration values in the calibration parameter file. it can.

  As described above, in this embodiment, normally, the stereo camera 3b used as a fixed camera is temporarily used as a calibration hand camera 7b, so that a hand camera used only for calibration is separately provided. No need to prepare. That is, the robot apparatus can be calibrated at a lower cost than when a hand camera dedicated to calibration is prepared.

  As mentioned above, although Embodiment 1-3 of this invention was demonstrated, all the above-mentioned structures are only illustrations, In the case of implementation of this invention, a person skilled in the art can change suitably in the range which does not deviate from the meaning of this invention. is there.

  For example, although a 6-axis vertical articulated robot arm has been illustrated as the robot arm 1, various types of robot arms such as a 7-axis robot arm, a parallel link structure, and a linear motion mechanism may be used. Can do.

  For example, the hand cameras 7 and 7a have been described as being attached to the wrist or palm of the robot hand 2. However, the hand visual sensor may be attached interchangeably by removing the robot hand once. Further, the hand vision sensor may be built in the hand and configured not to be attached or detached.

  Further, for example, the stereo camera is exemplified as the fixed visual sensor and the hand visual sensor, but the visual sensor that can be used is not limited to this, and various visual sensors that can measure the coordinate values of the feature points viewed from the camera coordinate system can be used. For example, a laser sensor or various pattern lights on a slit are projected onto an object to perform three-dimensional measurement based on the principle of trigonometry, or a visual sensor that obtains a distance image by the TOF (Time Of Flight) method. It may be used. It is also possible to use a normal monocular camera after calibrating information in the depth direction using a known external parameter calibration method in advance.

  In addition, for example, the mark 81 and the mark 81a are described as black circular marks for the sake of simplicity. However, the shape and color are not limited thereto, and the coordinates of the feature portion can be specified from the fixed visual sensor and the hand visual sensor. It doesn't matter if it exists. Various marks such as various polygons and checker marks in which black portions and white portions are arranged can be used.

  Furthermore, since the markers 8 and 8a only need to be able to measure the coordinates of the feature portion with a visual sensor, the feature points of the workpiece itself to be picked may be used for calibration as the markers 8 and 8a. For example, the vertices of the rectangular parallelepiped workpiece 6 as shown in FIG. 2 may be used in place of the marks 81 and 81a for calibration from the fixed camera 3 and measurement from the hand camera 7 or 7a. Absent.

  Further, the case where the robot arm 1 is controlled by an automatic program when taking a plurality of calibration postures has been described as an example, but the automatic posture is not executed and the calibration posture is captured by an inching operation using a teaching pendant. It may be allowed. In that case, manual operation is required every time calibration is performed, but since strict fine adjustment is not necessary, it can be performed in a relatively short time, and the effect of the accuracy of the present invention is obtained when an automatic program is used. Can be enjoyed as well.

Further, in each of the above-described embodiments, for the sake of simplicity, the example in which the relationship between the hand coordinate system T and the hand camera coordinate system V is known has been described. However, even when the coordinate transformation matrix ^T H _V is unknown, the coordinate transformation matrix ^T H _V and the coordinate transformation matrix ^R H _F that is the target calibration value can be calculated simultaneously. In that case, the two coordinate transformation matrices ^R H _F and ^T H _V in Equation (9) are unknowns. Since each coordinate transformation matrix has six degrees of freedom, a total of twelve parameters are obtained. However, by sufficiently increasing the number N of calibration postures, twelve parameters that minimize the evaluation formula of equation (9) Can be obtained by the method of least squares.

  Further, in each of the above-described embodiments, the case where the number of calibration postures is the same (N) as the markers 8 and 8a has been described in order to simplify the description. In applying the present invention, for example, measurement by a hand camera may be performed from a plurality of calibration postures with different angles with respect to one marker. Further, a plurality of markers may be arranged so as to be included in the field of view of the hand camera for one calibration posture, and a plurality of coordinate values may be acquired by one measurement from the hand camera. In this case, for example, after replacing the above index i with the index j for the calibration posture and the index k for the mark, Equation (8) is obtained for each combination of j and k measured from the hand visual sensor. One equivalent equation can be established. Then, as in the first to third embodiments, the least squares solution can be obtained by simultaneous equations for the number of acquired data.

  Further, in each of the above-described embodiments, an example has been described in which the coordinate transformation matrix representing the relationship between the robot coordinate system R and the fixed camera coordinate system F is defined as the calibration function. The calibration function only needs to be a function that can express the relationship between the measurement value of the fixed camera 3 and the command value of the robot arm 1, and an expression form other than the coordinate transformation matrix may be used. For example, the calibration function may be expressed by a three-dimensional vector representing translation and a Euler angle having three degrees of freedom representing rotation. As another example, a function in which the relationship between the measurement coordinate value of the fixed camera 3 and the command value of the robot arm 1 is directly expressed by a polynomial may be used as the calibration function. In particular, when the command value of the robot arm 1 and the coordinate value of the fixed camera 3 are expressed by a second-order or higher order M-order polynomial, the error outside the calibration range becomes an M-order function as the error moves away from the calibration range. growing. Therefore, with this configuration, a great improvement in calibration accuracy can be expected due to the feature that calibration can be performed by operating the robot in a posture close to that during actual work and in a sufficiently wide range.

  The robot calibration operation of the present invention is executed by, for example, a computer (for example, CPU 41) of the control device 4 reading out a robot control program including calibration control for realizing the measurement control operations of the above embodiments from a program memory (for example, ROM 42). Can be realized. In this case, the robot control program itself read from the recording medium realizes the functions of the above-described embodiments, and the robot control program itself and the recording medium recording the robot control program constitute the present invention. become.

  In the above embodiment, the case where the computer-readable recording medium is the ROM 42 has been described. However, the present invention is not limited to this configuration. For example, the program for realizing the present invention may be recorded on any recording medium as long as it is a computer-readable recording medium. As a recording medium for mounting the robot control program on the apparatus, an external storage device (not shown) other than the ROM 42 in FIG. 1 may be used. For example, as a computer-readable recording medium, a flexible disk, a hard disk, various optical disks, a magneto-optical disk, a magnetic tape, a rewritable nonvolatile memory (for example, a USB memory), a ROM, or the like can be used as the recording medium. . The robot control program in the above embodiment may be stored in a file server or the like, downloaded via the network 50, and executed by the CPU 41. The network 50 may be a public network such as the Internet, or a private network such as a so-called intranet closed to a manufacturing plant where the robot apparatus is installed or an operating organization (such as a company).

  The embodiment of the present invention is not limited to the implementation of the functions of the above-described embodiments by executing the robot control program code read by the computer. For example, an OS (operating system) operating on a computer performs part or all of actual processing based on an instruction of the program code, and the functions of the above-described embodiments may be realized by the processing. included.

  Furthermore, the program code read from the recording medium may be written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. This includes a case where the CPU of the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code, and the functions of the above embodiments are realized by the processing.

  In the above-described embodiment, the case has been described in which image processing is performed by a computer executing a robot control program recorded on a recording medium such as an HDD. However, the present invention is not limited to this. Part or all of the functions of the control unit that operates based on the robot control program may be implemented in the form of a dedicated LSI such as an ASIC or FPGA.

  DESCRIPTION OF SYMBOLS 1 ... Robot arm, 2 ... Robot hand, 3 ... Fixed camera, 4 ... Control apparatus, 6 ... Work, 7, 7a ... Hand camera, 8 ... Marker, 10 ... Robot system, 41 ... CPU, 42 ... ROM, 43 ... RAM, 44 ... interface unit, 50 ... network, 81, 81a ... mark, 411 ... robot control unit, 412 ... calibration calculation unit, 413 ... visual sensor control unit, 414 ... feature position calculation unit.

Claims (10)

A robot arm,
Have a proximal end and a predetermined positional relationship of the robot arm, measures the measurement range including the movable range of the robot arm, the first visual sensor which is not supported by the robot arm,
A second visual sensor supported by the robot arm;
A control device for controlling the position and orientation of the robot arm,
The controller is
First measurement data obtained by measuring a plurality of characteristic portions provided in the measurement range by the first visual sensor;
Second measurement data obtained by the robot arm taking a plurality of positions and orientations and measuring the characteristic portion by the second visual sensor;
Calibration posture data corresponding to each of the plurality of positions and postures of the robot arm;
And calculating a calibration value that relates the measured value of the first visual sensor and the command value given to the robot arm. 2. The robot apparatus according to claim 1, wherein the control device gives the robot arm using the calculated calibration value when controlling the position and orientation of the robot arm based on the measurement value of the first visual sensor. 3. A robot apparatus that generates a command value. 3. The robot apparatus according to claim 1 , wherein the control device sets an initial value set in advance with respect to the first measurement data and a relationship between a coordinate system of the robot arm and a coordinate system of the first visual sensor. The calibration posture data is calculated based on the data, and the robot arm takes a plurality of positions and postures based on the calculated calibration posture data, and performs control for measuring the characteristic portion by the second visual sensor. A robotic device characterized by 4. The robot apparatus according to claim 1, wherein the first measurement data is coordinate data obtained by measuring coordinate values of at least three points of the feature portion from the first visual sensor, and The second measurement data is coordinate data obtained by measuring at least one point of the feature part included in the measurement region of the second visual sensor for each of the plurality of positions and orientations. And the said control apparatus performs control which calculates the said calibration value by matching the coordinate data of said 1st measurement data and said 2nd measurement data, The robot apparatus characterized by the above-mentioned. 5. The robot apparatus according to claim 1, wherein the controller further includes a hand of the robot arm in addition to the first measurement data, the calibration posture data, and the second measurement data. A robot apparatus which performs control for calculating the calibration value using calibration data relating a coordinate system and a coordinate system of the second visual sensor. 6. The robot apparatus according to claim 4 or 5 , wherein the robot arm is provided in the robot arm and performs work on the workpiece, and the workpiece provided in a movable range of the robot hand moved through the robot arm. The first visual sensor is arranged such that the measurement range includes the work placement area, and a polygon connecting a plurality of the characteristic portions is formed in the work placement area. A robot apparatus comprising a plurality of the characteristic portions arranged to include a range that can be taken by a gripping center of the robot hand that grips the workpiece. The robot apparatus according to claim 6 , wherein the robot hand grips the second visual sensor. And the robot arm, said to have a predetermined positional relationship with the base end portion of the robot arm, measures the measurement range including the movable range of the robot arm, the first visual sensor which is not supported by the robot arm In a control method of a robot apparatus, comprising: a second visual sensor supported by the robot arm; and a control device that controls a position and orientation of the robot arm.
A first measurement step in which the control device measures a plurality of characteristic portions provided in the measurement range by the first visual sensor to obtain first measurement data;
Based on the calibration posture data corresponding to each of the plurality of positions and orientations of the robot arm , the control device obtains a plurality of positions and orientations, and causes the second visual sensor to measure the characteristic portion. A second measurement step for obtaining second measurement data;
A calibration value that relates the measurement value of the first visual sensor and the command value to be given to the robot arm using the first measurement data, the calibration posture data, and the second measurement data by the control device. A calibration process for calculating
A control method for a robot apparatus, comprising: A program for causing a computer to execute the control method according to claim 8. A computer-readable recording medium storing the program according to claim 9.

Images