JP5729219B2 - Method for coupling camera coordinate system and robot coordinate system of robot control system, image processing apparatus, program, and storage medium - Google Patents

Description

  The present invention relates to a method for combining a camera coordinate system and a robot coordinate system of a robot control system, an image processing apparatus, a program, and a storage medium.

  A conventional system that combines a camera coordinate system and a robot coordinate system will be described with reference to FIG. The system includes a handling robot, a robot control device that controls the handling robot, and a two-dimensional visual device that detects the position of a workpiece held by the handling robot. As shown in FIG. 16, the two-dimensional visual device 90 recognizes the workpiece position of the workpiece W on the workpiece table 150 based on the captured image obtained by imaging the workpiece W with the camera 110, and uses the position information of the workpiece W as a handling robot. The information is transmitted to 120 robot controllers 130. Based on the position information of the workpiece W, the robot control device 130 controls the handling robot 120 so that the hand tool 140 provided at the tip of the robot arm holds the specified position of the workpiece W.

  Such a system is generally correct from the two-dimensional visual device 90 to the robot controller 130 because the camera coordinate system (two-dimensional) of the camera 110 is different from the robot coordinate system (three-dimensional) as shown in FIG. In order to issue an instruction, it is necessary to check the relative positional relationship between the two coordinate systems and combine the two coordinate systems by some method. FIG. 17 shows an example when the camera coordinate system and the robot coordinate system are viewed from above.

  In the conventional method of combining the camera coordinate system and the robot coordinate system, as shown in FIG. 18, first, a hand tool (for example, an adsorber or a gripper) attached to the arm tip of the handling robot is removed, and the dedicated jig 160 is removed. Is precisely positioned and attached to the reference point of the tool. Next, the dedicated jigs 170 are respectively attached to a plurality of reference positions of the work table 150. The camera coordinate values of these reference positions are checked in advance and stored in a robot control device (not shown). The handling robot 120 is moved, the dedicated jigs 160 and 170 are accurately touched, and the robot coordinate value in the touched state is confirmed on a display of a robot control device (not shown). By acquiring corresponding points between such robot coordinate values and camera coordinate values at several locations, and using the obtained robot coordinate values for each reference value and the camera coordinate values for the reference value, the robot controller calculates them. The relative relationship between the two coordinate systems can be known (prior art 1). Conventionally, there are various methods for obtaining the reference position and calculating.

Note that Patent Documents 1 and 2 are known at the time of filing this application.
In Patent Document 1, a visual sensor is attached to the tip of a robot arm, and the robot at each point when the jig is placed at two different points while the jig is accurately grasped by the robot hand. By performing operations such as obtaining coordinate values, the relationship between the camera coordinate system and the robot coordinate system is obtained from the obtained coordinate values and the like (Prior Art 2).

  In the method proposed in Embodiment 3 of Patent Document 2, the robot grasping device grasps the recognition target object, and in this state, the first time is within the imaging range of the visual sensor arranged at a position other than the robot. Teaches the position after moving the recognition object. The second time teaches the position after the recognition object is translated in the X (+) direction, and the third time rotates the recognition object by a predetermined angle, translates it, and translates it. The position is taught (prior art 3).

  Patent Document 3 shows an example of position detection using a television camera in the article position detection method.

JP-A-1-72208 JP-A-6-190756, paragraphs 0046 to 0054, FIGS. 7 to 10 Japanese Patent Publication No. 7-104143

  The prior art 1 has a problem that the hand tool must be removed once and a dedicated jig must be accurately attached, which requires skill for the operator. Furthermore, in the prior art 1, in order to touch the dedicated jig attached to the robot side, it is necessary to provide at least three dedicated jigs to be attached to the work table side in order to obtain an accurate relationship between both coordinate systems. It is usually necessary to provide 10 or more places for measurement work. For this reason, not only the skill and labor of the operator are required, but also there are many places where the dedicated jig is used for touching, and there is a problem that a lot of calculation processing is required.

The prior art 2 has a problem that it is necessary to accurately grip the jig with the robot hand, and it takes time and effort to perform the operation of releasing the grip of the jig at two different points.
In the third embodiment of the prior art 3, there is a problem that a teaching work is required every time the recognition target object gripped by the handling robot is moved, which is troublesome.

  The object of the present invention is to solve the above-mentioned problems, and it is not necessary for the handling robot to grip the recognition target jig accurately, and it is possible to rotate and move linearly while holding the recognition target jig. By combining the camera coordinate system and the robot coordinate system, it is possible to easily obtain the tool axis center of the hand tool and the tilt of the robot coordinate system and the camera coordinate system by obtaining frame differences from the continuous images acquired when A camera coordinate system for a camera and a robot coordinate system for a robot, an image processing apparatus, a program, and a storage medium are provided.

  In order to solve the above problems, the invention of claim 1 is a camera of a robot control system comprising a hand tool provided at the tip of a robot via a tool axis, and a fixed camera for imaging work of the hand tool. In a method of combining a coordinate system and a robot coordinate system, a hand tool is rotated around a tool axis at the tip of a robot, and a moving image of a recognition target jig held by the hand tool (hereinafter referred to as a rotating moving image) is fixed camera. A first step of determining a closed curve from a trajectory drawn by a characteristic part of the recognition target jig based on a frame difference obtained from the rotational moving image and calculating a center point of the closed curve in a camera coordinate system; , A coordinate axis of the robot coordinate system, and at least one coordinate axis (hereinafter referred to as an inclination) of two coordinate axes constituting a coordinate plane that can intersect the tool axis. The hand tool holding the recognition target jig is linearly moved along the measurement coordinate axis), and a moving image of the recognition target jig (hereinafter referred to as a linear movement moving image) is acquired by the fixed camera. A second step of calculating an inclination index parameter of the inclination measurement coordinate axis with respect to the camera coordinate system based on the linear moving moving image; a camera coordinate system and a robot coordinate based on the center point of the closed curve and the inclination index parameter; The gist of the present invention is a method of combining a camera coordinate system and a robot coordinate system of a robot control system, comprising a third step of calculating a relative positional relationship between the systems.

  In this specification, the robot coordinate system includes both a coordinate system (base coordinate system) based on the base (base) of the robot and a coordinate system (user coordinate system) set by the user at an arbitrary position. It is. Both are orthogonal coordinate systems. The camera coordinate system is a two-dimensional orthogonal coordinate system developed within the imaging range of the camera.

  According to a second aspect of the present invention, in the first aspect, the closed curve is an ellipse, and in the second step, the two coordinate axes of the robot coordinate system are set as inclination measurement coordinate axes, and the inclination index parameter is set to the two inclination measurement coordinate axes. A vector based on the coordinate values of the start point and the end point in the camera coordinate system when the hand tool is moved along the line.

According to a third aspect of the present invention, in the first aspect, the closed curve is a circle, and in the second step, one coordinate axis of the robot coordinate system is set as an inclination measurement coordinate axis, and the inclination index parameter is set to the one inclination measurement coordinate axis. along, at the time of the hand tool is moved, characterized in that the angle between the straight line trajectory and inclination measuring axes.

A fourth aspect of the present invention is characterized in that, in any one of the first to third aspects, the characteristic portion of the recognition target jig is a protrusion provided at one end of the recognition target jig. To do.
The invention of claim 5 rotates a hand tool around a tool axis at the tip of a robot, acquires a moving image of a recognition target jig held by the hand tool (hereinafter referred to as a rotating moving image) from a fixed camera, First calculation means for calculating a closed curve from a trajectory drawn by a characteristic part of the recognition target jig based on a frame difference obtained from the rotating moving image, and calculating a center point of the closed curve in a camera coordinate system; and robot coordinates The recognition target jig is held so as to be along at least one coordinate axis (hereinafter referred to as an inclination measurement coordinate axis) of two coordinate axes constituting a coordinate plane that can intersect the tool axis. The hand tool is moved linearly, a moving image of the recognition target jig (hereinafter referred to as a linear moving moving image) is acquired by the fixed camera, and based on the linear moving moving image, Second calculating means for calculating an inclination index parameter of the inclination measurement coordinate axis with respect to the mela coordinate system; and a third position calculating a relative positional relationship between the camera coordinate system and the robot coordinate system based on the center point of the closed curve and the inclination index parameter. The gist of the present invention is an image processing apparatus including a calculation unit.

  The invention of claim 6 is the invention according to claim 5, wherein the closed curve is an ellipse, and the second calculation means uses two coordinate axes of the robot coordinate system as inclination measurement coordinate axes, and uses the two inclination measurement coordinate axes as the inclination index parameters. A vector based on the coordinate values of the start point and the end point in the camera coordinate system when the hand tool is moved along the line is calculated.

The invention according to claim 7 is the invention according to claim 5, wherein the closed curve is a circle, and the second calculation means uses one coordinate axis of the robot coordinate system as an inclination measurement coordinate axis, and uses the one inclination measurement coordinate axis as the inclination index parameter. along, at the time of the hand tool is moved, and calculates an angle between the straight line trajectory and inclination measuring axes.

According to the invention of claim 8, the computer rotates the hand tool around the tool axis at the tip of the robot, and the moving image of the recognition target jig held by the hand tool (hereinafter referred to as a rotating moving image) is obtained from the fixed camera. First calculating means for acquiring a closed curve from a trajectory drawn by a characteristic part of the recognition target jig based on a frame difference obtained from the rotational moving image, and calculating a center point of the closed curve in a camera coordinate system; The coordinate system of the recognition object is aligned along at least one coordinate axis (hereinafter referred to as an inclination measurement coordinate axis) of two coordinate axes constituting a coordinate plane that can intersect the tool axis. A hand tool holding a tool is moved linearly, and a moving image of the recognition target jig (hereinafter referred to as a linear moving moving image) is acquired by the fixed camera, and the linear moving motion is acquired. Second calculation means for calculating an inclination index parameter of the inclination measurement coordinate axis with respect to the camera coordinate system based on the image; and a relative positional relationship between the camera coordinate system and the robot coordinate system based on the center point of the closed curve and the inclination index parameter It is summarized as pulp program to function as a third calculation means for calculating a.

According to the ninth aspect of the present invention, a moving image of a recognition target jig held by the hand tool (hereinafter referred to as a rotating moving image) is rotated from a fixed camera by rotating a computer with a hand tool around a tool axis at the tip of a robot. First calculating means for acquiring a closed curve from a trajectory drawn by a characteristic part of the recognition target jig based on a frame difference obtained from the rotational moving image, and calculating a center point of the closed curve in a camera coordinate system; The coordinate system of the recognition object is aligned along at least one coordinate axis (hereinafter referred to as an inclination measurement coordinate axis) of two coordinate axes constituting a coordinate plane that can intersect the tool axis. A hand tool holding a tool is moved linearly, and a moving image of the recognition target jig (hereinafter referred to as a linear moving moving image) is acquired by the fixed camera, and the linear moving motion is acquired. Second calculation means for calculating an inclination index parameter of the inclination measurement coordinate axis with respect to the camera coordinate system based on the image; and a relative positional relationship between the camera coordinate system and the robot coordinate system based on the center point of the closed curve and the inclination index parameter It is summarized as third computer readable storage medium storing a Help program to function as a calculating means for calculating.

  According to the present invention, the handling robot does not need to be gripped accurately when gripping the recognition target jig, and the continuous obtained when the recognition target jig is gripped and rotated and linearly moved. By obtaining the frame differences from the images (moving images), the tool axis center of the hand tool and the inclinations of the robot coordinate system and the camera coordinate system can be easily obtained.

1 is an overall schematic diagram of a two-dimensional visual device, a robot control device, and a robot of an embodiment. (A) is the recognition object jig | tool of 1st Embodiment, (b) is the recognition object jig | tool of other embodiment. FIG. 2 is an electrical block diagram of a two-dimensional visual device, a robot control device, and a robot. (A), (b) is a flowchart of the image processing of a continuous image (moving image). (A)-(d) is explanatory drawing of the frame of a moving image, (e) is explanatory drawing which shows a frame difference result, (f) is explanatory drawing of the result of having calculated the circular arc and the curvature center of this circular arc. (A), (b) is explanatory drawing of a flame | frame at the time of the linear movement of the recognition object jig | tool. Explanatory drawing which shows the connection relation of a camera coordinate system and a robot coordinate system. (A), (b) is a flowchart of the image processing of the continuous image (moving image) of 2nd Embodiment. (A), (b) is explanatory drawing which shows the relationship between the camera coordinate system and robot coordinate system of 2nd Embodiment. FIG. 2 is an overall schematic diagram of a reference example of a two-dimensional visual device, a robot control device, and a robot. Explanatory drawing of the arrangement | positioning relationship of the illuminating device 100, the workpiece | work W, and the camera 30. FIG. Explanatory drawing of regular reflection and diffuse reflection. Explanatory drawing in case illumination light is a color complementary to a mall. Explanatory drawing in case illumination light is white light. Sectional drawing of the window glass provided with the mall. 1 is an overall schematic diagram of a conventional two-dimensional visual device, a robot control device, and a robot. Explanatory drawing which shows the relationship between a camera coordinate system and a robot coordinate system. Explanatory drawing of the connection method of the conventional camera coordinate system and robot coordinate system.

(First embodiment)
Hereinafter, a first embodiment in which a camera coordinate system and robot coordinate system coupling method, image processing apparatus, program, and storage medium of the present invention are embodied will be described with reference to FIGS. FIG. 1 shows an outline of the entire robot control system of this embodiment. The robot control system includes a robot control device 10, a handling robot (hereinafter simply referred to as a robot) 20, a camera 30 as a fixed camera, and a two-dimensional visual device 40 as an image processing device. The robot control system according to the present embodiment recognizes the position and orientation of one workpiece by recognizing the position of an object relative to a workpiece (not shown) such as a pallet, holds the workpiece on the workpiece table 60, and transports the workpiece to a target position. It is a system to do.

  The robot 20 is an articulated robot, and includes a plurality of arms 22 connected to a base 21 via a plurality of (six in this embodiment) joints. That is, the robot 20 of the present embodiment is composed of a 6-axis robot. Each joint includes a servo motor 23 and an encoder 24 (see FIG. 3) that detects the shaft angle of each servo motor 23. In FIG. 1, only a part of the servomotors 23 among the plurality of servomotors is illustrated. Each of the joints may be one of a swing joint that pivots one end of the arm 22 and pivotally supports the other end, and a rotary joint that pivotally supports the arm 22 itself about its longitudinal direction. It is configured.

  Further, as shown in FIG. 1, a hand tool is provided at the foremost part of all the arms 22 connected to each joint, that is, the robot tip (hereinafter referred to as “arm tip”) via a tool shaft 26 as a joint. 25 is provided. The hand tool 25 can be in an arbitrary posture by being positioned at an arbitrary position by the operation of the arm 22 through the joints. The hand tool 25 of the present embodiment includes a vacuum suction pad 27 that can suck a workpiece by a vacuum suction force.

  The robot control apparatus 10 includes a CPU, a ROM, and a RAM (all not shown). The ROM stores various programs including a control program for controlling the operation of the robot 20 and operation data indicating a movement target position of a workpiece (not shown). The CPU controls the robot 20 for workpiece transfer by executing various programs stored in the ROM. The RAM serves as a work area for storing various data by the processing of the CPU. The robot control device 10 includes a portable input device 15 for inputting, for example, teaching data for inputting teaching points of the robot 20 and other various settings. Further, the robot 20 can be manually controlled by an operator's operation on the input device 15. The robot control device 10 also includes a communication interface 11 for receiving analysis results and operation commands from the two-dimensional visual device 40.

  In FIG. 3, for convenience of explanation, only one servomotor 23 and encoder 24 of each joint of the robot 20 are representatively shown, and the other servomotors 23 and encoders 24 are not shown.

  The camera 30 includes a video camera having a solid-state imaging device such as a CCD or CMOS, and can capture a moving image at a predetermined frame rate (for example, 30 frames per second) based on an image signal output from the camera 30. It is. The camera 30 may be a color camera or a monochrome camera. The camera 30 is supported by support means (not shown) and is disposed above the work table 60. The work table is held in the actual work of holding the work (not shown) on the work table 60 and transporting it to the movement target position. The range including at least the entire range of the 60 workpiece placement surfaces 60a is defined as an imaging range.

  Based on the image signal output from the camera 30, the two-dimensional visual device 40 generates image data corresponding to an image within the imaging range (a range to be imaged where imaging is performed and image capture is performed). That is, the image signal obtained from the video camera is converted into a frame image composed of RGB color images (or monochrome images) that can be processed by a computer by the two-dimensional visual device 40. The resolution of the captured image is, for example, 640 pixels in the horizontal direction and 480 pixels in the vertical direction. Note that the frame rate, the image to be converted, and the resolution are not limited to the above example.

  The two-dimensional visual device 40 is composed of a computer and includes a CPU 41, a ROM 42, a RAM 43, an image memory 45, a camera interface 44, a monitor interface 46, an input interface 47 and a communication interface 48. The CPU 41 and each unit can input and output via the bus 49. The CPU 41 corresponds to a first calculation unit, a second calculation unit, and a third calculation unit.

  The ROM 42 stores various processing programs including an image processing program for workpiece posture position detection executed by the CPU 41, an image processing program for coupling the camera coordinate system and the robot coordinate system, and the like. In this embodiment, the robot coordinate system is a coordinate system (base coordinate system) based on the pedestal (base) of the robot. The ROM 42 corresponds to a storage medium. The RAM 43 is a work area that stores various data by the processing of the CPU 41. The camera interface 44 inputs an image signal output from the camera 30. The image memory 45 stores each frame image (image data) generated by the CPU 41 based on the image signal. The two-dimensional visual device 40 is connected to a display 50 composed of a liquid crystal display device or the like via a monitor interface 46. The display 50 performs predetermined display in various processes by the CPU 41 via the monitor interface 46 and also displays image data stored in the image memory 45. The input interface 47 is connected to an input device 51 for inputting various data necessary for various processes by the CPU 41. The communication interface 48 communicates various data with the communication interface 11 of the robot control apparatus 10.

  The two-dimensional visual device 40 recognizes the workpiece position of a workpiece (not shown) on the workpiece table 60 based on the image obtained by imaging the workpiece with the camera 30 and transmits the workpiece via the communication interfaces 48 and 11. The position information is transmitted to the robot controller 10. Based on the position information of the workpiece, the robot control apparatus 10 causes the robot 20 to grip the hand tool 25 provided at the tip of the arm at the specified position of the workpiece and transport the workpiece to a target position. Control.

(Operation of the first embodiment)
Next, a procedure for combining the camera coordinate system of the camera 30 and the robot coordinate system of the robot 20 in the robot control system of the present embodiment will be described.

  In this embodiment, the XY coordinate plane of the robot coordinate system and the XY coordinate plane of the camera coordinate system are parallel to each other, and the X axis and the Y axis of the robot coordinate system viewed from the camera 30 are orthogonal to each other. It is effective in the case.

<1. Center coordinates of robot hand tool>
4A, S10 to S18 are flowcharts of how to obtain the center coordinates of the robot hand tool 25 when the CPU 41 executes an image processing program for combining the camera coordinate system and the robot coordinate system. .

In S10, when the image processing program is executed, the CPU 41 of the robot control apparatus 10 controls the operation of the robot 20 described below.
This operation control may be manually controlled by the operator through an input operation on the input device 15, or when executing an image processing program for combining the camera coordinate system and the robot coordinate system, You may make it carry out by automatic control based on the automatic control program stored in ROM.

  The robot 20 sucks and holds the recognition target jig 70 shown in FIG. 2A by the vacuum suction pad 27 by the above-described manual control or automatic control, and is the position between the camera 30 and the work table 60 at the camera 30. The tool shaft 26 is rotated in a posture in which the tool shaft 26 is oriented in the vertical direction (that is, the direction perpendicular to the ground) in a state where the tool shaft 26 is positioned within the imaging range. The vertical direction is a direction parallel to the Z axis of the robot coordinate system and is orthogonal to the X axis and the Y axis of the robot coordinate system. Here, the tool axis 26 is rotated by the X coordinate value and the Y coordinate value (p, q) of the robot coordinate system by the control. The coordinate value is calculated by the CPU of the robot control device 10 based on the signal from each encoder 24 and the link parameter of the robot 20 in the robot coordinate system, and is stored in the RAM of the robot control device 10.

  As shown in FIG. 2A, the recognition target jig 70 is formed in a substantially flat rectangular shape as a whole, and a projection 71 as a characteristic portion is formed in the longitudinal direction at one end portion in the longitudinal direction. It is formed to extend. The overall shape of the recognition target jig 70 is not limited to a rectangle, and may be a square, a circle, an ellipse, or the like when viewed in plan. Moreover, it is not limited to a flat plate, The block shape which has thickness may be sufficient. When viewed from the camera 30, the characteristic part can be easily detected by frame difference processing if it is arranged in a part that is not hidden by the hand tool 25 in the recognition target jig 70 and has a protrusion shape.

  Further, the rotational posture of the recognition target jig 70 does not need to be exactly horizontal, and may be inclined. The holding position by the hand tool 25 may be an arbitrary position as long as the characteristic part is not hidden by the hand tool 25 when viewed from the camera 30 as described above.

  In S <b> 10, the recognition target jig 70 that is rotating as described above is captured by the camera 30, and the image signal is input to the two-dimensional visual device 40. The image signal input to the two-dimensional visual device 40 is generated by the CPU 41 in a time series in accordance with a predetermined frame rate and stored in the image memory 45. 5A to 5D show the frame images 1 to 4 acquired by the process of S10 and stored in the image memory 45. FIG. The frame image corresponds to a rotational moving image.

  In S12, the CPU 41 extracts a frame difference between temporally adjacent frame images in the frame images generated in chronological order. FIG. 5E shows image data as a frame difference result of the frame images 1 to 4.

  In S <b> 14, the CPU 41 acquires the coordinate values of the camera coordinate system of the plurality of feature points P <b> 1 to Pr regarding the feature part of the recognition target jig 70 in the image data (hereinafter referred to as a frame difference image) that is the frame difference result. To do. R is the frame number of the frame image. Acquisition of the coordinate values of the feature points is performed by performing known image processing such as binarization processing on the frame difference image with a predetermined threshold value to determine the feature points P1 to Pr.

  In S16, CPU41 calculates the common circular arc which passes each coordinate value based on the coordinate value of the feature points P1-Pr acquired by S14. The calculation of this arc is substantially the same as calculating (indexing) a circle as a closed curve. The arc E is an arc locus including the feature points P1 to Pr (see FIG. 5 (f)).

  In S 18, the CPU 41 calculates the coordinate value (m, n) of the center of curvature of the obtained arc, that is, the center of the calculated circle, and stores it in the RAM 43. The calculated coordinate value (m, n) of the center of the arc is the coordinate value of the camera coordinate system of the tool axis 26.

S10 to S18 correspond to the first step.
<2. Robot coordinate system tilt>
In FIG. 4B, S20 to S28 are flowcharts of how to obtain the tilt of the robot coordinate system with respect to the camera coordinate system when the CPU 41 executes an image processing program for combining the camera coordinate system and the robot coordinate system. is there.

In S20, when the image processing program is executed, the robot control device 10 controls the operation of the robot 20 described below.
This operation control may be manually controlled by the operator through an input operation on the input device 15, or when executing an image processing program for combining the camera coordinate system and the robot coordinate system, You may make it carry out by automatic control based on the automatic control program stored in ROM.

  The robot 20 sucks and holds the recognition target jig 70 shown in FIG. 2A by the vacuum suction pad 27 by the above-described manual control or automatic control, and is the position between the camera 30 and the work table 60 at the camera 30. The tool axis 26 is linearly moved along the X axis of the robot coordinate system while being positioned so as to fall within the imaging range.

  That is, in the present embodiment, in the robot coordinate system, along the X axis (tilt measurement coordinate axis) of the two coordinate axes (X axis and Y axis) constituting the robot coordinate system orthogonal to the tool axis 26. As described above, the hand tool 25 is moved by the recognition target jig 70.

  At this time, the posture of the recognition target jig 70 does not need to be exactly horizontal, and may be inclined. The holding position by the hand tool 25 may be an arbitrary position as long as the characteristic part is not hidden by the hand tool 25 when viewed from the camera 30 as described above.

In S <b> 20, the recognition target jig 70 moving linearly as described above is picked up by the camera 30, and the image signal is input to the two-dimensional visual device 40. The image signal input to the two-dimensional visual device 40 is generated by the CPU 41 at a predetermined frame rate according to a time series and stored in the image memory 45. FIG. 6A shows one of the frame images, which is the moving image of the hand tool 25 at the initial position, acquired by the processing of S20 and stored in the image memory 45. FIG. 6B shows one of the frame images, which is the moving image of the hand tool 25 at the final position, acquired by the processing of S20 and stored in the image memory 45. The frame image corresponds to a linear moving moving image. In FIG. 6 (b), Q ₁ indicates the position of the tip of the projection 71 of the recognition target jig 70 of the initial position. Q _s indicates the position of the tip of the protrusion 71 of the recognition target jig 70 at the final position. s is a frame number.

In S22, the CPU 41 extracts a frame difference between temporally adjacent frame images in the frame images generated in time series order.
In S24, the CPU 41 sets the coordinate values of the camera coordinate system of the plurality of feature points Q _{1 to} Q _s related to the feature parts of the recognition target jig 70 in the image data (hereinafter referred to as frame difference image) as the frame difference result. To get. The coordinate values of the feature points are acquired by performing known image processing such as binarization processing on the frame difference image with a predetermined threshold value and determining the feature points Q _{1 to} Q _s .

In S26, CPU 41 calculates a straight line passing through the coordinate values based on the coordinate values of the acquired feature point _Q 1 to Q _s in S24. This straight line is a straight line trajectory of the feature point _Q 1 to Q _s.
In S28, since the obtained straight line D is a straight line parallel to the X axis of the robot coordinate system, the CPU 41 calculates the inclination α between the straight line D and the X axis of the camera coordinate system, and stores it in the RAM 43. . Note that the X axis of the camera coordinate system is a horizontal axis with the upper left corner as the origin in the image data consisting of the number of horizontal pixels (640 pixels) and the number of vertical pixels (480 pixels) shown in FIG. It is. The Y axis of the camera coordinate system is an axis extending vertically downward from the origin.

S20 to S28 correspond to the second step. The slope α corresponds to a slope index parameter.
<3. Calculation of relative positional relationship between camera coordinate system and robot coordinate system>.
Based on the coordinate value (m, n) of the center of the arc of the camera coordinate system of the tool axis 26 acquired in S18 and S28 as described above, and the inclination α of the X axis of the robot coordinate system and the X axis of the camera coordinate system. The CPU 41 calculates the relative positional relationship between the camera coordinate system and the robot coordinates. This calculation corresponds to the third step.

Here, the relative positional relationship between the camera coordinate system and the robot coordinates will be described.
If the angle between the origin OR (x0, y0) of the robot coordinates and the camera coordinate system X-axis connecting the center (arc center) T of the tool axis is θ, it can be obtained by the following equation (1). Note that the robot coordinate value (p, q) of the center T of the tool axis 26 is acquired from the robot control apparatus 10 by communication by the CPU 41 and stored in the RAM 43.

θ = α + tan−1 (q / p) (1)
The length U of the line OR-T in the robot coordinate system is
√ (p ² + q ² ) [mm]
It is.

The actual distance per pixel (pixel) in the camera coordinate system is μ [mm / pixel], and the length U of the line segment OR-T in the camera coordinate system is
U = √ (p ² + q ² ) × 1 / μ [pixel] (2)
It is. Here, μ is a distance coefficient.

  Also, the difference between the X coordinate value (x0) and m of the origin of the robot coordinates and the difference between the Y coordinate value (y0) and n of the origin of the robot coordinates are obtained by multiplying OR-T by sin θ and cos θ, respectively. From the parameters m, n, p, q and α, the camera coordinate value (x0, y0) of the origin OR of the robot coordinates is obtained from the following equations (3) and (4).

x0 = m−√ (p ² + q ² ) × 1 / μ × cos θ (3)
y0 = n−√ (p ² + q ² ) × 1 / μ × sin θ (4)
For this reason, the CPU 41 calculates the camera coordinate value (x0, y0) of the origin OR of the robot coordinates using the equations (3) and (4). Equation (1) is used to calculate θ.

In this way, the relative positional relationship between the camera coordinate system and the robot coordinate system is calculated, and the result is stored in a memory such as the RAM 43.
Thereafter, since the relative positional relationship between the camera coordinate system and the robot coordinate system has been obtained, the correct instruction reflecting this relative positional relationship is issued from the two-dimensional visual device 40 to the robot control device 10 when performing actual work. It becomes possible.

This embodiment has the following features.
(1) In the first step, the camera coordinate system and the robot coordinate system of the robot control system according to this embodiment are coupled by rotating the hand tool 25 around the tool axis 26 at the tip of the robot and holding the hand tool 25. The rotation moving image of the recognition target jig 70 is acquired by the camera 30 (fixed camera), and the locus drawn by the protrusion 71 (characteristic part) of the recognition target jig 70 based on the frame difference obtained from the rotation moving image. An arc (that is, a closed curve) is determined from the above, and the center point of the arc curve is calculated in the camera coordinate system.

  Further, in the second step, with respect to the coordinate axis of the robot coordinate system and the X axis (tilt measurement coordinate axis) of one of the two coordinate axes constituting the coordinate plane (XY coordinate plane) where the tool axis 26 can intersect, The hand tool 25 holding the recognition target jig 70 is linearly moved so as to follow, and a moving image (linear movement moving image) of the recognition target jig 70 is acquired by the camera 30, and the camera is based on the linear movement moving image. An inclination α (inclination index parameter) of the X axis (inclination measurement coordinate axis) of the robot coordinate system with respect to the X axis of the coordinate system is calculated.

  In the third step, the camera coordinate system is determined based on the center point of the arc curve (closed curve) and the inclination α (inclination index parameter) of the X axis (inclination measurement coordinate axis) of the robot coordinate system with respect to the X axis of the camera coordinate system. The relative positional relationship of the robot coordinate system is calculated.

  As a result, when the robot 20 (handling robot) grips the recognition target jig 70, it is not necessary to accurately grip the recognition target jig 70, and when the recognition target jig 70 is gripped and rotated and linearly moved. From the acquired continuous images (moving images), the tool axis 26 center of the hand tool 25 and the inclinations of the robot coordinate system and the camera coordinate system can be easily obtained.

  In addition, since the recognition target jig of the present embodiment has a shape that can be held by the hand tool 25, it is not necessary to perform precise positioning when holding. In addition, when the jig to be recognized is gripped without strict positioning, the posture of the jig to be recognized varies, but the processing after being held is only rotation and linear movement, and is provided at the end of the jig. Since the moving image is only obtained for the purpose of acquiring the feature points related to the specified feature portion, the variation can be absorbed.

  Further, unlike the conventional example in which the hand tool 25 is removed, it is not necessary to remove the hand tool. Therefore, the operation of combining the camera coordinate system and the robot coordinate system does not require removal of the hand tool and attachment work after the operation is completed, and can be performed in a short time. Incidentally, the measurement work in the prior art 1 requires about 1 hour, but this method has the advantage of requiring about 5 minutes. In addition, when this method is applied to a production line for the above reasons, a great loss due to a line stop that occurs when a camera is replaced can be minimized.

  (2) In this embodiment, since the process of detecting the moving object, that is, the characteristic part of the recognition target jig 70 is performed using the frame difference, the trajectory of the recognition target jig can be automatically detected. Measurement work to obtain the relationship between the coordinate system and the robot coordinate system can be automated. That is, no special skill is required for the operator, and anyone can easily perform the measurement work.

  (3) In the method of combining the camera coordinate system and the robot coordinate system of the robot control system of this embodiment, the closed curve calculated in the second step is a circle, the X axis of the robot coordinate system is the tilt measurement coordinate axis, and the tilt index parameter Is the angle between the linear trajectory and the X axis (inclination measurement coordinate axis) when the hand tool 25 is moved along the X axis.

  As a result, when the XY coordinate plane of the robot coordinate system and the XY coordinate plane of the camera coordinate system are parallel to each other, and the X axis and the Y axis of the robot coordinate system viewed from the camera 30 are orthogonal, a description will be given later. Compared to the second embodiment, the operation of combining the camera coordinate system and the robot coordinate system can be performed more easily.

  (4) In this embodiment, the characteristic part of the recognition target jig 70 is a protrusion 71 provided at one end. As a result, since a moving image is only obtained for the purpose of obtaining a feature point related to the protrusion 71 (feature part) provided at the end of the jig, the recognition target jig can be gripped without strict positioning.

  (5) The two-dimensional visual device 40 (image processing device) of the present embodiment rotates the hand tool 25 around the tool axis 26 at the tip of the robot and rotates the recognition target jig 70 held by the hand tool 25. An image is acquired from the camera 30 (fixed camera), and an arc (that is, a closed curve) is calculated from the locus drawn by the protrusion 71 (feature part) of the recognition target jig 70 based on the frame difference obtained from the rotational moving image. CPU 41 (first calculation means) is provided for calculating the center point of the arc curve in the camera coordinate system.

  Further, the CPU 41 recognizes the recognition target jig so as to be along the X axis (tilt measurement coordinate axis) of the two coordinate axes constituting the robot coordinate system orthogonal to the tool axis 26, which is a coordinate axis of the robot coordinate system. The hand tool 25 holding 70 is moved linearly, and a moving image (linear moving moving image) of the recognition target jig 70 is acquired by the camera 30, and the robot with respect to the X axis of the camera coordinate system is acquired based on the linear moving moving image. It functions as second calculation means for calculating the inclination α (inclination index parameter) of the X axis (inclination measurement coordinate axis) of the coordinate system.

  The CPU 41 also determines the camera coordinate system and the robot coordinates based on the center point of the arc curve (closed curve) and the inclination α (inclination index parameter) of the X axis (inclination measurement coordinate axis) of the robot coordinate system with respect to the X axis of the camera coordinate system. It functions as third calculation means for calculating the relative positional relationship of the system.

As a result, it is possible to provide the two-dimensional visual device 40 that can achieve the effects described in (1) above.
Further, the coordinate value of the robot coordinate system at the position of the tool axis 26 for rotating the recognition target jig 70 is set and fixed in advance in the program of the robot controller, and is moved along the X axis of the robot coordinate system. Since all the operations can be automated by incorporating the program into the computer, the operator only has to move the robot as determined while operating the image processing device (in this embodiment, the two-dimensional visual device 40).

  (6) The image processing program for combining the camera coordinate system and the robot coordinate system according to the present embodiment is such that the hand tool 25 provided at the tip of the robot 20 via the tool axis 26 is moved around the tool axis 26. A rotating moving image of the recognition target jig 70 held by the hand tool 25 when rotating is acquired from the camera 30 (fixed camera), a frame difference is obtained from the rotating moving image, and the recognition target jig based on the frame difference is obtained. An arc (closed curve) is calculated from the trajectory drawn by the 70 characteristic parts, and functions as first calculation means for calculating the center point of the arc curve in the camera coordinate system.

  In addition, the program causes the computer to operate with respect to one X axis (tilt measurement coordinate axis) of two coordinate axes that constitute a coordinate plane (XY coordinate plane) that is a coordinate axis of the robot coordinate system and that the tool axis 26 can intersect. Then, the hand tool 25 holding the recognition target jig 70 is linearly moved along, and a moving image (linear movement moving image) of the recognition target jig 70 is acquired by the camera 30 and obtained from the linear movement moving image. The linear trajectory of the protrusion 71 of the recognition target jig 70 is obtained based on the frame difference, and the inclination α (inclination) of the X axis (inclination measurement coordinate axis) of the robot coordinate system with respect to the X axis of the camera coordinate system is obtained based on the linear trajectory. It functions as a second calculation means for calculating (index parameter).

  Further, the program causes the computer coordinates based on the center point of the arc curve (closed curve) and the inclination α (inclination index parameter) of the X axis (inclination measurement coordinate axis) of the robot coordinate system with respect to the X axis of the camera coordinate system. It functions as a third calculating means for calculating the relative positional relationship between the system and the robot coordinate system.

As a result, it is possible to provide a program that can achieve the effects described in (1) above.
(7) The ROM 42 as a storage medium of the present embodiment is readable by a computer and stores the image processing program described in (6) above.

  As a result, it is possible to provide a ROM 42 that stores a program that can achieve the effects described in (1) above. That is, by mounting a storage medium storing the program on a computer, the relationship between the camera coordinate system and the robot coordinate system can be easily obtained.

(Second Embodiment)
Next, a second embodiment will be described with reference to FIGS. Since the hardware configuration of the robot control system of the second embodiment is the same as that of the first embodiment, the same components are denoted by the same reference numerals and description thereof is omitted.

In the present embodiment, the CPU 41 corresponds to a first calculation unit, a second calculation unit, and a third calculation unit.
In the first embodiment, the XY coordinate plane of the robot coordinate system and the XY coordinate plane of the camera coordinate system are parallel to each other, and the X axis and the Y axis of the robot coordinate system viewed from the camera 30 are orthogonal to each other. It is effective. On the other hand, in the second embodiment, the XY coordinate plane of the robot coordinate system and the XY coordinate plane of the camera coordinate system are not parallel to each other, and the hand tool 25 is moved to the X of the robot coordinate system on the actual machine (robot). This is effective when the trajectories when moved along the axis and the Y axis are not orthogonal. Here, the reason why the two trajectories are not orthogonal is that there is a mechanical error in the connection configuration of the arms constituting the robot.

  In the present embodiment, the image processing program stored in the ROM 42 of the two-dimensional visual device 40 is different from the image processing program of the first embodiment. The ROM 42 corresponds to a storage medium that stores an image processing program.

(Operation of Second Embodiment)
In the robot control system of this embodiment, a procedure for coupling the camera coordinate system of the camera 30 and the robot coordinate system of the robot 20 will be described.

<1. Center coordinates of robot hand tool>
FIG. 8A is a flowchart of how to obtain the center coordinates of the hand tool 25 of the robot when the CPU 41 executes an image processing program for combining the camera coordinate system and the robot coordinate system. In the flowchart, since S10 to S14 are the same as those in the first embodiment, the description thereof will be omitted. However, in S10, as described in the first embodiment, the tool axis 26 is controlled by robot coordinates according to the above control. It rotates with the X coordinate value and Y coordinate value (p, q) of the system.

  In S16A, the CPU 41 calculates (determines) an elliptical arc (that is, an arc forming an ellipse as a closed curve) from the trajectory passing through each coordinate value based on the coordinate values of the feature points P1 to Pr acquired in S14.

  Here, when the XY coordinate plane of the robot coordinate system and the XY coordinate plane of the camera coordinate system are not parallel to each other, the trajectory of the characteristic part (projection 71) of the recognition target jig 70 obtained by the camera 30 is In order to draw an ellipse, this ellipse arc is calculated in S16A.

  In S18A, the CPU 41 calculates the center point (center) of the obtained elliptical arc, that is, the coordinate value (m, n) of the calculated center point of the ellipse, and stores it in the RAM 43. For example, when the ellipse is determined, the coordinate values of both ends of the major axis are calculated, and the coordinates of the half positions of both ends are set as the center coordinates of the ellipse. The method for obtaining the center is not limited to the above method.

The calculated coordinate value (m, n) of the center of the ellipse is the coordinate value of the camera coordinate system of the tool axis 26.
S10 to S18A correspond to the first step.

<2. Robot coordinate system tilt>
In FIG. 8B, S20A to S28A are flowcharts of how to obtain the tilt of the robot coordinate system relative to the camera coordinate system when the CPU 41 executes an image processing program for combining the camera coordinate system and the robot coordinate system. is there.

In S20A, when the image processing program is executed, the robot control device 10 performs operation control of the robot 20 described below.
This operation control may be manually controlled by the operator through an input operation on the input device 15, or when executing an image processing program for combining the camera coordinate system and the robot coordinate system, You may make it carry out by automatic control based on the automatic control program stored in ROM.

  The robot 20 sucks and holds the recognition target jig 70 shown in FIG. 2A by the vacuum suction pad 27 by the above-described manual control or automatic control, and is the position between the camera 30 and the work table 60 at the camera 30. The tool axis 26 is linearly moved by a linear distance L along the X axis of the robot coordinate system in a state where the tool axis 26 is positioned so as to fall within the imaging range. This linear distance is a preset distance.

  Thereafter, the recognition target jig 70 shown in FIG. 2A is sucked and held by the vacuum suction pad 27 by the manual control or the automatic control, and the camera 30 is positioned between the camera 30 and the work table 60. The tool axis 26 is linearly moved by a linear distance L along the Y axis of the robot coordinate system in a state where the tool axis 26 is positioned so as to fall within the imaging range.

  That is, in this embodiment, in the robot coordinate system, with respect to two coordinate axes constituting a coordinate plane that can intersect the tool axis 26, that is, the X axis (tilt measurement coordinate axis) and the Y axis (tilt measurement coordinate axis), The recognition object jig 70, that is, the hand tool 25 is moved along each of them.

  At this time, the posture of the recognition target jig 70 does not need to be exactly horizontal, and may be inclined. The holding position by the hand tool 25 may be an arbitrary position as long as the characteristic part is not hidden by the hand tool 25 when viewed from the camera 30 as described above.

  In S20A, the recognition target jig 70 moving linearly along the X axis and the Y axis as described above is picked up by the camera 30, and the image signal is input to the two-dimensional visual device 40. Is done. The image signal input to the two-dimensional visual device 40 is generated by the CPU 41 at a predetermined frame rate according to a time series for each axis (X axis and Y axis) and stored in the image memory 45.

The frame image corresponds to a linear moving moving image.
In S22A, the CPU 41 extracts frame differences between temporally adjacent frame images in the frame images generated in time series for each axis (X axis and Y axis).

  In S24A, when the CPU 41 linearly moves along each axis of the protrusion 71 (characteristic portion) of the recognition target jig 70 in the image data (hereinafter referred to as a frame difference image) that is the frame difference result, Get the coordinate values of the start and end camera coordinate systems. The coordinate values of the protrusions 71 are obtained by performing known image processing such as binarization processing on the frame difference image with a predetermined threshold value, and determining the start point and end point.

In the present embodiment, the coordinate values of these start points and end points are handled as vectors.
S20A to S24A correspond to the second step.
<3. Calculation of relative positional relationship between camera coordinate system and robot coordinate system>.
As described above, based on the two coordinate vectors (m, n) of the center of the elliptical arc of the camera coordinate system of the tool axis 26 acquired in S18A and the start and end coordinate values acquired in S24A. The CPU 41 calculates the relative positional relationship between the camera coordinate system and the robot coordinates in S28A. This calculation corresponds to the third step.

First, camera coordinate values (x ₀ , y ₀ ) at the origin of the robot coordinate system are obtained.
Specifically, when the linear distance L (mm) is moved, the start point and end point along the X axis are
U _s (x ₁ , y ₁ ), U _e (x ₂ , y ₂ )
And the start and end points along the Y axis,
V _s (x ₃ , y ₃ ), V _e (x ₄ , y ₄ )
And

  A unit vector of 1 mm in length along the X axis of the robot is

ロボットのＹ軸に沿った長さ１ｍｍの単位ベクトルは、

A unit vector of 1 mm in length along the Y axis of the robot is

となる。

It becomes.

  The unit vector along the X axis of the robot corresponds to a tilt index parameter with respect to the X axis (tilt measurement coordinate axis) of the camera coordinate system. A unit vector along the Y axis of the robot corresponds to a tilt index parameter with respect to the Y axis (tilt measurement coordinate axis) of the camera coordinate system.

The vector from the robot coordinate system origin O _R to the center T of the tool axis 26 shown in FIG. 9 (a), expressed as a composite vector of the two unit vectors, as follows.

これに、今までに取得したパラメータを代入すると、

Substituting the parameters obtained so far into this,

となる。

It becomes.

CPU41 is solved equation (8), obtains the camera coordinate values of the robot coordinate origin _(x 0, _{y 0).}
Next, the CPU 41 obtains a conversion formula from camera coordinate values to robot coordinate values. That is, the relative positional relationship between the camera coordinate system and the robot coordinates is calculated.

Specifically, the CPU 41 converts the camera coordinate value into the robot coordinate value by using x ₀ , y ₀ acquired as described above and the unit vector represented by the expressions (5) and (6). Find the conversion formula.

  If the coordinate value when the point P of the camera coordinate value (x, y) shown in FIG. 9B is converted into the robot coordinate system is (X, Y),

となる。この式（９）に、今までに取得したパラメータを代入すると、

It becomes. Substituting the parameters obtained so far into this equation (9),

となる。ＣＰＵ４１は、この式（１０）から得られる連立方程式を解いて、下記式（１１）、式（１２）を求める。

It becomes. The CPU 41 solves the simultaneous equations obtained from the equation (10) to obtain the following equations (11) and (12).

なお、式（１３）〜式（１６）は式（１１）、式（１２）中で使用したＵ_ｘ、Ｕ_ｙ、Ｖ_ｘ、Ｖ_ｙの定義のための式である。

Expressions (13) to (16) are expressions for defining U _x , U _y , V _x , and V _y used in expressions (11) and (12).

In this way, the relative positional relationship between the camera coordinate system and the robot coordinate system is calculated, and the result is stored in a memory such as the RAM 43.
Thereafter, since the relative positional relationship between the camera coordinate system and the robot coordinate system has been obtained, the correct instruction reflecting this relative positional relationship is issued from the two-dimensional visual device 40 to the robot control device 10 when performing actual work. It becomes possible.

The second embodiment has the following features in addition to the effects (2) and (4) of the first embodiment.
(1) In the first step, the camera coordinate system and the robot coordinate system of the robot control system are coupled by rotating the hand tool 25 around the tool axis 26 at the tip of the robot, and the recognition target treatment held by the hand tool 25. A rotating moving image of the tool 70 is acquired by the camera 30 (fixed camera), and an elliptical arc (that is, a closed curve) of the protrusion 71 (characteristic portion) of the recognition target jig 70 is obtained based on the frame difference obtained from the rotating moving image. The center point of the elliptical arc is calculated and calculated in the camera coordinate system.

  Further, in the second step, the coordinate axis of the robot coordinate system and the X axis (inclination measurement coordinate axis) and the Y axis (inclination measurement coordinate axis) constituting the coordinate plane on which the tool axis 26 can intersect are along. The hand tool 25 holding the recognition target jig 70 is linearly moved, and a linear moving moving image of the recognition target jig 70 is acquired by the camera 30 (fixed camera). Based on the linear moving moving image, a camera coordinate system is obtained. A vector that is a tilt index parameter for the X axis (tilt measurement coordinate axis) and the Y axis (tilt measurement coordinate axis) is calculated.

  In the third step, the center point of the elliptical arc (closed curve), the X-axis of the camera coordinate system, the X-axis (tilt measurement coordinate axis) of the robot coordinate system with respect to the Y-axis, the tilt index parameter for the Y-axis (tilt measurement coordinate axis), The relative positional relationship between the camera coordinate system and the robot coordinate system is calculated based on the vector.

  As a result, when the robot 20 (handling robot) grips the recognition target jig 70, it is not necessary to accurately grip the recognition target jig 70, and when the recognition target jig 70 is gripped and rotated and linearly moved. From the acquired continuous images (moving images), the tool axis 26 center of the hand tool 25 and the inclinations of the robot coordinate system and the camera coordinate system can be easily obtained.

  Further, according to the present embodiment, when the XY coordinate plane of the robot coordinate system and the XY coordinate plane of the camera coordinate system are not parallel to each other and the X and Y axes of the robot coordinate system are not orthogonal to each other. However, the tool axis center of the hand tool and the tilt of the robot coordinate system and the camera coordinate system can be easily obtained.

  (2) The two-dimensional visual device 40 (image processing device) of the second embodiment rotates the hand tool 25 around the tool axis 26 at the tip of the robot and rotates the recognition target jig 70 held by the hand tool 25. A moving image is acquired from the camera 30 (fixed camera), and an elliptical arc (that is, a closed curve) is calculated from the locus drawn by the protrusion 71 (feature part) of the recognition target jig 70 based on the frame difference obtained from the rotating moving image. And a CPU 41 (first calculation means) for calculating the center point of the elliptic arc (ellipse) in the camera coordinate system.

  Further, the CPU 41 is a recognition target along the X axis (inclination measurement coordinate axis) and the Y axis (inclination measurement coordinate axis) that constitute the coordinate plane of the robot coordinate system and that can intersect the tool axis 26. The hand tool 25 holding the jig 70 is linearly moved, and a linear moving moving image of the recognition target jig 70 is acquired by the camera 30 (fixed camera). Based on the linear moving moving image, the X axis of the camera coordinate system It functions as a second calculation means for calculating a vector that becomes an inclination index parameter for (inclination measurement coordinate axis) and Y axis (inclination measurement coordinate axis).

  Further, the CPU 41 is a vector serving as a tilt index parameter for the center point of the elliptical arc (closed curve) and the X axis (tilt measurement coordinate axis) and Y axis (tilt measurement coordinate axis) of the robot coordinate system with respect to the X axis and Y axis of the camera coordinate system. Functions as a third calculation for calculating the relative positional relationship between the camera coordinate system and the robot coordinate system.

As a result, it is possible to provide the two-dimensional visual device 40 that can achieve the effects described in (1) of the second embodiment.
In the two-dimensional visual device 40 of the second embodiment, the coordinate value of the robot coordinate system at the position of the tool axis 26 that rotates the recognition target jig 70 is set in advance in the program of the robot control device and is fixed. Since all calculations can be automated by incorporating a program to move along the X-axis and Y-axis of the system, the operator operates the image processing device (in this embodiment, the two-dimensional visual device 40). All you have to do is move the robot as determined.

  (3) The image processing program for combining the camera coordinate system and the robot coordinate system according to the present embodiment is such that the hand tool 25 provided at the tip of the robot 20 via the tool axis 26 is moved around the tool axis 26. A rotation moving image of the recognition target jig 70 held by the hand tool 25 when rotating is acquired from the camera 30 (fixed camera), and the protrusion of the recognition target jig 70 is based on the frame difference obtained from the rotation moving image. An elliptical arc (closed curve) is determined from the locus drawn by 71 (characteristic part), and functions as first calculation means for calculating the center point of the elliptical arc in the camera coordinate system.

  Further, the program runs the computer along the X axis (inclination measurement coordinate axis) and the Y axis (inclination measurement coordinate axis), which are coordinate axes of the robot coordinate system and constitute a coordinate plane on which the tool axis 26 can intersect. As described above, the hand tool 25 holding the recognition target jig 70 is linearly moved, and a linear moving moving image of the recognition target jig 70 is acquired by the camera 30 (fixed camera). Based on the linear moving moving image, camera coordinates are obtained. It is made to function as a second calculating means for calculating a vector that becomes a tilt index parameter for the X axis (tilt measurement coordinate axis) and Y axis (tilt measurement coordinate axis) of the system.

  In addition, the program causes the computer to set an inclination index for the center point of the elliptical arc (closed curve) and the X and Y axes of the robot coordinate system relative to the X and Y axes of the camera coordinate system, and the Y axis (inclination measurement coordinate axis). It is made to function as 3rd calculation which calculates the relative positional relationship of a camera coordinate system and a robot coordinate system based on the vector used as a parameter.

As a result, it is possible to provide a program that can achieve the effects described in (1) of the second embodiment.
(4) The ROM 42 as a storage medium of the present embodiment is readable by a computer and stores the image processing program described in (3) of the second embodiment.

  As a result, it is possible to provide a ROM 42 that stores a program capable of achieving the effects described in (1) of the second embodiment. That is, by mounting a storage medium storing the program on a computer, the relationship between the camera coordinate system and the robot coordinate system can be easily obtained.

(Reference example)
Next, a reference example will be described with reference to FIGS. The reference example is for solving the following problems.

  In the process of handling parts by handling or the like with a robot, the position / orientation of a workpiece is detected using image processing technology. In this case, in the image processing technique, a technique for detecting the outline of the workpiece outer shape is often performed.

  In the case of workpieces that consist only of transparent glass (for example, the front window and rear window of a vehicle), it is difficult to detect the edge of the camera's outer shape, so various methods have been devised to capture the edge well. (See Patent Document 3).

  On the other hand, in the case of a window that is integrated with glass and something other than glass (for example, a quarter window for killing a slug), the edge of the workpiece is formed because the molding made of rubber or resin is formed integrally. Is easy to detect with a camera, but has the following major problems.

  That is, the edge portion made of soft resin or rubber is soft and easily deformed, so that the edge is often deformed while being stored in a box or the like for protection. If an attempt is made to detect with the camera in this deformed state, there is a problem that the edge is recognized with an incorrect shape and misidentified as another type of workpiece. In the case of position measurement, there is a problem that an error occurs.

  As a method for measuring the position / orientation of a workpiece without using an outline, there is a method for detecting an alignment mark formed or printed on the workpiece surface. However, if such a work is performed, the appearance quality is impaired, so this method cannot be used in the case where the work surface is set to face the camera with the design surface side.

  Another method that does not use the outline is to detect the inner line of the molding (the boundary between glass and molding). As shown in FIG. 15, the molding 200 is regulated by a mold at the time of integral molding with the glass 210, and therefore has high accuracy and is bonded, so there is no deformation. However, since the same color ceramic as the molding is applied on one side (anti-design side) of the glass 210 to improve the appearance, and this ceramic can be seen through the transparent glass, the image obtained by imaging this part is Therefore, there is a problem that it is extremely difficult to recognize the boundary line between glass and molding by image processing.

  Furthermore, in the above example, a method of detecting the edge of the inner ceramic is also conceivable, but since the ceramic is applied by spraying, the positional accuracy is not good, and the window is made of colored glass such as privacy glass. In this case, it is difficult to see the ceramic edge, and it is difficult to detect the edge by image processing.

  In this way, the method of using the generally used outline contour for measuring the position and orientation of the workpiece cannot be used in the case of a mall where the outline is easily deformed. Even in this case, there is a problem that it is difficult to identify edges by image recognition processing.

  In order to solve the above problem, the robot control system of the present reference example includes a robot control device 10 and a handling robot (hereinafter simply referred to as a robot) 20, a camera 30 as a fixed camera, and a two-dimensional vision as a workpiece shape recognition device. A device 40 is provided. In these hardware configurations, the same components as those in the first embodiment are denoted by the same reference numerals as shown in FIG. Further, as shown in FIG. 10, the robot control system is provided with an illumination device 100 as illumination means for illuminating a work gripped by the robot.

  In the present embodiment, the workpiece W is a window glass of an automobile, and includes a main body Wa made of glass that is a transparent material, and a molding Wb disposed on the entire periphery of the main body Wa. The molding Wb corresponds to a diffuse reflection member that diffuses the projected light. The molding Wb is made of rubber or synthetic resin. As shown in FIG. 12, the molding Wb is fitted into the peripheral edge of the main body Wa by the groove Wc, and the molding Wb and a part on the counter-design surface side (the lower surface side in FIG. 12) of the main body Wa are The same color ceramic Wd is applied.

  As shown in FIG. 11, the lighting device 100 includes a case 101 having a covered square box shape, a plurality of light emitters 102 disposed on the inner top portion of the case 101, and a diffusion disposed by closing the bottom opening of the case 101. And a plate 103. In the present embodiment, the light emitter 102 is configured by a fluorescent lamp, an LED (Light Emitting Diode), or the like. The box shape of the case 101 is not limited.

  Further, the positional relationship among the illumination device 100, the workpiece W on the table (not shown), and the camera 30 is as follows. Note that the camera 30 includes an image pickup device 32 made of a CCD, a CMOS, or the like, and a lens 33.

  As shown in FIG. 11, a line connecting one line defining the imaging area of the image sensor 32 inside the camera 30 and the lens principal point 34 is a straight line A, and a line connecting the opposite edge and the lens principal point 34 is a straight line B. These straight lines A and B are the lines A ′ and B ′ that are folded back so that the incident angle θ1 = reflection angle θ2 and the incident angle θ3 = reflection angle θ4 on the surface of the window glass (main body Wa). . In FIG. 11, the straight lines A and B and the straight lines A ′ and B ′ are folded so that the incident angle θ1 = reflection angle θ2 and the incident angle θ3 = reflection angle θ4 on the virtual glass surface extended from the surface of the main body Wa. ing.

  Since the image pickup device 32 generally has a rectangular image pickup surface, when the straight lines A and A ′ are made to make a round along the edge of the image pickup surface of the image pickup device 32, a square pyramid is formed by the straight lines A and A ′. The Note that the quadrangular pyramid formed by the straight lines A and A ′ is folded back on the virtual glass surface and the surface of the main body Wa in FIG. 11.

  Further, when the straight lines B and B ′ are made to make a round along the edge of the imaging surface of the image sensor 32, a quadrangular pyramid is formed by the straight lines B and B ′. Note that the quadrangular pyramid formed by the straight lines B and B ′ is folded back on the virtual glass surface and the surface of the main body Wa in FIG. 11.

  The lighting device 100 is arranged and shaped so as to block all these quadrangular pyramids. Under this condition, the illumination angle of the illumination device 100 with respect to the workpiece W and the distance from the workpiece W may be set arbitrarily. Under this condition, as shown in FIG. 12, in the main body Wa that enters the camera field of view, the light projected from the illumination device 100 is regularly reflected and imaged by the image sensor 32. In addition, in the mall Wb entering the camera field of view, the light projected from the illumination device 100 is diffusely reflected and imaged by the image sensor 32 as shown in FIG.

Note that the illumination device 100 increases in size as the distance from the workpiece W increases. Therefore, it is desirable that the illumination device 100 be located at a position that does not interfere with the robot and is as close as possible.
In the present embodiment, the light emitter 102 is provided so as to emit light having a complementary color relationship with the surface color of the molding Wb.

  Complementary color is a hue circle that refers to colors that are in opposite positions, and is a color that has a relationship of becoming black when superimposed on a certain color. For example, the green complementary color is magenta, and when both are superimposed, it becomes black.

(Operation of the reference example)
Now, the operation of the two-dimensional visual device 40 configured as described above will be described.
When the work W is illuminated by the illumination device 100 under the above-described conditions, as shown in FIG. 12, the main body Wa is regularly reflected and the mall Wb is diffusely reflected in the camera field of view. Here, the color of the reflected light that is specularly reflected is governed by the light source color, and the main body Wa is imaged by the image sensor 32 with the light source color. On the other hand, since the mall Wb is diffusely reflected, the reflected light is governed by the surface color of the mall Wb.

  By the way, since the light emitter 102 projects on the molding Wb with a color complementary to the surface color of the molding Wb, the molding Wb looks black. In the example of FIG. 13, in the case of a magenta mall Wb, if the light emitter 102 emits light of a complementary green color and illuminates it, the mall Wb absorbs the green light and thus appears black.

The image sensor 32 images the black molding Wb.
The two-dimensional visual device 40 performs known image processing on the acquired image by the camera 30 and enters the field of view of the light source color (region of the main body Wa, that is, a bright portion) that enters the camera field of view. A boundary line with the area captured in black (the area of the mall Wb, that is, a dark portion) is extracted. By extracting the boundary line, the outer shape of the main body Wa defined by the boundary line is obtained.

  By acquiring the outer shape of the main body Wa of the workpiece W, the two-dimensional visual device 40 compares the outer shape of the workpiece W with the reference position set in advance in the camera coordinate system because the camera 30 is a fixed camera. Thus, it is possible to calculate the amount of positional deviation from the reference position.

  In this reference example, the illuminant 102 emits a color complementary to the surface color of the molding Wb. However, when the molding Wb is black, there is no complementary color, so You may make it light-emit white light.

FIG. 14 shows an example in the case of a black mall Wb, when the light emitter 102 emits white light and illuminates it, the mall Wb absorbs white light and thus appears black.
In the above reference example, the molding Wb is arranged on the entire circumference of the main body Wa, but the arrangement of the molding is not limited to the whole circumference. When the molding Wb is not arranged on the entire circumference, a boundary line (outline) between the mounting surface of the work table on which the workpiece W is placed and the portion of the main body Wa on which the molding Wb is not arranged can be identified. In addition, the placement surface is preferably a surface that diffusely reflects.

  In the reference example, the light emitter 102 itself emits the complementary color of the mall Wb. However, the diffusion plate 103 may transmit and diffuse only the complementary color. A filter that transmits a complementary color may be interposed between the light emitter 103 and the light emitter 102.

  In the reference example, the workpiece W is a window glass, but is not limited to the window glass. There is no limitation as long as the transparent body is a main body and a diffuse reflection member is provided around the main body. For example, it can be embodied as a shape recognition device such as a glass door or a glass lid.

In addition, embodiment of this invention is not limited to said each embodiment, You may change as follows.
In each of the embodiments, the hand tool 25 of the robot is a hand device with a vacuum suction pad, but may be replaced with a mechanical chuck device. In this case, for example, when the mechanical chuck device can grip the workpiece with the finger members facing each other, the shape of the recognition target jig 70 is opposite to each other extending in the longitudinal direction, as shown in FIG. The concave portions 72 are provided on both side portions located on the side. In addition, the shape of the recessed part 72 is not limited. Also in the recognition object jig of this modification, a protrusion 71 is provided at an end as a characteristic part.

  In each of the embodiments described above, the hand tool 25 of the robot is a hand device with a vacuum suction pad, but may be a magnetic suction chuck device that sucks a workpiece by magnetism. In this case, the recognition target jig is formed of a material that can be magnetically attracted by a magnetic attraction chuck device.

  In each of the above embodiments, the ROM 42 is a storage medium that stores an image processing program for coupling the camera coordinate system and the robot coordinate system. However, the storage medium that stores the image processing program is not limited to the ROM 42. Absent. Instead of the ROM 42, the CPU 41 may be embodied as a hard disk as an external storage device that can be written and read, or may be embodied as a semiconductor storage device such as a USB memory.

  -Although the robot of each said embodiment was materialized in the 6-axis robot, it is not limited to a 6-axis robot, It is a robot of 7 axes or more, or a robot of less than 5 axes, Comprising: A robot that can rotate the shaft and linearly move the tip of the arm may be used.

  In each of the above embodiments, the robot coordinate system is a coordinate system (base coordinate system) based on the pedestal (base) of the robot, but may be a coordinate system (user coordinate system) set by the user at an arbitrary position. .

  In each of the above embodiments, in S20, the tool axis 26 is parallel to the X axis of the robot coordinate system in a state where the tool axis 26 is positioned between the camera 30 and the work table 60 and is within the imaging range of the camera 30. The tool axis 26 may be linearly moved parallel to the Y axis of the robot coordinate system. Also in this case, since the tilt between the Y axis of the robot coordinate system and the camera coordinate system is obtained, the relative positional relationship between the camera coordinate system and the robot coordinate system can be calculated in the same manner based on this tilt.

-In 1st Embodiment, although S20-S28 was performed after performing S10-S18, you may perform S10-S18 after performing S20-S28.
In the second embodiment, S20A to S28A are executed after executing S10 to S18A. However, S10 to S18A may be executed after executing S20A to S28A.

  In the second embodiment, in S20A, the tool axis 26 is linearly moved by the same linear distance L along the X axis and Y axis of the robot coordinate system, but is moved by different linear distances L1 and L2. You may make it do. In this case, L1 and L2 shall be used instead of L used in Formula (5), Formula (6), Formula (8), Formula (10), and Formula (13) to Formula (14).

Next, technical ideas that can be recalled from the reference examples are described below.
Technical idea (1) is that, by setting the light source color to a color complementary to the surface color of the diffuse reflection member, in the image acquired by the image processing means, the image area where the diffuse reflection member is imaged is black, The image area where the specularly reflected light is obtained is white. For this reason, the shape recognition of the boundary between the main body and the diffuse reflection member in the workpiece can be accurately performed without being affected by the deformation of the diffuse reflection member other than the boundary.

(1) a camera for imaging a workpiece whose surface is specularly reflective and has a diffuse reflection member disposed on the periphery of a body made of a transparent material;
An illuminating unit that has a diffusing plate for diffusing light, and projects a color complementary to the surface color of the diffusive reflecting member to illuminate the workpiece as a surface light source by the diffusing plate;
The specular reflection in the image obtained by acquiring the specular reflection light projected from the illumination means and specularly reflected on the surface of the main body of the workpiece and the diffuse reflection light diffusely reflected by the diffuse reflection member with the camera. An apparatus for recognizing a workpiece shape, comprising image processing means for recognizing shapes of the main body and the diffuse reflection member based on a difference between a light source color of light and a surface color of the diffuse reflection member by the diffuse reflection light.

  Technical idea (2) is that the light source color is white light, and when the diffuse reflection member is black, the image area captured by the diffuse reflection member is black in the image acquired by the image processing means, and is regularly reflected. The image area where light is obtained is white. For this reason, the shape recognition of the boundary between the main body and the diffuse reflection member in the workpiece can be accurately performed without being affected by the deformation of the diffuse reflection member other than the boundary.

(2) a camera for imaging a workpiece whose surface is specularly reflective and has a diffuse reflection member disposed on the periphery of a body made of a transparent material;
An illuminating means for diffusing light and projecting white light to illuminate the workpiece as a surface light source by the diffusing plate;
The specular reflection in the image obtained by acquiring the specular reflection light projected from the illumination means and specularly reflected on the surface of the main body of the workpiece and the diffuse reflection light diffusely reflected by the diffuse reflection member with the camera. An apparatus for recognizing a workpiece shape, comprising image processing means for recognizing shapes of the main body and the diffuse reflection member based on a difference between a light source color of light and a surface color of the diffuse reflection member by the diffuse reflection light.

  According to the technical idea (3), the workpiece body is made of a transparent material such as glass or synthetic resin, and the workpiece shape can be accurately recognized when the diffuse reflection member is a molding.

  (3) The workpiece shape recognition apparatus according to the technical idea (1) or (2), wherein the main body of the workpiece is made of glass or synthetic resin, which is a transparent material, and the diffuse reflection member is a molding. .

10 ... Robot controller, 20 ... Robot, 25 ... Hand tool,
26 ... Tool axis, 30 ... Camera (fixed camera),
40 ... 2D visual device (image processing device),
41... CPU (first calculation means, second calculation means, third calculation means),
42 ... ROM (storage medium), 70 ... recognition target jig, 71 ... projection (characteristic part).

Claims (9)

  In a method of combining a camera coordinate system and a robot coordinate system of a robot control system having a hand tool provided at the tip of a robot via a tool axis and a fixed camera for imaging the work of the hand tool, Based on the frame difference obtained from the rotating moving image, obtained by obtaining a moving image of the recognition target jig held by the hand tool (hereinafter referred to as a rotating moving image) with a fixed camera. The first step of calculating a closed curve from the trajectory drawn by the characteristic part of the recognition target jig and calculating the center point of the closed curve in the camera coordinate system and the coordinate axis of the robot coordinate system, and the tool axis can intersect The recognition is carried out along at least one coordinate axis (hereinafter referred to as an inclination measurement coordinate axis) of two coordinate axes constituting a simple coordinate plane. The hand tool holding the elephant jig is linearly moved, and a moving image of the recognition target jig (hereinafter referred to as a linear moving moving image) is acquired by the fixed camera, and camera coordinates are obtained based on the linear moving moving image. A second step of calculating a tilt index parameter of the tilt measurement coordinate axis with respect to the system; and a third step of calculating a relative positional relationship between the camera coordinate system and the robot coordinate system based on the center point of the closed curve and the tilt index parameter. A combination method of a camera coordinate system and a robot coordinate system of a robot control system characterized by the above. The closed curve is an ellipse;
In the second step, the two coordinate axes of the robot coordinate system are used as tilt measurement coordinate axes,
The tilt index parameter is a vector based on coordinate values of a start point and an end point in a camera coordinate system when the hand tool is moved along the two tilt measurement coordinate axes, respectively. 2. A method for combining a camera coordinate system and a robot coordinate system of the robot control system according to 1. The closed curve is a circle;
In the second step, one coordinate axis of the robot coordinate system is set as a tilt measurement coordinate axis,
The gradient index parameter along said one tilt measurement axes, at the time of the hand tool is moved, the robot control according to claim 1, characterized in that the angle between the straight line trajectory and inclination measuring axes A method of combining the camera coordinate system of the system and the robot coordinate system.   The camera of the robot control system according to any one of claims 1 to 3, wherein the characteristic part of the recognition target jig is a protrusion provided at one end of the recognition target jig. Coupling method of coordinate system and robot coordinate system. The hand tool is rotated around the tool axis at the tip of the robot, and a moving image of a recognition target jig held by the hand tool (hereinafter referred to as a rotating moving image) is obtained from a fixed camera and obtained from the rotating moving image. Calculating a closed curve from a trajectory drawn by the characteristic part of the recognition target jig based on the frame difference, and calculating a center point of the closed curve in a camera coordinate system;
The recognition target jig so as to be along a coordinate axis of a robot coordinate system and at least one coordinate axis (hereinafter referred to as an inclination measurement coordinate axis) of two coordinate axes constituting a coordinate plane that can intersect the tool axis. The hand tool holding the image is moved linearly, and a moving image of the recognition target jig (hereinafter referred to as a linear moving moving image) is acquired by the fixed camera. Based on the linear moving moving image, Second calculation means for calculating an inclination index parameter of the inclination measurement coordinate axis;
An image processing apparatus comprising: third calculation means for calculating a relative positional relationship between a camera coordinate system and a robot coordinate system based on a center point of the closed curve and the tilt index parameter. The closed curve is an ellipse;
The second calculating means uses two coordinate axes of the robot coordinate system as tilt measurement coordinate axes, and the tilt coordinate parameter is obtained by moving the hand tool along the two tilt measurement coordinate axes as the tilt index parameter. The image processing apparatus according to claim 5, wherein a vector based on the coordinate values of the start point and the end point is calculated. The closed curve is a circle;
The second calculation means uses one coordinate axis of the robot coordinate system as a tilt measurement coordinate axis,
As the gradient index parameter, along said one tilt measurement axes, at the time of the hand tool is moved, according to claim 5, characterized in that to calculate the angle between the straight line trajectory and inclination measuring axes image Processing equipment. Computer
The hand tool is rotated around the tool axis at the tip of the robot, and a moving image of a recognition target jig held by the hand tool (hereinafter referred to as a rotating moving image) is obtained from a fixed camera and obtained from the rotating moving image. Calculating a closed curve from a trajectory drawn by the characteristic part of the recognition target jig based on the frame difference, and calculating a center point of the closed curve in a camera coordinate system;
The recognition target jig so as to be along a coordinate axis of a robot coordinate system and at least one coordinate axis (hereinafter referred to as an inclination measurement coordinate axis) of two coordinate axes constituting a coordinate plane that can intersect the tool axis. The hand tool holding the image is moved linearly, and a moving image of the recognition target jig (hereinafter referred to as a linear moving moving image) is acquired by the fixed camera. Based on the linear moving moving image, Second calculation means for calculating an inclination index parameter of the inclination measurement coordinate axis;
The third pulp program to function as a calculating means for calculating a relative positional relationship between the camera coordinate system and the robot coordinate system based on the center point and the gradient index parameter of the closed curve. Computer
The hand tool is rotated around the tool axis at the tip of the robot, and a moving image of a recognition target jig held by the hand tool (hereinafter referred to as a rotating moving image) is obtained from a fixed camera and obtained from the rotating moving image. Calculating a closed curve from a trajectory drawn by the characteristic part of the recognition target jig based on the frame difference, and calculating a center point of the closed curve in a camera coordinate system;
The recognition target jig so as to be along a coordinate axis of a robot coordinate system and at least one coordinate axis (hereinafter referred to as an inclination measurement coordinate axis) of two coordinate axes constituting a coordinate plane that can intersect the tool axis. The hand tool holding the image is moved linearly, and a moving image of the recognition target jig (hereinafter referred to as a linear moving moving image) is acquired by the fixed camera. Based on the linear moving moving image, Second calculation means for calculating an inclination index parameter of the inclination measurement coordinate axis;
Third computer-readable storage medium storing a Help program to function as a calculating means for calculating a center point and the gradient index parameter relative positional relationship between the camera coordinate system and the robot coordinate system on the basis of the closed curve.

Images