KR20110095700A - Industrial robot control method for workpiece object pickup - Google Patents

Abstract

PURPOSE: A method for controlling a robot for picking up working object is provided to measure the three-dimensional position/posture of a working object. CONSTITUTION: A method for controlling a robot for picking up working object is as follows. A working object candidate is detected by a stereo camera of a robot at a remote(S100). The three dimensional position/posture about the detected working object is detected using the stereo camera at a short range(S200). A gripping posture of the robot is instituted according to three dimensional position/posture of the working object(S300).

Description

Industrial robot control method for workpiece object pickup}

The present invention relates to an industrial robot, in particular, by using a visual sensor consisting of a stereo camera to detect candidates among the workpieces stacked in an arbitrary posture on a box or pallet, by measuring the position / posture of the detected workpiece The present invention relates to an industrial robot control method for picking up a workpiece that can easily set the grip posture of the robot.

Recently, in order to increase the productivity of the automation system and the adaptability to the environment, interest in the flexible manufacturing system (Flexible Manufacturing System) is increasing.

In order to build a flexible production system, it is essential to develop an intelligent robot that recognizes the environment, judges the situation, and autonomously operates.

To this end, industrial robot manufacturers are continuously researching reliable sensors and system integration technology to recognize the working environment. As part of the development of flexible production systems, the research on the empty picking system that draws the workpieces scattered in the boxes with robots and introduces them into a new process, but the scope of application is limited.

In the conventional bin picking method using the laser slit light, three-dimensional position / posture data can be obtained only at the point where the laser pattern is projected. Therefore, there is a disadvantage in that the area capable of measuring three-dimensional position / attitude data is narrow, and in order to measure the three-dimensional position / attitude, there is a problem that a feature point such as a hole must exist.

Although the research on the empty picking system using the stereo vision has been widely conducted, it is due to the lack of registration information in the homogeneous area where the feature points do not exist, the brightness difference of the left and right images, the occlusion, the noise, etc. There was a problem in that the application range is limited due to a matching error.

There is a method of attempting empty picking by comparing a 3D range image obtained by projecting a laser scanner or a structured pattern with the CAD data of an object. The information about the model has to be stored in the database for a long time by rotating, and the posture estimation accuracy and processing speed depend on the size of the database.

In addition, an active vision system for projecting a laser or light has a problem that it is difficult to obtain reliable three-dimensional point group data due to reflection and scattering of a workpiece made of a metallic material.

Accordingly, the present invention is to solve the conventional problems as described above, by using a visual sensor consisting of a stereo camera to set the pattern information for the reference workpiece in advance, and based on the pattern information of the reference workpiece geometric pattern matching ( Geometric Pattern Matching) detects the workpiece. Thereafter, the coordinate system of the workpiece is generated and measured using three-dimensional position / posture data of three feature points formed on the workpiece, and the gripping posture of the robot is set in real time according to the measured three-dimensional position / posture data of the workpiece. Accordingly, an object of the present invention is to provide an industrial robot control method for picking up a workpiece that can be used universally without limiting the shape, material, and processing time of the workpiece.

The present invention for achieving the above object, the first step of detecting a workpiece candidate using a stereo camera installed in the robot at a distance, and measuring the three-dimensional position / posture of the detected workpiece in a short distance with a stereo camera And a third step of setting the gripping position of the robot according to the second step and the three-dimensional position / posture of the workpiece.

In the first step, the robot having the stereo camera installed is moved to a location of the work object, and the stereo camera photographs a 2D image of a box in which the work objects are accommodated by the stereo camera at a long distance. Detecting a workpiece candidate having a two-dimensional pattern information of the workpiece and a geometric pattern matching ratio within a predetermined error range, wherein the second step includes capturing an image of the detected workpiece at a short distance with the stereo camera; After detecting the positions of the three feature points in the left and right images of the workpiece to be measured by geometric pattern matching, using the projection matrix preset to the control unit, the three-dimensional position / Generating coordinates by posture data, and measuring the position / posture of the detected workpiece; It is preferred.

In the second step, the three-dimensional position / posture data of the work object may preferably use a multi-pattern registration method of registering a plurality of patterns of brightness change or posture change of the work object.

In the second step, the three-dimensional position / posture data of the workpiece is preferably applied to a method of fitting an ellipse when the feature of the circular shape exists in the reference workpiece.

In the second step, when the 3D position / posture of the workpiece cannot be measured, the robot may change the measurement posture in 4 directions to re-measure the 3D position / posture of the workpiece. If the re-measurement fails to measure the three-dimensional position / posture of the workpiece, the object is moved to the position of another workpiece measured in the first step S100 and the three-dimensional of the workpiece. It is desirable to re-measure the position / posture.

On the other hand, before performing the first step, by moving the robot equipped with the stereo camera to the position of the reference workpiece, by taking a two-dimensional image of the box containing the reference workpiece with the stereo camera at a distance Setting two-dimensional pattern information of the reference workpiece to the control unit, three projection matrixes for the stereo camera using the detected reference workpiece at a short distance, and three left / right images for the detected reference workpiece; It is preferable to include the step of setting the coordinate system of the detected reference workpiece by measuring the three-dimensional position with respect to the feature point.

In the stereo camera, correction is performed by a correction plate having a plurality of square points arranged at equal intervals, and the correction plate is disposed at a fixed position, and the calibration is performed again at a predetermined time to calculate the preset calibration information and current calculation of the stereo camera. It is desirable to compare the stereo camera calibration information and check whether the setting has been changed.

In the third step, it is preferable to further include setting a trajectory for collision avoidance of the robot according to the six degrees of freedom change measured using the stereo camera.

Here, the step of setting the collision avoidance trajectory of the robot, in consideration of the posture of the workpiece and the position of the workpiece in the box, the collision of the robot that does not occur among the trajectories of the robot previously taught Work trace can be set in advance.

The setting of the collision avoidance trajectory of the robot may include changing the grip posture of the robot to be perpendicular to the bottom surface of the box when the workpiece is distributed in the outer region of the box in a severely inclined state. have.

In the third step, the collision detection function of the control unit and the collision detection function of the online monitoring program are monitored to determine whether there is a collision, and after modeling as a virtual environment using an OLP program (OLP) Monitoring in real time can change the gripping posture of the robot in real time to prevent a collision.

As described above, the present invention does not need to acquire three-dimensional point group data for the entire image, and precisely measures the position of the feature point using geometric pattern matching, so that the three-dimensional position caused by the stereo matching error in the conventional method. There is an advantage that can prevent the estimation error.

In addition, there is no need for a matching process between three-dimensional point group data for posture measurement, which has the advantage of being accurate and fast.

In addition, regardless of the shape and material of the workpiece, only three feature points that do not exist on the same line may be registered as a pattern, so the three-dimensional position / posture of the workpiece may be measured.

1 is an overall operation flowchart showing a preparation step and a production step of the industrial robot control method for the workpiece pickup according to the present invention.
2 is a flow chart showing in detail the production step in the industrial robot control method for the workpiece pickup according to the invention.
3 is a view showing the configuration of an industrial robot control method for the workpiece pickup according to the present invention.
4 is a view showing a state of setting the two-dimensional pattern information of the industrial robot control method for the workpiece pickup according to the present invention.
5 is a view showing a state of setting the pattern information of the stereo image for the three-dimensional position / posture measurement of the industrial robot control method for the workpiece pickup according to the present invention.
6 is a view for showing that the coordinate system of the reference workpiece is set through the three-dimensional position / attitude to the three feature points of the industrial robot control method for the workpiece pickup according to the present invention.
FIG. 7 is a view for detecting a workpiece having a geometric pattern matching ratio within a predetermined error range in a production step of an industrial robot control method for picking up a workpiece according to the present invention.
8 is a view showing the result of performing an elliptic fitting after performing a pattern recognition process for measuring the three-dimensional position / posture in the production step of the industrial robot control method for the workpiece pickup according to the present invention.
9 is a view for showing the calibration of the stereo camera using a correction plate in the industrial robot control method for the workpiece pickup according to the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In describing the present invention, the defined terms are defined in consideration of the function of the present invention, and should not be understood in a limiting sense of the technical elements of the present invention.

1 is an overall operation flowchart showing a preparation step and a production step of the industrial robot control method for the workpiece pickup according to the present invention. 2 is a flow chart showing in detail the production step in the industrial robot control method for the workpiece pickup according to the invention. 3 is a view showing the configuration of an industrial robot control method for the workpiece pickup according to the present invention. 4 is a view showing a state of setting the two-dimensional pattern information of the industrial robot control method for the workpiece pickup according to the present invention. 5 is a view showing a state of setting the pattern information of the stereo image for the three-dimensional position / posture measurement of the industrial robot control method for the workpiece pickup according to the present invention. 6 is a view for showing that the coordinate system of the reference workpiece is set through the three-dimensional position / attitude to the three feature points of the industrial robot control method for the workpiece pickup according to the present invention. FIG. 7 is a view for detecting a workpiece having a geometric pattern matching ratio within a predetermined error range in a production step of an industrial robot control method for picking up a workpiece according to the present invention. 8 is a view showing the result of performing an elliptic fitting after performing a pattern recognition process for measuring the three-dimensional position / posture in the production step of the industrial robot control method for the workpiece pickup according to the present invention. 9 is a view for showing the calibration of the stereo camera using a correction plate in the industrial robot control method for the workpiece pickup according to the present invention.

As shown in Figures 1 to 8, the industrial robot control method for picking up the workpiece of the present invention, the agent for detecting the workpiece 600 candidate using a stereo camera 300 installed in the robot 100 at a distance. Step 1 (S100), a second step (S200) of measuring the three-dimensional position / posture of the detected work object 600 at a short distance with the stereo camera 300, and the 3 of the measured work object 600 According to the dimensional position / posture, a third step (S300) of setting the grip position of the robot 100.

Before explaining the above steps, the present invention is composed of a PC-based vision unit and the robot 100 as shown in FIG. The PC-based vision unit includes a controller (PC) 200, a frame grabber, and a stereo camera 300, and is preferably connected to the controller 200 by RS 232C or Ethernet.

In the present invention, instead of using the conventional stereo matching method to find the matching point to detect the matching point of the left and right image using a geometric pattern matching (Geometric Pattern Matching) method, the posture of the workpiece is present inside the workpiece The coordinate system of the workpiece 600 may be generated and measured using three-dimensional position / posture data of three feature points.

That is, in order to grip the work object 600 in which the robot 100 is arbitrarily stacked in the present invention, the candidate selection process of the object to be worked on, the process of measuring the three-dimensional position / posture of the selected work, and for catching the work A grip attitude determination process and a strategic planning process for automatic operation are required.

First, the first step (S100) corresponding to the production stage (Operation Stage) of the present invention, using the stereo camera 300 installed in the robot 100 at a distance as shown in Figure 7 candidates for the workpiece 600 Detect.

The first step (S100), the step of moving the robot 100, the stereo camera 300 is installed to the position of the work object (600) (S110), the work with the stereo camera 300 at a distance After capturing the entire 2D image of the box such that the objects 600, that is, the plurality of workpieces 600 contained in the box are all visible, the 2D pattern information of the reference workpiece 500 preset in the control unit 200. (P) detects one or more candidates 600 for a workpiece having a geometric pattern matching ratio within a predetermined error range.

That is, the first step (S100) is to select one or a plurality of candidates 600 workable to transfer a plurality of workpieces 600 scattered one by one, which is the existing empty picking systems Unlike this, the present invention proposes a method of determining candidate objects by using shape information of a work without 3D distance information about a work area.

In the second step S200, the image of the workpiece 600 detected at a short distance is photographed by the stereo camera 300, and then the left and right images of the workpiece to be measured are geometrically matched as shown in FIG. 6. The position of the three feature points of is detected. Subsequently, coordinates are generated from three-dimensional position / posture data of three feature points of the workpiece to be measured using the projection matrix preset by the controller 200, and the position / posture of the detected workpiece 600 is generated. Measure

In other words, after generating the coordinate system of the workpiece to be measured using the three-dimensional position data for the three feature points calculated, the generated coordinate system is compared with the coordinate system of the reference workpiece 500 registered in advance in the control unit 200 It is determined by measuring the amount of change in the degrees of freedom in the 6-axis direction.

After that, if the feature exists in the workpiece to be measured according to the 6-axis degrees of freedom variation measured from the determination result, the three-dimensional position data of the workpiece to be measured can be calculated by performing a fitting process after performing geometric pattern matching. have.

On the other hand, in the second step (S200) the three-dimensional position / posture data of the workpiece 600, a multi-pattern registration method for registering a plurality of patterns for the brightness change or the posture change of the workpiece 600 Can be used.

In addition, in the second step, the three-dimensional position / posture data of the workpiece 600 may use a method of fitting an ellipse when the feature of the circular shape exists in the reference workpiece 500 as shown in FIG. 8. It may be.

In addition, when the three-dimensional position / posture of the workpiece 600 is not measured in the second step (S200), the robot 100 changes the measurement posture in four directions as shown in FIG. Re-measuring the three-dimensional position / posture of the workpiece 600, and if the re-measurement did not measure the three-dimensional position / posture of the workpiece 600, the error (alarm Re-measurement of the three-dimensional position / posture of the new workpiece 600 by moving to the position of the other workpiece 600 measured in the first step (S100) determined by.

In the third step (S300), according to the measured three-dimensional position / posture data of the workpiece 600, after setting the grip position of the robot 100 is to pick up the workpiece 600. That is, by transmitting the three-dimensional position data measured in the second step (S200) to the control unit 200 via a wired or wireless communication network, the three-dimensional position of the work trace of the industrial robot 100 set in the control unit 200 And correction can be corrected according to the posture.

To this end, the work trajectory of the robot is modified according to the six degrees of freedom variation measured using the stereo camera 300. There are two methods of selecting a trajectory for avoiding collision while the robot 100 is being driven.

In the first method, there is a method of selecting a work trajectory of a robot in which collision does not occur among the work trajectories of the robots previously taught in consideration of the posture of the work and the position placed on the box 400.

The second method is a method of correcting the grip posture of the robot 100 to be perpendicular to the bottom surface of the box 400 in consideration of the position where the workpiece 600 is placed on the box 400.

For example, when the workpiece is distributed in the outer region of the box 400 in a state in which it is inclined severely, when the workpiece is distributed in the outer region of the box 400 using the second method described above, the robot ( The grip posture of 100 may be changed to be perpendicular to the bottom surface of the box 400.

In addition, the collision detection function of the controller 200 and the collision detection function of the online monitoring program may be used to monitor the collision. In other words, the model can be modeled as a virtual environment using OLP (Off-Line Program) and monitored in real time to prevent robot collision.

Therefore, the posture of the robot 100 may be changed in real time according to the position / posture of the workpiece 600, and may be used universally regardless of the type and current state of the workpiece 600.

On the other hand, before performing the first step (S100), a preparatory stage in which the 2D pattern information P and the 3D position / posture data of the reference object are set in advance in the control unit 200 may be performed. Can be.

As shown in FIG. 1, the preparation step moves the robot 100 provided with the stereo camera 300 to the position of the reference workpiece 500, and then, the reference operation is performed with the stereo camera 300 at a distance. Taking a 2D image of the water 500 to detect the candidate for the reference workpiece 500 (S10), and the 2D pattern information P of the candidate for the detected reference workpiece 500, the controller 200. ), And setting the three-dimensional position / posture data for three feature points existing in the reference workpiece 500 at a short distance to the controller 200 (S20).

Here, the two-dimensional pattern information (P) of the reference workpiece 500 is a region inside the limit line surrounding the reference workpiece 500 in the form of a rectangular box is set to the two-dimensional pattern information (P). And the three-dimensional position / attitude data of the reference workpiece 500 is a coordinate system of the reference workpiece 500 is set through the three-dimensional position / attitude data for the three feature points specified in the reference workpiece 500.

That is, the three-dimensional position and posture of the workpiece 600 is performed by calculating a transformation matrix of the reference workpiece 500 coordinate system with respect to the robot 100 coordinate system as shown in FIG. 5, and the reference workpiece 500 coordinate system. May be generated using 3D position / posture data of the features F1, F2, and F3 stored in advance in the pattern registration process.

That is, the X axis of the workpiece coordinate system is generated along the F1F2 vector, the Y axis is set to be perpendicular to the X axis in the same plane where three feature points exist, and the Z axis is obtained externally of the X and Y axes. Then, the homogeneous transformation matrix of the robot 100 and the reference workpiece 500 is obtained and stored in a database, and the distance information between each feature point is also stored for future workpiece coordinate system verification.

In addition, it is preferable to include the step of calibrating the stereo camera 300 to convert the results measured in the three-dimensional image to the actual measurement. That is, in order to convert the 2D image coordinate system into the 3D world coordinate system in the second step S200, the projection matrix of the stereo camera 300 should be obtained.

In a perspective projection environment, the point X = [XYZ 1] ^T in three-dimensional space is projected onto one pixel x = [uv 1] ^T in the two-dimensional image plane. Equation (1) is an equation describing the projection matrix, and the projection matrix P is composed of an intrinsic parameter K, and an extrinsic parameter including a rotation matrix R and a translation vector T.

Equation (1):

In addition, the stereo camera 300 is corrected by the correction plate 700 is a plurality of square points are arranged at equal intervals, the correction plate 700 is placed in a fixed position, and recalibration every predetermined time It is possible to check whether the setting has been changed by comparing the calibration information of the set stereo camera with the currently calculated stereo camera calibration information.

As shown in FIG. 9, the stereo camera 300 may be corrected by detecting a point located at a corner of the quadrangular points arranged on the correction plate 700 in the captured image by using a corner detector.

The positions of the corner points existing in the stereo image may be detected using a Harris corner detector or SIFT. Here, the position of the corner point in the three-dimensional space is assigned by converting the three points to the robot 100 coordinate system reference position by teaching the robot 100.

In addition, the correction plate 700 is always placed in a fixed position and re-calibrates every predetermined time, the stereo camera 300 by comparing the preset stereo camera 300 correction information and the currently calculated stereo camera 300 correction information You can also check if the settings have changed.

,

이 획득되며, 3차원 공간상의 점

는 식(4),(5)와 같이 스테레오 영상의 일치점 정보 x_L, x_R, 를 이용하여 측정할 수 있게 된다.After performing the calibration of the stereo camera 300, the projection matrix for the stereo camera 300

,

Is obtained, a point in three-dimensional space

Equation (4), (5) can be measured using the matching point information x _L , x _R , of the stereo image.

In addition, hand eye calibration may be used. In this figure, {B}, {E}, {C}, and {W} are set to coordinate system for performing hand eye calibration. ) End effector coordinate system, camera coordinate system, world coordinate system. The relationship between each coordinate system can be transformed into a homogeneous transformation matrix.

는 식 (7)를 통하여 획득될 수 있는데, 여기서

는 로봇 제어부(200)로부터 제공되며,

는 로봇(100)으로 기준 물체를 교시하거나 스테레오카메라(300)를 이용하여 측정함으로써 획득할 수 있고,

는 스테레오카메라(300)로 기준 물체를 측정함으로써 계산할 수 있으므로 최종적인

는 쉽게 획득이 가능하다. This is a relationship between the robot 100 end effector and the camera 300 to obtain.

Can be obtained from equation (7), where

Is provided from the robot control unit 200,

Can be obtained by teaching a reference object with the robot 100 or by measuring using a stereo camera 300,

Can be calculated by measuring the reference object with the stereo camera 300, so

Can be easily obtained.

After the hand eye calibration is performed, all data measured by the stereo camera 300 can be converted based on the robot 100 coordinate system as shown in Equation (8).

Coordinate Systems for Hand Eye Calibration

As a result, the present invention does not need to acquire three-dimensional point group data for the entire image, and precisely measures the position of the feature point by using geometric pattern matching, and thus, the three-dimensional position estimation error caused by the stereo matching error in the conventional method. It is possible to prevent the problem, and the matching process between the three-dimensional point group data for posture measurement is also unnecessary, so it is accurate and fast. In addition, regardless of the shape and the material of the workpiece, only three feature points that do not exist on the same line can be registered as a pattern, so that the three-dimensional position / posture of the workpiece can be measured and thus can be used universally.

The technical idea of the industrial robot control method for the workpiece pickup according to the present invention has been described above with the accompanying drawings, but this is by way of example and not by way of limitation of the present invention.

Accordingly, it is a matter of course that various modifications and variations of the present invention are possible without departing from the scope of the present invention. And are included in the technical scope of the present invention.

100: robot 200: control unit
300: stereo camera 400: box
500: reference workpiece 600: workpiece
700: correction plate P: pattern information

Claims (11)

A first step (S100) of detecting a work object candidate using a stereo camera installed in a robot at a distance;
A second step (S200) of measuring a three-dimensional position / posture of the detected work object at a short distance with a stereo camera; And
And a third step (S300) of setting the holding posture of the robot according to the measured three-dimensional position / posture of the workpiece. The method of claim 1,
The first step (S100), the step of moving the robot equipped with the stereo camera to the position of the workpiece, and photographing the two-dimensional image of the box in which the workpiece is accommodated by the stereo camera at a distance, the control unit Detecting a workpiece candidate that the two-dimensional pattern information and the geometric pattern matching ratio of the predetermined reference workpiece is within a predetermined error range,
In the second step (S200), the image of the detected object is photographed by the stereo camera at a short distance, and after detecting the positions of three feature points in the left and right images of the object to be measured by geometric pattern matching. And generating coordinates by three-dimensional position / posture data of three feature points of the workpiece to be measured, using the projection matrix preset in the controller, and measuring the position / posture of the detected workpiece. An industrial robot control method for picking up a workpiece. The method of claim 2,
The three-dimensional position / posture data of the workpiece in the second step, using a multi-pattern registration method for registering a plurality of patterns for the brightness change or the posture change of the workpiece for the pickup of the workpiece Industrial robot control method. The method of claim 2,
In the second step, the three-dimensional position / posture data of the workpiece, if the feature of the circular shape on the reference workpiece, the method for fitting an ellipse, characterized in that for applying the industrial robot control Way. The method of claim 2,
The second step (S200), when the three-dimensional position / posture of the workpiece is not measured, the robot changes the measurement posture in four directions to re-measure the three-dimensional position / attitude of the workpiece Steps,
If the re-measurement fails to measure the three-dimensional position / posture of the workpiece, the three-dimensional position / movement of the workpiece is moved to another position of the workpiece measured in the first step S100. An industrial robot control method for picking up a workpiece, characterized in that the posture is measured again. The method of claim 2,
Before performing the first step (S100), the stereo camera is moved to the position of the reference workpiece, and then photographed the 2D image of the box in which the reference workpiece is accommodated by the stereo camera at a distance. Setting the two-dimensional pattern information of the reference workpiece to the controller; and using the detected reference workpiece at a short distance, the projection matrix for the stereo camera and the left and right / images of the detected reference workpiece. And measuring a three-dimensional position with respect to a dog feature point and setting a coordinate system of the detected reference workpiece to a control unit. The method of claim 6,
The stereo camera 300, the correction is made by the correction plate arranged a plurality of square points at equal intervals,
Placing the correction plate in a fixed position, and re-calibration every predetermined time to check whether the setting is changed by comparing the calibration information of the preset stereo camera and the currently calculated stereo camera correction information for the workpiece pickup Industrial robot control method. The method of claim 2,
In the third step (S300), the industrial robot for picking up the workpiece further comprises the step of setting the trajectory for collision avoidance of the robot according to the six degrees of freedom variation measured using the stereo camera. Control method. The method of claim 8,
The setting of the collision avoidance trajectory of the robot may include: a work trajectory of a robot in which collision does not occur among the trajectories of the robot previously taught in consideration of the posture of the work object and the position where the work is placed in a box. Industrial robot control method for the workpiece pickup, characterized in that for setting in advance. The method of claim 8,
The setting of the collision avoidance trajectory of the robot may include changing the grip posture of the robot to be perpendicular to the bottom surface of the box when the workpiece is distributed in the outer region of the box in a severely inclined state. Industrial robot control method for picking up the workpiece. The method of claim 2,
In the third step (S300), the collision detection function of the control unit and the collision detection function of the online monitoring program to monitor whether there is a collision,
An industrial robot control method for picking up a workpiece by modeling in a virtual environment using an OLP (OLP) program and then monitoring in real time to change the holding posture of the robot in real time to prevent a collision.

Images