KR102111655B1 - Automatic calibration method and apparatus for robot vision system - Google Patents

Abstract

One aspect of the present invention discloses an automatic calibration method in a calibration apparatus connected to a camera disposed on an end effector of a robot and a robot controller for controlling the robot. The method includes the following steps of: obtaining an image for a marker marked on a working area of the robot and a robot reference coordinate system from the camera and the robot controller, wherein the acquired image and the robot reference coordinate system are recorded by moving the end effector with a plurality of sample coordinates; and estimating a location of a robot coordinate system reference marker using the acquired image and the robot reference coordinate system.

Description

AUTOMATIC CALIBRATION METHOD AND APPARATUS FOR ROBOT VISION SYSTEM

The present invention relates to a calibration method, and more particularly, to an automatic calibration method in a robot vision system.

The vision sensor is a very important component in the usability of the robot system, but an accurate conversion between the coordinate system of the vision sensor and the coordinate system of the robot must be preceded. However, it is very difficult to perform calibration on an arbitrary robot and an arbitrary camera without a separate device such as a marker having fixed coordinates in advance.

In particular, in the case of a low-cost camera, the distortion of the camera is severe, and even if the intrinsic parameters are estimated through calibration, it is often impossible to obtain the coordinate system perfectly. In order to use the incomplete camera coordinate system for robotic work, it is necessary to over-fit the work area. If it is based on a pre-defined position or there are constraints, calibration cannot be performed for any work site.

The object according to an embodiment of the present invention is a camera robot mounted on the robot, or the camera is fixed separately, the robot vision system used with the robot, such as the focal length, offset, distortion constant, the camera's unique parameters and the camera and the robot end effector Or, it provides a method to automatically calibrate the position and angular offset between the camera and the robot coordinate system at once.

According to an aspect of the present invention for achieving the above object, an automatic calibration method in a calibration device connected to a camera disposed on an end effector of a robot and a robot controller for controlling the robot, a marker marked on the working area of the robot Obtaining an image and a robot reference coordinate system for the camera and the robot controller (the acquired image and robot reference coordinate system are recorded by moving the end effector with a plurality of sample coordinates) and the acquired image and robot reference The method may include estimating the position of the robot coordinate system reference marker using the coordinate system.

The step of estimating the position of the robot coordinate system reference marker using the acquired image and the robot reference coordinate system is based on the camera in a plurality of postures using the non-linear optimization algorithm with the camera-robot offset (T _off ) as the assumed value. The method may include estimating the camera-robot offset (T _off ) through a method of minimizing the error rate of the coordinates obtained by converting the position of the marker (T _m,j ) into a robot reference coordinate system.

The camera-robot offset T _off may be calculated as a function of variables including xyz coordinates and zyx Euler angles.

The camera-robot offset (T _off ) expresses the degree of distortion between the first sample coordinates and the second sample coordinates as the values of the error and the position (P) of the rotation matrix (R), and uses a nonlinear optimization algorithm to minimize this Can be estimated.

을 통해 산출되며, 여기서, j는 표본 좌표의 인덱스를, T_j는 표본 좌표 j의 변환 행렬을, T_m,j는 표본 좌표 j에서 카메라 기준 마커의 위치를, R은 로테이션 매트릭스와 관련된 변수를, P는 포지션 에러와 관련된 변수를 나타낼 수 있다.The camera-robot offset (T _off ) is

, Where j is the index of the sample coordinate, T _j is the transformation matrix of the sample coordinate j, T _m,j is the position of the camera reference marker in the sample coordinate j, R is the variable related to the rotation matrix. , P may indicate a variable related to position error.

The end effector is provided with a movable area taught, and the movable area may be defined as a plurality of reference points as an area that is a reference for calibration.

The plurality of sample coordinates may be arranged in an area within the plurality of reference points.

In the plurality of sample coordinates, recording a transformation matrix from the reference coordinate system of the robot to the robot end at the plurality of reference points, setting a motion center point commonly viewed by the plurality of reference points, based on the set motion center point Record the variables that determine the position of the plurality of reference coordinates, extract the minimum and maximum values of each variable, and extract through the step of extracting the plurality of sample coordinates through random weighting between the plurality of reference points Can be.

The variables may include at least two of azimuth, elevation, roll, and distance from the motion center point.

A transformation matrix T _j of each sample coordinate system may be calculated from the variables of the extracted plurality of sample coordinates.

According to another aspect of the present invention for achieving the above object, a calibration method in a calibration device connected to a camera disposed at a position spaced apart from a robot and a robot controller for controlling the robot includes a marker disposed in a robot end effector. Obtaining an image and a robot reference coordinate system for the camera from the camera and the robot controller (recorded by moving the end effector with multiple sample coordinates) and estimating a robot base reference camera position from the acquired image. Can be.

The step of estimating the position of the robot base reference camera using the acquired image and the robot reference coordinate system, the robot in a plurality of postures using the non-linear optimization algorithm with the robot end-marker offset (T _em ) as the assumed value The method may include estimating the robot end-marker offset (T _em ) through a method of minimizing the error rate of coordinates obtained by converting the position of the base reference camera to the robot reference coordinate system.

The robot end-marker offset (T _em ) may be calculated as a function of variables including xyz coordinates and zyx Euler angles.

The robot end-marker offset (T _em ) may be estimated by using a nonlinear optimization algorithm that expresses the degree of distortion between the first sample coordinates and the second sample coordinates as the values of the error and the position error of the rotation matrix and minimizes them. .

을 통해 산출되며, 여기서, j는 표본 좌표의 인덱스를, T_j는 로봇의 변환 행렬을 나타낸다. T_em는 로봇 말단으로부터 마커로의 오프셋 추정값을 나타내고, T_m,j는 카메라로부터 마커의 위치를, R은 로테이션 매트릭스와 관련된 변수를, P는 포지션 에러와 관련된 변수를 나타낼 수 있다.The robot end-marker offset (T _em ) is

It is calculated from, where j is an index of sample coordinates, and T _j is a transformation matrix of the robot. T _em may represent an offset estimate from the robot end to the marker, T _m,j may indicate the position of the marker from the camera, R may be a variable related to the rotation matrix, and P may be a variable related to position error.

According to another aspect of the present invention for achieving the above object, the camera disposed on the end effector of the robot and the calibration device connected to the robot controller for controlling the robot, the image for the marker marked on the working area of the robot and Using the input unit for obtaining the robot reference coordinate system from the camera and the robot controller (the acquired image and the robot reference coordinate system are recorded by moving the end effector with a plurality of sample coordinates) and the acquired image and the robot reference coordinate system It may include a processor for estimating the position of the robot coordinate system reference marker.

According to another aspect of the present invention for achieving the above object, a camera disposed at a position spaced apart from a robot and a calibration device connected to a robot controller for controlling the robot, the image for the marker and the robot reference coordinate system to the camera And an input unit obtained from the robot controller (the acquired image and the robot reference coordinate system are recorded by moving the end effector with a plurality of sample coordinates) and a processor for estimating the robot base reference camera position from the acquired image. Can be.

According to the automatic calibration method according to an embodiment of the present invention, there is an effect that the robot vision system can be easily calibrated in an arbitrary environment without a reference coordinate system or a separate constraint in which a marker is positioned at an exact coordinate in advance.

In particular, according to the automatic calibration method according to an embodiment of the present invention, since the calibration can be overfitted accordingly in any work area, the robot performs precise work in an environment where there are many constraints even with a low-cost camera with severe distortion. It has the effect of making it possible.

1 is a block diagram schematically showing a robot vision system using an automatic calibration method according to an embodiment of the present invention,
Figure 2 is a flow chart schematically showing an automatic calibration method according to an embodiment of the present invention,
3 is a conceptual diagram illustrating a concept of designating N reference points in an automatic calibration method according to an embodiment of the present invention;
Figure 4 is a detailed flow chart specifically showing a method for uniformly selecting the sample coordinates of the automatic calibration method according to an embodiment of the present invention,
5 is a block diagram schematically showing an automatic calibration apparatus according to an embodiment of the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may be referred to as a first component. The term and/or includes a combination of a plurality of related described items or any one of a plurality of related described items.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but there may be other components in between. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “include” or “have” are intended to indicate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the overall understanding in describing the present invention, the same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

1 is a block diagram schematically illustrating a robot vision system using an automatic calibration method according to an embodiment of the present invention. 1, a robot vision system according to an embodiment of the present invention may include a camera 110, a robot controller 120, and a calibration device 130.

Referring to FIG. 1, the calibration device 130 interlocks with the camera 110 and the robot controller 120 to obtain image information taken from the camera 110, particularly a marker, and the robot controller 120 Obtain information related to robot control from. Information related to robot control may include variable information (elevation angle, azimuth, etc.) that determines coordinate information or position in the robot coordinate system when the robot is located in a specific area, such as a reference point or a sample coordinate. The robot controller 120 may include an encoder.

According to an embodiment of the present invention, the camera 110 is mounted on the end of the robot, that is, the end effector, to photograph a marker existing in the movable region of the robot and acquire the captured image information. This can be the first embodiment. In addition, in the second embodiment, the camera 110 is fixed to the robot and is separated from the robot, but a marker is attached to the end of the robot and the area where the marker is visible in the camera image is set as a movable area so that calibration is performed. can do.

The calibration device 130 may obtain information from the camera 110 and the robot controller 120 to automatically calibrate the offset values between the robot and the camera as well as the unique parameters of the camera at once. This will be described in more detail through FIG. 2 below.

2 is a flowchart schematically illustrating an automatic calibration method according to an embodiment of the present invention.

Referring to FIG. 2, a marker is placed at an arbitrary position in the working area of the robot, and the user moves the robot to N reference points and teaches the movable area (S210). At this time, methods such as direct teaching and indirect teaching may be used. This process will be described in more detail through FIG. 3.

3 is a conceptual diagram illustrating a concept of designating N reference points in an automatic calibration method according to an embodiment of the present invention.

Referring to FIG. 3, in order to perform calibration at an arbitrary working position, it is preferable that a user variably designates an area that is a reference for performing calibration. This may be performed by a user through a method such as direct teaching or indirect teaching. According to an embodiment of the present invention, a marker is placed on a certain area (working area of the robot) from a position where the base of the robot is located or a corresponding position, and a reference point that becomes an edge of an area to be moved while performing calibration N is specified.

According to the embodiment of the present invention, it is preferable that there are four reference points. From this reference point, N seed views can be obtained.

Returning to FIG. 2 again, the apparatus obtains a marker image by setting sample coordinates in an area inside the taught reference points, that is, a movable area, and moving to the corresponding coordinates (S220). According to an embodiment of the present invention, it is preferable to set about 15 to 25 sample coordinates, and more preferably, set 20 pieces. At this time, it is preferable to go through the process of Figure 4 to uniformly select the sample coordinates.

4 is a detailed flow chart specifically showing a method of uniformly selecting a sample coordinate of an automatic calibration method according to an embodiment of the present invention.

Referring to FIG. 4, the transformation matrix (T _i , i∈{1, ..., N}) from the robot reference coordinate system to the robot end is recorded at the four reference points set in step S210. Here, i represents the index of the reference point (S410).

Then, in order to extract the center point of the spherical coordinate system to be moved by the robot, the device sets a motion center point common to the reference points (S420). To this end, it is preferable to solve the following minimization problem and extract the point closest to the z-axis of all terminal coordinate systems as the motion center point (S420).

Then, the azimuth angle θ _i of the reference points, the elevation angle Ψ _i , the roll ω _i , and the distance d _i from the center of operation are recorded based on the motion center point set in step S420, and the minimum value of each variable and The maximum value is extracted (S430). This is represented by the following variable values. (θ _min , θ _max , Ψ _min , Ψ _max , ω _min , ω _max , d _min , d _max )

Next, the device extracts a plurality of arbitrary sample coordinates through a random weighted sum between the reference points (S440). At this time, the number of sample coordinates is about 15 to 25 is suitable, more preferably 20 is good. At this time, in order to obtain uniform sample coordinates in the spherical coordinate system, it is preferable to use the following equation.

Here, r represents a random number between 0 and 1.

Then, a transformation matrix T _j of each sample coordinate system is calculated from at least one of azimuth, elevation angle, roll, and distance of approximately 20 sample coordinates newly extracted (S450 ).

Returning to FIG. 2 again, after setting the sample coordinates and calculating a transformation matrix therefor, the end effector of the robot is moved to the calculated sample coordinates to obtain an image of the marker at each sample coordinate.

Then, the camera-specific parameters are estimated from the acquired image (S230), and the position (T _m,j ) of the camera reference marker is estimated from each image (S240). Estimated camera-specific parameters may include focal length, offset, and distortion constant.

Then, the position of the reference marker of the robot coordinate system is estimated using the estimated position of the camera reference marker (S250).

According to an embodiment of the present invention, to estimate the position of the robot coordinate system reference marker, the device uses a nonlinear optimization algorithm. The device uses a method that minimizes the error of the coordinates (T _j , T _off , T _m,j ) that convert the position value of the camera reference marker to the robot reference coordinate system in various postures to minimize the camera-robot offset (T _off ). To estimate. Here, T _m,j represents the position of the camera reference marker in a specific sample coordinate. j represents the index of the sample coordinates. At this time, various algorithms such as BroydenFletcher-GoldfarbShanno can be used as the nonlinear optimization algorithm. In particular, for the efficiency and convergence of the operation, T _off can be calculated by placing it as a function of 6 variables of xyz coordinate and zyx Euler angle. This can be expressed by the following equation.

Here, T _off enters an assumption value, and a nonlinear algorithm is executed in a manner that minimizes the error of T _off . Looking at the bottom expression of Equation 3, the sample matrix (j) is the transformation matrix of the sample coordinate system, the T _off value, and the position of the marker from the camera (T _{m,j-) 1} ) is calculated, and an inverse is taken. In addition, as the term related to the sample coordinate (j) is calculated in the forward direction, only the degree of distortion between the two points remains in the path to the sample coordinate (j-1) and then to the sample coordinate (j). This is expressed by the above equation. And, looking at the top equation of Equation 3, ∥log(ΔR _j )∥ represents the rotation matrix norm, r∥ΔR _j ∥ becomes the norm of the position error, and collectively the total error in relation to T _off Is formed, and as the sigma (Σ) value is taken, a value obtained by summing the error values in the photographing of the entire sample coordinates is generated, and the calibration process operates in a direction to minimize this.

)를 추정할 수 있다. 여기서, T_j는 로봇의 변환 행렬을 나타낸다. T_em는 로봇 말단으로부터 마커로의 오프셋 추정값을 나타내고, T_m,j는 카메라로부터 마커의 위치이다. 이를 수학식으로 나타내면 다음과 같다. According to another embodiment of the present invention, in a working environment in which the camera is separately installed, a marker is attached to the end of the robot, and the zero point logger reference point where the marker is visible on the camera image is set to similarly perform the series of processes of FIG. 2. The end-marker offset (T _em ) is estimated, and through this, the camera position relative to the robot base (

) Can be estimated. Here, T _j represents the transformation matrix of the robot. T _em represents the offset estimate from the robot end to the marker, and T _m,j is the position of the marker from the camera. This is expressed by the following equation.

T _m,j ^-1 becomes the transformation matrix from the marker to the camera, and the robot's reference in the sample coordinate (j) and the position of the camera from the robot reference coordinate system in the previous sample coordinate (j-1) of the sample coordinate (j). The robot end-marker offset (T _em ) as well as the robot base reference camera position can be estimated by minimizing the error of the rotation matrix and the position matrix for errors between the camera positions from the coordinate system.

5 is a block diagram schematically showing an automatic calibration device according to an embodiment of the present invention. As shown in FIG. 5, the automatic calibration device according to an embodiment of the present invention may include an input unit 510, a memory 520, a processor 520, an output unit 540 and a user interface 550. have.

The input unit 510 may acquire information from a camera and a robot controller (or robot encoder). The captured image information can be obtained from the camera, and the robot controller can obtain information representing the position of the robot in the robot coordinate system when the corresponding image is captured, and information related to the control command of the robot.

The memory 520 stores instructions related to an automatic calibration procedure in the processor 530.

The processor 530 calibrates the unique parameters of the camera and the robot-camera offset at once. The processor 530 places a marker at an arbitrary position in the working area, moves the robot to a plurality of reference coordinates to teach the movable area, and then, when the robot automatically moves in the movable area, acquired through the input unit 510 Receives a number of marker images and a robot coordinate system, estimates camera-specific parameters from the image, and estimates the position of the camera reference marker in each image. At this time, when the position of the camera reference marker is converted to the reference coordinate system of the robot in various postures using a nonlinear algorithm, the camera-robot offset that minimizes the error rate is estimated.

The output unit 540 outputs estimated camera-specific parameters and camera-robot offset information.

The user interface 550 is a component used to set various setting values to be set in the corresponding device.

Although described above with reference to the drawings and examples, the protection scope of the present invention is not meant to be limited by the drawings or examples, and those skilled in the art will think of the present invention described in the claims below And it will be understood that various modifications and changes may be made to the present invention without departing from the scope.

Claims (17)

In the automatic calibration method in the calibration device connected to the robot controller for controlling the robot and the camera disposed in the end effector of the robot,
Acquiring an image and a robot reference coordinate system for the markers marked on the working area of the robot from the camera and the robot controller, the acquired image and the robot reference coordinate system recorded by moving the end effector into a plurality of sample coordinates; And
And estimating the position of the robot coordinate system reference marker using the acquired image and the robot reference coordinate system,
Estimating the position of the robot coordinate system reference marker using the acquired image and the robot reference coordinate system,
How to minimize the error rate of the coordinates that convert the position of the camera reference marker (T _m,j ) to the robot reference coordinate system in multiple postures using the non-linear optimization algorithm with the camera-robot offset (T _off ) as the assumed value. Estimating the camera-robot offset (T _off ) through
The camera-robot offset (T _off ) is calculated as a function of variables including xyz coordinates and zyx Euler angle,
The camera-robot offset (T _off ) expresses the degree of distortion between the first sample coordinates and the second sample coordinates as the values of the error and the position (P) of the rotation matrix (R), and uses a nonlinear optimization algorithm to minimize this Is estimated by
The camera-robot offset (T _off ) is

Is calculated through,
Where j is the index of the sample coordinate, T _j is the transformation matrix of the sample coordinate j, T _m,j is the position of the camera reference marker in the sample coordinate j, R is the variable related to the rotation matrix, and P is the position error. Represents a variable related to
The end effector is provided with a movable area taught,
The movable area is an area that is a reference for calibration, and is defined by a plurality of reference points.
The plurality of sample coordinates are arranged in an area within the plurality of reference points,
Multiple sample coordinates,
Recording a transformation matrix from the reference coordinate system of the robot to the robot end at the plurality of reference points;
Setting an operation center point common to the plurality of reference points;
Recording variables determining the positions of the plurality of reference coordinates based on the set operation center point and extracting minimum and maximum values of each variable; And
It is extracted through the step of extracting the plurality of sample coordinates through a weighted random number between the plurality of reference points,
The variables include at least two of azimuth, elevation, roll and distance from the motion center point,
An automatic calibration method in which a transformation matrix (T _j ) of each sample coordinate system is calculated from the extracted variables of a plurality of sample coordinates. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images