KR20190027584A - Method and apparatus for calibration of a robot hand and a camera - Google Patents

Abstract

The present invention relates to a method and an apparatus for calibration of a camera and a robot hand and, more specifically, relates to a method for performing calibration between a camera and a robot hand in an apparatus having the robot hand and the camera attached to the hand, which comprises the following steps of: converting a pose of the robot hand in accordance with inputted pose information; and calculating a rotation matrix (R) between the camera and the robot hand when the pose information has only a movement value, and calculating a moving vector (t) between the camera and the robot hand when the pose information has only a rotation value.

Description

Technical Field [0001] The present invention relates to a method and an apparatus for calibrating a camera and a robot hand,

The present invention relates to a method and an apparatus for calibrating a camera and a robot hand, and more particularly to a method of calibrating a robot hand and a robot hand using a rotation posture and a movement posture, And apparatus.

Generally, a mechanical device that performs an action similar to a human motion by using an electric or magnetic action is called a robot. Early robots were industrial robots, such as manipulators and conveyor robots, aimed at automating and unmanning operations on the production floor. They performed dangerous work, simple repetitive work, and tasks requiring great power on behalf of human beings Recently, research and development of a humanoid robot, which has a joint system similar to a human and provides various services coexisting with humans in a human work and living space, is actively under way.

Such a humanoid robot performs work using a manipulator designed to move close to the motion of an arm or a hand by an electrical and mechanical mechanism. Most manipulators currently in use are configured with multiple links linked together. The joint of each link is referred to as a joint, and the manipulator determines the motion characteristic according to the geometric relationship between the link and the joints. The mathematical representation of this geometric relationship is kinematics, and most manipulators move the robot hand (eg, the end effector) in a direction (target position) for performing a task with this kinematics characteristic .

On the other hand, the correction of the robot system with the camera is largely classified into the correction of the robot base and the robot hand, the correction of the camera, and the correction of the robot base or the hand / eye calibration of the robot hand and the camera.

The calibration method of the robot and the camera used in the industrial field generally uses a separate manipulator called a teach pendant. Using a separate manipulator called a teach pendant, the robot's hand is taught to a known position of the object, and the conversion relation between the camera and the robot's hand can be estimated through a plurality of iterative processes.

In the case of using Teach Pendant, it is required for the operator to understand the teaching menu and programming language, sense of space, robotic system and kinematic analysis ability, and it takes much time and troublesome operation for the operator to operate.

In addition, when the robot's hand is directly instructed, the operator must directly instruct the robot's hand on the side of the robot or in the vicinity of the robot, so that the operator's safety problem may occur according to the malfunction of the robot. The accuracy can be lowered.

In addition, the calibration of the camera itself has to be performed separately, which makes it difficult for the operator to obtain the parameters of the camera inside / outside the camera without prior knowledge of the camera, which is inconvenient and time consuming. A tendency to ignore the distortion of the lens is generally suggested, and a way to improve it is required.

Korean Prior Art 10-2014-0054927 (entitled "Automatic Calibration Method of Robot") is a prior art related to this.

The present invention provides a method and an apparatus for calibrating a camera and a robot hand capable of performing calibration between a camera and a robot hand as well as a calibration of a camera itself attached to the robot hand using a moving posture and a rotation posture of the robot hand .

The problems to be solved by the present invention are not limited to the above-mentioned problem (s), and another problem (s) not mentioned can be clearly understood by those skilled in the art from the following description.

A method of calibrating a camera and a robot hand according to an embodiment of the present invention is a method of calibrating a camera hand and a robot hand in an apparatus including a robot hand and a camera attached to the hand, Calculating a rotation matrix R between the camera and the robot hand when the attitude information includes only the movement value, and if the attitude information includes only the rotation value, And calculating a motion vector (t) between the camera and the robot hand.

Preferably, the method further includes a step of deriving a relational expression for calculating a posture conversion relation between the camera and the robot hand using the posture conversion relationship from the camera coordinate system after posture transformation to the robot hand coordinate system before posture transformation.

Preferably, the posture conversion relationship may include a rotation matrix and a motion vector.

Preferably, the relational expression may be a mathematical expression described below.

[Mathematical Expression]

Where Rc is the rotation matrix of the camera, tc is the motion vector of the camera, R _E is the rotation matrix of the robot hand, t _E is the motion vector of the robot hand, R is the rotation matrix between the camera and the robot hand, Hand motion vector.

Preferably, when the attitude information includes only the movement value, the step of calculating the rotation matrix R between the camera and the robot hand may include photographing the correction block in which the coordinate system is set in advance via the camera, Calculating a first camera coordinate system in a first camera posture and a second camera coordinate system in a second camera posture on the basis of a coordinate system of a correction block, calculating a posture of the camera using the first camera coordinate system and the second camera coordinate system, Calculating a conversion relationship between the camera and the robot hand, calculating a posture conversion relation of the robot hand using the posture information, and calculating a rotation matrix between the camera and the robot hand using the posture conversion relation of the camera and the posture conversion relation of the robot hand And a step of calculating the number of steps.

Preferably, the rotation matrix of the camera and the rotation matrix of the robot hand are I matrices, and the rotation matrix between the camera and the robot hand can be calculated by applying a pseudo inverse to the mathematical expression described below.

[Mathematical Expression]

Here, tc is the motion vector of the camera, t _E is the motion vector of the robot hand, and R is the rotation matrix between the camera and the robot hand.

Preferably, when the attitude information includes only the rotation value, calculating the motion vector t between the camera and the robot hand may include photographing a correction block in which a coordinate system is set in advance via the camera, Calculating a first camera coordinate system in a first camera posture and a second camera coordinate system in a second camera posture on the basis of a coordinate system of a block using a first camera coordinate system and a second camera coordinate system, Calculating a motion vector between the camera and the robot hand using the posture conversion relationship of the camera and the posture conversion relationship of the robot hand; .

Preferably, the motion vector t _e of the robot hand is '0', and the motion vector t between the camera and the robot hand can be calculated by applying a pseudo inverse to the following equation have.

[Mathematical Expression]

Here, Rc may be a rotation matrix of the camera, tc may be a motion vector of the camera, and t may be a motion vector between the camera and the robot hand.

Preferably, the step of performing the calibration between the camera and the robot hand using the posture conversion relationship between the calculated camera and the robot hand may be further included.

There is provided an apparatus for calibrating a robot hand and a camera attached to the robot hand according to another embodiment of the present invention, the apparatus comprising: a user interface unit for receiving attitude information of the robot hand; (R) between the camera and the robot hand when the robot hand does not rotate, and calculates a rotation matrix (R) between the camera and the robot hand when the robot hand only rotates, And a calibration control unit for calculating the vector t.

Preferably, the calibration control unit photographs a correction block in which a coordinate system has been set in advance through the camera when the robot hand moves only without rotating the camera, Calculating a first camera coordinate system of the first camera coordinate system and a second camera coordinate system of the second camera coordinate system, calculating a posture change relation of the camera using the first camera coordinate system and the second camera coordinate system, Calculating a rotation matrix between the camera and the robot hand using the posture conversion relation of the camera and the posture conversion relationship of the robot hand, wherein the rotation matrix between the camera and the robot hand is Can be calculated by applying a pseudo inverse to the described mathematical expression.

[Mathematical Expression]

Here, tc is the motion vector of the camera, t _E is the motion vector of the robot hand, and R is the rotation matrix between the camera and the robot hand.

Preferably, the calibration control unit photographs a correction block in which a coordinate system has been set in advance through the camera when the robot hand only rotates, and the first control unit Calculating a posture change relation of the camera using the first camera coordinate system and the second camera coordinate system, calculating a posture change relation of the camera using the first camera coordinate system and the second camera coordinate system, (T) between the camera and the robot hand using the posture conversion relationship of the camera and the posture conversion relationship of the robot hand, and calculates a motion vector (t) between the camera and the robot hand ) Can be calculated by applying a pseudo inverse to the equation described below.

[Mathematical Expression]

Here, Rc may be a rotation matrix of the camera, tc may be a motion vector of the camera, and t may be a motion vector between the camera and the robot hand.

Preferably, when the apparatus is provided with a stereo-based 3D camera equipped with a second camera in addition to the first camera attached on the robot hand, the calibration control unit may control the first camera Calculates a posture change relation between the first camera and the second camera using the coordinate system of the first camera and the coordinate system of the second camera, and calculates a posture change relation between the calculated two cameras using the coordinate system of the first camera and the coordinate system of the second camera, The relationship can be used to perform calibration of the 3D camera.

Preferably, when the camera attached to the robot hand is a structured light-based 3D camera equipped with a projector, the calibration control unit controls the coordinate system of the camera based on the coordinate system of the correction block, Calculating a posture change relationship between the camera and the projector using the calculated coordinate system of the camera and the coordinate system of the projector, and calculating a posture change relationship between the calculated camera and the projector, Calibration can be performed.

According to the present invention, the calibration between the camera and the robot hand can be performed as well as the calibration of the camera itself attached to the robot hand by using the movement posture and the rotation posture of the robot hand.

Further, according to the present invention, it is possible to acquire a more accurate positional relationship between the three-dimensional camera and the robot hand, and the calibration step of the three-dimensional camera itself is simplified, so that the production and installation of the robot manipulator of the automation system becomes easier and more accurate, This makes the factory automation system more active.

The effects of the present invention are not limited to the above-mentioned effects, and various effects can be included within the scope of what is well known to a person skilled in the art from the following description.

FIG. 1 is a view for explaining a robot that performs calibration of a camera and a manipulator end effector according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a control device for calibration of a camera and an end effecter according to an embodiment of the present invention.
3 to 7 are diagrams for explaining a method for calculating a posture change relation between a camera and an end effector according to an embodiment of the present invention.
FIG. 8 illustrates a method for calibrating a camera and an end effector according to an embodiment of the present invention. Referring to FIG.
9 is a flowchart illustrating a method of calculating a rotation matrix between a camera and an end effector according to an embodiment of the present invention.
10 is a flowchart illustrating a method of calculating a motion vector between a camera and an end effector according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

On the other hand, the robot hand is a term including an end effector, a gripper, and the like, and will be described below as an end effector only for convenience of explanation.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining a robot performing calibration between a camera and a manipulator end effector according to an embodiment of the present invention.

1, a robot 100 for performing calibration between a camera and a manipulator includes a first manipulator 111, a first connection unit 110 for connecting the first manipulator 111, An end effector 123 connected to the end of the second manipulator 122, a camera 120 coupled to an upper end of one side of the second manipulator 122, (Not shown). The robot 100 can perform three-dimensional motion by rotating the first connection part 110 and the second connection part 121 under the control of the controller. In addition, the second manipulator 122 includes the rotation unit 124 to allow the end effector 123 and the camera 120 to rotate integrally.

The end effector 123 refers to a part having a function of directly acting on a work object (e.g., a gripper, a welding torch, a spray gun, a nut runner, etc.) when the robot is working, and may correspond to a robot hand.

The camera 120 acquires an image necessary for operation of the robot. The object in the world coordinate system acquired by the camera 120 may be a two-dimensional plane or a three-dimensional solid object. In another embodiment, the camera 120 may be attached to the end effector 123. In another embodiment, the rotating portion 124 is created at the coupling portion of the second manipulator 122 and the end effector 123, and the camera is attached to the end effector 123 to rotate the end effector 123 without rotation of the second manipulator 121, Only the camera 123 and the camera 120 may rotate.

The camera 120 according to the present embodiment may be, for example, a three-dimensional camera.

The control device (not shown) moves the end effector 123 to a desired position by controlling the first connection part 120 and the second connection part 121 using the image acquired through the camera 120, 123 to operate the robot.

Further, when the posture of the end effector 123 and the camera 120 is changed, the controller calculates the camera coordinate system in each posture based on the predetermined correction block. Then, the controller calculates the posture conversion relationship of the camera 120 and the posture conversion relation of the end effector 123 using the camera coordinate system and the end effector coordinate system, and calculates the posture conversion relationship of the calculated camera 120 and the end effector The posture conversion relationship between the camera 120 and the end effector 123 is calculated using the posture conversion relationship of the camera 123 and the end effector 123. [ Such a control device may exist inside or outside the robot.

A detailed description of the control device will be given with reference to FIG.

The manipulators 111 and 122, the end effector 123 and the camera 120 may be connected to a control device and a wired communication means such as an I / O bus, a USB or an RS 232c or an IR infrared communication, a Bluetooth, a Zigbee ZigBee), and Radio Frequency Identification (RFID), so that information can be transmitted and received through wired communication or wireless communication.

The correction block 200 serves as an object to be imaged by the camera 220. The position information of the correction block 200 is set in advance and the correction block 200 is used to perform the calibration operation of the robot 100. [ (Correction block coordinate system) is used as a reference in calculating the camera coordinate system. As described above, the correction block 200 is used as a reference in calculating the camera coordinate system. For example, the correction block 200 may have various forms such as a block having an outer surface having a plurality of planes, a three-dimensional solid object, .

Hereinafter, the case where the correction block 200 is formed to have a plurality of planes will be described for convenience of explanation.

The correction block 200 is formed to have a plurality of planes, that is, a certain angle can be formed at a portion where the one face and the face are in contact with each other, and a specific pattern is printed on one face. At this time, the pattern is displayed in such a manner that each of the plurality of planes can be recognized and recognized differently, and the pattern may be in the form of, for example, a chessboard pattern or the like. These markers are used to discriminate the respective planes from each other and may be arbitrarily set marks, and it is preferable that any one of numbers, bar codes and points is selected and displayed.

In other words, the correction block 200 is formed such that a specific pattern is printed on each of multiple planes in a plurality of multiple planes, and a marking portion arbitrarily set in the pattern is included. That is, when the correction block is photographed in any one direction, two or more planes are formed so as to have multiple planes so that they can be identified and photographed.

On the other hand, it is preceded to set the three-dimensional position coordinate value (coordinate system) for each face of the correction block 200 in advance based on the information of the pattern and the mark portion displayed on each face of the correction block 120.

Therefore, in performing the calibration operation of the robot 100, the position coordinates of the correction block 200, that is, the three-dimensional coordinate values (positions and corner points) with respect to the positions of the respective faces of the correction block 200, And the position information of the correction block 200 is used as a coordinate system in the calculation of the position information of the camera 120 during the calibration operation.

Meanwhile, in an embodiment of the present invention, a chessboard board is used as a correction block for calibrating a camera and calibration between a camera and an end effector. In another embodiment of the present invention, a three- Can be used to perform calibration. In addition, the calibration of the camera itself can perform the camera calibration using the image acquired during the hand / iso self correction.

FIG. 2 is a block diagram showing a configuration of a control device for calibrating a camera and an end effecter according to an embodiment of the present invention, and FIGS. 3 to 7 are diagrams showing a posture change relationship between a camera and an end effector according to an embodiment of the present invention And Fig.

Referring to FIG. 2, a control device 150 for calibrating a camera and an end effector includes a user interface unit 152, a posture control unit 154, and a calibration control unit 156.

The user interface unit 152 receives a work instruction including attitude information for manipulating the manipulator and the end effector, and provides the work instruction to the posture control unit 154. At this time, the user interface unit 152 receives a work command to be performed by the manipulator, particularly, the end effector, through a switch operation, a voice, or the like. In addition, the user interface unit 152 can receive a work command through wired or wireless communication.

The posture control unit 154 controls the robot driving unit according to the work command received from the user interface unit 152 to control the posture of the end effector provided at the tip of the manipulator by driving the manipulator. At this time, the end effector can rotate or translate according to the work command.

The calibration control unit 156 acquires the rotation relationship between the three-dimensional camera and the robot manipulator using only the reference posture and the plurality of movement postures, and calculates the movement relationship between the three-dimensional camera and the robot manipulator .

)로부터 엔드 이펙터 좌표계(

) 간의 자세 변환 관계(

)를 산출할 수 있다. That is, when the work instruction including the attitude information is inputted and the pose of the camera installed on the end effector and the end effector of the manipulator is changed, the calibration control unit 156 sets the attitude change relation of the end effector And the posture conversion relation of the camera are respectively calculated. Then, the calibration control unit 156 calculates the posture conversion relationship between the camera and the end effector using the posture conversion relationship of the end effector and the posture conversion relation of the camera. That is, the calibration control unit 156 controls the camera coordinate system

) To the end effector coordinate system (

) Of the posture change relation (

) Can be calculated.

Hereinafter, a method of calculating the posture conversion relationship between the camera and the end effector will be described in detail.

)에서 제2 엔드 이펙터 자세(

)로 변환되면, 매니큘레이터 상에 설치된 카메라의 자세 또한 제1 카메라 자세(

)에서 제2 카메라 자세(

)로 변하게 된다. And the posture of the end effector is determined as the first end effector posture (

) To the second end effector posture (

), The posture of the camera provided on the manipulator is also changed to the first camera posture (

) To the second camera posture (

).

), 제2 카메라 자세에서의 제2 카메라 좌표계(

)를 각각 산출할 수 있다. 즉, 캘리브레이션 제어부(156)는 카메라를 통해 촬영된 보정블록의 영상을 인식하여 보정블록에 표시된 패턴 및 표식부를 확인하고, 기설정된 보정블록 좌표계(위치정보)를 기준으로 제1 카메라 좌표계(

)와 제2 카메라 좌표계(

)를 산출할 수 있다.Then, the calibration control unit 156 controls the first camera coordinate system (first camera coordinate system) in the first camera posture based on the predetermined correction block coordinate system (O _object )

), A second camera coordinate system (

Respectively. That is, the calibration control unit 156 recognizes the image of the correction block photographed by the camera and confirms the pattern and the marking portion displayed in the correction block, and detects the pattern and marking unit displayed on the first camera coordinate system

) And the second camera coordinate system

) Can be calculated.

A method of calculating the camera coordinate system from the correction block coordinate system will be described with reference to FIG. 5, the coordinates (X, Y, Z) of a point P on the correction block and the coordinates u, v of a point p projected on the camera image sensor are used to calculate a projection matrix P ) Can be obtained. Here, the projection matrix may be a matrix representing 3D space projected on the 2D screen.

The relationship of FIG. 5 can be expressed by the following equation (1).

Here, [lambda] may be a scale factor.

Equation (1) is summarized as Equation (2).

If the lambda expression is substituted into another expression in the expression (2), the expression (3) is summarized.

Equation (3) can be summarized to obtain a projection matrix (P) as shown in Equation (4).

Using the expression (4), the projection matrix P can be obtained.

On the other hand, if the projection matrix P is decomposed, an internal factor matrix based on internal factors and an external factor matrix based on external factors can be obtained as shown in Equation 5 below.

The intrinsic parameters include the position of the center of the image, the scale representation value between the CCD / CMOS sensor surface and the image plane, the focal length of the camera and the principal point, and the outsider describes the conversion relationship between the camera coordinate system and the world coordinate system. And is expressed as rotation and translation between two coordinate systems. Here, the outlier is represented by R and t values, and the values of R and t are values indicating the conversion relation between the object axis and the camera axis, R is the rotation conversion displacement, and t is the translation displacement displacement value.

On the other hand, the camera calibration uses five internal factors (fx, fy, fx, cx, and asymmetry coefficient as principal points) and six external factors in the camera correction block Camera calibration tool). These camera parameters are needed to calculate the geometric relationships for 3D measurements, and the error of the calibration factors will have a significant impact on the accuracy of the 3D measurement indices.

The calibration control unit 156 can obtain the rotation matrix R and the movement vector t for converting the world coordinate system into the camera coordinate system since the outsider expresses the rotation and the translation between the two coordinate systems. have.

As described above, the calibration control unit can calculate the first camera coordinate system and the second camera coordinate system from the projection matrix P, respectively.

When the first camera coordinate system and the second camera coordinate system are calculated, the calibration control unit 156 can calculate the posture change relationship from the first camera posture to the second camera posture using the first camera coordinate system and the second camera coordinate system .

)와 보정블록으로부터 제2 카메라 자세까지의 카메라 좌표계(

)를 연산하여 제1 카메라 좌표계에서 제2 카메라 좌표계의 자세변환관계(

)를 산출할 수 있다. 즉, 카메라의 자세변환관계(

)는 아래 기재된 수학식 6을 이용하여 산출할 수 있다.
Referring again to FIG. 4, the calibration control unit 156 calculates the camera coordinate system

And the camera coordinate system from the correction block to the second camera posture

) Of the second camera coordinate system in the first camera coordinate system

) Can be calculated. In other words,

) Can be calculated using Equation (6) described below.

)는 사용자가 입력한 자세정보에 따라 엔드 이펙터가 제1 엔드 이펙터 자세에서 제2 엔드 이펙터 자세로 이동시켜 얻어낸 변환 관계로, 사용자가 엔드 이펙터를 병진 및 회전 운동시킨 것이므로 알고 있는 관계이다.Post-conversion relation of end effector (

Is a relationship that the user knows and knows that the end effector has been translated and rotated by the conversion relationship obtained by moving the end effector from the first end effector attitude to the second end effector attitude according to the attitude information input by the user.

When the posture conversion relationship between the camera and the end effector is calculated as described above, the camera can calculate the posture conversion relationship between the camera and the end effector using the posture conversion relationship between the camera and the end effector .

)와 제2 카메라 좌표계에서 제2 엔드 이펙터 좌표계의 자세변환관계(

)는 카메라와 엔드 이펙터 간의 자세 변환 관계(

)로 동일함을 알 수 있다. 따라서, 캘리브레이션 제어부(156)는 카메라와 엔드 이펙터 간의 자세 변환 관계를 산출하기 위해, 카메라의 자세 변환 관계와 엔드 이펙터의 자세 변환 관계를 이용할 수 있다.Reference is made to Fig. 6 for explaining a method in which the calibration control unit 156 calculates the posture conversion relationship between the camera and the end effector. Referring to FIG. 6, in the first camera coordinate system, the posture conversion relationship (

) And the second posture conversion relationship of the second end-effector coordinate system in the second camera coordinate system

) Is the posture conversion relationship between the camera and the end effector

). ≪ / RTI > Therefore, in order to calculate the posture change relation between the camera and the end effector, the calibration control unit 156 can use the posture change relation of the camera and the posture change relation of the end effector.

)에서부터 자세 변환 전의 제1 엔드 이펙터 자세(

)까지의 자세 변환 관계(

)는 도 7에 도시된 점선 화살표와 같이 벡터 합에 의해 산출할 수 있다. 따라서, 제2 카메라 자세(

)에서부터 제1 엔드 이펙터 자세(

)까지의 자세 변환 관계(

)는 아래 기재된 수학식 7이 성립한다.On the other hand, the second camera posture

) To the first end effector posture before the posture change (

) Of the posture change relation (

Can be calculated by the vector sum as shown by the dotted arrow in Fig. Therefore, the second camera posture (

) To the first end effector posture (

) Of the posture change relation (

(7) < / RTI >

)에서부터 자세 변환 전의 제1 엔드 이펙터 자세(

)까지의 자세 변환 관계(

)는 제2 카메라 좌표계에서 제1 카메라 좌표계까지의 자세 변환 관계(

), 카메라와 엔드 이펙터 간의 자세변환관계(

)를 이용하여 산출할 수 있다. 또한, 자세 변환 후의 제2 카메라 자세(

)에서부터 자세 변환 전의 제1 엔드 이펙터 자세(

)까지의 자세 변환 관계(

)는 제2 엔드 이펙터 좌표계에서 제1 엔드 이펙터 좌표계까지의 자세 변환 관계(

), 카메라와 엔드 이펙터 간의 자세변환관계(

)를 이용하여 산출할 수 있다.That is, the second camera posture after posture conversion

) To the first end effector posture before the posture change (

) Of the posture change relation (

) Represents the posture change relationship from the second camera coordinate system to the first camera coordinate system

), Posture conversion relation between camera and end effector (

). ≪ / RTI > Further, the second camera posture after posture conversion

) To the first end effector posture before the posture change (

) Of the posture change relation (

) Is a posture change relationship from the second end effector coordinate system to the first end effector coordinate system

), Posture conversion relation between camera and end effector (

). ≪ / RTI >

)과 이동벡터(

)를 이용하여 아래 기재된 수학식 8과 같이 나타낼 수 있다.On the other hand, the matrix T indicating the posture change relation is a rotation matrix (

) And the motion vector (

) As shown in Equation (8) below.

(T) representing the posture change relation of the expression (7) are as shown in the following expression (9).

Where Rc is the camera rotation matrix, tc is the camera motion vector, R _E is the end effector rotation matrix, t _E is the end effector motion vector, R is the rotation matrix between the camera-end effector, It can mean. The camera rotation matrix Rc is a matrix representing a rotation transformation relation between the first camera coordinate system and the second camera coordinate system. The camera movement vector tc is a vector representing a translation transformation relation between the first camera coordinate system and the second camera coordinate system. . Between the end effector rotation matrix (R _E) has a first end effector coordinate system and the second is a matrix indicating the rotational transformation relation between the end effector coordinate system, the end effector motion vector (t _E) includes a first end effector coordinate system to the second end-effector coordinate system May refer to a vector representing a translational transformation relationship. The rotation matrix (R) between the camera and the end effector is a matrix representing the rotational transformation relationship between the camera coordinate system and the end effector coordinate system. The motion vector (t) between the camera and the end effector is a translation transformation relationship between the camera coordinate system and the end effector coordinate system May denote a vector representing < RTI ID = 0.0 >

The expression (9) can be summarized as the following expression (10).

Here, Rc denotes a rotation matrix of the camera, tc denotes a motion vector of the camera, t _E denotes a motion vector of the end effector, R denotes a rotation matrix between the camera and the end effector, and t denotes a motion vector between the camera and the end effector.

Since the camera rotation matrix Rc, the camera motion vector tc, the end effector rotation matrix R _E and the end effector motion vector t _E are known in Equation 10, the posture change relation between the camera and the end effector is calculated In order to do so, the rotation matrix R between the camera and the end effector and the motion vector t between the camera and the end effector must be calculated. Here, the camera rotation matrix Rc and the camera motion vector tc can be known from the posture conversion relationship of the camera, and the end effector rotation matrix R _E and the end effector motion vector t _E can be obtained by the posture conversion relationship .

The rotation matrix R between the camera and the end effector can be calculated when the end effector only moves without rotating and the motion vector t between the camera and the end effector can be calculated when the end effector rotates only .

First, a method of calculating the rotation matrix R between the camera and the end effector will be described.

When the posture information including only the movement value without rotation is input in the posture conversion of the end effector, the end effector and the camera are moved by the movement value. At this time, since the end effector and the camera do not rotate, the rotation matrix of the end effector and the rotation matrix of the camera can be substituted as shown in Equation (11).

Applying Equation (11) to Equation (10), Equation (10) can be simplified as Equation (12) below.

Since the rotation matrix R is a 3x3 matrix, the pseudo inverse can be applied to Equation (12) to convert to Equation (13).

)들을 산출할 수 있다. Since the camera motion vector t _c including the end effector motion vectors t _E , x _c , y _c , and z _c including x _E , y _E , and z _E in Equation 13 is known, ) Of each Element (

). ≪ / RTI >

Next, a method of calculating the motion vector t between the camera and the end effector will be described.

)일 수 있다. When the posture information including only the rotation value is input in the posture conversion of the end effector, the end effector and the camera rotate by the rotation value. At this time, the end effector rotates without moving, and the camera rotates and moves. Since the end effector does not move, the motion vector t _E of the end effector is '0' (

).

Thus, Equation (10) can be simplified as Equation (14) described below.

Equation (14) can be expressed by a matrix as shown in Equation (15).

Applying Pseudo Inverse to (15), the motion vector t between the camera and the end effector can be calculated.

The calibration control unit 156 can automatically perform calibration between the camera and the end effector as well as the camera itself using the rotation matrix and the motion vector between the camera and the end effector.

) 또는 카메라와 프로젝터 간의 자세 변환 관계(

)를 함께 산출할 있으며, 이를 통해 카메라 자체의 캘리브레이션을 수행할 수 있다.Meanwhile, in the embodiment of the present invention, the calibration is performed using one camera attached to the robot. However, in another embodiment of the present invention, the 3D camera based on the Stereo or Structured Light is calibrated by the calibration of the 3D camera itself . ≪ / RTI > That is, when using a stereoscopic or structured light based 3D camera, the coordinate system of the second camera or the projector can be calculated in the same manner as the first camera coordinate system (or the second camera coordinate system) based on the correction block , And the posture conversion relation between the first camera and the second camera

) Or the posture conversion relationship between the camera and the projector

), Which allows the camera itself to be calibrated.

)와 두번째 카메라의 좌표계(

)를 각각 산출할 수 있다. 첫번째 카메라의 좌표계(

)와 두번째 카메라의 좌표계(

)를가 산출되면, 캘리브레이션 제어부(156)은 첫번째 카메라의 좌표계(

)와 두번째 카메라의 좌표계(

)를 이용하여 첫번째 카메라와 두번째 카메라 간의 자세 변환 관계(

)를 산출할 수 있고, 이를 이용하여 카메라 자체의 캘리브레이션을 수행할 수 있다. For example, when a stereo camera equipped with a second camera other than the first camera attached on the robot hand is used, the calibration control unit 156 controls the first camera's coordinate system

) And the coordinate system of the second camera (

Respectively. The coordinate system of the first camera (

) And the coordinate system of the second camera (

) Is calculated, the calibration control unit 156 calculates the coordinate system of the first camera

) And the coordinate system of the second camera (

), The posture conversion relation between the first camera and the second camera

), And can calibrate the camera itself by using it.

)를 각각 산출할 수 있다. 카메라의 좌표계와 프로젝터 좌표계(

)가 산출되면, 캘리브레이션 제어부(156)은 카메라의 좌표계와 제 프로젝터 좌표계(

)를 이용하여 카메라와 프로젝터 간의 자세 변환 관계(

)를 산출할 수 있고, 이를 이용하여 카메라 자체의 캘리브레이션을 수행할 수 있다. In addition, when a 3D camera based on a Structured Light equipped with a camera and a projector attached to the robot hand is used, the calibration control unit 156 controls the camera coordinate system and the projector coordinate system

Respectively. The camera coordinate system and the projector coordinate system (

), The calibration control unit 156 controls the camera coordinate system and the projector coordinate system

), The posture change relation between the camera and the projector

), And can calibrate the camera itself by using it.

8 is a view for explaining a method for calibrating a camera robot manipulator according to an embodiment of the present invention.

Referring to FIG. 8, when the posture information of the end effector is inputted (S810), the posture of the end effector and the camera are converted according to the posture information (S820). That is, when the end effector is converted from the first end effector attitude to the second end effector attitude according to the attitude information, the camera installed on the end effector is also changed from the first camera attitude to the second camera attitude. Here, the first end effector attitude and the first camera attitude mean before the attitude change, and the second end effector attitude and the second camera attitude may mean after the attitude change.

If step S820 is performed, the controller calculates a rotation matrix R between the camera and the end effector (S830). If the control information includes only the rotation value, the controller calculates the rotation matrix R between the camera and the end effector The motion vector t is calculated (S840).

Referring to FIG. 9, a method of calculating a rotation matrix between a camera and an end effector will be described with reference to FIG. 10 for a method of calculating a motion vector between a camera and an end effector.

When step S840 is performed, the controller calculates a posture change relationship between the camera and the end effector, which is composed of the rotation matrix and the motion vector (S850).

The control device can perform calibration between the end effector and the camera by calculating the rotation matrix and the motion vector between the camera and the end effector.

9 is a flowchart illustrating a method of calculating a rotation matrix between a camera and an end effector according to an embodiment of the present invention.

), 제2 카메라 자세에서의 제2 카메라 좌표계(

)를 산출한다(S920).9, when the posture information including only the movement value is inputted (S910), the controller calculates the first camera coordinate system

), A second camera coordinate system (

(S920).

When step S920 is performed, the control device calculates the posture change relation of the camera using the first camera coordinate system and the second camera coordinate system (S930).

The posture conversion relationship of the end effector is previously known because the operator inputs the end effector of the robot to convert from the first end effector attitude to the second end effector attitude.

If the posture information includes only the movement value, the end effector and the camera do not rotate, so that the rotation matrix of the end effector and the camera may be I. Therefore, the posture conversion relation of the camera includes only the motion vector of the camera, and the posture conversion relation of the end effector may include only the motion vector of the end effector.

When step S930 is performed, the control device calculates a rotation matrix between the camera and the end effector using the posture change relation of the camera and the posture change relation of the end effector (S940). That is, since the posture change relation of the camera includes only the motion vector of the camera, and the posture change relation of the end effector includes only the motion vector of the end effector, the controller uses the above- The matrix can be calculated.

10 is a flowchart illustrating a method of calculating a motion vector between a camera and an end effector according to an embodiment of the present invention.

), 제2 카메라 자세에서의 제2 카메라 좌표계를 산출한다(S1020).Referring to FIG. 10, when the posture information including only the rotation value is inputted (S1010), the controller calculates a first camera coordinate system

, The second camera coordinate system in the second camera posture is calculated (S1020).

When step S1020 is performed, the controller calculates the posture change relation of the camera using the first camera coordinate system and the second camera coordinate system (S1030).

The posture conversion relationship of the end effector is previously known because the operator inputs the end effector of the robot to convert from the first end effector attitude to the second end effector attitude.

When the attitude information includes only the rotation value, the end effector does not move, so that the motion vector of the end effector may be '0'. Accordingly, the posture conversion relationship of the camera includes the rotation matrix and the motion vector of the camera, and the posture conversion relation of the end effector may include only the rotation matrix of the end effector.

When step S1030 is performed, the controller calculates a motion vector between the camera and the end effector using the posture conversion relationship of the camera and the posture conversion relation of the end effector (S1040). That is, the posture conversion relation of the camera includes the rotation matrix and the motion vector of the camera, and the posture conversion relation of the end effector includes only the rotation matrix of the end effector. Therefore, Can be calculated.

The calibration between the end effector and the camera according to another embodiment of the present invention is carried out by attaching one or more cameras attached to the end effector and the end effector and moving and rotating together with the end effector to perform the following operations. First, we define the azimuth to be the reference, and acquire images of feature points known at that location. Secondly, we acquire an image taken at the position of the end effector with pure translational motion that does not include rotation at the reference position. Third, the end effector is rotated twice about an arbitrary axis without moving at the first reference position, or the end effector is subjected to translational motion and rotational motion twice on an arbitrary axis to obtain an image. Therefore, one image at the initial position, three images with purely translational motion, and two arbitrary images are acquired by performing rotation or translation and rotation on arbitrary axes to obtain information through a total of 6 images, It is possible to calibrate the camera attached to the end effector and end effector.

Also, in an embodiment of the present invention, a chessboard board is used as a correction block for calibrating a camera and calibration between a camera and an end effector, but in another embodiment of the present invention, a three- Can be used to perform calibration. In addition, the calibration of the camera itself can be corrected using the above-described camera calibration method using the image acquired during the hand / iso self correction.

The above-described embodiments of the present invention can be embodied in a general-purpose digital computer that can be embodied as a program that can be executed by a computer and operates the program using a computer-readable recording medium.

The computer readable recording medium includes a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), optical reading medium (e.g., CD ROM, DVD, etc.).

The present invention has been described with reference to the preferred embodiments. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

100: Robot
111, 122: Manipulators
113: first connection portion
121: second connection portion
123: End effector
124:
150: Control device
152: User interface section
154:
156: Calibration control unit
200: correction block

Claims (14)

A method for performing a calibration between a camera hand and a robot hand, the apparatus including a robot hand and a camera attached to the robot hand,
Transforming a pose of the robot hand according to input attitude information; And
(R) between the camera and the robot hand when the attitude information includes only the movement value, and calculates a motion vector (t) between the camera and the robot hand when the attitude information includes only the rotation value step
And a method of calibrating a robot hand.
The method according to claim 1,
And deriving a relational expression for calculating a posture conversion relationship between the camera and the robot hand using the posture conversion relationship from the camera coordinate system after posture conversion to the robot hand coordinate system before posture conversion.
3. The method of claim 2,
Wherein the posture conversion relation includes a rotation matrix and a motion vector.
The method of claim 3,
Wherein the relational expression is a mathematical expression described below, and a calibration method of a camera and a robot hand
[Mathematical Expression]

Where Rc is the rotation matrix of the camera, tc is the motion vector of the camera, R _E is the rotation matrix of the robot hand, t _E is the motion vector of the robot hand, R is the rotation matrix between the camera and the robot hand, Hand movement vector.
5. The method of claim 4,
If the attitude information includes only the movement value, the step of calculating the rotation matrix R between the camera and the robot hand
A correction block in which a coordinate system is preset is photographed through the camera and a first camera coordinate system in a first camera posture and a second camera coordinate system in a second camera posture are calculated based on a coordinate system of the photographed correction block step;
Calculating a posture change relation of a camera using the first camera coordinate system and a second camera coordinate system, and calculating a posture change relation of the robot hand using the posture information; And
And calculating a rotation matrix between the camera and the robot hand using the posture conversion relation of the camera and the posture conversion relation of the robot hand.
In the fifth aspect,
Wherein the rotation matrix of the camera and the rotation matrix of the robot hand are I matrices,
Wherein the rotation matrix between the camera and the robot hand is calculated by applying a pseudo inverse to the equation described below.
[Mathematical Expression]

Here, tc is the motion vector of the camera, t _E is the motion vector of the robot hand, and R is the rotation matrix between the camera and the robot hand.
5. The method of claim 4,
The step of calculating the motion vector (t) between the camera and the robot hand when the attitude information includes only the rotation value
A correction block in which a coordinate system is preset is photographed through the camera and a first camera coordinate system in a first camera posture and a second camera coordinate system in a second camera posture are calculated based on a coordinate system of the photographed correction block step;
Calculating a posture change relation of a camera using the first camera coordinate system and a second camera coordinate system, and calculating a posture change relation of the robot hand using the posture information; And
And calculating a motion vector between the camera and the robot hand using the posture conversion relation of the camera and the posture conversion relation of the robot hand.
In the seventh aspect,
The motion vector t _E of the robot hand is '0'
Wherein the motion vector t between the camera and the robot hand is calculated by applying a pseudo inverse to the following equation:
[Mathematical Expression]

Here, Rc is the rotation matrix of the camera, tc is the motion vector of the camera, and t is the motion vector between the camera and the robot hand.
The method according to claim 1,
And performing a calibration between the camera and the robot hand using the calculated posture conversion relationship between the camera and the robot hand.

1. An apparatus for performing a calibration of a robot hand and a camera attached to the robot hand,
A user interface unit for receiving attitude information of the robot hand; And
A posture controller for converting a pose of the robot hand according to the posture information; And
(T) between the camera and the robot hand when the robot hand only rotates and calculates a rotation matrix (R) between the camera and the robot hand when the robot hand does not rotate,
And a calibration device for the robot hand.
11. The method of claim 10,
The calibration control unit
Wherein when the robot hand does not rotate but moves only, a correction block in which a coordinate system is set in advance is photographed through the camera, and a first camera coordinate system in a first camera posture and a second camera coordinate system in a second camera posture Calculating a posture change relation of the camera using the first camera coordinate system and the second camera coordinate system, calculating a posture change relation of the robot hand using the posture information, calculating a second camera coordinate system in the camera posture, And calculating a rotation matrix between the camera and the robot hand using the posture conversion relation of the camera and the posture conversion relation of the robot hand,
Wherein the rotation matrix between the camera and the robot hand is calculated by applying a pseudo inverse to the equation described below.
[Mathematical Expression]

Here, tc is the motion vector of the camera, t _E is the motion vector of the robot hand, and R is the rotation matrix between the camera and the robot hand.
11. The method of claim 10,
The calibration control unit
Wherein when the robot hand only rotates, a correction block having a coordinate system set in advance is photographed through the camera, and a first camera coordinate system in a first camera position and a second camera position in a second camera position Calculates a posture change relation of the camera using the first camera coordinate system and the second camera coordinate system, calculates a posture change relation of the robot hand using the posture information, Calculating a motion vector (t) between the camera and the robot hand using the posture conversion relationship of the camera and the posture conversion relation of the robot hand,
Wherein the motion vector t between the camera and the robot hand is calculated by applying a pseudo inverse to the following equation:
[Mathematical Expression]

Here, Rc is the rotation matrix of the camera, tc is the motion vector of the camera, and t is the motion vector between the camera and the robot hand.
13. The method according to claim 11 or 12,
If the apparatus is provided with a stereo-based 3D camera equipped with a second camera in addition to the first camera attached on the robot hand,
The calibration control unit
Calculating a coordinate system of the first camera and a coordinate system of the second camera on the basis of the coordinate system of the correction block and calculating a posture conversion relationship between the first camera and the second camera using the calculated coordinate system of the first camera and the coordinate system of the second camera And the calibration of the 3D camera is performed using the calculated posture conversion relationship between the two cameras.
13. The method according to claim 11 or 12,
If the camera attached to the robot hand is a structured light based 3D camera equipped with a projector,
The calibration control unit
Calculates a coordinate system of the camera and a coordinate system of the projector on the basis of the coordinate system of the correction block, calculates a posture change relationship between the camera and the projector using the calculated coordinate system of the camera and the coordinate system of the projector, And the calibration of the 3D camera is performed using the posture conversion relation between the camera and the projector.

Images