JP6282098B2 - Calibration apparatus and method - Google Patents

Description

  The present invention relates to a calibration method for a measuring apparatus including a camera.

  The three-dimensional measurement technique using an image can be used for various purposes such as generating a three-dimensional model from an actual object and measuring the position and orientation of an object. As a method of three-dimensional measurement using images, a method based on the principle of triangulation based on images taken by two cameras (stereo cameras) whose relative positions and orientations are known is generally used. It is. In order to increase the reliability of three-dimensional measurement, one of the stereo cameras is widely replaced with an illumination device such as a projector.

  In order to perform three-dimensional measurement based on the principle of triangulation, the internal parameters of the camera (projector) and the relative position and orientation between the camera and camera (or camera and projector) are calibrated in advance. There is a need.

  Patent Document 1 uses a calibration object that can be regarded as a three-dimensional distribution of an index by moving a plane on which the index is arranged by a known movement amount in a direction perpendicular to the plane using a moving stage. A method for calibrating a camera and a projector is disclosed. In this case, the three-dimensional coordinates of the index on the calibration object are determined by the coordinates on the plane of the index and the moving amount of the moving stage.

JP-A-11-166818

Z. Zhang, “A flexible new technique for camera calibration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, 2000.

  In the method disclosed in Patent Document 1, if the direction in which the plane on which the index is arranged is moved by the moving stage does not coincide with the direction perpendicular to the plane due to an attachment error or the like, an error occurs in the coordinates of the index on the calibration object. Calibration accuracy was low.

  The present invention has been made in view of the above problems, and an object of the present invention is to perform calibration of a measuring apparatus including a camera with high accuracy.

  In order to solve the above-described problem, the calibration device of the present invention is configured so that either one of two imaging devices whose relative positions and orientations are fixed and a calibration object are moved a plurality of times on a straight line at predetermined movement intervals. An acquisition means for acquiring a plurality of images obtained by capturing the calibration object by the two imaging devices each time it is moved, an approximate value of the direction of the straight line, and an approximate value of a parameter of each of the two imaging devices, An approximate value acquisition means for acquiring the position information of the index on the calibration object, and an index on the calibration object is detected from each of the plurality of images. Detection means for obtaining a correspondence between the image coordinates of the index and the position information of the index held in the holding means, and based on the correspondence between the image coordinates of the index and the position information in the plurality of images, The number of images is an image taken every time one of the two imaging devices and the calibration object is moved on the straight line at the predetermined movement interval, and the direction of the straight line is undetermined. Parameter deriving means for deriving a parameter for each of the two imaging devices by correcting the approximate value of the direction of the straight line and the approximate value of the parameters for each of the two imaging devices under the condition that there are The relative position and orientation of the two imaging devices are derived based on the correspondence between the image coordinates of the index and the position information in the image of the image and the parameters relating to the two imaging devices derived by the parameter deriving means. Position and orientation deriving means.

  According to the present invention, calibration of a measuring apparatus including a camera can be performed with high accuracy.

It is a block diagram which shows an example of schematic structure of the calibration apparatus 1 which concerns on 1st Embodiment. It is a figure explaining arrangement | positioning of the apparatus at the time of calibration in 1st Embodiment. It is a figure showing the projection model to the image of the point in three-dimensional space. It is a figure explaining the positional relationship of the calibration object and a camera. It is a flowchart which shows the process sequence of the calibration in 1st Embodiment. It is a flowchart which shows the process sequence of parameter calculation. It is a flowchart which shows the process sequence of the internal parameter calculation of a camera. It is a flowchart explaining the process sequence of calculation of the relative position and attitude | position between cameras. It is a figure explaining arrangement | positioning of the apparatus at the time of calibration in 2nd Embodiment. It is a block diagram which shows an example of schematic structure of the calibration apparatus 3 which concerns on 3rd Embodiment. 4 is a block diagram showing a detailed configuration of a feature detection unit 350. FIG. It is a flowchart explaining the process sequence of matching with the coordinate on the projector image of the parameter | index and world coordinate in 3rd Embodiment. It is a figure showing the pattern image of a gray code pattern. It is a figure explaining arrangement | positioning of the apparatus at the time of calibration in 4th Embodiment. It is a figure explaining the arrangement | positioning of the calibration object in 4th Embodiment, a camera, and a projector. It is a flowchart which shows the procedure of the parameter calculation in 5th Embodiment.

  Hereinafter, embodiments (embodiments) for carrying out the present invention will be described with reference to the drawings.

(First embodiment)
In the first embodiment, a case will be described in which calibration of a three-dimensional measurement apparatus configured by two cameras (stereo cameras) is performed. In the present embodiment, the calibration object is fixed in the scene, and the three-dimensional measurement apparatus (stereo camera) is moved by the moving stage. The parameters to be calibrated are the internal parameters of the two cameras and the relative position and orientation between the cameras. In the present embodiment, it is assumed that the posture of each camera with respect to the calibration object at each position when moved by the moving stage is the same.

  FIG. 1 is a block diagram illustrating an example of a schematic configuration of a calibration apparatus 1 according to the present embodiment. As shown in FIG. 1, the calibration apparatus 1 includes an image input unit 140, a feature detection unit 150, a parameter calculation unit 160, a movement information storage unit 170, and a calibration object information storage unit 180. The calibration device 1 is connected to a control unit 100, a first imaging unit 110, a second imaging unit 120, and a calibration information storage unit 190.

  FIG. 2 shows an arrangement of devices when performing calibration of the three-dimensional measuring apparatus in the present embodiment. As shown in FIG. 2, circular indicators are arranged at equal intervals on a plane constituting the calibration object. Further, in the present embodiment, in order to stabilize the calibration calculation, an index is also arranged on a portion other than the plane of the calibration object. Hereinafter, an index on the plane of the calibration object is called a plane marker, and an index on a portion other than the plane is called a three-dimensional marker. However, it is not essential to arrange the three-dimensional marker on the calibration object, and a calibration object in which only the planar marker is arranged may be used.

  By arranging a plane marker and a three-dimensional marker (three-dimensional index) on the calibration object, the distribution of the index on the calibration object becomes three-dimensional. In order to define a three-dimensional coordinate system (hereinafter referred to as a world coordinate system) on the calibration object, a part of the planar marker is concentric.

  Hereinafter, concentric planar markers are referred to as concentric markers. In the present embodiment, the concentric markers are arranged in an L shape, and the center position of the concentric marker corresponding to the corner portion of the L shape is set as the origin of the world coordinate system. Also, the normal direction of the plane is the Z axis of the world coordinate system, the L direction is the X axis, and the short direction is the Y axis. It is assumed that the plane marker is on the plane Z = 0 in the world coordinate system. The three-dimensional coordinates (position information) of each index in the world coordinate system are stored in the calibration object information storage unit 180. As these coordinates, the design values of the calibration object may be used as they are, or coordinates measured separately using a measuring machine such as a contact-type three-dimensional coordinate measuring machine may be used.

  The first image capturing unit 110 and the second image capturing unit 120 are cameras that capture grayscale or color images. The first imaging unit 110 and the second imaging unit 120 capture an image based on an instruction from the control unit 100. The first imaging unit 110 and the second imaging unit 120 correspond to the camera L and the camera R in FIG. The cameras L and R are fixed so that their relative positions and postures do not change, and constitute a stereo camera.

  The moving unit 130 is a moving stage for moving the stereo camera on a straight line. The moving unit 130 corresponds to the moving stage in FIG. At the time of calibration, the stereo camera moves on a straight line at a predetermined interval by the moving stage. The predetermined interval may be a constant interval or a different interval.

  The control unit 100 controls the first imaging unit 110, the second imaging unit 120, and the moving unit 130. The control unit 100 is realized by a computer. When taking an image for calibration, the control unit 100 instructs the moving unit 130 to move by a predetermined distance, and the first imaging unit 110 and the second imaging unit each time the movement is completed. The unit 120 is instructed to take an image.

  The control unit 100 and the moving unit 130 are connected by, for example, an RS-232C cable, and send a movement instruction via the RS-232C interface. The control unit 100, the first imaging unit 110, and the second imaging unit 120 are connected by, for example, a camera link cable, and send a shooting instruction via a camera link interface. However, the connection method between the control unit 100 and the moving unit 130, the control unit 100 and the first imaging unit 110, and the second imaging unit 120 is not limited to this, and other connection methods such as USB, IEEE 1394, and GbE are used. However, the essence of the present invention is not impaired.

  Hereinafter, each of the image input unit 140, the feature detection unit 150, the parameter calculation unit 160, the movement information holding unit 170, and the calibration object information holding unit 180 of the calibration apparatus 1 of the present embodiment will be described. The calibration device 1 is realized by a personal computer including a memory and a CPU, for example. The calibration apparatus 1 incorporates the configuration shown in FIG. 1 as a program or a circuit. Note that the movement information holding unit 170 and the calibration object information holding unit 180 include, for example, a memory and a hard disk.

  The image input unit 140 inputs an image of the calibration object captured by the first imaging unit 110 and the second imaging unit 120 each time the stereo camera moves by a predetermined interval by the moving unit 130. For example, an image is input through a camera link interface. However, the image input method is not limited to this, and may be input via another interface such as USB, IEEE 1394, or GbE.

  Further, it is not always necessary that the image input unit 140 is connected to the first imaging unit 110 and the second imaging unit 120 when an image is captured, and an image that has been captured in advance and saved on a storage medium (not illustrated). May be read and input. The input image is stored in, for example, a memory or hard disk in the calibration apparatus 1.

  The feature detection unit 150 detects an index on the calibration object from the image (stored in the memory area) input via the image input unit 140, and the index stored in the calibration object information storage unit 180. Is associated. The feature detection unit 150 outputs, as a result of the association, the correspondence between the coordinates on the image of the index and the world coordinates of the index stored in the calibration object information storage unit 180 to the parameter calculation unit 160.

  The movement information storage unit 170 holds a movement interval when the movement unit 130 moves the stereo camera on a straight line. The movement interval to be held may be, for example, a control value given to the moving unit 130 by the control unit 100, or may be obtained from an encoder value when an encoder for detecting the movement amount is attached to the moving unit 130. .

  The parameter calculation unit 160 determines the internal parameters of each camera and the camera based on the correspondence between the coordinates on the image of the index output from the feature detection unit 150 and the world coordinates and the movement amount stored in the movement information storage unit 170. The relative position and orientation between them are calculated. Details will be described later.

  The calibration information storage unit 190 stores the internal parameters of the two cameras calculated by the parameter calculation unit 160 and the relative positions and orientations between the cameras.

Next, a camera projection model in this embodiment will be described. As shown in FIG. 3, the camera coordinate system (three-dimensional) is taken such that the Z axis coincides with the optical axis, and the X axis and Y axis are parallel to the horizontal and vertical directions of the image, respectively. The origin of the camera coordinate system is the lens center of the camera. A point having a three-dimensional coordinate (X _c , Y _c , Z _c ) in the camera coordinate system is projected to a normalized image coordinate (x _c , y _c ) expressed by Equation 1 by a pinhole camera model.

ここで正規化画像座標とは、カメラ座標系においてＺ＝１の位置に画像面を設定したときの画像面上での位置である。正規化座標系の原点は光軸（Ｚ軸）と画像面の交点であり、ｘ軸、ｙ軸はそれぞれ画像の水平方向、垂直方向と平行であるとする。正規化画像座標が（ｘ_ｃ，ｙ_ｃ）の点の実画像上での座標（以下、ピクセル座標）（ｕ_ｃ，ｖ_ｃ）は数２で表される。

Here, the normalized image coordinates are positions on the image plane when the image plane is set at a position of Z = 1 in the camera coordinate system. The origin of the normalized coordinate system is the intersection of the optical axis (Z-axis) and the image plane, and the x-axis and y-axis are parallel to the horizontal and vertical directions of the image, respectively. The coordinates (hereinafter, pixel coordinates) (u _c , v _c ) on the actual image of the point having the normalized image coordinates (x _c , y _c ) are expressed by the following equation (2).

ここで、ｆ_ｘ、ｆ_ｙは焦点距離、（ｃ_ｘ，ｃ_ｙ）は画像中心の座標（正規化画像座標系の原点の実画像上でのピクセル座標）である。数３に示すように、実際のカメラによって撮影される場合には正規化画像座標にレンズ歪みの影響が加わる。

Here, f _x and f _y are focal lengths, and (c _x , c _y ) are image center coordinates (pixel coordinates on the actual image at the origin of the normalized image coordinate system). As shown in Equation 3, when the image is taken by an actual camera, the normalized image coordinates are affected by lens distortion.

ただし、

である。ｋ_１，ｋ_２，ｋ_３は半径方向の歪み係数、ｐ_１，ｐ_２は接線方向の歪み係数である。焦点距離ｆ_ｘ、ｆ_ｙ、画像中心ｃ_ｘ、ｃ_ｙ、歪み係数ｋ_１，ｋ_２，ｋ_３，ｐ_１，ｐ_２をカメラの内部パラメータと呼ぶ。

However,

It is. k ₁ , k ₂ , and k ₃ are radial distortion coefficients, and p ₁ and p ₂ are tangential distortion coefficients. The focal lengths f _x and f _y , the image centers c _x and c _y , and the distortion coefficients k ₁ , k ₂ , k ₃ , p ₁ and p ₂ are referred to as camera internal parameters.

Next, coordinate conversion from the coordinate system (world coordinate system) of the calibration object to the camera coordinate system will be described. As shown in FIG. 4, the world coordinates of the intersection point a between the straight line (hereinafter referred to as the straight line A) on which the camera (the origin of the camera coordinate system) is moved by the moving stage and the plane Z _w = 0 of the world coordinate system are expressed as t _l = [t _{x ty} 0] ^t .

In addition, the direction vector of the straight line A in the world coordinate system is assumed to be d ₁ = [d _x d _y d _z ] ^t (d _x ² + d _y ² + d _z ² = 1). The initial position of the camera from the intersection a between position away line A distance Z _0, the moving stage camera is assumed to be moved by each [Delta] Z. When the moving stage is moved i times (i = 0, 1,..., M−1), the relationship between the camera coordinates _Xc and the world coordinates _Xw is as shown in Equation 4.

ここでＲ_ｗｃは世界座標系におけるカメラの姿勢を表す３×３回転行列である。前述したように、本実施形態では、移動ステージによる移動により世界座標系におけるカメラの姿勢は変化しないという前提であるため、Ｒ_ｗｃはカメラが移動しても変化しない。数４を変形することにより数５が得られる。

Here, R _wc is a 3 × 3 rotation matrix representing the posture of the camera in the world coordinate system. As described above, in the present embodiment, since it is premised that the posture of the camera in the world coordinate system does not change due to movement by the moving stage, R _wc does not change even when the camera moves. Equation 5 is obtained by transforming Equation 4.

ただし

である。

However,

It is.

  FIG. 5 is a flowchart showing a calibration processing procedure of the three-dimensional measurement apparatus according to this embodiment.

(Step S1010)
In step S1010, calibration initialization processing in the present embodiment is performed. First, information relating to a calibration object is read from a storage device (not shown) and held in the calibration object information holding unit 180. As described above, the information held by the calibration object information holding unit 180 is the three-dimensional coordinates (X _w , Y _w , Z _w ) of each index in the world coordinate system. Further, the movement information storage unit 170 holds the movement interval ΔZ when the stereo camera moves on a straight line by the movement unit 130. Here, it is assumed that the movement interval ΔZ is always constant.

(Step S1020)
In step S1020, an image of the calibration object photographed by the camera L and the camera R is input via the image input unit 140. The camera L and the camera R are moved on the straight line at predetermined intervals by the moving stage, and an image taken every time is stopped is input. That is, when the stereo camera rotates M-1 by the moving stage, 2M images including the initial position are input.

(Step S1030)
In step S1030, an index (circular marker) is detected from the image input in step S1020. The index detection process is performed on all the images input in step S1020. As shown in FIG. 2, since the calibration object is white and the index is a black circle, binarization of the image and labeling processing of the black area are performed to extract circular area candidates. A region corresponding to the index is detected by performing threshold processing on the candidate area.

  Similarly, a concentric marker that defines a coordinate system is detected as a region having a white region in a black region and having a predetermined area or more based on the results of binarization and labeling.

  After the index is detected from the image, the detected index is associated. First, two straight lines with concentric markers on the image are estimated, and an index near the intersection of the two straight lines is associated with the index at the origin. Based on the index associated with the origin and the direction of the two straight lines, the other indices are associated. The association result is held for each camera. Corresponding information is the coordinates of the indicator in the world coordinate system

及びカメラ画像上の座標

移動ステージの移動回数ｉの組として保持する。

And coordinates on the camera image

It is held as a set of the number of times i of movement stage movement.

(Step S1040)
In step S1040, using the corresponding information obtained up to step S1030, the internal parameters of camera L and camera R and the relative position and orientation between camera L and camera 2 (between the imaging devices) are calculated. FIG. 6 is a flowchart showing the procedure of parameter calculation (derivation) in step S1040. Hereinafter, a detailed procedure of parameter calculation will be described based on the flowchart of FIG.

(Step S1110)
In step S1110, first, internal parameters of the camera L are calculated. FIG. 7 is a flowchart illustrating a processing procedure for calculating the internal parameters of the camera. In this step, first, an approximate value of an unknown parameter is calculated (approximate value acquisition) by a method that does not require an initial value, and then an unknown parameter that minimizes the evaluation function shown in Equation 6 is nonlinearly optimized. calculate.

ただし、

は算出される未知パラメータに基づいて計算される指標の座標、

は画像上で検出された指標の座標である。また、Ｎは対応付けされた指標の総数である。

However,

Is the index coordinates calculated based on the calculated unknown parameters,

Is the coordinates of the index detected on the image. N is the total number of associated indexes.

(Step S1210)
In step S1210, the approximate value of the unknown parameter is calculated. In calculating the approximate value, the influence of lens distortion is ignored. Here, the unknown parameters are the camera internal parameters (focal length, image center) other than the lens distortion coefficient, the camera posture in the world coordinate system, the straight line parameter that the camera moves on the moving stage, and the initial position of the camera on the straight line It is.

The approximate value is calculated in the following order. First, the position (A _x , A _y ) where the straight line A is projected on the captured image is obtained by image processing to be described later. By using (A _x , A _y ), the image center of the camera's internal parameters can be excluded from the unknown parameters. Next, a position at which the straight line A passes through the plane Z _w = 0 is obtained based on the correspondence information between (A _x , A _y ) and the plane marker. Finally, the remaining parameters are calculated.

(1) Calculation of A _x , A _{y When} the direction vector of the straight line A in the camera coordinate system is expressed as d _c = [d _cx d _cy d _cz ] ^t , if the influence of lens distortion is ignored, The pixel coordinates (A _x , A _y ) of the projected straight line A are expressed as follows:

ただし、ｄ_ｃ＝ Ｒ_ｃｗ ｄ_ｌである。
一方、カメラを直線Ａ上で移動させた場合、校正用物体上の各指標の画像座標（ピクセル座標）の移動軌跡は理論的には一点（Ａ_ｘ，Ａ_ｙ）で交わる。そこで、カメラの位置を変えて撮影したときの校正用物体上の各指標の移動軌跡をもとに （Ａ_ｘ，Ａ_ｙ）を算出する。

However, d _c = R _cw d _l .
On the other hand, when the camera is moved on the straight line A, the movement locus of the image coordinates (pixel coordinates) of each index on the calibration object theoretically intersects at one point (A _x , A _y ). Therefore, (A _x , A _y ) is calculated based on the movement trajectory of each index on the calibration object when the camera position is changed.

₍₂₎ t x, calculation of _{t y} is then calculated plane coordinates of the intersection a of the straight line A and the plane _{Z w = 0 (t x,} t y). From the correspondence between the pixel coordinates (u, v) and the plane coordinates (X _w , Y _w ) of the plane marker obtained in step S1030, a transformation matrix between the pixel coordinates and the plane coordinates is calculated, and (A _x , A _y ) is determined as a plane. The coordinates (t _x , t _y ) are converted into

(3) Estimation of other parameters Since d _c = R _cw d _l , the transformation from the world coordinate system to the camera coordinate system in Equation 5 is re-expressed as follows:

数２、数７、数８より正規化画像座標からピクセル座標への変換は次式のように表すことができる。

回転行列Ｒ_ｃｗを次のように表す。

数１、数８、数９、数１０より

ただし、

である。数１１は、対応付けられたすべての指標について成り立つ。そこで、数１１を各指標について立式したものを連立方程式として未知パラメータ

The conversion from the normalized image coordinates to the pixel coordinates can be expressed by the following equation from Equations 2, 7, and 8.

The rotation matrix R _cw is expressed as follows.

From Equation 1, Equation 8, Equation 9, and Equation 10

However,

It is. Equation 11 holds for all the associated indices. Therefore, unknown parameters are obtained by formulating Equation 11 for each index as simultaneous equations.

を算出する。Ｐの各成分は例えば最小二乗法により算出する。Ｐの各成分を算出した後、数１２の１１個の式及び回転行列の正規直交性及びｄ_ｃが単位ベクトルであることを利用して、回転行列Ｒ_ｃｗの各要素、直線上のカメラの初期位置Ｚ_０、方向ベクトルｄ_ｃの各成分、焦点距離ｆ_ｘ、ｆ_ｙを算出する。なお、複数の解の組合せが存在する場合には、焦点距離が正の値になるといった妥当な解の組合せを選択することで一意な解を得る。さらに、算出されたｆ_ｘ、ｆ_ｙ、Ａ_ｘ、Ａ_ｙ、ｄ_ｃを用いて数７からｃ_ｘ、ｃ_ｙを算出する。最後に

Is calculated. Each component of P is calculated by, for example, the least square method. After calculating the components of P, by utilizing the fact that the number 12 of the 11 pieces of the formula and the rotation matrix orthonormality and d _c of a unit vector, each element of the rotation matrix R _cw, the straight line of the camera initial position _{Z 0,} calculates each component of the direction vector _{d c,} the focal length _f x, the _{f y.} When there are a plurality of solution combinations, a unique solution is obtained by selecting an appropriate solution combination in which the focal length is a positive value. Furthermore, the calculated _{f x, f y, A x} , A y, having 7 with _{d c c} x, calculates the _{c y.} Finally

からｄ_ｌを算出する。

D ₁ is calculated from

As described above, the focal lengths f _x and f _y that are unknown parameters, the image centers c _x and c _y , the direction vector d _{l of the} straight line along which the camera moves, the 3 × 3 rotation matrix R _cw that represents the posture of the camera, calculating the approximate values of the initial position Z ₀ of the straight line of the camera. The approximate values of the lens distortion coefficients [k ₁ k ₂ k ₃ p ₁ p ₂ ] ^t are all 0.

  In step S1210, the approximate value of the unknown parameter is calculated by calculation based on the correspondence information of the detected index. However, if the information that can be used as the approximate value such as the design parameter is known in advance, the approximate value is not necessarily obtained. May not be obtained by calculation.

(Step S1220)
In step S1220, the unknown parameter is optimized so that the evaluation function shown in Equation 6 is minimized using the approximate value of the unknown parameter obtained in step S1210 as an initial value. The unknown parameters to be optimized are the internal parameters of the camera, the direction vector d _{1 of} the straight line along which the camera moves, the initial position Z ₀ of the camera on the straight line, and the attitude of the camera with respect to the world coordinate system. The optimization calculation is performed by, for example, the Gauss-Newton method. However, the optimization calculation method is not limited to this, and any other method may be used as long as it minimizes the evaluation function shown in Equation 6. For example, a Newton method, steepest descent method, or conjugate gradient method may be used.

(Step S1120)
In step S1120, the internal parameters of the camera R are calibrated. Since the process of step S1120 is the same as the process of step S1110, description thereof is omitted.

(Step S1130)
After the internal parameters of the cameras L and R are calibrated in steps S1110 and S1120, the relative positions and orientations of the cameras L and R are calibrated in step S1030. FIG. 8 is a flowchart illustrating a processing procedure for calculating the relative positions and orientations of the camera L and the camera R. In this step, first, an approximate value of an unknown parameter is calculated by a method that does not require an initial value, and then an unknown parameter is calculated by nonlinear optimization so that the evaluation function represented by Equation 13 is minimized.

ただし、

は算出される未知パラメータに基づいて計算されるカメラＬの画像上での指標の座標、

However,

Is the coordinate of the index on the image of the camera L calculated based on the calculated unknown parameter,

はカメラＬの画像上で検出された指標の座標、

Is the coordinates of the index detected on the image of the camera L,

は算出される未知パラメータに基づいて計算されるカメラＲの画像上での指標の座標、

Is the coordinates of the index on the image of the camera R calculated based on the calculated unknown parameter,

はカメラＲの画像上で検出された指標の座標である。また、Ｎ_１、Ｎ_２はそれぞれカメラＬ、カメラＲにより撮影された画像における対応付けされた指標の総数である。

Is the coordinates of the index detected on the image of the camera R. N ₁ and N ₂ are the total number of indices associated with the images taken by the cameras L and R, respectively.

(Step S1310)
In step S1310, an approximate value of the relative position and orientation (position and orientation) of the camera R with respect to the camera L is calculated. The approximate values of the relative position and orientation of the camera R with respect to the camera L are calculated based on the position and orientation of each camera with respect to the world coordinate system calculated at the time of calibration of internal parameters in steps S1110 and S1120. Further, as the direction vector of the straight line along which the camera moves and the approximate value of the initial position of the camera on the straight line, values calculated during the calibration of the camera internal parameters of the camera L are used.

  However, the method of calculating the approximate value is not limited to this, and a combination of values calculated at the time of calibration of the internal parameters of each camera may be used as the approximate value. .

(Step S1320)
In step S1320, the unknown parameter is optimized so that the evaluation function shown in Equation 13 is minimized with the approximate value of the unknown parameter obtained in step S1310 as an initial value.

  The unknown parameters in this step are the relative position and orientation of the camera R with respect to the camera L, the direction vector of the straight line to which the camera moves, the posture of the camera L in the world coordinate system, and the initial position of the camera L on the straight line. Since optimization in this step is the same as that described in step S1220, description thereof is omitted.

  By the method described above, the calibration of the three-dimensional measuring apparatus constituted by the stereo camera is performed.

  As described above, in the first embodiment, the method for calibrating the three-dimensional measurement apparatus is described with the movement interval when the three-dimensional measurement apparatus (stereo camera) is moved by the moving stage being known.

  By using the constraint condition that the 3D measuring device moves on a straight line and the movement interval is known for calibration calculation at the time of calibration, calibration can be performed with high accuracy even if parameters related to the moving direction are unknown. It becomes possible to do. Further, it is not necessary to make the specific direction of the calibration object coincide with the direction of movement by the moving stage in advance, and the calibration of the three-dimensional measuring apparatus is simplified.

  An example of the three-dimensional measuring apparatus calibrated by the present invention is a stereo camera composed of two cameras.

  However, the configuration of the three-dimensional measurement apparatus is not limited to this, and three or more cameras may be used. In this case, after estimating the internal parameters of each camera for each camera, the relative positions and postures between the cameras are minimized so as to minimize the sum of the reprojection errors of the indices detected on the images of all the cameras. Can be estimated.

(Second Embodiment)
In the first embodiment, the calibration method when the present invention is applied to the case where the calibration object is fixed in the scene and the three-dimensional measuring device (stereo camera) is moved by the moving stage has been described. In the second embodiment, a calibration method in the case where the three-dimensional measurement apparatus is fixed and the calibration object is moved by the moving stage will be described.

  Since the schematic configuration of the three-dimensional measuring apparatus 1 in the second embodiment is the same as that in FIG. However, as shown in FIG. 9, in the second embodiment, the moving unit 130 moves the calibration object instead of the stereo camera on a straight line at a predetermined interval. Each camera constituting the stereo camera captures an image of the calibration object each time the calibration object is moved by the moving unit 130 by a predetermined interval. As described in the first embodiment, the interval of movement by the movement stage may be constant as long as it is obtained as known information, or may be different for each movement.

Next, conversion of the calibration object from the coordinate system to the camera coordinate system in the second embodiment will be described. In the second embodiment, the coordinate system of the calibration object at the initial position on the moving stage is assumed to be the world coordinate system. It is assumed that the calibration object moves by ΔZ each time by the moving stage. In addition, a vector in the moving direction by the moving stage in the world coordinate system is set to d _l = [d _x d _y d _z ] ^t (d _x ² + d _y ² + d _z ² = 1).

The moving stage i times the relationship between the coordinate X _b and world coordinates X _w on the calibration object when (i = 0,1, ..., M -1) is moved is expressed as Expression 14.

When a 3 × 3 rotation matrix that represents the attitude of the world coordinate system with respect to the camera coordinate system is R _cw and a three-dimensional vector that represents the position is t _cw , the relationship between the camera coordinates and the world coordinates is expressed by Equation 15.

On the other hand, the position of the camera coordinate system with respect to the world coordinate system is expressed as _twc as follows.

但し

であり、ｄ_ｌとｔ_ｌは平行でないものとする。ｔ_ｗｃとｔ_ｃｗの間には次式のように関係がある。

However,

And d _l and t _l are not parallel. There is a relationship between t _wc and t _cw as shown in the following equation.

数１４、数１５、数１６、数１７から次式が得られる。

The following equations are obtained from Equations 14, 15, 16, and 17.

Expression 18 has substantially the same form as Expression 5 in the first embodiment (the sign of (Z ₀ + iΔZ) d ₁ is reversed because the camera moves in the first embodiment. This is because the calibration object moves in the opposite direction in the second embodiment).

This indicates that the case where the calibration object is fixed and the camera is moved by the moving stage is equivalent to the case where the camera is fixed and the calibration object is moved by the moving stage. Therefore, calibration can be performed by the method disclosed in the first embodiment even when using an image obtained by fixing the camera and moving the calibration object by the moving stage. Subsequent processing is the same as that of the first embodiment, and thus description thereof is omitted.
As described above, in the second embodiment, the method for calibrating the three-dimensional measurement apparatus is described with the movement interval when the calibration object is moved on the straight line by the moving stage as known.

  Even when the calibration object is moved by the moving stage, the parameter regarding the moving direction is unknown by using the constraint that the calibration object moves on a straight line and the movement interval is known. Calibration with high accuracy becomes possible.

  Further, it is not necessary to make the specific direction of the calibration object coincide with the direction of movement by the moving stage in advance, and the calibration of the three-dimensional measuring apparatus is simplified.

(Third embodiment)
In the first embodiment and the second embodiment, the calibration method for a stereo camera including two cameras as the three-dimensional measurement apparatus has been described. In the present embodiment, a case will be described in which the present invention is applied to calibration of a three-dimensional measurement apparatus in which one of the stereo cameras is replaced with a projector.

  FIG. 10 is a block diagram illustrating an example of a schematic configuration of the calibration apparatus 3 according to the present embodiment.

  The calibration device 3 includes an image input unit 340, a feature detection unit 350, a parameter calculation unit 360, a movement information storage unit 370, and a calibration object information storage unit 380. The calibration device 3 is connected to a control unit 300, an imaging unit 310, an illumination unit 320, and a calibration information storage unit 390. Note that the calibration object used in the present embodiment is the same as the calibration object used in the first embodiment, and a description thereof will be omitted.

  The imaging unit 310 is a camera that captures a grayscale or color image. The illumination unit 320 is a projection type projector. The projector may always project a fixed pattern or image into the space, or may switch and project an arbitrary image.

  Here, a projector capable of switching and projecting an arbitrary image is used. Similar to the two cameras shown in FIG. 2, the camera and the projector are fixed so that their relative positions and postures do not change.

  The projector emits a pattern having a uniform luminance value and a pattern image for calibration. An image taken by irradiating a uniform pattern is used to detect an index on the calibration object. When a projector that always projects a fixed pattern is used, an image taken under ambient light is used for detection of an index on a calibration object without performing projection by the projector.

  In the present embodiment, a pattern of the spatial encoding method is projected as a pattern image for calibration. In the spatial coding method, a plurality of fringe pattern images are switched and projected and imaged in order to determine the spatial code of each pixel. In the present embodiment, it is assumed that the projector irradiates horizontal and vertical stripe patterns. The projector projects a pattern based on an instruction from the control unit 300. The camera captures an image when the projector projects a pattern based on an instruction from the control unit 300.

  The moving unit 330 is a moving stage for moving the camera and the projector on a straight line. In the present embodiment, as in the first embodiment, the camera and the projector are moved on a straight line at a predetermined interval by the moving stage, and an image of the calibration object fixed in the scene is taken. The calibration object is irradiated with a uniform pattern and a pattern image for calibration of the projector, and images are taken separately.

  The control unit 300 has the same function as that of the control unit 100 of the first embodiment except that it controls the illumination unit 320 instead of the second imaging unit 120, and thus description thereof is omitted.

  The image input unit 340 inputs an image of a calibration object imaged by the imaging unit 310 to the computer every time the moving unit 330 moves by a predetermined distance by the control unit 300. Similar to the image input unit 140 of the first embodiment, the image input unit 340 and the imaging unit 310 are not necessarily connected, and an image that has been captured in advance and saved on a storage medium (not shown) is read out. May be entered.

  Since the function of the calibration object information storage unit 380 is the same as the function of the calibration object information storage unit 180 of the first embodiment, a description thereof will be omitted.

  The feature detection unit 350 detects an index on the calibration object from the image input via the image input unit 340 and associates the index with the index stored in the calibration object information storage unit 380.

  FIG. 11 is a block diagram illustrating a detailed configuration of the feature detection unit 350. The feature detection unit 350 includes an index detection unit 351 and a pattern detection unit 352. Since the function of the index detection unit 351 is the same as the function of the feature detection unit 150 of the first embodiment, description thereof is omitted.

  The pattern detection unit 354 calculates the horizontal and vertical spatial codes of each pixel and the boundary coordinates of the horizontal and vertical spatial codes of each pixel based on the image irradiated with the stripe pattern input via the image input unit 340. To do. Using the coordinate of the index detected by the index detection unit 351 and the boundary coordinates of the horizontal and vertical spatial codes calculated by the pattern detection unit 352, the coordinate of the index on the projector image is calculated and associated with the world coordinate.

  The movement information storage unit 370 holds a movement interval when the camera and the projector constituting the three-dimensional measurement apparatus are moved on a straight line by the movement unit 330.

  Based on the correspondence between the coordinates on the image of the index obtained from the feature detection unit 350 and the world coordinates and the movement amount stored in the movement information storage unit 370, the parameter calculation unit 360 The relative position and orientation between the projector and the projector are calculated.

  The calibration information storage unit 390 stores the internal parameters of the camera and the projector calculated by the parameter calculation unit 360 and the relative position and orientation between the camera and the projector.

  The calibration processing procedure of the three-dimensional measuring apparatus in the third embodiment is basically the same as the processing procedure in the first embodiment shown in FIG. There are two differences between the first embodiment and the third embodiment.

  One is that, in step S1020, an image irradiated with a pattern having a uniform luminance value and a pattern image for calibration is input. Another difference is that in step S1030, coordinates on the projector image, not coordinates on the image on the camera R, are calculated and associated with world coordinates.

Here, when the projector is viewed as an optical system using a lens, the only difference from the camera is whether the image is input or output through the lens, so the projection model of the projector is described by the same model as the camera. can do. Therefore, there is no essential difference between the calibration of the camera and the projector, and it can be performed by the same method.
Hereinafter, a method for associating the coordinates of the index on the projector image with the world coordinates will be described. Since the processing in step S1040 after the association information is obtained is the same as that in the first embodiment, description thereof is omitted.

  FIG. 12 is a flowchart illustrating a processing procedure for associating coordinates on the projector image with indices and world coordinates according to the third embodiment.

(Step S3110)
In step S 3110, the index is detected from the image of the calibration object captured by irradiating the uniform pattern input in step S 1020, and the index stored in the calibration object information storage unit 380 is associated with the world coordinates. Do. Since the processing in this step is the same as that in step S1030 of the first embodiment, description thereof is omitted.

(Step S3120)
In step S3120, the horizontal and vertical spatial code boundaries are calculated from the image irradiated with the stripe pattern input in step S1020. A typical fringe pattern used in the spatial coding method is a gray code pattern (FIG. 13).

  First, an image captured by irradiating each stripe pattern constituting a gray code pattern is binarized, and a code (space code) is calculated by arranging binarization results (0 or 1) for each pixel.

  The space code represents the ID of a line on the projector image irradiated to the corresponding pixel, and is calculated for each of the horizontal direction and the vertical direction. Based on this result, the boundary of the spatial code on the captured image is calculated for each of the horizontal direction and the vertical direction.

  Next, the coordinates of the intersection of the vertical and horizontal stripe patterns are calculated using the boundary between the horizontal and vertical spatial codes. The coordinates on the camera image of each intersection and the coordinates on the projector image can be associated with each other by the space code. This correspondence is used in step S3130.

(Step S3130)
In step S3130, next, the world coordinates of the index are associated with the coordinates on the projector image.

  First, for each index detected on the camera image that is associated with the world coordinates in step S3110, the intersection of spatial codes in the vicinity area (here, within radius r pixels) on the camera image ( Search in step S3120.

  Next, assuming that the area around the index is a plane locally, the coordinates on the camera image are converted to the coordinates on the projector image based on the coordinates on the camera image and the coordinates on the projector image of the searched intersection. Calculate the homography transformation. Homographic transformation is described in, for example, Hartley & A. It is calculated by the Direct Linear Transform (DLT) method disclosed in Zisserman “Multiple View Geometry” (Cambridge University Press). Finally, using the calculated homography conversion, the coordinates of the index on the camera image are converted into the coordinates on the projector image. In this way, the world coordinates of the index are associated with the coordinates on the projector image.

  Using the result of associating the coordinates of the index on the projector image and the world coordinates obtained as described above, the internal parameters of the projector and the relative parameters between the camera and the projector are processed by the same process as in the first embodiment. Calculate the position and orientation.

  As described above, in the third embodiment, a method for calibrating a three-dimensional measurement apparatus with a known movement interval when the three-dimensional measurement apparatus constituted by a camera and a projector is moved by a moving stage is known. explained.

  Even if the 3D measuring device is composed of a camera and a projector, the direction of movement can be obtained by using the constraint that the 3D measuring device moves on a straight line and the movement interval is known for the calibration calculation. It is possible to calibrate with high accuracy even if the parameter is unknown.

  Further, as a secondary effect, it is not necessary to make the specific direction of the calibration object coincide with the direction of movement by the moving stage in advance, and the calibration of the three-dimensional measuring apparatus is simplified.

  In the present embodiment, when calibrating a three-dimensional measuring apparatus including a camera and a projector, the projector projects a fringe pattern image used for the spatial coding method. However, the pattern image projected by the projector is not limited to this, and any other image may be used as long as the pattern image can correspond to the camera image.

  For example, for projecting and photographing a plurality of pattern images, a pattern image for the phase shift method may be projected.

  Moreover, the pattern image which can take the correspondence of a pattern image and a camera image from the image which image | photographed one pattern image with the camera may be sufficient.

  Further, in the present embodiment, in order to calculate the coordinates of the index on the projector image, the index of the index on the camera image is calculated by homography conversion estimated from the coordinates of the intersection of the vertical and horizontal patterns of the spatial code projected by the projector. The coordinates were converted. However, the method of calculating the coordinates of the index on the projector image is not limited to this, and interpolation is performed based on the distance between the coordinates of the index on the camera image and the intersection of the vertical and horizontal patterns of the nearby spatial code. It may be calculated.

(Fourth embodiment).
In the third embodiment, a method for calibrating the internal parameters of the projector and the relative positions and orientations of the camera and the projector by using the correspondence between the world coordinates of the index on the calibration object and the coordinates on the projector image. explained. On the other hand, in the fourth embodiment, without using the index on the calibration object, the above parameters are set based on the correspondence between the coordinates on the projector image and the coordinates on the camera image of the pattern projected by the projector. A method for performing calibration will be described.

  In this embodiment, as in the second embodiment, a calibration method in the case where the calibration object is moved by the moving stage will be described.

  In this embodiment, it is assumed that the internal parameters of the camera are calibrated in advance by the method described in the first embodiment or the second embodiment. Further, it is assumed that approximate values of parameters to be calibrated such as internal parameters of the projector and relative positions and orientations between the camera and the projector are obtained from design values and the like.

  FIG. 14 shows an arrangement of devices when performing calibration of the three-dimensional measurement apparatus in the fourth embodiment. In the fourth embodiment, an object composed of a plurality of planes having different heights having a common normal vector is used as a calibration object. Note that the height difference Δr between planes is known. Here, it is assumed that the number of planes is two. However, the number of planes is not limited to two, and may be constituted by three or more planes.

  The procedure of the calibration process in the fourth embodiment is basically the same as the procedure of the first embodiment shown in FIG. However, the image input in the image input in step S1020 is an image of the calibration object on which the fringe pattern is projected.

  In step S1030, the index is not detected, and only the calculation and association of the coordinates of the intersections of the vertical and horizontal stripe patterns are performed. The correspondence information is held as a set of coordinates on the camera and projector image at the intersection of the vertical and horizontal stripe patterns, and the plane ID when the light beam corresponding to each intersection is reflected on the calibration object And

  Hereinafter, the parameter calculation process in step S1040 will be described. In the fourth embodiment, since the internal parameters of the camera are known, the internal parameters of the projector and the relative position and orientation between the camera and the projector are calculated. In the previous embodiments, the relative position and orientation were calculated after calculating the internal parameters, whereas in the fourth embodiment, the internal parameters of the projector and the relative positions of the camera and the projector were calculated. Calculate the posture at the same time.

  Since the processing procedure of the calibration calculation in the fourth embodiment is basically the same as that shown in the flowchart of FIG. 7, the processing contents will be described with step numbers S1210 to S1220 of FIG. 7 as S4210 to S4220. To do. Here, an unknown parameter that minimizes the evaluation function shown in Equation 19 is calculated by nonlinear optimization.

ただし、

は算出される未知パラメータに基づいて計算されるプロジェクタ画像上のパターンの交点の座標、

However,

Is the coordinates of the intersection of the pattern on the projector image calculated based on the calculated unknown parameter,

はプロジェクタが投影するパターン画像上でのパターンの交点の座標である。また、Ｎは対応付けされたパターンの交点の総数である。

Is the coordinates of the intersection of the patterns on the pattern image projected by the projector. N is the total number of intersections of the associated patterns.

(Step S4210)
In step S4210, an approximate value of an unknown parameter is input. Here, the unknown parameters are the internal parameters of the projector (focal length, image center, lens distortion coefficient), the relative position and orientation of the camera and projector, the position and orientation of the calibration object, and the movement vector of the calibration object. .

  Since the calibration object is composed of a plurality of planes having the same normal vector and different heights, the position and orientation of the calibration object are parameters of a certain plane constituting the calibration object. As described above, approximate values of these unknown parameters are input based on design values and the like. Alternatively, the movement vector disclosed in Patent Document 1 is calculated by a known method.

(Step S4220)
In step S4220, the unknown parameter is optimized so that the evaluation function shown in Equation 19 is minimized with the approximate value of the unknown parameter obtained in step S4210 as an initial value.

First, the relationship between the normalized image coordinates (x _c , y _c ) on the camera side and the normalized image coordinates (x _p , y _p ) on the projector side will be described.

First, using the known camera internal parameters, pixel coordinates (u _c , v _c ) on the camera image at the intersection of the pattern based on Equations 2 and 3 are normalized image coordinates (x _c ) without distortion. , Y _c ).

The pixel coordinates (u _c , v _c ) are converted into distorted normalized image coordinates (x ′ _c , y ′ _c ) based on Equation 2, and further distorted normalized image coordinates (x ' _c , y' _c ) is transformed into normalized image coordinates (x _c , y _c ) without distortion. However, since it is difficult to analytically obtain the normalized image coordinates (x _c , y _c ) without distortion from the normalized image coordinates (x ′ _c , y ′ _c ) with distortion, the initial values are set to x _c , y _c . Values (for example, x ′ _c , y ′ _c ) are given, and x _c and y _c are calculated by iterative calculation such as Newton's method.

Next, the normalized image coordinates (x _c , y _c ) on the camera side are associated with the normalized image coordinates (x _p , y _p ) on the projector side. FIG. 15 is a diagram for explaining the arrangement of the calibration object, the camera, and the projector in the fourth embodiment. An equation in the camera coordinate system at the initial position (when there is no movement by the moving stage) of one plane (plane ID: 1, hereinafter referred to as plane 1) constituting the calibration object is

とする。ただしｎ_ｃは平面１の法線ベクトル（単位ベクトル）、ｒ_ｃはカメラ座標系の原点から平面１までの距離である。平面１が移動ステージによって方向ｄ_ｃ（ｄ_ｃは単位ベクトル）にｉΔＺ（ｉ＝０，１，…，Ｍ−１）だけ移動する場合、平面の方程式は次式のようになる。

And However n _c is the normal vector of the plane 1 (unit vector), the r _c is the distance from the origin of the camera coordinate system to the plane 1. IΔZ the plane 1 moves the stage in a direction _d c _{(d c} is the unit vector) (i = 0,1, ..., M-1) when moving only plane equation is expressed by the following equation.

平面１と校正用物体を構成する他方の平面（平面ＩＤ：２、以下平面２とする）は平面１と法線ベクトルが共通で高さの差異がΔｒである。平面２が移動ステージによって方向ｄ_ｃにｉΔＺ（ｉ＝０，１，…，Ｍ−１）だけ移動する場合、平面の方程式は次式のようになる。

The plane 1 and the other plane constituting the calibration object (plane ID: 2, hereinafter referred to as plane 2) have the same normal vector as that of plane 1 and have a height difference Δr. IΔZ direction _{d c} plane 2 by moving the stage (i = 0,1, ..., M -1) when moving only plane equation is expressed by the following equation.

ここで、カメラ座標系からプロジェクタ座標系への変換を数２２のように表す。

Here, the transformation from the camera coordinate system to the projector coordinate system is expressed as in Expression 22.

平面１上のある点のカメラ側の正規化画像座標を（ｘ_ｃ，ｙ_ｃ）、プロジェクタ側の正規化画像座標を（ｘ_ｐ，ｙ_ｐ）と両者には次のような関係がある。

The normalized image coordinates on the camera side of a certain point on the plane 1 are (x _c , y _c ), the normalized image coordinates on the projector side are (x _p , y _p ), and both have the following relationship.

ただし、

となる。また、平面２の場合には

ただし

となる。

However,

It becomes. In the case of plane 2,

However,

It becomes.

As described above, the normalized image coordinates (x _c , y _c ) on the camera side and the normalized image coordinates (x _p , y _p ) on the projector side are related by Equations 23 and 24. Further, the normalized image coordinates (x _p , y _p ) on the projector side are converted into pixel coordinates on the projector side according to Equations 2 and 3.

  In step S4220, the unknown parameters are optimized so that the evaluation function shown in Equation 19 is minimized based on Equations 2, 3, 23, and 24.

  The unknown parameters optimized in this step are the internal parameters of the projector, the relative position and orientation of the camera and the projector, the position and orientation of the calibration object, and the movement vector of the calibration object. Since optimization in this step is the same as that described in step S1220 of the first embodiment, description thereof is omitted.

  As described above, in the fourth embodiment, based on the correspondence between the coordinates on the projector image of the pattern projected by the projector and the coordinates on the camera image, the movement interval of the calibration object by the moving stage is known. A method for calibrating a three-dimensional measuring apparatus has been described.

  Even when using the correspondence between the coordinates on the projector image of the pattern projected by the projector and the coordinates on the camera image, the constraint condition that the 3D measuring device moves on a straight line and the movement interval is known is calibrated. By using this for the calculation of the calibration, it is possible to perform calibration with high accuracy even if the parameter relating to the moving direction is unknown.

  In the fourth embodiment, the calibration method has been described using the correspondence between the coordinates on the projector image and the coordinates on the camera image at the intersection of the vertical and horizontal stripe patterns. However, the correspondence of coordinates is not limited to two-dimensional coordinates such as the coordinates of intersections.

  For example, it may be a correspondence between a straight line parameter on the projector image and a two-dimensional coordinate on the camera image. In the fourth embodiment, the calibration is performed based on the constraint that when the conversion is performed using the parameter for estimating the two-dimensional coordinates of the intersection detected on the camera image, the two-dimensional coordinates on the corresponding projector image are matched. went.

  On the other hand, when the correspondence with the straight line parameter is used, the two-dimensional coordinate on the projector image obtained by converting the two-dimensional coordinate on the camera image by the estimation parameter is on the straight line on the projector image obtained as the correspondence. Perform calibration based on constraint conditions.

(Fifth embodiment)
In the embodiment described above, the parameters of the straight line along which the camera (projector) or the calibration object moves by the moving stage are obtained together with other parameters by numerical calculation.

  However, the method for obtaining the straight line parameters is not limited to this, and a plurality of straight line parameter candidates are set, each straight line parameter is known and calibration is performed multiple times, and the parameter with the best evaluation value is selected. May be.

  In the present embodiment, as in the second embodiment, calibration of a stereo camera when the calibration object is moved by a moving stage will be described. The calibration processing procedure of the three-dimensional measuring apparatus according to the present embodiment is the same as the flowchart of the first embodiment shown in FIG. 5 except for the parameter calculation processing in step S1040. Below, the parameter calculation process in this embodiment is demonstrated.

  FIG. 16 is a flowchart illustrating a parameter calculation procedure according to the fifth embodiment. The detailed procedure for parameter calculation will be described below based on the flowchart of FIG.

(Step S5110)
In step S5110, a plurality of straight line direction candidates in which the calibration object moves on the moving stage are set. Here, assuming that the moving direction of the calibration object substantially coincides with the normal direction of the plane constituting the calibration object, candidates are set by sampling N directions in the vicinity of the normal direction of the plane.

  Note that the movement direction of the calibration object does not necessarily coincide with the normal direction of the plane constituting the calibration object, and any movement direction can be used as long as information on the approximate movement direction serving as a sampling reference can be obtained. There may be.

  Sampling may be performed by connecting points set at equal intervals on the unit sphere and the origin of the calibration object, or set at regular intervals on a plane installed at a fixed distance from the calibration object. It is also possible to connect the origin and the origin. In addition, any method may be used as long as it is a method for setting a plurality of directions.

(Step S5120)
In step S5120, 1 is substituted into variable i.

(Step S5130)
In step S5130, calibration calculation is performed using the i-th direction set in step S5110 as known information. Based on the known movement direction of the calibration object, the three-dimensional coordinates of each marker on the calibration object in the same coordinate system (reference coordinate system) are calculated.
The reference coordinate system is assumed to be the same as the coordinate system of the calibration object at the initial position, for example. When the three-dimensional coordinates of each marker in the reference coordinate system are calculated, unknown parameters are calculated in the same manner as the flowchart shown in FIG. The unknown parameter is calculated by a method disclosed in Patent Document 1, for example.

(Step S5140)
In step S5140, the parameter evaluation value calculated in step S5130 is calculated. For example, the value of Equation 13 is used as the evaluation value. Further, the combination of the parameter calculated in step S5130 and its evaluation value is stored.

(Step S5150)
In step S5150, it is determined whether the calculation of parameters and evaluation values has been completed for all straight line direction candidates. If completed, the process advances to step S5170. If not completed, 1 is added to i in step S5160, and the process returns to step S5130.

(Step S5170)
In step S5170, the best evaluation value calculated in step S5140 is selected, and the parameter when the selected evaluation value is calculated is output as the final calibration result. When using the value of Equation 13 as the evaluation value, the one with the smallest evaluation value is selected as the best evaluation value.

  As described above, in the fifth embodiment, there is a method for setting a plurality of straight line parameter candidates, performing calibration a plurality of times with each straight line parameter being known, and selecting the parameter having the best evaluation value. explained. Note that the method described in the fifth embodiment is applied not only to calibration of a stereo camera including a plurality of cameras but also to calibration of a three-dimensional measurement apparatus including an arbitrary number of cameras and projectors. May be. Moreover, you may apply to the calibration of a monocular camera by using Formula 6 as an evaluation value. Further, not only when the calibration object is moved by the moving stage, but also the three-dimensional measuring device may be moved. Furthermore, the processing shown in the flowchart of FIG. 16 may be performed stepwise. In other words, parameter calculation is performed by roughly sampling the direction of the straight line and calculating the parameter again by repeating the process of calculating the parameter again with a fine sampling interval near the direction of the straight line of the best evaluation value. May be.

(Modification 1)
In the embodiment described above, the posture of each camera with respect to the calibration object is assumed to be constant, and is not changed by the movement of the moving stage.

However, the posture of each camera is not necessarily constant and may be estimated as an unknown parameter. Specifically, as a parameter that changes each time the stage moves without _sharing R _cw in _Equation 5, a correction value for R _cw may be calculated and corrected for each movement position.

  In addition, when the change in posture is small, the posture of each camera with respect to the calibration object is fixed when calculating the approximate value of the internal parameter, and the posture of each camera is set as an unknown parameter when performing the optimization calculation. May be estimated.

  According to this modification, even when the posture of each camera changes minutely due to a mechanical problem of the moving stage, calibration can be performed with high accuracy, so that restrictions on the moving stage are reduced.

Similarly, in the fourth embodiment, the direction of the normal vector of the plane constituting the calibration object with respect to the camera is assumed to be constant. However, the orientation of the normal vector of the plane with respect to the camera need not be constant for each movement, and may be estimated as an unknown parameter individually. In this case, nc in _Equations 20 and 21 are unknown parameters that differ for each movement.

(Modification 2)
In the embodiment described above, the coordinates of each index on the calibration object in the coordinate system defined on the calibration object are known. However, when the calibration object is composed of a plurality of objects and the coordinates of each index on the object in the coordinate system defined on each object are known, the relative position and orientation between the objects (relative The position and orientation may also be estimated as an unknown parameter. Specifically, the coordinate system defined in one object among the plurality of objects in several 5 and (X _w in Equation ₅₎ world coordinates, the coordinates between the coordinate system X _b and the world coordinate system of the other objects Convert

と表して、Ｒ_ｗｂ、ｔ_ｗｂも未知パラメータとして同時に推定する。この場合、内部パラメータの概略値を算出する際には複数の物体のうち一つの物体上の指標のみを利用し、最適化計算を行う際に複数の物体上の指標を利用し物体間の相対的な位置及び姿勢も推定する。

R _wb and t _wb are also estimated simultaneously as unknown parameters. In this case, when calculating the approximate value of the internal parameter, only the index on one object among the plurality of objects is used, and the index on the plurality of objects is used when performing the optimization calculation. The estimated position and orientation are also estimated.

  When calculating the approximate value, the calculation may be performed for each object constituting the calibration object, and the approximate value that minimizes the evaluation function expressed by Equation 6 may be selected from them. When performing the optimization calculation, the posture of each camera described in the first modification may be estimated by including it in the unknown parameter. That is, the coordinate information of the index on each partial calibration object is also updated by correcting the relative position and orientation of each partial calibration object during the calculation of the optimum. Thereby, it is not necessary to manufacture a huge calibration object when performing calibration for a wide space, and calibration can be performed with high accuracy.

  Similarly, in the fourth embodiment, the positional relationship between a plurality of planes constituting the calibration object is known. Here, a calibration object composed of a plane having a known positional relationship is referred to as a partial calibration object. However, when the calibration object is composed of a plurality of partial calibration objects whose positional relationships are unknown, the positional relationship between the partial calibration objects may also be estimated using unknown parameters.

(Modification 3)
In the embodiment described above, the moving stage is used to move the three-dimensional measuring apparatus and the calibration object on a straight line at predetermined intervals. However, moving on a straight line at a predetermined interval is not limited to the moving stage. For example, it may be moved on a straight line using a moving stage or a robot with a higher degree of freedom. Alternatively, the translation may be realized by placing a calibration object on an object whose thickness is measured in advance with high accuracy, instead of mechanically moving it. With this modification, the present invention is implemented under more general conditions by increasing the choice of hardware used to move the three-dimensional measuring device or calibration object on the straight line at predetermined intervals during calibration. It becomes possible to do.

(Modification 4) The movement amount may not be equal. In the above-described embodiments, the intervals at which the three-dimensional measuring device and the calibration object are moved by the movement stage are equal. However, the movement interval by the movement stage does not need to be constant for each movement, and may be different for each movement as long as it is available as a known value.

(Modification 5) The camera model (lens distortion model) may be another model. In the embodiment described above, a model having a radial distortion and a tangential distortion as a lens distortion model (Equation 3) Was used. However, the lens distortion model is not limited to this. For example, a model proposed by Lenz et al. (R. Lenz and D. Fritsch, “Accuracy of videometry with CCD sensors,” .45, pp. 90-110, 1999.). The Lenz model is expressed by the following equation.

ただし、

である。すなわち、歪みのない座標と歪みの加わった座標の関係を関数として表すことができればいかなるレンズ歪みのモデルを用いてもよい。

However,

It is. In other words, any lens distortion model may be used as long as the relationship between coordinates without distortion and coordinates with distortion can be expressed as a function.

(Modification 6) The index on the calibration object may not be a point. In the embodiment described above, the position information on the calibration object such as a circular marker is represented as the position as the index on the calibration object. Index was used. However, the index on the calibration object is not limited to this, and the arrangement information may be expressed by an expression method other than the position.

  For example, it may be a straight line such as each side of a checkered pattern on the calibration object. In this case, the arrangement information of the index on the calibration object is a straight line parameter. An image feature (for example, an edge) corresponding to the straight line is detected, and a constraint condition that the image feature exists on a straight line that is a projected image on the straight image on the calibration object calculated by the estimation parameter is used. Perform calibration.

<Definition>
The calibration object in the present embodiment may be, for example, a calibration object in which indices with known coordinates on the calibration object are distributed on the surface, or from a plurality of planes whose positional relationships are known. It may be a calibration object configured. On the calibration object, the indices may be distributed three-dimensionally or may be distributed in a plane. Further, the plurality of planes are not necessarily parallel, and may be a plurality of planes forming an arbitrary angle. The index on the calibration object may be any index as long as it can be detected on the image and uniquely identified. For example, it may be a checkered intersection or a square index having a unique identifier. The calibration object may be a plurality of partial calibration objects whose relative positional relationships are fixed. Furthermore, the relative positional relationship may be unknown.

  The imaging device to be calibrated in the present embodiment may be any camera or projector as long as the projection model can be described by a pinhole model and a lens distortion model. In addition, the three-dimensional measurement apparatus to be calibrated in the present invention may be a stereo camera composed of two cameras, or may be composed of three or more cameras. Further, it may be a three-dimensional measuring device including a camera and a projector.

  The moving means in the present embodiment moves the calibration object while fixing the imaging apparatus in the scene, or moves the imaging apparatus while fixing the calibration object. The moving means may be a moving stage that linearly moves the calibration object / imaging device, or may be a moving stage or a robot with a higher degree of freedom. Further, it may be realized by placing a calibration object on an object whose thickness is measured with high accuracy in advance, instead of mechanically moving it. Further, the interval of movement by the moving means may be a constant interval or a different interval for each movement as long as it is obtained as known information.

  The feature detection means in the present embodiment detects features on the image for calibration. The feature detection means detects an index on the calibration object. Further, the projector detects a pattern projected on the calibration object. The pattern projected by the projector may be a pattern composed of a plurality of patterns, such as a spatial coding method pattern, or a single pattern.

  The parameter calculation means in this embodiment is based on the coordinates of the feature image detected by the feature detection means, the geometric information about the calibration object, the known movement amount by the movement means, the internal parameters of the camera and the projector, the cubic The relative position and orientation between the camera and the camera (projector) constituting the original measuring apparatus are calculated. The parameter calculation means may calculate the calibration object and the orientation of the imaging device as different parameters for each movement by the moving stage. Further, when the calibration object is composed of a plurality of partial calibration objects whose mutual positional relationships are unknown, the positional relationship between the partial calibration objects may be calculated. The parameter calculation means may perform calibration using the coordinates of the index on the calibration object, or may perform calibration using the coordinates of the pattern projected by the projector. Furthermore, the parameters of the straight line moving by the moving stage may be obtained by numerical calculation, or when a plurality of straight line parameter candidates are set, each straight line parameter is known and calibration is performed multiple times, and the best evaluation value is obtained The parameters may be selected.

  In the calibration in the present embodiment, other parameters may be changed as long as the moving direction and the moving interval by the moving stage. For example, the posture of the imaging device or the calibration object may be changed by the movement of the moving stage.

Claims (10)

A straight line in any movement interval one of a predetermined and two imaging devices relative position and orientation is fixed and the calibration object are moved a plurality of times, the calibration by the two imaging devices for each Before moving Acquisition means for acquiring a plurality of images obtained by photographing the object ;
An approximate value acquisition means for acquiring an approximate value in the direction of the straight line and an approximate value of a parameter of each of the two imaging devices;
Holding means for holding position information of an index on the calibration object;
Detecting means for detecting an index on the calibration object from each of the plurality of images , and obtaining a correspondence relationship between image coordinates of the detected index image on the image and position information of the index held in the holding means ;
Based on the correspondence between the image coordinates and the position information of the index in the plurality of images, the plurality of images at said predetermined movement interval one of the calibration object and the two imaging devices It is an image taken every time it moves on the straight line , and the approximate value of the straight line direction and the approximate value of the parameters relating to each of the two imaging devices are corrected under the condition that the direction of the straight line is undetermined. Parameter deriving means for deriving parameters relating to each of the two imaging devices ,
Based on the correspondence between the image coordinates and position information of the index in the plurality of images and the parameters relating to each of the two imaging devices derived by the parameter deriving means, the relative positions and orientations of the two imaging devices are calculated. A calibration apparatus having position and orientation deriving means for deriving . The parameter derivation means, and an approximate value of the parameter for each direction of the approximate values and the two imaging devices of the linear as an initial value, and wherein the deriving the parameters for each said two imaging devices by iterative calculation The calibration device according to claim 1. The parameter deriving means includes image coordinates on the image of the index, the index is the holding on the basis of the outline parameters of the imaging device so as to reduce the deviation between the projected coordinates projected onto the image, the straight line by correcting the approximate value of the parameter for each approximate value and the two imaging devices in the direction of, the calibration according to claim 1 or 2, wherein the deriving the parameters for each said two imaging devices apparatus. When the calibration object is composed of a plurality of partial calibration objects,
The holding means holds position information of indices arranged on the plurality of partial calibration objects,
The parameter derivation unit includes a correspondence relationship between the position information of the index image coordinates to be the holding of the image of each indicator disposed on the plurality of partial calibration object detected by the detecting means, the advance Based on the determined movement interval , the relative position between the plurality of partial calibration objects is further corrected by correcting the relative position and orientation between the plurality of partial calibration objects given as initial values. 4. The calibration apparatus according to claim 3, wherein an orientation is derived, and the projection coordinates are derived based on the derived relative position and orientation and an approximate parameter of the imaging apparatus.   5. The calibration apparatus according to claim 1, wherein the calibration object has a flat surface, and a plurality of indices exist on the flat surface.   The calibration apparatus according to claim 5, wherein the calibration object further includes a three-dimensional index on the plane. The parameter derivation unit estimates an attitude of the imaging apparatus with respect to the calibration object at each position when the imaging apparatus or the calibration object moves at each predetermined movement interval. The calibration device according to any one of 1 to 6.   The calibration apparatus according to claim 1, wherein the parameter relating to the imaging apparatus includes an internal parameter of a camera included in the imaging apparatus. Acquisition means of the calibration device, a straight line in any movement interval one of a predetermined relative position and orientation is fixed with two imaging devices and the calibration object are moved a plurality of times, each time Before moving An obtaining step of obtaining a plurality of images obtained by photographing the calibration object by the two imaging devices;
An approximate value acquisition unit for acquiring an approximate value in the direction of the straight line and an approximate value of a parameter of each of the two imaging devices;
The acquisition unit of the calibration apparatus detects an index on the calibration object from each of the plurality of images , and the correspondence between the image coordinates of the detected index on the image and the position information of the index held in the holding unit A detection process for determining the relationship ;
Based on the correspondence between the image coordinates of the index and the position information in the plurality of images, the parameter deriving unit of the calibration device is configured so that the plurality of images are either the two imaging devices or the calibration object. An image taken each time one of the images is moved on the straight line at the predetermined movement interval, and the approximate value of the direction of the straight line and the two by correcting the approximate value of the parameter for each imaging device, and the parameter deriving step of deriving the parameters for each said two imaging devices,
The position and orientation deriving means of the calibration device is based on the correspondence between the image coordinates of the index and the position information in the plurality of images, and the parameters relating to each of the two imaging devices derived by the parameter deriving step. A calibration method including a position and orientation deriving step of deriving a relative position and orientation of the two imaging devices . The program for functioning a computer as each means of the calibration apparatus of any one of Claims 1 thru | or 8 .

Images