JP2010186265A - Camera calibration device, camera calibration method, camera calibration program, and recording medium with the program recorded threin - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a camera calibration device for easily calibrating a camera. <P>SOLUTION: This camera calibration device is provided with a reference point adjustment part 104 for setting a prescribed two-dimensional coordinate system in an image acquired by photographing a reference object wherein geometric information of a reference point is known by use of a camera 102 of a calibration target, and extracting a prepared two-dimensional coordinate value of the reference point, and a reference point measurement part 105, and is provided with a camera calibration processing part 106 for adjusting a position or an attitude of the reference object until a part of the obtained reference points satisfies a specific geometric restraint condition, and calibrating external information or internal information of the camera 102 from the coordinate value of the reference point obtained on two-dimensional coordinates and the geometric information of the reference point according to calculation derived from a perspective projection condition when three-dimensional points of all the reference points on the reference object are projected to the two-dimensional coordinate system. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention can be used for camera calibration of a monocular camera or a multi-view camera arranged so as to surround a subject, and a small number of reference points or a part of the reference points is used for the purpose of reducing the cost of camera calibration work. The present invention relates to a camera calibration apparatus that photographs a reference object for calibration including the camera and calibrates the camera based on the projection relationship (determines a camera parameter).

  Camera calibration (camera calibration) is performed to obtain internal parameters such as a camera-specific focal length and image center, and external parameters such as posture and position information at the time of camera placement. In the conventional method, a two-dimensional object having a lattice pattern as shown in FIG. 1 is photographed with respect to the camera to be calibrated, and from the geometric information about the reference object and the projection information on the image observed by the camera, the camera Internal and external parameters (both are referred to as camera parameters) are estimated (see Non-Patent Document 1).

  As an example using a simpler reference object, a method of using a one-dimensional object with a known length as shown in FIG. Patent Document 2).

  In computer vision, there is a basic principle called Structure From Motion, and there is a method for restoring the three-dimensional shape and movement of a subject by moving the camera relative to a stationary subject (for moving subjects). On the other hand, the principle is the same even if the camera is stationary). Self-calibration (also called auto-calibration) is one of the methods based on this principle. It does not require the above-mentioned known reference object and uses a feature point observed from a static video scene. Estimate the parameters.

  Non-Patent Document 3 proposes a self-calibration applied to a multi-view camera system.

Z. Zhang, “A flexible new technology for camera calibration”, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 22, no. 11, pp. 1330-1334, 2000. Z. Zhang, “Camera calibration with one-dimensional objects”, IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 26, no. 7, pp. 892-699, 2004. Isao Miyagawa, Yoshihiro Hiyama, Yuichi Iwabuchi, “Calibration of multi-viewpoint cameras using one-dimensional objects”, IEICE Technical Report (PRMU 2006-177), Vol. 106, no. 429, pp. 37-42, 2006.

  In conventional camera calibration methods, it is common practice to use a reference object as represented by Non-Patent Documents 1 and 2. In FIG. 1, the two-dimensional coordinates of the reference point on the two-dimensional object are known (Non-Patent Document 1). In FIG. 2, the length L of the one-dimensional object, the distance from point A to point C, and the point B to point C are shown. It is a premise that the ratio of the distance to is known (Non-Patent Document 2).

  In camera calibration using such a reference object, it is necessary to photograph a sufficient amount of reference points. In other words, a lot of time is taken for the work of photographing the reference object before the camera calibration. Further, in a multi-view camera system environment, the same camera calibration work must be repeated for each camera.

  On the other hand, according to self-calibration, the camera can be calibrated based on feature points observed from a static video scene without requiring a known reference object. However, since the observation accuracy of the feature point sensitively affects the calibration result, in order to stably estimate the camera parameters, the image coordinates of the feature point must be measured with high accuracy from a static scene. . In Non-Patent Document 3, self-calibration is applied to a multi-view camera system using artificially prepared reference points. However, a huge amount of reference points are observed for each camera, and work efficiency is improved. There is no simplification.

  Thus, one of the factors that reduce the work efficiency of the camera calibration and increase the work amount is the amount of feature points or reference points observed by the camera. In particular, in an outdoor environment, it is not possible to assume a blue back for automatically extracting reference points, so manual operation is required to obtain image coordinates of the reference points more accurately. In order to stably estimate camera parameters, it is desirable to measure a large number of reference points over a wide range. However, in a real environment where a security camera is arranged, it is not always guaranteed to capture a sufficient amount of reference points for camera calibration, and there is not always enough time to capture them. Depending on the operational situation, it is necessary to respond to single-eye or multi-view camera calibration from a small number of reference points.

  The present invention uses a special reference object in a monocular or multi-viewpoint camera system to reduce the imaging work cost for camera calibration, and easily calibrates a monocular camera from a small number of reference points observed from the object. Alternatively, a main problem is to easily calibrate a multi-viewpoint camera arranged so as to surround a subject.

  In order to explain the means for solving the above problems, a basic principle of projective geometry and a calculation algorithm based on the principle will be described. The principle and method of the solution means here are described focusing on a monocular camera, but can be easily extended to a multi-view camera. The expansion will be described in detail in Example 4.

  In explaining the principle of projective geometry, an overview of the reference object and the positional relationship with the camera must be defined. FIG. 3 shows an overview of the reference object used in the present invention. The reference object is roughly composed of a vertical bar 11 and a horizontal bar 12, which are in a relationship orthogonal to each other at a point O. In the vertical bar 11, the point O is an end point, the point A is another end point, and the point B is arranged therebetween. Similarly, the horizontal bar 12 has a point O as an end point, a point C as another end point, and a point D is arranged therebetween. It is assumed that the horizontal bar 12 and the vertical bar 11 can rotate freely as shown in FIG. Points O, A, B, C, and D are observed as reference points in an image captured by the camera.

For convenience, the point O is the origin on the world coordinate system, the vertical bar 11 is coincident with the Z _w axis of the world coordinate system, and points A and B are set at heights h ₁ and h ₂ from the origin O, respectively. It shall be. On the other hand, it is assumed that the horizontal bar 12 is on the X _w Y _w plane, and the horizontal bar 12 coincides with the Y _w axis of the world coordinate system, and the point C is located at a depth of d ₃ and d ₄ from the origin O, respectively. , D are set. When the reference object in FIG. 3 taken by the camera C _1, FIG. 5 (a), the without loss of generality be set the world coordinate system X _w Y _w Z _w onto the reference object shown in (b). FIG. 4 is an example of an image obtained by photographing the reference object of FIG. 3, and it is assumed that image coordinates: p _{0 to} p ₄ of each reference point can be measured from this image.

Next, a camera coordinate system XYZ is set, and it is assumed that the geometric projection by this camera can be modeled as a perspective projection according to a pinhole camera. If the three-dimensional point on the reference object is P _j = (X _j , Y _j , Z _j ) and the camera viewpoint position is (Tx, Ty, Tz), the two-dimensional point of the point P _j observed on the image (X _j , y _j ) is

Given in. f is a focal length, and R ₁₁ , R ₁₂ ,..., R ₃₃ correspond to elements of a 3 × 3 rotation matrix. In the camera model handled in the present invention, the center of the image is the projection center, and lens distortion is ignored.

FIG. 5 (a), arranged as reference points to (b) the, Z _w point on axis A, B, and Y point on the _w axis C, D. Therefore, since it can be assumed that the camera is arranged in such a posture that the rotation around the Y axis is 0 with respect to these reference points, the rotation matrix is

  And can be given. φ is the rotation angle around the X axis, and θ is the rotation angle around the Z axis.

The basic principle of the present invention is that five reference points on a reference object: O, A, B, C, and D, coordinate values: p _{0 to} p ₄ on an image are observed according to equations (1) and (2). Assuming that the metric information about the image coordinates and reference points: h ₁ , h ₂ , d ₃ , d ₄ , the camera focal length f, the rotation angle in the camera coordinate system: φ, θ, and The camera viewpoint position (Tx, Ty, Tz) in the world coordinate system is solved as an inverse problem. It is necessary to derive a calculation formula or algorithm necessary for the estimation from the relational expression of the perspective projection linked by the formulas (1) and (2). Thereafter, the calculation formula is derived.

If the points A and B are points at the heights h ₁ and h ₂ from the origin O, respectively, the projection images p _j , j = 0, 1, and 2 including the origin O are

Can be described. From the relationship of the perspective projection image of the origin O shown in equations (4) and (5),
−f (Txcos (θ) + Tysin (θ) cos (φ) + Tzsin (θ) sin (φ)) = x ₀ (Tysin (φ) −Tzcos (φ)), (8)
f (Txsin (θ) −Tycos (θ) cos (φ) −Tzcos (θ) sin (φ)) = y ₀ (Tysin (φ) −Tzcos (φ)) (9)
Therefore, substituting equations (8) and (9) into equations (6) and (7) respectively,
(X _j −x ₀ ) Tysin (φ) − (x _j −x ₀ ) Tz cos (φ) −fh _j sin (θ) sin (φ) = − x _j h _j cos (φ), (j = 1, 2), (10)
(Y _j −y ₀ ) Tysin (φ) − (y _j −y ₀ ) Tz cos (φ) −fh _j cos (θ) sin (φ) = − y _j h _j cos (φ), (j = 1, 2) (11)
Get. Furthermore, from equations (10) and (11), respectively

  Can guide you.

On the other hand, the points C and D are defined as points on the _Yw axis for convenience. Here, the rotation parameter ω is introduced for the reference points C and D. Points C ′ and D ′ are points obtained by rotating the points C and D around the _Zw axis at a rotation angle ω. FIG. 9 shows an example of an image when the horizontal bar 12 of the reference object is rotated at the rotation angle ω and the horizontal bar matches the straight line x = x ₀ passing through the x coordinate of the observation point p ₀ . That is, the coordinates of the observation points of the points C ′ and D ′ are obtained by p ₃ = (x ₀ , y ₃ ) and p ₄ = (x ₀ , y ₄ ), respectively.

In FIG. 5B, the three-dimensional coordinates of the points C ′ and D ′ can be given as (−d _j sin (ω), d _j cos (ω), 0), j = 3,4. From (1) and Equation (3), x _j {(d _j cos (ω) −Ty) sin (φ) + Tz cos (φ)} = f {(d _j sin (ω) + Tx) cos (θ) − (d _j cos (ω) −Ty) sin (θ) cos (φ) + Tzsin (θ) sin (φ)}, (j = 3,4) (15)
It becomes. Here, using equation (8),

  The relationship is obtained.

On the other hand, the y coordinate on the image of the points C ′ and D ′ is −y _j {(d _j cos (ω) −Ty) sin (φ) + Tz cos (φ)} = f {(d _j sin (ω) + Tx) sin (θ) + (d _j cos (ω) −Ty) cos (θ) cos (φ) −Tzcos (θ) sin (φ)}, (j = 3,4) (17)
So, using equation (9),
(Y _j −y ₀ ) Tysin (φ) − (y _j −y ₀ ) Tz cos (φ) −fd _j {sin (θ) sin (ω) + cos (θ) cos (φ) cos (ω)} = y _j d _j sin (φ) cos (ω), (j = 3,4) (18)
It becomes. If formula (18) is rearranged,

  Can be derived. Substituting equation (16) into equation (19) and rearranging,

  When substituting equation (16) into equation (13) and rearranging,

  Furthermore, if formula (20) is substituted into formula (21) and rearranged,

  (A negative solution is selected assuming that −π / 2 <φ <0 in the camera arrangement of FIG. 5A). Since φ can be obtained from Equation (22), ω can be calculated using Equation (20), and the focal length f can be obtained from Equation (16).

  Finally, an equation that combines equations (4) to (7) and (18):

  By solving the above, the viewpoint position (Tx, Ty, Tz) of the camera can be restored. However, in Formula (23),

  Is replaced.

Using the above derivation formula, the camera parameter can be obtained from the image coordinates of the reference point. First, image coordinates: p ₀ , p ₁ , p ₂ of reference points: O, A, B are measured from the image of FIG. Next, the horizontal bar 12 is rotated at the rotation angle ω so as to coincide with x = x _0, and the image coordinates: p ₀ , p ₃ , p ₄ of the reference points C ′, D ′ in the coincidence state are measured. To do. Next, camera parameters can be obtained by using the image coordinates of the reference point as input data and sequentially executing the following processing according to the calculation formula derived above.
Step 1: The rotation angle θ is obtained from equation (14).
Step 2: The rotation angle φ is obtained from equation (22).
Step 3: The parameter ω is obtained from equation (20).
Step 4: The focal length f is obtained from equation (16).
Step 5: The viewpoint position (Tx, Ty, Tz) of the camera is obtained by Expression (23).

  As described above, the present invention uses the reference object of FIG. 3 to measure five reference points on the reference object, and obtains camera parameters by the above calculation procedure. Therefore, the basic configuration of the present invention is a configuration for executing these processes.

  According to a first aspect of the present invention, there is provided a camera constituting apparatus that calibrates internal information of an image input device and external information including posture and position from an image acquired using a single eye or a plurality of image input devices. Then, a predetermined two-dimensional coordinate system is set in an image obtained by photographing a reference object whose reference point geometric information is known using the image input device to be calibrated, and the two-dimensional coordinate value of the prepared reference point A reference point extracting means for extracting the reference object, and a reference object for adjusting a posture or a position of the reference object until a part of the reference points satisfy a specific geometric constraint condition among the reference points obtained by the reference point extracting means It is derived from the perspective projection conditions when the three-dimensional points of all the reference points on the reference object are projected onto the two-dimensional coordinate system on the image with respect to the adjustment means and the reference object adjusted by the reference object adjustment means. Calculation It, from the geometric information of the reference point and the coordinate value of the reference point obtained in the two-dimensional coordinate system, is characterized by and a camera calibration calculating means for calibrating the internal information or external information of the image input device.

  Since the present invention performs camera calibration using a small number of reference points, it does not require an enormous amount of reference points unlike conventional camera calibration (Non-Patent Documents 1 to 3). In the present invention, the use of a special reference object for smoothly measuring the coordinate value of the reference point on the image simplifies the camera calibration operation and shortens the processing time.

  Furthermore, if the plurality of cameras are arranged so as to surround the reference object, the plurality of cameras can be calibrated at the same time, which contributes to seamless 3D image generation, mosaic image generation, or panoramic image generation using a multi-viewpoint camera. .

It is explanatory drawing which shows the camera calibration operation | work using the two-dimensional object in the conventional camera calibration method. It is explanatory drawing which shows the camera calibration operation | work using the one-dimensional object in the conventional camera calibration method. It is explanatory drawing which shows the reference object comprised from the vertical bar and horizontal bar which are used by this invention. It is explanatory drawing which shows the example of an image when the reference object of FIG. 3 is image | photographed. It is explanatory drawing which shows the positional relationship of the camera and reference point in the example of embodiment of this invention. It is a basic composition figure of Example 1 of the present invention. It is a flowchart which shows the whole process of Example 1 of this invention. It is a flowchart which shows the process of the reference point adjustment part in Example 1 of this invention. It is explanatory drawing which shows the example of an image when rotation of a horizontal bar is completed in Example 1 of this invention, and the reference object was image | photographed. It is a flowchart which shows the process of the camera calibration calculation in Example 1 of this invention. It is explanatory drawing which shows the example of the reference object used in Example 2 of this invention. It is a flowchart which shows the process of the reference point measurement in Example 2 of this invention. It is a flowchart which shows the process of the camera calibration calculation in Example 3 of this invention. It is explanatory drawing which shows the example of arrangement | positioning of the multiview camera in Example 4 of this invention. It is explanatory drawing which shows the relationship of the world coordinate system regarding the two cameras in Example 4 of this invention. It is a basic composition figure of Example 4 of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to the following embodiments.

  FIG. 6 is a basic configuration diagram relating to the camera calibration invention described in claim 1, and FIG. 7 is an overall processing flow. The first embodiment is an embodiment using the reference object of FIG. 3 (claim 2), and will be described with reference to FIGS. The present invention presents a reference object in front of a camera 102 to be calibrated and cooperates with a reference point adjustment unit 104 to control a subject position, a subject control unit 101 as a reference object adjustment unit, a calibration target camera 102, a calibration An image input unit 103 that acquires an image to be used for the reference, and a reference point adjustment unit 104 that performs preprocessing for camera calibration processing in conjunction with the subject control unit 101 on the image from the image input unit 103, A reference point measuring unit 105 that measures image coordinates necessary for calibration processing from the captured image, and a camera calibration processing unit 106 as camera calibration calculation means for executing camera calibration.

  In the first embodiment, the reference point extraction unit of the present invention is configured by a reference point adjustment unit 104 and a reference point measurement unit 105.

  In this configuration, the camera 102 to be calibrated does not necessarily have to be connected as a component, and a sample image necessary for calibration may be acquired through the image input unit 103. The image input unit 103 includes a hard disk, a RAID, and the like. Either a device, a recording medium such as a CD-ROM, or a remote data resource via a network may be used. Furthermore, it is possible to acquire an image in real time according to the necessity of processing, and the present invention does not necessarily require a storage device.

First, in the subject control unit 101 in FIG. 6, the reference object in FIG. 3 is presented in front of the camera 102 as a photographing subject (step S11 in FIG. 7; subject presentation). The reference object shown in FIG. 3 is composed of a vertical bar 11, a horizontal bar 12, and a plane board serving as a base 13 thereof. The vertical bar 11 is provided with reference points at positions A and B at the heights h ₁ and h _{2 with} the position indicated by O as the origin. The subject control unit 101 can freely rotate the horizontal bar 12 around the vertical bar 11 as a rotation axis in response to a command. On the horizontal bar 12, reference points C and D are installed at distances of d ₃ and d ₄ from the point O, respectively. It is assumed that the reference points are colored with a special paint that stands out even outdoors (this makes extraction by image processing easier).

  A sample image is obtained from the image input unit 103 through the camera 102 (step S12 in FIG. 7; image acquisition). FIG. 4 shows an example of the image obtained at this time. In this figure, the image center is the center of the image coordinate system, and the xy image coordinate system with the point as the origin is displayed in an overlapping manner. Camera calibration is performed using this sample image, but pre-processing is performed by the reference point adjustment unit 104 in FIG. 6 before measuring the image coordinates of each reference point.

The reference point adjustment unit 104 first detects the vertical bar 11 (one-dimensional object composed of points A, B, and O) by image processing, and extracts straight line parameters. This can be applied to image processing such as Hough transform. Next, scanning is performed along the detected straight line, and reference points O, A, and B colored with special ink are extracted. If the shape of the reference point is designed as a sphere, the sphere portion can be extracted using Blob analysis or the like, and the center of the sphere region can be observed as the image coordinates of the points O, A, and B. it can. In any case, the image coordinates: p ₀ = (x ₀ , y ₀ ), p ₁ = (x ₁ , y ₁ ), p ₂ = (x ₂ , y ₂ ) are obtained by this observation means.

Next, as shown in FIG. 9, the horizontal bar 12 is rotated in accordance with steps S13 and S14 of FIG. 7 until the reference points C and D on the screen coincide with the straight line x = x ₀ (reference when they coincide). The points are C ′ and D ′). The rotation operation of the horizontal bar 12 is performed according to the processing flow of FIG. First, a default rotation angle is set (step S21). Subsequently, straight line parameters of the horizontal bar 12 are obtained from the area of the horizontal bar 12 using image processing such as Hough transform (step S22). The horizontal bar 12 is rotated at a small rotation angle Δω until the extracted straight line overlaps with the straight line x = x ₀ passing through the previously extracted p ₀ = (x ₀ , y ₀ ) (steps S23 to S23). S25). When the horizontal bar 12 overlaps the straight line x = x ₀ within the allowable error range, the rotation operation is stopped.

The image of FIG. 9 is obtained by this series of processing. On this image, the image coordinates of the reference points C ′ and D ′ are measured. According to the processing flow of FIG. 8, since the horizontal bar 12 overlaps with the straight line x = x ₀ , it is possible to scan the top of the straight line x = x ₀ from top to bottom and detect the special ink portion by image processing. Coordinates p ₃ = (x ₀ , y ₃ ) and p ₄ = (x ₀ , y ₄ ) are obtained. Through the above processing flow, the reference point measurement unit 105 in FIG. 6 uses image coordinates on the vertical bar 11: p ₀ = (x ₀ , y ₀ ), p ₁ = (x ₁ , y ₁ ), p ₂ = (X ₂ , y ₂ ) and image coordinates p ₃ = (x ₀ , y ₃ ), p ₄ = (x ₀ , y ₄ ) on the horizontal bar 12 are obtained (step S15 in FIG. 7; reference points) Measurement).

If the image coordinates of the reference point can be measured, the camera calibration processing unit 106 in FIG. 6 performs geometric information on each reference point: h ₁ , h ₂ , d ₃ , d ₄ and the image coordinate of the corresponding reference point: p ₀ = (x ₀ , y ₀ ), p ₁ = (x ₁ , y ₁ ), p ₂ = (x ₂ , y ₂ ), p ₃ = (x ₀ , y ₃ ), p ₄ = (x ₀ , Using y ₄ ) as input data, camera parameters are calculated according to the processing flow of FIG. Since the derivation of the calculation formula and the basic principle have been described above, how to obtain the camera parameters using the calculation formula already derived above will be described below.

  First, the values p and q replaced by the equations (12) and (13) are calculated. If these values are obtained, the rotation angle θ is calculated by the equation (14) (step S31 in FIG. 10; rotation angle θ calculation). Next, using the value r (replaced by equation (19)) and the previously obtained rotation angle θ, the rotation angle φ is calculated by equation (22) (step S32 in FIG. 10; rotation angle φ calculation). ). Next, a rotation parameter ω necessary for later calculation is calculated. For this calculation, the value r and the rotation angle φ are used to obtain the equation (20) (step S33 in FIG. 10; rotation parameter ω calculation). Further, using the rotation angles θ, φ and the rotation parameter ω, the focal length f is calculated by the equation (16) (step S34 in FIG. 10; focal length f calculation). Finally, the camera viewpoint position (Tx, Ty, Tz) is calculated by solving the simultaneous linear equations of Expression (23) using the camera parameters and rotation parameters obtained so far (step S35 in FIG. 10; position ( Tx, Ty, Tz) calculation).

  As described above, the reference object designed in FIG. 3 is used as a photographic subject for calibration, and processing is performed according to the processing flow of FIG. 7 under the basic configuration of FIG. , (B), the camera viewpoint position (Tx, Ty, Tz), the rotation angles θ and φ in the camera coordinate system, and the camera focal length f can be obtained.

  The second embodiment is an embodiment relating to claim 3. Although the basic configuration diagram is the same as that of the first embodiment, only the difference between the overview of the reference object and the processing based thereon is described.

In the present embodiment, the subject control unit 101 in FIG. 6 presents the reference object in FIG. 11 as a photographic subject in front of the camera 102. In the reference object of FIG. 11, instead of rotating the horizontal bar 12, a plane depicting concentric circles 14 with radii d ₃ and d ₄ is attached perpendicular to the vertical bar 11. In the case of a monocular camera, calibration can be performed only with the vertical bar 11 and the concentric circle 14. However, assuming that the camera can be expanded to a multi-view camera, an azimuth bar 22 that rotates about the vertical bar 11 along the concentric circle 14 as a rotation axis is provided. Prepare (an example of using this azimuth bar 22 will be described in Example 4).

FIG. 12 shows the processing flow of the second embodiment, which makes the rotation of the horizontal bar 12 unnecessary compared with the first embodiment. In step S41, the subject is presented in the same manner as in step S11 in FIG. 7, and in step S42, an image is acquired in the same manner as in step S12 in FIG. If concentric circles with radii d ₃ and d ₄ are prepared as shown in FIG. 11, the rotation of the horizontal bar 12 becomes unnecessary. Instead, as in FIG. 9, the linear x = x ₀ scans down the point which intersects the concentric 14 from above the upper straight line x = x _0, it is detected in the edge portion intersecting the concentric circle 14. Thus, reference point C ′, D ′ image coordinates: p ₃ = (x ₀ , y ₃ ), p ₄ = (x ₀ , y ₄ ) are measured (step S43 in FIG. 12; measurement of reference point). If the image coordinates of the reference point are obtained, camera calibration can be performed as in the first embodiment (step S44 in FIG. 12; calculation of camera calibration).

  The reference object of FIG. 11 is easy to design and manufacture when the shooting range of the camera is relatively small. However, when the shooting scale increases, the concentric circles 14 corresponding to the scale must be designed and manufactured. For this reason, it is advantageous to rotate the horizontal bar 12 of the first embodiment when the photographing scale becomes large.

  The third embodiment relates to the fourth aspect. Although the basic configuration diagram is the same as that of the first embodiment, since the processing flow of camera calibration is different, only this difference will be described below.

  Before that, it supplements about the subject which a present Example solves. When the reference object of FIG. 3 or FIG. 11 is photographed, the vertical bar 11 in FIG. 4 may be photographed substantially along the y axis.

  Therefore, once the camera parameters are obtained by the camera calibration processing flow described in the first embodiment, φ, Ty, Tz are substituted into formula (31) described later, and θ is calculated again. When θ is obtained, parameters φ, ω, f, (Tx, Ty, Tz) are sequentially obtained according to the procedure described above. When the camera parameter is updated, θ is obtained again by Expression (31), and the camera parameter is obtained by the same procedure. By continuing this until the solution stabilizes, the camera parameters can be adjusted iteratively.

  That is, first, when camera parameters for calibration (calibration external information) are obtained, camera parameters for evaluation (evaluation external information) are recalculated from the camera parameters, and the error between the calibration external information and the evaluation external information is large Replaces the evaluation external information with the calibration external information, recalculates the camera parameters, and repeats the calibration calculation until the error becomes smaller than a predetermined error.

  FIG. 13 shows a processing flow in the camera calibration processing unit 106 of the third embodiment. Unlike the first embodiment, in preparation for the situation described above, the third embodiment repeatedly calculates camera parameters. When the number of iterations is the first, camera parameters are obtained by the same flow as the camera calibration process of the first embodiment (steps S31 to S35 described in FIG. 10). Next, it is determined whether the number of iterations is a predetermined number N (step S36). If the number of iterations does not exceed the predetermined number N,

  The rotation angle θ ′ is recalculated by calculating (Step S37). The rotation angle θ ′ is replaced with the rotation angle θ, and the camera parameters are calculated again according to the processing flow of FIG. After this iteration is repeated until the iteration count N is exceeded, camera parameters are output.

  The fourth embodiment is a method of calibrating a plurality of cameras (multi-viewpoint cameras) using the reference object shown in FIG. First, in order to extend the calibration method using the monocular camera described in the first to third embodiments to the calibration method using the multi-view camera, the multi-view camera is supplemented.

FIG. 14 shows an example in which a total of five cameras C ₁ to C ₅ are arranged so as to surround the reference object of FIG. This figure is a bird's eye view as viewed from above, the coordinate system of the camera C ₁ is assumed as the world coordinate system X _w Y _w Z _w is set. That is, as in the _first to third embodiments, the rotation matrix related to the camera C ₁ is the posture given by the expression (3) (the horizontal bar 12 of the reference object without rotation around the _Zw axis with respect to the camera C ₁ . assume that is associated with the world coordinate system X _w Y _w Z _w in is directed). Therefore, for the other cameras C _i , i = 2,..., 5, the reference object is not placed in such a posture, and is freely arranged so as to shoot around the reference object of FIG. And Here, in order to estimate the external parameters of the multi-viewpoint camera on the same coordinate system, for any one camera (camera C _{1 in} this embodiment) that matches the reference, FIG. Consider the world coordinate system of (b).

If the same method as in the _first to third embodiments is used, the camera parameters of the camera C ₁ can be obtained. On the other hand, for the other cameras C _i , i = 2,..., 5, the points C and D are simply rotated around the _Zw axis, and the same relationship (respectively) between the camera C ₁ and the world coordinate system. It can be easily imagined that the relationship between the camera rotation matrix and the orientation given by equation (3) may be related to the world coordinate system.

That is, the present invention can be applied to a plurality of cameras C _i , i = 1, 2,..., M (hereinafter suffix i is attached for convenience for camera identification). Reference points observed by each camera are A _i , B _i , O _i , C ′ _i , D ′ _i , and their observation coordinates are p _ij , i = 1, 2,..., M, j = 1, 2, 3, 4 Using the calculation formula described above, the camera parameters at each viewpoint: f _i , φ _i , θ _i , Tx _i , Ty _i , Tz _i , ω _i , i = 1, 2, ..., M can be obtained. Here, the reference points on the vertical bar 11: A _i = A, B _i = B, and O _i = O are reference points common to the cameras. On the other hand, the projection points of the reference points C ′ _i and D ′ _i rotate the horizontal bar 12 so as to be constrained on x = x ₀ in the image obtained by each camera. Three-dimensional coordinates are different. Therefore, since the same reference point on the reference object is not photographed by all cameras, even if the calibration method of the first to third embodiments is applied to each camera and external parameters are obtained, the same reference point on the same world coordinate system is obtained. It is not expressed. The fourth embodiment performs multi-viewpoint camera calibration by converting this into the same world coordinate system.

First, the basic principle of calibrating two cameras will be described with reference to the example of FIG. When the camera calibration method with a _single eye described in the first to third embodiments is applied to the camera C ₁ and the camera C ₂ independently, the world coordinate systems are different as shown in FIG. The camera C ₁ has a world coordinate system of X _w Y _w Z _w and the camera C ₂ has a world coordinate system of X ′ _w Y ′ _w Z ′ _w . In this figure, the world coordinate system X ′ _w Y ′ _w Z ′ _w can be considered as a rotation of the world coordinate system X _w Y _w Z _w by rotation around the Z _w axis. That is, if this rotation angle is known, the external parameters of the camera C ₂ restored on the world coordinate system X ′ _w Y ′ _w Z ′ _w can be converted to the world coordinate system X _w Y _w Z _w .

In the fourth embodiment, the horizontal bar 12 in FIG. 3 is used to restore the external parameters of each camera on the same coordinate system. Reference points C ₀ and D ₀ when the horizontal bar 12 is appropriately rotated are observed by the camera C ₂ . Alternatively, when the horizontal bar 12 is not properly rotated but is calibrated using the reference object shown in FIG. 3, the reference point on the horizontal bar 12 that overlaps x = x ₀ in the image obtained by the camera C ₁ is defined as the camera C. You may shoot with ₂ . In this case, only the image acquired by camera C ₁ is one, no effect on the method of calibration since. When the reference object shown in FIG. 11 is used, the azimuth bar 22 is fixed in an appropriate orientation as a reference, and the camera C ₂ is used to measure a point intersecting the concentric circle. The camera can be calibrated. Therefore, hereinafter, an embodiment using the reference object of FIG. 3 will be described.

In the image obtained by the camera C _1, reference point C ₀ on the horizontal bar 12, D ₀ of the image _{coordinates: (x 13, y 13)} , in (x _14, y _14), an image acquired by the camera C _2, Assume that the image coordinates (x ₂₃ , y ₂₃ ) and (x ₂₄ , y ₂₄ ) of the reference points C ₀ and D ₀ on the horizontal bar 12 are obtained. Since the reference points C ₀ and D ₀ are three-dimensional coordinates on the X _w Y _w plane, from the equations (1), (2), and (3),

And can be transformed. Substituting the image coordinates of the reference point and the camera parameters in these equations, the three-dimensional coordinates of the reference point _{C 0 (X 3, Y 3} , 0), 3 -dimensional coordinates of the reference point D ₀ (X _4, Y ₄ , 0) can be restored.

Using the equations (32) and (33), the three-dimensional reference points C ₀ and D ₀ restored from the camera parameters and image coordinates of the camera C ₁ : (x ₁₃ , y ₁₃ ) and (x ₁₄ , y ₁₄ ) The coordinates are (X ₃ , Y ₃ , 0) and (X ₄ , Y ₄ , 0), respectively. On the other hand, the camera parameters and image coordinates of the camera C ₂ and the three-dimensional coordinates of the reference points C ₀ and D ₀ restored from (x ₂₃ , y ₂₃ ) and (x ₂₄ , y ₂₄ ) are respectively (X ′ ₃ , Y ′ ₃ , 0), (X ′ ₄ , Y ′ ₄ , 0). From the 3D coordinate values restored by each camera,

(The rotation angle can be calculated at any one point, but here the average value from two points), the difference between the rotation angles:
ΔΩ ₂ = Ω ₂ −Ω ₁ (36)
, As shown in Figure 15, with respect to the world coordinate system X _w Y _w Z _w camera C ₁ are assumed, the world coordinate system camera C ₂ is assumed X _'w Y' of _w Z _'w The rotation angle around the _Zw axis.

Therefore, the conversion of the external parameters of camera C ₂ is the rotation matrix:

, The rotation matrix [R] composed of the external parameters of the camera C ₂ and the translation vector (Tx, Ty, Tz)

And matrix transformation. Equation (38), the external parameter obtained by the matrix transformation of (39) is intended to camera C ₁ is represented on the world coordinate system assumed. If this calculation is applied to M viewpoint cameras C _i , i = 2, 3,..., M, camera calibration with a multi-view camera can be performed. The above is the basic principle of calibrating with a multi-viewpoint camera using the monocular camera calibration method of the first to third embodiments.

FIG. 16 shows a basic configuration diagram of the fourth embodiment. In FIG. 16, the same parts as those in FIG. 6 are denoted by the same reference numerals. Compared to the first to third embodiments, a switching unit (camera switching unit 110) of a plurality of cameras C _i , i = 1, 2, 3,..., M (cameras C _{1 to} C _M ), and external parameters of each camera Is newly added to the parameter conversion unit (parameter conversion processing unit 111) for converting to the same coordinate system. The fourth embodiment is the same as the monocular camera calibration described in the first to third embodiments except for the processing unit newly added. Therefore, only this difference will be described below.

In the camera switching unit 110, when the cameras C _i , i = 1, 2, 3,..., M are sequentially switched, and the reference object of FIG. An image (reference image) is obtained by shooting the horizontal bar 12 at an appropriate rotation once, and then the horizontal bar 12 is rotated to take an image (calibration image) when x = x ₀ is matched. . The image coordinates of the reference points C ₀ and D ₀ are measured from the standard image from the two images (the standard image and the calibration image) acquired by each camera, and then the calibration image is used in the same manner as in the first to third embodiments. Image coordinates of reference points A, B, O, C ′, D ′ are obtained. Using the image coordinates obtained from the calibration image, the camera calibration processing unit 106 calculates the camera parameters of each camera by the method described in the first to third embodiments. Further, the parameter conversion processing unit 111 obtains the rotation angle Ω _i of each camera with respect to the reference camera (in the examples so far, the camera is the camera C ₁ ) using the expression (36), and the expression ( Using the rotation matrix obtained in accordance with 37), matrix transformations of equations (38) and (39) are performed. After this conversion, external parameters of each camera C _i , i = 2, 3,..., M are extracted.

  As described above, by applying the monocular camera calibration method according to the first to third embodiments to each camera, and converting the calculated camera parameters of each camera into a reference camera coordinate system, a multi-view camera Can be calibrated.

  Further, a part or all of the functions of each means in the camera calibration apparatus of the present embodiment can be configured by a computer program, and the program can be executed using the computer to realize the present invention. Needless to say, the procedure in the camera calibration method can be constituted by a computer program and the program can be executed by the computer, and the program for realizing the function by the computer can be read by a computer-readable recording medium such as an FD. (Floppy (registered trademark) Disk), MO (Magneto-Optical disk), ROM (Read Only Memory), memory card, CD (Compact Disk) -ROM, DVD (Digital Versatile) Disk) -ROM, CD-R, CD-RW, HDD, removable disk, etc., and can be stored and distributed. It is also possible to provide the above program through a network such as the Internet or electronic mail.

101 ... object control unit 102, C ₁ ~C _M ... camera 103 ... image input unit 104 ... reference point adjuster 105 ... reference point measuring unit 106 ... camera calibration processing unit 110 ... camera switching section 111 ... parameter conversion processing unit

Claims (10)

A device for calibrating external information including internal information and posture and position of an image input device from an image acquired using a single eye or a plurality of image input devices,
A predetermined two-dimensional coordinate system is set in an image obtained by photographing a reference object with known reference point geometric information using an image input device to be calibrated, and a two-dimensional coordinate value of the prepared reference point is extracted. Reference point extracting means for
Among the reference points obtained by the reference point extracting means, reference object adjusting means for adjusting the posture or position of the reference object until some reference points satisfy a specific geometric constraint condition;
According to the calculation derived from the perspective projection conditions when the three-dimensional points of all the reference points on the reference object are projected onto the two-dimensional coordinate system on the image with respect to the reference object adjusted by the reference object adjusting means, Camera calibration calculation means for calibrating internal information or external information of the image input device from the geometric information of the reference point and the coordinate value of the reference point obtained on the two-dimensional coordinate;
A camera calibration device comprising: The reference object is composed of a reference object for calibration photography including a vertical bar and a horizontal bar orthogonal to each other,
The reference point extraction unit sets an image coordinate system in the image of the subject acquired using the image input device to be calibrated, and points at a position orthogonal to an arbitrary point on the vertical bar, as well as on the horizontal bar. The reference object is taken as a reference point, the reference object is photographed using the image input device to be calibrated, a predetermined two-dimensional coordinate system is set in the acquired image, and a position orthogonal to the arbitrary point on the vertical bar And 2D coordinate values of any point on the horizontal bar,
The reference object adjusting unit is integrated with the vertical bar until the reference point on the horizontal bar satisfies a specific geometric constraint condition among the reference points obtained by the reference point extracting unit. With means to rotate the horizontal bar around the axis,
The camera calibration apparatus according to claim 1. The reference object is composed of a reference object for calibration photography including a disk that forms a plurality of circles centered on a reference point on a certain base, and a bar perpendicular to a concentric circle extending from the center of the concentric circle,
The reference point extraction unit sets an image coordinate system in the image of the subject acquired using the image input device to be calibrated, and passes through the center of the concentric circle and the center of the concentric circle with an arbitrary point on the vertical bar. Using the image input device to be calibrated as a reference point at the point where the straight line and the concentric circle intersect, the reference object is photographed, a predetermined two-dimensional coordinate system is set in the acquired image, and an arbitrary point on the vertical bar 2D coordinate values of the center of the concentric circle and the point where the concentric circle intersects with a predetermined straight line passing through the center of the concentric circle,
The camera calibration apparatus according to claim 1. The camera configuration calculation means obtains calibration external information from internal information or external information of the image input device obtained by the calibration calculation, recalculates evaluation external information from the calibration external information, and the calibration external information and evaluation external information If the error is larger than the predetermined error, replace the evaluation external information with the calibration external information and recalculate the internal information of the image input device or external information other than the predetermined external information. Repeating the calibration calculation until the error between the information is less than a predetermined error,
The camera calibration apparatus according to claim 1, wherein the camera calibration apparatus is provided. A method of calibrating external information including internal information and posture and position of an image input device from an image acquired using a single eye or a plurality of image input devices,
The reference point extraction unit sets a predetermined two-dimensional coordinate system in an image acquired by photographing a reference object whose reference point geometric information is known using an image input device to be calibrated, and A reference point extracting step for extracting a two-dimensional coordinate value;
A reference object adjusting step in which a reference object adjusting means adjusts the posture or position of the reference object until some of the reference points satisfy a specific geometric constraint condition among the reference points obtained in the reference point extracting step; ,
From the perspective projection conditions when the camera calibration calculation means projects the three-dimensional points of all the reference points on the reference object to the two-dimensional coordinate system on the image with respect to the reference object adjusted in the reference object adjusting step. A camera calibration calculation step for calibrating internal information or external information of the image input device from the geometric information of the reference point and the coordinate value of the reference point obtained on the two-dimensional coordinates in accordance with the derived calculation;
A camera calibration method comprising: The reference object is composed of a reference object for calibration photography including a vertical bar and a horizontal bar orthogonal to each other,
In the reference point extraction step, an image coordinate system is set in the image of the subject acquired using the image input device to be calibrated, a point at a position orthogonal to an arbitrary point on the vertical bar, and the horizontal bar The reference object is taken as a reference point, the reference object is photographed using the image input device to be calibrated, a predetermined two-dimensional coordinate system is set in the acquired image, and a position orthogonal to the arbitrary point on the vertical bar And 2D coordinate values of any point on the horizontal bar,
In the reference object adjustment step, the vertical bar integrated with the vertical bar until the reference point on the horizontal bar satisfies a specific geometric constraint condition among the reference points obtained in the reference point extraction step. Rotate the horizontal bar around the axis,
The camera calibration method according to claim 5. The reference object is composed of a reference object for calibration photography including a disk that forms a plurality of circles centered on a reference point on a certain base, and a bar perpendicular to a concentric circle extending from the center of the concentric circle,
In the reference point extraction step, an image coordinate system is set in the image of the subject acquired using the image input device to be calibrated, and a predetermined point that passes through the center of the concentric circle with the arbitrary point on the vertical bar and the center of the concentric circle. Using the image input device to be calibrated as a reference point at the point where the straight line and the concentric circle intersect, the reference object is photographed, a predetermined two-dimensional coordinate system is set in the acquired image, and an arbitrary point on the vertical bar 2D coordinate values of the center of the concentric circle and the point where the concentric circle intersects with a predetermined straight line passing through the center of the concentric circle,
The camera calibration method according to claim 5. The camera configuration calculation step obtains calibration external information from internal information or external information of the image input device obtained by the calibration calculation, recalculates evaluation external information from the calibration external information, and the calibration external information and evaluation external information. If the error is larger than the predetermined error, replace the evaluation external information with the calibration external information and recalculate the internal information of the image input device or external information other than the predetermined external information. Repeating the calibration calculation until the error between the information is less than a predetermined error,
The camera calibration method according to claim 5, wherein the camera calibration method is performed.   A camera calibration program for causing a computer to execute each step according to any one of claims 5 to 8.   A computer-readable recording medium in which the camera calibration program according to claim 9 is recorded.

Images