JP2016213783A - Camera calibration apparatus, camera calibration method, camera calibration program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a camera calibration apparatus, a camera calibration method, a camera calibration program, and a recording medium capable of realizing calibration of a camera model even when there is an unknown refractive layer between a camera and a subject.SOLUTION: A camera calibration apparatus comprises: a correspondence relationship acquiring unit that acquires information on a correspondence relationship between a pixel and a position on a calibration object for each attitude of a calibration object from an image group obtained by photographing the calibration object with different attitudes; an attitude estimation unit for estimating the attitude of the calibration object to output attitude information; and a ray information acquisition unit that acquires ray information indicating a correspondence relationship between the pixel and a light beam using the information on the correspondence relationship and the attitude information. The attitude estimation unit estimates the attitude of the calibration object using a fact that the position on the calibration object sampled with the same pixel among the plurality of images exists on a straight line in the photographing space.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a camera calibration device, a camera calibration method, a camera calibration program, and a recording medium when a refractive layer exists between a camera and a subject.

  2. Description of the Related Art Conventionally, a technique for extracting, for example, the shape of a subject and the positional relationship between the subject and the camera as three-dimensional information from an image taken with a camera is known. Such a technique is very important in a robot vision, a system that provides augmented reality, and the like. In order to extract such three-dimensional information, it is necessary to acquire information related to the correspondence between each pixel on the image and light rays in the real space and information related to the positional relationship between a plurality of cameras. And the process which acquires such information is called camera calibration. In particular, the process of acquiring information about the correspondence between each pixel on the image and the light beam in the real space is called internal calibration, and the process of acquiring information about the positional relationship between a plurality of cameras is called external calibration. being called.

  In general camera calibration, it is assumed that a mapping between an object and an image plane can be expressed by a perspective camera model such as a pinhole camera model. In such a perspective projection camera model, it is assumed that the light rays from the object converge while moving straight toward the projection center of the camera. Therefore, when there is a refractive layer between the camera and the object, camera calibration is realized by modeling the refraction of the refractive layer based on Snell's law. In order to model refraction, information on the refractive index, position, and shape of the refractive layer is required. However, since it is generally difficult to obtain such information in advance, there is a problem that refraction modeling cannot be performed. Even if such information is obtained, if the shape of the refractive layer is complicated, there is a problem that the modeling process of refraction is complicated.

As a camera model different from the perspective projection camera model, there is a Raxel (ray-pixel) camera model proposed in Non-Patent Document 1. In the Raxel camera model, each pixel on an image is directly associated with a light beam in a three-dimensional space. Therefore, it is possible to obtain the correspondence between pixels and light rays in camera calibration without modeling the refraction of light rays in the refraction layer.
For example, in the technique described in Non-Patent Document 2, a calibration object such as a checkerboard is installed in a space to be imaged including a refractive layer between the camera and a pixel to a point on the calibration object plane. Calibration is realized by obtaining an approximation function that represents the mapping of.

Michael D. Grossbarg, Shree K. Nayar, “The Raxel Imaging Model and Ray-Based Calibration,” International Journal of Computer Vision, 61 (2), pp. 119-137, 2005. Tomohiko Yano, Shohei Nobuhara, Takashi Matsuyama, “3D Shape from Silhouettes in Water for Online Novel-View Synthesis,” 16th Image Recognition and Understanding Symposium (MIRU), 2013.

  However, the technique described in Non-Patent Document 2 has a problem that the shape of the refracting layer is limited due to the problem of accuracy of approximation with an approximation function, and it is difficult to apply to any refracting layer. Further, in the calibration, information regarding the posture of the calibration object with respect to the camera is required. And in order to acquire information about the posture, it is necessary to photograph a calibration object installed in a space including a refractive layer in the path, and simultaneously image another calibration object installed in a space not including the refractive layer. is there. That is, the method described in Non-Patent Document 2 has a problem that a space that is originally targeted can be captured only by a part of the camera, and a high-definition image cannot be acquired. Further, the technique described in Non-Patent Document 2 has a problem that it can be used only in a situation where both a space including a refractive layer and a space not including a refractive layer can be photographed.

  In view of the above circumstances, the present invention provides a camera calibration that can realize camera model calibration even when an unknown refractive layer or a complicated refractive layer exists between the camera and the subject. An object is to provide an apparatus, a camera calibration method, a camera calibration program, and a recording medium.

  One aspect of the present invention is a camera calibration device for associating each pixel of an image obtained by photographing a subject through a refractive layer with a light beam from the subject, the posture of the subject as the subject Acquired by the correspondence acquisition unit for acquiring information on the correspondence between the pixel and the position on the calibration object for each posture of the calibration object from the group of images obtained by photographing different calibration objects. Using the information on the correspondence relationship, the posture estimation unit that estimates the posture of the calibration object and outputs posture information, the information on the correspondence relationship acquired by the correspondence relationship acquisition unit, and the posture estimation unit A light ray information acquisition unit that acquires light ray information indicating a correspondence relationship between pixels and light rays using the estimated posture information, and the posture estimation unit includes a plurality of images. Position on the calibration object sampled in one pixel, a camera calibration device for estimating the pose of the calibration object using that exist on a straight line in the imaging space.

  In one aspect of the present invention, the posture estimation unit obtains two or more line segments for one pixel from a position on the calibration object sampled by the same pixel, and a direction vector corresponding to the line segment. The posture of the calibration object is estimated so as to minimize the outer product.

  In one embodiment of the present invention, two or more line segments are obtained for one pixel from the position on the calibration object sampled by the same pixel using the information on the correspondence and the posture information. And a calibration evaluation unit for obtaining a calibration evaluation value based on the outer product between the direction vectors corresponding to the line segment.

  One aspect of the present invention is an object for selecting three or more image groups in which calibration objects having different postures with respect to the camera are photographed with a small difference in distance from the camera, from the image groups in which the calibration objects having different postures are photographed. An image selection unit is further included, and the correspondence relationship acquisition unit acquires information regarding the correspondence relationship between the pixel and the position on the calibration object for each posture of the calibration object from the image group selected by the target image selection unit.

  One aspect of the present invention is a distance estimation unit that estimates a distance from the camera to the calibration object according to a ratio of a region where the calibration object is captured with respect to an image obtained by capturing the calibration object; A degree-of-similarity estimation unit that estimates the degree of similarity of the posture of the calibration object with respect to the camera according to the difference in the shape of the area where the calibration object is imaged with respect to the image group obtained by imaging the calibration object; The target image selection unit is a calibration having a small difference in distance from the camera and a different posture with respect to the camera based on the distance estimated by the distance estimation unit and the similarity estimated by the similarity estimation unit. A group of three or more images in which an object is photographed is selected.

  One aspect of the present invention is a camera calibration method for associating each pixel of an image obtained by photographing a subject through a refraction layer with a light beam from the subject, the posture of the subject as the subject Acquired in the correspondence acquisition step and the correspondence acquisition step for acquiring information related to the correspondence between the pixel and the position on the calibration object for each posture of the calibration object from the group of images obtained by photographing different calibration objects. Using the information on the correspondence relationship, the posture estimation step of estimating the posture of the calibration object and outputting posture information, the information on the correspondence relationship acquired in the correspondence relationship acquisition step, and the posture estimation step A light ray information obtaining step for obtaining light ray information indicating a correspondence relationship between a pixel and a light ray using the estimated posture information; The posture estimation step is a camera calibration method that estimates the posture of the calibration object using the fact that the position on the calibration object sampled by the same pixel between a plurality of images exists in a straight line in the imaging space. .

  One aspect of the present invention is a camera calibration program for causing a computer to execute the camera calibration method.

  One aspect of the present invention is a recording medium on which the camera calibration program is recorded.

  According to the present invention, calibration of a camera model can be realized even when an unknown refraction layer or a complicated refraction layer exists between a camera and a subject.

It is a figure which shows the imaging | photography space which performs camera calibration in 1st Embodiment. It is a block diagram which shows the structure of the 1st calibration apparatus in 1st Embodiment. It is a flowchart which shows operation | movement of the 1st calibration apparatus 10 in 1st Embodiment. It is a figure which shows the imaging | photography space which performs camera calibration in 2nd Embodiment. It is a block diagram which shows the structure of the 2nd calibration apparatus in 2nd Embodiment. It is a flowchart which shows operation | movement of the 2nd calibration apparatus 20 in 2nd Embodiment. It is a figure which shows the structural example of the hardware of the 1st calibration apparatus 10 in 1st Embodiment, or the 2nd calibration apparatus 20 in 2nd Embodiment.

Hereinafter, a camera calibration apparatus according to an embodiment of the present invention will be described with reference to the drawings.
(First embodiment)
First, as a first embodiment, a mapping process (internal calibration) between a light beam in an imaging space and a pixel of an image captured by the camera will be described as calibration in one camera. FIG. 1 is a diagram illustrating an imaging space in which camera calibration is performed in the first embodiment. As shown in FIG. 1, an unknown refractive layer 2 exists between the camera 1 and the imaging space, and calibration objects 3 and 3 'are installed in the imaging space. Here, the calibration object 3 ′ is the same object as the calibration object 3, and shows a state after the posture of the calibration object 3 is changed. Note that any object may be used as the calibration object 3, but it is necessary to use an object that can obtain a correspondence between a pixel and a position on the calibration object 3, which will be described later. In the present embodiment, a flat display or the like that displays a gray code is used as the calibration object 3.

  FIG. 2 is a block diagram illustrating a configuration of the first calibration apparatus according to the first embodiment. As illustrated in FIG. 2, the first calibration device 10 includes an image input unit 11, a correspondence relationship acquisition unit 12, a posture estimation unit 13, and a light ray information output unit 14. Although not shown in FIG. 1, the first calibration device 10 is a computer terminal that can input an image captured by the camera 1 and performs image processing on the input image.

  The image input unit 11 inputs an image (group) obtained by photographing the calibration object 3. The correspondence relationship acquisition unit 12 acquires information that associates pixels in the image input by the image input unit 11 with positions on the calibration object 3. The posture estimation unit 13 estimates a relative posture between the plurality of calibration objects 3 and 3 ′. The light ray information output unit 14 associates light rays with pixels.

Next, the operation of the first calibration device 10 shown in FIG. 2 will be described. FIG. 3 is a flowchart illustrating the operation of the first calibration device 10 according to the first embodiment.
First, the image input unit 11 inputs a group of images {I _{n, m} | n = 0,..., N−1, m = 0,... M _n −1} taken while changing the posture of the calibration object 3. (Step S101). Here, N is the number of postures of the calibration object 3 included in the input image group, and M _n is the number of images when the calibration object 3 is in the nth posture. Here, N is 3 or more, and M _n is the number of images necessary for associating the pixel with the position on the calibration object 3 in the nth posture. M _n depends on the type of the calibration object 3 and the size of the imaging space.

  Next, the correspondence acquisition unit 12 associates a pixel with a position on the calibration object 3 for each posture of the calibration object 3 (step S102). Specifically, the coordinate value (pixel position) on the image plane is associated with the coordinate value in the coordinate system defined on the calibration object 3. The method here differs depending on the type of the calibration object 3.

  For example, a case will be described in which the calibration object 3 is configured by a flat display and the calibration object 3 displays a series of gray code patterns in which coordinate values are encoded. The image input unit 11 inputs an image group obtained by photographing the calibration object 3 that displays a gray code pattern. The correspondence acquisition unit 12 can obtain a coordinate value on the calibration object 3 corresponding to the pixel by obtaining a sequence of signals photographed at each pixel from the input image group and decoding it. Regarding the correspondence between the pixel using the gray code and the position on the plane of the calibration object 3, details are described in Non-Patent Document 1, for example.

Here, the coordinate value of the point P _{k, m} on the calibration object 3 in the nth posture with respect to the pixel k, obtained as a result of step S102, is expressed as p _{k, n} (k = 1,..., NumPixs, n = 1, ..., N). Here, numPixs is the number of pixels for which a correspondence relationship with a light ray is obtained.

After associating each pixel with the coordinate value on the calibration object 3, the posture estimation unit 13 estimates information regarding the posture of the calibration object 3 (step S103). How the posture estimation unit 13 represents information related to the posture of the calibration object 3 is not limited. For example, a calibration object 3 ′ having a different posture with a certain posture of the calibration object 3 as a reference may be expressed in a format using a rotation matrix R _n and a translation vector t _n for converting to a reference posture. In the following description, the posture estimation unit 13 expresses a posture using a rotation matrix and a translation vector based on the posture with respect to n = 1.

In the processing for obtaining the rotation matrix R _n and the translation vector t _n , the property that the light beam sampled by each pixel has been traveling straight through the imaging space is used. That is, it is used that points on the calibration object 3 sampled at the pixel k exist on the same straight line in the imaging space. When the rotation matrix R _n and the translation vector t _n are used, the coordinate value p _{k, n} on the calibration object 3 of the nth posture with respect to the pixel k is expressed as the coordinate for the first posture as shown in (Equation 1) below. It can be represented by a coordinate value P _{k, n} ^[1] in a system (hereinafter referred to as a reference coordinate system).

At this time, the direction vector of the line segment P _{k, l} P _{k, n in} the reference coordinate system is given by the following (formula 2).

When the points on the calibration object 3 sampled by the pixel k exist on the same straight line in the imaging space, the outer product of the direction vectors of the posture i and the posture j is zero. That is, the rotation upper row where the outer product represented by the following (Equation 3) is 0 and the translation vector are obtained.

Here, x indicates an outer product, M _k is a matrix composed only of coordinate values p _{k, n} , and h is a vector composed of a rotation matrix and a translation vector. Here, in order to satisfy the condition that exists on the same straight line, it is necessary to have three or more points. That is, it is necessary that N ≧ 3. Here, the description will be made assuming that N = 3, but it can be easily inferred that the expansion is possible when N> 3.

姿勢推定部１３は、この連立方程式を解くことで回転行列と並進ベクトルを求めて、較正物体３の姿勢を推定する。なお、（式４）の連立方程式は、どのような方法を用いて解いても構わない。 Since the above condition is obtained for each pixel, the simultaneous equations shown in the following (Equation 4) can be obtained by collecting all the conditions.

The posture estimation unit 13 calculates the rotation matrix and the translation vector by solving the simultaneous equations, and estimates the posture of the calibration object 3. Note that the simultaneous equations of (Equation 4) may be solved using any method.

There is also a method for obtaining the posture of the calibration object 3 without solving the simultaneous equations shown in (Equation 4). For example, the posture of the calibration object 3 may be estimated by solving an optimization problem that minimizes the size of the outer product. In other words, a vector representing the posture of the calibration object 3 is represented by h ^ (actual hat (^) is on h), and a line segment (two or more lines) constituted by points on the calibration object 3 sampled by the pixel k. When the outer product of the line segment) is represented by g _k , the vector h indicating the posture of the calibration object 3 can be obtained by solving the minimization problem shown in the following (Equation 5).

Further, instead of G using the outer product g _k , a vector h indicating the posture of the calibration object 3 is obtained by minimizing G ′ using the outer product norm g ′ _k using the following (Equation 6). You can ask for it. By using the norm of the vector h indicating the posture of the calibration object 3, not all components of the outer product are minimized, but all components are efficiently minimized. As a result, the amount of calculation required for minimization can be reduced, and an appropriate vector h indicating the posture of the calibration object 3 can be obtained.

Furthermore, the posture estimation unit 13 may estimate the posture of the calibration object 3 using a condition in which the direction vectors match without using an outer product. For example, assuming that the unit vector d _k of the difference vector of the direction vector of the point on the calibration object 3 sampled by the pixel k is to solve the minimization problem using the following (Equation 7) and (Equation 8): A vector h indicating the posture of the calibration object 3 can be obtained.

  After estimating information related to the posture of the calibration object 3, the light ray information output unit 14 generates light ray information in which light rays sampled by the pixels are associated with each pixel (step S104). The generated light ray information becomes an output of the first calibration device 10. Note that the association may be expressed in any form. The light ray information in the present embodiment is represented using, for example, a look-up table that returns an expression of a straight line representing a light ray with each pixel as an index.

  Note that after the process of estimating the posture of the calibration object 3 (step S103) is completed, an evaluation value indicating the accuracy of calibration, that is, the accuracy of estimation of the posture of the calibration object 3, may be obtained. This evaluation value may be any evaluation value as long as it can indicate whether or not the posture of the calibration object 3 has been correctly estimated. Specific example 1 of the evaluation value is obtained by obtaining the sum, maximum value, and variance of the norm of the outer product of the line segment constituted by the points on the calibration object 3 obtained for each pixel, and setting the evaluation value. Specific example 2 of the evaluation value is an evaluation value obtained by calculating the sum, maximum value, and variance of the norm of the difference vector of the direction vector formed by the points on the calibration object 3 obtained for each pixel. is there. Furthermore, the specific example 3 of the evaluation value is obtained by calculating the sum, the maximum value, and the variance of the distance between the light ray associated with each pixel and the point on the calibration object 3 and obtaining the evaluation value. Furthermore, specific example 4 of the evaluation value is the evaluation value obtained by calculating the sum, maximum value, and variance of the distance between the ray associated with each pixel and the point on the calibration object 3 on the calibration object 3 plane. Is.

  When the obtained evaluation value represents that the predetermined accuracy is not satisfied, the first calibration device 10 adds an image obtained by photographing the calibration object 3 ′ of another posture, and uses the updated image group. The process of estimating the posture of the calibration object 3 ′ (step S103) may be performed again.

  In addition, when adding an image, you may add the image | photographed image of the calibration object 3 of what kind of attitude | position. For example, it is possible to add an image that is the same calibration object 3 and the size of the calibration object 3 on the image is significantly different from that of the already input image group. An image having the same size but different shape of the calibration object 3 on the image may be added. The former means that the distance from the camera that captured the image to the calibration object 3 is different from that of the already input image. By inputting such an image, it is possible to perform calibration in consideration of the straightness of the light beam through a longer path. On the other hand, the latter image can be more robustly calibrated in consideration of the straightness of light rays in a narrower space.

  In addition, when adding a new image, it is not necessary to store all the image groups acquired in the past, but to replace the images so that the total number of images does not change. By keeping the total number of images constant, it is possible to keep the necessary memory amount and calculation amount constant.

  In the above description, the calibration process is performed by using all the input images from the image input unit 11, but only a part of the input images may be selected and used. For example, the image input unit 11 may select an image group in which the distance from the camera 1 to the calibration object 3 changes greatly based on the input image group, or the camera 1 and the calibration object 3 may be selected. It is also possible to select an image group in which the distance up to the same level and the posture of the calibration object 3 change greatly.

  Note that in order to accurately obtain information on the distance from the camera 1 to the calibration object 3 and the posture of the calibration object 3, it is necessary to perform calibration. Such information may be given separately or estimated from an image. For example, the size of the calibration object on the image may be used to estimate the distance from the camera to the calibration object, and the similarity of the posture of the calibration object can be determined by comparing the shape of the calibration object on the image. It may be estimated. For example, the correspondence acquisition unit 12 uses the camera 1 as an estimation method of the distance from the camera 1 to the calibration object 3 according to the ratio of the area where the calibration object 3 is captured with respect to the image obtained by capturing the calibration object 3. To the calibration object 3 is estimated. In addition, the correspondence acquisition unit 12 estimates the similarity of the posture of the calibration object 3 with respect to the camera 1 in accordance with the difference in the shape of the area where the calibration object 3 is photographed with respect to the image group obtained by photographing the calibration object 3. To do. Next, the correspondence relationship acquisition unit 12 selects three or more image groups obtained by photographing the calibration object 3 with a small difference in distance from the camera 1 and a different posture with respect to the camera 1 based on the estimated distance and similarity. To do.

(Second Embodiment)
Next, as a second embodiment, calibration for obtaining a relationship between a plurality of cameras will be described. Specifically, calibration (external calibration) that estimates a relative positional relationship between two cameras and obtains a description using a common coordinate system between the cameras for the light rays mapped to each pixel. Will be described.

  FIG. 4 is a diagram illustrating an imaging space in which camera calibration is performed in the second embodiment. As shown in FIG. 4, an unknown refractive layer 2 exists between the plurality of cameras 1 a to 1 c and the calibration objects 3 a, 3 a ′, 3 b, and 3 b ′ that are subjects. It is assumed that the calibration objects 3a, 3a ', 3b, 3b' are installed at different times and that only one calibration object exists at a time. For example, the calibration object 3a 'is obtained by changing the posture of the calibration object 3a. Similarly, the calibration object 3b' is obtained by changing the posture of the calibration object 3b. The calibration object 3a and the calibration object 3b may be different objects or the same object. Further, any object may be used as the calibration objects 3a and 3b, but it is necessary to use an object that can obtain the correspondence between the pixel and the position on the calibration object. As the calibration object 3a and the calibration object 3b in the present embodiment, a flat display or the like that displays a gray code is used.

  The following method can be considered as the simplest method for obtaining the mapping of light rays and pixels in the description using a common coordinate system for the plurality of cameras 1a to 1c. First, each of the plurality of cameras 1a to 1c has the configuration of the first calibration device 10 shown in the first embodiment. And it is a method which inputs the image group of the same calibration object with respect to several cameras 1a-1c, describes a light ray on the basis of the attitude | position of the calibration object of the same image, and calculates | requires mapping with a pixel. In the present embodiment, a description will be given of a method for obtaining a mapping between rays and pixels based on a description using a common coordinate system even between cameras for which such a method cannot be used.

  As an imaging space to which the above simplest method cannot be applied, for example, there is an imaging space shown in FIG. In the shooting space of FIG. 4, the camera 1a and the camera 1b can obtain the ray description and the mapping from the pixel using the common calibration object 3a, and the camera 1b and the camera 1c have the common calibration object 3b. Can be used to determine ray descriptions and pixel mappings. However, the cameras 1a and 1c are in a situation where a common calibration object cannot be photographed. In the present embodiment, a method for obtaining ray-pixel mapping by description using a coordinate system common to the three cameras 1a to 1c in the imaging space shown in FIG. 4 will be described.

FIG. 5 is a block diagram illustrating a configuration of the second calibration device according to the second embodiment. As shown in FIG. 5, the second calibration device 20 includes a light ray information input unit 21, a light ray information storage unit 22, a camera position information estimation unit 23, and a light ray description conversion unit 24.
The light ray information input unit 21 inputs light ray information (hereinafter referred to as light ray information) associated with each pixel of the camera. The light ray information storage unit 22 stores light ray information associated with each input camera pixel. The camera position information estimation unit 23 estimates the positional relationship of the camera using the light ray information. The light ray description conversion unit 24 obtains a description expressed in a common coordinate system for the light rays based on the obtained positional relationship of the cameras.

Next, the operation of the second calibration device 20 in the second embodiment will be described. FIG. 6 is a flowchart showing the operation of the second calibration device 20 in the second embodiment.
First, the light ray information input unit 21 inputs light ray information associated with each pixel of the camera to the camera pair (camera 1a and camera 1b or camera 1b and camera 1c), and stores the light ray information in the light ray information storage unit 22. (Step S201). Here, the ray information input for each camera pair is described in a common coordinate system.

  Note that any method may be used for obtaining the light ray information described in a common coordinate system for each camera pair. For example, by using the first calibration apparatus 10 shown in the first embodiment, an image group of the same calibration object is input, and pixel and light ray mapping in each camera is performed with reference to the posture of the calibration object of the same image. By obtaining, it is possible to obtain ray information described in a common coordinate system.

  That is, in the present embodiment, the light ray information input unit 21 inputs light ray information for mapping light rays and pixels described in a common coordinate system for the camera 1a and the camera 1b, and the camera 1b and the camera 1c. Are input with light ray information for mapping light rays and pixels described in a common coordinate system.

Next, the camera position information estimation unit 23 obtains a set of pixels {p _k | k = 1,..., NumCPixs} to which a description of light rays expressed in a plurality of different coordinate systems is given (step S202). This set of pixels is represented by a set of pixels having a mapping to the light beam l ₁₂ (k) represented in the coordinate system common to the camera 1a and a coordinate system common to the camera 1c. And a set of pixels having a mapping to ray l ₂₃ (k). Here cameras 1a and common rays represented by a coordinate system l ₁₂ corresponding to the pixel p _{k (k),} represents the light ray represented by the camera 1c common coordinate system l _{23 (k).}

Next, when a set of pixels having a ray description in an ordinary coordinate system is obtained, the camera position information estimation unit 23 estimates the relationship between the coordinate systems (step S203). Specifically, a rotation matrix R and a translation vector t that satisfy (Equation 9) below are obtained. In (Equation 9), the coordinate system to which the light beam of the camera 1a is given is used as a reference, but the coordinate system to which the light beam of the camera 1c is given may be used as a reference.

The rotation matrix R and the translation vector t may be obtained by any method. For example, the reference (B. Kamgar-Parsi and B. Kamgar-Parsi, “Algorithms for matching 3D line sets,” TPAMI, vol.26, No. 5, pp.582-593, May 2004). At this time, it may also be determined translation vector and rotation matrix that satisfies the above equation (9) for all p _k. However, if the number of pixels is large, the amount of calculation becomes enormous, and if incorrect ray information is included, the accuracy of estimation decreases due to the influence. Therefore, in step S203, estimation may be performed using only some pixels.

When estimation is performed using only some pixels, estimation accuracy of the estimated rotation matrix and translation vector is obtained. Any method may be used to express the estimation accuracy. For example, using the estimated rotation matrix and translation vector, the point q _i (k) on the calibration object 3b expressed in the common coordinate system with the camera 1c is converted by the following (Equation 10). The sum, maximum value, or dispersion of the distance between the point q ′ _i (k) and the light beam l ₁₂ (k) may be used. Further, the sum, maximum value, and dispersion of the distance between q ′ _i (k) and the light beam l ₁₂ (k) on the calibration object may be used.

なお、得られた推定精度が予め定めた閾値を超えていない場合は、別の画素の集合を用いて回転行列と並進ベクトルの推定をやり直す。 Further, a point s ′ _j (k) obtained by converting the point s _j (k) on the calibration object 3a expressed in the common coordinate system with the camera 1a by the following (formula 11) and the light beam l ₂₃ ( k) may be used as the sum, maximum value, or variance of the distances, or using the sum, maximum value, or variance of the distances between the s ′ _j (k) and the ray l ₂₃ (k) on the calibration object. It doesn't matter.

When the obtained estimation accuracy does not exceed a predetermined threshold value, the rotation matrix and the translation vector are estimated again using another set of pixels.

Next, when the estimation of the rotation matrix and the translation vector is completed, the ray description conversion unit 24 converts the ray description using the obtained rotation matrix and the translation vector, so that the ray and the pixel in the common coordinate system are converted. Is generated and output (step S204). Specifically, in accordance with the following (Equation 12), for the light beam group {l ₂₃ (m)} expressed in the coordinate system common to the camera 1c, the light beam group expressed in the coordinate system common to the camera 1a. Conversion is performed by obtaining {l ₁₂ (m)}.

(Hardware configuration example)
FIG. 7 is a diagram illustrating a hardware configuration example of the first calibration device 10 according to the first embodiment or the second calibration device 20 according to the second embodiment.
As shown in FIG. 7, it is a block diagram showing a hardware configuration when the first calibration device 10 or the second calibration device 20 is configured by a computer. The first calibration device 10 or the second calibration device 20 includes a CPU (Central Processing Unit) 100 that executes a program, a memory 101 such as a RAM that stores programs and data accessed by the CPU 100, a camera, and the like. An image input unit 102 that inputs an image signal to be processed, a program storage device 103 that stores a camera calibration program 105 that is a software program that causes the CPU 100 to execute camera calibration processing, and the CPU 100 is loaded into the memory 101. A light beam information output unit 104 that outputs the correspondence between pixels and light beams generated by executing the camera calibration program 105 is connected by a bus.

  Note that the image input unit 102 may be configured by a storage unit that stores an image signal by a disk device or the like and inputs the image signal by reading the stored image signal. The light beam information output unit 104 may be configured by a storage unit that stores light beam information indicating the correspondence between pixels and light beams by a disk device or the like. Also, it includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and functions as a device having camera calibration by executing a camera calibration program. Note that all or part of each function of camera calibration may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). The camera calibration program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. The camera calibration program may be transmitted via a telecommunication line.

  As described above, the first calibration device 10 in the first embodiment or the second calibration device 20 in the second embodiment can be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As described above, the camera and the subject are generated by generating the correspondence relationship between the pixel and the light from the group of images obtained by photographing the same calibration object in a plurality of postures using the fact that the light travels straight in the photographing space. Even when an arbitrary refractive layer (an unknown refractive layer or a complicated-shaped refractive layer) is included between the two, a camera calibration can be realized. In addition, the camera calibration can be realized even in a situation where the subject cannot be photographed through a space that does not include a refractive layer between the camera and the subject.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The camera calibration device, the camera calibration method, the camera calibration program, and the recording medium of the present invention can associate pixels with light rays even when there is an unknown refractive layer between the camera and the subject. Applicable to essential applications.

DESCRIPTION OF SYMBOLS 1, 1a, 1b, 1c ... Camera, 2 ... Refraction layer, 3, 3 ', 3a, 3a', 3b, 3b '... Calibration object, 10 ... 1st calibration apparatus, 11 ... Image input part, 12 ... Correspondence Relationship acquisition unit, 13 ... Posture estimation unit, 14 ... Ray information output unit, 20 ... Second calibration device, 21 ... Ray information input unit, 22 ... Ray information storage unit, 23 ... Camera position information estimation unit, 24 ... Ray Description converter

Claims (8)

A camera calibration device for associating each pixel of an image obtained by photographing a subject through a refractive layer with a light beam from the subject,
A correspondence acquisition unit that acquires information on a correspondence between the pixel and the position on the calibration object for each posture of the calibration object from an image group obtained by photographing calibration objects having different postures as the subject;
A posture estimation unit that estimates posture of the calibration object and outputs posture information using information on the correspondence acquired by the correspondence acquisition unit;
A light ray information acquisition unit that acquires light ray information indicating a correspondence relationship between a pixel and a light beam using the information related to the correspondence relationship acquired by the correspondence relationship acquisition unit and the posture information estimated by the posture estimation unit. And
The posture estimation unit is a camera calibration device that estimates the posture of the calibration object by using a position on the calibration object sampled by the same pixel between a plurality of images on a straight line.   The posture estimation unit obtains two or more line segments for one pixel from positions on the calibration object sampled by the same pixel, and minimizes an outer product between direction vectors corresponding to the line segment. The camera calibration apparatus according to claim 1, wherein the posture of the calibration object is estimated as described above.   Using the information on the correspondence relationship and the posture information, two or more line segments are obtained for one pixel from the position on the calibration object sampled by the same pixel, and the line segment corresponds to the line segment. The camera calibration apparatus according to claim 1, further comprising a calibration evaluation unit that obtains a calibration evaluation value based on an outer product between the direction vectors. A target image selection unit that selects three or more image groups in which calibration objects having different postures with respect to the camera are photographed with a small difference in distance from the camera from the image groups in which the calibration objects having different postures are captured;
4. The information processing apparatus according to claim 1, wherein the correspondence relationship acquisition unit acquires information about a correspondence relationship between a pixel and a position on the calibration object for each posture of the calibration object from the image group selected by the target image selection unit. The camera calibration device according to claim 1. A distance estimation unit that estimates a distance from the camera to the calibration object according to a ratio of a region where the calibration object is captured with respect to an image of the calibration object;
A similarity estimation unit for estimating the degree of similarity of the posture of the calibration object with respect to the camera according to the difference in the shape of the area where the calibration object is captured with respect to the image group obtained by capturing the calibration object; ,
The target image selection unit is a calibration having a small difference in distance from the camera and a different posture with respect to the camera based on the distance estimated by the distance estimation unit and the similarity estimated by the similarity estimation unit. The camera calibration device according to claim 4, wherein three or more image groups obtained by photographing an object are selected. A camera calibration method for associating each pixel of an image obtained by photographing a subject through a refraction layer with a light beam from the subject,
A correspondence acquisition step for acquiring information on a correspondence relationship between the pixel and the position on the calibration object for each posture of the calibration object from an image group obtained by photographing calibration objects having different postures as the subject;
A posture estimation step of estimating posture of the calibration object and outputting posture information using information on the correspondence acquired in the correspondence relationship acquisition step;
A ray information acquisition step of obtaining ray information indicating a correspondence relationship between a pixel and a ray by using the information related to the correspondence acquired in the correspondence acquisition step and the posture information estimated in the posture estimation step; With
The posture estimation step is a camera calibration method for estimating a posture of the calibration object by using a position on the calibration object sampled with the same pixel between a plurality of images on a straight line.   A camera calibration program for causing a computer to execute the camera calibration method according to claim 6.   A recording medium on which the camera calibration program according to claim 7 is recorded.

Images