JP2017011537A - Camera calibration method, camera calibration device and camera calibration program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a camera calibration method, a camera calibration device and a camera calibration program capable of achieving calibration of a camera model, even if an unknown refractive layer exists between a camera and a subject.SOLUTION: A camera calibration method has: a correspondence acquisition step of acquiring information about correspondence of a pixel and the position on a calibration object, for each posture thereof, from an image group capturing calibration objects of different postures as subjects; intersection line detection step of detecting a pixel group corresponding to an intersection line of the calibration objects of different postures in a photographic space; a posture estimation step of estimating the posture of the calibration object by using information about correspondence in the intersection line, and outputting posture information; and a light ray information acquisition step of acquiring light ray information indicating the correspondence of the pixel and light rays in the photographic space, by using the information about the correspondence, and the posture information.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a camera calibration method, a camera calibration device, and a camera calibration program when a refractive layer exists between a camera and a subject.

  2. Description of the Related Art Conventionally, a technique for extracting, for example, the shape of a subject and the positional relationship between the subject and the camera as three-dimensional information from an image taken with a camera is known. Such a technique is very important in a robot vision, a system that provides augmented reality, and the like. In order to extract such three-dimensional information, it is necessary to acquire information related to the correspondence between each pixel on the image and light rays in the real space and information related to the positional relationship between a plurality of cameras. And the process which acquires such information is called camera calibration. In particular, the process of acquiring information about the correspondence between each pixel on the image and the light beam in the real space is called internal calibration, and the process of acquiring information about the positional relationship between a plurality of cameras is called external calibration. being called.

  In general camera calibration, it is assumed that a mapping between an object and an image plane can be expressed by a perspective camera model such as a pinhole camera model. In such a perspective projection camera model, it is assumed that the light rays from the object converge while moving straight toward the projection center of the camera. Therefore, when there is a refractive layer between the camera and the object, camera calibration is realized by modeling the refraction of the refractive layer based on Snell's law. In order to model refraction, information on the refractive index, position, and shape of the refractive layer is required. However, since it is generally difficult to obtain such information in advance, there is a problem that refraction modeling cannot be performed. Even if such information is obtained, if the shape of the refractive layer is complicated, there is a problem that the modeling process of refraction is complicated.

As a camera model different from the perspective projection camera model, there is a Raxel (ray-pixel) camera model proposed in Non-Patent Document 1. In the Raxel camera model, each pixel on an image is directly associated with a light beam in a three-dimensional space. Therefore, the correspondence between the pixel and the light beam is obtained in the camera calibration without modeling the refraction of the light beam in the refractive layer.
For example, in the technique described in Non-Patent Document 2, a calibration object such as a checkerboard is installed in a space to be imaged including a refractive layer between the camera and a pixel to a point on the calibration object plane. Calibration is realized by obtaining an approximation function that represents the mapping of.

Michael D. Grossbarg, Shree K. Nayar, “The Raxel Imaging Model and Ray-Based Calibration,” International Journal of Computer Vision, 61 (2), pp. 119-137, 2005. Tomohiko Yano, Shohei Nobuhara, Takashi Matsuyama, “3D Shape from Silhouettes in Water for Online Novel-View Synthesis,” 16th Image Recognition and Understanding Symposium (MIRU), 2013.

  However, the method described in Non-Patent Document 2 has a problem that the shape of the refractive layer is limited due to the accuracy of approximation with an approximation function, and it is difficult to apply the method to any refractive layer. Further, in the calibration, information regarding the posture of the calibration object with respect to the camera is required. And in order to acquire information about the posture, it is necessary to photograph a calibration object installed in a space including a refractive layer in the path, and simultaneously image another calibration object installed in a space not including the refractive layer. is there. That is, the method described in Non-Patent Document 2 has a problem that a space that is originally targeted can be captured only by a part of the camera, and a high-definition image cannot be acquired. Further, the technique described in Non-Patent Document 2 has a problem that it can be used only in a situation where both a space including a refractive layer and a space not including a refractive layer can be photographed.

  In view of the above circumstances, the present invention provides a camera calibration that can realize camera model calibration even when an unknown refractive layer or a complicated refractive layer exists between the camera and the subject. It is an object to provide a method, a camera calibration device, and a camera calibration program.

  One aspect of the present invention is a camera calibration method for associating each pixel of an image obtained by photographing a subject through a refraction layer with a light beam from the subject, the posture of the subject as the subject Correspondence acquisition step for acquiring information on the correspondence between the pixel and the position on the calibration object for each posture of the calibration object from a group of images obtained by photographing different calibration objects; Using the information about the correspondence acquired in the intersection detection step for detecting a pixel group corresponding to the intersection line intersecting in the imaging space and the correspondence acquisition step in the intersection detected in the intersection detection step , The posture estimation step of estimating the posture of the calibration object and outputting posture information, and the correspondence relationship acquired in the correspondence relationship acquisition step. And a ray calibration information acquisition step of acquiring ray information indicating a correspondence relationship between pixels and rays in the imaging space using the information to be obtained and the posture information estimated in the posture estimation step. It is.

  One aspect of the present invention is the above-described camera calibration method, wherein in the posture estimation step, the posture of the calibration object is estimated using the fact that the intersection line is defined on the two calibration objects.

  One aspect of the present invention is the above-described camera calibration method, wherein in the posture estimation step, the posture of the calibration object is estimated using a plurality of intersecting lines intersecting at one point on the imaging space. .

  One aspect of the present invention is the camera calibration method described above, wherein, in the intersection detection step, the calibration plane density in the pixel at the same position is the same between the images obtained by photographing the calibration objects of the different postures. The pixel is detected as a pixel where the intersection line exists.

  One aspect of the present invention is the above-described camera calibration method, using the information related to the correspondence acquired in the correspondence acquisition step and the posture of the calibration object estimated in the posture estimation step. The method further includes an evaluation value acquisition step of acquiring an evaluation value for estimating the posture of the calibration object.

  One aspect of the present invention is the camera calibration method described above, wherein in the evaluation value acquisition step, two or more line segments for one pixel from a position on the calibration object sampled at the same pixel. And an evaluation value for estimating the posture of the calibration object is obtained using the outer product between the direction vectors corresponding to the line segment.

  One aspect of the present invention is a camera calibration device for associating each pixel of an image obtained by photographing a subject through a refractive layer with a light beam from the subject, the posture of the subject as the subject For each posture of the calibration object, from a group of images obtained by photographing different calibration objects, a correspondence acquisition unit that acquires information on the correspondence between the pixel and the position on the calibration object, and a calibration object having a different posture By using the information on the correspondence acquired by the intersection detection unit for detecting a pixel group corresponding to the intersection line intersecting in the imaging space, and the correspondence acquisition unit for the intersection detected by the intersection detection unit A posture estimation unit that estimates the posture of the calibration object and outputs posture information, information about the correspondence acquired by the correspondence relationship acquisition unit, and information estimated by the posture estimation unit. It was used and the posture information, a camera calibration device and a light information acquiring section for acquiring light information indicating a correspondence relationship between the light beam in the imaging space and pixel.

  One aspect of the present invention is a camera calibration program for causing a computer to execute the camera calibration method.

  According to the present invention, calibration of a camera model can be realized even when an unknown refraction layer or a complicated refraction layer exists between a camera and a subject.

It is a figure which shows the imaging | photography space which performs camera calibration in this embodiment. It is a block diagram which shows the structure of the camera calibration apparatus in this embodiment. It is a flowchart which shows operation | movement of the camera calibration apparatus 10 in this embodiment. It is a figure for demonstrating the 1st method of estimating the attitude | position of the calibration object. It is a figure for demonstrating the 2nd method of estimating the attitude | position of the calibration object. It is a figure which shows the structural example of the hardware of the camera calibration apparatus 10 in this embodiment.

Hereinafter, a camera calibration apparatus according to an embodiment of the present invention will be described with reference to the drawings.
First, as a calibration in one camera, a mapping process (internal calibration) between a light beam in an imaging space and a pixel of an image captured by the camera will be described. FIG. 1 is a diagram illustrating an imaging space in which camera calibration is performed in the present embodiment. As shown in FIG. 1, an unknown refractive layer 2 exists between the camera 1 and the imaging space, and a calibration object 3 or a calibration object 3 ′ is arranged in the imaging space. Here, the calibration object 3 ′ is the same object as the calibration object 3, and shows a state after the posture of the calibration object 3 is changed. Although any object may be used as the calibration object 3, it is necessary to use an object that can obtain the correspondence between the pixel and the position on the calibration object 3, as will be described later. In the present embodiment, a flat display that displays a gray code is used as the calibration object 3.

  FIG. 2 is a block diagram illustrating a configuration of the camera calibration apparatus according to the present embodiment. As shown in FIG. 2, the camera calibration apparatus 10 includes an image input unit 11, a correspondence relationship acquisition unit 12, an intersection detection unit 13, a posture estimation unit 14, and a ray information output unit 15. . Although not shown in FIG. 1, the camera calibration device 10 is a computer terminal that can input an image captured by the camera 1 and performs image processing on the input image.

  The image input unit 11 inputs an image (group) obtained by photographing the calibration objects 3 and 3 ′. The correspondence relationship acquisition unit 12 acquires information that associates pixels in the image input by the image input unit 11 with positions on the calibration object 3. The intersection line detection unit 13 detects an intersection line formed by the calibration object 3 and the calibration object 3 ′ having different postures in the imaging space on the image. The posture estimation unit 14 estimates a relative posture between the plurality of calibration objects 3 and 3 ′. The light ray information output unit 15 associates a light ray with each pixel using the information acquired by the correspondence acquisition unit 12 and the information on the postures of the plurality of calibration objects 3 and 3 ′ estimated by the posture estimation unit 14. Process.

Next, the operation of the camera calibration apparatus 10 shown in FIG. 2 will be described. FIG. 3 is a flowchart showing the operation of the camera calibration apparatus 10 in the present embodiment.
First, the image input unit 11 inputs a group of images {In, m | n = 1,..., N, m = 1,... Mn} taken while changing the posture of the calibration object 3 (step S101). Here, N is the number of postures of the calibration object 3 included in the input image group, and Mn is the number of images when the calibration object 3 is in the nth posture. Here, N is 3 or more, and Mn is the number of images necessary for associating the pixel with the position on the calibration object 3 in the nth posture. Mn depends on the type of the calibration object 3 and the size of the imaging space.

Next, the correspondence relationship acquisition unit 12 selects an image group including the calibration object 3 having three different postures from the input image group (step S102). Here, the correspondence relationship acquisition unit 12 can determine whether there is a difference in posture between the calibration object 3 and the calibration object 3 ′ using various methods, and can select an image including the calibration object 3 having a different posture. The correspondence relationship acquisition unit 12 is configured to select a plurality of images having the same size of the area of the calibration object 3 in the image and a large change in the shape of the area of the calibration object 3 in the image, for example. With such a configuration, it is possible to increase the possibility that there is an intersection line of the calibration object 3 in the imaging space detected by the processing described later. In the following description, the posture (unknown) of the calibration object 3 in the selected image is Φ ₁ , Φ ₂ , and Φ ₃ .

  Next, the correspondence relationship acquisition unit 12 associates the pixel with the position on the calibration object 3 for each posture of the calibration object 3 (step S103). Specifically, the coordinate value (pixel position) on the image plane is associated with the coordinate value in the coordinate system defined on the calibration object 3. The method here differs depending on the type of the calibration object 3.

  For example, a case will be described in which the calibration object 3 is configured by a flat display and the calibration object 3 displays a series of gray code patterns in which coordinate values are encoded. The image input unit 11 inputs an image group obtained by photographing the calibration object 3 that displays a gray code pattern. The correspondence acquisition unit 12 can obtain a coordinate value on the calibration object 3 corresponding to the pixel by obtaining a sequence of signals photographed at each pixel from the input image group and decoding it. For example, Non-Patent Document 1 describes in detail the correspondence between the pixel using the gray code and the position on the plane (calibration plane) of the calibration object 3.

Here, the coordinate value in the coordinate system based on the calibration object 3 of the posture Φ _i of the point P _{k, i} on the calibration object 3 of the posture Φ _i with respect to the pixel k obtained as a result of step S103 is _represented by p _{k, i.} ^[i] (k = 1,..., numPixs _i , i = 1,..., 3) Here, numPixs _i is the number of pixels that can be correlated with a point on the calibration object 3 in the posture Φ _i . After associating each pixel with the coordinate value on the calibration object 3, the intersection line detection unit 13 detects on the image an intersection line where the calibration objects 3 having different postures intersect in the imaging space (step S104). Note that an intersection line where calibration objects 3 having different postures intersect in the imaging space is assumed that calibration objects 3 having different postures are simultaneously present in the imaging space, and calibration objects 3 having different virtual postures are assumed. This is the part where the planes intersect.

The pixels on the configured intersection line out with orientation [Phi _i calibration object 3 and the orientation [Phi _j calibration object 3 of '| and _{{K i, j (a)} a = 1, ..., numPixs (i, j)} . numPixs (i, j) is the position [Phi _i calibration object 3 in an image obtained by photographing the calibration object 3 orientation [Phi _i, be a number of pixels on the intersecting line composed out with orientation [Phi _j calibration object 3 ' , K _{i, j} (a) = K _{j, i} (a).

Incidentally, the intersection line detection unit 13, and any way calibration object 3 orientation [Phi _i with, may be detected intersection of the orientation [Phi _j calibration object 3 '. For example, the intersection detection unit 13 may use a method of using a calibration plane density that is a ratio of the area of the captured calibration plane in each pixel on the image or in a certain region centered on each pixel. . In other words, if pixels at the same position have the same calibration plane density between images taken of calibration objects 3 of different orientations, an intersection line exists, and the intersection line is detected by obtaining a set of such pixels. can do. When the calibration object 3 is configured by a flat display, the number of pixels of the flat display is used as the area of the calibration plane, and the intersection line is calculated using the pixel density that is the density of the flat display pixels per pixel on the image. Can be detected.

  When a flat display is used as the calibration object 3, it is preferable that the correspondence between the pixel and the position on the calibration object can be taken as in the Gray code described above. Moreover, even if it is other than the Gray code, an image obtained by capturing a pattern that can easily acquire the pixel density may be displayed on the flat display as the calibration object 3. Even when a flat display is not used, each pixel on the image or an area on the calibration object 3 photographed in a certain area centered on each pixel or a unit area on the calibration object 3 is photographed. An object that can detect the number of pixels on the image may be used as the calibration object 3. By using such a calibration object 3, the intersection line detection unit 13 can detect the intersection line.

The intersection line detection unit 13 determines that the intersection line cannot be detected when the intersection line is not obtained with sufficient accuracy. Further, the intersection line detection unit 13 may use any scale as the accuracy of the intersection line detection. For example, the intersection line detection unit 13 detects the intersection line by using the linearity of the point group {p _{Ki, j (a)} ^[i] , i} on the calibration object 3 corresponding to the pixel group detected as the intersection line. Assess accuracy. Further, the intersection line detection unit 13 obtains a likely straight line obtained for the point group, for example, as an evaluation value of the linearity of the point group, and uses a total value or a dispersion value of the distance between the straight line and each point. May be. Further, the intersection detection unit 13 may use the number of points within a certain distance from the straight line as the evaluation value of the linearity of the point group.

Further, the intersection line detection unit 13 may exclude the point group {p _{Ki, j (a)} ^[i] , i} that does not satisfy linearity from the pixel group corresponding to the intersection line. . In this case, the intersection line detection unit 13 may determine that the intersection line has not been detected when the number of remaining pixels is equal to or less than a certain number.

  Here, when three intersection lines cannot be detected between all postures, or when two or more points on the intersection line cannot be detected in each intersection line (NO in step S105), the correspondence acquisition unit 12 selects a group of images including the calibration object 3 in another posture (step S106), and returns to step S103. Note that the process of associating the pixel with the position on the calibration object 3 for each posture of the calibration object 3 in step S103 is not limited to a procedure that is repeated every time a combination of different postures is selected, as shown in FIG. It may be a procedure. For example, this is a procedure in a case where a sufficient storage area is secured in the camera calibration apparatus 10 and the result of associating the pixel and the position on the calibration object 3 for each posture is accumulated in the storage area. For example, when the camera calibration device 10 has already accumulated an image group for which the correspondence relationship between the pixel and the position on the calibration object 3 has already been acquired, among the newly selected combinations of different postures. Then, a procedure of reading out the result of associating the accumulated pixel with the position on the calibration object 3 from the storage area is performed.

  Note that the procedure is not limited to the procedure shown in FIG. 3. For example, the procedure of performing step S104 after step S102 or the procedure of performing step S103 after checking the conditions of step S105 may be used. . However, when evaluating the linearity of the detected intersection line on the calibration object 3 described above, it is necessary to perform the process of obtaining the correspondence relationship in step S103 in step S104 for some pixels.

Next, when three intersection lines can be detected and two or more points can be detected on each intersection line (YES in step S105), the posture estimation unit 14 obtains information on the detected points on the intersection line. Using this, the posture of the calibration object 3 is obtained (step S107). Note that the posture estimation unit 14 may express the posture of the calibration object 3 to be obtained with any information. In addition, the posture estimation unit 14, for example, forms of a rotation matrix R _i and a translation vector t _i for converting a coordinate system for the calibration object 3 with the posture Φ _i into a coordinate system for the calibration object 3 with a certain reference posture. It may be expressed as

In the following description, a _{first method} for expressing the posture of the calibration object 3 using a rotation matrix and a translation vector based on the posture with respect to the posture Φ ₁ and estimating the rotation matrix and the translation vector will be described. In the process of obtaining the rotation matrix R _i and the translation vector t _i , it is used that the detected intersection line is defined on the two calibration objects 3. That is, it is utilized that one intersection line is expressed in two different coordinate systems.

Hereinafter, the posture estimation process of the calibration object 3 in the posture estimation unit 14 will be specifically described.
FIG. 4 is a diagram for explaining a first method for estimating the posture of the calibration object 3. FIG. 4 shows an example of three intersection lines where the calibration planes of the calibration object 3 in the postures Φ _{1 to} Φ ₃ detected by the intersection line detection unit 13 intersect. As shown in FIG. 4, points P _i (i = 1,..., 6) on each intersection line are defined. At this time, since each point _Pi is a point on two calibration objects, the relationship shown in the following (Equation 1) is established.

From this relationship, if the coordinate value in the coordinate system with the calibration object 3 of the posture Φ _j of the point P _i is represented by p _i ^[j] , the following relationships (Expression 2) to (Expression 4) hold. .

Further, for the inner product of the straight line P ₁ P ₂ and the straight line P ₅ P _{6 and} the inner product of the straight line P ₃ P ₄ and the straight line P ₅ P ₆ , the following relationships (Equation 5) and (Equation 6) are established, respectively. The straight line P _i P _j indicates a straight line connecting the points P _i and P _j .

If these are organized for _{R 2, R 3, t 2} , t 3, and the vector x comprising the elements of _{R 2, R 3, t 2} , t 3, the elements of _{R 2, R 3, t 2} , t 3 The simultaneous equations expressed in the form of Mx = b are obtained by using the matrix M and the vector b which are not included. Therefore, by solving this simultaneous equation, rotation matrices R ₂ and R ₃ and translation vectors t ₂ and t ₃ are obtained. Note that any method may be used to obtain the rotation matrix and the translation vector. For example, the rotation matrix and the translation vector may be obtained using singular value decomposition.

  In the above description for obtaining the rotation matrix and the translation vector, two points on three intersecting lines (6 points in total) are used. However, if there are three or more points on the intersecting line, the points can be determined by an arbitrary method. You may select and perform the said process. Furthermore, a method of performing robust estimation against noise by performing estimation / verification using a plurality of combinations may be used. A representative example of such a method is a method called RANSAC (RANdom Sample Consensus).

In addition, when the three calibrating objects 3 are not parallel to each other and the intersecting lines of the three calibrating objects 3 do not intersect with one straight line, the three calibrating objects 3 are arranged on one side of the triangular pyramid. The rotation matrix R _i and the translation vector t _i may be obtained by using the part. More specifically, the rotation matrix and the translation vector may be obtained by utilizing the fact that three intersecting lines intersect at one point that is the apex of the triangular pyramid.

Hereinafter, the second method for estimating the posture of the calibration object 3 using the rotation matrix and the translation vector will be specifically described.
FIG. 5 is a diagram for explaining a second method for estimating the posture of the calibration object 3. As shown in FIG. 5, a point P _i (i = 1, 3, 5) on each intersection line where the calibration planes of the calibration object ₃ having postures Φ _{1 to} Φ 3 intersect and an intersection point O of the three intersection lines are defined. To do. At this time, for the point P _i (i = 1, 3, 5), in addition to the same relationship as described above, the coordinate value in the coordinate system based on the calibration object 3 of the posture Φ _i of the intersection O is used. o ^[i] can be calculated by obtaining the intersection of straight lines representing each intersection line. At this time, the following relations (Expression 7) to (Expression 11) are established from the equivalence of the line segment and the inner product.

When these organizing the rotation matrix _R 2, _{R 3,} vector x containing the elements of the rotation matrix _R 2, _{R 3} and 'a rotation matrix _R 2, matrix does not include an element of _{R 3} M' and the vector b 'using Thus, simultaneous equations expressed in the form of M′x ′ = b ′ are obtained. Therefore, the rotation matrices R ₂ and R ₃ can be obtained by solving the simultaneous equations. Note that any method may be used as a method of obtaining the rotation matrices R ₂ and R ₃ . When the rotation matrices R ₂ and R ₃ are obtained, translation vectors are obtained by the following (Expression 12) and (Expression 13).

  In the above description for calculating the rotation matrix, one point on each intersection line (three points in total) was used. However, if there are two or more points on the intersection line, select a point by any method. The above processing may be performed. Furthermore, a method of performing robust estimation against noise by estimating / verifying using a combination of a plurality of points on the intersection line may be used. A representative example of such a method is a method called RANSAC (RANdom Sample Consensus).

  When the posture relationship of the calibration object 3 is obtained, the light ray information output unit 15 generates a result of associating each pixel with the light ray sampled by that pixel (step S108). The generated result is an output of the camera calibration device 10. The expression for associating the pixel with the light beam may be expressed in any format. For example, each pixel may be used as an index, and may be expressed using a look-up table that returns a straight line expression representing a ray.

The light beam sampled at the pixel k can be obtained by estimating a straight line from the coordinate value {p _{k, i} ^[i] | (i = 1, 2, 3)} on the calibration object 3. Here, the light beam is expressed in a coordinate system based on the calibration object 3 having the posture Φ ₁ , but a coordinate system based on another posture may be used.

  Further, after the process of estimating the posture of the calibration object 3 (step S107) is completed, an evaluation value representing the accuracy of calibration, that is, the accuracy of estimation of the posture of the calibration object 3 may be obtained. This evaluation value may be any evaluation value as long as it can indicate whether or not the posture of the calibration object 3 has been correctly estimated. Specific example 1 of the evaluation value is an evaluation value obtained by calculating the sum, maximum value, and variance of the norm of the outer product of two or more line segments composed of points on the calibration object 3 obtained for each pixel. It is. Further, specific example 2 of the evaluation value obtains two or more line segments constituted by points on the calibration object 3 obtained for each pixel, and regarding the norm of the difference vector of the direction vector corresponding to the line segment, The sum, maximum value, and variance are obtained and used as evaluation values. Furthermore, the specific example 3 of the evaluation value is obtained by calculating the sum, the maximum value, and the variance of the distance between the light ray associated with each pixel and the point on the calibration object 3 and obtaining the evaluation value. Furthermore, specific example 4 of the evaluation value is the evaluation value obtained by calculating the sum, maximum value, and variance of the distance between the ray associated with each pixel and the point on the calibration object 3 on the calibration object 3 plane. Is.

  Alternatively, an image obtained by capturing the calibration object 3 having another posture may be added, an image group for the calibration object 3 having a different posture may be selected from the updated image group, and the process may return to step S103. The image to be added may be an image obtained by photographing the calibration object 3 in any posture. For example, an image that is the same calibration object 3 and the size of the calibration object 3 on the image is significantly different from that of the already input image group may be added. Moreover, you may add the image which is the same calibration object 3, and the same size but the shape of the calibration object 3 on an image differs. The former means that the distance from the camera that captured the image to the calibration object 3 is different from that of the already input image. By inputting an image including the calibration object 3 having a different posture as described above, it is possible to perform calibration in consideration of the straightness of the light beam through a longer path. On the other hand, the latter makes it possible to perform calibration more robustly by considering the straightness of the light beam in a narrower space.

(Hardware configuration example)
FIG. 6 is a diagram illustrating a hardware configuration example of the camera calibration device 10 according to the present embodiment.
As shown in FIG. 6, it is a block diagram which shows the hardware constitutions when the camera calibration apparatus 10 is comprised by a computer. The camera calibration apparatus 10 receives a CPU (Central Processing Unit) 100 that executes a program, a memory 101 such as a RAM that stores programs and data accessed by the CPU 100, and an image signal to be processed from the camera or the like. The image input unit 102, a program storage device 103 that stores a camera calibration program 105 that is a software program that causes the CPU 100 to execute camera calibration processing, and the CPU 100 executes the camera calibration program 105 that is loaded into the memory 101. In this configuration, a light ray information output unit 104 that outputs the correspondence between the generated pixel and the light ray is connected by a bus.

  Note that the image input unit 102 may be configured by a storage unit that stores an image signal by a disk device or the like and inputs the image signal by reading the stored image signal. The light beam information output unit 104 may be configured by a storage unit that stores light beam information indicating the correspondence between pixels and light beams by a disk device or the like. Also, it includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and functions as a device having camera calibration by executing a camera calibration program. Note that all or part of each function of camera calibration may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). The camera calibration program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. The camera calibration program may be transmitted via a telecommunication line.

  Thus, the camera calibration device 10 in the present embodiment can be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As described above, the camera and the subject are generated by generating the correspondence relationship between the pixel and the light from the group of images obtained by photographing the same calibration object in a plurality of postures using the fact that the light travels straight in the photographing space. Even when an arbitrary refractive layer (an unknown refractive layer or a complicated-shaped refractive layer) is included between the two, a camera calibration can be realized. In addition, the camera calibration can be realized even in a situation where the subject cannot be photographed through a space that does not include a refractive layer between the camera and the subject.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The camera calibration method, camera calibration apparatus, and camera calibration program of the present invention are indispensable for associating pixels with light rays when there is an unknown refractive layer between the camera and the subject. Can be applied.

DESCRIPTION OF SYMBOLS 1 ... Camera, 2 ... Refraction layer 3, 3 '... Calibration object, 10 ... Camera calibration apparatus, 11 ... Image input part, 12 ... Correspondence acquisition part, 13 ... Intersection detection part, 14 ... Posture estimation part, 15: Ray information output unit

Claims (8)

A camera calibration method for associating each pixel of an image obtained by photographing a subject through a refraction layer with a light beam from the subject,
A correspondence acquisition step for acquiring information on a correspondence relationship between the pixel and the position on the calibration object for each posture of the calibration object from an image group obtained by photographing calibration objects having different postures as the subject;
An intersection detection step of detecting a pixel group corresponding to an intersection of the calibration objects having different postures in the imaging space;
A posture estimation step of estimating posture of the calibration object and outputting posture information using information on the correspondence acquired in the correspondence acquisition step in the intersection detected in the intersection detection step;
Using the information on the correspondence acquired in the correspondence acquisition step and the posture information estimated in the posture estimation step, light information indicating the correspondence between the pixels and the light rays in the imaging space is acquired. And a light beam information acquisition step.   The camera calibration method according to claim 1, wherein, in the posture estimation step, the posture of the calibration object is estimated using the fact that the intersection line is defined on two calibration objects.   The camera calibration method according to claim 1, wherein in the posture estimation step, the posture of the calibration object is estimated using a plurality of intersecting lines intersecting at one point on the imaging space.   The crossing detection step detects the pixel as a pixel where the crossing line exists when the calibration plane density of the pixel at the same position is the same between the images obtained by photographing the calibration objects of the different postures. The camera calibration method according to claim 3.   An evaluation value acquisition step of acquiring an evaluation value for estimating the posture of the calibration object using the information regarding the correspondence relationship acquired in the correspondence relationship acquisition step and the posture of the calibration object estimated in the posture estimation step. The camera calibration method according to any one of claims 1 to 4, further comprising:   In the evaluation value acquisition step, two or more line segments are obtained for one pixel from the position on the calibration object sampled by the same pixel, and an outer product between direction vectors corresponding to the line segment is used. The camera calibration method according to claim 5, wherein an evaluation value for estimating the posture of the calibration object is acquired. A camera calibration device for associating each pixel of an image obtained by photographing a subject through a refractive layer with a light beam from the subject,
A correspondence acquisition unit that acquires information on a correspondence between the pixel and the position on the calibration object for each posture of the calibration object from an image group obtained by photographing calibration objects having different postures as the subject;
An intersection detection unit for detecting a pixel group corresponding to an intersection line in which the calibration objects having different postures intersect in an imaging space;
A posture estimation unit that estimates posture of the calibration object and outputs posture information using information about the correspondence acquired by the correspondence acquisition unit in the intersection detected by the intersection detection unit;
Using the information on the correspondence acquired by the correspondence acquisition unit and the posture information estimated by the posture estimation unit, light information indicating the correspondence between the pixels and the light rays in the imaging space is acquired. And a light beam information acquisition unit.   A camera calibration program for causing a computer to execute the camera calibration method according to claim 1.

Images