WO2023166617A1

WO2023166617A1 - Camera parameter estimation device, camera parameter estimation method, and computer-readable recording medium

Info

Publication number: WO2023166617A1
Application number: PCT/JP2022/008910
Authority: WO
Inventors: 学中野
Original assignee: 日本電気株式会社
Priority date: 2022-03-02
Filing date: 2022-03-02
Publication date: 2023-09-07

Abstract

A camera parameter estimation device 10 comprises: a corresponding point acquisition unit 11 that acquires image plane coordinates of respective corresponding points included in a plurality of pairs of corresponding points, which are obtained from a first image captured by a first camera and a second image captured by a second camera; an epipole acquisition unit 12 that acquires one or both of a first epipole indicating the image plane coordinates of the second camera projected onto the first image and a second epipole indicating the image plane coordinates of the first camera projected onto the second image; and a camera parameter estimation unit 13 that estimates a focal distance and a fundamental matrix representing an epipolar constraint, which is a geometric constraint between corresponding points, from simultaneous equations represented by: an epipolar equation for the fundamental matrix and the plurality of pairs of corresponding points; a constraint condition relating to the fundamental matrix and the first epipole or second epipole; and a constraint condition enabling decomposition of the focal distance from the fundamental matrix.

Description

Camera parameter estimation device, camera parameter estimation method, and computer-readable recording medium

The present disclosure relates to a camera parameter estimation device and a camera parameter estimation method for estimating camera parameters, and further to a computer-readable recording medium for realizing them.

The problem of restoring camera parameters and three-dimensional information of an object from an image sequence containing a plurality of images of the same object (subject) photographed using a plurality of cameras is SfM (Structure-from-Motion), and the multi-view geometry problem. Camera parameters include two types of "internal parameters" and "external parameters".

Intrinsic parameters are lens-specific parameters such as focal length, lens distortion, and optical center coordinates, and extrinsic parameters are three-dimensional rotation matrices and translation vectors between cameras. The intrinsic parameters can be measured in advance if the lens is fixed, and a camera with known intrinsic parameters is called a calibrated camera. On the other hand, a camera whose intrinsic parameters are unknown is called an uncalibrated camera.

In two-viewpoint geometry with uncalibrated cameras, the "relative parameters between cameras" are expressed as a 3 × 3 fundamental matrix (F matrix: Fundamental matrix). The term "relative" here is used to express the rotation matrix and translation vector of the second camera based on the coordinate system of the first camera when the definition of the absolute world coordinate system is unknown. is. The F matrix is obtained by multiplying this relative translation vector, rotation matrix, and internal parameters. It is known that this F-matrix has constraints such that one of the eigenvalues is zero and the degrees of freedom are six or seven.

R. Hartley's 8-point method is widely used as the method for calculating the F matrix. R. Hartley's 8-point method uses at least eight sets of "corresponding point pairs" containing two corresponding points in two images. Using 8 or more corresponding point pairs and ignoring the constraints of the F matrix, one least-squares solution can be determined by the linear least-squares method. Then, the estimated F-matrix is decomposed to obtain the focal length and the extrinsic parameters. Another well-known method is H. Stewenius' six-point method, which uses six corresponding point pairs. Unlike the R. Hartley 8-point method, H. Stewenius's 6-point method can simultaneously estimate the F matrix and the focal length by solving a polynomial problem that satisfies the aforementioned constraints on the F matrix. H. Stewenius' six-point method assumes that the camera's lens center coincides with the image center, and that the two cameras have the same focal length. This assumption is seen as valid in digital cameras and is standard in SfM. Patent Document 1 discloses a method of calibrating a multi-viewpoint camera using the 6-point method.

Non-Patent Document 1 discloses a method of using the image coordinates of another camera captured in the image (hereinafter referred to as an epipole) in order to stably obtain the F matrix with a smaller number of corresponding points. When each camera is captured in each image, that is, when two sets of epipoles are both observable, the F matrix can be calculated with three or more sets of corresponding points by the linear least squares method (M.Ito's 3 dot method). Also, when only one camera is reflected in the other image, that is, when only one set of epipoles can be observed, the F matrix can be similarly calculated with five or more sets of corresponding points (M. Ito's 5-point method). While being a linear least squares method, it satisfies the constraint that one eigenvalue of the F matrix is zero.

Each corresponding point pair is a pair of image coordinates on each image obtained by observing a point in real space with two cameras. In other words, the three-dimensional coordinates in the real space of corresponding point pairs are the same. In general, tens to thousands of corresponding point pairs are obtained from one image pair. The obtained corresponding point pair group includes erroneous corresponding points, that is, corresponding point pairs in which projected points of different three-dimensional coordinates are associated. Therefore, from all corresponding point pairs, corresponding point pairs with the minimum number of points required for calculation (for example, 6 pairs for the 6-point method, 6 pairs for the 8-point method) are randomly selected, and an F matrix with a small error is calculated. The operation must be repeated. Here, the error represents the residual of the epipolar equation and the reprojection error. This operation is called RANSAC (Random SAmple Consensus), and is widely used as a standard method for searching for correct corresponding points with a small error from a group of corresponding points. Since it is a type of combinatorial optimization, the smaller the minimum number of points, the smaller the number of iterations and the shorter the calculation execution time.

Patent No. 6569769

By the way, the method of calculating the F matrix described above has the following problems.

First, R. Hartley's 8-point method is a least-squares method that ignores the constraints of the F matrix described above. Therefore, with this calculation method, there is a possibility that the internal parameters such as the focal length cannot be restored from the F matrix. Geometrically accurate three-dimensional information cannot be estimated if the internal parameters are unknown. For example, when restoring a three-dimensional shape, a problem arises that a cube cannot be distinguished from a distorted hexahedron.

Next, H. Stewenius' 6-point method can estimate the focal length, but the estimation error is large due to the observation noise included in the corresponding points, and the focal length is calculated as an unrealistic negative value. I have something to do.

Furthermore, although M. Ito's 3-point method and 5-point method satisfy some of the constraints, they do not take into consideration the decomposability condition of the F matrix, like R. Hartley's 8-point method. Therefore, even though one eigenvalue of the F matrix is zero, there is a possibility that internal parameters such as focal length cannot be restored.

As described above, in calculating the F matrix by the conventional 3-point method, 5-point method, 6-point method, or 8-point method, the F matrix that satisfies the decomposability condition must be stably and highly accurately estimated. is difficult.

An example of an object of the present disclosure is to solve the above problem and provide a camera parameter estimation device, a camera parameter estimation method, and a computer-readable recording medium capable of estimating a highly accurate F matrix that satisfies the resolvability condition. It is in.

In order to achieve the above object, a camera parameter estimation device according to one aspect of the present disclosure includes:
a corresponding point acquisition unit that acquires the coordinates in the image plane of each corresponding point included in each of the plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera;
one of a first epipole that is the image plane coordinates of the second camera projected onto the first image and a second epipole that is the image plane coordinates of the first camera projected onto the second image; or an epipole acquisition unit that acquires both;
An epipolar equation for a base matrix representing an epipolar constraint, which is a geometric constraint between two corresponding points of a corresponding point pair, and the plurality of corresponding point pairs, the base matrix and the first epipole or the second epipole a camera parameter estimating unit that estimates the fundamental matrix and the focal length from simultaneous equations represented by a constraint for and a constraint that allows the focal length to be decomposed from the fundamental matrix;
characterized by comprising

Further, in order to achieve the above object, a camera parameter estimation method according to one aspect of the present disclosure includes:
Acquiring the coordinates in the image plane of each corresponding point included in each of the plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera,
one of a first epipole that is the image plane coordinates of the second camera projected onto the first image and a second epipole that is the image plane coordinates of the first camera projected onto the second image; or get both,
An epipolar equation for a base matrix representing an epipolar constraint that is a geometric constraint between two corresponding points of a corresponding point pair and the plurality of corresponding point pairs, and for the base matrix and the first epipole or the second epipole and a constraint that allows the focal length to be decomposed from the fundamental matrix, estimating the fundamental matrix and the focal length from the simultaneous equations expressed by
It is characterized by

Furthermore, in order to achieve the above object, a computer-readable recording medium in one aspect of the present disclosure includes:
to the computer,
Acquiring the coordinates in the image plane of each corresponding point included in each of the plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera,
one of a first epipole that is the image plane coordinates of the second camera projected onto the first image and a second epipole that is the image plane coordinates of the first camera projected onto the second image; or get both,
An epipolar equation for a base matrix representing an epipolar constraint that is a geometric constraint between two corresponding points of a corresponding point pair and the plurality of corresponding point pairs, and for the base matrix and the first epipole or the second epipole and a constraint that allows the focal length to be decomposed from the fundamental matrix, estimating the fundamental matrix and the focal length from the simultaneous equations represented by
It is characterized by recording a program including instructions.

As described above, according to the present disclosure, it is possible to estimate a highly accurate F matrix that satisfies the decomposability condition.

FIG. 1 is a block diagram showing an example of the configuration of a camera parameter estimation device according to the first embodiment. FIG. 2 is a diagram for explaining a specific example of two-viewpoint geometry. FIG. 3 is a flow chart showing an example of the operation of the camera parameter estimation device according to the first embodiment. FIG. 4 is a block diagram showing an example of the configuration of a camera parameter estimation device according to the second embodiment. FIG. 5 is an explanatory diagram showing a specific example of the coordinate system in the second embodiment. FIG. 6 is a flowchart showing an example of the operation of the camera parameter estimation device according to the second embodiment. FIG. 7 is a block diagram showing an example of a computer that implements the camera parameter estimation device according to the first and second embodiments.

Embodiments will be described below with reference to the drawings. In addition, in the following embodiments, the same or equivalent elements are given the same reference numerals, and overlapping descriptions are omitted.

(First embodiment)
A camera parameter estimation device, a camera parameter estimation method, and a program according to the first embodiment will be described below with reference to FIGS. 1 to 3. FIG.

[Device configuration]
First, the configuration of the camera parameter estimation device according to the first embodiment will be explained using FIG. FIG. 1 is a block diagram showing an example of the configuration of a camera parameter estimation device according to the first embodiment.

The camera parameter estimation device 10 according to the first embodiment shown in FIG. 1 is a device for estimating the basic matrix (F matrix) and the focal length of the camera. The F matrix is a matrix representing an epipolar constraint, which is a geometric constraint between two corresponding points included in a corresponding point pair. As shown in FIG. 1 , the camera parameter estimation device 10 includes a corresponding point acquisition section 11 , an epipole acquisition section 12 and a camera parameter estimation section 13 .

The corresponding point acquisition unit 11 acquires the in-image plane coordinates of each corresponding point included in each of a plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera. The epipole acquisition unit 12 obtains a first epipole, which is the coordinates in the image plane of the second camera projected onto the first image, and a second epipole, which is the coordinates in the image plane of the first camera projected onto the second image. one or both of them.

The camera parameter estimation unit 13 estimates the F matrix and the focal length from predetermined simultaneous equations. Simultaneous equations are epipolar equations for the F matrix and a plurality of corresponding point pairs, constraints for the F matrix and the first epipole or the second epipole (hereinafter referred to as "epipole constraints"), and from the F matrix It is represented by a constraint condition that allows the focal length to be resolved (hereinafter referred to as “resolvability condition”).

In this way, the camera parameter estimation device 10 solves the "predetermined simultaneous equations" represented by the epipolar equation, the epipole constraint, and the decomposability condition, and estimates the F matrix and the focal length. Therefore, according to the camera parameter estimation device 10, it is possible to estimate the F matrix with high accuracy that satisfies the decomposability condition.

Next, using FIG. 2 in addition to FIG. 1, functions of the camera parameter estimation device 10 in the first embodiment will be described in detail. FIG. 2 is a diagram for explaining a specific example of two-viewpoint geometry.

As described above, the corresponding point acquisition unit 11 acquires the coordinates in the image plane of each corresponding point included in each of two or more "pairs of corresponding points". In the first embodiment, the coordinates within the image plane are also referred to as "observed image coordinates". Each “corresponding point pair” includes, for example, two corresponding points that are included in the “first image” and the “second image” and correspond to each other.

Here, a specific example of the geometric relationship between two images obtained by photographing the same object with two cameras, that is, "two-viewpoint geometry" will be described with reference to FIG. FIG. 2 is an explanatory diagram showing a specific example of the coordinate system in the first embodiment.

FIG. 2 shows the camera coordinate system of camera 1 and the camera coordinate system of camera 2. In FIG. 2 , the first image is an image captured by camera 1 and the second image is an image captured by camera 2 . That is, for example, the first image corresponds to the above "first image", and the second image corresponds to the above "second image".

Then, in FIG. 2, a certain three-dimensional point X in the world coordinate system is observed by the camera 1 as the coordinate m within the image plane. The coordinates in the image plane are also denoted as "image coordinates" or "observed image coordinates", for example. 2, the three-dimensional point X is observed by the camera 2 as the observed image coordinate m'. The observed image coordinate m and the observed image coordinate m' are points corresponding to each other, and thus are "corresponding points". The observation image coordinate m and the observation image coordinate m' can be collectively called a "corresponding point pair". Camera 1 is observed as epipole e' in the second image, and camera 2 is observed as epipole e in the first image.

In the first embodiment, since the world coordinate system is not explicitly given, the coordinate values of the three-dimensional point X are unknown. Since the world coordinate system can be set arbitrarily, it is assumed to match the camera coordinate system of camera 1 . That is, in the extrinsic parameters of camera 1, the position (that is, three-dimensional translation vector) is [0,0,0], and the rotation matrix is a 3×3 identity matrix. The extrinsic parameters of the camera 2 are represented by the translation vector t and the rotation matrix R. It is assumed that the two cameras 1 and 2 have the same focal length f.

In the first embodiment, the corresponding point acquisition unit 11 obtains the observation image coordinates of the corresponding points using a widely known technique such as SIFT (Scale Invariant Feature Transform) or SURF (Speeded Up Robust Features). can be obtained. The corresponding point obtaining unit 11 can also obtain observation image coordinates of corresponding points by accepting designation of corresponding points and removal of incorrect corresponding points from the user. In this case, the corresponding points are displayed by a display for displaying the first image and the second image, and the corresponding point obtaining unit 11 obtains the corresponding points via an input device such as a mouse, keyboard, or touch panel that receives input from the user. Accepts point specification and removal of erroneous corresponding points.

In the first embodiment, the epipole acquisition unit 12 obtains the position of a camera other than the camera that captured the first image and the second image, that is, the image coordinates of each of the epipoles e and e′. get. In addition, the epipole acquisition unit 12 can also acquire the position of the epipole by accepting the designation of the position of the epipole by the user. In this case, the epipole is displayed on the display, and the epipole acquisition unit 12 receives designation of the position of the epipole via the input device.

In the first embodiment, the camera parameter estimation unit 13 uses two or more sets of corresponding points, epipoles e and epipoles e′ to solve the “predetermined simultaneous equations” to obtain the F matrix and the focal length f and This "predetermined simultaneous equations" expresses not only the F matrix and the epipolar equations for two or more corresponding point pairs, but also the decomposable conditions under which the focal length f can be obtained from the F matrix. A specific example of the "predetermined simultaneous equations" will be described later.

[Device operation]
Next, an example of the operation of the camera parameter estimation device 10 according to the first embodiment will be explained using FIG. FIG. 3 is a flow chart showing an example of the operation of the camera parameter estimation device according to the first embodiment. 1 and 2 will be referred to as necessary in the following description. Also, in the first embodiment, the camera parameter estimation method is implemented by operating the camera parameter estimation device 10 . Therefore, the description of the camera parameter estimation method in the first embodiment is replaced with the description of the operation of the camera parameter estimation device 10 below.

As shown in FIG. 3, first, when image data of two images (first image and second image) are input to the camera parameter estimation device 10, the corresponding point acquisition unit 11 obtains images of each image data. Two or more sets of corresponding point pairs are specified from . Then, the corresponding point obtaining unit 11 obtains the in-image coordinates of each corresponding point included in each identified pair of corresponding points (step S11).

Next, the epipole acquisition unit 12 uses the input image data of the first image and the second image to obtain the first epipole and the second image, which are coordinates in the image plane of the second camera projected onto the first image. A second epipole, which is the coordinates in the image plane of the first camera projected onto , is obtained (step S12).

Next, the camera parameter estimation unit 13 uses two or more sets of corresponding points and two epipoles to solve the "predetermined simultaneous equations" represented by the epipolar equation, the epipole constraint, and the decomposability condition. Then, the F matrix is calculated, and the focal length f is calculated by decomposing the F matrix (step S13). After that, the camera parameter estimation unit 13 outputs the F matrix and the focal length f calculated in step S13 (step S14). By executing step S14, the operation of the camera parameter estimation device 10 ends.

[Concrete example]
Here, a specific example of the first embodiment will be described below. In the following description of the specific example, the symbols shown in FIG. 2 are used, and it is assumed that the origin of the world coordinate system coincides with the camera coordinate system of the camera 1, as in the example of FIG. A certain 3D point X is observed as m and m' in the two camera images, respectively. Let e be the epipole of camera 2 in camera 1, and let e be the epipole of camera 1 in camera 2. FIG. Furthermore, m, m', e, and e' are expressed as three-dimensional homogeneous coordinate representations, that is, three-dimensional vectors obtained by adding 1 to the two-dimensional image coordinates. In this specific example, transposition of a vector and transposition of a matrix are represented by a superscript T.

First, define the problem to be solved. It is known that the 3×3 F matrix and the i-th corresponding points m and m′ satisfy the epipolar equation expressed by Equation 1 below.

The F matrix has the feature that one eigenvalue is zero. This feature is represented by Equation 2 or Equation 3 below, which are equivalent.

Here, det() in Equation 2 above represents the determinant of the matrix. Equation 3 above indicates that the epipole e is the eigenvector corresponding to the zero eigenvalue of the F matrix, and the epipole e' is the eigenvector corresponding to the zero eigenvalue of the transposed F matrix.

Then, it is known that the conditions under which the focal length f can be resolved from the F matrix are represented by Equation 4 or Equation 4 below.

where Fij represents the (i,j) component of the F matrix. It is known that the focal length f can be calculated from the F matrix that satisfies Equation 4 or Equation 5 using known methods such as Kruppa's equation and Bougnoux's formula.

Next, given two sets of corresponding points and two epipoles e(=[e _x , e _y , e _z ] ^T ) and e'(=[e' _x , e' _y , e' _z ] ^T ) We describe how to calculate the F matrix and the focal length f when First, from Equations 1 and 3, the relationship between corresponding points, epipoles, and the F matrix is expressed by Equation 6 below.

In Equation 6 above, x represents the 9x1 vector representation of the F matrix, I represents the 3x3 identity matrix, and 0 _1x3 represents the 1x3 zero vector. Furthermore, the symbol obtained by enclosing "X" with O represents the Kronecker product.

The matrix M is an 8×9 matrix, of which 7 rows are linearly independent. Therefore, x is represented by a linear combination of the singular vectors (null vectors) _n1 and _n2 of matrix M, as shown in Equation 7 below.

In Equation 7 above, a and b are arbitrary real numbers. From Equation 1 above, the F matrix in the epipolar equation has constant multiple indefiniteness. Therefore, in Equation 7, either a or b may be fixed as an arbitrary value.

Next, a method for calculating the F matrix will be described. To remove ambiguity, let b=1, ie x=an ₁ +n ₂ , as an example below.

Substituting “x=an ₁ +n ₂ ” into Equation 4 yields a simultaneous polynomial with two variables a and w. For example, solutions to simultaneous polynomials can be calculated using Gröbner basis or homotopy continuity method.

Substituting “x=an ₁ +n ₂ ” into Equation 5 yields a quintic equation for a. For example, a companion matrix can be used to compute the solution of a quintic equation (maximum of 5 real solutions). The focal length f can be determined from the calculated a. Then, the focal length f can be calculated from the F matrix by using the Kruppa's equation or Bougnoux's formula described above.

When the simultaneous equations are solved using Equation 4 or Equation 5 above, multiple real number solutions are generally obtained. As a method of narrowing down the solution, for example, there is a method of selecting an F matrix in which the focal length f is positive. Another method is to select the F matrix and the focal length f that minimize the error (residue of epipolar equation or reprojection error, etc.) from other corresponding points not used in the calculation.

The above solution method will be explained along each step shown in FIG. First, in step S11, when the image data of two images (the first image and the second image) are input, the corresponding point obtaining unit 11 obtains two or more sets of corresponding point pairs (corresponding point pairs) from each image. m and m').

Next, in step S12, the epipole acquisition unit 12 acquires the epipoles e and e' of each other's cameras from the respective images using the input image data of the first image and the second image.

Next, in step S13, the camera parameter estimation unit 13 substitutes “x=an ₁ +n ₂ ” into Equation 4 or Equation 5 above, solves the resulting “predetermined simultaneous equations”, and calculates the coefficient Calculate a and determine the F matrix and the focal length f. After that, in step S14, the camera parameter estimation unit 13 outputs the determined F matrix and focal length f.

[Effects of the first embodiment]
As described above, the camera parameter estimation device 10 according to the first embodiment can calculate the F matrix and the focal length using two or more corresponding point pairs and two epipoles. The reason is as follows.

Matrix M ₁ and M ₂ in Equation 6 contain the epipole constraint based on Equation 3 above. Therefore, transforming the vector x represented by Equation 7 into a 3×3 matrix satisfies the property of the F matrix (Equation 1) that one eigenvalue is zero. Furthermore, Equation 4 or Equation 5 above is a constraint expression that expresses the decomposability condition of the F matrix. Therefore, determining the coefficients of the singular vectors using Equation 4 or Equation 5 above is equivalent to determining the F matrix with which the focal length f can be calculated. For the above reasons, the F matrix and the focal length can be calculated according to the first embodiment.

[Modification]
The first embodiment is not limited to the example described above. In the first embodiment, various modifications that can be understood by those skilled in the art can be applied to the above example. For example, the first embodiment can also be implemented in the form shown in the following modifications.

Modification 1
In the examples shown in FIGS. 1 to 3, the “first image” and the “second image” are two still images of the same object captured at a certain time by two different cameras from different viewpoints. Although described as an example, the first embodiment is not limited to this. For example, the “first image” and the “second image” may be two frame images of a time-series continuous moving image. At this time, the camera 1 that captures the "first image" and the camera 2 that captures the "second image" may be the same camera or different cameras. In short, the “first image” and the “second image” are images obtained by photographing the same object or the same scene from different viewpoints using one or more cameras having the same focal length. good.

Modification 2
In the examples shown in FIGS. 1 to 3, the camera parameter estimation unit 13 removes the ambiguity of the F matrix by setting b=1, but the first embodiment is not limited to this. In the camera parameter estimation unit 13, for example, in order to eliminate the possibility of b=0, "a+b=1", that is, "x=an ₁ +(1−a)n ₂ " may be set. . In addition, in the camera parameter estimation unit 13, in order to eliminate the possibility of b=0 and limit the value range to −1≦a and b≦1 to improve numerical stability, “a ² +b ² =1” and “x=an ₁ +bn ₂ ” may be reduced to a two-variable polynomial.

Modification 3
In the examples shown in FIGS. 1 to 3, the case where two corresponding point pairs are used in the camera parameter estimation unit 13 is described, but the first embodiment is not limited to this. For example, when three or more corresponding point pairs are obtained, the camera parameter estimation unit 13 may execute the RANSAC algorithm to determine the final output F matrix and focal length f. . The number of samplings in the RANSAC algorithm is two sets, which exponentially reduces the number of trials compared to the 6-point and 8-point methods. Furthermore, in the camera parameter estimating unit 13, non-linear optimization may be used in which Equation 1 is the objective function and Equation 3 is the constraint condition. As the objective function, a well-known Sampson error or reprojection error may be used in addition to Equation 1 above. Also, a weighted least squares method may be used to constrain Equation 3 above.

[program]
The program in the first embodiment may be any program that causes a computer to execute steps S11 to S14 shown in FIG. By installing this program in a computer and executing it, the camera parameter estimation device 10 and the camera parameter estimation method in the first embodiment can be realized. In this case, the processor of the computer functions as the corresponding point acquisition unit 11, the epipole acquisition unit 12, and the camera parameter estimation unit 13 to perform processing. Examples of computers include general-purpose PCs, smartphones, and tablet-type terminal devices.

Also, the program in the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the corresponding point acquisition unit 11, the epipole acquisition unit 12, and the camera parameter estimation unit 13, respectively.

(Second embodiment)
Next, the camera parameter estimation device, camera parameter estimation method, and program in the second embodiment will be described with reference to FIGS. 4 to 6. FIG.

[Device configuration]
First, the configuration of the camera parameter estimation device according to the second embodiment will be explained using FIG. FIG. 4 is a block diagram showing an example of the configuration of a camera parameter estimation device according to the second embodiment.

The camera parameter estimation device 20 in the second embodiment shown in FIG. 4 is also a device that estimates the F matrix and the focal length of the camera, like the camera parameter estimation device 10 in the first embodiment.

As shown in FIG. 4, in the second embodiment, as in the first embodiment, a camera parameter estimation device 20 includes a corresponding point acquisition unit 21, an epipole acquisition unit 22, and a camera parameter estimation unit 23. It has

However, the second embodiment differs from the first embodiment in that only one of the two epipoles e and e' is input. The following description focuses on differences from the first embodiment.

Here, a specific example of "two-viewpoint geometry" in the second embodiment will be described using FIG. FIG. 5 is an explanatory diagram showing a specific example of the coordinate system in the second embodiment. In the example of FIG. 5, the camera 2 is observed as the epipole e in the first image, but the camera 1 is observed as the epipole e' in the second image because it is outside the imaging range of the second image. do not have.

The corresponding point acquisition unit 21 acquires, for example, four or more "corresponding point pairs" in the second embodiment. Note that the method of acquiring the corresponding point pairs by the corresponding point acquiring unit 21 is the same as that of the corresponding point acquiring unit 11 in the first embodiment, so the description thereof will be omitted.

The epipole acquisition unit 22 acquires only the image coordinates of the epipole e in the first image in the second embodiment. The epipole image coordinate acquisition method by the epipole acquisition unit 22 is the same as that of the epipole acquisition unit 12 in the first embodiment, so the description thereof will be omitted.

In the second embodiment, the camera parameter estimation unit 23 calculates the F matrix and the focal length f by solving a "predetermined simultaneous equation" using four or more corresponding point pairs and one epipole e. do. This "predetermined simultaneous equation" is represented by an epipolar equation for the F matrix and the four or more corresponding point pairs, an epipolar constraint, and a resolvability condition that allows the focal length f to be decomposed from the F matrix. ing.

Thus, in the second embodiment, the camera parameter estimating device 20 solves the "predetermined simultaneous equations" using four or more sets of corresponding points and one epipole e. and the focal length f are calculated.

[Device operation]
Next, an example of the operation of the camera parameter estimation device 20 according to the second embodiment will be explained using FIG. FIG. 6 is a flowchart showing an example of the operation of the camera parameter estimation device according to the second embodiment. 4 and 5 will be referred to as necessary in the following description. Also, in the second embodiment, the camera parameter estimation method is implemented by operating the camera parameter estimation device 20 . Therefore, the description of the camera parameter estimation method in the second embodiment is replaced with the description of the operation of the camera parameter estimation device 20 below.

As shown in FIG. 6, first, when image data of two images (a first image and a second image) are input to the camera parameter estimation device 10, the corresponding point obtaining unit 21 obtains images of each image data. 4 or more sets of corresponding points are specified from . Then, the corresponding point obtaining unit 21 obtains the in-image coordinates of each corresponding point included in each identified pair of corresponding points (step S21).

Next, the epipole acquisition unit 22 acquires the epipole e, which is the coordinates within the image plane of the second camera projected onto the first image, using the input image data of the first image and the second image (step S22 ).

Next, the camera parameter estimation unit 23 uses four or more sets of corresponding points and one epipole to solve the "predetermined simultaneous equations" represented by the epipolar equation, epipole constraint conditions, and decomposability conditions. Then, the F matrix is calculated, and the focal length f is calculated by decomposing the F matrix (step S23). After that, the camera parameter estimation unit 23 outputs the F matrix and the focal length f calculated in step S23 (step S24). By executing step S24, the operation of the camera parameter estimation device 20 ends.

[Concrete example]
Here, a specific example of the second embodiment will be described below. In the following explanation of the specific example, the symbols shown in FIG. 5 are used, and it is assumed that the origin of the world coordinate system coincides with the camera coordinate system of the camera 1, as in the examples of FIGS. . Also, in FIG. 5, the symbols that are the same as the symbols shown in FIG. 2 have the same meanings, so the description of the symbols will be omitted below.

Next, when four corresponding point pairs (m, m') and one epipole e (=[e _x , e _y , e _z ] ^T ) are given, the F matrix and the focal length f are Explain how to calculate.

By modifying Equation 6 above, the relationship between corresponding points, epipoles, and the F matrix is expressed by Equation 8 below.

In Equation 8, the matrix M is 7×9, and x is represented by a linear combination of singular vectors (null vectors) _n1 and _n2 of the matrix M, as in Equation 7 above. That is, in the second embodiment, as in the first embodiment, it is expressed as "x=an ₁ +n ₂ ". is calculated.

The above solution method will be explained along each step shown in FIG. First, in step S21, when the image data of two images (the first image and the second image) are input, the corresponding point obtaining unit 21 obtains four or more corresponding point pairs (m, m').

Next, in step S22, the epipole acquisition unit 22 uses the image data of the first image to acquire the epipole e of the camera 2 that captured the second image.

Next, in step S23, the camera parameter estimation unit 23 substitutes “x=an ₁ +n ₂ ” into Equation 4 or Equation 5, solves the resulting “predetermined simultaneous equations”, and calculates the coefficient a to determine the F matrix and the focal length f. After that, in step S24, the camera parameter estimation unit 23 outputs the determined F matrix and focal length f.

[Effects of Second Embodiment]
As described above, the camera parameter estimation device 20 according to the second embodiment can calculate the mF matrix and the focal length using four or more corresponding point pairs and one epipole. The reason is as follows.

The matrix _M1 in (8) contains the epipole constraints based on (3) above. As in the first embodiment, the F matrix is represented by Equation 7 above, and the F matrix calculated using Equation 4 or Equation 5 above satisfies the decomposability condition. Therefore, even in the second embodiment, determining the coefficients of the singular vectors using Equation 4 or Equation 5 above is equivalent to determining the F matrix that allows the focal length f to be calculated. For the above reasons, the F matrix and the focal length are calculated also in the case of the second embodiment.

[Modification]
2nd Embodiment is not limited to the example mentioned above. In the second embodiment, various modifications that can be understood by those skilled in the art can be applied to the above example. For example, the second embodiment can also be implemented by the following modified examples.

Modification 4
In the examples shown in FIGS. 4 to 6, the epipole acquisition unit 22 acquires the epipole using the image data of the first image, but in the second embodiment, the image data of the second image is used. You can also get an Epipole. In that case, the epipole acquisition unit 22 acquires the epipole e' of the camera of the first image, and the matrix _M1 in Equation 8 above is replaced with the matrix _M2 in Equation 8 above.

Modification 5
In the examples shown in FIGS. 4 to 6, the camera parameter estimator 23 removes the ambiguity of the F matrix by setting b=1, but the second embodiment is not limited to this. Also in Modification 5, the same method as in Modification 2 described above can be applied.

Modification 6
In the examples shown in FIGS. 4 to 6, the case where the number of corresponding point pairs used by the camera parameter estimation unit 23 is four is described, but the second embodiment is not limited to this. For example, when four or more corresponding point pairs are obtained, the camera parameter estimating unit 23 also executes the RANSAC algorithm or nonlinear optimization in the same manner as in Modification 3 described above, and finally An output F matrix and a focal length f may be determined.

[program]
The program in the second embodiment may be any program that causes a computer to execute steps S21 to S24 shown in FIG. By installing this program in a computer and executing it, the camera parameter estimation device 20 and the camera parameter estimation method in the second embodiment can be realized. In this case, the processor of the computer functions as a corresponding point obtaining section 21, an epipole obtaining section 22, and a camera parameter estimating section 23, and performs processing. Examples of computers include general-purpose PCs, smartphones, and tablet-type terminal devices.

Also, the program in the second embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the corresponding point acquisition unit 21, the epipole acquisition unit 22, and the camera parameter estimation unit 23, respectively.

(physical configuration)
Here, a computer (hardware configuration) that implements the camera parameter estimation device by executing the programs in the first and second embodiments will be described with reference to FIG. FIG. 7 is a block diagram showing an example of a computer that implements the camera parameter estimation device according to the first and second embodiments.

As shown in FIG. 7, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader/writer 116, and a communication interface 117. and These units are connected to each other via a bus 121 so as to be able to communicate with each other.

Also, the computer 110 may include an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), or FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, a GPU or FPGA can execute the programs in the embodiments.

The CPU 111 expands the program in the embodiment, which is composed of a code group stored in the storage device 113, into the main memory 112 and executes various operations by executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

Also, the programs in the first and second embodiments are provided in a state stored in a non-transitory computer readable medium 120. Note that the programs in the first and second embodiments may be distributed on the Internet connected via the communication interface 117. FIG. Also, the programs in the first and second embodiments may be supplied to the computer via a wired communication path such as electric wires and optical fibers, or a wireless communication path. In this case, the programs in the first and second embodiments are carried by electrical signals, optical signals and electromagnetic waves.

Further, as a specific example of the storage device 113, in addition to a hard disk drive, a semiconductor storage device such as a flash memory can be cited. Input interface 114 mediates data transmission between CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls display on the display device 119 .

The data reader/writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results in the computer 110 to the recording medium 120. Communication interface 117 mediates data transmission between CPU 111 and other computers.

Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, and CD- Optical recording media such as ROM (Compact Disk Read Only Memory), CD-R and CD-R/W can be mentioned.

Further, a specific example of the recording medium 120 is a semiconductor memory. The semiconductor memory includes, for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, and RAM (Random Access Memory).

The camera parameter estimation devices in the first and second embodiments can also be realized by using hardware corresponding to each part, instead of a computer in which a program is installed. Furthermore, the camera parameter estimating devices in the first and second embodiments may be partly implemented by a program and the rest by hardware.

Some or all of the above-described embodiments can be expressed by the following (Appendix 1) to (Appendix 9), but are not limited to the following descriptions.

(Appendix 1)
Corresponding point acquisition means for acquiring the coordinates in the image plane of each corresponding point included in each of the plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera;
one of a first epipole that is the image plane coordinates of the second camera projected onto the first image and a second epipole that is the image plane coordinates of the first camera projected onto the second image; or an epipole obtaining means for obtaining both;
An epipolar equation for a base matrix representing an epipolar constraint that is a geometric constraint between two corresponding points of a corresponding point pair and the plurality of corresponding point pairs, and for the base matrix and the first epipole or the second epipole a camera parameter estimating means for estimating the fundamental matrix and the focal length from a simultaneous equation represented by a constraint of and a constraint that allows the focal length to be decomposed from the fundamental matrix;
A camera parameter estimation device comprising:

(Appendix 2)
the epipole acquisition means acquires both the first epipole and the second epipole;
The camera parameter estimation means estimates the base matrix and the focal length by solving the simultaneous equations using two or more sets of corresponding point pairs, the first epipole, and the second epipole. do,
The camera parameter estimation device according to appendix 1.

(Appendix 3)
the epipole acquisition means acquires one of the first epipole and the second epipole;
The camera parameter estimating means solves the simultaneous equations using the four or more corresponding point pairs and one of the first epipole and the second epipole to obtain the base matrix and the focal point to estimate the distance,
The camera parameter estimation device according to appendix 1.

(Appendix 4)
Acquiring the coordinates in the image plane of each corresponding point included in each of the plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera,
one of a first epipole that is the image plane coordinates of the second camera projected onto the first image and a second epipole that is the image plane coordinates of the first camera projected onto the second image; or get both,
An epipolar equation for a base matrix representing an epipolar constraint that is a geometric constraint between two corresponding points of a corresponding point pair and the plurality of corresponding point pairs, and for the base matrix and the first epipole or the second epipole and a constraint that allows the focal length to be decomposed from the fundamental matrix, estimating the fundamental matrix and the focal length from the simultaneous equations expressed by
Camera parameter estimation method.

(Appendix 5)
obtaining both the first epipole and the second epipole in obtaining the epipole;
In estimating the base matrix and the focal length, by solving the simultaneous equations using two or more pairs of corresponding points, the first epipole, and the second epipole, the base matrix and the focal length to estimate the focal length,
A camera parameter estimation method according to appendix 4.

(Appendix 6)
obtaining one of the first epipole and the second epipole in obtaining the epipole;
In estimating the base matrix and the focal length, the base matrix and the focal length are estimated by solving the simultaneous equations using four or more of the corresponding point pairs and one of the first epipole and the second epipole. estimating a matrix and the focal length;
A camera parameter estimation method according to appendix 4.

(Appendix 7)
to the computer,
Acquiring the coordinates in the image plane of each corresponding point included in each of the plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera,
one of a first epipole that is the image plane coordinates of the second camera projected onto the first image and a second epipole that is the image plane coordinates of the first camera projected onto the second image; or get both,
An epipolar equation for a base matrix representing an epipolar constraint that is a geometric constraint between two corresponding points of a corresponding point pair and the plurality of corresponding point pairs, and for the base matrix and the first epipole or the second epipole and a constraint that allows the focal length to be decomposed from the fundamental matrix, estimating the fundamental matrix and the focal length from the simultaneous equations represented by
A computer-readable recording medium recording a program containing instructions.

(Appendix 8)
obtaining both the first epipole and the second epipole in obtaining the epipole;
In estimating the base matrix and the focal length, by solving the simultaneous equations using two or more pairs of corresponding points, the first epipole, and the second epipole, the base matrix and the focal length to estimate the focal length,
The computer-readable recording medium according to appendix 7.

(Appendix 9)
obtaining one of the first epipole and the second epipole in obtaining the epipole;
In estimating the base matrix and the focal length, the base matrix and the focal length are estimated by solving the simultaneous equations using four or more of the corresponding point pairs and one of the first epipole and the second epipole. estimating a matrix and the focal length;
The computer-readable recording medium according to appendix 7.

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

As described above, according to the present invention, a highly accurate F matrix that satisfies the decomposability condition can be estimated. INDUSTRIAL APPLICABILITY The present invention is suitable for three-dimensional shape reconstruction from images (Structure-from-Motion).

10 Camera parameter estimation device (Embodiment 1)
11 Corresponding Point Acquisition Unit 12 Epipole Acquisition Unit 13 Camera Parameter Estimation Unit 20 Camera Parameter Estimation Apparatus (Embodiment 2)
21 corresponding point acquisition unit 22 epipole acquisition unit 23 camera parameter estimation unit 110 computer 111 CPU
112 main memory 113 storage device 114 input interface 115 display controller 116 data reader/writer 117 communication interface 118 input device 119 display device 120 recording medium 121 bus

Claims

Corresponding point acquisition means for acquiring the coordinates in the image plane of each corresponding point included in each of the plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera;
one of a first epipole that is the image plane coordinates of the second camera projected onto the first image and a second epipole that is the image plane coordinates of the first camera projected onto the second image; or an epipole obtaining means for obtaining both;
An epipolar equation for a base matrix representing an epipolar constraint that is a geometric constraint between two corresponding points of a corresponding point pair and the plurality of corresponding point pairs, and for the base matrix and the first epipole or the second epipole a camera parameter estimating means for estimating the fundamental matrix and the focal length from a simultaneous equation represented by a constraint of and a constraint that allows the focal length to be decomposed from the fundamental matrix;
A camera parameter estimation device comprising:
the epipole acquisition means acquires both the first epipole and the second epipole;
The camera parameter estimation means estimates the base matrix and the focal length by solving the simultaneous equations using two or more sets of corresponding point pairs, the first epipole, and the second epipole. do,
The camera parameter estimation device according to claim 1.
the epipole acquisition means acquires one of the first epipole and the second epipole;
The camera parameter estimating means solves the simultaneous equations using the four or more corresponding point pairs and one of the first epipole and the second epipole to obtain the base matrix and the focal point to estimate the distance,
The camera parameter estimation device according to claim 1.
Acquiring the coordinates in the image plane of each corresponding point included in each of the plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera,
one of a first epipole that is the image plane coordinates of the second camera projected onto the first image and a second epipole that is the image plane coordinates of the first camera projected onto the second image; or get both,
An epipolar equation for a base matrix representing an epipolar constraint that is a geometric constraint between two corresponding points of a corresponding point pair and the plurality of corresponding point pairs, and for the base matrix and the first epipole or the second epipole and a constraint that allows the focal length to be decomposed from the fundamental matrix, estimating the fundamental matrix and the focal length from the simultaneous equations expressed by
Camera parameter estimation method.
obtaining both the first epipole and the second epipole in obtaining the epipole;
In estimating the base matrix and the focal length, by solving the simultaneous equations using two or more pairs of corresponding points, the first epipole, and the second epipole, the base matrix and the focal length to estimate the focal length,
A camera parameter estimation method according to claim 4.
obtaining one of the first epipole and the second epipole in obtaining the epipole;
In estimating the base matrix and the focal length, the base matrix and the focal length are estimated by solving the simultaneous equations using four or more of the corresponding point pairs and one of the first epipole and the second epipole. estimating a matrix and the focal length;
A camera parameter estimation method according to claim 4.
to the computer,
Acquiring the coordinates in the image plane of each corresponding point included in each of the plurality of corresponding point pairs obtained from the first image by the first camera and the second image by the second camera,
one of a first epipole that is the image plane coordinates of the second camera projected onto the first image and a second epipole that is the image plane coordinates of the first camera projected onto the second image; or get both,
An epipolar equation for a base matrix representing an epipolar constraint that is a geometric constraint between two corresponding points of a corresponding point pair and the plurality of corresponding point pairs, and for the base matrix and the first epipole or the second epipole and a constraint that allows the focal length to be decomposed from the fundamental matrix, estimating the fundamental matrix and the focal length from the simultaneous equations represented by
A computer-readable recording medium recording a program containing instructions.
obtaining both the first epipole and the second epipole in obtaining the epipole;
In estimating the base matrix and the focal length, by solving the simultaneous equations using two or more pairs of corresponding points, the first epipole, and the second epipole, the base matrix and the focal length to estimate the focal length,
A computer-readable recording medium according to claim 7.
obtaining one of the first epipole and the second epipole in obtaining the epipole;
In estimating the base matrix and the focal length, the base matrix and the focal length are estimated by solving the simultaneous equations using four or more of the corresponding point pairs and one of the first epipole and the second epipole. estimating a matrix and the focal length;
A computer-readable recording medium according to claim 7.