JP2013246779A - Unified optimal calculation method and program for two-dimensional or three-dimensional geometric transformation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a calculation method for optimally calculating various transformation parameters only by describing an algebraic formula for conditions of constraint without special parameterization for each problem or model, in parameter estimation of two-dimensional/three-dimensional geometric transformation appearing frequently in image processing or image recognition.SOLUTION: Various geometric transformation parameters can be optimally calculated with extended FNS, only by describing an algebraic formula for conditions of constraint without parameterization for each model. In the extended FNS, images captured by two cameras automatically satisfy an internal constraint of scalar equation, called an Epi-double linear equation, where the determinant thereof is 0 and an optimal solution is calculated. The expanded FNS is applied to a vector equation in the two-dimensional/three-dimensional geometric transformation and a plurality of conditions of constraint, so that a solution obtained achieves theoretical limitation of accuracy.

Description

  The present invention relates to a unified optimal calculation method for two-dimensional or three-dimensional geometric transformation, and more particularly to a calculation method applied to the vector equation and a plurality of constraint conditions.

  A method that has been often used in the past to calculate the parameters of a two-dimensional or three-dimensional geometric transformation is the least square method. Parameter estimation is written down as simultaneous equations, resulting in a matrix eigenvalue problem. The calculation algorithm is performed by eigenvalue decomposition or singular value decomposition of a matrix.

  In the “image synthesizing apparatus” disclosed in Patent Document 1 below, in order to generate a panoramic image by synthesizing a plurality of images, it is assumed that geometric deformation between the plurality of images is a two-dimensional affine transformation. The conversion parameter is calculated by the least square method.

  Further, in the following Patent Document 2 (US Patent “GEOMETRIC REGISTRATION OF IMAGES BY SIMILARITY TRANSFORMATION USING TWO REFERENCE POINTS”), a two-dimensional similarity transformation is used to join two images together, and its rotation and translation. It is disclosed to use two reference points to calculate.

  Also, Patent Document 3 “Image processing apparatus” below discloses that a two-dimensional similarity transformation parameter is calculated by a least square method for the synthesis of a plurality of images. When calculating the geometric transformation between images from the correspondence of characteristic points in the image, the image coordinates of the feature points are not related to the extraction method, even if the correct feature points are associated. Uncertainty, so-called observation error, occurs.

  To overcome the observation error, more feature points are taken to use the least squares method. However, the least squares method assumes that all observation errors are isotropic with no directionality and are all uniform.

JP-A-10-126665 US Patent No. 8078004 Japanese Unexamined Patent Publication No. 2007-047552

Kenichi Kanaya and Yasuyuki Shibuya, “Extended FNS Method for Constrained Parameter Estimation” Information Processing Society of Japan, 2007-CVIM-158-4, (2007-3), Kagoshima City (Kagoshima Univ.), 25-32.

  However, in reality, depending on the sensor used for observation, it is not uncommon to have no error characteristics as described above. For example, when three-dimensional restoration is performed by stereo viewing with two cameras, the restoration error increases in the depth direction, and the error distribution is non-uniform.

  In addition, when a sufficient number of feature points cannot be obtained and the transformation parameters must be estimated from only a small number of limited feature points, an estimation method that is as accurate as possible must be used.

  If the data contains an error, the covariance matrix of the estimated solution does not become smaller than a certain value no matter what estimation method is used. In other words, there is a theoretical limit that cannot be exceeded in the accuracy of estimation.

  That is, the least square method is a simple calculation method, but the accuracy of the obtained solution is low. This is because the distribution of errors included in the data is not sufficiently considered. In order to overcome data errors, the Gauss-Newton method or the like is used as an iterative optimization method. However, parameterization must be performed for each model for each problem to be estimated. In particular, it represents a three-dimensional rotation. There are various ways, and it becomes a nonlinear complex expression. If the iteration does not start from a good initial value, it tends to fall into a local solution as the error increases.

  The present invention has been made in view of the above-described problems, and in the parameter estimation of 2D / 3D geometric transformation that often appears in image processing and image recognition, special parameterization is performed for each problem and each model. It is an object of the present invention to propose a calculation method that can optimally calculate various transformation parameters by simply writing down an algebraic constraint condition expression.

  In addition, the present invention can be applied to any geometric transformation, whether 2D / 3D data is translated, rotated, scaled, or combined. An object of the present invention is to propose a calculation method that can use the same calculation method without performing parameterization by special formula modification for each model.

  In addition, since the present invention also supports a plurality of constraint conditions, the present invention proposes a calculation method that has a considerably wide application range and can be applied to parameter calculations other than two-dimensional / three-dimensional geometric transformations. Objective.

  The unified optimal calculation method for two-dimensional or three-dimensional geometric transformation of the present invention is a method for uniformly computing two-dimensional or three-dimensional geometric transformation in image processing, and is a constraint imposed on projective transformation. An extended FNS method that automatically satisfies the conditions is used.

  The program of the present invention is a program that causes a computer to execute the above-described unified optimal calculation method for two-dimensional or three-dimensional geometric transformation.

  The parameter estimation method for image processing or image recognition according to the present invention uses the above-described unified optimal calculation method for two-dimensional or three-dimensional geometric transformation.

  In addition, the motion parameter estimation method in the video processing of the present invention uses the above-described unified optimal calculation method of two-dimensional or three-dimensional geometric transformation.

  The camera parameter estimation method in the virtual studio or AR system of the present invention uses the above-described unified optimal calculation method for two-dimensional or three-dimensional geometric transformation.

  Further, the color correction parameter estimation method in the color correction processing of the present invention uses the above-described unified optimal calculation method for two-dimensional or three-dimensional geometric transformation.

  Moreover, the Gauss-Newton method, which is a conventional parameter calculation method for iterative optimization, has to be specially parameterized for each model to be estimated, and is complicated to apply, such as a complicated nonlinear expression. However, if the extended FNS method according to the present invention is used, parameterization is not performed for each model, and various geometric transformation parameters are optimized by simply writing algebraic constraint conditions. Can be calculated. The extended FNS method has been proposed for the optimal calculation of the basic matrix used for three-dimensional reconstruction from images, but the determinant of a scalar equation called an epi-double-track equation for images taken by two cameras. An optimal solution is calculated by automatically satisfying the internal constraint of 0. The present invention applies this to a vector equation and a plurality of constraints in 2D / 3D geometric transformation, and the calculated solution can achieve the theoretical limit of accuracy.

  Specifically, the 2D / 3D geometric transformation model is expressed by linearization, the maximum likelihood estimation that minimizes the sum of squares of the Mahalanobis distance, the internal constraint of the geometric transformation is expressed by an algebraic expression, A calculation method that automatically satisfies internal constraints is proposed. Furthermore, the evaluation is based on a theoretical limit that represents the reliability of the obtained solution.

  In the present invention, parameter estimation that frequently appears in image processing and image recognition is described by two-dimensional or three-dimensional geometric transformation. Two-dimensional and three-dimensional geometric transformations are the most complex general models with the highest degree of freedom (number of unknown parameters), and by imposing restrictions on them, affine transformations, similarity transformations, and rigid body transformations with lower degrees of freedom. , Rotation conversion and the like. The “extended FNS method”, which is an optimal calculation method that automatically satisfies such constraint conditions, is used. The extended FNS method is proposed for optimal calculation of a basic matrix used for three-dimensional reconstruction from an image. The basic matrix calculates the optimal solution that automatically satisfies the internal constraint that the determinant is 0 for the scalar equation obtained from the images from the two cameras satisfying the epi doublet equation. In the present invention, this is applied to a vector equation and a plurality of constraint conditions in 2D / 3D geometric transformation.

  In parameter estimation of 2D / 3D geometric transformations that often appear in image processing and image recognition, simply write down algebraic constraint conditions without performing special parameterization for each problem or model. Various conversion parameters can be calculated optimally.

  In addition, whether the 2D / 3D data is translated, rotating, changing in scale, combined, or any geometric transformation, a special formula The same calculation method can be used for all models without performing parameterization by deformation for each model.

  In addition, since it supports a plurality of constraint conditions, the application range is considerably wide, and it can be applied to parameter calculation other than two-dimensional / three-dimensional geometric transformation.

It is a figure explaining the example about the hierarchy of 2D / 3D geometric transformation. It is a block diagram explaining the color calibration and color correction processing by a color chart. It is a figure which compares and explains each conversion image about (a) rotation, (b) rigid body, (c) similarity, (d) affine, and (e) projection in two-dimensional geometric transformation. It is a figure explaining each transformation about (a) rotation, (b) rigid body, (c) similarity, (d) affine, and (e) projection in three-dimensional geometric transformation. The RMS error E of the three-dimensional similarity transformation matrix by the extended FNS method, the LM method, and the least square method for the error of the standard deviation σ added to the stereo image is plotted together with the KCR lower bound. For a representative example in the case of σ = 1, the fitting residual J by the extended FNS method and the state of convergence of the constraint condition are plotted against the number of iterations. Extended FNS method taking σ on the horizontal axis, LM method is a plot RMS error _{E R,} E t, the _{E S} for each parameter by the least squares method. It is a figure explaining the change of fitting residual J (paper surface left side) and constraint condition (PHI) m (u) (paper surface right side) in the repetition process of an extended FNS method.

  The present invention expresses the characteristics of the observation error included in the data in a two-dimensional or three-dimensional geometric transformation by a covariance matrix, calculates a solution that minimizes the sum of squares of the Mahalanobis distance, and increases the estimation accuracy. The present invention relates to a calculation method capable of increasing to a theoretical limit.

  Two-dimensional or three-dimensional geometric transformations can be obtained as affine transformations, similarity transformations, rigid transformations, rotation transformations, etc., with a lower degree of freedom by applying projective transformation as a general model with the highest degree of freedom and imposing constraints on it. However, as for the calculation methods used so far, each calculation method in each conversion has been used, such as a projection conversion calculation method for a projective transformation and a similarity conversion calculation method for a similarity transformation. This is because, in particular, there are various methods for expressing the three-dimensional rotation, and the parameter expression for conversion differs depending on the expression method.

  According to the present invention, paying attention to the hierarchical nature of geometric transformation, by simply writing down algebraic constraints, projection transformation to affine transformation, similarity transformation, rigid transformation, rotational transformation, etc., or translation, All geometric transformations by a combination of rotation and scale change can be calculated in a unified manner. FIG. 1 is a diagram for explaining an example of the hierarchy of 2D / 3D geometric conversion.

  Parameter estimation that frequently appears in image processing and image recognition is described by two-dimensional or three-dimensional geometric transformation. Two-dimensional and three-dimensional geometric transformations are the most complex general models with the highest degree of freedom (number of unknown parameters), and by imposing restrictions on them, affine transformations, similarity transformations, and rigid body transformations with lower degrees of freedom. , Rotation conversion and the like. The “extended FNS method”, which is an optimal calculation method that automatically satisfies such constraint conditions, is used.

  The extended FNS method is proposed for optimal calculation of a basic matrix used for three-dimensional reconstruction from an image. The basic matrix calculates an optimal solution that automatically satisfies the internal constraint that the determinant is 0, which is a scalar equation obtained from the fact that the images from the two cameras satisfy the epipolar equation.

  The present invention is typically an algorithm for optimally calculating geometric transformation parameters. The specific calculation method is eigenvalue decomposition of a matrix, and it is preferably a software program that implements this. Although a hardware (FPGA) implementation is possible, it seems that a software program is more suitable for realizing the eigenvalue decomposition of the matrix used for the calculation.

  When image processing or image recognition is performed, it is preferable to perform processing for directly handling pixels by hardware (FPGA), and to pass data obtained as a result to the CPU to perform parameter calculation by software. However, a hybrid configuration in which a part of numerical calculation is handled as an “accelerator” can be considered. As an accelerator, use of hardware (FPGA), GPU, etc. can be considered. In addition, all software implementation by GPU implementation is possible.

  FIG. 2 is a block diagram illustrating color calibration and color correction processing using a color chart. A simple block diagram of an example applied to estimation of color correction parameters and color correction processing in a three-dimensional RGB color space is one example, and the present invention is not limited to only color correction.

  In FIG. 2, the color correction model is a combination of independent gain, offset and rotation in the three-dimensional RGB color space (three-dimensional similarity transformation) for each RGB. Therefore, the degree of freedom (the number of unknown parameters) is 2 × 3 + 3 = 9 degrees of freedom. The degree of freedom of the three-dimensional affine transformation is 12, and there are three internal constraint equations.

(Color correction by three-dimensional similarity transformation)
The three-dimensional affine transformation is expressed by the following equation (1).

  However, Z [•] is a normalization operator in which the fourth component of the vector is 1. When a 13-dimensional vector u in which elements of the 3-dimensional affine transformation matrix (1) are arranged is used, the above equation (1) can be expressed as the following equation (2).

ただし、

であり、ベクトルａ，ｂの内積を（ａ，ｂ）で示すものとする。ｕにはスケールの不定性があるので、

と正規化する。

However,

And the inner product of the vectors a and b is represented by (a, b). Since u has indefiniteness of scale,

And normalize.

In the three-dimensional similarity transformation, the upper left 3 × 3 matrix of the above equation (1) is represented by the following equation (4).

Therefore, the constraint condition for the three-dimensional affine transformation to become the three-dimensional similarity transformation can be expressed by the following equation (5), and the internal constraint equation is expressed by equation (6).

と正規化するので自由度が１だけ減る）が３個の内部拘束式により自由度９となり、これは３次元相似変換の自由度９と一致する。 12 degree of freedom of 3D affine transformation (13D vector u

And the degree of freedom is reduced by 1), the degree of freedom becomes 9 due to the three internal constraint equations, which coincides with the degree of freedom of 3D similarity transformation.

(About unified optimal calculation of 2D / 3D geometric transformation)
A method for optimally calculating two-dimensional and three-dimensional geometric transformations that frequently appear in computer vision problems will be described below. By using projective transformation as a general model and imposing constraints on it, affine transformation, similarity transformation, rigid transformation, and rotation transformation can be obtained. The "extended FNS method" is an optimal calculation method that automatically satisfies such constraints. Is used. In addition, a simulation experiment of three-dimensional similarity transformation in stereo three-dimensional reconstruction data is performed, and it is experimentally shown that the theoretical limit of the calculated solution or the estimation accuracy is achieved even when the error distribution of the data is not uniform.

1. Introduction Many problems that appear in computer vision are formulated as parameter estimation problems for two-dimensional and three-dimensional geometric transformations. The two-dimensional / three-dimensional geometric transformation uses a projective transformation as a general model, and imposes restrictions on parameters to obtain affine transformation, similarity transformation, rigid body transformation, and rotation transformation, which are partial models. Mathematically, it is called a group, and is called a transformation group that forms a set of transformations. When a group includes another group, the included group is said to be a subgroup of the included group. It is known that these partial models are estimated using the Gauss-Newton method or the Levenberg-Marquett method with the gradient method in order to force convergence by parameterizing variously. The Gauss-Newton method, also called bundle adjustment (minimizing reprojection error), is not only for estimating parameters of 2D / 3D geometric transformation, but also for various problems such as camera calibration, calculation of basic matrix, and 3D reconstruction. Is used as a standard non-linear optimization method, but its parameterization varies depending on how to express three-dimensional rotations in particular. It can be a complex nonlinear expression or iterates from an appropriate initial value. Otherwise, it tends to fall into a local solution as the error increases. On the other hand, according to publicly known literature (Mr. Kanaya), the theory of statistical optimization is constructed, and the renormalization method is proposed as a linear solution method for optimal calculation of geometric models, and various methods such as fitting ellipses and three-dimensional reconstruction are proposed. Applied to various problems. After this, the HEIV method and the FNS method have been proposed.

  Even if there are constraints between parameters, Kanaya proposed an optimal correction that takes into account the statistical nature of the error so that the solution can be satisfied without considering such internal constraints. ing. Furthermore, an extended FNS method that extends the FNS method so as to automatically satisfy internal constraints is also proposed. Mr. Kanaya and Mr. Shibuya showed the optimal calculation of the basic matrix by the extended FNS method. Matsunaga et al. Used the extended FNS method for estimating level-constrained color correction parameters by three-dimensional affine transformation in a three-dimensional RGB color space.

  In the present embodiment, affine transformation, similarity transformation, rigid transformation, and rotation transformation obtained by imposing a restriction on projective transformation, which is a general model, using the extended FNS method for parameters of two-dimensional and three-dimensional geometric transformation appearing in computer vision. In addition to formulating a method for optimally calculating the partial model of, we experimentally show that the theoretical limit of the obtained solution or estimation accuracy is achieved.

  In addition, we observe the behavior of the fitting residuals and the convergence of constraints, and compare them with the least squares method and the Levenberg-Marquardt method. Chapter 2 summarizes 2D / 3D geometric transformations. In Chapter 3, based on the theory of statistical optimization by Kanaya, the geometric fitting problem is formulated, and maximum likelihood estimation, estimation accuracy evaluation and theoretical limits are described. Chapter 4 shows the unified optimal calculation procedure using the extended FNS method for 2D / 3D geometric transformation, and Chapter 5 conducts a 3D similarity transformation simulation experiment on 3D reconstruction data by stereo vision, and summarizes it in Chapter 6. I will do it.

から２次元平面上の点

への回転変換は回転行列Ｒを用いて次のように表される。

ここで、Ｒは回転を表す２×２行列であり、回転角θとすると、次のように表される。

３次元の回転にはいろいろな表現法がある。よく使われるのはオイラー角とロール・ピッチ・ヨー（各座標軸周りの回転角）である。一方、回転軸と回転角、四元数による表現もある。３×３行列である回転行列には９個の成分があるが、その自由度は３しかない。

例えば、単位ベクトル

を回転軸として、その周りに正の回転方向に角度Ωだけ回転する３次元回転行列は次のように与えられる。

Ｏｈｔａ氏らは３次元回転を四元数で表現することによりくりこみ法により最適に計算した。Ｎｉｉｔｓｕｍａ氏らはＦＮＳ法により最適に計算した。 2 2D and 3D geometric transformation 2.1 Rotation transformation A point on a 2D plane

A point on the 2D plane from

The rotation conversion to is expressed as follows using the rotation matrix R.

Here, R is a 2 × 2 matrix representing rotation, and is represented as follows, assuming the rotation angle θ.

There are various representations for three-dimensional rotation. Commonly used are Euler angles and roll, pitch, and yaw (rotation angles around each coordinate axis). On the other hand, there is also an expression by a rotation axis, a rotation angle, and a quaternion. The rotation matrix, which is a 3 × 3 matrix, has nine components, but has only three degrees of freedom.

For example, the unit vector

A three-dimensional rotation matrix that rotates around the rotation axis by an angle Ω in the positive rotation direction is given as follows.

Ohta et al. Optimally calculated by the renormalization method by expressing the three-dimensional rotation as a quaternion. Niitsuma et al. Optimally calculated by the FNS method.

から２次元平面上の点

への剛体変換は

と表される。ここで、ｔは新たに加えた２次元の並進ベクトルであり、ｘ方向とｙ方向の並進成分ｔ_ｘ，ｔ_ｙを持つ。
上式（４）は同次座標

を使えば、次のようにひとつの行列により表すことができる。

ここで、Ｚ［・］はベクトルの第３成分を１とする正規化作用素であり、Ｈ_Ｅは次のような３×３行列である。

空間中の点の同次座標を

とすると、３次元の剛体変換は次のように表せる。

Ｚ［・］は３次元の場合は、ベクトルの第４成分を１とする正規化作用素になる。Ｈ_Ｅは３次元回転行列Ｒと３次元並進ベクトルｔからなる次のような４×４行列になる。

2.2 Rigid body transformation Rotation plus translation is called rigid body transformation or Euclidean transformation. Point on 2D plane

A point on the 2D plane from

Rigid body transformation to

It is expressed. Here, t is a newly added two-dimensional translation vector having translation components t _x and t _y in the _x and y directions.
The above equation (4) is homogeneous coordinates

Can be represented by one matrix as follows.

Here, Z [·] is a normalization operator to 1 the third component of the vector, H _E is the following 3 × 3 matrix.

The homogeneous coordinates of a point in space

Then, the three-dimensional rigid transformation can be expressed as follows.

In the case of three dimensions, Z [•] is a normalization operator in which the fourth component of the vector is 1. H _E is the following 4 × 4 matrix consisting of 3-dimensional rotation matrix R and the three-dimensional translation vector t.

ただし、ｓはスケール係数である。相似変換は剛体変換とスケール変換の積により

と表せる。ここで、Ｈｓは相似変換を表す３×３行列であり、次のようになる。

ｓ＝１のとき、相似変換Ｈｓは、剛体変換Ｈ_Ｅと等しくなる。
また、３次元の相似変換は次のように表せる。

ここで、ＨｓはＨ_Ｅにスケール変換Ｓを加えた次のような４×４行列になる。

Ｓは定数倍のスケール係数ｓにより次のように表せる。

2.3 Similarity transformation Two-dimensional rigid transformation plus scale transformation (enlargement / reduction) is called two-dimensional similarity transformation. Since the scale transformation acting on the point x on the two-dimensional plane is equal to the horizontal and vertical directions, it is represented by the following matrix.

Here, s is a scale factor. Similarity transformation is the product of rigid transformation and scale transformation.

It can be expressed. Here, Hs is a 3 × 3 matrix representing similarity transformation, and is as follows.

When s = 1, similarity transformation Hs is equal to the rigid body transformation _{H E.}
The three-dimensional similarity transformation can be expressed as follows.

Here, Hs becomes following 4 × 4 matrix plus scaling S into H _E.

S can be expressed as follows by a scale factor s that is a constant multiple.

ここで、ρはｘ方向に関する拡大（縮小）率であり、ψは回転角である。
２次元の相似変換に対して、このようなせん断変形Ｄを加えたものがアフィン変換である。アフィン変換は次のように表せる。

ここで、Ｈ_Ａは次のような３×３行列である。

アフィン変換で新たに加えたせん断変形Ｄはせん断の方向ψとせん断の大きさρで決まるものであり、その自由度は２である。したがって、アフィン変換の自由度は６である。行列Ｈ_Ａを要素で表すと、第３行が（０，０，１）になり、その左上２×２行列が正則であるような自由度６の任意行列であるとしてもよい。
３次元のアフィン変換は次のように表せる。

行列Ｈ_Ａは第４行が（０，０，０，１）であり、その左上３×３行列が正則であるような任意の４×４行列である。
2.4 Affine transformation A deformation that makes a square a rhombus is called a shear deformation. That is, the shear deformation is a deformation in which the right angle is not maintained but the parallelism is maintained. Such deformation can be performed by rotating the figure at a certain angle on the plane, then enlarging (or reducing) the x-direction, and then rotating it back to the original angle. The matrix D that performs such deformation can be expressed as follows.

Here, ρ is an enlargement (reduction) rate in the x direction, and ψ is a rotation angle.
An affine transformation is obtained by adding such a shear deformation D to a two-dimensional similarity transformation. The affine transformation can be expressed as follows:

Here, _HA is a 3 × 3 matrix as follows.

The shear deformation D newly applied by the affine transformation is determined by the shear direction ψ and the shear magnitude ρ, and the degree of freedom is two. Therefore, the degree of freedom of affine transformation is 6. When the matrix _HA is represented by elements, the third row is (0, 0, 1), and the upper left 2 × 2 matrix may be an arbitrary matrix with 6 degrees of freedom.
The three-dimensional affine transformation can be expressed as follows.

The matrix _HA is an arbitrary 4 × 4 matrix whose fourth row is (0, 0, 0, 1) and whose upper left 3 × 3 matrix is regular.

ここで、Ｂは、

である。
２次元のアフィン変換に扇形変形を加えたものが２次元射影変換である。射影変換は次のように表される。

ここで、Ｈはアフィン変換行列Ｈ_Ａと扇形変形行列Ｂの積であり、次のように表せる。

射影変換は自由度６のアフィン変換に自由度２の扇形変形を加えたものであるから、その自由度は８である。射影変換のもとでは、長さや角度のみでなく平行度も保たれない。
行列Ｈは任意の３×３正則行列であるとしてもよい。３×３行列には９個の要素があるが、すべての要素を定数倍しても全く同じ変換を表すことから、その自由度は８である。
３次元の射影変換は次のように表せる。

Ｈは任意の４×４正則行列である。４×４行列には１６個の要素があるが、すべての要素を定数倍しても全く同じ変換を表すことから、その自由度は１５である。
図３は、２次元幾何学変換における（ａ）回転と（ｂ）剛体と（ｃ）相似と（ｄ）アフィンと（ｅ）射影とについて各変換イメージを比較説明する図である。
2.5 Projective transformation The transformation into a sector is called sector transformation. With fan-shaped deformation, linearity is maintained, but parallelism and orthogonality are not maintained. That is, the sector deformation is a deformation that moves an infinite point on a parallel line to a finite point. Such a deformation is expressed as follows.

Where B is

It is.
A two-dimensional projective transformation is a two-dimensional affine transformation with a sectoral deformation. Projective transformation is expressed as follows.

Here, H is the product of the affine transformation matrix _HA and the sector deformation matrix B, and can be expressed as follows.

Since the projective transformation is an affine transformation with six degrees of freedom plus a fan-shaped deformation with two degrees of freedom, the degree of freedom is eight. Under projective transformation, not only the length and angle but also the parallelism is not maintained.
The matrix H may be an arbitrary 3 × 3 regular matrix. There are 9 elements in the 3 × 3 matrix, but the degree of freedom is 8 because all the elements are multiplied by a constant to represent the same transformation.
The three-dimensional projective transformation can be expressed as follows.

H is an arbitrary 4 × 4 regular matrix. Although there are 16 elements in the 4 × 4 matrix, the degree of freedom is 15 because all elements are expressed by the same conversion even if they are multiplied by a constant.
FIG. 3 is a diagram for comparing and explaining conversion images for (a) rotation, (b) rigid body, (c) similarity, (d) affine, and (e) projection in two-dimensional geometric conversion.

は、それらの誤差がないときの値

が未知のｐ次元パラメータベクトルｕをもつｒ個の拘束条件

を満たすとする。これを（幾何学的）モデルと称する。データ

の定義される空間

をデータ空間、パラメータｕの定義される空間Ｕをパラメータ空間、ｒを拘束条件のランクと称する。ｒ個の方程式（２４）はｘの方程式として互いに代数的に独立で、データ空間

に余次元ｒの（代数）多様体Ｓを定義する。
問題は誤差のあるデータ

からパラメータｕ推定することである。これはデータ空間のＮ個の点に多様体Ｓを当てはめる問題と解釈できる。この問題を幾何学的当てはめと称する。 3 Geometric fit 3.1 Geometric model N m-dimensional data vectors

Is the value when there is no error

R constraints with unknown p-dimensional parameter vector u

Suppose that This is called a (geometric) model. data

The space defined by

Is a data space, a space U in which a parameter u is defined is called a parameter space, and r is called a constraint condition rank. The r equations (24) are algebraically independent from each other as an equation for x, and the data space

Define a (algebraic) manifold S of codimension r.
The problem is erroneous data

To estimate the parameter u. This can be interpreted as a problem of applying the manifold S to N points in the data space. This problem is referred to as geometric fitting.

と第２画像上の点

が対応すれば、その対応が４次元ベクトル

で表せる。３次元空間上の点

と点

が対応すれば、その対応が６次元ベクトル

で表せる。２次元幾何学変換の場合、

に対する独立な方程式は２個であるからｒ＝２となり、３次元幾何学変換の場合、

に対する独立な方程式は３個であるからｒ＝３となり、ｕはそれぞれ２次元／３次元幾何学変換行列の要素を並べたベクトルである。
式（２４）の

は一般にｘの複雑な非線形関数であるが、コンピュータビジョンに現れる多くの問題では、未知パラメータｕに関しては線形であったり、パラメータを付け直して線形に表せることが多い。そのような場合は式（２４）が次の形に表せる。

ここで、

はｍ次元ベクトルからｐ次元ベクトルヘの（一般に非線形の）写像である。以下、ベクトルａ，ｂの内積を（ａ，ｂ）と書く。ｕにはスケールの不定性があるので

と正規化する。 [Example 1] The problem of estimating a 2D / 3D geometric transformation is also a typical geometric fitting problem. In this case, each data is a vector representing the position of a point on a 2D image or 3D space. It is. For example, a point on the first image

And points on the second image

Corresponds to a four-dimensional vector

It can be expressed as Point in 3D space

And point

Corresponds to a 6-dimensional vector

It can be expressed as For 2D geometric transformation,

Since there are two independent equations for, r = 2, and in the case of a three-dimensional geometric transformation,

Since there are three independent equations for, r = 3, and u is a vector in which the elements of the 2D / 3D geometric transformation matrix are arranged.
Of formula (24)

Is generally a complex non-linear function of x, but in many problems appearing in computer vision, the unknown parameter u is often linear or can be linearized by re-parameterizing. In such a case, equation (24) can be expressed in the following form.

here,

Is a (generally non-linear) mapping from an m-dimensional vector to a p-dimensional vector. Hereinafter, the inner product of the vectors a and b is written as (a, b). Because u has indefiniteness of scale

And normalize.

が第２画像の点

に対応するとする。カメラの焦点距離が十分大きい場合には、両者は２次元アフィン変換で結ばれると見なせる。真の特徴点位置

は次の関係を満たす。

ここで、

であり、ｆ_０は要素ｈ_ｉｊのオーダーを揃えるためのほぼ画像サイズの大きさの任意定数である。２次元アフィン変換行列Ｈ_Ａを誤差のあるデータ

から推定する。

と置くと、式（２６）は式（２５）の形に書ける。ｕはアフィン変換行列Ｈ_Ａの要素を並べたベクトルである。これにより、データ空間

が７次元空間

の４次元（代数）多様体として埋め込まれ、パラメータ空間Ｕは

の原点を中心とする６次元単位球面

となる。
ここで、図４は、３次元幾何学変換における（ａ）回転と（ｂ）剛体と（ｃ）相似と（ｄ）アフィンと（ｅ）射影とについての各変換を説明する図である。 [Example 2] N feature points are extracted from two images obtained by photographing the same plane from different positions, and points of the first image are extracted.

Is the point of the second image

Suppose that When the focal length of the camera is sufficiently large, it can be considered that both are connected by two-dimensional affine transformation. True feature point location

Satisfies the following relationship:

here,

And f ₀ is an arbitrary constant of almost the size of the image for aligning the order of the elements h _ij . Two-dimensional affine transformation matrix H _A is data with error

Estimated from

Then, equation (26) can be written in the form of equation (25). u is a vector in which elements of the affine transformation matrix H _A are arranged. This allows the data space

Is a 7-dimensional space

Embedded in a four-dimensional (algebraic) manifold, and the parameter space U is

6-dimensional unit sphere centered at the origin of

It becomes.
Here, FIG. 4 is a diagram for explaining each conversion of (a) rotation, (b) rigid body, (c) similarity, (d) affine, and (e) projection in the three-dimensional geometric conversion.

が点

に対応する場合、真の特徴点位置を

とすると、３次元アフィン変換は２次元の場合同様、式（２６）の関係を満たす。ただし、

３次元アフィン変換行列Ｈ_Ａを誤差のあるデータ

から推定する。

と置くと、３次元アフィン変換も式（２５）の形に書ける。これにより、データ空間

が１３次元空間

の６次元（代数）多様体として埋め込まれ、パラメータ空間Ｕは

の原点を中心とする１２次元単位球面

となる。 [Example 3] Points in 3D space

Is a point

The true feature point position

Then, the three-dimensional affine transformation satisfies the relationship of Expression (26) as in the two-dimensional case. However,

3D affine transformation matrix _HA

Estimated from

Then, the three-dimensional affine transformation can also be written in the form of equation (25). This allows the data space

Is a 13-dimensional space

Embedded in a 6-dimensional (algebraic) manifold, and the parameter space U is

12-dimensional unit sphere centered at the origin of

It becomes.

の誤差の挙動を記述する共分散行列を

とする。以下

を

と略記する。
εは誤差の絶対量を表す定数であり、ノイズレベルと呼ぶ。

は誤差のデータや方向への依存を定性的に表すものであり、正規化共分散行列と称して既知とする。
誤差がデータ毎に独立で正規分布に従うとすると、

の最尤推定は線形化された拘束条件のもとでマハラノビス距離の二乗和

を最小にするように

を推定することである。
データの誤差は小さいと仮定して、線形近似を用いてラグランジュ乗数により拘束条件（２５）を消去すると式（３２）は次のように書ける。

ただし、

は

を（ｋｌ）要素とするｒｘｒ行列の逆行列の（ｋｌ）要素であり、次のように書く。

ここで、

は、

の正規化共分散行列である。
データの誤差は小さいと仮定しているから、

はデータ

の正規化共分散行列

から線形近似によって次のように計算できる。

ただし、

は、

のｍ×ｐヤコビ行列である。 3.2 Maximum likelihood estimation Mapped data

A covariance matrix describing the error behavior of

And Less than

The

Abbreviated.
ε is a constant representing the absolute amount of error and is called a noise level.

Represents qualitatively the dependence of error on data and direction, and is known as a normalized covariance matrix.
If the error is independent for each data and follows a normal distribution,

Is the sum of squares of Mahalanobis distance under linearized constraints

To minimize

Is to estimate.
Assuming that the error of the data is small, if the constraint condition (25) is eliminated by a Lagrange multiplier using linear approximation, equation (32) can be written as follows.

However,

Is

Is the (kl) element of the inverse matrix of the rxr matrix with (kl) element, and is written as follows.

here,

Is

Is the normalized covariance matrix.
Since we assume that the error in the data is small,

Is the data

Normalized covariance matrix of

Can be calculated by linear approximation as follows.

However,

Is

M × p Jacobian matrix.

の正規化共分散行列

は、特に特徴的な誤差の現れ方をしない場合はデフォルト値としてＩ_４×４を用いる。データ

のいずれか一方が誤差を含まない理想的な参照モデルの場合には、

とすればよい。
３次元アフィン変換の場合、６次元ベクトル

の正規化共分散行列

は、特に特徴的な誤差の現れ方をしない場合はデフォルト値としてＩ_６×６を用いる。データ

のいずれか一方が誤差を含まない理想的な参照モデルの場合には、

あるいは

とすればよい。
これらはいずれも誤差の分布が一様等方であることを仮定しているが、センサによっては誤差の分布が不均一な非等方性であることは珍しくない。 [Example 4] Two-dimensional affine transformation, four-dimensional vector

Normalized covariance matrix of

Uses I _{4 × 4} as a default value when no particular error appears. data

If either is an ideal reference model with no error,

And it is sufficient.
For 3D affine transformation, 6D vector

Normalized covariance matrix of

Uses I _{6 × 6} as a default value when no particular error appears. data

If either is an ideal reference model with no error,

Or

And it is sufficient.
All of these assume that the error distribution is uniform and isotropic, but it is not uncommon for some sensors to have non-uniform anisotropy.

の誤差△ｕを次のように定義する。

ここで，Ｐ_Ｕは

にいて、点ｕにおけるパラメータ空間Ｕの接空間

への直交射影行列である。Ｉ_ｐ×ｐはｐ次の単位行列である。これから推定値

の共分散行列が次のように定義できる。

式（３７）は、具体的に次のように計算できる。

ただし、行列Ｍを次のように置いた。

の正規化のためにランクが（ｐ−１）の（ムーア・ペンローズの）一般逆行列となっていることに注意する。
このとき，どのような推定方法で

を計算しても、データに誤差がある限り、

がある値より小さくならない。すなわち、精度には超えることができない理論的な限界が存在する。これを式で書くと、次のようになる。

記号

は左辺引く右辺が半正値対称行列であることを表す。行列

は式（３９）のＭをデータの真値

を用いて計算した値である。式（４０）の右辺はＫＣＲ （Ｋａｎａｔａｎｉ−Ｃｒａｍｅｒ−Ｒａｏ）下界と呼ばれる。最尤推定を行えば、ＫＣＲ下界が

の項を除いて等号で成立する。最尤推定はこの意味で最適な推定法である。 3.3 Accuracy evaluation and theoretical limit Estimated value

Is defined as follows.

Where _PU is

And the tangent space of the parameter space U at the point u

Is an orthogonal projection matrix. I _{p × p} is a p-th order unit matrix. Estimated value

Can be defined as follows.

Equation (37) can be specifically calculated as follows.

However, the matrix M was placed as follows.

Note that it is a generalized inverse matrix (Moore-Penrose) of rank (p-1) for normalization of.
At this time, what kind of estimation method

As long as there is an error in the data,

Is not less than a certain value. In other words, there is a theoretical limit that cannot be exceeded. This can be written as follows:

symbol

Represents that the left side minus the right side is a semi-positive symmetric matrix. matrix

Is the true value of the data in equation (39)

This is a value calculated using. The right side of equation (40) is called the KCR (Kanatani-Cramer-Rao) lower bound. If maximum likelihood estimation is performed, the KCR lower bound

Except for the term of Maximum likelihood estimation is the optimal estimation method in this sense.

と正規化するが、これは一つの内部拘束である。そして、これ以外にｕに次のｑ個の内部拘束が存在するとする。

これらｑ個の式は代数的に独立であるとする。式（４１）はスケール不定のｕに対して成立するとし、各φｍ（ｕ）は

次同次式であると仮定する。最尤推定量

は式（３３）を内部拘束

のもとで最小化することになる。 4 Unified Optimal Calculation of Geometric Transformation 4.1 Internal Constraint u has scale indefiniteness

This is one internal constraint. In addition to this, the following q internal constraints exist in u.

These q expressions are assumed to be algebraically independent. Equation (41) holds for u with an indefinite scale, and each φm (u) is

Assume that the following homogeneous expression. Maximum likelihood estimator

Is the internal constraint for equation (33)

Will be minimized.

としたものである。２次元相似変換行列は次の内部拘束を満たさなければならない。

ここで、Ｉは２次の単位行列である。Ｓを次のように書く。

従って、拘束条件（４２）は次のようになる。

式（２８）の２次元アフィン変換を表す７次元ベクトルｕの要素を用いると、２次元相似変換行列における内部拘束式は次のような２次同次式で書ける。

これにより、データ空間

が７次元空間

の４次元（代数）多様体として埋め込まれ、パラメータ空間Ｕは

の

で定義される４次元（代数）多様体となる。
[Example 5] The two-dimensional similarity transformation matrix is the upper left 2 × 2 matrix of the two-dimensional affine transformation matrix (26).

It is what. The two-dimensional similarity transformation matrix must satisfy the following internal constraints.

Here, I is a secondary unit matrix. Write S as:

Therefore, the constraint condition (42) is as follows.

Using the elements of the seven-dimensional vector u representing the two-dimensional affine transformation of Equation (28), the internal constraint equation in the two-dimensional similarity transformation matrix can be written by the following quadratic homogeneous equation.

This allows the data space

Is a 7-dimensional space

Embedded in a four-dimensional (algebraic) manifold, and the parameter space U is

of

Is a four-dimensional (algebraic) manifold defined by

としたものである。
Ｓを次のように書く。

したがって、拘束条件は次のようになる。

式（３０）の３次元アフィン変換を表す１３次元ベクトルｕの要素を用いると、３次元相似変換行列における内部拘束式は次のような２次同次式で書ける。

これにより、データ空間

が１３次元空間

の６次元（代数）多様体として埋め込まれ、パラメータ空間Ｕは

の

で定義される７次元（代数）多様体となる。 [Example 6] The three-dimensional similarity transformation matrix is the upper left 3 × 3 matrix of the three-dimensional affine transformation matrix (29).

It is what.
Write S as:

Therefore, the constraint conditions are as follows.

Using the elements of the 13-dimensional vector u representing the 3-dimensional affine transformation of Expression (30), the internal constraint expression in the 3-dimensional similarity transformation matrix can be written by the following quadratic homogeneous expression.

This allows the data space

Is a 13-dimensional space

Embedded in a 6-dimensional (algebraic) manifold, and the parameter space U is

of

It is a 7-dimensional (algebraic) manifold defined by.

を満たさなければならない。回転行列Ｒを次のように書くと、

拘束条件は次のようになる。

式（２８）の２次元アフィン変換を表す７次元ベクトルｕの要素を用いると、２次元剛体変換行列における内部拘束式は、２次元相似変換における内部拘束式（４５）と次のような２次同次式になる。

これにより、データ空間

が７次元空間

の４次元（代数）多様体として埋め込まれ、パラメータ空間Ｕは

の

で定義される３次元（代数）多様体となる。
純粋な２次元回転変換行列は、並進ベクトル

になり、データおよびパラメータ空間はそれぞれ５次元空間

に縮退するが、内部拘束式は式（４５）（５１）と同じである。 [Example 7] Two-dimensional rigid transformation matrix is a constraint condition

Must be met. If we write the rotation matrix R as

The constraint conditions are as follows.

Using the elements of the seven-dimensional vector u representing the two-dimensional affine transformation of Equation (28), the internal constraint equation in the two-dimensional rigid transformation matrix is the internal constraint equation (45) in the two-dimensional similarity transformation and the following quadratic: It becomes a homogeneous expression.

This allows the data space

Is a 7-dimensional space

Embedded in a four-dimensional (algebraic) manifold, and the parameter space U is

of

It becomes a three-dimensional (algebraic) manifold defined by.
Pure two-dimensional rotation transformation matrix is a translation vector

The data and parameter space are each a 5-dimensional space

However, the internal constraint equation is the same as the equations (45) and (51).

拘束条件は次のようになる。

式（３０）の３次元アフィン変換を表す１３次元ベクトルｕの要素を用いると、３次元剛体変換行列における内部拘束式は、３次元相似変換における内部拘束式（４８）と次のような２次同次式になる。

これにより、データ空間

が１３次元空間

の６次元（代数）多様体として埋め込まれ、パラメータ空間Ｕは

の

で定義される６次元（代数）多様体となる。
純粋な３次元回転変換行列は、並進ベクトル

になり、データおよびパラメータ空間はそれぞれ１０次元空間

に縮退するが、内部拘束式は式（４８）（５４）と同じである。

以外の内部拘束が存在する場合の推定値

の共分散行列は次のようになる。

ここで、射影行列Ｐ_Ｕは、内部拘束

の勾配ベクトル

にシュミットの直交化を施した正規直交系

を用いると次のように書ける。

ただし、ｐ次元ベクトルｕが内部拘束を受けて、パラメータ空間Ｕがｐ次元空間

の（ｐ−ｑ−１）次元多様体となるため、式（３９）の行列ＭをＰ_ＵＭＰ_Ｕに置き換え、ランクが（ｐ−ｑ−１）の（ムーア・ペンローズの）一般逆行列になる。
[Example 8] A three-dimensional rigid body transformation matrix can be written as a rotation matrix R as follows:

The constraint conditions are as follows.

Using the elements of the 13-dimensional vector u representing the three-dimensional affine transformation of Equation (30), the internal constraint equation in the three-dimensional rigid body transformation matrix is the internal constraint equation (48) in the three-dimensional similarity transformation and the following secondary equation: It becomes a homogeneous expression.

This allows the data space

Is a 13-dimensional space

Embedded in a 6-dimensional (algebraic) manifold, and the parameter space U is

of

It is a 6-dimensional (algebraic) manifold defined by.
Pure three-dimensional rotation transformation matrix is a translation vector

And the data and parameter space are each 10-dimensional space

However, the internal constraint equation is the same as the equations (48) and (54).

Estimated value when there is an internal constraint other than

The covariance matrix of is

Here, the projection matrix P _U is an internal constraint

Gradient vector

Orthonormal system with Schmidt orthogonalization

Can be written as follows.

However, the p-dimensional vector u is subject to internal constraints, and the parameter space U is a p-dimensional space.

Therefore, the matrix M in equation (39) is replaced with P _U MP _U , and the general inverse matrix of (pq-1) (Moore-Penrose) is obtained. Become.

ここで、行列Ｍは式（３９）であり、行列Ｌは次のように置いた。

上式中の

は次のように定義する。

式（３３）を最小にするには式（５７）より

を解けばよく、固有値問題を反復して解くＦＮＳ法を用いることができるが、ｕには内部拘束が存在するので、内部拘束を自動的に満たすようにＦＮＳ法を拡張した拡張ＦＮＳ法の手順を以下に示す。 4.2 Extended FNS method Differentiating equation (33) by u gives the following.

Here, the matrix M is Equation (39), and the matrix L is set as follows.

In the above formula

Is defined as follows.

To minimize Equation (33) from Equation (57)

Can be used, and the FNS method that solves the eigenvalue problem repeatedly can be used. However, since there is an internal constraint in u, the procedure of the extended FNS method in which the FNS method is extended to automatically satisfy the internal constraint. Is shown below.

  1. gives the initial solution for u.

  2. The matrices M and L of the equations (39) and (58) are calculated.

を計算し、これにシュミットの直交化を施した正規直交系

を求める。 3. Internal constraint gradient vector

And an orthonormal system with Schmidt orthogonalization

Ask for.

を計算する。

4). Next projection matrix

Calculate

5. The next matrix Y is calculated.

の小さいｑ＋１個の固有値に対する単位固有ベクトル

を求める。 6). Eigenvalue problem

Unit eigenvectors for small q + 1 eigenvalues

Ask for.

に射影した

を計算する。

7). The current solution u is as follows

Projected onto

Calculate

は単位ベクトルヘの正規化作用素である

。 8). Next u ′ is calculated.

Is the normalization operator to the unit vector

.

なら

を返して終了する。そうでなければ、

としてステップ２に戻る。
ｑ＝０とすると、

であり、ｕ´はＹの最小固有値に対する固有ベクトルとなり、通常の

以外の内部拘束が存在しない場合のＦＮＳ法に帰着する。
9.

Then

Return and exit. Otherwise,

Return to step 2.
If q = 0,

And u ′ is the eigenvector for the smallest eigenvalue of Y,

This results in the FNS method when there is no other internal constraint.

の正規化以外の制約を無視することである。近似解を求めるのによく用いられるのは、最小二乗法である。式（３３）で

（クロネッカのデルタ：ｋ＝ｌで１，それ以外で０）と置くと次のようになる。

これは次のように変形される。

ただし、次のように置いた。

式（６６）はｕの２次形式であるから、これを最小化する単位ベクトルｕは（２次）モーメント行列Ｍ_ＬＳの最小固有値に対する単位固有ベクトルである。 4.3 Least-squares method The iterations of the extended FNS method require initial values, but u have internal constraints. A commonly used simplification is

Ignore constraints other than normalization. The least square method is often used to find an approximate solution. In equation (33)

(Kronecca Delta: 1 if k = 1 and 0 otherwise)

This is transformed as follows.

However, it was placed as follows.

Since equation (66) is a quadratic form of u, the unit vector u that minimizes this is the unit eigenvector for the minimum eigenvalue of the (second-order) moment matrix M _LS .

ただし

はａ回目の試行の３次元相似変換行列の解のベクトル表現であり、Ｐ_Ｕは式（５６）の射影行列である。３次元相似変換における拘束条件は式（４８）より得られる。式（５５）の両辺の平方根を取り、式（３７）より

であることに注意すると、３次元相似変換行列におけるＲＭＳ誤差のＫＣＲ下界が次のように得られる。

比較のために、回転を特異値分解により計算する最小二乗法、リー代数により表現することによるレーベンバーグ・マーカート（ＬＭ）法を適用する。そして、３次元相似変換における回転行列、並進、スケ−ル変化を

とし、それらの真値を

とするとき、回転の誤差は相対回転

の回転角を

（゜）で評価し、並進、スケール変化の誤差をそれぞれ

として評価する。拡張ＦＮＳ法による３次元相似変換行列は特異値分解により回転行列とスケール変化に分解して評価する。 5. Simulation Experiment As shown in FIG. 3, the curved grid is arranged so that the center grid point is at the origin of the world coordinates, and then rotated around a certain rotation axis passing through the origin, translated, and changed in scale. And the three-dimensional position before and after the similarity transformation of each lattice point is measured by stereo vision. The two cameras are arranged at a position where the curved grid is viewed at 10 degrees. It is assumed that the image size is 500 × 800 pixels and the focal length is 600 pixels. Using each grid point as a corresponding point, a normal random number with an expected value of 0 and a standard deviation σ pixel is added to the x and y coordinates, and the three-dimensional position of each grid point is optimally calculated by the method of Kanatani et al. The normalized covariance matrix is predicted by the method of Niitsuma et al. Using the three-dimensional restoration position as data, and the three-dimensional similarity transformation matrix is calculated by applying the extended FNS method described in the previous section. For the initial value of the extended FNS method, a solution by the least square method ignoring the constraints described in the previous section is used.
The calculated three-dimensional similarity transformation matrix is expressed in the form of a unit vector u in Expression (30), and the error is calculated from Expression (36). This is tried 1000 times with different errors for each σ, and the square mean square (RMS) error is evaluated as follows.

However,

Is a vector representation of the solution of the three-dimensional similarity transformation matrix of the a-th trial, and _PU is the projection matrix of Equation (56). The constraint condition in the three-dimensional similarity transformation is obtained from equation (48). Taking the square root of both sides of Equation (55), from Equation (37)

Note that the KCR lower bound of the RMS error in the three-dimensional similarity transformation matrix is obtained as follows.

For comparison, the least square method for calculating the rotation by singular value decomposition and the Levenberg-Markert (LM) method by expressing it by the Lie algebra are applied. And the rotation matrix, translation, and scale change in 3D similarity transformation

And the true value of them

The rotation error is relative rotation

Rotation angle

(°) to evaluate the translation and scale change errors.

Evaluate as The three-dimensional similarity transformation matrix by the extended FNS method is evaluated by decomposing it into a rotation matrix and a scale change by singular value decomposition.

  FIG. 5 is a plot of the RMS error E of the three-dimensional similarity transformation matrix by the extended FNS method, the LM method, and the least square method with respect to the error of the standard deviation σ added to the stereo image together with the KCR lower bound. As can be understood from FIG. 5, the least-square method does not take into account the non-uniformity of the error distribution of the three-dimensional restored data, and therefore has a very low accuracy. On the other hand, the extended FNS method is almost the same as the LM method, and it can be seen that the theoretical limit of accuracy is almost reached.

の収束の様子を反復回数に対してプロットしたものである。すべての拘束条件が急速に０に収束している。 FIG. 6 shows a typical example in the case of σ = 1, the fitting residual J by the extended FNS method and the constraint condition.

The state of convergence of is plotted against the number of iterations. All constraint conditions converge rapidly to zero.

Figure 7 is a plot of the horizontal axis taken extended FNS method, LM method, RMS error _E R of each parameter by the least squares _method, E t, the _{E S} the sigma. As can be seen from FIG. 7, the accuracy of all parameters is extremely low in the least square method. When the LM method or the extended FNS method is used, the accuracy is remarkably improved.

FIG. 8 is a diagram for explaining changes in the fitting residual J (left side of the paper) and the constraint condition Φm (u) (right side of the paper) in the iterative process of the extended FNS method.

6 Summary The 2D / 3D geometric transformation, the geometric fitting problem, the maximum likelihood estimation, the estimation accuracy evaluation and the theoretical limits that appear in computer vision are summarized. A unified optimal calculation method using the extended FNS method for 2D / 3D geometric transformation parameters was presented. The simulation experiment of the three-dimensional similarity transformation in the three-dimensional reconstruction data by stereo vision showed that the solution obtained by the extended FNS method achieved the theoretical limit of the estimation accuracy. We observed the behavior of the fitting residuals and the convergence of constraints, and also compared the least squares method and the Leppenberg-Marquett method. The extended FNS method can be constrained by the same calculation procedure as long as it simply writes an algebraic constraint expression without performing complicated expression transformation by sophisticated parameterization for each model of 2D / 3D geometric transformation. The optimal estimation result of the partial model with the condition is obtained. Since it is possible to cope with a plurality of constraint conditions, the applicable range is considerably wide. It is also possible to select an optimal model from a plurality of different partial models estimated by the extended FNS method.

  The content described in the embodiment is not limited to the description in the embodiment, and the configuration can be changed and combined as appropriate within the obvious range, and the processing can be changed as appropriate.

  The two-dimensional / three-dimensional geometric transformation optimal calculation method described in the present embodiment calculates two-dimensional and three-dimensional geometric transformations that frequently appear in image processing and image recognition. Is the method. The affine transformation, similarity transformation, rigid transformation, and rotation transformation can be obtained by imposing the projective transformation as the most complex and highly flexible general model and imposing constraints on it, but the optimal calculation method that automatically satisfies such constraints The extended FNS method is used. If there is an error in the data, the parameter estimation will not be below a certain value using any method. In other words, there is a theoretical limit of accuracy, but this method achieves that theoretical limit.

  The present invention can be applied to a system conversion apparatus and video processing in general for the broadcasting equipment field. It can be applied to various image processing and overall parameter estimation in image recognition, and in particular, motion parameter estimation in video processing, camera parameter estimation in a virtual studio / AR system, color correction parameter estimation processing in color correction processing, etc. It can be suitably deployed.

210... Color chart recognition processing unit 220... Color correction parameter calculation processing unit 230.

Claims (6)

A method for uniformly and optimally calculating a two-dimensional or three-dimensional geometric transformation in image processing,
A unified optimal calculation method for two-dimensional or three-dimensional geometric transformation, characterized by using an extended FNS method that automatically satisfies the constraints imposed on projective transformation.   The program which makes a computer perform the unified optimal calculation method of the two-dimensional or three-dimensional geometric transformation of Claim 1.   A parameter estimation method in image processing or image recognition using the unified optimal calculation method for two-dimensional or three-dimensional geometric transformation according to claim 1.   A motion parameter estimation method in video processing using the unified optimal calculation method for two-dimensional or three-dimensional geometric transformation according to claim 1.   A camera parameter estimation method in a virtual studio or AR system using the unified optimal calculation method for two-dimensional or three-dimensional geometric transformation according to claim 1. A color correction parameter estimation method in color correction processing using the unified optimal calculation method for two-dimensional or three-dimensional geometric transformation according to claim 1.

Images