JP3901552B2 - Omnidirectional camera viewpoint movement and object shape restoration method, apparatus, omnidirectional camera viewpoint movement and object shape restoration program, and recording medium recording the program - Google Patents

Description

【００３８】
一方、外界の物体の点は、この全方位カメラの光学的性質により、全方位画像上において、図５に示すようにＷ×Ｈの画像サイズで、Ｈ／２の半径のイメージサークルより内側の点（ｘ_ij，ｙ_ij）に射影される。ここで、Ｘ₀軸との為す角ρ_ijとすると、光学的性質から、式（３）が成り立つ。
【００３９】
【数２】

【００４０】
これで、外界の物体と全方位カメラ視点の位置の幾何学的配置、全方位カメラの光学的性質、並びに、カメラ内部パラメータを定義しておく。
【００４１】
図１，図２にもどり、本発明の説明を続ける。
【００４２】
始めに、特徴点計測部２は、時系列画像が時間管理で格納されている時系列画像データベース１から時系列画像を取り出して入力し、図２での時系列画像処理のステップを行う。
【００４３】
これらのステップでは、まず、時系列画像データベース１から時系列画像を１枚取り出し、これを初期画像として、その画像上に特徴点を配置する。このとき、エッジ検出、ハフ変換、並びに、濃淡の２次元勾配などの画像処理により自動的に特徴点を配置するなどして、初期画像上に特徴点を配置する。このとき配置する特徴点の数をＰ個（ｊ＝１，２，…，Ｐ）とし、配置したときの特徴点の２次元座標値として、図５での画像座標系での原点からの座標値（ｘ_ij，ｙ_ij）：ｊ＝１，２，…，Ｐを記録しておく。
【００４４】
次に、特徴点計測部２は、初期画像に続く時系列画像をデータベース１から１枚ずつ読み込み、時系列画像処理のステップにて、初期画像に配置した特徴点を、時系列画像間の濃淡の変化などに着目した手法などを利用することで画像追跡し、各時系列画像（初期画像から第ｉ番目の画像）の特徴点の画像座標値として、図５での画像座標系での原点からの２次元座標値（ｘ_ij，ｙ_ij）を記録する。時系列画像を読み出し続けた場合、初期画像に配置した特徴点の中で、画像中から消失したり、オクルージョンなどにより隠れてしまったときは、画像追跡を停止し、特徴点追跡を終了する。特徴点追跡が終了した時点で、読み出した時系列画像の数ｉ＝１，２，…，Ｆは、初期画像を含めてＦ枚とする。
【００４５】
次に、特徴点計測部２は、計測行列記憶を行う。計測行列記憶のステップでは、各時系列画像における特徴点の時間的な画像座標的配置の変化量が記録される。特徴点の時間的な画像座標的配置の変化を行列としてデータ化したものを計測行列［Ｘ］と称し、式（４）のデータ形式とする。
【００４６】
【数３】

【００４７】
次に、計測行列変換部３は、計測行列変換のステップにて、全方位画像座標値での計測行列に変換する。ここでの全方位画像座標値とは、図５に示すように、画像中心、すなわち、光軸の位置を示し、光学中心の座標値（Ｃ_x，Ｃ_y）とした場合、この座標値からの相対的な座標値に一時変換しておく。さらに、これを全時系列にわたり、式（２）（３）のρ_ijとΦ_ijを全特徴点から測定し、式（５）で定義するような全方位画像平面上の座標値へ幾何変換する。これを式（６）のようなデータ配列にする。
【００４８】
ｕ_ij＝cot（φ_ij）cos（ρ_ij） …式（５（その１））
ｕ_ij＝cot（φ_ij）sin（ρ_ij） …式（５（その２））
【００４９】
【数４】

【００５０】
次に、変換された計測行列は計測行列入力部４により行列分解処理部５に入力され、行列分解処理部２において因子分解法処理される。この因子分解法処理のステップでは、全方位カメラ視点の運動と外界の物体の形状、または構造情報を復元する。図３に、因子分解法処理での処理フローを示す。
【００５１】
まず、それまでのステップで全方位画像平面上へ変換した計測行列［Ａ］を、計測行列入力部４による計測行列データ入力において読み込む。次に、特異値分解なる数学的手法により、式（７）のように、一意に、３つの行列に行列分解する。
【００５２】
【数５】

【００５３】
さらに、行列分解した後、ランク３により、さらに行列を式（８）のように分離する。式（８）の第二項を雑音成分と見なして、式（９）のように雑音除去する。
【００５４】
【数６】

【００５５】
次に、雑音成分が除去された運動行列と形状行列とを獲得し、運動行列についてカメラ運動に関する拘束条件を設定し、これを満足するように、［Ｑ］を決定する。すなわち、式（１２）に示すように、行列［Ｑ］を作用させた後の行列を［Ｍ］とし、各行についてｍ_i，ｎ_iとし、要素が式（１３）であるとする。この行列の要素ｍ_i，ｎ_iが、式（１４），（１５），（１６）の拘束条件を満足するようにする。ただし、行列［Ｑ］の要素を、式（１７）に示す要素とする。このとき、式（１２）の要素から、式（１４）から式（１８）を経て（１９）の計算式を設定する。また、式（１２）の要素から、式（１５）から式（２０）を経て（２１）の計算式を設定する。また、式（１２）の要素から、式（１６）から式（２２）を経て（２３）の計算式を設定する。式（１９），（２１），（２３）から、式（２４）の行列［Ｑ］［Ｑ］^Tを求める計算式を設定する。行列［Ｑ］［Ｑ］^Tを求めるときは、式（２５）の演算を行い、この計算で得た要素から、式（２６）のように固有値分解し、式（２７）であるとする。次に、式（２８）を使って、式（２５）で得た元の行列［Ｑ］の要素Ｑ₁₁，Ｑ₁₂，Ｑ₂₁，Ｑ₂₂を求める。
【００５６】
【数７】

【００５７】
ｍ² _ix＋ｍ² _iy＝１ …式（１４）
ｎ² _ix＋ｎ² _iy＝１ …式（１５）
ｍ_ixｎ_ix＋ｍ_iyｎ_iy＝０ …式（１６）
【００５８】
【数８】

【００５９】
【数９】

【００６０】
【数１０】

【００６１】
または、式（２６）の固有値分解を行い、式（２６）の右辺の２番目の行列の固有値の平方根を対角要素とした行列、すなわち、式（２８）の右辺の１番目の行列を求める。次に、式（２８−１）に示すように固有ベクトルの符号判定を行い、（ｑ_ix，ｑ_iy）ｉ＝１，２を決定し、式（２８）により［Ｑ］を求める。
【００６２】
次に、変換行列算出部６は、求めた行列［Ｑ］によって、式（２９）により運動行列の変換処理を行い、並びに、式（３０）により形状行列変換の処理を行う。これで得られた運動成分［Ｍ'］、並びに、物体形状成分［Ｓ'］を出力し、因子分解法処理を終える。
【００６３】
【数１１】

【００６４】
図１、図２に戻り、回転運動復元部１１は回転成分計算を、平面並進運動復元部１０は平面並進成分計算を、並びに空間情報復元分解８は空間情報復元計算を、それぞれ行う。
【００６５】
まず、視点運動抽出部９により、全フレームでの回転成分を抽出し、回転運動復元部１１により、初期画像での運動成分が式（３１）となるように、全フレームにおいて、方位正規化計算をする。
【００６６】
【数１２】

【００６７】
すなわち、まず、式（２９）で求めた行列［Ｍ］について、式（３２）の運動成分とし、初期フレームでの行ベクトルを、式（３３）のように取り出す。次に、このベクトルに直交するベクトルｋ₁を式（３４）のように生成し、これらベクトルを要素とする行列［Ｎ］を生成する。この行列を使って、式（３６）のように変換をすることで、方位正規化計算を行い、行列［Ｍ］を得る。
【００６８】
ｉ^T ₁＝［Ｕ'₁₁ Ｕ'₁₂ ０］ …式（３３（その１））
ｊ^T ₁＝［Ｕ'_1+F,₁ Ｕ'_1+F,₂ ０］ …式（３３（その２））
ｋ^T ₁＝［０ ０ １］］ …式（３４）
【００６９】
【数１３】

【００７０】
次に、回転運動復元部１１は、各フレームに対して、式（３７）、または、式（３８）を使って方位回転θ_iを計算する。
【００７１】
【数１４】

【００７２】
一方、図４では、物体形状抽出部７により並進成分を抽出して、平面並進運動復元部１０により、各フレームでの成分から、平面運動としての並進成分を計算する。ここでは、式（１３）の第３列目の要素を抽出し、各フレームに対して、式（３９）に従って、平面運動での並進成分（Ｔ⁽ⁱ⁾ｘ，Ｔ⁽ⁱ⁾ｙ）を復元する。
【００７３】
【数１５】

【００７４】
これで、図４の回転成分計算、および並進運動計算を終了する。
【００７５】
続いて、図１、図２に戻り、空間情報復元部８による空間情報復元処理を行う。ここでの空間情報復元処理では、式（３０）の行列要素が、式（４０）であるとすると、式（４１）に従って、外界の物体形状、または、構造に関する空間情報を表現する３次元座標値が復元できる。
【００７６】
以上、本発明の実施形態例により、全方位映像の特徴点の時間的動きから、全方位カメラ視点の運動、すなわち、方位運動とＸＹ平面での並進運動、並びに、外界の物体の形状、または構造の空間情報、すなわち、３次元座標値を復元することができる。
【００７７】
【数１６】

【００７８】
なお、本発明では、全方位光学系において、式（２）に示す光軸からの入射角に比例した光学射影の場合、時系列の全方位画像上の特徴点座標値について、原点と設定した位置から、式（２）の全方位射影式を使って仰角のφ_ijと位相ρ_ijを求め、式（５（その１））と式（５（その２））で表現する座標系へ幾何変換し、この座標平面上での特徴点座標値の計測行列［Ａ］から行列分解を行うようにしても良いし、式（４２）に示す光軸からの入射角の半角の正弦に比例した光学射影の場合、時系列の全方位画像上の特徴点座標値について、原点と設定した位置から、式（４２）の全方位射影式を使って仰角のφ_ijと位相ρ_ijを求め、式（５（その１））と式（５（その２））で表現する座標系へ幾何変換し、この座標平面上での特徴点座標値の計測行列［Ａ］から行列分解を行うようにしても良いし、式（４３）に示す光軸からの入射角の半角の正接に比例した光学射影の場合、時系列の全方位画像上の特徴点座標値について、原点と設定した位置から、式（４３）の全方位射影式を使って仰角のφ_ijと位相ρ_ijを求め、式（５（その１））と式（５（その２））で表現する座標系へ幾何変換し、この座標平面上での特徴点座標値の計測行列［Ａ］から行列分解を行うようにしても良い。
【００７９】
【数１７】

【００８０】
なお、図１で示した装置における各部の一部もしくは全部の機能をコンピュータのプログラムで構成し、そのプログラムをコンピュータを用いて実行して本発明を実現することができること、あるいは、図２〜図４で示した処理の手順をコンピュータのプログラムで構成し、そのプログラムをコンピュータに実行させることができることは言うまでもなく、コンピュータでその機能を実現するためのプログラム、あるいは、コンピュータにその処理の手順を実行させるためのプログラムを、そのコンピュータが読み取り可能な記録媒体、例えば、ＦＤ（フロッピーディスク（登録商標））や、ＭＯ、ＲＯＭ、メモリカード、ＣＤ、ＤＶＤ、リムーバブルディスクなどに記録して、保存したり、配布したりすることが可能である。また、上記のプログラムをインターネットや電子メールなど、ネットワークを通して提供することも可能である。
【００８１】
【発明の効果】
本発明によれば、全方位カメラを使って取得した時系列画像全般（移動手段を利用して撮影した車載映像、海上映像、空撮映像、屋内映像など）から、対象物に関する物体形状等の空間情報を高精度に獲得、復元することが可能となる。また、これまでの測量技術並の高精度な３次元立体視が可能となる。さらに、全方位カメラ視点のカメラ光軸周りの回転、並びに、ＸＹ平面運動からなる方位運動を正確に復元することが可能となる。
【図面の簡単な説明】
【図１】本発明の基本的な実施形態例を示す構成図である。
【図２】本実施形態例での処理フローを説明する図である。
【図３】本実施形態例での因子分解法処理フローを説明する図である。
【図４】本実施形態例でのカメラ視点の回転運動、及び並進運動の復元を説明する図である。
【図５】全方位画像面の座標値を説明する図である。
【図６】ＸＹ平面でのカメラパラメータを説明する図である。
【図７】垂直平面とＹＺ平面、光学系の関係を説明する図である。
【図８】等距離射影を説明する図である。
【符号の説明】
１…時系列画像データベース
２…特徴点計測部
３…計測行列変換部
４…計測行列入力部
５…行列分解処理部
６…変換行列算出部
７…物体形状抽出部
８…空間情報復元部
９…視点運動抽出部
１０…平面並進運動復元部
１１…回転運動復元部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to time series image data acquired by an image input device or the like, and measurement or acquisition and restoration of a spatial shape or a spatial structure of an object, and motion related to an optical viewpoint of an image input device (camera). In addition, it relates to restoring the temporal motion of the subject (object). In particular, it relates to camera movement and object shape restoration in computer vision.
[0002]
In addition, the present invention can be used for in-vehicle images acquired using an omnidirectional camera, marine video from a ship, time series images such as aerial photography, and in particular, posture and plane motion related to an omnidirectional camera viewpoint. In addition, it relates to the restoration of spatial information such as the shape of the outside world reflected in the time-series video, that is, the shape of the appearance of the subject (object).
[0003]
[Prior art]
In the computer vision field, methods for measuring or acquiring the shape of an object from time-series image data include three-dimensional analysis methods using stereo measurement and epipolar plane analysis. Recently, there is a factorization method (the following document [1]) as a representative method for simultaneously measuring or acquiring three-dimensional information related to camera motion and the shape of a subject (object). According to these methods, it is possible to acquire and restore information related to the spatial shape or spatial structure and motion related to the camera viewpoint from a plurality of time-series images in which the object is photographed.
[0004]
Reference [1] C. Tomasi and T. Kanade; “Shape and Motion from Image Streams Under Orthography: A Factorization Method”, International Journal of Computer Vision, Vol. 9, No. 2, 1992.
[0005]
However, it is difficult to obtain images seamlessly in time-series images taken by moving the shooting camera using moving means, etc. due to the shooting environment and minute movements of the shooting camera. In some cases, random noise is mixed in and it is difficult to accurately restore the camera motion.
[0006]
Furthermore, camera motion and object shape reconstruction in Euclidean space can be performed by factorization from image coordinate values that can be expressed in an orthogonal coordinate system with a plane image plane. In order to frame-in or frame-out, the rotational motion of the image input device is restored globally or in the long-term, and the shape of the external object is large-scaled, globally, or long-term It is difficult to restore. On the other hand, in order to avoid such a problem of a finite field angle, as a means for acquiring a 360-degree panoramic image at a time, the camera motion and the object shape are restored from a time-series image acquired using an omnidirectional camera. Can be considered. However, conventional approaches based on epipolar analysis and stereo vision have been applied to time-series omnidirectional images. From time-series images with noise added, the camera motion and object shape in Euclidean space can be easily detected. There are problems such as inability to restore.
[0007]
By the way, outdoors, it is possible to sense a state related to position information and posture information of image input by using a sensing device such as GPS, differential GPS, and gyro, which has become highly accurate in recent years.
[0008]
[Problems to be solved by the invention]
In general, when simultaneously restoring camera movement and the shape of an object from a time-series image acquired from an image input device (camera), the influence of random noise mixed in the time-series image or the minuteness of the camera at the time of shooting It is difficult to accurately restore correct movements. In order to deal with such a problem, the above-described factorization method (reference [1]) exists in computer vision.
[0009]
On the other hand, position information measurement using a GPS device or the like is limited to a geometrical arrangement of GPS satellites or a time zone with good sensing conditions. In addition, there is a problem that a place where trees or high-rise buildings stand in the environment surrounding position measurement is a disadvantageous condition for sensing.
[0010]
In addition, in a time-series image acquired in a shooting environment that visualizes a 360-degree landscape at a time, such as an omnidirectional camera, using a moving means, a measurement method that applies the principle of stereo vision can be used to Spatial information can be acquired and restored, but since the camera moves minutely depending on the arrangement of the moving means and the imaging device and the shooting environment, seamless time-series images cannot be easily acquired. Therefore, the influence of random noise is large, and it is impossible to restore the camera movement and the shape of the object at the same time and with high accuracy at all times.
[0011]
The present invention provides position information that cannot be measured even by surveying using remote sensing, etc., and spatial noise of external objects using methods such as stereo vision. It is an object of the present invention to provide a method and apparatus for accurately restoring temporal fluctuations relating to the posture and position of a camera viewpoint, that is, reconstructing motion, and acquiring and restoring spatial information of external objects.
[0012]
[Means for Solving the Problems]
In order to solve the above-described problems, the present invention relates to the movement of the omnidirectional camera viewpoint regarding the time series in the time-series omnidirectional image obtained by the image input means, and the shape or structure of the external object as the subject. Or a method of reconstructing spatial information including appearance, in an image coordinate system set on a time-series image, measuring a temporal variation amount of a feature point in each time-series image, and calculating the feature point coordinate value A first step of converting to another coordinate system; and a measurement matrix, which is data obtained by aggregating the converted feature point coordinate values, is subjected to matrix decomposition, a motion matrix relating to the camera viewpoint, and a shape matrix expressing the appearance of an object in the outside world In the second step of decomposing, and in the decomposed motion matrix, a transformation matrix is obtained so as to satisfy the conditions set for defining motion, and the transformation matrix is used to Information corresponding to rolling motion, a third step of calculating and restoring information corresponding to translational motion, a fourth step of restoring a shape matrix expressing the appearance of the object using the transformation matrix, and the restoration The fifth step of restoring the spatial information of the object in the outside world using the shape matrix thus obtained is used as an omnidirectional camera viewpoint motion and object shape restoration method.
[0013]
Alternatively, in the omnidirectional camera viewpoint motion restoration and object shape restoration method described above, the second step is a matrix decomposition by factorization of a measurement matrix that is data obtained by tabulating converted feature point coordinate values; A omnidirectional camera viewpoint motion and object shape restoration method comprising the steps of denoising from the decomposed matrix and obtaining a motion matrix and a shape matrix from the denoised matrix .
[0014]
Alternatively, in the omnidirectional camera viewpoint motion restoration and object shape restoration method described above, in the first step, in converting the feature point coordinate value in each time-series image into another coordinate system, Based on the omnidirectional projection proportional to the incident angle from the optical axis, the image coordinate value is converted into a coordinate value of another coordinate system, and in the second step, the matrix is decomposed by factorization. An omnidirectional camera viewpoint motion and object shape restoration method characterized in that factorization is performed on a measurement matrix consisting of values.
[0015]
Alternatively, in the omnidirectional camera viewpoint motion restoration and object shape restoration method described above, in the first step, in converting the feature point coordinate value in each time-series image into another coordinate system, , Based on the projection proportional to the sine of the half angle of the incident angle from the optical axis, the image coordinate value is converted into a coordinate value of another coordinate system, and the conversion is performed in the matrix decomposition by factorization in the second step. An omnidirectional camera viewpoint motion and object shape restoration method characterized in that factorization is performed on a measurement matrix composed of coordinate values obtained.
[0016]
Alternatively, in the omnidirectional camera viewpoint motion restoration and object shape restoration method described above, in the first step, in converting the feature point coordinate value in each time-series image into another coordinate system, The image coordinate value is converted to a coordinate value of another coordinate system based on the projection proportional to the tangent of the half angle of the incident angle from the optical axis, and the conversion is performed in the matrix decomposition by factorization in the second step. An omnidirectional camera viewpoint motion and object shape restoration method characterized in that factorization is performed on a measurement matrix composed of coordinate values obtained.
[0017]
Alternatively, in the omnidirectional camera viewpoint motion restoration and object shape restoration method described above, in the first step, when converting the feature point coordinate values in each time series image to another coordinate system, the time series omnidirectional The coordinate value of the feature point on the image is geometrically transformed into a coordinate system expressed by the phase from the origin and the set position, and in the second step, matrix decomposition is performed from the measurement matrix of the feature point coordinate value on the phase plane. The omnidirectional camera viewpoint movement and the object shape restoration method characterized by the above are used as means.
[0018]
Alternatively, in the omnidirectional camera viewpoint motion restoration and object shape restoration method, the fourth step uses the transformation matrix obtained in the third step when calculating and restoring information corresponding to the rotational motion. In the obtained motion matrix, an omnidirectional camera viewpoint motion and object shape restoration method characterized by restoring rotational motion around the optical axis is used as means.
[0019]
Alternatively, in the omnidirectional camera viewpoint motion restoration and object shape restoration method described above, in the fourth step, when calculating and restoring information corresponding to the translational motion, the transformation matrix obtained in the third step is used. The omnidirectional camera viewpoint is characterized by taking out information corresponding to the translational movement of the omnidirectional camera viewpoint and converting it into a planar movement in the image coordinate system set when the coordinate values of the feature points on the time-series image are measured The motion and object shape restoration method is used as means.
[0020]
Alternatively, in the omnidirectional camera viewpoint motion restoration and object shape restoration method described above, in the fifth step, one component in the shape matrix obtained by converting the shape matrix representing the appearance of the object obtained in the third step And omnidirectional camera viewpoint motion and object shape restoration method characterized in that the spatial information of feature points is restored.
[0021]
Alternatively, in the above-described omnidirectional camera viewpoint motion restoration and object shape restoration method, in the third step, when obtaining a transformation matrix, for a matrix element satisfying the conditional expression, its principal component and its principal component An omnidirectional camera viewpoint motion and object shape restoration method characterized in that a transformation matrix is obtained from such a code vector.
[0022]
Alternatively, in a time-series omnidirectional image acquired by an image input device, it is a device that restores spatial information including the motion of the omnidirectional camera viewpoint with respect to time-series and the shape, structure, or appearance of an external object. In the image coordinate system set on the time series image, a feature point measurement unit that measures the temporal variation of the feature point in each time series image, and the feature point coordinate value is converted into another coordinate system. A shape that represents the appearance of the motion matrix related to the camera viewpoint and the appearance of the object in the outside world by decomposing the measurement matrix, which is the data summing up the converted feature point coordinate values, decomposing the matrix, and removing noise A matrix decomposition processing unit for decomposing the matrix, a transformation matrix calculating unit for obtaining a transformation matrix so as to satisfy a condition set for defining motion in the decomposed motion matrix, and using the transformation matrix, Bearing Using the transformation matrix, the shape matrix expressing the appearance of the object is restored by using the transformation matrix and the information corresponding to the rotational movement of the viewpoint and the viewpoint movement restoring unit that calculates and restores the information corresponding to the translational movement. An omnidirectional camera viewpoint movement and object shape restoration device characterized by having a spatial information restoration unit that restores the spatial information of an object in the outside world using the shape matrix thus formed.
[0023]
Alternatively, an omnidirectional camera viewpoint movement and object shape restoration program is provided as a program for causing a computer to execute the steps in the omnidirectional camera viewpoint movement and object shape restoration method.
[0024]
Alternatively, an omnidirectional camera viewpoint characterized in that a program for causing a computer to execute the steps in the omnidirectional camera viewpoint movement and object shape restoration method described above is recorded on a recording medium readable by the computer. A recording medium on which a motion and object shape restoration program is recorded is used as means.
[0025]
In the present invention, in the omnidirectional projection formula using the optical properties of the omnidirectional camera, by applying the factorization method (reference [1]), the temporal movement of the feature points measured from the time-series images, that is, Conditions for reconstructing camera motion by removing random noise from temporal fluctuation values in omnidirectional space, and decomposing the camera motion into omnidirectional camera viewpoints and elements representing spatial information of external objects Using the transformation matrix that satisfies this conditional expression, acquire and restore motions consisting of posture and position related to the omnidirectional camera viewpoint and spatial information of external objects, that is, object shape, structure, appearance, etc. This makes it possible to accurately determine the shape of an object related to an object from all time-series images acquired using an omnidirectional camera (vehicle-mounted video, sea video, aerial video, indoor video, etc. taken using a moving means). Obtained makes it possible to restore. In addition, high-accuracy three-dimensional stereoscopic viewing similar to conventional surveying techniques is possible. Furthermore, it is possible to accurately restore the rotation of the omnidirectional camera viewpoint around the camera optical axis and the camera motion including the XY plane motion.
[0026]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0027]
FIG. 1 is a block diagram showing an embodiment of the present invention. In this embodiment, a case will be described in which an omnidirectional camera imaging apparatus incorporating an optical system of equidistant projection (omnidirectional projection proportional to the incident angle from the optical axis) using a fisheye lens or the like is described. The present invention can also be applied to a time-series image acquired by an omnidirectional camera imaging apparatus incorporating an optical system of other omnidirectional projection (equilateral solid angle projection, stereoscopic projection, etc.).
[0028]
In FIG. 1, 1 is a time-series image database, 2 is a feature point measurement unit, 3 is a measurement matrix conversion unit, 4 is a measurement matrix input unit, 5 is a matrix decomposition processing unit, 6 is a transformation matrix calculation unit, and 7 is an object shape. An extraction unit, 8 is a spatial information restoration unit, 9 is a viewpoint motion extraction unit, 10 is a plane translational motion restoration unit, and 11 is a rotational motion restoration unit.
[0029]
In the present embodiment, the feature point measurement unit 2 measures temporal variation of feature points in the time-series omnidirectional images stored in the time-series image database 1, and the measurement matrix conversion unit 3 measures the omnidirectional image plane. The measurement matrix that is the aggregated data of the converted feature point coordinate values is input by the measurement matrix input unit 4, and the matrix decomposition processing unit 5 uses the factorization method to apply the motion matrix related to the camera viewpoint, The transformation matrix calculation unit 6 decomposes the shape matrix representing the appearance of the object in the outside world, sets camera motion constraint conditions from the components of the motion matrix among the matrices decomposed by the matrix decomposition processing unit 5, and satisfies this condition The transformation matrix is obtained, and the transformation matrix is used to transform the motion matrix and the shape matrix. The viewpoint motion extraction unit 9 extracts the motion information from the transformed motion matrix and restores the plane translational motion. 10, the plane translational motion related to the camera viewpoint is restored, the rotational motion restoring unit 11 restores the rotational motion, the object shape extracting unit 7 extracts the transformed shape matrix, and the spatial information restoring unit 8 extracts the object of the outside world. Restore spatial information such as shape, structure, or appearance.
[0030]
FIG. 2 is a process flow relating to the inventions of claims 1 to 10. The present invention will be described in detail based on these drawings.
[0031]
Here, the geometric positional relationship between the omnidirectional camera and the object in the outside world, and the camera parameters used in the present invention will be described.
[0032]
As shown in FIG. 6, an XYZ coordinate system set in the space is used as a reference coordinate system, and space information (X_j, Y_j, Z_jIs restored according to the invention. In this XYZ coordinate system, the omnidirectional camera viewpoint moves in time. At this time, the omnidirectional camera, that is, the coordinate system X set at the optical center₀Y₀In the XY plane, as shown in FIG. 6, the rotation about the optical axis parallel to the Z axis of the omnidirectional camera viewpoint, that is, the angle made with the X axis is θ_iAnd the position on the plane is (Tx_i, Ty_i). Translational movement of this XY plane (Tx_i, Ty_i) And orientation θ_iIs restored by the present invention.
[0033]
For convenience of explanation, in the initial state, X set to the omnidirectional camera viewpoint₀Y₀It is assumed that the direction of the coordinate system matches the direction of the XY coordinate system of the reference coordinate system. Also, it is assumed that there is no translational movement of the omnidirectional camera viewpoint, that is, the optical axis Z axis, and an angle other than the azimuth, that is, the traveling axis X₀Rotation around as well as Y₀It is assumed that there is no temporal change in the surrounding rotation. Also, the viewpoint of the omnidirectional camera, that is, the X when viewed from the XY plane when the optical center and a point on the object in the outside world are connected by a straight line.₀The angle made with the axis is ρ_ijAnd
[0034]
Further, FIG. 7 shows the geometric relationship between the external object on the YZ plane and the omnidirectional camera viewpoint, and the camera parameters. The omnidirectional camera viewpoint, that is, the position of the optical center (Tx_i, Ty_i), The point of the external object (X_j, Y_j, Z_j) Is connected by a straight line, the angle made with the XY plane is Φ_ijAnd
[0035]
In addition, as shown in FIG. 8, the optical property of the fisheye lens in the present invention is that an object point (X_j, Y_j, Z_j) And the omnidirectional camera viewpoint position, that is, the position of the optical center (Tx_i, Ty_i), The incident angle is Φ_ijThe light incident on the optical axis (Φ_ij= 90 degrees) is the image center (C_x, C_y) And the incident light is incident on the XY plane (Φ_ij= 0 degree) is projected onto an image circle (limit in an image displayed as an omnidirectional image).
[0036]
Optically, equidistant projection is established. The incident angle is Φ_ijThe center of the image (C_x, C_yDistance R from_ijTo be projected. Here, the focal length is f, and Q represents a proportionality constant between the optical distance and the pixel, that is, the correspondence with the area sensor per pixel. At this time, Formula (1) is established in equidistant projection. Also, the image coordinate value of the projection point is set to (x_ij, Y_ij), It can be expressed in equation (2).
[0037]
[Expression 1]

[0038]
On the other hand, due to the optical properties of this omnidirectional camera, the point of the object in the outside world is an image size of W × H as shown in FIG. Point (x_ij, Y_ij). Where X₀The angle ρ with the axis_ijThen, Formula (3) is formed from optical properties.
[0039]
[Expression 2]

[0040]
Thus, the geometric arrangement of the positions of the external object and the omnidirectional camera viewpoint, the optical properties of the omnidirectional camera, and the camera internal parameters are defined.
[0041]
Returning to FIG. 1 and FIG. 2, the description of the present invention will be continued.
[0042]
First, the feature point measurement unit 2 takes out and inputs a time-series image from the time-series image database 1 in which time-series images are stored by time management, and performs the steps of time-series image processing in FIG.
[0043]
In these steps, first, one time-series image is extracted from the time-series image database 1, and this is used as an initial image, and feature points are arranged on the image. At this time, the feature points are arranged on the initial image by automatically arranging the feature points by image processing such as edge detection, Hough transform, and two-dimensional gradient. At this time, the number of feature points to be arranged is P (j = 1, 2,..., P), and coordinates from the origin in the image coordinate system in FIG. Value (x_ij, Y_ij): J = 1, 2,..., P are recorded.
[0044]
Next, the feature point measurement unit 2 reads the time-series images following the initial image one by one from the database 1, and the feature points arranged in the initial image in the time-series image processing step are displayed with the density between the time-series images. The image is tracked by using a method focusing on a change in the image, and the origin in the image coordinate system in FIG. 5 is used as the image coordinate value of the feature point of each time-series image (i-th image from the initial image). 2D coordinate value from (x_ij, Y_ij). When the time-series image is continuously read out, when the feature points arranged in the initial image disappear from the image or are hidden by occlusion, the image tracking is stopped and the feature point tracking is ended. When the feature point tracking is completed, the number of read time-series images i = 1, 2,..., F is F including the initial image.
[0045]
Next, the feature point measurement unit 2 performs measurement matrix storage. In the measurement matrix storage step, the amount of change in temporal image coordinate arrangement of feature points in each time-series image is recorded. A data obtained by converting changes in temporal image coordinate arrangement of feature points into a matrix is referred to as a measurement matrix [X], which is a data format of Expression (4).
[0046]
[Equation 3]

[0047]
Next, the measurement matrix conversion unit 3 converts it into a measurement matrix with omnidirectional image coordinate values in the measurement matrix conversion step. As shown in FIG. 5, the omnidirectional image coordinate value here indicates the center of the image, that is, the position of the optical axis, and the coordinate value (C_x, C_y), It is temporarily converted into a relative coordinate value from this coordinate value. Further, this is applied over the entire time series, and ρ in equations (2) and (3)_ijAnd Φ_ijAre measured from all feature points and geometrically transformed into coordinate values on the omnidirectional image plane as defined by equation (5). This is converted into a data array as shown in equation (6).
[0048]
u_ij= Cot (φ_ij) Cos (ρ_ij) ... Formula (5 (part 1))
u_ij= Cot (φ_ij) Sin (ρ_ij) ... Formula (5 (2))
[0049]
[Expression 4]

[0050]
Next, the converted measurement matrix is input to the matrix decomposition processing unit 5 by the measurement matrix input unit 4, and subjected to factorization processing in the matrix decomposition processing unit 2. In this factorization method processing step, the movement of the omnidirectional camera viewpoint and the shape or structure information of the external object are restored. FIG. 3 shows a processing flow in the factorization method processing.
[0051]
First, the measurement matrix [A] converted to the omnidirectional image plane in the previous steps is read in the measurement matrix data input by the measurement matrix input unit 4. Next, matrix decomposition is uniquely performed into three matrices as shown in Expression (7) by a mathematical method called singular value decomposition.
[0052]
[Equation 5]

[0053]
Further, after matrix decomposition, the matrix is further separated as shown in Expression (8) by rank 3. The second term of equation (8) is regarded as a noise component, and noise is removed as in equation (9).
[0054]
[Formula 6]

[0055]
Next, the motion matrix and shape matrix from which the noise component has been removed are obtained, a constraint condition regarding camera motion is set for the motion matrix, and [Q] is determined so as to satisfy this. That is, as shown in Expression (12), the matrix after applying the matrix [Q] is [M], and m for each row_i, N_iAnd the element is represented by equation (13). Element m of this matrix_i, N_iHowever, the constraint conditions of the expressions (14), (15), and (16) are satisfied. However, an element of the matrix [Q] is an element shown in Expression (17). At this time, the calculation formula (19) is set from the formula (14) to the formula (18) from the elements of the formula (12). Further, the calculation formula (21) is set from the formula (12) through the formula (15) to the formula (20). Also, the calculation formula (23) is set from the formula (12) through the formula (16) from the formula (12). From equations (19), (21), and (23), the matrix [Q] [Q] of equation (24)^TSet the formula to calculate. Matrix [Q] [Q]^TIs obtained by performing the calculation of Expression (25), eigenvalue decomposition is performed as shown in Expression (26) from the elements obtained by this calculation, and Expression (27) is assumed. Next, using equation (28), element Q of the original matrix [Q] obtained by equation (25)₁₁, Q₁₂, Q_{twenty one}, Q_{twenty two}Ask for.
[0056]
[Expression 7]

[0057]
m² _ix+ M² _iy= 1 ... Formula (14)
n² _ix+ N² _iy= 1 ... Formula (15)
m_ixn_ix+ M_iyn_iy= 0 ... Formula (16)
[0058]
[Equation 8]

[0059]
[Equation 9]

[0060]
[Expression 10]

[0061]
Alternatively, eigenvalue decomposition of Expression (26) is performed to obtain a matrix having the square root of the eigenvalue of the second matrix on the right side of Expression (26) as a diagonal element, that is, the first matrix on the right side of Expression (28). . Next, as shown in Expression (28-1), sign determination of the eigenvector is performed, and (q_ix, Q_iy) I = 1, 2 is determined, and [Q] is obtained by the equation (28).
[0062]
Next, the transformation matrix calculation unit 6 performs a motion matrix transformation process according to Equation (29) using the obtained matrix [Q], and performs a shape matrix transformation process according to Equation (30). The motion component [M ′] and the object shape component [S ′] obtained in this way are output, and the factorization method processing ends.
[0063]
## EQU11 ##

[0064]
Returning to FIGS. 1 and 2, the rotational motion restoration unit 11 performs rotational component calculation, the planar translational motion restoration unit 10 performs planar translation component calculation, and the spatial information restoration decomposition 8 performs spatial information restoration calculation.
[0065]
First, the viewpoint motion extraction unit 9 extracts rotational components in all frames, and the rotational motion restoration unit 11 calculates orientation normalization in all frames so that the motion components in the initial image are expressed by Equation (31). do.
[0066]
[Expression 12]

[0067]
That is, first, the matrix [M] obtained by Expression (29) is used as the motion component of Expression (32), and the row vector at the initial frame is extracted as Expression (33). Next, a vector k orthogonal to this vector₁Is generated as shown in Expression (34), and a matrix [N] having these vectors as elements is generated. Using this matrix, transformation is performed as shown in Expression (36) to perform azimuth normalization calculation to obtain matrix [M].
[0068]
i^T ₁= [U '₁₁  U '₁₂  0] Formula (33 (Part 1))
j^T ₁= [U '_{1 + F},₁  U '_{1 + F},₂  0] Formula (33 (2))
k^T ₁= [0 0 1]] ... Formula (34)
[0069]
[Formula 13]

[0070]
Next, the rotational motion restoring unit 11 performs azimuth rotation θ for each frame using the equation (37) or the equation (38)._iCalculate
[0071]
[Expression 14]

[0072]
On the other hand, in FIG. 4, the translational component is extracted by the object shape extraction unit 7, and the translational component as the planar motion is calculated from the component in each frame by the planar translational motion restoration unit 10. Here, the elements in the third column of Expression (13) are extracted, and for each frame, the translational component (T in plane motion) according to Expression (39)⁽ⁱ⁾x, T⁽ⁱ⁾Restore y).
[0073]
[Expression 15]

[0074]
This completes the rotation component calculation and the translational motion calculation of FIG.
[0075]
Subsequently, returning to FIG. 1 and FIG. 2, the spatial information restoring process by the spatial information restoring unit 8 is performed. In the spatial information restoration processing here, assuming that the matrix element of the equation (30) is the equation (40), the three-dimensional coordinates expressing the spatial information related to the object shape or structure of the outside world according to the equation (41). The value can be restored.
[0076]
As described above, according to the embodiment of the present invention, from the temporal movement of the feature point of the omnidirectional video, the movement of the omnidirectional camera viewpoint, that is, the azimuth movement and the translational movement in the XY plane, and the shape of the object in the outside world, or The spatial information of the structure, that is, the three-dimensional coordinate value can be restored.
[0077]
[Expression 16]

[0078]
In the present invention, in the omnidirectional optical system, in the case of optical projection proportional to the angle of incidence from the optical axis shown in equation (2), the feature point coordinate value on the time-series omnidirectional image is set as the origin. From the position, the elevation angle φ using the omnidirectional projection formula (2)_ijAnd phase ρ_ijAnd geometrically transform into the coordinate system expressed by Equation (5 (Part 1)) and Equation (5 (Part 2)), and perform matrix decomposition from the measurement matrix [A] of the feature point coordinate values on this coordinate plane. In the case of optical projection proportional to the half angle sine of the incident angle from the optical axis shown in equation (42), the feature point coordinate value on the time-series omnidirectional image is set as the origin. From the position, the elevation angle φ using the omnidirectional projection formula (42)_ijAnd phase ρ_ijAnd geometrically transform into the coordinate system expressed by Equation (5 (Part 1)) and Equation (5 (Part 2)), and perform matrix decomposition from the measurement matrix [A] of the feature point coordinate values on this coordinate plane. In the case of optical projection proportional to the tangent of the half angle of the incident angle from the optical axis shown in Expression (43), the feature point coordinate value on the time-series omnidirectional image is set as the origin. From the position, using the omnidirectional projection formula (43), the elevation angle φ_ijAnd phase ρ_ijAnd geometrically transform into the coordinate system expressed by Equation (5 (Part 1)) and Equation (5 (Part 2)), and perform matrix decomposition from the measurement matrix [A] of the feature point coordinate values on this coordinate plane. You may make it do.
[0079]
[Expression 17]

[0080]
1 may be realized by configuring a part or all of the functions of each unit in the apparatus shown in FIG. 1 with a computer program and executing the program using the computer, or FIG. It goes without saying that the processing procedure shown in 4 can be constituted by a computer program and the program can be executed by the computer, and the program for realizing the function by the computer or the processing procedure is executed by the computer. The program to be recorded on a computer-readable recording medium such as FD (floppy disk (registered trademark)), MO, ROM, memory card, CD, DVD, removable disk, etc. Can be distributed. It is also possible to provide the above program through a network such as the Internet or electronic mail.
[0081]
【The invention's effect】
According to the present invention, from general time-series images acquired using an omnidirectional camera (vehicle-mounted video, marine video, aerial video, indoor video, etc. captured using a moving means) Spatial information can be acquired and restored with high accuracy. In addition, high-accuracy three-dimensional stereoscopic viewing similar to conventional surveying techniques is possible. Furthermore, it is possible to accurately restore the rotation of the omnidirectional camera viewpoint around the camera optical axis and the azimuth motion consisting of the XY plane motion.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a basic embodiment of the present invention.
FIG. 2 is a diagram illustrating a processing flow in the present embodiment.
FIG. 3 is a diagram for explaining a factorization method processing flow in the present embodiment example;
FIG. 4 is a diagram for explaining the rotational movement of a camera viewpoint and the restoration of translational movement in the present embodiment.
FIG. 5 is a diagram illustrating coordinate values on an omnidirectional image plane.
FIG. 6 is a diagram illustrating camera parameters on an XY plane.
FIG. 7 is a diagram illustrating a relationship between a vertical plane, a YZ plane, and an optical system.
FIG. 8 is a diagram for explaining equidistant projection.
[Explanation of symbols]
1. Time series image database
2 ... Feature point measurement unit
3 ... Measurement matrix converter
4 ... Measurement matrix input section
5 ... Matrix decomposition processor
6 ... Conversion matrix calculation unit
7 ... Object shape extraction unit
8 ... Spatial information restoration unit
9 ... Viewpoint motion extraction unit
10 ... plane translational motion restoration part
11 ... Rotational motion restoration part

Claims (13)

In a time-series omnidirectional image acquired by the image input means, a method of restoring spatial information including the movement of the omnidirectional camera viewpoint with respect to the time-series and the shape, structure, or appearance of the external object that is the subject. There,
A first step of measuring a temporal variation amount of a feature point in each time series image in an image coordinate system set on the time series image, and converting the feature point coordinate value to another coordinate system;
A second step of decomposing a measurement matrix, which is data obtained by aggregating the converted feature point coordinate values, into a motion matrix related to a camera viewpoint and a shape matrix expressing the appearance of an object in the outside world;
In the decomposed motion matrix, a transformation matrix is obtained so as to satisfy the conditions set for defining the motion, and information corresponding to the rotational motion of the omnidirectional camera viewpoint and translational motion are obtained using the transformation matrix. A third step of calculating and restoring information corresponding to
A fourth step of restoring a shape matrix representing the appearance of the object using the transformation matrix;
An omnidirectional camera viewpoint motion and object shape restoration method, comprising: a fifth step of restoring space information of an object in the outside world using the restored shape matrix. An omnidirectional camera viewpoint motion restoration and object shape restoration method according to claim 1,
The second step is
A matrix decomposition by factorization of a measurement matrix that is data obtained by aggregating converted feature point coordinate values;
Denoising from the decomposed matrix;
Obtaining a motion matrix and a shape matrix from the denoised matrix, and an omnidirectional camera viewpoint motion and object shape restoration method. An omnidirectional camera viewpoint motion restoration and object shape restoration method according to claim 2,
In the first step, when converting the feature point coordinate values in each time-series image to another coordinate system, the image coordinates are calculated in the omnidirectional optical system based on the omnidirectional projection proportional to the incident angle from the optical axis. Convert the value into a coordinate value in another coordinate system,
An omnidirectional camera viewpoint movement and object shape restoration method characterized in that, in the step of performing matrix decomposition by factorization in the second step, factorization is performed on the measurement matrix composed of the converted coordinate values. An omnidirectional camera viewpoint motion restoration and object shape restoration method according to claim 2,
In the first step, when converting the feature point coordinate value in each time-series image to another coordinate system, in the omnidirectional optical system, based on the projection proportional to the sine of the half angle of the incident angle from the optical axis, Convert image coordinate values to coordinate values in another coordinate system,
An omnidirectional camera viewpoint movement and object shape restoration method characterized in that, in the step of performing matrix decomposition by factorization in the second step, factorization is performed on the measurement matrix composed of the converted coordinate values. An omnidirectional camera viewpoint motion restoration and object shape restoration method according to claim 2,
In the first step, when converting the feature point coordinate value in each time series image to another coordinate system, in the omnidirectional optical system, based on the projection proportional to the tangent of the half angle of the incident angle from the optical axis, Convert image coordinate values to coordinate values in another coordinate system,
An omnidirectional camera viewpoint movement and object shape restoration method characterized in that, in the step of performing matrix decomposition by factorization in the second step, factorization is performed on the measurement matrix composed of the converted coordinate values. An omnidirectional camera viewpoint motion restoration and object shape restoration method according to any one of claims 1 to 5,
In the first step, when converting the feature point coordinate values in each time-series image to another coordinate system, the coordinate values of the feature points on the time-series omnidirectional image are calculated based on the phase from the origin and the set position. Transform to the coordinate system to represent,
In the second step, an omnidirectional camera viewpoint motion and object shape restoration method comprising performing matrix decomposition from a measurement matrix of feature point coordinate values on the phase plane. An omnidirectional camera viewpoint motion restoration and object shape restoration method according to any one of claims 1 to 6,
In the fourth step, when the information corresponding to the rotational motion is calculated and restored, the rotational motion around the optical axis is restored in the motion matrix obtained by using the transformation matrix obtained in the third step. Omnidirectional camera viewpoint movement and object shape restoration method. An omnidirectional camera viewpoint motion restoration and object shape restoration method according to any one of claims 1 to 7,
In the fourth step, when calculating and restoring the information corresponding to the translational motion, the information corresponding to the translational motion of the omnidirectional camera viewpoint is extracted using the transformation matrix obtained in the third step, An omnidirectional camera viewpoint motion and object shape restoration method, characterized in that it is converted into a planar motion in an image coordinate system set when the coordinate values of feature points are measured. An omnidirectional camera viewpoint motion restoration and object shape restoration method according to any one of claims 1 to 8,
In the fifth step, the omnidirectional feature is characterized in that one component of the shape matrix obtained by converting the shape matrix representing the appearance of the object obtained in the third step is extracted and the spatial information of the feature points is restored. Camera viewpoint movement and object shape restoration method. An omnidirectional camera viewpoint motion restoration and object shape restoration method according to any one of claims 1 to 9,
In the third step, when obtaining a transformation matrix, an omnidirectional camera viewpoint motion characterized by obtaining a transformation matrix from its principal component and a code vector related to the principal component for a matrix element satisfying the conditional expression, Object shape restoration method. In a time-series omnidirectional image acquired by an image input device, a device that restores spatial information including motion of an omnidirectional camera viewpoint related to time-series, and the shape or structure of an external object or appearance,
In the image coordinate system set on the time series image, a feature point measurement unit that measures a temporal variation amount of the feature point in each time series image,
A measurement matrix conversion unit that converts the feature point coordinate values into another coordinate system and totals them, and
Matrix decomposition processing unit that decomposes a measurement matrix, which is data obtained by aggregating the converted feature point coordinate values, and removes noise into a motion matrix related to the camera viewpoint and a shape matrix that expresses the appearance of an object in the external world When,
In the decomposed motion matrix, a transformation matrix calculation unit for obtaining a transformation matrix so as to satisfy a condition set for defining motion;
Using the transformation matrix, a viewpoint motion restoring unit that calculates and restores information corresponding to the rotational motion of the omnidirectional camera viewpoint and information corresponding to the translational motion;
A spatial information restoration unit that restores a shape matrix that expresses the appearance of the object using the transformation matrix, and that restores the spatial information of the object in the outside world using the restored shape matrix. An omnidirectional camera viewpoint movement and object shape restoration device. An omnidirectional camera viewpoint motion and object shape restoration, characterized in that the step in the omnidirectional camera viewpoint motion and object shape restoration method according to any one of claims 1 to 10 is a program for causing a computer to execute. program. A program for causing a computer to execute the steps in the omnidirectional camera viewpoint movement and the object shape restoration method according to any one of claims 1 to 10,
A recording medium on which an omnidirectional camera viewpoint motion and object shape restoration program is recorded, wherein the program is recorded on a recording medium readable by the computer.

Images