JP3555159B2 - Three-dimensional shape restoration method and apparatus - Google Patents

Description

【００１４】
ただし、
【００１５】
【数２】

【００１６】
上式において、添え字ｉ（＝１，・・・，Ｎ）は画像上の各々の点に対応する番号、ｐ_ｉ は画像上の点ｉでのＺ座標値の逆数（インバースデプスと呼ぶ）である。一方、カメラの速度ｋは次式で与えられる。
【００１７】
ｋ＝｜Ｔ｜＝（Ｔ_ｘ ^２ ＋Ｔ_ｙ ^２ ＋Ｔ_ｚ ^２ ）^１／２ （２）
ここで、距離情報計算手段２０の入力となる量であるオプティカルフローとカメラの速度を１つのベクトルｙ、計算すべき量であるカメラの並進・回転速度とインバースデプスを１つのベクトルθにまとめて表記する。
【００１８】
ｙ＝［ｕ_１ ｖ_１ ｕ_２ ｖ_２ ・・・ｕ_Ｎ ｖ_Ｎ ｋ］^Ｔ （３）
θ＝［Ｔ_ｘ Ｔ_ｙ Ｔ_ｚ Ｒ_ｘ Ｒ_ｙ Ｒ_ｚ ｐ_１ ｐ_２ ・・・ｐ_Ｎ ］^Ｔ
ベクトルｙはθの関数ｙ（θ）であり、詳細は式（１）および（２）で与えられる。
【００１９】
次に、オプティカルフロー計算手段１０によって実際の動画像から計算されるオプティカルフローと速度計などにより計測されるカメラの速度から作られる観測ベクトルｙ^＊ は、真の値ｙに平均０の多次元ガウスノイズｎが加わったものとして確率的なモデル化を行う。
【００２０】
ｙ^＊ ＝ｙ（θ）＋ｎ （４）
ノイズｎの共分散行列をＶ_ｙ とすると、パラメータθが与えられたときに計測値ｙ^＊ が観測される確率Ｐ（ｙ^＊ ｜θ）は次式となる。
【００２１】
Ｐ（ｙ^＊ ｜θ）＝（２π）^−Ｎ／２｜Ｖ_ｙ ｜^−１／２ｅｘｐ（−Ｊ／２） （５）
Ｊ＝（ｙ^＊ −ｙ（θ））^Ｔ Ｖ_ｙ ^−１（ｙ^＊ −ｙ（θ））
計測値ｙ^＊ が観測されたときのパラメータθの最ゆう推定値θ_ＭＬは、式（５）を最大にするθ、すなわちＪを最小にするθである。Ｊの最小化は、最急勾配法などの非線形最適化の手法を用いることができる。また、文献
太田直哉：“信頼性情報を持ったオプティカルフローからの形状復元とその移動物体検出への応用”，電子情報通信学会論文誌，Ｄ−ＩＩ，Ｖｏｌ．Ｊ７６−Ｄ−ＩＩ，Ｎｏ．８，ｐｐ．１５６２−１５７１（１９９３年８月）
の第２．３節に示されている最小化アルゴリズムにおいて、カメラの並進速度の大きさ｜Ｔ｜を１に正規化するステップをカメラの速度の計測値ｋ^＊ の大きさにするように変更することで、このアルゴリズムがそのまま適用可能である。
【００２２】
次に、推定されたパラメータθ_ＭＬの誤差の共分散行列の計算法について述べる。関数ｙ（θ）の非線形性により、厳密に共分散行列を計算することは容易ではないので、ｙ（θ）をθ_ＭＬ周りでテーラー展開し、１次の項までとることによって線形化する。
【００２３】
ｙ_Ｌ ＝ｙ（θ_ＭＬ）＋（ｄｙ／ｄθ）（θ−θ_ＭＬ） （６）
ここで、ｄｙ／ｄθはヤコビ行列である。式（６）をθについて解くと次式となる。
【００２４】
θ＝（ｄｙ／ｄθ）^−１（ｙ_Ｌ −ｙ（θ_ＭＬ））＋θ_ＭＬ （７）
ベクトルｙ_Ｌ の共分散行列をＶ_ｙ とすれば、θの共分散行列Ｖ_ｈ は
Ｖ_ｈ ＝（ｄｙ／ｄθ）^−１Ｖ_ｙ （（ｄｙ／ｄθ）^−１）^Ｔ （８）
となる。オプティカルフロー検出方法によって得られる分散情報は、共分散行列の逆行列である関係上、実際の計算では、Ｖ_ｙ の逆行列を用いて計算を行う方が効率的である。そこで、式（８）の逆行列を作り、実際の計算ではこれを用いる。
【００２５】
Ｖ_ｈ ^−１ ＝（ｄｙ／ｄθ）^Ｔ Ｖ_ｙ ^−１（ｄｙ／ｄθ） （９）
一方、距離情報融合手段３０において、計算の容易性からカメラの運動パラメータｍと各インバースデプスｐ_ｉ に関して独立にカルマンフィルタを構成するので、そのためには、式（９）からそれぞれの共分散行列を作り出す必要がある。ここで、運動パラメータｍとは
ｍ＝［Ｔ_ｘ Ｔ_ｙ Ｔ_ｚ Ｒ_ｘ Ｒ_ｙ Ｒ_ｚ ］^Ｔ （１０）
である。そこでＶ_ｈ ^−１から対応する部分行列を取り出し、それをそれぞれの変数に対する分散の逆行列として使用する。すなわち、カメラの運動パラメータに関しては、Ｖ_ｈ ^−１の１行１列から６行６列までの部分行列を運動パラメータの分散の逆行列Ｖ_ｍ ^−１とし、ｉ番目のインバースデプスに関しては、ｉ＋６行ｉ＋６列の対角成分を分散の逆数１／σ_ｐｉ ^２ とする。この操作は、各変数の分散を他の変数の正しい値が得られた場合の条件付き確率で見積もっていることに相当する。
【００２６】
ここで、今まで述べたことをまとめておく。まず、オプティカルフロー計算手段１０によって、画像上の各点での移動ベクトルｕ_ｉ ^＊ とその共分散行列の逆行列Ｖ_ｕｉ ^−１が計算されると共に、速度計などからカメラの移動速度ｋが得られる。移動速度ｋ^＊ にはσ_ｋ ^２ の分散の誤差が含まれているとする。このときｕ_ｉ ^＊ およびｋ^＊ の誤差は独立であると仮定する。これらから観測ベクトルｙ^＊ と共分散行列の逆行列Ｖ_ｙ ^−１を作り、距離情報計算手段２０の入力とする。観測ベクトルｙ^＊ は、式（３）で示した順序に各要素を並べる。行列Ｖ_ｙ ^−１は、２行２列の行列Ｖ_ｕｉ ^−１をその対角要素、すなわち２ｉ−１行２ｉ−１列から２ｉ行２ｉ列までの２行２列の部分行列になるように並べ、最後の対角成分２Ｎ＋１行２Ｎ＋１列の要素は１／σ_ｋ ^２ とする。残りの部分は０とする。このようにして作られたｙ^＊ とＶ_ｙ ^{− １} を用いて式（５）を最小化し、θ_ＭＬを計算すると共に、式（９）によりＶ_ｈ ^−１を求める。さらに、式（１０）を参考にθ_ＭＬを分解してｍ_ＭＬとｐ_ＭＬｉ 、Ｖ_ｈ ^−１を分解してＶ_ｍ ^−１と１／σ_ｐｉ ^２ を得る。Ｖ_ｍ ^−１と１／σ_ｐｉ ^２ の具体的な形は、式（１）および（９）から計算され、次のようになる。
【００２７】
【数３】

【００２８】
距離情報融合手段３０においては、前述のごとくカメラの運動パラメータｍと各インバースデプスｐ_ｉ に関して独立にカルマンフィルタを構成する。距離情報計算手段２０によってθ_ＭＬとその共分散行列の逆行列Ｖ_ｈ ^−１が得られているので、理論的にはθを状態ベクトルとしてカルマンフィルタを構成することができる。しかし、このようなカルマンフィルタを構成すると、実際にＶ_ｈ ^−１の逆行列を数値計算により求める必要があり、その大きな次元数のため処理量と安定性の面で問題がある。それを避けるために、θをカメラの運動パラメータｍと各インバースデプスｐ_ｉ に分解し、それぞれ独立にカルマンフィルタを構成する。
【００２９】
最初にカメラの運動パラメータｍのカルマンフィルタについて述べる。カメラの並進・回転速度は、微少時間では一定であると考え、状態遷移行列を単位行列とし、システムノイズの共分散行列をＱ_ｍ とすると、時間更新アルゴリズムは次のようになる。
【００３０】
ｍ_ｔ＋１ ⁻ ＝ｍ_ｔ ^＋ （１２）
Ｖ_ｍｔ＋１ ⁻ ＝Ｖ_ｍｔ ^＋ ＋Ｑ_ｍ
一方、観測更新アルゴリズムは次式で与えられる。
【００３１】
ｍ_ｔ ^＋ ＝ｍ_ｔ ⁻ ＋Ｋ_ｍｔ（ｍ_ｔ ^＊ −ｍ_ｔ ⁻ ） （１３）
Ｖ_ｍｔ ^＋ ＝（Ｉ−Ｋ_ｍｔ）Ｖ_ｍｔ ⁻
Ｋ_ｍｔ＝Ｖ_ｍｔ ⁻ （Ｖ_ｍｔ ⁻ ＋Ｖ_ｍｔ ^＊ ）^−１
これらの式で、ベクトルおよび行列の添字ｔおよびｔ＋１は時間ステップを、肩の＋および−はそれぞれ観測更新値と時間更新値を表す。また、Ｉは単位行列であり、ｍ_ｔ ^＊ およびＶ_ｍｔ ^＊ は時刻ｔにおける観測値とその分散で、距離情報計算手段２０で計算されたｍ_ＭＬとＶ_ｍ ^−１の逆行列を使用する。
【００３２】
次に、インバースデプスｐ_ｉ のカルマンフィルタを構成する。時刻ｔに３次元座標がＸ_ｔ ^＊ であった物体上の点は、その時刻のカメラの並進・回転速度がＴ_ｔ ＊ _{およびＲ ｔ} ^＊ とすれば、単位時間後の座標値Ｘ_ｔ＋１ ^＊ は次式で与えられる。
【００３３】
Ｘ_ｔ＋１ ^＊ ＝−Ｔ_ｔ ^＊ −Ｒ_ｔ ^＊ ×Ｘ_ｔ ^＊ （１４）
Ｘ_ｔ ^＊ ＝（Ｘ_ｔ Ｙ_ｔ Ｚ_ｔ ）^Ｔ
Ｘ_ｔ＋１ ^＊ ＝（Ｘ_ｔ＋１ Ｙ_ｔ＋１ Ｚ_ｔ＋１ ）^Ｔ
上式のＺ_ｔ とＺ_ｔ＋１ の関係から両時刻のインバースデプスの関係は次のようになる。
【００３４】
ｐ_ｉｔ＋１＝ｐ_ｉｔ／（１＋Ｒ_ｙｔｘ_ｉｔ−Ｒ_ｘｔｙ_ｉｔ−Ｔ_ｚｔｐ_ｉｔ） （１５）
ここで、上式をｐ_ｉｔで微分すると次式となる。
【００３５】
ｄｐ_ｉｔ＋１／ｄｐ_ｉｔ＝（Ｔ_ｚｔｐ_ｉｔ）／（１＋Ｒ_ｙｔｘ_ｉｔ−Ｒ_ｘｔｙ_ｉｔ−Ｔ_ｚｔｐ_ｉｔ）^２ ＋１／（１＋Ｒ_ｙｔｘ_ｉｔ−Ｒ_ｘｔｙ_ｉｔ−Ｔ_ｚｔｐ_ｉｔ） （１６）
式（１５）および（１６）より、時間更新アルゴリズムが得られる。
【００３６】
ｐ_ｉｔ＋１ ⁻ ＝ｐ_ｉｔ ^＋ ／（１＋Ｒ_ｙｔｘ_ｉｔ−Ｒ_ｘｔｙ_ｉｔ−Ｔ_ｚｔｐ_ｉｔ ^＋ ） （１７）
σ_{ｐｉｔ＋１} ^２−＝（ｄｐ_ｉｔ＋１／ｄｐ_ｉｔ）^２ σ_ｐｉｔ ^２＋＋Ｑ_ｐ
Ｒ_ｙｔ，Ｒ_ｘｔ，Ｔ_ｚｔは、カメラの運動パラメータのカルマンフィルタの推定値ｍ_ｔ ＋ _{の対応する要素を用いる。また、Ｑ ｐ} は、インバースデプスに関するシステムノイズである。
【００３７】
一方、観測更新アルゴリズムは次のようになる。
【００３８】
ｐ_ｉｔ ^＋ ＝ｐ_ｉｔ ⁻ ＋Ｋ_ｐｔ（ｐ_ｉｔ ^＊ −ｐ_ｉｔ ⁻ ） （１８）
σ_ｐｉｔ ^２＋＝（１−Ｋ_ｐｔ）σ_ｐｉｔ ^２−
Ｋ_ｐｔ＝σ_ｐｉｔ ^２−／（σ_ｐｉｔ ^２−＋σ_ｐｉｔ ^２＊）
ここで、ｐ_ｉｔ ^＊ およびσ_ｐｉｔ ^２＊は、時刻ｔに距離情報計算手段２０によって計算された値ｐ_ＭＬｉ およびσ_ｐｉ ^２ に対応する。さて、以上で物体上に固定された点のインバースデプスに関するカルマンフィルタが構成されたが、その点は、カメラの運動によって画像上を移動する。従って、ある時刻でのインバースデプスｐ_ｉ に対応する点は、次の時刻では画像上の異なった位置にあるので、補間によってこれらを関係づける必要があるが、これは次のようにして達成される。式（１８）で示される観測更新アルゴリズムを適用する際に、式（１３）で計算されるその時刻のカメラの運動パラメータの推定値ｍ_ｔ ^＋ から式（１）を用いて推定されるオプティカルフローｕ_ｉ ^＋ を計算する。観測値ｐ_ｉｔ ^＊ に対応する予測値ｐ_ｉｔ ⁻ は、単位時間前には画像上で−ｕ_ｉ ^＋ だけ離れた位置に存在することになるので、ｐ_ｉｔ ⁻ の画像を内挿してその位置での値を求め、式（１８）内のｐ_ｉｔ ⁻ として使用する。この様子を１次元の図として図２に示した。内挿の方法には、最近隣内挿法、共１次内挿法、３次たたみ込み内挿法などがあり、そのいずれも使用することができる。詳細は以下の参考文献に述べられている。
【００３９】
高木幹雄，下田陽久 監修：画像解析ハンドブック，東京大学出版会，ｐｐ．４１１−４４４（１９９１）
また、予測値ｐ_ｉｔ ⁻ の位置を単位時間だけ前の運動パラメータの推定値ｍ_ｔ−１ ^＋ から計算する方法もある。これにはｍ_ｔ−１ ^＋ を式（１）に代入して時刻ｔ−１のオプティカルフローｕ_ｉ ^＋ を推定し、図３に示すようにｐ_ｉｔ ⁻ の位置を移動する。移動された位置を標本点として内挿し、観測値ｐ_ｉｔ ^＊ の位置の値を得る。この方法では、内挿すべき画像の標本点が規則正しい格子上に存在しないので、内挿問題がやや複雑になる。この場合の内挿の１例として以下の方法がある。まず、計算しようとする点に最も近く、かつその点を内部に含むような３つの標本点を決定する。これらの標本点の位置をそれぞれ（ｘ_ａ ，ｙ_ａ ），（ｘ_ｂ ，ｙ_ｂ ），（ｘ_ｃ ，ｙ_ｃ ）とし、値をｐ_ａ ，ｐ_ｂ ，ｐ_ｃ とすると、位置（ｘ，ｙ）の点での値ｐは次式で与えられる。
【００４０】
ｐ＝（ｃ_１ ｘ＋ｃ_２ ｙ＋ｃ_３ ）／ｃ_４ （１９）
ｃ_１ ＝（ｙ_ａ −ｙ_ｂ ）ｐ_ａ ＋（ｙ_ａ −ｙ_ｃ ）ｐ_ｂ ＋（ｙ_ｂ −ｙ_ａ ）ｐ_ｃ
ｃ_２ ＝（ｘ_ｂ −ｘ_ｃ ）ｐ_ａ ＋（ｘ_ｃ −ｘ_ａ ）ｐ_ｂ ＋（ｘ_ａ −ｘ_ｂ ）ｐ_ｃ
ｃ_３ ＝（ｘ_ｃ ｙ_ｂ −ｘ_ｂ ｙ_ｃ ）ｐ_ａ ＋（ｘ_ａ ｙ_ｃ −ｘ_ｃ ｙ_ａ ）ｐ_ｂ ＋（ｘ_ｂ ｙ_ａ −ｘ_ａ ｙ_ｂ ）ｐ_ｃ
ｃ_４ ＝（ｘ_ｂ −ｘ_ｃ ）ｙ_ａ ＋（ｘ_ｃ −ｘ_ａ ）ｙ_ｂ ＋（ｘ_ａ −ｘ_ｂ ）ｙ_ｃ
予測値ｐ_ｉｔ ⁻ と同様に分散σ_ｐｉｔ ^２−の内挿も必要であるが、これも以上述べた方法によって達成される。ただし、分散の場合には平方根を求めて標準偏差に直してから内挿を行い、自乗して分散に戻すという操作を行う。
【００４１】
【発明の効果】
本発明によって示された方法および装置を使用することで、動画像の各フレームから得られる距離情報とカメラの運動を時間的に融合し、より正確な情報を安定して得ることができる。
【図面の簡単な説明】
【図１】本発明の３次元形状復元装置の一実施例を示す構成図である。
【図２】インバースデプスの内挿方法を説明する図である。
【図３】インバースデプスの内挿方法を説明する図である。
【符号の説明】
１０ オプティカルフロー計算手段
２０ 距離情報計算手段
３０ 距離情報融合手段
４０ 距離情報カルマンフィルタ
５０ 運動パラメータカルマンフィルタ[0001]
[Industrial applications]
The present invention relates to a method and an apparatus for reconstructing a three-dimensional shape that obtains distance information by analyzing an image when information on a distance to an object in an environment is required.
[0002]
[Prior art]
For mobile robots that move autonomously in the environment or autonomous vehicles that run on their own understanding of the road environment, have an artificial vision system that obtains the necessary information by analyzing images obtained by TV cameras with a computer. There are many. In these systems, information on the distance to an object in the environment is required, for example, for detecting an obstacle. As a technique for analyzing information of an image to obtain distance information, there is a technique called Structure from Motion that uses movement information of a moving image in addition to stereoscopic viewing.
[0003]
As a technique for detecting an optical flow from a moving image, for example, Ohta N.A. : "Image Movement Detection with Reliability Indices", IEICE Trans. , E74,10, pp. 3379-3388 (Oct. 1991)
As an example of a method for calculating the motion parameters of the camera and the distance from the object to be photographed from the optical flow, Naoya Ota: "Shape recovery from optical flow with reliability information And its application to moving object detection, "Transactions of the Institute of Electronics, Information and Communication Engineers, D-II, Vol. J76-D-II, No. 8, pp. 1562-1571 (August 1993)
There is a method reported in With these two methods, the distance from the moving image to the object, the motion parameters of the camera, and the variance of the error estimated for those values can be calculated. However, these methods use two frames of a moving image. A method in which distance information calculated from each frame of a moving image sequentially obtained from a camera and motion of the camera are temporally fused and a Kalman filter is used to obtain a more accurate result is described in Mattheys L. et al. M. , Kanade T .; , Szeliski R .; : "Kalman Filter-based Algorithms for Estimating Depth from Image Image Sequence", Int. J. Comput. Vision, 3, pp. 209-236 (1989)
Indicated by However, this report discusses only a case where the camera motion does not include rotation as a specific example, and Heel J. et al. : "Dynamic Motion Vision", Proc. of Image Understands Working Shop '89, pp. 702-713 (1989)
Indicated by
[0004]
[Problems to be solved by the invention]
The method proposed by Heel linearizes the relational expression between optical flow and distance information and applies an extended Kalman filter. However, since the relationship between the optical flow and the distance information has a strong nonlinearity, the movement of the camera for properly operating this method is limited, and there is also a problem in the stability of the operation. In order to enable a stable operation with respect to general camera motion, it is necessary to configure an algorithm with due consideration of the nonlinearity of this problem.
[0005]
An object of the present invention is to provide a three-dimensional shape restoration method and apparatus capable of temporally fusing distance information obtained from each frame of a moving image with camera motion and stably obtaining more accurate information. It is in.
[0006]
[Means for Solving the Problems]
The present invention provides an optical flow calculation means for calculating an estimated value of an optical flow, which is movement information on an image, and a covariance matrix thereof using two frames in a series of images constituting a moving image, and the optical flow Distance information calculating means for calculating an estimated value of the distance to the object being imaged and the motion of the camera and their covariance matrices using the estimated value and the covariance matrix, and applying these two methods to a moving image By temporally fusing the distance to many objects obtained by continuous application and the estimated value of the camera motion and their covariance matrix, a highly accurate estimated value of the distance to the object and the camera motion is obtained. A three-dimensional shape restoration device comprising: a distance information fusion unit that calculates the covariance matrix;
The distance information fusing means constitutes an independent Kalman filter for each of the estimated value of the distance to the object calculated by the distance information calculating means and the estimated value of the motion of the camera.
[0007]
In addition, the present invention uses the magnitude of the translational motion of the camera and the variance of the noise included in the motion as well as the estimated value of the optical flow and its covariance matrix as inputs to the distance information calculating means, and calculates the distance to the object to be calculated. It is characterized by employing distance information fusion means for reflecting the estimated value of the distance and the estimated value of the motion of the camera.
[0008]
[Action]
Applying the extended Kalman filter directly to the relationship between the optical flow having strong nonlinearity, the distance information, and the motion of the camera results in an unstable operation. Therefore, the distance information calculation means performs a process in which the nonlinearity of the problem is sufficiently taken into consideration, obtains values at which the linear approximation is effective, and then applies a Kalman filter to those values. The optical flow calculation means calculates an optical flow and a covariance matrix of an error estimated therefrom from the moving image, and the distance information calculation means calculates a motion parameter of the camera, distance information, and a covariance matrix of an error included therein. These values are temporally fused by the Kalman filter of the distance information fusion unit, and become more accurate values. Since there is a correlation between the distance information calculated by the distance information calculation means and the motion parameters of the camera, it is conceivable to configure a Kalman filter collectively, but since the dimension of the state vector becomes very large, It is not realistic as an actual method. Therefore, an independent Kalman filter is configured for each of the distance information and the motion parameters of the camera. Note that the input to the distance information calculation means is the motion speed of the camera separately from the optical flow for the following reason. The distance information obtained by analyzing the optical flow is only a value relative to the magnitude of the translational movement of the camera, and an absolute value cannot be known. An absolute value is necessary for the Kalman filter of the distance information fusion means to perform a correct operation, and in order to obtain the absolute value, the absolute value of the translational movement of the camera, that is, the movement speed of the camera, is calculated by the distance information calculation means. This is because it is necessary to input from a device other than the camera such as a speedometer. However, in reality, the movement of the camera is often smooth, and if the time is short, there is often no problem even if the speed is assumed to be constant. In this case, the measured value of the camera speed and the variance of the noise included in the measured value may be set to appropriate values.
[0009]
【Example】
Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a configuration diagram showing one embodiment of a three-dimensional shape restoration device of the present invention. The three-dimensional shape restoration apparatus shown in FIG. 1 includes an optical flow calculation unit 10 that calculates an estimated value of an optical flow and a covariance matrix of an error estimated therefrom from a moving image, an estimated value of an optical flow, a covariance matrix, and a camera. Distance information calculating means 20 for calculating distance information to the object being imaged using the magnitude of the translational motion and the variance of the noise contained therein, the estimated values of the motion parameters of the camera, and the covariance matrix of the errors contained therein; Independent Kalman filters 30 and 40 are formed for each of the estimated value of the distance information and the estimated value of the motion parameter of the camera, and the highly accurate estimated value of the distance information and the motion parameter of the camera and its covariance matrix are calculated. And a distance information fusion unit 50.
[0010]
First, the optical flow calculation means 10 of FIG. 1 can be used in the method of the present invention as long as it outputs a covariance matrix of an optical flow and a noise included therein. For example, the above-mentioned document Ohta N. : "Image Movement Detection with Reliability Indices", IEICE Trans. , E74,10, pp. 3379-3388 (Oct. 1991)
Or Naoya Ota: "Shape restoration from optical flow and detection of moving object", Society of Instrument and Control Engineers, 25th Pattern Measurement Subcommittee, pp. 7-13 (November 19, 1993)
You can use the techniques reported in This approach estimates the motion vector there at the point i on the image u _i ^* and covariance matrix V _ui of the error in the estimated value _(more precisely its inverse matrix V _{ui ^-1)} to output a.
[0011]
Next, in the distance information calculating means 20, the aforementioned document Naoya Ota: "Shape restoration from optical flow having reliability information and its application to moving object detection", IEICE Transactions, D-II, Vol. J76-D-II, No. 8, pp. 1562-1571 (August 1993)
However, this method does not describe the case of inputting the magnitude of the translational movement of the camera, and will be described below.
[0012]
A central projection system with a focal length of 1 is used as a model of the imaging system. Camera still in the environment translational velocity T, when moving at a speed R, (the position (x _{i, y i)} to) the point i on the image optical flow occurs u _i is expressed by the following formula .
[0013]
(Equation 1)

[0014]
However,
[0015]
(Equation 2)

[0016]
In the above equation, the subscript i (= 1,..., N) is a number corresponding to each point on the image, and p _i is the reciprocal of the Z coordinate value at the point i on the image (called an inverse depth). It is. On the other hand, the camera speed k is given by the following equation.
[0017]
^{_{k = | T | = (T}} x 2 + T y 2 + T z 2) 1/2 (2)
Here, the optical flow and the speed of the camera, which are the amounts to be input to the distance information calculating means 20, are combined into one vector y, and the translation / rotation speed and the inverse depth of the camera, which are the amounts to be calculated, are combined into one vector θ. write.
[0018]
y = [u ₁ v ₁ u ₂ v ₂ ... u _N v _N k] ^T (3)
_{θ = [T x T y T} z R x R y R z p 1 p 2 ··· p N] T
The vector y is a function y (θ) of θ, and details are given by equations (1) and (2).
[0019]
Next, an observation vector y ^* generated from the optical flow calculated from the actual moving image by the optical flow calculation means 10 and the speed of the camera measured by a speedometer or the like is converted into a true value y and a multidimensional Gaussian whose average is 0. Probabilistic modeling is performed assuming that noise n is added.
[0020]
y ^* = y (θ) + n (4)
Assuming that the covariance matrix of the noise n is V _y , the probability P (y ^* | θ) that the measured value y ^* is observed when the parameter θ is given is as follows.
[0021]
^{P (y * | θ) =} (2π) -N / 2 | V y | -1/2 exp (-J / 2) (5)
J = (y ^* −y (θ)) ^T V _y ⁻¹ (y ^* −y (θ))
The maximum likelihood estimation value θ _ML of the parameter θ when the measurement value y ^* is observed is θ that maximizes Expression (5), that is, θ that minimizes J. To minimize J, a non-linear optimization method such as the steepest gradient method can be used. Also, Naoya Ota: "Shape recovery from optical flow with reliability information and its application to moving object detection", IEICE Transactions, D-II, Vol. J76-D-II, No. 8, pp. 1562-1571 (August 1993)
In the minimization algorithm shown in section 2.3 of the above, the step of normalizing the magnitude of the translation speed | T | of the camera to 1 is changed to the magnitude of the measured value k ^* of the camera speed. By doing so, this algorithm can be applied as it is.
[0022]
Next, a method for calculating the covariance matrix of the error of the estimated parameter θ _ML will be described. Since it is not easy to calculate the exact covariance matrix due to the nonlinearity of the function y (θ), y (θ) is _tailored around θ _ML and linearized by taking the first-order term.
[0023]
_{y L = y (θ ML)} + (dy / dθ) (θ-θ ML) (6)
Here, dy / dθ is a Jacobian matrix. When equation (6) is solved for θ, the following equation is obtained.
[0024]
θ = (dy / dθ) ⁻¹ (y _L −y (θ _ML )) + θ _ML (7)
Assuming that the covariance matrix of the vector y _L is V _y , the covariance matrix V _h of θ is V _h = (dy / dθ) ⁻¹ V _y ((dy / dθ) ⁻¹ ) ^T (8)
It becomes. Distributed information obtained by the optical flow detection method, the inverse matrix is related of the covariance matrix, the actual calculation, who performed the calculation using the inverse matrix of V _y is efficient. Therefore, an inverse matrix of Expression (8) is created, and this is used in actual calculation.
[0025]
V _h ⁻¹ = (dy / dθ) ^T V _y ⁻¹ (dy / dθ) (9)
On the other hand, in the distance information fusion unit 30, so constituting a Kalman filter from the ease of calculation independently for camera motion parameters m and the inverse depth p _i, For this purpose, produce a respective covariance matrix from equation (9) There is a need. Here, the motion parameters _{m m = [T x T y} T z R x R y R z] T (10)
It is. Then, the corresponding sub-matrix is extracted from V _h ^-1 and used as the inverse matrix of the variance for each variable. In other words, regarding the motion parameters of the camera, the sub-matrix from 1 row 1 column to 6 rows 6 columns of V _h ⁻¹ is defined as the inverse matrix V _m ⁻¹ of the variance of the motion parameters, and the i-th inverse depth is i + 6 The diagonal component in the row i + 6 is defined as the reciprocal of the variance 1 / σ _pi ² . This operation is equivalent to estimating the variance of each variable by the conditional probability when the correct value of the other variable is obtained.
[0026]
Here is a summary of what we have said so far. First, the optical flow calculating unit 10, the inverse matrix V _ui ^-1 of movement in each point on the image vectors u _i ^* and its covariance matrix is calculated, the moving speed k of the camera etc. speedometer obtained Can be It is assumed that the moving speed k ^* includes a variance error of σ _k ² . _U ^{i *} and ^{k *} error at this time are assumed to be independent. From these, the observation vector y ^* and the inverse matrix V _y ^-1 of the covariance matrix are created and input to the distance information calculation means 20. Observation vector y ^* arranges each element in the order shown by Formula (3). The matrix V _y ^-1 is obtained by _{transforming the} 2-by-2 matrix V _ui ^-1 into a diagonal element thereof, that is, a 2-by-2 sub-matrix from 2i-1 row 2i-1 column to 2i row 2i column. The elements of the last diagonal component 2N + 1 row and 2N + 1 column are 1 / σ _k ² . The remaining part is set to 0. Equation (5) is minimized using y ^* and V _y ^-1 created in this manner, θ _ML is calculated, and V _h ^-1 is obtained by equation (9). Furthermore, to obtain a _V ^{m -1} and 1 / sigma _pi ² decomposes _{m ML} and _{p MLi,} the ^{V h -1} to decompose theta _ML equation (10) as a reference. The specific forms of V _m ^-1 and 1 / σ _pi ² are calculated from equations (1) and (9) and are as follows.
[0027]
(Equation 3)

[0028]
In the distance information fusion unit 30 constitute a Kalman filter independently for motion parameters m and the inverse depth p i of the camera as described above. Since θML and the inverse matrix V h -1 of its covariance matrix have been obtained by the distance information calculation means 20, a Kalman filter can be theoretically configured using θ as a state vector. However, when such a Kalman filter is configured, it is necessary to actually obtain the inverse matrix of V h -1 by numerical calculation, and there is a problem in terms of processing amount and stability due to the large number of dimensions. To avoid it, decomposing θ to the camera motion parameters m and the inverse depth p i, constitute a Kalman filter independently.
[0029]
First, a Kalman filter of a camera motion parameter m will be described. Translation and rotation speed of the camera is considered to be constant in the short time, the state transition matrix and a unit matrix and the covariance matrix of system noise and Q m, time update algorithm is as follows.
[0030]
m t + 1 - = m t + (12)
V mt + 1 - = V mt + + Q m
On the other hand, the observation update algorithm is given by the following equation.
[0031]
m t + = m t - + K mt (m t * -m t -) (13)
V mt + = (I-K mt) V mt -
K mt = V mt - (V mt - + V mt *) -1
In these equations, the suffixes t and t + 1 of the vector and the matrix denote the time step, and + and-on the shoulder denote the observation update value and the time update value, respectively. Also, I is the identity matrix, m t * and V mt * is an observation value and its dispersion at the time t, using the inverse matrix of the calculated m ML and V m -1 at a distance information calculating unit 20.
[0032]
Next, a Kalman filter with inverse depth p i is configured. The point on the object whose three-dimensional coordinates were Xt * at time t is represented by a coordinate value Xt + 1 * after a unit time if the translation and rotation speeds of the camera at that time are Tt * and Rt *. It is given by the following equation.
[0033]
X t + 1 * = -T t * -R t * × X t * (14)
X t * = (X t Y t Z t) T
Xt + 1 * = ( Xt + 1Yt + 1Zt + 1 ) T
Above formula relationship inverse depth of both time from the relationship between the Z t and Z t + 1 of it is as follows.
[0034]
p it + 1 = p it / (1 + R yt x it -R xt y it -T zt p it) (15)
Here, differentiating the above equation with pit gives the following equation.
[0035]
dp it + 1 / dp it = (T zt p it) / (1 + R yt x it -R xt y it -T zt p it) 2 + 1 / (1 + R yt x it -R xt y it -T zt p it) (16 )
From equations (15) and (16), a time update algorithm is obtained.
[0036]
p it + 1 - = p it + / (1 + R yt x it -R xt y it -T zt p it +) (17)
σ pit + 1 2- = (dp it + 1 / dp it) 2 σ pit 2+ + Q p
R yt, R xt, T zt is using the corresponding elements of the estimated value m t + Kalman filter of the camera motion parameters. In addition, Q p is a system noise on the inverse depth.
[0037]
On the other hand, the observation update algorithm is as follows.
[0038]
p it + = p it - + K pt (p it * -p it -) (18)
σ pit 2+ = (1-K pt ) σ pit 2-
K pt = σ pit 2 − / (σ pit 2− + σ pit 2 * )
Here, p * and sigma pit 2 * corresponds to the distance information calculated by the calculation means 20 the values p MLi and sigma pi 2 at time t. Now, the Kalman filter for the inverse depth of a point fixed on the object has been configured as described above. The point moves on the image by the movement of the camera. Therefore, the point corresponding to the inverse depth p i at a certain time, since the next time in a different position on the image, it is necessary to relate these by interpolation, this is accomplished as follows You. When applying the observation update algorithm represented by equation (18), optical flow estimation using Equation estimate m t + color type parameter of the motion of that time of the camera calculated in (13) (1) Calculate u i + . Observations p it * predicted value corresponding to the p it -, it means that exists at a position apart on the image -u i + only before a unit time, p it - the position image by interpolating the Is obtained and used as pit − in the equation (18). This state is shown in FIG. 2 as a one-dimensional figure. As the interpolation method, there are a nearest neighbor interpolation method, a bilinear interpolation method, a tertiary convolution interpolation method, and the like, and any of them can be used. Details are given in the following references.
[0039]
Mikio Takagi, Hirohisa Shimoda Supervision: Image Analysis Handbook, University of Tokyo Press, pp. 411-444 (1991)
Further, the predicted value p it - there is a method of calculating the estimated value of only the previous motion parameter position unit time m t-1 +. It estimates the time t-1 of the optical flow u i + by substituting m t-1 + in formula (1), p it as shown in Figure 3 - to move the position. The moved position is interpolated as a sample point to obtain the value of the position of the observed value pit * . In this method, the interpolation problem is somewhat complicated because the sample points of the image to be interpolated are not on a regular grid. The following method is an example of interpolation in this case. First, three sample points that are closest to the point to be calculated and that include that point are determined. The position of these sampling points each (x a, y a), (x b, y b), (x c, y c) and the value p a, p b, When p c, the position (x, The value p at point y) is given by:
[0040]
p = (c 1 x + c 2 y + c 3 ) / c 4 (19)
c 1 = (y a -y b ) p a + (y a -y c) p b + (y b -y a) p c
c 2 = (x b -x c ) p a + (x c -x a) p b + (x a -x b) p c
c 3 = (x c y b -x b y c) p a + (x a y c -x c y a) p b + (x b y a -x a y b) p c
c 4 = (x b -x c ) y a + (x c -x a) y b + (x a -x b) y c
Predicted value p it - and it is also necessary likewise distributed sigma pit 2-interpolation, which is also achieved by the method described above. However, in the case of variance, an operation is performed in which a square root is obtained and converted to a standard deviation, interpolation is performed, and then squared to return to variance.
[0041]
【The invention's effect】
By using the method and the apparatus shown by the present invention, the distance information obtained from each frame of the moving image and the movement of the camera can be temporally fused, and more accurate information can be stably obtained.
[Brief description of the drawings]
FIG. 1 is a configuration diagram showing one embodiment of a three-dimensional shape restoration device of the present invention.
FIG. 2 is a diagram illustrating an inverse depth interpolation method.
FIG. 3 is a diagram illustrating an inverse depth interpolation method.
[Explanation of symbols]
Reference Signs List 10 optical flow calculation means 20 distance information calculation means 30 distance information fusion means 40 distance information Kalman filter 50 motion parameter Kalman filter

Claims (5)

An optical flow calculation procedure for calculating an optical flow that is movement information on an image and a covariance matrix thereof using two frames in a series of images forming a moving image, and an optical flow estimate and Distance information calculation procedure for calculating the distance to the object being imaged and camera motion using the covariance matrix and their covariance matrix, and these two procedures are continuously applied to moving images By applying the independently constructed Kalman filter to the estimated value of the distance to many objects and the estimated value of the camera motion obtained by performing A three-dimensional shape restoring method, which includes a distance information fusion procedure for calculating an estimated value having a high value and a covariance matrix thereof. 2. A method according to claim 1, wherein, in addition to the estimated value of the optical flow and its covariance matrix, the magnitude of the translational motion of the camera and the variance of the noise contained therein are input and used to calculate the object. A three-dimensional shape restoration method, which is reflected in an estimated value of the distance to the camera and an estimated value of the motion of the camera. An optical flow calculation means for calculating an optical flow estimation value, which is movement information on an image, and a covariance matrix thereof using two frames in a series of images constituting a moving image; and the optical flow estimation value, Distance information calculation means for calculating the distance to the object being imaged using the covariance matrix, the estimated value of the motion of the camera, and their covariance matrix, and continuously applying these two methods to moving images Of the distance to many objects and the estimated values of the camera's motion and their covariance matrices in time to obtain a highly accurate estimate of the distance to the object and the motion of the camera and its covariance matrix A three-dimensional shape restoration device comprising:
The three-dimensional shape reconstruction device according to claim 3, wherein the distance information fusion unit constitutes an independent Kalman filter for each of the estimated value of the distance to the object and the estimated value of the motion of the camera calculated by the distance information calculating unit. apparatus. 4. The three-dimensional shape restoration apparatus according to claim 3, wherein the input of the distance information calculating means uses, in addition to the estimated value of the optical flow and its covariance matrix, the magnitude of the translational motion of the camera and the variance of the noise included therein, A three-dimensional shape restoration apparatus characterized by employing distance information fusion means for reflecting an estimated value of a distance to an object to be calculated and an estimated value of a motion of a camera. Optical flow calculation means for calculating an estimated value of the optical flow from the moving image and a covariance matrix of an error estimated for the optical flow,
The estimated value of the optical flow and the covariance matrix and the magnitude of the translational motion of the camera and the variance of the noise included therein are used to estimate the distance information to the object being imaged, the estimated value of the motion parameters of the camera, the distance information, and the camera. Distance information calculating means for calculating a covariance matrix of an error included in the motion parameter,
A distance information Kalman filter to be applied to the estimated value of the distance information and the covariance matrix; and a motion parameter Kalman filter to be applied to the estimated value of the motion parameter of the camera and the covariance matrix. A three-dimensional shape restoration apparatus comprising: a distance information fusion unit that calculates a highly accurate estimated value of a parameter and a covariance matrix of an error.

Images