JP2004108836A - Azimuth angle computing method of imaging apparatus and system, attitude sensor of imaging apparatus, tilt sensor of imaging device and computer program and three-dimentional model configuration device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make a sensor for obtaining a yaw angle unnecessary and obtain the yaw angle more accurately than the conventional general sensor of comparatively low cost for obtaining a yaw angle by obtaining an azimuth angle i.e., a yaw angle of an imaging apparatus from computing, only by obtaining a tilt angle of the imaging apparatus, i.e., roll angle and a pitch angle. <P>SOLUTION: A tilt angle (roll angle and pitch angle) of a camera 1 is obtained by a tilt sensor 2. From the obtained tilt angle, horizon is set as a virtual horizontal line on an imaging face of a camera 1. On the imaging face S of an image picked up with the camera 1, an actual horizontal line L<SB>1</SB>is established on the basis of information expressing an actual horizontal segment L. On the basis of a formula expressing relation on the imaging face S of the horizon and the actual horizontal line L<SB>1</SB>, deviation with the direction of the actual horizontal segment L is obtained as an azimuth angle (yaw angle) of the camera 1. <P>COPYRIGHT: (C)2004,JPO

Description

【００５２】
ここで、ｆはカメラ１のレンズ１１の焦点距離を、ｋ_ｕはカメラ１の画素サイズ（具体的には、ＣＣＤの個々の画素のサイズ）の縦方向のサイズの逆数を、ｋ_ｖは同じく横方向のサイズの逆数を、φは前述した如く画像面Ｓ上の座標軸、即ち画像座標のｕ軸とｖ軸とが空間的になす角度をそれぞれ表している。また、（ｕ_０，ｖ_０）は画像面Ｓ上で光軸、即ちカメラ座標のｚ軸が横切る点（画像中心）の座標値を表している。なお、このカメラ１のパラメータの実際の取得方法としては、テストチャートを撮像することにより、その画像から取得するようにしてもよい。
【００５３】
更に、センサ座標系とカメラ座標系との間のキャリブレーションも行なっておき、両者の相対的位置関係を表す回転行列Ｒｇｃを求めておく。前述したように、カメラ座標とセンサ座標とは、カメラ１に傾きセンサ２が固定されているという前提である場合は相対的には固定された関係にあるので、この時点で求められた回転行列Ｒｇｃは定数となる。
【００５４】
次にステップＳ２として、現実の水平な線分Ｌが撮像されている画像を取得し、そのカメラ１の画像面Ｓ上での像Ｌ_０とし、この像Ｌ_０も線分であるので、これを通る直線、即ち現実の水平な直線Ｌ_１を設定する。そして、この直線Ｌ_１を表す式、即ち画像座標上での座標値を求める。
【００５５】
次にステップＳ３として、傾きセンサ２が取得したカメラ１のロール角及びピッチ角から、画像面Ｓ上の地平線Ｌ_{ｉｎｆｉｎｉ}を下記式（２）により計算して求める。
【００５６】
【数２】

【００５７】
ここで、Ｒｙ（α）はｙ軸周りの回転角αを表す回転行列であり、Ｒｘ（β）はｘ軸周りの回転角βを表す回転行列である。
【００５８】
次に、ステップＳ４として、現実の水平な直線Ｌ_１と地平線Ｌ_{ｉｎｆｉｎｉ}との画像面Ｓ上での位置関係（相対的な位置関係）を表す数式に基づいてカメラ１のヨー角を求める。具体的な例としては、両者の交点の位置と両者のなす角度に関係する数式とに基づいて以下のような手順により、ヨー角を求めることが可能である。
【００５９】
まず、現実の水平な直線Ｌ_１と地平線の像Ｌ_{ｉｎｆｉｎｉ}との交点を求め、その座標を（ｘ_{ｉｎｆｉｎｉ}，ｙ_{ｉｎｆｉｎｉ}）とする。また、ワールド座標系のＸＹ平面上の無限遠点ｍの画像面Ｓ上の像を下記式とおく。
ｍ_{ｉｎｆｉｎｉ}＝［ｘ_{ｉｎｆｉｎｉ}，ｙ_{ｉｎｆｉｎｉ}，１］^Ｔ
そして、ｍ_{ｉｎｆｉｎｉ}を用いて下記式（３）に示すベクトルｐを計算する。
【００６０】
【数３】

【００６１】
このベクトルｐは、ｐ＝［ｐ１，ｐ２，０］^Ｔであるので第３成分が０になっている。従って、センサ基準座標系（傾きセンサ２がリセットされた時点の座標系）を基準としたときのある視点（カメラ位置）のセンサ座標系の方位角をγ、ワールド座標系とセンサ基準座標系とのなす方位角をγ_ｗｇｏとすると、求めるべき方位角（ヨー角）は下記式（４）の右辺にて計算される。
γ＋γ_ｗｇｏ＝　ａｒｃｔａｎ（ｐ２／ｐ１）　…（４）
【００６２】以上に、本発明に係る撮像装置の方位角（ヨー角）の計算方法の概略の手順について説明したが、以下により詳細に説明する。なお、図５の模式図は、現実の水平な線分Ｌの画像面Ｓ上での像Ｌ_０（太い実線）と、これを通って画像面Ｓ上に設定された直線Ｌ_１、即ち現実の水平な直線（破線）と、ワールド座標系との関係を、画像座標系上で見た状態を示している。
【００６３】
まず、現実の水平な線分Ｌの方向をヨー角（方位角）の基準として定め、これをワールド座標系のＸ軸とする。従って、以下においてはこのワールド座標系のＸ軸が方位角の基準となり、実際に算出されるヨー角（方位角）はこの基準の方位角に対する相対的な偏差になる。勿論、必要な場合には、現実の水平な線分Ｌの方向を何らかの方法によって厳密に南北方向、または特定の方位に一致させることにより、絶対的な方位角が算出可能であることはいうまでもない。またこの場合、ワールド座標系のＺ軸は重力加速度の方向（鉛直線の天空向き）とする（ワールド座標系のＹ軸はＸ軸及びＺ軸と直交する方向となる）。
【００６４】
ここで、先のステップＳ２において、現実の水平な線分Ｌの像Ｌ_０を取得する。この場合、線分Ｌの画像面Ｓ上での像Ｌ_０は当然線分になるので、この線分Ｌ_０を
Ｌ_０＝［ａ，ｂ，ｃ］^Ｔ
と表す。その意味は、この線分Ｌを通る画像面Ｓ上の画像系の座標値（ｕ，ｖ）の点ｍ_１を
ｍ_１＝［ｕ，ｖ，１］^Ｔ
とした場合に下記式（５）を満たすということである。
Ｌ_０ ^Ｔｍ_１＝０　　…（５）
【００６５】
最初に地平線（ワールド座標系のＺ＝０において、Ｘ，Ｙが∞（ｉｎｆｉｎｉ）、即ち無限遠となる点の画像面Ｓ上の像の集合）の表現を求める。但し、地平線も当然ながら画像面Ｓ上では直線になるので、前述同様にＬ_{ｉｎｆｉｎｉ}とする。ＸＹ平面上の無限遠点ｍ_{ｉｎｆｉｎｉ}は、射影空間の斉次座標として下記式（６）にて表される。但し、Ｍは三次元空間中の無限遠点である。
【００６６】
【数４】

【００６７】
その画像上の像、即ちＸＹ平面上の無限遠点ｍの像は画像面Ｓ上の像ｍ_{ｉｎｆｉｎｉ}として下記式（７）にて表される。
ｍ_{ｉｎｆｉｎｉ}∝Ｋ［Ｒ｜ｔ］Ｍ　　…（７）
但し、「∝」は両辺が定数倍を許せば等しいことを表す。Ｒ及びｔはワールド座標系（ＸＹＺ座標系）からカメラ座標系（ｘｙｚ座標系）への座標変換を表す回転行列及び平行移動ベクトルをそれぞれ表している。また、［　］内の縦線は両側の行列の要素を単純に並べて配列することを意味している。従って、ここで前述の式（６）を上記の式（７）に代入すると下記式（８）が得られる。
【００６８】
【数５】

【００６９】
地平線Ｌ_{ｉｎｆｉｎｉ}は、ワールド座標系のＸ，Ｙには関係なく、Ｌ_{ｉｎｆｉｎｉ} ^Ｔｍ＝０となること、及び式（８）とから下記式（９）のように表すことができる。
【００７０】
【数６】

【００７１】
本発明においては、傾きセンサ２はヨー角（方位角）は取得しないが、傾きセンサ２をリセットした時点においては傾き角であるロール角及びピッチ角と、方位角であるヨー角とはいずれも０であり、その際のセンサ座標系をセンサ基準座標系と称することは前述した。従って、傾きセンサ２から得られるロール角及びピッチ角をそれぞれα及びβとし、更に傾きセンサ２からヨー角（方位角）が得られると仮定してそれをγとする。更に、センサ基準座標系からある視点（カメラ位置）のセンサ座標系への変換を表す回転行列をＲｇｏｇｎとし、ワールド座標系からセンサ基準座標系への変換を表す回転行列をＲｗｇｏ　とする。また、ｙ軸周りの回転角（ロール角）αを表す回転行列をＲｙ（α）とし、ｘ軸周りの回転角（ピッチ角）βを表す回転行列をＲｘ（β）とし、ｚ軸周りの回転（ヨー角）γを表す回転行列Ｒｚ（γ）とする。以上のようにした場合、ワールド座標系からカメラ座標系への変換を表す回転行列Ｒは下記式（１０）にて表される。
Ｒ＝ＲｇｃＲｇｏｇｎＲｗｇｏ
＝ＲｇｃＲｙ　（α）　Ｒｘ　（β）　Ｒｚ　（γ）　Ｒｗｇｏ　　…（１０）
【００７２】
ワールド座標系からセンサ基準座標系への変換はヨー角（方位角）成分のみであるので、その角度をγ_ｗｇｏとすると、
Ｒｗｇｏ　＝Ｒｚ（γ_ｗｇｏ）
となり、上記式（１０）は下記式（１１）のように表される。
Ｒ＝ＲｇｃＲｙ　（α）　Ｒｘ　（β）　Ｒｚ　（γ＋γ_ｗｇｏ）　　…（１１）
【００７３】
上記式（１１）において未知であるのはγ及びγ_ｗｇｏのみである。ここで、式（１１）を前述の式（９）に代入すると下記式（１２），　（１３）となる。
【００７４】
【数７】

【００７５】
上記式（１３）にはヨー角（方位角）γ及びγ_ｗｇｏを含まないので、ロール角α及びピッチ角βのみから式（１３）により地平線の画像面Ｓ上の像Ｌ_{ｉｎｆｉｎｉ}を作成することが可能になる。一方、画像面Ｓ上に撮像された現実の水平な線分Ｌを空間中で無限遠まで延伸した場合（即ち現実の水平な直線Ｌ_１）の延伸先の無限遠点Ｍ_{ｉｎｆｉｎｉ}は下記式（１４）で表すことができる。
【００７６】
【数８】

【００７７】
無限遠点Ｍ_{ｉｎｆｉｎｉ}の像ｍ_{ｉｎｆｉｎｉ}は前述の式（７）に示されているように、
ｍ_{ｉｎｆｉｎｉ}∝Ｋ［Ｒ｜ｔ］Ｍ
であるので、下記式（１５）のように表される。
【００７８】
【数９】

【００７９】
現実の水平な線分Ｌの画像面Ｓ上で求めた像Ｌ_０を通る直線Ｌ_１と、傾きセンサ２により取得されたロール角α及びピッチ角βとから、前述の式（１３）で求めたＬ_{ｉｎｆｉｎｉ}との交点がこの点である。従って、この点の座標とヨー角（方位角）との関係は前述の式（１１）及び式（１５）から下記式（１６）のように表される。
【００８０】
【数１０】

【００８１】
ここで、式（１６）の左辺を前述の概略の手順のステップＳ５の式（３）に示されているように
ｐ＝［ｐ１，ｐ２，０］^Ｔ
とおくと、ヨー角（方位角）は下記式（１８）の右辺から計算可能である。
γ＋γ_ｗｇｏ＝　ａｒｃｔａｎ（ｐ２／ｐ１）　…（１８）
【００８２】
以上のような手順により、カメラ１のヨー角（方位角）をいかなるセンサによっても実際に検出することなしに、カメラ１が撮像した画像とその際に傾きセンサ２により取得されたカメラ１の傾き角、即ちロール角α及びピッチ角βとから計算により求めることが可能であることが理解される。なお、カメラ１により撮像された画像が１枚のみである場合には、カメラ１のヨー角は現実の水平な線分Ｌの方向に対する偏差、換言すれば相対的な方位角として求まるのみである。しかし、カメラ１を移動させつつ複数の画像を撮像した場合には、各画像が撮像された際のカメラ１の他の画像が撮像された際のカメラ１に対する方位角の偏差、即ちヨー角が判明することになる。
【００８３】
図６は上述のような各座標系の間での変換の状態を示す模式図であり、図７はワールド座標系とセンサ基準座標系とある視点（カメラ位置）でのセンサ座標系とのヨー角（方位角）の関係を示す模式図である。
【００８４】
なお、上述の実施の形態においては、線分Ｌの画像面Ｓ上での像Ｌ_０（または像Ｌ_０を通る現実の水平な直線Ｌ_１）と地平線の像Ｌ_{ｉｎｆｉｎｉ}との交点を求め、この交点の座標（ｘ_{ｉｎｆｉｎｉ}，ｙ_{ｉｎｆｉｎｉ}）に基づいて現実の水平な線分Ｌの方向に対する方位角の偏差としてカメラ１のヨー角を求めるようにしている。具体的には、前述の式（７）、即ち
ｍ_{ｉｎｆｉｎｉ}∝Ｋ［Ｒ｜ｔ］Ｍ　　…（７）
の左辺のｍ_{ｉｎｆｉｎｉ}に上述の交点の値を代入しているが、これに代えて「Ｌ_０×Ｌ_{ｉｎｆｉｎｉ}」、即ちベクトルＬ_０とベクトルＬ_{ｉｎｆｉｎｉ}との外積を代入することにより、同様の結果を得ることが可能である。
【００８５】
より具体的には、ｍ_{ｉｎｆｉｎｉ}は直線Ｌ_０及びＬ_{ｉｎｆｉｎｉ}の交点である、換言すればｍ_{ｉｎｆｉｎｉ}は直線Ｌ_０及びＬ_{ｉｎｆｉｎｉ}が共に通る点であるので、前述した式（５）と同様にそれぞれに関して、下記式（１９）、（２０）が成立する。
Ｌ_０ ^Ｔｍ_{ｉｎｆｉｎｉ}＝０　　…（１９）
Ｌ_{ｉｎｆｉｎｉ} ^Ｔｍ_{ｉｎｆｉｎｉ}＝０　　…（２０）
【００８６】
このことは、ベクトルｍ_{ｉｎｆｉｎｉ}がベクトルＬ_０及びベクトルＬ_{ｉｎｆｉｎｉ}となす角が９０度であることを意味している。従って、ｍ_{ｉｎｆｉｎｉ}はベクトルＬ_０とベクトルＬ_{ｉｎｆｉｎｉ}との外積である、即ち「Ｌ_０×Ｌ_{ｉｎｆｉｎｉ}」と定義できる。以上のことから、上記式（７）によりカメラ１のヨー角を求めることが可能である。
【００８７】
以上のようにして、複数の画像情報とそれぞれの画像が撮像された際のカメラ１の傾き角、即ちロール角α及びピッチ角βが計算機１へ入力され、計算機１においてそれぞれの画像が撮像された際のカメラ１の現実の水平な線分Ｌの方向に対する偏差として方位角、即ちヨー角が算出される。従って、以後は公知の手法、たとえば特許文献１に開示されているような手法により、カメラ１により撮像された複数の画像を使用して対象物ＯＢの三次元モデル化を行なうことが可能になる。
【００８８】
なお参考として、対象物ＯＢの三次元モデル化は特許文献１に開示されている手法に従えば以下のようにして行うことが可能である。計算機３は、前述のようにしてカメラ１により撮像した複数の画像とそれぞれに対応する姿勢情報（傾き角、即ちロール角及びピッチ角と、方位角、即ちヨー角）から、各画像を撮像した際のカメラ１の位置を計算する。具体的には、計算機３はカメラ１の傾き情報（ロール角、ピッチ角及びヨー角）を使用して各画像上に他の画像の輪郭の像を投影し、各画像上で他の画像の投影中心を始点として輪郭に接する線分を計算する。そして、計算機３は各輪郭に接する線分が等しくなるようにカメラ１の位置を計算し直し、この処理を収束するまで反復することにより、各画像が撮像された際のカメラ１の位置が判明する。
【００８９】
次に、計算機３は対象物ＯＢの輪郭を使用して計算を行なうことによりその対象物ＯＢの三次元モデルを構成する。この際の計算は、たとえば公知のＳｈａｐｅ−ｆｒｏｍ−Ｓｉｌｈｏｕｅｔｔｅｓ法を利用することにより容易に可能である。具体的には、カメラ１の撮像位置が決定した後、ボリュームデータが作成され、このボリュームデータに三角メッシュが設定される。そして、各メッシュの頂点の平滑化及び間引きを行なうことにより、三次元形状が作成される。最後に、画像から対象物ＯＢの表面のテクスチャを取得し、これを復元された三次元形状の表面にマッピングすることにより三次元モデルが完成する。
【００９０】
なお、上述の実施の形態では、方位角であるヨー角の計算と三次元モデルの構成とを同一の計算機３で行なうようにしているが、対象物体が存在する場所、計算量等を考慮して個別のコンピュータで行なうようにしてもよいことはいうまでもない。
【００９１】
図８はコンピュータである計算機３に上述のような方位角（ヨー角）の計算を行なわせるためのコンピュータプログラム４１の内容を示す模式図であり、当初から記憶装置４に記憶させておいてもよいし、たとえばＣＤ−ＲＯＭ４２等のような可搬型の記録媒体に記録されていてもよい。但し、コンピュータプログラム４１が記録媒体に記録されている場合は、記録媒体に記録されているコンピュータプログラムを計算機３に読み込ませる必要があるが、これは一般的な汎用コンピュータに備えられているＣＤ−ＲＯＭドライブ等により読み込ませるか、または通信回線を介してダウンロードすることも可能である。
【００９２】
図８に示すように、本発明のコンピュータプログラム４１は、カメラ１の傾き角、即ちロール角及びピッチ角に基づいてカメラ１の方位角、即ちヨー角をコンピュータに計算させるためのコンピュータプログラムであり、具体的には以下の手順を含む。
【００９３】
カメラ１が被写体を撮像した場合に被写体が撮像される状態を示すパラメータを取得する手順（Ｐ１）。カメラ１のロール角及びピッチ角と、取得したパラメータとから、カメラ１の画像面Ｓ上に仮想的な水平な直線を設定する手順（Ｐ２）。カメラ１が撮像した画像を読み込む手順（Ｐ３）。読み込んだ画像に含まれる現実の水平な線分Ｌを表す情報に基づいて画像面Ｓ上に現実の水平な直線Ｌ_１を設定する手順（Ｐ４）。画像面Ｓ上にそれぞれ設定された仮想的な水平な直線Ｌ_{ｉｎｆｉｎｉ}と現実の水平な直線Ｌ_１との画像面Ｓ上での関係を表す数式に基づいてカメラ１のヨー角を求める手順（Ｐ５）。
【００９４】
なお、コンピュータプログラム４１は更に、上述のようにして求められたヨー角を含む傾き情報と画像情報とから、たとえば特許文献１に開示されている手法に従って三次元モデルを構成するコンピュータプログラム（Ｐ６）をも含んでいる。
【００９５】
【発明の効果】
以上に詳述したように本発明の方位角計算方法及び装置によれば、撮像装置の傾き角、即ちロール角及びピッチ角を取得するのみにて、その際の撮像装置の方位角、即ちヨー角が計算により求まる。このため、ヨー角を取得するためのセンサが不要になると共に、従来一般的な比較的安価なヨー角取得用のセンサよりも正確にヨー角を取得することが可能になる。
【００９６】
また本発明の撮像装置のための姿勢検出装置によれば、センサは撮像装置の傾き角、即ちロール角及びピッチ角のみを取得し、このセンサが取得した撮像装置のロール角及びピッチ角に基づいて方位角計算装置が撮像装置の方位角、即ちヨー角を計算により求める。このため、ヨー角を取得するためのセンサが不要になると共に、従来一般的な比較的安価なヨー角取得用のセンサよりも正確にヨー角を取得することが可能になる。
【００９７】
更に本発明の撮像装置の傾きセンサによれば、撮像装置の傾き角、即ちロール角及びピッチ角のみを取得して撮像装置の姿勢検出装置に与える。姿勢検出装置は、傾きセンサから与えられた撮像装置の傾き角（ロール角及びピッチ角）に基づいて撮像装置の方位角（ヨー角）を計算により求める。このため、ヨー角を取得するためのセンサが不要になると共に、従来一般的な比較的安価なヨー角取得用のセンサよりも正確にヨー角を取得することが可能になる。
【００９８】
また更に本発明のコンピュータプログラムによれば、汎用のコンピュータにインストールすることにより、上述のような本発明の方位角計算方法が実行され、またそのような汎用コンピュータは前述のような本発明の方位角計算装置として機能する。換言すれば、そのような汎用コンピュータを前述のような本発明の方位角計算装置として機能させることが可能になるので、ロール角及びピッチ角のみを取得するセンサと、汎用コンピュータとで本発明の撮像装置の傾きセンサ及び三次元モデル構成装置を構成することが可能になる。
【００９９】
また更に本発明の三次元モデル構成装置によれば、撮像装置の傾き角、即ちロール角及びピッチ角を取得するのみで、これらに基づいて撮像装置の方位角、即ちヨー角が求められ、複数の画像の撮像時の撮像装置の位置が求められる。そして、撮像装置で撮像した複数の画像それぞれの撮像時の撮像装置の位置と、撮像された画像の対象物の像の輪郭とから、対象物の三次元モデルが構成される。従って、ロール角及びピッチ角のみを取得するセンサと、汎用コンピュータとで本発明の三次元モデル構成装置を構成することが可能になると共に、より良質な三次元モデルを構成することが可能になる。
【図面の簡単な説明】
【図１】本発明に係る三次元モデル構成装置のシステム構成例を示すブロック図である。
【図２】本発明の三次元モデル構成装置により対象物の三次元モデルを構成するために対象物ＯＢを撮像する状況を示す模式図である。
【図３】本発明に係る撮像装置のヨー角計算方法の概略の手順を示すフローチャートである。
【図４】ワールド座標、カメラ座標、画像座標、及びセンサ座標それぞれの間の関係を示す模式図である。
【図５】水平な線分の画像面上での像とワールド座標系との関係を、画像座標系上で見た場合の状態を示す模式図である。
【図６】各座標系の間での変換の状態を示す模式図である。
【図７】ワールド座標系とセンサ基準座標系とある視点（カメラ位置）でのセンサ座標系とのヨー角（方位角）の関係を示す模式図である。
【図８】本発明のヨー角の計算を行なわせるためのコンピュータプログラムの内容を示す模式図である。
【符号の説明】
１　　カメラ
２　　傾きセンサ
３　　計算機（コンピュータ）
４　　記憶装置
４１　コンピュータプログラム
４２　ＣＤ−ＲＯＭ（記録媒体）
α　　ロール角
β　　ピッチ角
γ　　ヨー角
ＯＢ　三次元モデル化の対象物
Ｌ　　現実の水平な線分
Ｌ_０　現実の水平な線分Ｌの画像面上の像
Ｌ_１　画像面上に設定された現実の水平な直線
Ｌ_{ｉｎｆｉｎｉ}　　画像面上に設定された仮想的な地平線の像[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an azimuth angle calculation method for an imaging device, an apparatus therefor, a posture detection device of the imaging device, a tilt sensor of the imaging device, and a computer program. Further, the present invention uses the azimuth angle (yaw angle) of the imaging device calculated in this way together with the inclination angle (roll angle and pitch angle) of the imaging device obtained by the sensor to obtain an object imaged by the imaging device. The present invention relates to a three-dimensional model forming apparatus for forming a three-dimensional model.
[0002]
[Prior art]
The three-dimensional model mainly includes three-dimensional shape data of an object, a surface pattern (image thereof), and the like as data. With respect to an object in which such a three-dimensional model is configured by a computer, an image when the object is viewed from an arbitrary direction (viewpoint) can be generated and displayed on the computer. For this reason, in recent years, by constructing a three-dimensional model based on CAD data, various tests are performed as if the real object exists even though the real object does not exist. The range of use is expanding for such purposes as creating an electronic catalog using the three-dimensional model.
[0003]
There is no problem if a three-dimensional model can be constructed from CAD data or a product design drawing, but a simple object such as a blueprint, such as a Buddha statue or ceramics, can easily be used as a three-dimensional model. There is a great demand for a method or an apparatus for acquiring the information in the recent era of multimedia. In response to such demands, conventionally, an object to be three-dimensionally modeled is mounted on a turntable and the turntable is rotated, and a plurality of images are captured by an imaging device fixed outside the turntable. A method for obtaining a three-dimensional model from an image has already been put to practical use. According to such a conventional technique, it is possible to obtain a relatively high-quality three-dimensional model without requiring a large-scale device.
[0004]
As a typical example of the above-described conventional techniques, a technique (the Shape-from-Silhouettes method) of restoring the shape of an object from the appearance of an outline when the object is viewed from a plurality of angles is relatively wide. Are known.
[0005]
However, the related art using the turntable as described above has the following problems. First, there is a limit to the size and weight of the object to be three-dimensionally modeled. In other words, it is necessary to prepare a turntable that can withstand the size and weight of the object to be three-dimensionally modeled, but this naturally has limitations. In addition, it cannot be applied to objects that cannot be moved due to other factors besides size and weight, in other words, objects that cannot be put on a turntable, such as a Buddha statue carved on a rock wall. Needless to say.
[0006]
Further, in the related art using a turntable as described above, there is a problem that the direction in which an object is viewed is limited. In the Shape-from-Silhouettes method, it is important to use a contour when the object is viewed from more angles to construct a better three-dimensional model. However, when a turntable is used, the viewpoint (the position of the imaging device) is limited, so that the viewpoint must be insufficient from the viewpoint described above.
[0007]
For example, there is a problem that the shape of the concave portion on the surface of the object cannot be restored. Specifically, if there is a part whose main curvature is negative on the surface of the object, the part cannot be accurately restored, and conversely, only that part is replaced by the convex hull of that part. In some cases, the shape was restored to a proper shape. However, such restoration is only possible in an ideal case. In reality, a portion where one of the curvatures is negative exists on the surface of the object, and the change in the viewing direction (the imaging direction by the imaging device) is not sufficient, and only from the direction having such a negative curvature. When looking at the contour of an object, it is very unlikely that that part will be correctly restored. In this case, a so-called visual {hull} effect was observed in which the part was restored as a convex hull.
[0008]
As described above, when the image pickup apparatus is fixed and the object is rotated on the turntable, no matter how much the rotation angle is reduced and the image is taken from a number of viewpoints (image pickup apparatus positions), the view point of the object (the image pickup apparatus position) Since the angle from the image pickup device position) is restricted, there is an unavoidable aspect that the above-described problem occurs.
[0009]
In order to solve the above problems, it is sufficient to image the object from all possible angles around the object. However, for that purpose, it is possible to freely change the position / posture of the imaging apparatus, and it is necessary to accurately know the values representing the position / posture (inclination and direction). As a conventional method for this purpose, feature points are set on a background image or on the surface of an object to be three-dimensionally modeled, and feature points in a plurality of images are tracked or associated with each other by a structure {from} motion method. A technique for restoring the position and orientation of an imaging device from an image is known.
[0010]
For example, Niem et al. Previously set a calibration pattern around an object to be three-dimensionally modeled, and restored the position and orientation of the imaging device from the state of the calibration pattern in the captured image. We propose a method to construct a three-dimensional model. However, in this method, even if feature points can be extracted from a plurality of images and the feature points can be extracted, it may be difficult to associate the plurality of feature points with each other. The necessity of a calibration pattern means that the range of application is limited. For example, a relatively large object such as a building such as a building or a car may be imaged while moving the imaging device. In that case, if a pattern for calibration is required, the pattern must be used. It must be captured in the image. However, since such a pattern is arranged at a fixed position, the movement of the imaging device must be a great restriction.
[0011]
In view of the above circumstances, the following invention has been made by one of the present inventors (see Patent Document 1).
[0012]
[Patent Document 1]
JP 2001-307074 A
[0013]
The invention disclosed in Patent Document 1 includes a posture sensor that obtains posture information of an imaging device when an image is captured, and captures each image based on the posture information of the imaging device obtained by the posture sensor. The position of the imaging device at the time is calculated. Then, a three-dimensional model of the target object is configured using the positions of the imaging device obtained by the calculation regarding the plurality of images and the contour of the target object for three-dimensional modeling from the obtained images. The invention of Patent Document 1 is very effective not only for an object that cannot be placed on a turntable but also for an object that is placed on a turntable. For example, even if the planar positional relationship of the imaging device with respect to the turntable is fixed, the image of the target object can be captured from a greater number of positions by moving the imaging device up and down, that is, moving up and down. Because it becomes.
[0014]
[Problems to be solved by the invention]
However, in the invention of Patent Document 1 described above, a gyro sensor is used as a posture sensor, that is, a sensor that detects a roll angle and a pitch angle as tilt angles and a yaw angle as an azimuth angle. However, since a small and high-accuracy gyro sensor is very expensive, it is actually necessary to use a relatively inexpensive gyro sensor, and therefore, the accuracy of the gyro sensor, especially the azimuth (yaw angle) There is a problem that the accuracy is insufficient and a high-accuracy three-dimensional model cannot be obtained.
[0015]
In addition to measuring the inclination angle (inclination with respect to a plane perpendicular to the direction of gravitational acceleration), that is, not only the gyro sensor, but also the roll angle (inclination in the left and right direction) and the pitch angle (elevation angle) Acquisition) is relatively easy because the direction of the gravitational acceleration or the inclination with respect to the horizontal plane may be detected. However, it is easy to measure the azimuth (direction in a plane perpendicular to the direction of gravitational acceleration), that is, the yaw angle (obtain the data) when the imaging device is not moved, but it is easy to measure a large number of images. It is relatively difficult to move the image pickup apparatus to pick up an image, since the reference target is actually only the geomagnetic direction.
[0016]
The present invention has been made in view of the above circumstances. That is, in view of the fact that the angle of inclination (roll angle and pitch angle) can be detected with sufficiently high accuracy by an angle sensor having a relatively simple configuration, a three-dimensional model is obtained in the invention disclosed in Patent Document 1. The main purpose is to obtain the azimuth angle (yaw angle) of the imaging device necessary for the calculation from the inclination angle (roll angle and pitch angle). Specifically, at least the inclination angle (roll angle and pitch angle) of the imaging device is acquired by an angle sensor having a relatively simple configuration capable of detecting the inclination angle (roll angle and pitch angle) with sufficiently high accuracy, An object of the present invention is to provide a method and an apparatus for calculating an azimuth angle of an image pickup apparatus that enable the azimuth angle (yaw angle) of the image pickup apparatus to be calculated with high accuracy based on this.
[0017]
The present invention also relates to a posture detection device, a tilt sensor, and a computer program of an imaging device used in the above-described azimuth angle calculation method and device, and further relates to a tilt angle (roll angle and pitch angle) detected by the tilt sensor, and It is an object of the present invention to provide a three-dimensional model forming apparatus for forming a three-dimensional model from an azimuth (yaw angle) calculated based on the above.
[0018]
[Means for Solving the Problems]
An azimuth calculation method for an imaging device according to the present invention is a method for calculating an azimuth angle of the imaging device based on a tilt angle of the imaging device, wherein a virtual azimuth angle is calculated on an image plane of the imaging device based on the tilt angle. A horizontal straight line is set, and a real horizontal straight line is set on the image plane based on information representing a real horizontal line segment included in the image captured by the imaging device, and each is set on the image plane. The azimuth of the imaging device is obtained based on a mathematical expression representing a relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line.
[0019]
In such an azimuth calculation method of the present invention, a virtual horizontal straight line is set on the image plane of the imaging device based on the inclination angle (roll angle and pitch angle), and is included in the image captured by the imaging device. Information representing an actual horizontal line segment, for example, a bar of an appropriate length placed on a horizontal plane, a line segment marked on the horizontal plane, and at least two lines set to represent a line segment parallel to the horizontal plane And the like, a real horizontal straight line is set on the image plane. Then, the azimuth (yaw angle) of the imaging device, specifically, based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the actual horizontal straight line respectively set on the image plane, The deviation from the direction of the actual horizontal straight line is obtained.
[0020]
An azimuth calculation device for an imaging device according to the present invention is an azimuth calculation device for an imaging device that obtains an azimuth angle of the imaging device based on a tilt angle of the imaging device. Means for setting a virtual horizontal straight line on the image plane, and a real horizontal straight line on the image plane based on information representing a real horizontal line segment included in the image captured by the imaging device. Means for setting, the azimuth of the imaging device based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line respectively set on the image plane And means for requesting.
[0021]
In such an azimuth angle calculation device of the present invention, a virtual horizontal straight line is set on the image plane of the imaging device based on the inclination angle (roll angle and pitch angle) of the imaging device, and the image captured by the imaging device is obtained. The information representing the actual horizontal line segment included in, for example, a bar of an appropriate length placed on the horizontal plane, a line segment marked on the horizontal plane, and a line segment parallel to the horizontal plane were set. An actual horizontal straight line is set on the image plane based on at least two points and the like. Then, the azimuth (yaw angle) of the imaging device, specifically, based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the actual horizontal straight line set on these image planes, respectively. , The deviation from the direction of the actual horizontal straight line is obtained.
[0022]
Further, the present invention is a posture detecting device for an imaging device provided with the azimuth calculating device as described above. Specifically, a sensor that obtains the tilt angle of the imaging device, a unit that sets a virtual horizontal straight line on the image plane of the imaging device based on the tilt angle obtained by the sensor, Means for setting a real horizontal straight line on the image plane based on information representing a real horizontal line segment included in the captured image, and the virtual horizontal straight lines respectively set on the image plane And an azimuth calculation device having means for calculating an azimuth of the imaging device based on a mathematical expression representing a relationship between the actual horizontal straight line and the image plane.
[0023]
In such an attitude detection device for an imaging device according to the present invention, the sensor acquires a tilt angle (a roll angle and a pitch angle) of the imaging device. The azimuth angle calculation device sets a virtual horizontal straight line on the image plane of the imaging device based on the inclination angles (roll angle and pitch angle) acquired by the sensor, and sets the actual straight line included in the image captured by the imaging device. Information representing a horizontal line segment, for example, a bar of appropriate length placed on a horizontal plane, a line segment marked on the horizontal plane, and at least two points set to represent a line segment parallel to the horizontal plane And the like, to set a real horizontal straight line on the image plane. Then, the azimuth (yaw angle) of the imaging device, specifically, based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the actual horizontal straight line set on these image planes, respectively. , The deviation from the direction of the actual horizontal straight line is obtained.
[0024]
Further, the tilt sensor of the imaging device of the present invention is a sensor for a posture detection device of the imaging device. The attitude detection device of the imaging device includes: a unit that sets a virtual horizontal straight line on an image plane of the imaging device based on a tilt angle of the imaging device; and a real horizontal line included in an image captured by the imaging device. Means for setting a real horizontal straight line on the image plane based on information representing a line segment, and the real horizontal straight line and the real horizontal straight line respectively set on the image plane. Means for obtaining an azimuth of the imaging device based on a mathematical expression representing a relationship on an image plane. The tilt sensor according to the present invention is characterized in that the tilt angle of the imaging device is obtained and given to such a posture detecting device of the imaging device.
[0025]
The tilt sensor (roll angle and pitch angle) of the image capturing apparatus according to the present invention acquires the tilt angle (roll angle and pitch angle) of the image capturing apparatus and provides it to the attitude detecting apparatus of the image capturing apparatus. Accordingly, the posture detection device of the imaging device sets a virtual horizontal straight line on the image plane of the imaging device based on the inclination angle (roll angle and pitch angle) of the imaging device, and outputs the image to the image captured by the imaging device. Information representing a real horizontal line segment included, for example, a bar of appropriate length placed on a horizontal plane, a line segment marked on the horizontal plane, at least set to represent a line segment parallel to the horizontal plane An actual horizontal straight line is set on the image plane based on two points and the like. Then, the attitude detection device of the imaging device determines the orientation of the imaging device based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the actual horizontal straight line respectively set on these image surfaces. An angle (yaw angle), specifically, a deviation from the direction of an actual horizontal straight line is obtained.
[0026]
Further, the present invention is a computer program for causing a computer to calculate an azimuth angle of an imaging device based on a tilt angle of the imaging device, and causes the computer to execute the following procedure. As a first procedure, a procedure for acquiring a parameter indicating a state in which a subject is imaged when the imaging device has captured the subject. As a second procedure, a procedure of setting a virtual horizontal straight line on the image plane of the imaging device from the inclination angle of the imaging device and the acquired parameters. A third procedure is to read an image captured by the imaging device. As a fourth procedure, a procedure of setting a real horizontal straight line on the image plane based on information representing a real horizontal line segment included in the read image. As a fifth procedure, the azimuth angle of the imaging device based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line respectively set on the image plane Steps to ask.
[0027]
By installing such a computer program of the present invention on a general-purpose computer, the azimuth angle calculation method of the present invention as described above is executed, and such a general-purpose computer executes the azimuth angle of the present invention as described above. Functions as a computing device.
[0028]
Furthermore, a three-dimensional model forming apparatus according to the present invention is a three-dimensional model forming apparatus configured to form a three-dimensional model of an object from a plurality of images obtained by imaging the object with an imaging device. It is characterized by having requirements. That is, a means for acquiring the tilt angle of the imaging device is provided. Means for setting a virtual horizontal straight line on the image plane of the imaging device based on the obtained inclination angle of the imaging device; and a real horizontal line included in the image captured by the imaging device. Means for setting an actual horizontal straight line on the image plane based on information representing a line segment. Furthermore, the azimuth angle of the imaging device based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line respectively set on the image plane by these means. And means for calculating the position of the imaging device from the plurality of images captured by the imaging device and the tilt angle and azimuth of the imaging device at the time of capturing each image. And a means for configuring a three-dimensional model of the object from the position of the imaging device at the time of capturing each of the plurality of images captured by the imaging device and the outline of the image of the target in the captured image. .
[0029]
In such a three-dimensional model configuration device of the present invention, the inclination angle (roll angle and pitch angle) of the imaging device is acquired, and based on these, a virtual horizontal straight line is set on the image plane of the imaging device, Information representing the actual horizontal line segment included in the image captured by the imaging device, for example, a bar of an appropriate length placed on a horizontal plane, a line segment marked on the horizontal plane, a line segment parallel to the horizontal plane An actual horizontal straight line is set on the image plane based on at least two points and the like set to represent. Further, the azimuth angle (yaw angle) of the image pickup device is determined based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the actual horizontal straight line respectively set on these image planes. Is calculated from the direction of the actual horizontal straight line, and the inclination angle (roll angle and pitch angle) and azimuth angle (yaw angle) of the plurality of images captured by the imaging device and the imaging device when each image is captured. ), The position of the imaging device is obtained. Then, a three-dimensional model of the object is formed from the position of the imaging device at the time of capturing each of the plurality of images captured by the imaging device and the outline of the captured image of the object.
[0030]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, the present invention will be described in detail with reference to the drawings showing the embodiments. FIG. 1 is a block diagram illustrating a system configuration example of a three-dimensional model configuration device according to the present invention.
[0031]
The three-dimensional model forming apparatus according to the present invention includes a camera 1 that is an imaging device for capturing an object to be three-dimensionally modeled (hereinafter, referred to as an object), and a tilt sensor 2 that obtains a tilt angle of the camera 1. And a computer (computer) 3 to which image information is input from the camera 1 and tilt information is input from the tilt sensor 2, respectively. Further, a large-scale storage device 4 such as a hard disk is connected to the computer 3, and the storage device 4 stores the computer program 41 of the present invention and various other programs.
[0032]
In the present embodiment, the tilt angle obtained by the tilt sensor 2 is a roll angle and a pitch angle indicating a tilt with respect to a plane orthogonal to the direction of gravitational acceleration, that is, a horizontal plane, and is obtained based on the tilt information. The azimuth is a yaw angle representing a direction in a plane orthogonal to the direction of gravitational acceleration, that is, in a horizontal plane.
[0033]
The camera 1 may be a video camera, a digital still camera, or the like as long as the camera 1 can input the captured image information and the tilt information acquired by the tilt sensor 2 when the respective image information is captured to the computer 3. Various imaging devices can be used. The image information and the tilt information may be directly input to the computer 3 online, or may be indirectly input to the computer 3 offline, for example, once recorded on some recording medium. If a video camera that captures image information using analog signals is used, the data may be input to the computer 3 and then converted to digital data, or may be converted to digital data using a converter and then input to the computer 3.
[0034]
The camera 1 is provided with a lens 11, and the focal length f of the lens 11 is assumed to be fixed in the present embodiment. However, in principle, the present invention can be implemented even if the focal length f of the lens 11 of the camera 1 is variable.
[0035]
The tilt sensor 2 can be of various configurations, but as the sensor attached to the camera 1 itself, for example, a known acceleration sensor that detects tilt using gravitational acceleration can be used. is there. In the case where the camera 1 is fixed to a tripod for imaging, a sensor that measures the inclination of the camera platform with reference to a state where the camera platform of the tripod is horizontal may be used. In any case, the inclination sensor 2 may be any sensor that can detect at least the roll angle and the pitch angle. For example, various sensors such as a sensor capable of detecting a tilt with respect to the direction of the gravitational acceleration and a sensor capable of detecting a tilt with respect to a horizontal plane can be used as the tilt sensor 2. 1 and its fixing method, imaging target, surrounding environment, and the like can be appropriately selected.
[0036]
In addition, it is desirable that the relative positional relationship between the tilt sensor 2 and the camera 1 does not change, and that the relative position between the two does not change. In such a case, there is always a constant relative relationship between the roll angle and the pitch angle measured by the tilt sensor 2 and the roll angle and the pitch angle of the camera 1, so that correction between them is performed. There is no need to perform such operations. However, even in the case where the relative positional relationship between the two changes or the relative change in the posture occurs, it is of course possible to correct as long as those states are known. . In the present embodiment, it is assumed that the information used by the computer 3 among the inclination information input from the inclination sensor 2 to the computer 3 is only the roll angle and the pitch angle.
[0037]
When the camera 1 actually captures an image of the object OB, if the roll angle and the pitch angle of the camera 1 are fixed to predetermined angles, it is not necessary to use the tilt sensor 2. Further, since the roll angle is a left-right inclination (more specifically, a left-right inclination with respect to a horizontal plane), it is fixed to 0. In other words, the camera 1 can capture an image of an object with no left-right inclination. If possible, the inclination sensor 2 may measure only the pitch angle. In such a state, for example, the object OB is placed on a turntable and rotated, and the camera 1 itself is raised and lowered while keeping the roll angle fixed at 0 and not moving in a planar manner. This is effective in a case where an image is taken at a different position. In addition, fixing the roll angle and the pitch angle of the camera 1 or fixing either the roll angle or the pitch angle to 0 requires a tripod with a pan head that can precisely adjust the roll angle and the pitch angle. It can be easily realized by using it.
[0038]
The computer 3 inputs the image of the object captured by the camera 1 as image information, and inputs the tilt information given from the tilt sensor 2 at that time, that is, the roll angle and the pitch angle measured by the tilt sensor 2. Then, the computer 3 calculates the yaw angle (azimuth angle) of the camera 1 based on the roll angle and the pitch angle measured by the tilt sensor 2 at the time when the camera 1 captures the image of the object based on the information in the procedure described later. ) Functions as an azimuth angle calculation device. At the same time, the computer 3 functions as a posture detection device of the camera 1 by calculating the yaw angle of the camera 1 together with the inclination sensor 2 for obtaining the roll angle and the pitch angle of the camera 1. Further, the computer 3 performs a three-dimensional model of the object based on a roll angle, a pitch angle, and a yaw angle (azimuth angle) of the camera 1 by a known method, for example, a method disclosed in Patent Document 1. It also functions as a constituent device.
[0039]
FIG. 2 is a schematic diagram showing a situation where the camera 1 to which the tilt sensor 2 is fixed captures an image of the object OB in order to form a three-dimensional model of the object OB by the three-dimensional model forming apparatus of the present invention as described above. It is.
[0040]
In the present invention, as will be described later, the azimuth, that is, the yaw angle is calculated from the tilt angle, which is the tilt information acquired by the tilt sensor 2, that is, the roll angle and the pitch angle. When imaging the object OB from a plurality of positions, a straight line parallel to the horizontal plane, in fact, the direction of the actual line segment L is used as a temporary reference. Instead of the actual line segment L, various information that can represent an actual horizontal line segment may be used instead. Such information includes, for example, a bar of an appropriate length placed on a horizontal plane, a line segment marked on the horizontal plane, at least two points set to represent a line segment parallel to the horizontal plane, and the like. Is available. More specifically, when the object OB is a building, a horizontal portion (for example, a window frame or the like) of the building may be used. If it is impossible to set a horizontal plane, for example, a string of an appropriate length may be stretched in parallel with the horizontal plane, or a line segment connecting at least two points to each other as described above. You may use it by setting so that it may become parallel to a horizontal surface.
[0041]
In the case where at least two points are set and used as information representing a real horizontal line segment, for example, a sign such as a cross or a multiplex circle is set, and the center point is used. Then, identification on the image plane of the camera 1 becomes easy. It is also possible to use a light emitter for at least two points. In this case, the use of a light emitter having a specific wavelength makes it extremely easy to specify the image on the image plane of the camera 1.
[0042]
In any case, as shown in FIG. 2, image information obtained as a result of imaging the object OB from a plurality of positions by the camera 1 and inclination information (at least the roll angle and the roll angle) acquired by the inclination sensor 2 at each imaging time point. (Pitch angle) is input to the computer 3.
[0043]
Next, a method of calculating the azimuth angle (yaw angle) of the imaging apparatus according to the present invention will be described. The principle of the calculation method of the azimuth angle (yaw angle) of the imaging apparatus according to the present invention is summarized as follows. A horizon (a set of points at infinity) can be drawn on an image based on the roll angle and the pitch angle detected by the tilt sensor 2 attached to the camera 1. This horizon is a virtual horizontal straight line. In the present embodiment, this horizon is used as a virtual horizontal straight line, but it is possible to set a virtual horizontal straight line on an image captured by the camera 1 even if it is not such a horizon. If so, you may use it. Also, regarding the actual line segment L parallel to the above-mentioned horizontal plane, when it is possible to use a plurality of such line segments (information representing an actual horizontal line segment), the above-described virtual line segment is used. If a horizontal line that is as close to an orthogonal state as possible is selected, the accuracy of the calculation described below is improved.
[0044]
Next, the horizon drawn on this image and the image L on the image obtained by capturing the actual horizontal line segment L₀Find the intersection with However, in consideration of the fact that the two generally do not have intersections, the image L of the actual line segment L on the image is taken into consideration.₀L passing through₁That is, an actual horizontal straight line is set, and this straight line L₁And the intersection with the horizon drawn on the image. Since this intersection is a fixed point located at a virtual infinity, the position of this fixed point on the image plane is determined to determine the yaw angle of the camera 1 (the straight line L₁(The azimuth angle based on the azimuth) can be obtained by calculation.
[0045]
A schematic procedure of the calculation method of the azimuth angle (yaw angle) of the imaging apparatus according to the present invention is as shown in a flowchart of FIG. FIG. 4 is a schematic diagram showing a relationship among world coordinates, camera coordinates, image coordinates, and sensor coordinates.
[0046]
Here, the world coordinates (XYZ coordinate system: origin is Ow) is an absolute coordinate system temporarily used as a reference, and is a real horizontal line segment L, that is, a real horizontal line L₁Is made coincident with the X-axis of the world coordinate system and used as a reference for the yaw angle of the camera 1. In this case, the Z axis of the world coordinate system is set to the direction of the gravitational acceleration (the vertical direction of the sky).
[0047]
The camera coordinates (xyz coordinate system: the origin is Oc) is a coordinate system unique to the camera 1, and in the present embodiment, the optical axis of the camera 1 is the z axis, and the image plane is a plane orthogonal to the z axis. The left-right direction of S is an x-axis, and the up-down direction is a y-axis. However, the camera coordinates may be different from those described above, and in such a case, the process of converting to the above coordinate system may be performed.
[0048]
The image coordinate (uv coordinate system) is a coordinate system set on the image plane S of the camera 1 described above. The left and right directions of the image plane S are generally defined as u axis, and the vertical direction is defined as v axis. Is an angle formed spatially by φ.
[0049]
The sensor coordinates are a three-dimensional coordinate system set in the tilt sensor 2. Initially, all three-dimensional directions are 0, and this state is called a sensor reference coordinate system. Note that the camera coordinates and the sensor coordinates maintain a relatively constant relationship on the assumption that the tilt sensor 2 is fixed to the camera 1.
[0050]
First, as a step S1, the camera 1 used for capturing an image is calibrated to obtain internal parameters of the camera 1, specifically, parameters representing a state in which a subject is imaged. The internal parameters of the camera 1 are expressed as a matrix K by equation (1).
[0051]
(Equation 1)

[0052]
Here, f is the focal length of the lens 11 of the camera 1 and k is_uIs the reciprocal of the vertical size of the pixel size of the camera 1 (specifically, the size of each pixel of the CCD), k_vRepresents the reciprocal of the horizontal size, and φ represents the spatial angle between the coordinate axes on the image plane S, that is, the u-axis and the v-axis of the image coordinates, as described above. Also, (u₀, V₀) Indicates the coordinate value of the point (image center) at which the optical axis, that is, the z axis of the camera coordinates, crosses on the image plane S. Note that as an actual method of acquiring the parameters of the camera 1, a test chart may be imaged and acquired from the image.
[0053]
Further, calibration between the sensor coordinate system and the camera coordinate system is also performed, and a rotation matrix Rgc representing a relative positional relationship between the two is obtained. As described above, the camera coordinates and the sensor coordinates have a relatively fixed relationship when it is assumed that the tilt sensor 2 is fixed to the camera 1. Rgc is a constant.
[0054]
Next, as step S2, an image in which a real horizontal line segment L is captured is acquired, and the image L on the image plane S of the camera 1 is acquired.₀And this image L₀Is also a line segment, so a straight line passing through it, that is, an actual horizontal straight line L₁Set. And this straight line L₁, That is, a coordinate value on the image coordinates is obtained.
[0055]
Next, as step S3, the horizon L on the image plane S is calculated from the roll angle and the pitch angle of the camera 1 acquired by the tilt sensor 2._infiniIs calculated by the following equation (2).
[0056]
(Equation 2)

[0057]
Here, Ry (α) is a rotation matrix representing the rotation angle α around the y axis, and Rx (β) is a rotation matrix representing the rotation angle β around the x axis.
[0058]
Next, as step S4, the actual horizontal straight line L₁And horizon L_infiniThe yaw angle of the camera 1 is obtained based on a mathematical expression representing a positional relationship (relative positional relationship) on the image plane S with the camera. As a specific example, the yaw angle can be obtained by the following procedure based on the position of the intersection between the two and the mathematical expression relating to the angle between the two.
[0059]
First, the actual horizontal straight line L₁And the horizon image L_infiniAnd the coordinates of (x_infini, Y_infini). The image on the image plane S of the infinity point m on the XY plane in the world coordinate system is represented by the following equation.
m_infini= [X_infini, Y_infini, 1]^T
And m_infiniIs used to calculate a vector p shown in the following equation (3).
[0060]
(Equation 3)

[0061]
This vector p is expressed as p = [p1, p2, 0]^TTherefore, the third component is 0. Therefore, the azimuth of the sensor coordinate system at a certain viewpoint (camera position) based on the sensor reference coordinate system (the coordinate system at the time when the tilt sensor 2 is reset) is γ, and the world coordinate system and the sensor reference coordinate system Azimuth angle_wgoThen, the azimuth angle (yaw angle) to be calculated is calculated on the right side of the following equation (4).
γ + γ_wgo= {Arctan (p2 / p1)} ... (4)
The outline of the calculation method of the azimuth angle (yaw angle) of the imaging apparatus according to the present invention has been described above, and will be described in more detail below. The schematic diagram of FIG. 5 shows an image L of the actual horizontal line segment L on the image plane S.₀(Thick solid line) and a straight line L passing through it and set on the image plane S₁In other words, the relationship between the actual horizontal straight line (broken line) and the world coordinate system is shown as viewed on the image coordinate system.
[0063]
First, the direction of the actual horizontal line segment L is determined as a reference for the yaw angle (azimuth angle), and this is defined as the X axis of the world coordinate system. Therefore, hereinafter, the X axis of the world coordinate system is used as a reference for the azimuth, and the actually calculated yaw angle (azimuth) is a relative deviation from the reference azimuth. Of course, if necessary, the absolute azimuth can be calculated by making the direction of the actual horizontal line segment L exactly match the north-south direction or a specific azimuth by some method. Nor. In this case, the Z axis of the world coordinate system is set to the direction of the gravitational acceleration (the vertical direction of the sky) (the Y axis of the world coordinate system is a direction orthogonal to the X axis and the Z axis).
[0064]
Here, in the previous step S2, the image L of the actual horizontal line segment L₀To get. In this case, the image L of the line segment L on the image plane S₀Is naturally a line segment, so this line segment L₀To
L₀= [A, b, c]^T
It expresses. This means that a point m of the coordinate value (u, v) of the image system on the image plane S passing through the line segment L₁To
m₁= [U, v, 1]^T
In this case, the following expression (5) is satisfied.
L₀ ^Tm₁= 0 (5)
[0065]
First, a representation of a horizon (a set of images on the image plane S at a point where X and Y are at infinity at X = 0 in the world coordinate system Z = 0) is obtained. However, since the horizon is also a straight line on the image plane S, L_infiniAnd Infinity point m on XY plane_infiniIs represented by the following equation (6) as homogeneous coordinates in the projective space. Here, M is a point at infinity in the three-dimensional space.
[0066]
(Equation 4)

[0067]
An image on the image, that is, an image at a point m at infinity on the XY plane is an image m on the image plane S._infiniIs represented by the following equation (7).
m_infini{K [R | t] M} (7)
However, “∝” indicates that both sides are equal if a constant multiple is allowed. R and t represent a rotation matrix and a translation vector, respectively, representing a coordinate transformation from a world coordinate system (XYZ coordinate system) to a camera coordinate system (xyz coordinate system). A vertical line in [] means that the elements of the matrix on both sides are simply arranged. Therefore, when the above equation (6) is substituted into the above equation (7), the following equation (8) is obtained.
[0068]
(Equation 5)

[0069]
Horizon L_infiniIs L regardless of the X and Y of the world coordinate system._infini ^TFrom m = 0 and equation (8), it can be expressed as equation (9) below.
[0070]
(Equation 6)

[0071]
In the present invention, the tilt sensor 2 does not acquire the yaw angle (azimuth angle), but when the tilt sensor 2 is reset, both the roll angle and the pitch angle, which are the tilt angles, and the yaw angle, which is the azimuth angle, are not obtained. 0, and the sensor coordinate system at that time is referred to as a sensor reference coordinate system, as described above. Accordingly, the roll angle and the pitch angle obtained from the tilt sensor 2 are set to α and β, respectively, and further, it is assumed that the yaw angle (azimuth angle) is obtained from the tilt sensor 2 and set to γ. Further, a rotation matrix representing a conversion from the sensor reference coordinate system to a sensor coordinate system of a certain viewpoint (camera position) is Rgogn, and a rotation matrix representing a conversion from the world coordinate system to the sensor reference coordinate system is Rwgoｗ. Further, a rotation matrix representing a rotation angle (roll angle) α around the y axis is Ry (α), a rotation matrix representing a rotation angle (pitch angle) β around the x axis is Rx (β), and a rotation matrix around the z axis is Rx (β). It is assumed that the rotation matrix Rz (γ) represents the rotation (yaw angle) γ. In the case described above, the rotation matrix R representing the transformation from the world coordinate system to the camera coordinate system is represented by the following equation (10).
R = RgcRgogRwgo
= RgcRy (α) Rx (β) Rz (γ) Rwgo (10)
[0072]
Since the conversion from the world coordinate system to the sensor reference coordinate system is only the yaw angle (azimuth) component, the angle is represented by γ_wgoThen
Rwgo = Rz (γ_wgo)
The above equation (10) is expressed as the following equation (11).
R = RgcRy (α) Rx (β) Rz (γ + γ_wgo)… (11)
[0073]
The unknowns in the above equation (11) are γ and γ_wgoOnly. Here, substituting equation (11) into equation (9) yields the following equations (12) and (13).
[0074]
(Equation 7)

[0075]
In the above equation (13), yaw angles (azimuth angles) γ and γ_wgoIs not included, the image L of the horizon on the image plane S is obtained from the roll angle α and the pitch angle β by the equation (13)._infiniCan be created. On the other hand, when the real horizontal line segment L imaged on the image plane S is extended to infinity in space (that is, when the real horizontal straight line L₁Infinity point M at the extension point of)_infiniCan be represented by the following equation (14).
[0076]
(Equation 8)

[0077]
Infinity point M_infiniStatue m_infiniIs, as shown in equation (7) above,
m_infini∝K [R | t] M
Therefore, it is expressed as the following equation (15).
[0078]
(Equation 9)

[0079]
Image L obtained on image plane S of actual horizontal line segment L₀L passing through₁And the roll angle α and the pitch angle β obtained by the tilt sensor 2, L_infiniThis is the point of intersection with Therefore, the relationship between the coordinates of this point and the yaw angle (azimuth angle) is expressed by the following equation (16) from the above equations (11) and (15).
[0080]
(Equation 10)

[0081]
Here, the left side of equation (16) is calculated as shown in equation (3) of step S5 of the above-described general procedure.
p = [p1, p2, 0]^T
In other words, the yaw angle (azimuth angle) can be calculated from the right side of the following equation (18).
γ + γ_wgo= {Arctan (p2 / p1)} ... (18)
[0082]
According to the procedure described above, the image captured by the camera 1 and the tilt of the camera 1 acquired by the tilt sensor 2 at that time without actually detecting the yaw angle (azimuth) of the camera 1 by any sensor. It is understood that the angle can be obtained by calculation from the roll angle α and the pitch angle β. When only one image is captured by the camera 1, the yaw angle of the camera 1 is obtained only as a deviation from the direction of the actual horizontal line segment L, in other words, as a relative azimuth. . However, when a plurality of images are captured while moving the camera 1, the deviation of the azimuth angle with respect to the camera 1 when another image is captured when each image is captured, that is, the yaw angle is Will be found out.
[0083]
FIG. 6 is a schematic diagram showing a state of conversion between coordinate systems as described above, and FIG. 7 is a diagram showing yaw between a world coordinate system, a sensor reference coordinate system, and a sensor coordinate system at a certain viewpoint (camera position). It is a schematic diagram which shows the relationship of an angle (azimuth angle).
[0084]
In the above embodiment, the image L of the line segment L on the image plane S₀(Or image L₀Real horizontal straight line L passing through₁) And horizon image L_infiniWith the coordinates of this intersection (x_infini, Y_infini), The yaw angle of the camera 1 is determined as the deviation of the azimuth angle with respect to the direction of the actual horizontal line segment L. Specifically, the above equation (7), that is,
m_infini{K [R | t] M} (7)
M on the left side of_infiniIs substituted for the value of the above-mentioned intersection, but "L₀× L_infini, Ie, the vector L₀And the vector L_infiniThe same result can be obtained by substituting the outer product of
[0085]
More specifically, m_infiniIs a straight line L₀And L_infini, In other words, m_infiniIs a straight line L₀And L_infiniAre both passing points, and the following equations (19) and (20) are established for each of them in the same manner as the above-mentioned equation (5).
L₀ ^Tm_infini= 0 (19)
L_infini ^Tm_infini= 0 (20)
[0086]
This means that the vector m_infiniIs the vector L₀And the vector L_infiniIs 90 degrees. Therefore, m_infiniIs the vector L₀And the vector L_infini, Ie, "L₀× L_infini"Can be defined. From the above, it is possible to obtain the yaw angle of the camera 1 by the above equation (7).
[0087]
As described above, a plurality of pieces of image information and the tilt angle of the camera 1 when each image is captured, that is, the roll angle α and the pitch angle β are input to the computer 1, and each image is captured by the computer 1. The azimuth, that is, the yaw angle, is calculated as the deviation from the direction of the actual horizontal line segment L of the camera 1 when the camera 1 is moved. Therefore, it becomes possible to perform three-dimensional modeling of the object OB using a plurality of images taken by the camera 1 by a known method, for example, a method disclosed in Patent Document 1. .
[0088]
For reference, the three-dimensional modeling of the object OB can be performed as follows according to the method disclosed in Patent Document 1. The computer 3 captures each image from the plurality of images captured by the camera 1 as described above and the corresponding posture information (tilt angle, ie, roll angle and pitch angle, and azimuth, ie, yaw angle). Then, the position of the camera 1 is calculated. Specifically, the computer 3 uses the tilt information (roll angle, pitch angle, and yaw angle) of the camera 1 to project an image of the outline of another image on each image, and displays the image of the other image on each image. A line segment tangent to the contour is calculated starting from the projection center. Then, the computer 3 recalculates the position of the camera 1 so that the line segments tangent to each contour become equal, and repeats this process until the convergence, whereby the position of the camera 1 at the time of capturing each image is determined. I do.
[0089]
Next, the computer 3 configures a three-dimensional model of the object OB by performing calculations using the contour of the object OB. The calculation at this time can be easily performed by using, for example, the well-known Shape-from-Silhouettes method. Specifically, after the imaging position of the camera 1 is determined, volume data is created, and a triangular mesh is set in the volume data. Then, a three-dimensional shape is created by smoothing and thinning out the vertices of each mesh. Finally, the texture of the surface of the object OB is obtained from the image, and is mapped on the restored three-dimensional surface, thereby completing the three-dimensional model.
[0090]
In the above-described embodiment, the calculation of the yaw angle, which is the azimuth, and the configuration of the three-dimensional model are performed by the same computer 3, but the location where the target object exists, the amount of calculation, and the like are taken into consideration. Needless to say, it may be performed by an individual computer.
[0091]
FIG. 8 is a schematic diagram showing the contents of a computer program 41 for causing the computer 3 as a computer to calculate the azimuth angle (yaw angle) as described above, and may be stored in the storage device 4 from the beginning. Alternatively, the program may be recorded on a portable recording medium such as the CD-ROM 42. However, when the computer program 41 is recorded on a recording medium, it is necessary to read the computer program recorded on the recording medium into the computer 3, which is a CD-ROM provided in a general-purpose computer. The data can be read by a ROM drive or the like, or can be downloaded via a communication line.
[0092]
As shown in FIG. 8, a computer program 41 of the present invention is a computer program for causing a computer to calculate the azimuth angle, ie, the yaw angle of the camera 1 based on the tilt angle of the camera 1, ie, the roll angle and the pitch angle. Specifically, it includes the following procedure.
[0093]
Step (P1) of acquiring a parameter indicating a state where a subject is imaged when the camera 1 captures an image of the subject. Procedure for setting a virtual horizontal straight line on the image plane S of the camera 1 from the roll angle and the pitch angle of the camera 1 and the acquired parameters (P2). Procedure for reading an image captured by the camera 1 (P3). A real horizontal straight line L on an image plane S based on information representing a real horizontal line segment L included in the read image.₁(P4). Virtual horizontal straight lines L respectively set on the image plane S_infiniAnd the actual horizontal straight line L₁(P5) of obtaining the yaw angle of the camera 1 based on a mathematical expression representing the relationship on the image plane S with the yaw angle.
[0094]
The computer program 41 further includes a computer program (P6) for forming a three-dimensional model from the tilt information including the yaw angle and the image information obtained as described above, for example, according to the method disclosed in Patent Document 1. Also included.
[0095]
【The invention's effect】
As described in detail above, according to the azimuth angle calculation method and apparatus of the present invention, only the tilt angle of the imaging device, that is, the roll angle and the pitch angle are obtained, and the azimuth angle of the imaging device at that time, that is, the yaw angle, is acquired. The angle is calculated. For this reason, a sensor for acquiring the yaw angle is not required, and the yaw angle can be acquired more accurately than a conventional and relatively inexpensive yaw angle acquisition sensor.
[0096]
Further, according to the attitude detection device for an imaging device of the present invention, the sensor acquires only the inclination angle of the imaging device, that is, the roll angle and the pitch angle, and the sensor acquires the roll angle and the pitch angle of the imaging device acquired by the sensor. Then, the azimuth calculation device calculates the azimuth of the imaging device, that is, the yaw angle. For this reason, a sensor for acquiring the yaw angle is not required, and the yaw angle can be acquired more accurately than a conventional and relatively inexpensive yaw angle acquisition sensor.
[0097]
Further, according to the tilt sensor of the imaging device of the present invention, only the tilt angle of the imaging device, that is, the roll angle and the pitch angle, are acquired and given to the attitude detection device of the imaging device. The attitude detection device calculates the azimuth angle (yaw angle) of the imaging device based on the tilt angle (roll angle and pitch angle) of the imaging device given from the tilt sensor. For this reason, a sensor for acquiring the yaw angle is not required, and the yaw angle can be acquired more accurately than a conventional and relatively inexpensive yaw angle acquisition sensor.
[0098]
Still further, according to the computer program of the present invention, the azimuth calculation method of the present invention as described above is executed by being installed in a general-purpose computer. Functions as an angle calculator. In other words, such a general-purpose computer can be made to function as the azimuth angle calculation device of the present invention as described above. Therefore, a sensor that acquires only the roll angle and the pitch angle and the general-purpose computer according to the present invention are used. It is possible to configure a tilt sensor and a three-dimensional model configuration device of the imaging device.
[0099]
Further, according to the three-dimensional model forming apparatus of the present invention, only the tilt angle of the imaging device, that is, the roll angle and the pitch angle are obtained, and based on these, the azimuth angle of the imaging device, that is, the yaw angle is obtained. The position of the imaging device at the time of imaging of the image is obtained. Then, a three-dimensional model of the object is formed from the position of the imaging device at the time of capturing each of the plurality of images captured by the imaging device and the outline of the captured image of the object. Therefore, the sensor for acquiring only the roll angle and the pitch angle and the general-purpose computer can configure the three-dimensional model forming apparatus of the present invention, and can configure a higher-quality three-dimensional model. .
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a system configuration example of a three-dimensional model configuration device according to the present invention.
FIG. 2 is a schematic diagram showing a situation in which an object OB is imaged in order to constitute a three-dimensional model of the object by the three-dimensional model construction device of the present invention.
FIG. 3 is a flowchart illustrating a schematic procedure of a yaw angle calculation method of the imaging apparatus according to the present invention.
FIG. 4 is a schematic diagram showing a relationship among world coordinates, camera coordinates, image coordinates, and sensor coordinates.
FIG. 5 is a schematic diagram showing a state in which a relationship between an image on a horizontal line segment on an image plane and a world coordinate system is viewed on the image coordinate system.
FIG. 6 is a schematic diagram showing a state of conversion between coordinate systems.
FIG. 7 is a schematic diagram showing a relationship between a yaw angle (azimuth angle) between a world coordinate system, a sensor reference coordinate system, and a sensor coordinate system at a certain viewpoint (camera position).
FIG. 8 is a schematic diagram showing the contents of a computer program for causing a yaw angle to be calculated according to the present invention.
[Explanation of symbols]
1 Camera
2 tilt sensor
3 Computer (computer)
4 Storage device
41 Computer program
42 CD-ROM (recording medium)
α roll angle
β pitch angle
γ yaw angle
Object of OB @ 3D modeling
L real horizontal line segment
L₀像 Image of the actual horizontal line segment L on the image plane
L₁現 実 Real horizontal line set on the image plane
L_infini像 A virtual horizon image set on the image plane

Claims (6)

In a method of calculating an azimuth of the imaging device based on a tilt angle of the imaging device,
Setting a virtual horizontal straight line on the image plane of the imaging device based on the tilt angle,
Setting a real horizontal straight line on the image plane based on information representing a real horizontal line segment included in the image captured by the imaging device;
Determining the azimuth of the imaging device based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line respectively set on the image plane. Of calculating the azimuth angle of the imaging device to perform. In the azimuth calculation device of the imaging device for obtaining the azimuth of the imaging device based on the inclination angle of the imaging device,
Means for setting a virtual horizontal straight line on the image plane of the imaging device based on the inclination angle of the imaging device;
Means for setting a real horizontal straight line on the image plane based on information representing a real horizontal line segment included in the image captured by the imaging device,
Means for obtaining an azimuth of the imaging device based on a mathematical expression representing a relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line respectively set on the image plane. An azimuth calculation device for an image pickup device. A sensor for acquiring an inclination angle of the imaging device;
Means for setting a virtual horizontal straight line on the image plane of the imaging device based on the tilt angle obtained by the sensor; and information representing an actual horizontal line segment included in the image captured by the imaging device. Means for setting a real horizontal straight line on the image plane based on the image plane, and a relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line respectively set on the image plane. And an azimuth calculating device having means for calculating an azimuth of the imaging device based on a mathematical expression representing: Means for setting a virtual horizontal straight line on the image plane of the imaging device based on the tilt angle of the imaging device, and information based on information representing a real horizontal line segment included in the image captured by the imaging device. Means for setting a real horizontal straight line on the image plane, and a relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line respectively set on the image plane. A tilt sensor for the image capturing apparatus, wherein the tilt angle of the image capturing apparatus is obtained and given to a posture detecting apparatus of the image capturing apparatus having means for calculating an azimuth angle of the image capturing apparatus based on a mathematical formula. . A computer program for causing a computer to calculate an azimuth angle of the imaging device based on a tilt angle of the imaging device,
A procedure for acquiring a parameter indicating a state where the subject is imaged when the imaging device has captured the subject,
A procedure for setting a virtual horizontal straight line on the image plane of the imaging device, from the inclination angle of the imaging device and the acquired parameters,
Reading an image captured by the imaging device;
A procedure for setting a real horizontal straight line on the image plane based on information representing a real horizontal line segment included in the read image,
Calculating the azimuth of the imaging device based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line respectively set on the image plane. A computer program characterized by being executed by a computer. In a three-dimensional model configuration device that configures a three-dimensional model of the target object from a plurality of images of the target object captured by the imaging device
Means for acquiring a tilt angle of the imaging device;
Means for setting a virtual horizontal straight line on the image plane of the imaging device based on the tilt angle of the imaging device,
Means for setting a real horizontal straight line on the image plane based on information representing a real horizontal line segment included in the image captured by the imaging device,
Means for determining the azimuth of the imaging device based on a mathematical expression representing the relationship on the image plane between the virtual horizontal straight line and the real horizontal straight line respectively set on the image plane,
Means for determining the position of the imaging device from the inclination angle and the azimuth angle of the imaging device at the time of capturing the plurality of images captured by the imaging device and each image,
Means for configuring a three-dimensional model of the object from the position of the imaging device at the time of imaging each of the plurality of images captured by the imaging device and the outline of the image of the object in the captured image. Characteristic three-dimensional model construction device.

Images