JP3752063B2 - Omnidirectional stereo imaging device - Google Patents

Description

【００２６】
ここで、ａ₁、ｂ₁、ｃ₁は、双曲面の形状を決めるパラメータで、以下の関係がある。
【００２７】
【数２】

【００２８】
図２のように、外側の焦点を原点（０，０，０）に固定した場合、もう一つの内側の焦点２０５の座標は、（０，０，２Ｃ₁）となる。
【００２９】
同様に双曲面ミラー２に関して
【００３０】
【数３】

【００３１】
【数４】

【００３２】
の関係が成り立つ。ここで双曲面ミラー２の内側の焦点２０６の座標は、（０，０，２Ｃ₂）となる。
【００３３】
いま図２の２０７で示される点Ｐ（Ｘｐ、Ｙｐ、Ｚｐ）が、投影面２０４にどのように写るかを示す。点Ｐから焦点２０５に向かう光は、双曲面ミラー１で反射され、原点におかれたカメラのレンズに入射する。このとき投影面２０４に写る像が、２０８の点Ｐ₁（ｘ_p1、ｙ_p1、ｆ）である。
【００３４】
同様に２０７の点Ｐから、双曲面ミラー２の焦点２０６に向かう光が、投影面２０４に結ぶ像は、２０９の点Ｐ₂（ｘ_p2、ｙ_p2、ｆ）で表される。
【００３５】
投影面の像の様子を図３に示す。３０１はカメラの撮影範囲を示し、３０２のリング状の部分が双曲面ミラー１に反射した像の写る範囲であり、３０３の円状の部分が双曲面ミラー２の反射像の写る範囲を示している。
【００３６】
この３０２，３０３が、全方位に対応するステレオ画像対である。このように同一カメラの投影面に、領域が分かれてステレオ画像対が投影されるので、カメラを動画記録可能なビデオカメラなどにすれば、リアルタイムで全方位のステレオ画像が撮影できる。
【００３７】
なお、以上の説明では、全方位を写すためのミラーを、双曲面ミラーで構成した例で説明したが、その他の角錐ミラーでも同様に実施可能である。これについては、更に後述する。
【００３８】
次に、双曲面ミラーから得られたステレオ画像対の関係から、計測点の３次元情報を計測する手法について説明する。
【００３９】
図２の２０７で示された点Ｐが投影面２０４に投影された点Ｐ₁，Ｐ₂は、図３では、それぞれ３０４，３０５で示される位置に投影される。投影面上でのＰ₁、Ｐ₂の座標は、それぞれ（ｘ_p1、ｙ_p1）、（ｙ_p2、ｙ_p2）となる。
【００４０】
３次元空間で同一の点が、２つの双曲面ミラー１，２によって投影された点Ｐ１，Ｐ２の間には、双曲面ミラーがＺ軸に関して回転対称であるという性質から、双曲面ミラーの軸の中心を通る直線上に並ぶという制約が生じる。
言い換えると、空間上の同一点の投影面上での対応点を探索するステレオ対の探索の時に、探索範囲を２次元の平面から、１次元の直線に限定でき、探索にかかる処理を低減できる。
【００４１】
処理の流れを図４を用いて説明する。４０１は、図３で示したカメラでの撮影画像である。４０２は、前記撮影画像からステレオ対応点を探索する対応点計算部であり、４０３は前記対応点計算部からのステレオ対応点の組から、その点の３次元座標を計算する奥行き計算部である。
【００４２】
対応点計算部４０２においては、撮影画像を中心から外側に向かって放射方向に輝度値の一次微分を行い、エッジを抽出する。このエッジデータに対して、ステレオ対応点を相関を計算して探索する。探索範囲は中心から放射状にのびる直線状に限定する。相関値が予め与えたしきい値より高い点の組の座標Ｐ₁（ｘ_p1，ｙ_p1）、Ｐ₂（ｘ_p2，ｙ_p2）を、４０３の奥行き計算部に出力する。
【００４３】
奥行き計算部４０３においては、与えられた２つのステレオ対応点の組から、図２の２０７に示す対応点Ｐの３次元座標（Ｘｐ、Ｙｐ、Ｚｐ）を計算する。
【００４４】
図５に示すように、双曲面のパラメータａ₁、ｂ₁、ｃ₁、カメラの焦点距離ｆ、点Ｐから双曲面の内側の焦点への直線が水平線とのなす角を−α１、双曲面ミラー１からの原点への反射光と水平線がなす角をβ１とすると、
【００４５】
【数５】

【００４６】
【数６】

【００４７】
【数７】

【００４８】
の関係が得られる。ここで、図５では、双曲面ミラー１と点Ｑとの関係を描いた説明図であり、その説明の都合上、双曲面ミラー２の記載を省略している。従って、同図の双曲面ミラー１を双曲面ミラー２と見なすことにより、双曲面ミラー２についても、上記と同様に、
【００４９】
【数８】

【００５０】
【数９】

【００５１】
【数１０】

【００５２】
の関係が成り立つ。
【００５３】
また、ステレオ対応点が、投影面で中心を通る直線の上に制限されるという拘束は、その直線が投影面のＸ軸となす角をθとして
【００５４】
【数１１】

【００５５】
の関係が得られる。
【００５６】
（式５）から（式１１）を元に、Ｘｐ、Ｙｐ、Ｚｐについて求めると
【００５７】
【数１２】

【００５８】
よって、双曲面ミラーの形状のパラメータａ₁，ｂ₁、ｃ₁、ａ₂、ｂ₂、ｃ₂、カメラの焦点距離ｆは判っているので、ステレオ対応点（ｘ_p1，ｙ_p1）、（ｘ_p2，ｙ_p2）の組から（式１２）により、対応する点の３次元座標を計算する。計算された出力が図４の４０４の座標出力である。
【００５９】
（実施の形態２）
図６は、本発明の全方位ステレオ画像撮影装置の実施の形態２の概略構成図を示す。
【００６０】
図６において、６０１は上部角錐ミラー、６０２は下部角錐ミラーであり、その中心線６０５と一致する６０３は上部角錐ミラーの各面の反射像を撮影する上部カメラ群、６０４は下部角錐ミラーの各反射像を撮影する下部カメラ群である。
【００６１】
光学的な配置を図７を用いて説明する。図７は、図６の中心線６０５を含み、角錐の１つの面に垂直に交わる平面に沿った断面の模式図である。７０１、７０２は、それぞれ上部角錐ミラー、下部角錐ミラー、７０３、７０４はそれぞれ上部カメラ、下部カメラである。７０５を２つの角錐ミラーの中心線上の中間点Ｏとする。上部カメラ７０３に写る映像は、ミラー７０１によって反射され、上部カメラ７０３の虚像７０６の位置から見た映像と等価なものになる。同様に下部カメラ７０４の虚像を７０７とする。
【００６２】
水平方向に分割された像を、全方位のステレオに隙間なく接合するためには、各方向のカメラのレンズ中心を一致させる必要がある。１つのカメラで時間をずらして撮影する場合には、レンズ中心も回転の中心として、水平方向に回転すればよいが、同時に複数のカメラで撮影する場合には、同時に２つのカメラを同じ場所に置けないので不可能である。しかしミラーを使って、すべてのカメラの虚像のレンズ中心を１つの点に集めれば、カメラの実体は別々の場所に位置し、等価的なレンズ中心だけを一点に集めることができる。角錐ミラーの場合、このレンズ中心の集まる場所は、角錐ミラーの中心線上になり、各カメラは虚像のレンズ中心が、中心線上の一点に位置するよう配置される。
【００６３】
カメラ７０３の撮影範囲を、線分７０８，７０９で囲まれる領域、カメラ７０４の撮影範囲を線分７１０，７１１で囲まれる領域としたとき、両方のカメラに投影像が写るステレオ計測可能な領域は、７１２で示す斜線の領域となる。
【００６４】
次に、この領域とミラー、カメラの画角、位置との関係を説明する。図７に示すように、角錐ミラーの角度をγとする。上部、下部とも同じ形状のミラーとする。またカメラ７０３，７０４の光軸は鉛直線上を向いているとする。このときミラーによって反射されたカメラの光軸７１３、７１４が水平方向となす仰角をψとすれば、γとψとの関係は次の式のようになる。
【００６５】
【数１３】

【００６６】
７０５の原点０からカメラの虚像のレンズ中心までの距離をＬとして、カメラの上下方向の画角を２ρとする。７０５の原点０から外側に向かって水平な方向をＹ軸、鉛直方向をＺ軸とし、計測領域７１２が始まる点７１５の座標を（０，ｙ₀，０）とおけば、ｙ₀は、
【００６７】
【数１４】
ｙ₀ = Ｌ ／ tan(ψ+ρ)
となる。
【００６８】
計測領域７１２の形状が変化する点７１６のＹ座標をｙ_cとすれば、ｙ₀＜ｙ＜ｙ_cでの計測領域７１２の境界上の点７１６のｚ座標は、
【００６９】
【数１５】
ｚ ＝ ｙ tan(ψ+ρ) ― Ｌ
となる。
【００７０】
ｙ_c＜ｙでの計測領域７１２の境界上の点７１７のｚ座標は、
【００７１】
【数１６】
ｚ ＝ Ｌ ― ｙ tan(ψ-ρ)
となる。
（式１４）、（式１５）から、点７１７の座標ｙ_cは、
【００７２】
【数１７】
ｙ_c ＝ ２Ｌ ／ （ tan(ψ＋ρ) ＋ tan(ψ−ρ) ）
となる。
【００７３】
計測したい対象が存在する領域を、計測領域７１２が覆うように、角錐ミラーの距離Ｌ、仰角ψを調整することにより、望む計測領域のステレオ画像対を取得することができる。
【００７４】
なお、以上の説明では、全方位を写すためのミラーを、角錐ミラーで構成した例で説明したが、その他の球面ミラー、円錐ミラー、双曲面ミラーでも同様に実施可能である。
【００７５】
以上述べたことから明らかな様に本実施の形態にれば、例えば、移動ロボットの視覚系や、人工現実感などの分野における実環境の取得などにおいて、全方位のリアルタイムの画像だけでなく、その画像中の対象物までの距離などの情報を与える全方位のステレオ画像対を得ることのできる全方位のステレオ画像撮影装置を提供することが出来る。
【００７６】
即ち、水平方向の全方位を写すミラー２組を、その互いが他方のミラーに写りこまないよう上下対称に組み合わせる、あるいは、２つの反射像を同一カメラの投影面の異なる位置に像を結ぶよう形状の違うミラーを上下方向に配置することにより、全方位のステレオ画像対をリアルタイムに取得可能となる。
【００７７】
特に曲率の違う２つの双曲面ミラーを組み合わせることにより、全方位のステレオ画像対を１つの映像の中に撮影可能となり、リアルタイムに全方位のステレオ画像撮影が可能となる。また得られた画像からステレオ対応点を探索する場合、放射状の直線上でのみ探索すればよく、高速に対応点の３次元座標を求めることが可能となる。
【００７８】
尚、上記実施の形態１では、全方位を写すためのミラーを、双曲面ミラーで構成した例で説明したが、これに限らず例えば、以下に示すような角錐ミラーで構成しても良い。
【００７９】
即ち、ここでは、全方位を写すためのミラーを、角錐ミラーで構成した実施例について説明する。
【００８０】
図８は、角錐ミラーで構成した本発明の全方位ステレオ画像撮影装置の概略構成図である。図８において、１００１は上部角錐ミラーであり、１００２は下部角錐ミラーであり、１００３は上部、下部角錐ミラーの対応する平面の反射像を同時に撮影するカメラ群である。この場合、１つのカメラには対応する上下の角錐ミラーの面のステレオ像が得られる。この像は、全方位を水平方向に分割したものであり、全カメラの像を合成することで、水平方向３６０度の像を得ることができる。
【００８１】
次に、光学的な配置を、図９を用いて説明する。図９は、図８の角錐ミラーの中心軸を含み、角錐ミラーを構成する平面ミラーの１つに垂直な平面で切断した断面図を示している。１１０１が注目する上部角錐ミラー面、１１０２は、それに対応する下部角錐ミラー面である。１１０３は、それら２つのミラー面に写る反射像を撮影するカメラである。
【００８２】
１１０４に示す点Ｑが、上部、下部ミラー面で反射され、カメラ１１０３の投影面１１０５に写る像は、それぞれｑ１、ｑ２となる。よって、同一の点からの２つのステレオ対が、１つのカメラの投影面上に得られる。
【００８３】
この様にして得られた点ｑ１，ｑ２の座標を利用して、以下に述べる計算により、点Ｑの３次元座標が得られる。
【００８４】
ここで、点Ｑの３次元座標と、カメラの投影像の関係について、図９，１０を参照しながら述べる。
【００８５】
即ち、図９に示すように、１１０３のカメラから見た像は、カメラの虚像１１０６，１１０７から見た場合の像と等価である。よってカメラ１１０３に撮影される像を考える場合、仮想的に虚像１１０６，１１０７の位置にカメラをおいたものとして、考えても良い。ただし、鏡像の関係にあるため、撮影される像も、反転した虚像の関係にあることに注意する必要がある。
【００８６】
次に、図１０を参照しながら、点Ｑの３次元座標の算出方法について更に述べる。
【００８７】
図１０は、１２０１の下部ミラー面に関して１２０４の点Ｑと、１２０６の投影像ｑ２の関係を示したものである。今１２０２のカメラのレンズ中心を座標系の原点として、カメラの光軸方向をＺ軸、ミラー面に垂直な横方向をＹ軸、図面手前方向をＸ軸とする。１２０１のミラー面は、Ｚ軸と成す角がγで、Ｚ軸と交わる点が原点OからＲの距離にあるとする。
【００８８】
このＸＹＺ座標系で、１２０４の点Ｑの座標を（Ｘ_Q、Ｙ_Q、Ｚ_Q）とし、焦点距離ｆのカメラの投影面１２０５に写る像１２０６をｑ２（ｘ_q2、ｙ_q2、ｆ）として、その２つの関係を求める。
【００８９】
１２０２のカメラの虚像を１２０３とし、そのレンズ中心をＯ’とする。点Ｑの投影像ｑ２は、カメラの虚像の投影面１２０７に写る像ｑ２’（１２０８）と鏡像の関係にある。よって１２０７の投影面上の座標系での、ｑ２’の座標からｑ２の座標は求められる。
【００９０】
まず、原点が１２０３のカメラの虚像のレンズ中心Ｏ’にあり、カメラの虚像の光軸にＺ’軸が一致し、Ｘ’軸が前記ＸＹＺ座標系のＸ軸と平行な、Ｘ’Ｙ’Ｚ’座標系を考える。この座標系でのｑ２’の座標は、(ｘ_q2’ 、ｙ_q2’ 、ｆ)となる。ｑ２との鏡像の関係にあるので、
【００９１】
【数１８】
ｘ_q2 ＝ −ｘ_q2’
ｙ_q2 ＝ −ｙ_q2’
となる。
【００９２】
次に、点ＱのＸ’Ｙ’Ｚ’座標系での座標を求める。ＸＹＺ座標系とＸ’Ｙ’Ｚ’座標系の変換は、ＸＹＺ座標系の原点Ｏを、Ｏ’へ平行移動し、Ｚ軸がＺ’軸へ一致するよう、Ｘ軸周りに回転することにより、求められる。
【００９３】
今、Ｚ軸とＺ’軸の成す角をψとすれば、
【００９４】
【数１９】
ψ ＝ ２γ
の関係がある。一方、ＯからＯ’への平行移動により、Ｑ（Ｘ_Q、Ｙ_Q、Ｚ_Q）の座標値は、
【００９５】
【数２０】
（Ｘ_Q、Ｙ_Q＋Ｒsinψ、Ｚ_Q＋Ｒ＋Ｒcosψ）
となる。次に、Ｘ軸周りのψ回転により
【００９６】
【数２１】

【００９７】
となる。
【００９８】
Ｚ'でｆの距離にある投影面１２０７の投影により、点Ｑの投影像ｑ２’(ｘ_q2’、ｙ_q2’、ｆ)の座標が求まり、（式１８）より、カメラへの投影像ｑ２（ｘ_q2、ｙ_q2、ｆ）が求まる。
【００９９】
【数２２】
ｘ_q2 ＝ −ｘ_q2’ ＝ −ｆ・Ｘ_Q’／Ｚ_Q’
ｙ_q2 ＝ −ｙ_q2’ ＝ −ｆ・Ｙ_Q’／Ｚ_Q’
以上により、ミラーの位置R、傾きγ、カメラの焦点距離ｆを用いて、任意の点Ｑ（Ｘ_Q、Ｙ_Q、Ｚ_Q）と、そのカメラでの像ｑ２（ｘ_q2、ｙ_q2、ｆ）には、（式２１）、（式２２）の関係が成り立つ。
【０１００】
一方、図９におけるような、上部、下部の２つのミラー面では、その傾きγ、カメラからの距離Ｒが異なる。よって、一点Ｑのカメラでの投影像は、それぞれのγ、Ｒと（式２１）、（式２２）により、別の点ｑ１、ｑ２となる。
【０１０１】
従って、逆に、それぞれのγ、Ｒが判っていれば、点ｑ１、ｑ２の座標と、（式２１）、（式２２）により、点Ｑの３次元座標（Ｘ_Q、Ｙ_Q、Ｚ_Q）を計算できる。尚、上述した点Ｑの３次元座標の計算方法は、上記実施の形態２において点Ｐの３次元座標を求める場合にも同様に用いることが出来ることは言うまでもない。
【０１０２】
【発明の効果】
以上述べたところから明らかな様に本発明は、従来に比べてより完全な全方位のステレオ画像対を、リアルタイムに取得することが出来るという長所を有する。
【図面の簡単な説明】
【図１】本発明の一実施の形態の全方位ステレオ画像撮影装置の概略構成図
【図２】同実施の形態の全方位ステレオ画像撮影装置の鉛直断面図
【図３】同実施の形態のカメラにより得られる投影像を示す図
【図４】同実施の形態の３次元座標を計算する処理の流れを示す図
【図５】同実施の形態の光学系の関係を示す説明図
【図６】本発明の全方位ステレオ画像撮影装置の第２の実施の形態の概略構成図
【図７】本発明の第２の実施の形態の測定範囲を説明する図
【図８】本発明の全方位ステレオ画像撮影装置の他の実施の形態の角錐ミラーを用いた概略構成図
【図９】同実施の形態の鉛直断面図
【図１０】同実施の形態の光学系の関係を示す説明図
【図１１】従来の双曲面型全方位撮影装置の説明図
【図１２】従来の角錐型全方位撮影装置の説明図
【符号の説明】
１０１ 上部の双曲面ミラー
１０２ 下部の双曲面ミラー
１０４ カメラ
２０１、 ２０２ 双曲面ミラー
２０４ 投影面
２０５、 ２０６ 双曲面の焦点
２０７ 計測点
２０８、 ２０９ 投影面へ投影された点
３０１ 撮影像
３０２ 上部双曲面ミラーの投影範囲
３０３ 部双曲面ミラーの投影範囲
４０１ 投影像
４０２ 対応点計算部
４０３ 奥行き計算部
４０４ ３次元座標データ
６０１、６０２ 角錐ミラー
６０３、６０４ カメラ群
６０５ 角錐中心
７０１、７０２ 角錐ミラー
７０３、７０４ カメラ
７０６、７０７ カメラの虚像
７０８、７０９、７１０、７１１ カメラの撮影範囲
７１２ ステレオ計測可能な領域
７１３、７１４ カメラの光軸
７１５、７１６、７１７、７１８ ステレオ計測可能な領域の境界上の点
８０１ 双曲面ミラー
８０２ カメラ
８０３ レンズ中心（外焦点）
８０４ 内焦点
８０５ 観測点
８０６ 観測点からの光線
９０１ 角錐ミラー
９０２ カメラ群
１００１ 上部角錐ミラー
１００２ 下部角錐ミラー
１００３ カメラ群
１１０１ 上部ミラー面
１１０２ 下部ミラー面
１１０３ カメラ
１１０４ 計測点
１１０５ カメラの投影面
１１０６ 上部ミラー面に対応するカメラの虚像
１１０７ 下部ミラー面に対応するカメラの虚像
１２０１ ミラー面
１２０２ カメラ
１２０３ カメラの虚像
１２０４ 計測点
１２０５ カメラの投影面
１２０６ 計測点の投影像
１２０７ 虚像の投影面
１２０８ 虚像の投影面への投影像[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an omnidirectional stereo image photographing apparatus capable of obtaining an omnidirectional stereo image pair.
[0002]
[Prior art]
Conventionally, as an apparatus for capturing an omnidirectional image in real time, there is one disclosed in JP-A-6-295333.
FIG. 11 is a configuration diagram of this conventional apparatus, in which 801 is a rotating hyperboloidal mirror and 802 is a camera. The convex portion of the hyperboloid mirror 801 is a reflecting surface, and the convex portion is arranged vertically downward, and the camera 802 is arranged so that the lens center is at the outer focal point 803 of the hyperboloid.
[0003]
The light 806 from the point 805 toward the inner focal point 804 is reflected by the mirror 801 and passes through the outer focal point 803 which is the center of the camera lens. For this reason, the image obtained by the camera 802 is equivalent to the image of the central projection viewed from the inner focal point 804.
[0004]
In this way, an omnidirectional image can be acquired in real time.
[0005]
Japanese Patent Laid-Open No. 8-125835 discloses an omnidirectional image capturing apparatus using a pyramid-shaped mirror and a plurality of cameras in order to acquire a high-resolution omnidirectional image. is there.
[0006]
FIG. 12 is a block diagram of this conventional apparatus. Reference numeral 901 denotes a pyramid type mirror, and reference numeral 902 denotes a camera group arranged corresponding to each surface so as to capture an image reflected on each surface of the pyramid mirror 901. .
[0007]
By arranging the position of each camera so that the virtual image created by the mirror surface to which the lens center of each camera corresponds overlaps at one point of the center of the pyramid, it is equivalent to seeing the outer peripheral direction from the intersection of the virtual images A divided image is acquired by each camera.
[0008]
A panoramic image of the entire periphery is generated by joining the images obtained by dividing in this way by image processing.
[0009]
In this case, compared with the case where the entire surrounding image is acquired by one camera, when the resolution of the cameras is the same, a higher resolution image can be obtained by sharing with a plurality of cameras.
[0010]
[Problems to be solved by the invention]
However, when the above method is used for photographing a stereo image used for distance measurement or the like, the following problems occur.
[0011]
Stereo images are taken by taking two stereo image pairs taken from the same object from different camera positions. Even when the omnidirectional imaging apparatus is used, the two imaging apparatuses are arranged at a certain interval.
[0012]
In this case, due to the characteristics of the omnidirectional imaging apparatus, the other imaging apparatus is reflected in the image of one imaging apparatus to close the field of view. This impairs omnidirectional shooting.
[0013]
In order to solve the above-mentioned problem, it is also possible to use a moving stereo technique in which a single imaging device is moved and images at different positions are taken at a time. However, in this case, real-time video acquisition is impossible.
[0014]
In consideration of such a problem of the conventional omnidirectional stereo photographing apparatus, the present invention provides an omnidirectional stereo image photographing apparatus capable of acquiring in real time a more complete omnidirectional stereo image pair than the conventional one. The purpose is to provide.
[0015]
[Means for Solving the Problems]
A first hyperboloid mirror that reflects an omnidirectional image in a predetermined direction;
A second hyperboloid mirror that reflects an omnidirectional image in a predetermined direction with a curvature different from that of the first mirror;
A camera that images the same object reflected by the first hyperboloid mirror and the second hyperboloid mirror as a stereo image,
The first hyperboloidal mirror and the second hyperboloidal mirror are arranged such that the center axes of both hyperboloids coincide and the positions of the outer focal points coincide.
The camera is an omnidirectional stereo image photographing device arranged so that the position of the outer focal point coincides with the position of the lens center of the camera .
[0016]
According to a second aspect of the present invention, there is provided a first pyramid mirror that reflects an image of substantially all directions in a predetermined direction;
A second pyramid mirror that reflects a substantially omnidirectional image in a predetermined direction substantially coincident with the first pyramid mirror;
Imaging the same object reflected by one reflecting surface of the first pyramidal mirror and one reflecting surface of the second pyramidal mirror corresponding to the reflecting surface as a stereo image ; A camera provided for each pair of two reflecting surfaces corresponding to the second pyramidal mirror,
The one reflection surface of the first pyramid mirror, the one reflection surface of the second pyramid mirror, and the camera form images at different positions in the camera. And an omnidirectional stereo image photographing device in which the lens center of the virtual image of each camera with respect to the reflecting surface of each pyramidal mirror is located at one point on the central axis for each pyramid mirror. .
[0019]
The third aspect of the present invention uses the image data photographed by the omnidirectional stereo image photographing apparatus of the first or second aspect of the present invention , and is reflected by the two mirrors or the two reflecting surfaces corresponding to each other. A corresponding point calculation unit for obtaining a stereo corresponding point between the image regions of each image,
An omnidirectional stereo image photographing apparatus comprising: a depth calculation unit that calculates a three-dimensional coordinate of the corresponding point based on the stereo corresponding point obtained by the corresponding point calculation unit.
[0020]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to FIGS.
[0021]
(Embodiment 1)
FIG. 1 is a schematic configuration diagram of an omnidirectional stereo image photographing apparatus of the present invention.
[0022]
In FIG. 1, 101 and 102 are hyperboloid mirrors having hyperboloid cross sections with different curvatures, 103 indicates a vertical direction, and the axes of the hyperboloid mirrors 101 and 102 coincide with the vertical direction 103. To do. A camera 104 is arranged along the vertical direction 103 so that the center of the lens is located at the focal point outside the hyperboloid.
[0023]
This optical arrangement will be described with reference to FIG. FIG. 2 is a schematic view of a cross section along a plane including the vertical line 103 in FIG. 201 is a hyperboloidal mirror 1 positioned at the top, and 202 is a hyperboloidal mirror 2 positioned at the bottom. The hyperboloid mirror 1 (201) in FIG. 2 corresponds to the hyperboloid mirror 101 in FIG. 1, and the hyperboloid mirror 2 (202) corresponds to the hyperboloid mirror 102 in FIG. Here, the coordinate system is set such that the origin 0 is at the focal point 203 of the two hyperboloid mirrors, the upper direction is the Z axis, the horizontal direction is the Y axis, and the front side of the screen is the X axis. Assume that the focal length of the camera is f, and the projection plane of Z = f connecting the images is 204.
[0024]
The shape of the hyperboloid mirror 1 can be expressed by the following equation.
[0025]
[Expression 1]

[0026]
Here, a ₁ , b ₁ , and c ₁ are parameters that determine the shape of the hyperboloid, and have the following relationship.
[0027]
[Expression 2]

[0028]
As shown in FIG. 2, when the outer focus is fixed at the origin (0, 0, 0), the coordinates of the other inner focus 205 are (0, 0, 2C ₁ ).
[0029]
Similarly for the hyperboloid mirror 2 [0030]
[Equation 3]

[0031]
[Expression 4]

[0032]
The relationship holds. Here, the coordinates of the focal point 206 inside the hyperboloid mirror 2 are (0, 0, 2C ₂ ).
[0033]
Now, it is shown how the point P (Xp, Yp, Zp) indicated by 207 in FIG. The light traveling from the point P toward the focal point 205 is reflected by the hyperboloidal mirror 1 and enters the lens of the camera placed at the origin. At this time, the image shown on the projection plane 204 is 208 point P ₁ (x _p1 , y _p1 , f).
[0034]
Similarly, an image obtained by connecting light from the point P 207 toward the focal point 206 of the hyperboloid mirror 2 to the projection plane 204 is represented by a point P ₂ (x _p2 , y _p2 , f) 209.
[0035]
The state of the image on the projection surface is shown in FIG. Reference numeral 301 denotes a shooting range of the camera, 302 is a range where an image reflected by the hyperboloid mirror 1 is reflected, and 303 is a range where a reflection image of the hyperboloid mirror 2 is reflected. Yes.
[0036]
These 302 and 303 are a stereo image pair corresponding to all directions. As described above, the stereo image pair is projected on the projection plane of the same camera, so that if the camera is a video camera capable of recording moving images, a stereo image in all directions can be taken in real time.
[0037]
In the above description, the example in which the mirror for omnidirectional imaging is configured by a hyperboloidal mirror has been described. However, other pyramid mirrors can be similarly implemented. This will be further described later.
[0038]
Next, a method for measuring three-dimensional information of measurement points from the relationship between stereo image pairs obtained from a hyperboloid mirror will be described.
[0039]
The points P ₁ and P _{2 obtained} by projecting the point P indicated by 207 in FIG. 2 on the projection plane 204 are projected to the positions indicated by 304 and 305 in FIG. 3, respectively. The coordinates of P ₁ and P ₂ on the projection plane are (x _p1 , y _p1 ) and (y _p2 , y _p2 ), respectively.
[0040]
Since the hyperboloidal mirror is rotationally symmetric with respect to the Z axis between the points P1 and P2 projected by the two hyperboloidal mirrors 1 and 2, the same point in the three-dimensional space, the axis of the hyperboloidal mirror There is a restriction that they are arranged on a straight line passing through the center of the.
In other words, the search range can be limited from a two-dimensional plane to a one-dimensional straight line when searching for a stereo pair that searches for a corresponding point on the projection plane of the same point in space, and the processing involved in the search can be reduced. .
[0041]
The flow of processing will be described with reference to FIG. Reference numeral 401 denotes an image captured by the camera shown in FIG. Reference numeral 402 denotes a corresponding point calculation unit that searches for a stereo corresponding point from the photographed image, and 403 denotes a depth calculation unit that calculates a three-dimensional coordinate of the point from a set of stereo corresponding points from the corresponding point calculation unit. .
[0042]
The corresponding point calculation unit 402 performs first-order differentiation of the luminance value in the radial direction from the center toward the outside of the captured image, and extracts an edge. For this edge data, a stereo correspondence point is calculated and searched. The search range is limited to a straight line extending radially from the center. The coordinates P ₁ (x _p1 , y _p1 ) and P ₂ (x _p2 , y _p2 ) of a set of points whose correlation values are higher than a predetermined threshold value are output to the depth calculation unit 403.
[0043]
The depth calculation unit 403 calculates the three-dimensional coordinates (Xp, Yp, Zp) of the corresponding point P indicated by 207 in FIG. 2 from the set of two given stereo corresponding points.
[0044]
As shown in FIG. 5, the parameters a ₁ , b ₁ , c _{1 of} the hyperboloid, the focal length f of the camera, the angle formed by the straight line from the point P to the focal point inside the hyperboloid with the horizontal line are −α1, the hyperboloid If the angle between the reflected light from the mirror 1 to the origin and the horizontal line is β1,
[0045]
[Equation 5]

[0046]
[Formula 6]

[0047]
[Expression 7]

[0048]
The relationship is obtained. Here, FIG. 5 is an explanatory diagram depicting the relationship between the hyperboloid mirror 1 and the point Q, and the hyperboloid mirror 2 is omitted for convenience of explanation. Therefore, by regarding the hyperboloidal mirror 1 in the figure as the hyperboloidal mirror 2, the hyperboloidal mirror 2 is also similar to the above.
[0049]
[Equation 8]

[0050]
[Equation 9]

[0051]
[Expression 10]

[0052]
The relationship holds.
[0053]
Further, the constraint that the stereo correspondence point is restricted on a straight line passing through the center on the projection plane is that the angle between the straight line and the X axis of the projection plane is θ.
## EQU11 ##

[0055]
The relationship is obtained.
[0056]
Based on (Expression 5) to (Expression 11), Xp, Yp, and Zp are obtained.
[Expression 12]

[0058]
Therefore, since the parameters a ₁ , b ₁ , c ₁ , a ₂ , b ₂ , c ₂ and the focal length f of the camera are known, the stereo correspondence points (x _p1 , y _p1 ), ( From the set of x _p2 , y _p2 ), the three-dimensional coordinates of the corresponding point are calculated by (Equation 12). The calculated output is the coordinate output 404 in FIG.
[0059]
(Embodiment 2)
FIG. 6 shows a schematic configuration diagram of Embodiment 2 of the omnidirectional stereo image photographing apparatus of the present invention.
[0060]
In FIG. 6, reference numeral 601 denotes an upper pyramid mirror, 602 denotes a lower pyramid mirror, 603 that coincides with the center line 605 denotes an upper camera group that captures a reflection image of each surface of the upper pyramid mirror, and 604 denotes each lower pyramid mirror. It is a lower camera group which takes a reflected image.
[0061]
The optical arrangement will be described with reference to FIG. FIG. 7 is a schematic diagram of a cross section along a plane including the center line 605 of FIG. 6 and perpendicular to one surface of the pyramid. Reference numerals 701 and 702 denote an upper pyramid mirror and a lower pyramid mirror, respectively. Reference numerals 703 and 704 denote an upper camera and a lower camera, respectively. 705 is an intermediate point O on the center line of the two pyramid mirrors. An image captured by the upper camera 703 is reflected by the mirror 701 and is equivalent to an image viewed from the position of the virtual image 706 of the upper camera 703. Similarly, a virtual image of the lower camera 704 is assumed to be 707.
[0062]
In order to join horizontally divided images to stereo images in all directions without gaps, it is necessary to match the camera lens centers in the respective directions. When shooting with a single camera at different times, the center of the lens can be rotated in the horizontal direction, but when shooting with multiple cameras at the same time, place the two cameras at the same place at the same time. It is impossible because it cannot be placed. However, if the mirrors are used to collect the lens centers of all the camera's virtual images at one point, the camera entities can be located at different locations and only the equivalent lens centers can be collected at one point. In the case of a pyramid mirror, this lens center gathers on the center line of the pyramid mirror, and each camera is arranged so that the lens center of the virtual image is located at one point on the center line.
[0063]
When the shooting range of the camera 703 is an area surrounded by line segments 708 and 709, and the shooting range of the camera 704 is an area surrounded by line segments 710 and 711, a stereo-measurable area in which projected images are captured on both cameras is , 712, a hatched area.
[0064]
Next, the relationship between this area and the mirror, the camera angle of view, and the position will be described. As shown in FIG. 7, the angle of the pyramid mirror is γ. The upper and lower mirrors have the same shape. Further, it is assumed that the optical axes of the cameras 703 and 704 are directed on the vertical line. At this time, if the elevation angle between the optical axes 713 and 714 of the camera reflected by the mirror and the horizontal direction is ψ, the relationship between γ and ψ is expressed by the following equation.
[0065]
[Formula 13]

[0066]
The distance from the origin 0 of 705 to the lens center of the virtual image of the camera is L, and the vertical field angle of the camera is 2ρ. If the horizontal direction from the origin 0 of 705 to the outside is the Y axis, the vertical direction is the Z axis, and the coordinates of the point 715 where the measurement region 712 starts are (0, y ₀ , 0), y ₀ is
[0067]
[Expression 14]
y ₀ = L / tan (ψ + ρ)
It becomes.
[0068]
If the y coordinate of the point 716 where the shape of the measurement region 712 changes is y _c , the z coordinate of the point 716 on the boundary of the measurement region 712 at y ₀ <y <y _c is
[0069]
[Expression 15]
z = y tan (ψ + ρ)-L
It becomes.
[0070]
The z coordinate of the point 717 on the boundary of the measurement region 712 when y _c <y is
[0071]
[Expression 16]
z = L-y tan (ψ-ρ)
It becomes.
From (Expression 14) and (Expression 15), the coordinate y _{c of the} point 717 is
[0072]
[Expression 17]
y _c = 2L / (tan (ψ + ρ) + tan (ψ−ρ))
It becomes.
[0073]
By adjusting the distance L of the pyramid mirror and the elevation angle ψ so that the measurement region 712 covers the region where the target to be measured exists, a stereo image pair of the desired measurement region can be acquired.
[0074]
In the above description, the example in which the mirror for capturing all directions is a pyramid mirror has been described. However, other spherical mirrors, conical mirrors, and hyperboloid mirrors can be similarly applied.
[0075]
As is clear from the above description, according to the present embodiment, for example, in the acquisition of a real environment in the field of the visual system of a mobile robot, artificial reality, etc., not only real-time images in all directions, It is possible to provide an omnidirectional stereo image photographing apparatus capable of obtaining an omnidirectional stereo image pair that gives information such as a distance to an object in the image.
[0076]
In other words, two sets of mirrors that capture all azimuths in the horizontal direction are combined symmetrically so that they do not appear on the other mirror, or the two reflected images are connected to different positions on the projection plane of the same camera. By arranging mirrors with different shapes in the vertical direction, stereo image pairs in all directions can be acquired in real time.
[0077]
In particular, by combining two hyperboloid mirrors having different curvatures, it is possible to shoot an omnidirectional stereo image pair in one video, and omnidirectional stereo image shooting in real time. Further, when searching for a stereo corresponding point from the obtained image, it is only necessary to search on a radial straight line, and the three-dimensional coordinates of the corresponding point can be obtained at high speed.
[0078]
In the first embodiment, the mirror for omnidirectional imaging is described as an example of a hyperboloidal mirror. However, the present invention is not limited to this. For example, the mirror may be formed of a pyramid mirror as shown below.
[0079]
That is, here, an embodiment in which a mirror for capturing all directions is formed of a pyramid mirror will be described.
[0080]
FIG. 8 is a schematic configuration diagram of an omnidirectional stereo image photographing apparatus of the present invention constituted by a pyramid mirror. In FIG. 8, 1001 is an upper pyramid mirror, 1002 is a lower pyramid mirror, and 1003 is a camera group that simultaneously captures reflection images of corresponding upper and lower pyramid mirror planes. In this case, a stereo image of the surfaces of the corresponding upper and lower pyramid mirrors can be obtained with one camera. This image is obtained by dividing all directions in the horizontal direction, and an image of 360 degrees in the horizontal direction can be obtained by combining the images of all cameras.
[0081]
Next, the optical arrangement will be described with reference to FIG. FIG. 9 shows a cross-sectional view taken along a plane perpendicular to one of the plane mirrors including the central axis of the pyramid mirror of FIG. 8 and constituting the pyramid mirror. Reference numeral 1101 denotes an upper pyramid mirror surface, and reference numeral 1102 denotes a corresponding lower pyramid mirror surface. Reference numeral 1103 denotes a camera that captures a reflected image reflected on the two mirror surfaces.
[0082]
Points Q shown in 1104 are reflected by the upper and lower mirror surfaces, and the images that appear on the projection surface 1105 of the camera 1103 are q1 and q2, respectively. Therefore, two stereo pairs from the same point are obtained on the projection plane of one camera.
[0083]
Using the coordinates of the points q1 and q2 obtained in this way, the three-dimensional coordinates of the point Q are obtained by the calculation described below.
[0084]
Here, the three-dimensional coordinates of the point Q, the relationship of the projected image of the camera is described with reference to FIGS. 9 and 10.
[0085]
That is, as shown in FIG. 9 , the image viewed from the camera 1103 is equivalent to the image viewed from the virtual images 1106 and 1107 of the camera. Therefore, when an image photographed by the camera 1103 is considered, it may be considered that the camera is virtually placed at the positions of the virtual images 1106 and 1107. However, it should be noted that because the images are mirror images, the captured images are also inverted virtual images.
[0086]
Next, referring to FIG. 10, further describes a method of calculating the three-dimensional coordinates of the point Q.
[0087]
FIG. 10 shows the relationship between the point Q at 1204 and the projected image q2 at 1206 with respect to the lower mirror surface at 1201. FIG. Assume that the lens center of the camera 1202 is the origin of the coordinate system, the optical axis direction of the camera is the Z axis, the horizontal direction perpendicular to the mirror surface is the Y axis, and the front direction of the drawing is the X axis. Suppose that the mirror surface 1201 has an angle of γ with the Z axis and a point intersecting the Z axis at a distance of R from the origin O.
[0088]
In this XYZ coordinate system, the coordinates of the point Q of 1204 are (X _Q , Y _Q , Z _Q ), and the image 1206 on the projection plane 1205 of the camera at the focal length f is _{q 2} (x _q2 , y _q2 , f). Find the relationship between the two.
[0089]
The virtual image of the camera 1202 is 1203, and the lens center is O ′. The projected image q2 at the point Q is in a mirror image relationship with an image q2 ′ (1208) that appears on the projection surface 1207 of the virtual image of the camera. Therefore, the coordinate of q2 is obtained from the coordinate of q2 ′ in the coordinate system on the projection surface 1207.
[0090]
First, X′Y ′, where the origin is at the lens center O ′ of the virtual image of the camera 1203, the Z ′ axis coincides with the optical axis of the virtual image of the camera, and the X ′ axis is parallel to the X axis of the XYZ coordinate system. Consider the Z ′ coordinate system. The coordinates of q2 'in this coordinate system are ( _xq2 ', _yq2 ', f). Since it has a mirror image relationship with q2,
[0091]
[Formula 18]
x _q2 = -x _q2 '
y _q2 = -y _q2 '
It becomes.
[0092]
Next, the coordinates of the point Q in the X′Y′Z ′ coordinate system are obtained. The transformation between the XYZ coordinate system and the X′Y′Z ′ coordinate system is performed by translating the origin O of the XYZ coordinate system to O ′ and rotating around the X axis so that the Z axis coincides with the Z ′ axis. ,Desired.
[0093]
Now, if the angle between the Z axis and the Z ′ axis is ψ,
[0094]
[Equation 19]
ψ = 2γ
There is a relationship. On the other hand, due to the parallel movement from O to O ′, the coordinate value of Q (X _Q , Y _Q , Z _Q ) becomes
[0095]
[Expression 20]
_{(X Q, Y Q + Rsinψ} , Z Q + R + Rcosψ)
It becomes. Next, by ψ rotation around the X axis,
[Expression 21]

[0097]
It becomes.
[0098]
The coordinates of the projection image q2 ′ (x _q2 ′, y _q2 ′, f) of the point Q are obtained by projection of the projection plane 1207 at a distance of f at Z ′, and the projection image q2 to the camera is obtained from (Equation 18). (X _q2 , y _q2 , f) is obtained.
[0099]
[Expression 22]
x _q2 = -x _q2 '= -f · X _Q ' / Z _Q '
y _q2 = -y _q2 '= -f · Y _Q ' / Z _Q '
Thus, using the mirror position R, tilt γ, and camera focal length f, an arbitrary point Q (X _Q , Y _Q , Z _Q ) and an image q2 (x _q2 , y _q2 , f on the camera) ) Holds the relationship of (Expression 21) and (Expression 22).
[0100]
On the other hand, the two upper and lower mirror surfaces in FIG. 9 have different inclinations γ and distances R from the camera. Therefore, the projected image of the camera at one point Q becomes different points q1 and q2 by the respective γ and R and (Expression 21) and (Expression 22).
[0101]
Therefore, conversely, if the respective γ and R are known, the three-dimensional coordinates (X _Q , Y _Q , Z _Q ) of the point Q by the coordinates of the points q 1 and q 2 and (Equation 21) and (Equation 22). ) Can be calculated. Needless to say, the above-described method for calculating the three-dimensional coordinates of the point Q can also be used in the same manner when obtaining the three-dimensional coordinates of the point P in the second embodiment.
[0102]
【The invention's effect】
As is apparent from the above description, the present invention has the advantage that a more complete omnidirectional stereo image pair can be acquired in real time as compared with the prior art.
[Brief description of the drawings]
FIG. 1 is a schematic configuration diagram of an omnidirectional stereo image photographing apparatus according to an embodiment of the present invention. FIG. 2 is a vertical sectional view of the omnidirectional stereo image photographing apparatus according to the embodiment. FIG. 4 is a diagram showing a projection image obtained by a camera. FIG. 4 is a diagram showing a flow of processing for calculating three-dimensional coordinates according to the embodiment. FIG. 5 is an explanatory diagram showing a relationship between optical systems according to the embodiment. FIG. 7 is a schematic configuration diagram of the second embodiment of the omnidirectional stereo image photographing apparatus of the present invention. FIG. 7 is a diagram for explaining the measurement range of the second embodiment of the present invention. FIG. 9 is a schematic configuration diagram using a pyramid mirror of another embodiment of a stereo image photographing apparatus. FIG. 9 is a vertical sectional view of the embodiment. FIG. 10 is an explanatory diagram showing a relationship of the optical system of the embodiment. 11 is an explanatory diagram of a conventional hyperboloid omnidirectional imaging apparatus. FIG. 12 is a conventional pyramid omnidirectional imaging apparatus. Illustration DESCRIPTION OF SYMBOLS
101 Upper hyperboloid mirror 102 Lower hyperboloid mirror 104 Camera 201, 202 Hyperboloid mirror 204 Projection surface 205, 206 Focal point of hyperboloid 207 Measurement point 208, 209 Point projected onto projection surface 301 Photographed image 302 Upper hyperboloid Projection range of mirror 303 Projection range of hyperboloid mirror 401 Projected image 402 Corresponding point calculation unit 403 Depth calculation unit 404 Three-dimensional coordinate data 601 and 602 Pyramid mirrors 603 and 604 Camera group 605 Pyramid centers 701 and 702 Pyramidal mirrors 703 and 704 Camera 706, 707 Camera virtual image 708, 709, 710, 711 Camera imaging range 712 Stereo measurable area 713, 714 Camera optical axis 715, 716, 717, 718 Point 801 on the boundary of stereo measurable area Curved mirror 802 Camera 803 Center (outside focus)
804 In-focus 805 Observation point 806 Ray 901 from observation point Pyramid mirror 902 Camera group 1001 Upper pyramid mirror 1002 Lower pyramid mirror 1003 Camera group 1101 Upper mirror surface 1102 Lower mirror surface 1103 Camera 1104 Measurement point 1105 Camera projection surface 1106 Upper mirror Virtual image 1107 of the camera corresponding to the surface Virtual image 1201 of the camera corresponding to the lower mirror surface Mirror surface 1202 Camera 1203 Virtual image 1204 of the camera 1204 Measurement point 1205 Projection surface 1206 Measurement point projection image 1207 Virtual image projection surface 1208 Virtual image projection surface Projection image

Claims (3)

A first hyperboloid mirror that reflects an omnidirectional image in a predetermined direction;
A second hyperboloid mirror that reflects an omnidirectional image in a predetermined direction with a curvature different from that of the first mirror;
A camera that images the same object reflected by the first hyperboloid mirror and the second hyperboloid mirror as a stereo image,
The first hyperboloidal mirror and the second hyperboloidal mirror are arranged such that the center axes of both hyperboloids coincide and the positions of the outer focal points coincide.
The omnidirectional stereo image photographing apparatus , wherein the camera is arranged such that a position of the outer focal point coincides with a position of a lens center of the camera . A first pyramidal mirror that reflects a substantially omnidirectional image in a predetermined direction;
A second pyramid mirror that reflects a substantially omnidirectional image in a predetermined direction substantially coincident with the first pyramid mirror;
Imaging the same object reflected by one reflecting surface of the first pyramidal mirror and one reflecting surface of the second pyramidal mirror corresponding to the reflecting surface as a stereo image ; A camera provided for each pair of two reflecting surfaces corresponding to the second pyramidal mirror,
The one reflection surface of the first pyramid mirror, the one reflection surface of the second pyramid mirror, and the camera form images at different positions in the camera. And an omnidirectional stereo system in which the center of the virtual image of each camera with respect to the reflecting surface of each pyramidal mirror is located at one point on the central axis for each pyramid mirror. Image shooting device. A stereo between image regions of each image reflected by the two mirrors or the two reflecting surfaces corresponding to each other using image data photographed by the omnidirectional stereo image photographing device according to claim 1 or 2. A corresponding point calculation unit for calculating corresponding points;
A depth calculation unit that calculates the three-dimensional coordinates of the corresponding points based on the stereo corresponding points obtained by the corresponding point calculation unit;
An omnidirectional stereo image photographing device characterized by comprising:

Images