JP4110501B2 - Random pattern generation apparatus and method, distance image generation apparatus and method, and program providing medium - Google Patents

Description

【０１０８】
次に、ステップ６０３において、正規乱数に基づくランダムテクスチャ画像ｒｄ＿ｉｍ［ｋ］が生成される。ここで、ｋ＝０〜Ｋ−１，ｒｄ＿ｉｍ［ｋ］＝Ｒｋ，Ｋ＝ＸＳＩＺＥ×ＹＳＩＺＥ（画像サイズ）である。
【０１０９】
さらに、ステップ６０４で、ステップ６０３で生成された正規乱数に基づくランダムテクスチャ画像ｒｄ＿ｉｍ［ｋ］の二値化処理を行なう、これは予め定められた閾値Ｔに基づいて実行され。この結果得られる二値画像がｆ（ｉ，ｊ）とされる。
【０１１０】
次に、ステップ６０５において、二値画像ｆ（ｉ，ｊ）のパターン適性度指標値としての変化率ｒが算出される。変化率ｒの算出は、以下の式に基づいて実行される。
【０１１１】
【数２】

【０１１２】
なお、上記式においてＡは、正規化係数である。
【０１１３】
次に、ステップ６０６において、変化率ｒと、予め定めた閾値Ｔ１との比較処理が行なわれ、変化率ｒが閾値Ｔ１より大であれば、十分なランダム性を有する二値画像であると判定され、ステップ６０７において投光パターン画像生成処理を終了する。
【０１１４】
ステップ６０６における判定がＮｏ、すなわち変化率ｒが予め定めた閾値Ｔ１より大でないと判定された場合には、生成された二値画像が投光に不適切な周期成分を含むと判定され、ステップ６０８に進み、標準偏差ｓまたは、閾値Ｔの少なくともいずけか一方の値を変更して、ステップ６０２以下、少なくともステップ６０４以下の処理を繰り返して、新たな二値画像ｆ（ｉ，ｊ）を生成する。
【０１１５】
最終的にステップ６０６において、変化率ｒ＞閾値Ｔ１がＹｅｓとなった時点で、その二値画像が十分なランダム性を有する二値画像であると判定され、ステップ６０７に進み、投光パターン画像生成処理を終了する。
【０１１６】
以上、図５、図６を用いて説明した本発明のランダムパターン生成装置、およびランダムパターン生成方法によれば、一様乱数に基づいて生成された二値画像の変化率ｒを予め定めた閾値Ｔより大となること保証して投光用ランダム画像パターンを生成するように構成したので、ステレオ画像処理における対応付け処理において問題となる、すなわちミスマッチングの原因となる周期成分が十分取り除かれた良好なランダムパターンの生成が可能となる。
【０１１７】
［ランダムパターン生成装置−その他の実施例］
以下、本発明のランダムパターン生成装置、およびランダムパターン生成方法における二値画像パターンの評価方法の異なる態様について、図７、図８、図９を用いて説明する。図５、図６における生成された二値画像パターンの評価は、図６におけるステップ６０５，６０６において示されるように、生成した二値画像全体からパターン適性度指標値としての変化率ｒを算出して、この変化率ｒと閾値Ｔ１を比較することによって行なっていた。図７〜９では、二値画像ｆ（ｘ，ｙ）から一部領域を抽出して、その一部領域についてのパターン適性度指標値として変化率、特徴量による判定、さらにパターンマッチングによる判定処理を行なう態様である。
【０１１８】
まず、図７を用いて、二値画像ｆ（ｘ，ｙ）から一部領域（ウィンドー）のパターンを抽出して、パターン変化率パラメータを用いるランダム画像パターンの評価方法を説明する。
【０１１９】
図７のステップ７０１において、サイズがＸＳＩＺＥ×ＹＳＩＺＥの二値画像ｆ（ｘ，ｙ）を取得する。例えば図７右上に示すランダムパターンである。
【０１２０】
ステップ７０２では、評価基準サイズとして適用されるウィンドーのサイズ：Ｎｘ×Ｎｙを設定する。例えば図７右下に示すＮｘ×Ｎｙのウィンドーである。ステップ７０３では、ウィンドーの位置を設定する座標ｘｍ，ｙｍを設定する。これらの座標は、ｘ座標値：ｘｍ＝Ｎｘ／２、ｙ座標値：ｙｍ＝Ｎｙ／２として設定される。
【０１２１】
次に、ステップ７０４において、（ｘｍ，ｙｍ）によって定義されるウィンドーの変化率γｍを以下の式に従って求める。
【０１２２】
【数３】

【０１２３】
次に、ステップ７０５において、（ｘｍ，ｙｍ）によって定義されるウィンドーのパターン適性度指標値としての変化率γｍと予め定めた閾値Ｔ２との比較を実行して、変化率γｍが閾値Ｔ２より大であれば、そのウィンドーは十分なランダム性を有する二値画像であると判定され、ステップ７０６において、さらに比較すべきウインドゥ領域があるか否かを判定して、ある場合は、ステップ７０８に進み、ｘｍ，ｙｍを更新して、新たなウィンドー領域の変化率の算出および閾値との比較を実行する。
【０１２４】
ステップ７０５における判定がＮｏ、すなわち変化率γｍが予め定めた閾値Ｔ２より大でないと判定された場合には、生成された二値画像における当該ウィンドーが投光に不適切な周期成分を含むと判定され、ステップ７０７に進み、投光パターンとして不適切であると判定し、ステップ７０９において評価処理を終了する。
【０１２５】
このように投光パターンとして不適切であるとの判定がなされた場合において、新たなランダムパターンが必要である場合は、先に説明した図６のステップ６０８において標準偏差ｓ、または閾値Ｔを変更して新たな二値画像ｆ’（ｘ，ｙ）の生成処理を行なうことになる。
【０１２６】
図７のフローにおいて、ステップ７０５の評価判定がすべてＹｅｓとなり、評価ウィンドーが終了した場合は、ステップ７０９に進み、評価が終了し、生成された二値画像ｆ（ｘ，ｙ）が投光パターンとして適切であると判定する。
【０１２７】
次に、図８を用いて、二値画像ｆ（ｘ，ｙ）から一部領域（ウィンドー）のパターンを抽出して、パターンの特徴量パラメータを用いるランダム画像パターンの評価方法を説明する。
【０１２８】
図８のステップ８０１において、サイズがＸＳＩＺＥ×ＹＳＩＺＥの二値画像ｆ（ｘ，ｙ）を取得する。例えば図８右上に示すランダムパターンである。
【０１２９】
ステップ８０２では、評価基準サイズとして適用されるウィンドーのサイズ：Ｎｘ×Ｎｙを設定する。例えば図８右下に示すＮｘ×Ｎｙのウィンドーである。ステップ８０３では、ウィンドーの位置を設定する座標ｘｍ，ｙｍを設定する。これらの座標は、ｘ座標値：ｘｍ＝Ｎｘ／２、ｙ座標値：ｙｍ＝Ｎｎ／２として設定される。
【０１３０】
次に、ステップ８０４において、（ｘｍ，ｙｍ）によって定義されるウィンドーの特徴行列Ｒを以下の式に従って求める。
【０１３１】
【数４】

【０１３２】
次に、ステップ８０５において、特徴行列Ｒの固有値：λｍ１、λｍ２を求め、ステップ８０６において、（ｘｍ，ｙｍ）によって定義されるウィンドーの特徴行列Ｒの固有値：λｍ１、λｍ２から求められるパターン適性度指標値としての特徴量：λ＝ｍｉｎ（λｍ１，λｍ２）と予め定めた閾値λＴとの比較を実行して、特徴量：λが閾値λＴより大であれば、そのウィンドーは十分なランダム性を有する二値画像であると判定され、ステップ８０７において、さらに比較すべきウインドー領域があるか否かを判定して、ある場合は、ステップ８０９に進み、ｘｍ，ｙｍを更新して、新たなウィンドー領域の特徴量の算出および閾値との比較を実行する。
【０１３３】
ステップ８０６における判定がＮｏ、すなわち特徴量：λが予め定めた閾値λＴより大でないと判定された場合には、生成された二値画像における当該ウィンドーが投光に不適切な周期成分を含むと判定され、ステップ８０８に進み、投光パターンとして不適切であると判定し、ステップ８１０において評価処理を終了する。
【０１３４】
このように投光パターンとして不適切であるとの判定がなされた場合において、新たなランダムパターンが必要である場合は、先に説明した図６のステップ６０８において標準偏差ｓ、または閾値Ｔを変更して新たな二値画像ｆ’（ｘ，ｙ）の生成処理を行なうことになる。
【０１３５】
図８のフローにおいて、ステップ８０６の評価判定がすべてＹｅｓとなり、評価ウィンドーが終了した場合は、ステップ８１０に進み、評価が終了し、生成された二値画像ｆ（ｘ，ｙ）が投光パターンとして適切であると判定する。
【０１３６】
次に、図９を用いて、二値画像ｆ（ｘ，ｙ）から一部領域（ウィンドー）のパターンを複数抽出して、ウィンドー間のマッチングスコアを用いてランダム画像パターンの評価を行なう方法を説明する。前述の図７、図８の処理フローでは１つの部分領域における画像の特徴について評価していたが、ここでは、異なる部分領域間において、類似するパターンが存在するか否かを判定して、類似パターンがあると判定された場合は、投光パターンとしては不適切であるとの判定を行なうものである。
【０１３７】
図９のステップ９０１において、サイズがＸＳＩＺＥ×ＹＳＩＺＥの二値画像ｆ（ｘ，ｙ）を取得する。
【０１３８】
ステップ９０２では、評価基準サイズとして適用されるウィンドーのサイズ：Ｎｘ×Ｎｙを設定する。例えば図９右下に示すＮｘ×Ｎｙのウィンドーである。この場合比較対象となるウィンドーも同一サイズのウィンドーとして設定される。ステップ９０３では、ウィンドーの位置を設定する座標ｘｍ，ｙｍを設定する。これらの座標は、ｘ座標値：ｘｍ＝Ｎｘ／２、ｙ座標値：ｙｍ＝Ｎｎ／２として設定される。
【０１３９】
次に、ステップ９０４において、画像内の２つのウィンドーのパターンＰ１（ｉ，ｊ）とＰ２（ｉ，ｊ）との領域間相関値がパターン適性度指標値として以下の式に従って求められる。
【０１４０】
【数５】

【０１４１】
次に、ステップ９０５において、上記式に従って求められるパターン適性度指標値としての相関値ｒｍと予め定めた閾値Ｔ３との比較を実行して、相関値ｒｍが閾値Ｔ３より小であれば、ステップ９０４で比較した２つのウィンドーは不適切な類似パターンを有さない画像であると判定され、ステップ９０６において、さらに比較すべきウインドゥ領域があるか否かを判定して、ある場合は、ステップ９０８に進み、ｘｍ，ｙｍを更新して、新たなウィンドー領域を設定してステップ９０４以下のパターンマッチング処理を行なう。
【０１４２】
ステップ９０５における判定がＮｏ、すなわち相関値ｒｍが予め定めた閾値Ｔ３より小でないと判定された場合には、ステップ９０４で比較した２つのウィンドーは不適切な類似パターンを有すると判定され、ステップ９０７に進み、投光パターンとして不適切であると判定し、ステップ９０９において評価処理を終了する。
【０１４３】
このように投光パターンとして不適切であるとの判定がなされた場合において、新たなランダムパターンが必要である場合は、先に説明した図６のステップ６０８において標準偏差ｓ、または閾値Ｔを変更して新たな二値画像ｆ’（ｘ，ｙ）の生成処理を行なうことになる。
【０１４４】
図９のフローにおいて、ステップ９０５の評価判定がすべてＹｅｓとなり、評価ウィンドーが終了した場合は、ステップ９０９に進み、評価が終了し、生成された二値画像ｆ（ｘ，ｙ）が投光パターンとして適切であると判定する。
【０１４５】
以上では、主として二値画像によるパターンを投光する例について説明したが、二値画像ではなく濃淡パターンを投光する場合におけるパターン生成処理とそのパターン評価方法の例について、以下説明する。
【０１４６】
まず、濃淡画像によるランダムパターンの生成処理フローを図１０に示す。図１０のフローの各ステップについて説明する。まず、ステップ１００１、ステップ１００２において濃淡パターンを生成する画像領域の座標値の初期値をｉ＝０，ｊ＝０として設定する。
【０１４７】
次に、ステップ１００３において、０−１間の一様乱数を発生し、これを配列ａ［ｎ］として格納する。さらに、ステップ１００４において、ステップ１００３で発生した一様乱数の配列ａ［ｎ］に基づいて標準偏差ｓ、平均値ａｍとなる正規乱数Ｒを発生し、設定した座標（ｉ，ｊ）に対応する濃淡データをＲ＝ｆ（ｉ，ｊ）として設定する。これが座標（ｉ，ｊ）の濃淡値となる。
【０１４８】
ステップ１００５、ステップ１００６では、濃淡値を設定した座標と、投光パターン画像の設定サイズとの比較を実行して濃淡値の未設定領域の有無を判定する。未設定領域がある場合は、それぞれステップ１００７、ステップ１００８で座標を更新し、さらに、ステップ１００３以降の処理を繰り返し実行して、各座標の濃淡値を乱数に基づいて設定する。ＸＳＩＺＥ×ＹＳＩＺＥの領域に含まれる座標すべての濃淡値が決定されると、ステップ１００９に進み、濃淡データからなる投光パターンの生成処理が終了する。
【０１４９】
なお、乱数の発生方法としては様々な手法が適用可能である。図１０のフロー中に示す方法は一例を示しているにすぎず、本発明に適用される乱数発生方法は図１０に示した方法に限定されるものではない。
【０１５０】
図１０のフローにしたがって生成されたランダムパターンは、二値画像パターンではなく、濃淡値をもっているので、濃淡パターン画像としてのパターンとしての適性度を評価することが必要となる。
【０１５１】
濃淡値を有する投光パターンの評価処理フローを図１１に示す。図１１は、図１１の右上に示す濃淡画像ｆ（ｘ，ｙ）から、例えば図１１の右下に示す一部領域（ウィンドー）のパターンを抽出して分散値を求めて濃淡画像パターンを評価する方法を示したものである。
【０１５２】
図１１のステップ１１０１において、サイズがＸＳＩＺＥ×ＹＳＩＺＥの濃淡画像ｆ（ｘ，ｙ）を取得する。例えば図１１右上に示す濃淡パターンである。
【０１５３】
ステップ１１０２では、評価基準サイズとして適用されるウィンドーのサイズ：Ｎｘ×Ｎｙを設定する。例えば図１１右下に示すＮｘ×Ｎｙのウィンドーである。ステップ１１０３では、ウィンドーの位置を設定する座標ｘｍ，ｙｍを設定する。これらの座標は、ｘ座標値：ｘｍ＝Ｎｘ／２、ｙ座標値：ｙｍ＝Ｎｙ／２として設定される。
【０１５４】
次に、ステップ１１０４において、（ｘｍ，ｙｍ）によって定義されるウィンドー内の分散値σｍを以下の式に従って求める。
【０１５５】
【数６】

【０１５６】
次に、ステップ１１０５において、（ｘｍ，ｙｍ）によって定義されるウィンドーのパターンとしての適性度を示すパターン適性度指標値としての分散値σｍと予め定めた閾値σＴとの比較を実行して、分散値σｍが閾値σＴより大であれば、そのウィンドーは十分なランダム性を有する濃淡画像であると判定され、ステップ１１０６において、さらに比較すべきウインドゥ領域があるか否かを判定して、ある場合は、ステップ１１０８に進み、ｘｍ，ｙｍを更新して、新たなウィンドー領域の分散値の算出および閾値との比較を実行する。
【０１５７】
ステップ１１０５における判定がＮｏ、すなわち分散値σｍが予め定めた閾値σＴより大でないと判定された場合には、生成された濃淡画像における当該ウィンドーが投光に不適切であると判定され、ステップ１１０７に進み、投光パターンとして不適切であると判定し、ステップ１１０９において評価処理を終了する。
【０１５８】
このように投光パターンとして不適切であるとの判定がなされた場合において、新たなランダムパターンが必要である場合は、先に説明した図１０のステップ１００４において標準偏差ｓを変更して新たな濃淡画像の生成処理を行なうことが可能である。
【０１５９】
図１１のフローにおいて、ステップ１１０５の評価判定がすべてＹｅｓとなり、評価ウィンドーが終了した場合は、ステップ１１０９に進み、評価が終了し、生成された濃淡画像ｆ（ｘ，ｙ）が投光パターンとして適切であると判定する。
【０１６０】
図１２は、上述した方法によって生成された投光用画像パターンを対象（平面）に投光し、観測された２枚の画像から距離画像を求めた例を示した図である。図１２（ａ）は、上述した本発明のランダムパターン生成方法に従って生成したランダム点群からなる投光用画像パターンを示す。図１２（ｂ）（ｃ）は、そのパターンを測定対象である無地の壁に投光し、それぞれカメラ１とカメラ２（例えば、基準カメラと参照カメラ）により観測された画像である。
【０１６１】
図１２（ｄ）は、結果として得られる測定対象である無地の壁の距離画像である。この距離画像は、基準カメラ画像における任意のＮｘ×Ｎｙウィンドー内のパターンを用いて、参照カメラ画像におけるエピポーラライン上の対応付け、すなわち基準カメラ画像における任意のＮｘ×Ｎｙウィンドー内のパターンと同様のパターンが参照カメラ画像のエピポーラライン上のどの部分に表れるかを求めることによって視差データを取得し、その視差データに基づいて得られる距離画像である。図１２（ｄ）の距離画像は、測定対象である無地の壁に一律の距離データが反映されており、この結果は測定対象が無地の壁でも、本発明のランダムパターン生成方法に従って生成したランダム点群からなる投光用画像パターンを投光することによって、カメラとの距離を精度良く計測することを示している。
【０１６２】
図１３は、上述のランダム点群からなる画像パターンを顔に投光し、基準カメラと参照カメラにより観測されたステレオ画像から求められた距離画像の計測結果の例を模式的に示した概念図である。
【０１６３】
図１３（ａ）は、上述した本発明のランダムパターン生成方法に従って生成したランダム点群からなる投光用画像パターンを示す。図１３（ｂ）は、そのパターンを測定対象である顔に投光した模式図である。
【０１６４】
図１３（ｃ）（ｄ）は、測定対象である顔に投光用画像パターン投光し、それぞれ基準カメラと参照カメラにより観測された画像である。
【０１６５】
図１３（ｅ）は、結果として得られる測定対象である顔の距離画像である。この距離画像は、説明のために抽象的に示す概念図であり、実際のものとは異なるが本発明のランダムパターン生成方法に従って生成したランダム点群からなる投光用画像パターンを投光した場合の対応点付け処理においてはミスマッチングの発生可能性が低減され、測定対象のほぼ全ての面で正確な距離が求められる。すなわち基準カメラから測定対象である顔の各部位までの距離に応じた濃淡が示される精度の高い距離画像が得られる。本発明のランダムパターン生成方法を用いることにより、測定対象が人の顔のような特徴がはっきりしない対象であっても、基準カメラ画像と参照カメラ画像との対応点付け処理においてミスマッチングが発生する可能性が低減され、距離画像（または３次元形状）を高精度に生成することができる。
【０１６６】
なお、上述の実施例では、画像撮り込みを行なうカメラを２台として、２つの投光パターン画像から距離画像を求める構成について説明したが、さらに３台以上のカメラを用いたマルチベースラインステレオシステム（例えば特開平１０−２８９３１５号に開示）において本実施例を適用することも可能である。また、固定されたカメラ装置群ではなく、一台一台を自由に動かすことの出来るカメラ群を用いてステレオ撮影するような場合にも、本発明のランダムパターン生成装置とその方法は適用可能である。
【０１６７】
［距離画像生成装置−実施例１］
次に、上述のランダムパターン生成装置、方法を適用した距離画像生成装置および距離画像生成方法について説明する。
【０１６８】
図１４に、本発明の距離画像生成装置の構成ブロック図を示す。距離画像生成装置は、先に説明したステレオ画像法を適用して計測対象の三次元形状を計測するものであり、立体形状を持つ、例えば人の顔等の被測定対象物に対して角度の異なる位置に配置した２つのカメラ、いわゆる基準カメラと参照カメラを用いて像を撮り込み、これら２つのカメラの撮影する画像に基づいて図１で説明したステレオ画像法に基づいて被測定対象物の表面形状を計測する。
【０１６９】
前述した本発明のランダムパターン生成装置は、図１４における投光パターン生成装置１２１１に相当する。投光パターン生成装置１２１１において生成されたランダムパターンは、投光パターン照射手段１２１２において照射される。この投光パターンは一様乱数や正規乱数に基づく非周期的投光パターンである。
【０１７０】
投光パターン評価手段１２１０は、基準カメラ２０１、参照カメラ２０４によって撮影された投光パターン画像の少なくともいずれか一方の画像の評価を実行する。この画像評価は、前述のランダムパターン生成装置に関する説明中で述べた図５の構成、および図６〜図９の評価フローにおける評価を基準カメラ２０１、または参照カメラ２０４によって撮影された投光パターン画像に適用して実行される。または、投光パターンとして濃淡画像を使用することも可能であり、図１０、図１１の処理にしたがって、投光パターンの生成、および評価を実行してもよい。
【０１７１】
すなわち、例えば、投光パターン評価手段１２１０が、図１４の基準カメラ２０１の撮り込んだ投光パターン画像について図６の評価フローに従って評価を行なう場合は、フレームメモリ１ａ，２０３からの画像を図６の評価フローのステップ６０５に従って、撮り込み画像ｐ（ｉ，ｊ）、または撮り込み画像ｐ（ｉ，ｊ）を所定の閾値Ｔｐで処理して得られる二値画像ｐ’（ｉ，ｊ）の変化率ｒをパターンとしての適性度を示すパターン適性度指標値として算出して、ステップ６０６の閾値との比較を実行して適切な投光パターンであるか否かの判定を行なう。
【０１７２】
また、先に説明した図７の評価フローに従って基準カメラ２０１、または参照カメラ２０４によって撮影された投光パターン画像の評価を行なう場合は、撮り込み画像ｐ（ｉ，ｊ）、または撮り込み画像ｐ（ｉ，ｊ）を所定の閾値Ｔｐで処理して得られる二値画像ｐ’（ｉ，ｊ）をステップ７０１に示される二値画像として、図７に示す以下のステップ７０２〜７０９を実行して、各ウィンドーの変化率ｒｍをパターンとしての適性度を示すパターン適性度指標値として算出して評価を行なう。
【０１７３】
また、先に説明した図８の評価フローに従って基準カメラ２０１、または参照カメラ２０４によって撮影された投光パターン画像の評価を行なう場合は、撮り込み画像ｐ（ｉ，ｊ）、または撮り込み画像ｐ（ｉ，ｊ）を所定の閾値Ｔｐで処理して得られる二値画像ｐ’（ｉ，ｊ）をステップ８０１に示される二値画像として、図８に示す以下のステップ８０２〜８１０を実行して、各ウィンドーの特徴行列Ｒの固有値をパターンとしての適性度を示すパターン適性度指標値として求め、これを閾値と比較して評価を行なう。
【０１７４】
また、先に説明した図９の評価フローに従って基準カメラ２０１、または参照カメラ２０４によって撮影された投光パターン画像の評価を行なう場合は、撮り込み画像ｐ（ｉ，ｊ）、または撮り込み画像ｐ（ｉ，ｊ）を所定の閾値Ｔｐで処理して得られる二値画像ｐ’（ｉ，ｊ）をステップ９０１に示される二値画像として、図９に示す以下のステップ９０２〜９０９を実行して、異なるウィンドー間のマッチング率を求め、これをパターンとしての適性度を示すパターン適性度指標値として閾値と比較して評価を行なう。
【０１７５】
また、上記以外にもパターン画像の分散値（図１１参照）を使用した評価も可能であり、さらに投光パターンとして濃淡画像を使用する場合には、図１０、図１１の処理にしたがって、投光パターンの生成、および評価処理を実行する。
【０１７６】
なお、投光パターン評価手段１２１０は、基準カメラ２０１の撮り込み画像、参照カメラ２０２の撮り込み画像のいずれか一方を評価する構成であっても、あるいは両画像の評価を行なう構成としてもよい。
【０１７７】
上述の各種評価方法によって投光パターンとして不適切であると判定された場合は、図１４の投光パターン生成手段１２１１において、当該パターンの使用を中止し、新たな投光パターンを選択、または生成して測定対象に照射する。
【０１７８】
投光パターン生成装置１２１１は、異なる非周期パターンを予めスライド等に形成した投光パターンスライドを複数有し、これら複数の投光パターンから順次投光パターンを選択して照射し、投光パターン評価手段１２１０において投光パターンとして適切であると認められたものを最終的な投光パターン、すなわち距離画像を生成するためのパターンとして選択する。あるいは、投光パターン照射手段１２１２が透過型液晶表示素子を有し、投光パターン生成装置１２１１が透過型液晶表示素子を制御して動的にランダムパターンを生成して投影する構成としてもよい。動的なランダムパターン生成手法については、図１５を用いて詳細に説明する。
【０１７９】
投光パターン評価手段１２１０において投光パターンとして適切であると認められた場合は、その結果が画像対応付け手段２０５に通知され、その投光パターンにおいて計測対象とともに基準カメラ２０１と参照カメラ２０２により撮影され、フレームメモリ２０３、２０４に格納された画像に基づいて、画像間対応付け手段２０５が投光パターン画像間の対応付け処理を行なう。すなわち、基準カメラ２０１の撮影した画像の各画素に対する参照カメラ２０２の撮影した参照画像の画素の対応付け処理を行ない、基準画像に対する参照画像の視差データを距離画像生成手段２０６に出力し、距離画像生成手段２０６が距離画像の生成処理を実行する。
【０１８０】
本発明の距離画像生成装置において、距離画像の生成される際に使用される投光パターンは、投光パターン生成手段１２１１によって生成または選択され、投光パターン評価手段１２１０において適切であると認められた投光パターンであるので、画像の対応付け処理においてミスマッチングの発生が抑制されることとなり、画像間対応付け手段２０５における画像間の対応付け処理、および距離画像生成手段２０６における距離画像生成処理が高精度に行われる。
【０１８１】
なお、図１４に示す構成の各ブロックにおける処理、およびブロック間データ転送制御等は、図示しない制御手段によって制御され、所定のメモリ、例えばＲＡＭ，ＲＯＭ等の半導体メモリ、磁気ディスク、光ディスク等の記憶媒体に記録された制御プログラムによって制御することができる。また、図示しない入力手段によってユーザがコマンド、パラメータ等を入力して各制御態様を変更することが可能である。
【０１８２】
次に、投光パターン照射手段１２１２が透過型液晶表示素子を有し、投光パターン生成装置１２１１が透過型液晶表示素子を制御して動的にランダムパターンを生成して投影する構成ブロック図を図１５に示す。
【０１８３】
図１５に示されるように、投光パターン照射手段１２１２は透過型液晶表示素子１３２０を有し、光源１３２１からの光が透過型液晶表示素子１３２０を介して測定対象２００に照射される。
【０１８４】
投光パターン生成手段１２１１は、液晶制御手段１３０７を有し、液晶制御手段１３０７が透過型液晶表示素子１３２０に様々な非周期パターンを形成する制御を実行する。また、投光パターン生成手段１２１１は、先にランダムパターン生成装置の説明中で用いた図５の構成部５０１〜５０４，５０７，５０８と同様の構成を持ち、例えば２値の一様乱数を発生する一様乱数発生手段１３０１、一様乱数発生手段１３０１において発生した一様乱数に基づいて、予め設定された標準偏差ｓ、平均値ａｍに基づいて正規乱数Ｒｋを発生する正規乱数発生手段１３０２、正規乱数発生手段１３０２において発生された正規乱数Ｒｋに基づいて画像ｒｄ＿ｉｍ［ｋ］を生成する画像生成手段１３０３、予め定めた閾値Ｔにより画像ｒｄ＿ｉｍ［ｋ］の二値化処理を行ない、二値画像ｆ（ｉ，ｊ）を生成する二値画像生成手段１３０４を有する。
【０１８５】
液晶制御手段１３０７は、二値画像生成手段１３０４の生成した二値画像に対応するパターンを形成するように透過型液晶表示素子１３２０を制御する。
【０１８６】
標準偏差ｓは、標準偏差設定手段１３０５において様々な値を設定可能であり、設定された標準偏差に基づいて様々な正規乱数が正規乱数発生手段１３０２において発生可能であり、閾値Ｔについても閾値設定手段１３０６において様々な値を設定可能であり、これら各種設定値に基づいて様々な二値画像が二値画像生成手段１３０４において生成される。
【０１８７】
一方、投光パターン評価手段１２１０は、基準カメラ２０１の撮り込み画像、参照カメラ２０２の撮り込み画像をフレームメモリ１ａ，２０３、フレームメモリ１ａ，２０４を介して受信し、撮影画像変化率検出手段１３０８において撮り込み画像ｐ（ｉ，ｊ）、または撮り込み画像ｐ（ｉ，ｊ）を所定の閾値Ｔｐで処理して得られる二値画像ｐ’（ｉ，ｊ）の変化率ｒをパターンとしての適性度を示すパターン適性度指標値として算出する。さらに変化率判定手段１３０９が予め定めた閾値Ｔ１との比較処理を実行して適切な投光パターンであるか否かの判定を行なう。
【０１８８】
変化率判定手段１３０９における判定結果は、投光パターン生成手段１２１１にフィードバックされ、比較結果が不適切な投光パターンであると判定した場合には、投光パターン生成手段１２１１の閾値Ｔ設定手段１３０６、または標準偏差ｓ設定手段１３０５において新たな閾値Ｔ、または標準偏差ｓが設定されて新たな二値画像が生成される。この新たな二値画像に基づくパターンが液晶制御手段１３０７の制御のもとに投光パターン照射手段１２１２中の透過型液晶表示素子１３２０に形成される。
【０１８９】
変化率判定手段１３０９における比較結果が適切な投光パターンであると判定した場合には、図１５には示されていないが、図１４で説明したように画像対応付け手段２０５に通知され、各フレームメモリに格納された画像データに基づく対応付け処理が開始される。
【０１９０】
なお、図１５においては、投光パターン評価手段１２１０が、先のランダムパターン生成装置の説明における図５、６の画像評価構成を採用したものについて示したが、この評価構成に限らず、図７で説明した各ウィンドーの変化率ｒｍに基づく評価、図８で説明した各ウィンドーの特徴行列の固有値に基づく評価、図９で説明した２つのウィンドー間のマッチング率に基づく評価構成のいずれを適用することも可能である。
【０１９１】
図１６に本発明の距離画像生成装置における投光パターン決定処理フローを示す。以下、図１６の各ステップについて図１４、図１５を参照しながら、その処理を説明する。
【０１９２】
ステップ１４０１は、測定対象２００に投影する投光パターンを設定するステップである。投光パターン生成手段１２１１が、予め複数の非周期パターンを形成したスライドを有する場合は、その中から１つのスライドを投光パターンとして選択する。投光パターン生成手段１２１１が透過型液晶表示素子１３２０を有する場合は、投光パターン生成手段１２１１が図１５に示す二値画像生成手段１３０４で生成した二値画像に基づくパターンを透過型液晶表示素子１３２０に形成するように液晶制御手段１３０７によって制御する。
【０１９３】
ステップ１４０２は測定対象２００にある投光パターンを照射して、基準カメラ２０１、参照カメラ２０２を用いて撮影するステップである。各カメラで撮影された画像は、フレームメモリ１ａ，２０３、フレームメモリ２ａ，２０４に格納される。
【０１９４】
ステップ１４０３は、撮影された画像に基づく投光パターンの評価処理ステップである。投光パターン評価手段１２１０において、撮影された画像に基づく投光パターンの評価処理が実行される。これら評価処理は、先に説明した図５〜図９の評価処理構成を適用して行なわれる。
【０１９５】
ステップ１４０４は、投光パターン評価手段１２１０における評価処理において投光パターンとして不適切な部分があるか否かを判定するステップであり、不適切な部分がない場合は、ステップ１４０５に進み、画像間対応付け手段２０５、距離画像生成手段において撮り込み画像に基づく画像間対応付け処理および距離画像生成処理が実行される。
【０１９６】
ステップ１４０４における判定がＮｏ、すなわち投光パターン評価手段１２１０における評価処理において投光パターンとして不適切な部分があると判定された場合は、ステップ１４０６に進み、新たな投光パターンの生成、または選択処理がなされ、さらにステップ１４０１以下の処理、すなわち新たな投光パターンの照射、撮影、評価処理が実行される。
【０１９７】
本発明の距離画像生成装置における、実際の画像取得例を図１７に示す。図１７は測定対象に予め模様が付されており、この模様の付された測定対象に非周期投光パターンを照射して撮影すると、撮影結果が測定対象の本来の模様と投光パターンの重なりが発生して対応付け処理が困難になる例を示したものである。
【０１９８】
図１７（ａ）が測定対象に付されている模様であり、（ｂ）が最初の投光パターンである。投光パターン自体は非周期なものであるが、この非周期パターンが、測定対象に投光し（ｃ）、これを撮影すると、（ｄ）のように測定対象の本来の模様と投光パターンの重なりが発生して対応付け処理が困難になる。図１４または１３で示す投光パターン評価手段は、この図１７（ｄ）に示す画像の評価を、前述のように例えば図５〜９で説明した各方法で行なう。
【０１９９】
図１７（ｄ）に示す画像の評価は、投光パターンとして不適切であるとの評価となるので、投光パターン生成手段１２１１は、新たな投光パターン（ｅ）を生成して、測定対象に投光（ｆ）して、その結果（ｇ）を得る。この画像（ｇ）についても評価処理がなされ、評価の結果が適切であると判定されることにより、対応付け処理、距離画像生成処理が開始される。
【０２００】
本発明の距離画像生成装置は、上述のように、投光パターンを、その撮影画像に基づいて評価・検討して、ミスマッチングの発生が抑制された適正な投光パターン照射画像についての距離画像生成が可能な構成としたので、より高精度な距離画像を得ることができる。
【０２０１】
また、不適切な投光パターン、すなわち対応付け処理においてミスマッチングが多く発生するような投光パターン画像についての対応点付け処理、距離画像生成処理を行なう以前に、適切な投光パターンを選択することが可能となるので、処理の複雑な対応点付け処理、距離画像生成処理を何度も繰り返して行なう必要なく１回のみの処理で高精度な距離画像を得ることができる。
【０２０２】
なお、上述の実施例では、画像撮り込みを行なうカメラを２台として、２つの投光パターン画像から距離画像を求める構成について説明したが、さらに３台以上のカメラを用いたマルチベースラインステレオシステム（例えば特開平１０−２８９３１５号に開示）において本実施例を適用することも可能であり、各カメラにおける取り込み画像のいずれか、または複数の画像に対して上述の評価を実行する構成としてもよい。
【０２０３】
また、固定されたカメラ装置群ではなく、一台一台を自由に動かすことの出来るカメラ群を用いてステレオ撮影するような場合にも、本発明の構成は適応でき、各カメラの移動後の撮影画像の評価が容易に実行可能であり、任意の位置にカメラを設定することが容易となる。
【０２０４】
［距離画像生成装置−実施例２］
上述の距離画像生成装置の実施例１の構成は、取得投光パターン画像の評価を距離画像生成以前に実行する構成であるが、実際に距離画像を生成してそのミスマッチング領域の有無を判定して、その判定に基づいて新たな投光パターンを生成する構成について実施例２として説明する。
【０２０５】
距離画像は、例えば基準カメラと測定対象との距離値を撮影画像の各画素に例えば濃淡値として反映させた画像データであり、三次元グラフィックデータの生成等に用いられる。前述の実施例１では、距離画像の生成ステップ以前に投光パターンの適性を判定する構成であったが、実際の距離画像においてミスマッチング領域が検出される可能性を皆無にすることは保証されるものではなく、そこで実施例２では、距離画像生成後にミスマッチング領域が検出された場合に、さらに新たな投光パターンを照射してミスマッチング領域を削減した高精度な距離画像を得る構成を提供する。なお、ミスマッチング領域とは、距離画像において、正確な距離を示していない画素領域であり、画像間対応付け処理が正確に実行されなかった場合に発生する。
【０２０６】
実施例２の構成を図１８に示す。実施例１と異なる点は、ミスマッチング領域検出手段１６０１であり、ミスマッチング領域検出手段１６０１は、距離画像生成手段２０６によって生成された距離画像を格納した画像メモリ２０７から距離画像を取得してミスマッチング領域を検出する。
【０２０７】
ミスマッチング領域検出手段１６０１によって距離画像中にミスマッチング領域が検出された場合は、投光パターン生成手段１６０２に通知され、新たな投光パターンが生成、または選択され、投光パターン照射手段１６０３によって新たな投光パターンが照射される。投光パターン照射手段１６０３の構成は、実施例１と同様であり、説明を省略する。
【０２０８】
投光パターン生成手段１６０２は、測定対象に対して投光するパターンを生成する構成を持ち、乱数を用いた異なる複数の非周期的投光パターンを生成可能である。さらに、実施例１における投光パターン評価手段と同様の評価構成を持ち、生成した投光パターンについてパターンとしての適性度を示すパターン適性度指標値に基づく判定を行なって判定の良好なもののみを投光パターンとして設定する。
【０２０９】
すなわち、生成投光パターンのパターンとしての適性度を示すパターン適性度指標値、例えば前述した変化率、特徴量、分散値等を算出し、算出したパターンとしての適性度を示すパターン適性度指標値を予め定めた閾値と比較することで距離画像生成用投光パターンとしての適性を判定し、適性ありと判定されたパターンのみを投光パターンとして設定する。従って、投光パターン生成手段１６０２によって生成され、投光パターン照射手段１６０３によって照射される投光パターンは基本的に周期性を生じにくい投光パターンとなる。
【０２１０】
ミスマッチング領域検出手段１６０１の処理を中心として、以下実施例２の構成を説明する。図１９に本実施例の距離画像生成装置の処理フローを示す。
【０２１１】
ステップ１７０１は、測定対象２００に投影する投光パターンを設定するステップである。投光パターン生成手段１６０２が、予め複数の非周期パターンを形成したスライドを有する場合は、その中から１つのスライドを投光パターンとして選択する。投光パターン生成手段１６０２が、図１５で説明した透過型液晶表示素子を有する場合は、投光パターン生成手段１６０２が図１５に示す二値画像生成手段１３０４で生成した二値画像に基づくパターンを透過型液晶表示素子１３２０に形成するように液晶制御手段１３０７によって制御する。
【０２１２】
ステップ１７０２は投光パターン画像に基づく距離画像の生成処理ステップであり、基準カメラ２０１、参照カメラ２０２で撮影された画像、すなわちフレームメモリ１ａ，２０３、フレームメモリ２ａ，２０４に格納された画像に基づいて、画像間対応付け手段２０５、距離画像生成手段２０６において実行される。
【０２１３】
ステップ１７０３は、生成された距離画像に基づくミスマッチング領域の検出処理であり、ミスマッチング領域検出手段１６０１によって実行される。ミスマッチング領域の検出処理について、図２０、図２１を用いて説明する。
【０２１４】
図２０は、ミスマッチングの検出処理の概略を示すものであり、ステップ１８０１において距離画像のラベリング処理を実行し、ラベリング画像１８０２を取得し、ラベリング画像１８０２に基づいて、ステップ１８０３においてミスマッチング領域を検出する。
【０２１５】
図２０に示すステップ１８０１におけるラベリング処理の詳細処理フローを図２１に示す。まず、図２１のステップ２１０１において、図１８に示すミスマッチング領域検出手段１６０１は、画像メモリ２０７から距離画像Ｄ１を読み出す。
【０２１６】
ステップ２１０２において、ミスマッチング領域検出手段１６０１は、ラベリング処理を開始する座標を設定するための初期化処理として、ラベリング未処理画素数ｋを初期化し、ミスマッチング領域検出手段１６０１は、内蔵する座標記憶用バッファｘ［ｋ］，ｙ［ｋ］を初期化する。
【０２１７】
ステップ２１０３において、ミスマッチング領域検出手段１６０１は、距離画像Ｄ１の中の計測対象（被写体）に対応する部分の任意の位置座標を注目画素の座標（ｉ，ｊ）として設定する。
【０２１８】
ステップ２１０４において、ミスマッチング領域検出手段１６０１は、距離画像Ｄ１の注目画素の画素値ｇ（ｉ，ｊ）を抽出する。画素値ｇ（ｉ，ｊ）は、画素（ｉ，ｊ）の距離値を示している。
【０２１９】
次に、ステップ２１０５において、ミスマッチング領域検出手段１６０１は、注目画素（ｉ，ｊ）に隣接する画素の内の１つの未処理画素の画素値ｇ（ｉ＋ｍ，ｊ＋ｎ）（ただしｍ，ｎ＝−１，０，１）を抽出する。
【０２２０】
ステップ２１０６において、ミスマッチング領域検出手段１６０１は、注目する画素（ｉ，ｊ）の画素値ｇ（ｉ，ｊ）が、ラベル値Ｌ０であるか否かを判定し、さらに注目する画素の画素値ｇ（ｉ，ｊ）と隣接する画素の画素値ｇ（ｉ＋ｍ，ｊ＋ｎ）の差の絶対値が所定の閾値より小さいか否かを判定する。
【０２２１】
この閾値との比較により、画素（ｉ，ｊ）と隣接画素（ｉ＋ｍ，ｊ＋ｎ）の連続性が判定される。すなわち画素（ｉ，ｊ）と隣接画素（ｉ＋ｍ，ｊ＋ｎ）の画素値、ｇ（ｉ，ｊ）とｇ（ｉ＋ｍ，ｊ＋ｎ）との差分が閾値より小さければその２つの画素は連続する領域にあると判定され、閾値より大であれば不連続の領域であると判定する。
【０２２２】
注目する画素の画素値ｇ（ｉ，ｊ）がＬ０でなく、かつ、隣接する画素の画素値ｇ（ｉ＋ｍ．ｊ＋ｎ）が画素値ｇ（ｉ，ｊ）に対して連続していると判定された場合ステップ２１０７に進む。
【０２２３】
ステップ２１０７において、ミスマッチング領域検出手段１６０１は、ラベリング未処理画素数ｋを１だけインクリメントし、隣接する画素のｘ座標ｉ＋ｍを座標記憶用バッファｘ［ｋ］に記憶し、ｙ座標ｊ＋ｎを座標記憶用バッファｙ［ｋ］に記憶する。
【０２２４】
ステップ２１０８において、ミスマッチング領域検出手段１６０１は、注目する画素に隣接する８つの全ての画素に対してステップ２１０５，２１０６の処理を施したか否かを判定し、隣接する画素にステップ２１０５，２１０６の処理を施していないものが存在すると判定した場合、ステップ２１０５に戻り、それ以降の処理を繰り返す。その後、ステップ２１０８で、隣接する８つの全ての画素に対してステップ２１０５，２１０６の処理を施したと判定された場合、ステップ２１０９に進む。
【０２２５】
ステップ２１０９において、ミスマッチング領域検出手段１６０１は、注目画素の画素値ｇ（ｉ，ｊ）をラベル値Ｌ０に置換する。ここで、ラベル値Ｌ０は、距離画像Ｄ１の測定対象（被写体）に対応する部分の画素に対して付されるものである。
【０２２６】
ステップ２１１０において、ミスマッチング領域検出手段１６０１は、ラベリング未処理画素数ｋが０であるか否かを判定し、ラベリング未処理画素数ｋが０ではないと判定した場合、ステップ２１１１に進む。
【０２２７】
ステップ２１１１において、ミスマッチング領域検出手段１６０１は、座標記憶用バッファｘ［ｋ］に記憶されている値を注目画素のｘ座標とし、座標記憶用バッファｙ［ｋ］に記憶されている値を注目画素のｙ座標とする。これにより、先ほどまでの注目画素に隣接し、かつ画素値が連続する画素が新たな注目画素とされる。その後、ミスマッチング領域検出手段１６０１は、ラベリング未処理画素数ｋを１だけインクリメントしてステップ２１０４に戻る。
【０２２８】
その後、ステップ２１１０において、ラベリング未処理画素数ｋが０であると判定されるまで、ステップ２１０４以降の処理が繰り返され、ラベリング未処理画素数ｋが０であると判定された場合、ステップ２１１２に進む。ステップ２１１２において、ミスマッチング領域検出手段１６０１は、距離画像Ｄ１の画素で画素値がラベル値Ｌ０に置換されていない画素の画素値をミスマッチング領域であることを示すラベル値Ｌ１に置換する。
【０２２９】
上述のようなラベリング処理が終了すると、距離画像Ｄ１において距離データが所定の閾値以下の差分を有する連続領域には同一のラベルが付与され、例えば図２０のラベリング画像１８０２が得られる。
【０２３０】
ミスマッチング領域検出手段１６０１は、同一ラベルが付与された画素に囲まれていて、かつ周囲と異なるラベルが付与された小さな孤立領域が存在するか否かを上記のラベル値にしたがって判定する。図２０のラベリング画像１８０２では、領域Ｂ，Ｃ，Ｄがこれらの条件に該当し、この領域Ｂ，Ｃ，Ｄをミスマッチング領域であると判定（ステップ１８０３）する。
【０２３１】
図１９に戻って、さらに説明を続ける。上述のようなミスマッチング領域検出処理の結果、ミスマッチング領域が検出されなかった場合は、ステップ１７０４の判定がＮｏとなり、ステップ１７０５に進み、生成された距離画像が出力される。
【０２３２】
また、ミスマッチング領域が検出された場合は、ステップ１７０６に進み、新たな投光パターンの生成、または選択処理がなされ、さらにステップ１７０１以下の処理、すなわち新たな投光パターンに基づく距離画像生成処理、ミスマッチング領域検出処理が実行される。新たな投光パターンの生成処理は、実施例１と同様の処理であり、説明を省略する。
【０２３３】
なお、図１９のステップ１７０４は、ミスマッチング領域が検出されたか否かで処理を変更する構成としてあるが、ある一定の閾値を定めて、ミスマッチング領域が設定閾値以上の量を超える場合にのみ新たな投光パターンに基づく距離画像生成処理を行なうように構成してもよい。
【０２３４】
本実施例の距離画像生成装置および方法によれば、生成された距離画像に基づくミスマッチング領域を検出して新たな投光パターンの設定を行なう構成であり、また、投光パターン生成手段が生成し使用されるパターンは、変化率等の各種のパターンとしての適性度を示すパターン適性度指標値に基づいて距離画像生成用投光パターンとしての適性が良好であると判定されたパターンのみとしたので、適切な距離画像の生成が可能であり、ミスマッチング領域の解消された高精度な距離画像を出力することができる。
【０２３５】
以上、特定の実施例を参照しながら、本発明について詳解してきた。しかしながら、本発明の要旨を逸脱しない範囲で当業者が該実施例の修正や代用を成し得ることは自明である。すなわち、例示という形態で本発明を開示してきたのであり、限定的に解釈されるべきではない。また、上述の実施例を適宜組み合わせて構成したものも、本発明の範囲に含まれるものであり、本発明の要旨を判断するためには、冒頭に記載した特許請求の範囲の欄を参酌すべきである。
【０２３６】
【発明の効果】
以上詳記したように、本発明によれば、投光用ランダムパターンを、パターンあるいは撮影画像に基づいて、パターンの変化率、特徴量、分散値、あるいはパターンマッチング等のパターンとしての適性度を示すパターン適性度指標値により評価することにより、適切な投光パターンを得る構成としたので、様々な測定対象に対して、適切な投光パターン画像を取得することが可能となり、高精度な距離画像が得られる。
【０２３７】
さらに、本発明によれば、投光パターンを、パターンあるいはその撮影画像に基づいて評価・検討して、ミスマッチングの発生が抑制された適正な投光パターン照射画像についての距離画像生成を可能としたので、より高精度な距離画像を得ることができ、ミスマッチングが多く発生するような投光パターン画像についての対応点付け処理、距離画像生成処理を行なう以前に、適切な投光パターンを選択することが可能となるので、処理の複雑な対応点付け処理、距離画像生成処理を何度も繰り返して行なう必要なく１回のみの処理で高精度な距離画像を得ることができる。
【０２３８】
さらに、本発明によれば、生成された距離画像に基づくミスマッチング領域を検出して新たな投光パターンの設定を行なう構成とし、変化率等の各種のパターンとしての適性度を示すパターン適性度指標値に基づいて距離画像生成用投光パターンとしての適性が良好であると判定されたパターンのみを使用するように構成したので、ミスマッチング領域の解消された高精度な距離画像を出力することが可能となる。
【図面の簡単な説明】
【図１】本発明において使用される三次元情報取得構成として適用可能なステレオ画像法について説明した図である。
【図２】本発明のランダムパターン生成装置を適用した距離画像生成装置の一実施例の構成を示すブロック図である。
【図３】本発明における投光パターン生成及び照射手段において用いられる非周期投光パターンの例を示す図である。
【図４】本発明における投光パターン生成及び照射手段の構成例を示す図である。
【図５】本発明のランダムパターン生成装置の構成ブロック図である。
【図６】本発明のランダムパターン生成装置の投光パターン生成処理フローを説明する図である。
【図７】本発明のランダムパターン生成装置における投光パターン評価処理フロー（その１）を説明する図である。
【図８】本発明のランダムパターン生成装置における投光パターン評価処理フロー（その２）を説明する図である。
【図９】本発明のランダムパターン生成装置における投光パターン評価処理フロー（その３）を説明する図である。
【図１０】本発明のランダムパターン生成装置の濃淡投光パターン生成処理フローを説明する図である。
【図１１】本発明のランダムパターン生成装置における濃淡投光パターン評価処理フローを説明する図である。
【図１２】本発明のランダムパターン生成装置において生成した投光パターンによる距離画像生成例（その１）を示す図である。
【図１３】本発明のランダムパターン生成装置において生成した投光パターンによる距離画像生成例（その２）を示す図である。
【図１４】本発明の距離画像生成装置の一実施例（実施例１）の構成を示すブロック図である。
【図１５】本発明の距離画像生成装置の投光パターン生成手段、投光パターン評価手段の詳細構成を示す図である。
【図１６】本発明の距離画像生成装置における距離画像生成処理のフローチャートを示す図である。
【図１７】本発明の距離画像生成装置における投光パターン生成手段、投光パターン評価手段の処理過程を説明する図である。
【図１８】本発明の距離画像生成装置の一実施例（実施例２）の構成を示すブロック図である。
【図１９】本発明の距離画像生成装置の処理フローを示す図である。
【図２０】本発明の距離画像生成装置におけるミスマッチング領域の検出処理フローを示す図である。
【図２１】本発明の距離画像生成装置における距離画像に基づくラベリング処理の処理フローを示す図である。
【符号の説明】
２０１…基準カメラ
２０２…参照カメラ
２０３，２０４…フレームメモリ
２０５…画像間対応付け手段
２０６…距離画像生成手段
２０７…画像メモリ
２０８…出力手段
２２０…投光パターン生成及び照射手段
４０１…制御手段
４０２…カメラ１（基準カメラ）
４０３…カメラ２（参照カメラ）
４０４…画像メモリ
５０１…一様乱数発生手段
５０２…正規乱数発生手段
５０３…画像生成手段
５０４…二値画像生成手段
５０５…変化率検出手段
５０６…変化率判定手段
５０７…標準偏差設定手段
５０８…閾値設定手段
１２１０…投光パターン評価手段
１２１１…投光パターン生成手段
１２１２…投光パターン照射手段
１３０１…一様乱数発生手段
１３０２…正規乱数発生手段
１３０３…画像生成手段
１３０４…二値画像生成手段
１３０５…標準偏差設定手段
１３０６…閾値設定手段
１３０７…液晶制御手段
１３０８…撮影画像変化率検出手段
１３０９…変化率判定手段
１６０１…ミスマッチング領域検出手段
１６０２…投光パターン生成手段
１６０３…投光パターン照射手段[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a random pattern generation device and method for determining the three-dimensional shape of an object, a distance image generation device and method, and a program providing medium. In particular, a method for measuring the three-dimensional shape of the surface of an object using images obtained by photographing an object existing in a distance range in a three-dimensional space from a plurality of different positions, that is, a random pattern generation apparatus suitable for a stereo image method and the method thereof The present invention relates to a distance image generating apparatus and method, and a program providing medium.
[0002]
[Prior art]
Conventional methods for obtaining a three-dimensional shape of an object are roughly classified into an active method (active method) and a passive method (passive method). Active methods include projecting light and measuring the time it takes for the light to be reflected and returning, as well as obtaining the distance to each object to be measured, or applying slit-shaped pattern light to the object to be measured. There is a method such as a light cutting method in which a three-dimensional shape is measured by examining the shape of the pattern light projected to the measurement object.
[0003]
A typical passive method is a stereo image method that uses the principle of triangulation, which uses two or more cameras to find the corresponding points between the images, finds the parallax, This is a method of measuring distance.
[0004]
The principle of the stereo imaging method will be briefly described. The stereo image method uses a plurality of cameras to associate the pixels in a plurality of images obtained by photographing the same object from two or more viewpoints (different gaze directions), thereby positioning the measurement object in a three-dimensional space. Is to seek. For example, the same object is photographed from different viewpoints by a standard camera and a reference camera, and the distance of the measurement object in each image is measured by the principle of triangulation.
[0005]
FIG. 1 is a diagram for explaining the principle of the stereo imaging method. The reference camera (Camera 1) and the reference camera (Camera 2) capture the same object from different viewpoints. Consider obtaining the depth of a point “mb” in an image taken by a reference camera.
[0006]
The objects that appear at the point “mb” in the image captured by the reference camera are “m1”, “m2”, and “m3” in the image captured by the reference camera that captures the same object from different viewpoints. It will be developed on a straight line. This straight line is referred to as an epipolar line Lp.
[0007]
The position of the point “mb” in the reference camera appears on a straight line called “epipolar line” in the image by the reference camera. As long as the point P to be imaged (a point existing on a straight line including P1, P2, and P3) exists on the line of sight of the reference camera, regardless of the depth, that is, the distance from the reference camera, on the reference image It appears at the same observation point “mb”. On the other hand, the point P on the image captured by the reference camera appears on the epipolar line at a position corresponding to the distance between the reference camera and the observation point P.
[0008]
FIG. 1 illustrates the correspondence between the epipolar line and the observation point “mb” in the reference image. As shown in the figure, as the position of the observation point P changes to P1, P2, and P3, the observation point in the reference image shifts to “m1”, “m2”, and “m3”.
[0009]
The distance of the point P can be identified by searching the observation point “mb” on the epipolar line using the above geometric optical properties. This is the basic principle of the “stereo imaging method”. In this way, three-dimensional information about all pixels on the screen is acquired. The acquired three-dimensional information can be used as pixel attribute data corresponding to each pixel.
[0010]
The stereo image method described above has a configuration using one reference camera and one reference camera. However, an evaluation value is obtained by a multi-baseline stereo method using a plurality of reference cameras, You may comprise so that the three-dimensional information for every pixel may be acquired based on an evaluation value. In the multi-baseline stereo image method, images obtained by one reference camera and a plurality of reference cameras are used, evaluation values representing correlations with the reference camera images are obtained for each of the plurality of reference camera images, and the respective evaluation values are obtained. Addition is performed, and the addition value is used as a final evaluation value. Details of the multi-baseline stereo imaging method are described in, for example, “Stereo matching using a plurality of baseline lengths”, IEICE Transactions D-11 Vol. J75-D-II No. 8 pp. 1317-1327, August 1992.
[0011]
As described above, the stereo imaging method associates pixels in a plurality of images obtained by photographing the same object from two or more viewpoints (different gaze directions) using a plurality of cameras. By performing “scoring”, the position of the measurement object in the three-dimensional space is determined.
[0012]
Conventionally, the method of “corresponding points” that is often used is roughly divided into Pixel-based matching, Area-based matching, and Feature-based matching. Pixel-based matching is a method in which correspondence between points in one image is searched as it is in the other image (C. Lawrence Zitnick and Jon A. Webb: Multi-baseline Stereo Usage Surface Re-Technology, Tec. CS-96-196,
(1996)).
[0013]
Area-based matching is a method of searching for a correspondence of a point in one image using a local image pattern around the point when searching in the other image (Okutomi, Kaide: multiple baselines) Stereo matching using the head, IEICE Transactions D-II, Vol. J75-D-II, No. 8, pp. 1317-1327, (1992), Yokoyama, Miwa, Kasumigahara, Koyanazu, Hayashi , After: Stereo Camera System and Its Application, SRF '97, (1997), Kinade, Kimura: Video rate stereo machine, Journal of the Robotics Society of Japan, Vol.13, No.3, pp.322-326 (1995) ), Kaide, Mosquito field, Kimura, Kawamura, Yoshida, Oda: Development of video rate stereo machine, Nippon Vol.15, No.2, pp.261-267, (1997), Yamaguchi, Takachi, Iguchi: Stereo correspondence for stone image measurement using adaptive window method, humanities and computer, Vol. 32, No. 10, pp. 55-60, (1996), Yokoya: Recent Special Issue on Signal Processing Recent Topics in Computer Vision, System / Control / Information, Vol.38, No.8, pp.436 ~ 441, (1994)).
[0014]
Feature-based matching is a method of extracting features such as shading edges from images and performing association using only features between images (H. H. Baker and TO B. Inford: Depth from edge intensity). based stereo, In Proc. IJCAI '81, (1981), Ishiyama, Kakuho, Kawai, Ueshiba, Tomita: Search for corresponding candidates in segment-based stereo, IEICE Technical Report, Vol. 96, No. 136, (1997), W. E. L. Grimson: Computational experience with a feature based stereo algorithm, IEEE Trans. PAMI, Vol. 17-34, (1985)).
[0015]
The characteristics of the above methods are summarized as follows.
(1) Since Pixel-based matching and Area-based matching search for corresponding points for each pixel, the obtained distance image is dense. On the other hand, in Feature-based matching, since only the feature points are associated, the obtained distance image is sparse.
[0016]
(2) Area-based matching performs a kind of correlation operation, and thus requires a calculation cost as compared with Pixel-based matching and Feature-based matching, but is not necessarily a problem that cannot be solved by increasing the algorithm speed.
[0017]
(3) Pixel-based matching performs only association between pixels, and thus has a considerably high calculation speed. However, association using gray values between pixels is not easy due to a difference in characteristics between the left and right cameras.
[0018]
From the above-mentioned features, in general, Area-based matching is effective and often used as a method for obtaining a three-dimensional shape (or depth) of an object for each pixel with high accuracy.
[0019]
Corresponding point marking processing in the stereo image method is performed by the various methods described above. However, it is necessary to measure a three-dimensional shape with any of the above-described methods for an object having almost no features (shading, shape, color, etc.) such as a white wall or a human face. Corresponding point marking processing becomes difficult. This is one big problem in the processing by the stereo image method.
[0020]
Periodic patterns such as checkers and frequency-modulated sine waves are used in order to more accurately perform matching processing on objects that have almost no features (such as shading, shape, and color) such as white walls and human faces. Several techniques have been disclosed for reducing image matching errors by multi-baseline stereo systems by projecting light (SB Kang, JA Webb, CL Zitnick and T. Kanade). : A Multibaseline Stereo System with Active Illumination and Real-time Image Acquisition, Proc IEEE Int Conf. Comput. Vis., Vol. guchi and S.K.Nayar:... Microscopic Shape from Focus Using Active Illumination, Proc Int Conf Pattern Recogn, Vol.12, No.Vol 1, pp.147~152, (1994) see).
[0021]
[Problems to be solved by the invention]
However, with the above-described conventional method, it is impossible to achieve complete removal of errors in matching processing, that is, mismatching. In addition, the configuration in which the above-described frequency-modulated periodic pattern is projected and the association process is performed reduces the mismatch region in a multi-baseline stereo machine using a plurality of (three or more) cameras. Although it is effective for two-lens stereo vision using two cameras, an error may occur in matching similar patterns because the generated pattern is periodic, There is a drawback that it is not always effective.
[0022]
Therefore, for example, we have proposed a configuration that enables high-accuracy range image generation by generating a non-periodic projection pattern using random numbers and projecting this random pattern onto the measurement target. Filed in Japanese Patent Application No. 11-269838).
[0023]
As described above, according to the measurement method in which a non-periodic random pattern is projected onto an object having no pattern and the association process is performed, there are almost no features (shading, shape, color, etc.) such as a white wall or a human face. The associating process with respect to the object is performed more accurately, and the distance image generation is performed with high accuracy.
[0024]
However, even when a non-periodic pattern is projected, depending on the shape of the measurement target or the positional relationship between the measurement target, the random pattern projection unit, and the imaging device, etc., it is repeated with a certain period on the shooting screen. In some cases, a pattern may be generated. In such a case, the association processing becomes difficult, and the distance image generation may be hindered. Even when an aperiodic random pattern is projected, whether or not the projected result includes a repetitive pattern cannot be determined unless it is actually projected.
[0025]
There is also a system that performs measurement by sequentially projecting multiple types of random patterns. In this case as well, it can be expected that a portion that cannot be measured well by using a plurality of random patterns can be reduced, but it is not possible to make up for all the portions that cannot be measured well with several predetermined patterns. It is also difficult to keep up with various environmental changes including camera placement.
[0026]
The present invention has been made in view of the above-described problems, and relates to a random pattern generation device and method, a distance image generation device and method, and a program providing medium in a distance image generation method according to a stereo image method. The present invention provides a random texture in which mismatching regions are unlikely to occur in the corresponding scoring process, for example, a non-periodic random image pattern configuration composed of binary or shaded random point groups and a generation method thereof.
[0027]
Furthermore, the present invention provides a new random light to be projected based on an image photographed by pattern projection so that the pattern on the measurement object is appropriate, that is, mismatching does not occur in the matching process. The present invention provides a distance image generation apparatus and method that reduce the occurrence of mismatching regions by determining a pattern.
[0028]
Furthermore, the present invention is configured to generate a pattern based on an image photographed by pattern projection so that the pattern on the measurement target is appropriate, that is, no mismatching occurs in the corresponding marking process. Specifically, for example, stereo cameras are not good at patterns that are repeated at regular intervals, so the projection result is calculated so that it does not have repetitive components, an appropriate random pattern is created, and the distance image is projected Provides a configuration for generating
[0029]
[Means for Solving the Problems]
  The present invention has been made in consideration of the above problems, and the first aspect thereof is
  Followed stereo imagingIn a random pattern generation device that generates a light projection pattern that projects a measurement target in measurement of a three-dimensional shape,
  A projection pattern generating means capable of generating a plurality of different aperiodic projection patterns using random numbers;
  A projection pattern evaluation unit that performs evaluation of the projection pattern generated in the projection pattern generation unit;
  The light projection pattern evaluation unit calculates a pattern suitability index value indicating the suitability of the light projection pattern generated by the light projection pattern generation unit based on the generated light projection pattern, and calculates the pattern suitability By comparing the index value with a predetermined threshold, it is possible to determine whether or not the generated projection pattern is suitable as a projection pattern for three-dimensional shape measurement.And
The light projection pattern evaluation means includes:
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
The value of any one of (1) to (4) is calculated as the pattern suitability index value, and the generated light projection pattern is projected for three-dimensional shape measurement by comparing the calculated value with a predetermined threshold value. It is a configuration that determines whether or not it has suitability as a patternThe present invention resides in a random pattern generation device.
[0030]
Furthermore, in one embodiment of the random pattern generation device of the present invention, the projection pattern generation means has a configuration for generating different aperiodic projection patterns by changing a standard deviation s that defines a normal random number. When the generated light projection pattern is determined not to be suitable as a light projection pattern for three-dimensional shape measurement in the light projection pattern evaluation unit, a new aperiodic pattern in which the standard deviation s is changed is determined. It has the structure which produces | generates a light projection pattern.
[0031]
Furthermore, in one embodiment of the random pattern generation device of the present invention, the projection pattern generation means has a configuration for generating a binary image obtained by binarizing an image composed of normal random numbers using a threshold T, and By changing the threshold value T, a different aperiodic projection pattern is generated, and the generated projection pattern is suitable as a projection pattern for three-dimensional shape measurement in the projection pattern evaluation unit. When it is determined that the threshold value T is not included, a new aperiodic projection pattern in which the threshold value T is changed is generated.
[0032]
Furthermore, in one embodiment of the random pattern generation device of the present invention, the projection pattern generation unit has a configuration for generating a grayscale image based on a normal random number, and the generated projection pattern is the projection pattern evaluation unit. When it is determined that the projection pattern is not suitable as a projection pattern for measuring a three-dimensional shape, it has a configuration for generating a new aperiodic projection pattern in which a parameter defining a normal random number is changed. To do.
[0037]
Furthermore, in one embodiment of the random pattern generation device of the present invention, the projection pattern evaluation unit extracts at least one or more windows that are partial areas from the projection pattern generated by the projection pattern generation unit, and the pattern The configuration is characterized in that the aptitude index value is calculated.
[0038]
  Furthermore, the second aspect of the present invention provides
  Followed stereo imagingIn a random pattern generation method for generating a light projection pattern to be projected onto a measurement object in measurement of a three-dimensional shape
(A) a projection pattern generation step of generating an aperiodic projection pattern using random numbers;
(B) an evaluation step for evaluating the projection pattern generated in the projection pattern generation step;
  (C) A pattern suitability index value indicating the suitability of the projection pattern generated in the projection pattern generation step as the patternAs
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
Any one of the above values (1) to (4)A pattern suitability index value calculating step for calculating based on the generated projection pattern;
  (D) an aptitude determination step for determining whether or not the generated light projection pattern has suitability as a three-dimensional shape measurement light projection pattern by comparing the calculated pattern suitability index value with a predetermined threshold;
  An evaluation step including:
  Until it is determined that there is appropriateness in the suitability determination step, different projection patterns are generated in step (a), and steps (a) to (d) are repeatedly executed.
  In the random pattern generation method, the light projection pattern determined to be appropriate in the suitability determination step in the evaluation step is used as a light projection pattern for three-dimensional shape measurement.
[0047]
  Furthermore, the third aspect of the present invention provides
  In a distance image generation device that generates a distance image by measuring a three-dimensional shape of the measurement object using images taken from different viewpoints,
  A projection pattern generating means capable of generating a plurality of different aperiodic projection patterns using random numbers;
  A reference camera for capturing a projection pattern image of the measurement target;
  A reference camera that captures a projection pattern image at a different viewpoint from the reference camera;
  Evaluate the pattern suitability of at least one of the projection pattern images captured by the reference camera or the reference camera, and evaluate whether it is suitable as a projection pattern image for generating a distance image. Light projection pattern evaluation meansAs a pattern suitability index value indicating the suitability of the light projection pattern,
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
By calculating one of the above values (1) to (4) and comparing the calculated pattern suitability index value with a predetermined threshold value, the generated light projection pattern becomes a three-dimensional shape measurement light projection pattern. A projection pattern evaluation means for determining whether or not it has aptitude,
  An image-to-image association unit that detects correspondence between the projection pattern images captured by the reference camera and the reference camera and outputs parallax data;
  Distance image generating means for generating a distance image based on the parallax data output by the image correlation means;
  When the projection pattern image obtained based on the generated projection pattern is determined not to be suitable as the projection pattern image for distance image generation by the projection pattern evaluation unit, Generate a periodic flood pattern,
  The inter-image association unit and the distance image generation unit determine that the projection pattern image obtained based on the generated projection pattern is appropriate as the projection pattern image for generating the distance image by the projection pattern evaluation unit. The distance image generating apparatus is characterized in that the processing is executed on the condition of the above.
[0049]
Furthermore, in one embodiment of the distance image generation device of the present invention, the light projection pattern generation means has a configuration for generating different aperiodic light projection patterns by changing a standard deviation s that defines a normal random number. When the projection pattern image obtained based on the generated projection pattern is determined to be not suitable as the projection pattern for distance image generation in the projection pattern evaluation unit, the standard deviation s is changed. It has the structure which produces | generates a new aperiodic light projection pattern.
[0050]
Furthermore, in one embodiment of the distance image generating apparatus of the present invention, the projection pattern generating means has a configuration for generating a binary image obtained by binarizing an image composed of normal random numbers using a threshold T, and By changing the threshold value T, a different aperiodic projection pattern is generated, and a projection pattern image obtained on the basis of the generated projection pattern is used for distance image generation in the projection pattern evaluation unit. When it is determined that there is no suitability as a light projection pattern, a new aperiodic light projection pattern in which the threshold value T is changed is generated.
[0057]
Furthermore, the fourth aspect of the present invention provides
In a distance image generation device that generates a distance image by measuring a three-dimensional shape of the measurement object using images taken from different viewpoints,
This is a projection pattern generation means for generating a pattern to be projected onto a measurement target, and can generate a plurality of different aperiodic projection patterns using random numbers, and also shows a pattern suitability indicating the suitability of the pattern The index value is calculated based on the generated light projection pattern, and the suitability as the distance image generation light projection pattern is determined by comparing the calculated pattern suitability index value with a predetermined threshold value. A projection pattern generating means for setting only a pattern as a projection pattern;
A reference camera for capturing a projection pattern image of the measurement target;
A reference camera that captures a projection pattern image at a different viewpoint from the reference camera;
An image-to-image association unit that detects correspondence between the projection pattern images captured by the reference camera and the reference camera and outputs parallax data;
A distance image generating means for generating a distance image based on the parallax data output by the inter-image association means;
A mismatching area detecting means for detecting a mismatching area in the distance image generated by the distance image generating means;
Have
The projection image generation unit has a configuration for generating a new aperiodic projection pattern when the mismatching region detection unit detects a mismatching region equal to or greater than a predetermined threshold. In the generator.
[0066]
  Furthermore, the fifth aspect of the present invention provides
  In a distance image generation method for generating a distance image by measuring a three-dimensional shape of the measurement object using images taken from different viewpoints of the measurement object,
  A projection pattern generation step for generating a non-periodic projection pattern using random numbers;
  The projection pattern image of the measurement target,With reference cameraApply a reference camera and synchronize from different viewpointsImage capture step to capture,
  Evaluate the pattern suitability of at least one of the projection pattern images captured by the reference camera or the reference camera, and evaluate whether it is suitable as a projection pattern image for generating a distance image. Projection pattern evaluation stepAs a pattern suitability index value indicating the suitability of the light projection pattern,
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
By calculating one of the above values (1) to (4) and comparing the calculated pattern suitability index value with a predetermined threshold value, the generated light projection pattern becomes a three-dimensional shape measurement light projection pattern. Projection pattern evaluation step for determining whether or not there is suitabilityWhen,
  An image-corresponding step for detecting a correspondence between each projection pattern image captured by the reference camera and the reference camera and outputting parallax data;
  A distance image generation step for generating a distance image based on the parallax data output by the inter-image association means;
  In the projection pattern generation step, when it is determined that the projection pattern image obtained based on the generated projection pattern is not suitable as the projection pattern image for distance image generation in the projection pattern evaluation step, Repeatedly generate a periodic flood pattern,
  In the association step between images and the distance image generation step, it is determined that the projection pattern image obtained based on the generated projection pattern is appropriate as the projection pattern image for distance image generation in the projection pattern evaluation step. The distance image generation method is characterized in that the process is executed on the condition.
[0076]
  Furthermore, the sixth aspect of the present invention provides
  In a distance image generation method for generating a distance image by measuring a three-dimensional shape of the measurement object using images taken from different viewpoints of the measurement object,
  A non-periodic projection pattern using random numbers is generated, a pattern suitability index value indicating suitability as a pattern is calculated based on the generated projection pattern, and the calculated pattern suitability index value is a predetermined threshold value A light projection pattern generation step for determining suitability as a light projection pattern for distance image generation by comparing with, and setting only a pattern determined to be suitable as a light projection pattern;
  The projection pattern image of the measurement target,With reference cameraApply a reference camera and synchronize from different viewpointsImage capture step to capture,
  An image-to-image association step of detecting correspondence between each projection pattern image captured by the reference camera and the reference camera and outputting parallax data;
  A distance image generation step for generating a distance image based on the parallax data output in the inter-image association step;
  A mismatching region detection step for detecting a mismatching region in the distance image based on the projection pattern image generated in the distance image generation step;
  Have
  The light projection pattern generation step includes a new aperiodic light projection when a mismatching area equal to or greater than a predetermined threshold is detected in the mismatching area detection step in the distance image obtained based on the generated light projection pattern. A distance image generation method is characterized by generating a pattern.
[0085]
  Furthermore, the seventh aspect of the present invention provides
  A computer program for executing a projection pattern generation process used for executing a three-dimensional shape measurement for measuring a three-dimensional shape of the measurement target using images obtained by photographing the measurement target from different viewpoints on a computer system. A program providing medium provided tangibly, the computer program comprising:
(A) a projection pattern generation step of generating an aperiodic projection pattern using random numbers;
(B) an evaluation step for evaluating the projection pattern generated in the projection pattern generation step;
  (C) A pattern suitability index value indicating the suitability of the projection pattern generated in the projection pattern generation step as the patternAs
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
Any one of the above values (1) to (4)A pattern suitability index value calculating step for calculating based on the generated projection pattern;
  (D) an aptitude determination step for determining whether or not the generated light projection pattern has suitability as a three-dimensional shape measurement light projection pattern by comparing the calculated pattern suitability index value with a predetermined threshold;
  An evaluation step including:
  Until it is determined that there is appropriateness in the suitability determination step, different projection patterns are generated in step (a), and steps (a) to (d) are repeatedly executed.
  And determining a light projection pattern determined to be appropriate in the suitability determination step in the evaluation step as a light projection pattern for three-dimensional shape measurement.
[0086]
The program providing medium according to the seventh aspect of the present invention is a medium that provides a computer program in a computer-readable format to, for example, a general-purpose computer system that can execute various program codes. The form of the medium is not particularly limited, such as a storage medium such as a CD, FD, or MO, or a transmission medium such as a network.
[0087]
Such a program providing medium defines a structural or functional cooperative relationship between a computer program and a providing medium for realizing a function of a predetermined computer program on a computer system. . In other words, by installing a computer program in the computer system via the provided medium, a cooperative action is exhibited on the computer system, and the same effects as the other aspects of the present invention are obtained. Can do it.
[0088]
Other objects, features, and advantages of the present invention will become apparent from a more detailed description based on embodiments of the present invention described later and the accompanying drawings.
[0089]
DETAILED DESCRIPTION OF THE INVENTION
[Random pattern generation apparatus-Example 1]
First, FIG. 2 shows a configuration example of a distance image generation apparatus to which the random pattern generation apparatus of the present invention is applied, and an outline of the distance image generation apparatus will be described. The distance image generation apparatus measures the three-dimensional shape of the measurement target by applying the stereo image method described above, and has a three-dimensional shape, for example, an angle with respect to the measurement target object such as a human face. Images are taken using two cameras arranged at different positions, a so-called reference camera and a reference camera, and the object to be measured is based on the stereo image method described in FIG. The surface shape is measured.
[0090]
The random pattern generated by the random pattern generation device of the present invention is irradiated by the projection pattern generation and irradiation unit 220 of the distance image generation device. This projection pattern is an aperiodic projection pattern based on a uniform random number or a regular random number. The aperiodic projection pattern projected onto the measurement target 200 by the projection pattern generation and irradiation unit 220 is a pattern having no periodicity in, for example, dot size, line length, thickness, position, density, etc. For example, the pattern is as shown in FIGS. 3 (a) and 3 (b). These pattern generation methods will be described in detail later.
[0091]
The non-periodic pattern generated by the random pattern generation device of the present invention is formed in advance on a slide or the like, for example, and the pattern can be projected onto the measurement object by irradiating flash light through the slide. Alternatively, a random pattern can be dynamically generated and projected using a transmissive liquid crystal display element or the like. The dynamic random pattern generation method will be described in detail later. By capturing an aperiodic pattern formed on the surface of the measurement target 200 together with the measurement target, it is possible to easily associate the image data of the reference camera and the reference camera when generating the distance image.
[0092]
The description of the configuration of the distance image generation device of the present invention will be continued. The reference camera 201 and the reference camera 202 are arranged in different line-of-sight directions with respect to the measurement object 200, and take images of the respective line-of-sight directions, that is, the measurement object 200 onto which a random pattern is projected. Images taken by the reference camera 201 and the reference camera 202 are taken at a synchronized timing. Images captured from the reference camera 201 and the reference camera 202 are stored in the frame memories 203 and 204 via the A / D converters, respectively.
[0093]
The inter-image association unit 205 performs association processing between the projection pattern images of the base camera and the reference camera of the frame memories 1a and 203 and the frame memories 2a and 204, respectively. That is, the process of associating the pixels of the reference image captured by the reference camera 202 with the pixels of the image captured by the standard camera 201 is performed, and the parallax data of the reference image for the standard image is output to the distance image generation unit 206.
[0094]
The projection pattern generated by the projection pattern generation and irradiation unit 220 is a non-periodic pattern as described above, and the images captured by the cameras can be associated with each other relatively easily. The distance image generation unit 206 generates a distance image based on the parallax data generated by the inter-image association unit 205 and stores it in the image memory 207. The distance image stored in the image memory 207 is output via the output unit 208.
[0095]
Note that the processing in each block having the configuration shown in FIG. 2 and inter-block data transfer control are controlled by a control means (not shown) and stored in a predetermined memory, for example, a semiconductor memory such as RAM or ROM, a magnetic disk, an optical disk, or the like. It can be controlled by a control program recorded on the medium. In addition, the control mode can be changed by a user inputting a command, a parameter, or the like using an input unit (not shown).
[0096]
FIG. 4 shows a specific configuration example of the light projection pattern generation / irradiation means 220, that is, the configuration of a light projection device for an image pattern composed of random point groups, and the relationship with the image photographing device.
[0097]
Generally, when projecting a texture image pattern, the brightness of the image becomes a problem. Here, in order to obtain a sufficient amount of light, a strobe projection system is used as a light source. On the other hand, on the photographing apparatus side, the CCD accumulation time is controlled by adjusting the electronic shutters of the reference camera 402 and the reference camera 403 in order to suppress the influence of background light and improve the contrast of the projection pattern. Further, by sharing the control unit 401 between the light projecting device and the light projecting pattern generation / irradiation unit 220, the reference camera 402 is instantly used when the texture image pattern is projected by the light projection pattern generation / irradiation unit 220. The reference camera 403 captures the measurement object at the same time. Images captured by each camera are separately stored in the image memory 404 (corresponding to the frame memories 203 and 204 in FIG. 2), and are used for association processing between images and distance image generation processing.
[0098]
The slide in the projection pattern generation and irradiation unit 220 shown in FIG. 4 is a slide on which an aperiodic pattern is recorded. As described above, the dot size, line length, thickness, position, density, etc. have no periodicity, so-called non-periodic projection patterns, for example, patterns as shown in FIGS. Is a recorded slide.
[0099]
The details of the non-periodic projection pattern generation processing configuration will be described below. FIG. 5 is a block diagram showing a functional configuration of the random pattern generation device of the present invention, and FIG. 6 shows a processing flow for generating a random texture pattern that is an aperiodic projection pattern in the random pattern generation device of the present invention. Hereinafter, the random pattern generation device of the present invention will be described with reference to FIGS. 5 and 6.
[0100]
Based on the uniform random number generated by the uniform random number generating means 501 in FIG. 5, the normal random number generating means 502 generates a normal random number Rk based on a preset standard deviation s and average value am. The standard deviation s can be set to various values in the standard deviation setting means 507, and various normal random numbers can be generated in the normal random number generation means 502 based on the set standard deviation.
[0101]
The image generation unit 503 generates an image rd_im [k] based on the normal random number Rk generated by the normal random number generation unit 502.
[0102]
Next, the binary image generation unit 504 performs binarization processing of the image rd_im [k] with a predetermined threshold T, and generates a binary image f (i, j). Various values can be set for the threshold value T in the threshold value setting means 508, and various binary images can be generated in the binary image generation means 504 based on the set threshold value.
[0103]
Next, the change rate detection unit 505 obtains the change rate r of the binary image f (i, j) generated by the binary image generation unit 504, and the change rate determination unit 506 further determines the change rate detection unit 505. A comparison between the detected change rate r and a predetermined threshold value T1 is executed. The rate of change r is a value that increases when the generated binary image has no periodicity, and decreases when the periodicity has a periodicity, for example, when the pixel value includes repetition such as 010101, etc. It is an index value. The calculation formula for the rate of change r will be described in detail in the description of the flow of FIG.
[0104]
In the comparison process with the threshold value T1 of the change rate determination unit 506, when the change rate r exceeds the threshold value T1, it is determined that the random pattern has sufficient aperiodicity, and the binary image generation unit 504 is determined. The binary image generated in (1) is used as a projection pattern.
[0105]
Next, more specific processing will be described according to the flow of FIG. In step 601 of FIG. 6, a uniform random number generation process between 0 and 1 is performed and stored in the array a [n]. These are uniform random numbers that are either binary (0 or 1) or gray values (can have values between 0 and 1).
[0106]
In step 602, processing for generating a normal random number (gauss) having a standard deviation s and an average value am is performed. The normal random number Rk is generated based on the 0-1 uniform random number according to the following equation.
[0107]
[Expression 1]

[0108]
Next, in step 603, a random texture image rd_im [k] based on normal random numbers is generated. Here, k = 0 to K−1, rd_im [k] = Rk, and K = XSIZE × YSIZE (image size).
[0109]
Further, in step 604, binarization processing of the random texture image rd_im [k] based on the normal random number generated in step 603 is performed, which is executed based on a predetermined threshold T. The binary image obtained as a result is defined as f (i, j).
[0110]
Next, in step 605, a change rate r as a pattern suitability index value of the binary image f (i, j) is calculated. The change rate r is calculated based on the following equation.
[0111]
[Expression 2]

[0112]
In the above formula, A is a normalization coefficient.
[0113]
Next, in step 606, a comparison process between the rate of change r and a predetermined threshold value T1 is performed. If the rate of change r is greater than the threshold value T1, it is determined that the binary image has sufficient randomness. In step 607, the projection pattern image generation process is terminated.
[0114]
If the determination in step 606 is No, that is, if it is determined that the rate of change r is not greater than the predetermined threshold value T1, it is determined that the generated binary image includes a periodic component that is inappropriate for light projection, and step Proceeding to step 608, the standard deviation s or at least one of the threshold values T is changed, and the processing of step 602 and the following steps and step 604 and the subsequent steps is repeated to obtain a new binary image f (i, j). Is generated.
[0115]
Finally, at step 606, when the rate of change r> threshold value T1 becomes Yes, it is determined that the binary image is a binary image having sufficient randomness, and the process proceeds to step 607 to project the light projection pattern image. The generation process ends.
[0116]
As described above, according to the random pattern generation device and the random pattern generation method of the present invention described with reference to FIGS. 5 and 6, the threshold value for which the rate of change r of the binary image generated based on the uniform random number is determined in advance. Since it is configured to generate a random image pattern for projection by guaranteeing that it is larger than T, periodic components that cause problems in stereo image processing, that is, cause mismatching, are sufficiently removed. A good random pattern can be generated.
[0117]
[Random pattern generator-other embodiments]
Hereinafter, different aspects of the binary image pattern evaluation method in the random pattern generation apparatus and random pattern generation method of the present invention will be described with reference to FIGS. 7, 8, and 9. The evaluation of the generated binary image pattern in FIGS. 5 and 6 is performed by calculating a change rate r as a pattern suitability index value from the entire generated binary image, as shown in steps 605 and 606 in FIG. The change rate r is compared with the threshold value T1. 7 to 9, a partial area is extracted from the binary image f (x, y), the pattern suitability index value for the partial area is determined based on the change rate, the feature amount, and the determination process based on pattern matching. It is the aspect which performs.
[0118]
First, a random image pattern evaluation method using a pattern change rate parameter by extracting a pattern of a partial region (window) from a binary image f (x, y) will be described with reference to FIG.
[0119]
In step 701 in FIG. 7, a binary image f (x, y) having a size of XSIZE × YSIZE is acquired. For example, the random pattern shown in the upper right of FIG.
[0120]
In step 702, the size of the window to be applied as the evaluation reference size: Nx × Ny is set. For example, it is an Nx × Ny window shown in the lower right of FIG. In step 703, coordinates xm and ym for setting the window position are set. These coordinates are set as x coordinate value: xm = Nx / 2 and y coordinate value: ym = Ny / 2.
[0121]
Next, in step 704, a window change rate γm defined by (xm, ym) is obtained according to the following equation.
[0122]
[Equation 3]

[0123]
Next, in step 705, the change rate γm as the window pattern suitability index value defined by (xm, ym) is compared with a predetermined threshold T2, and the change rate γm is greater than the threshold T2. If so, it is determined that the window is a binary image having sufficient randomness. In step 706, it is determined whether there is a window area to be compared. If there is, the process proceeds to step 708. , Xm, ym are updated, and a new window area change rate is calculated and compared with a threshold value.
[0124]
If the determination in step 705 is No, that is, if it is determined that the rate of change γm is not greater than the predetermined threshold value T2, it is determined that the window in the generated binary image includes a periodic component that is inappropriate for light projection. Then, the process proceeds to step 707, where it is determined that the projection pattern is inappropriate, and in step 709, the evaluation process is terminated.
[0125]
When it is determined that the projection pattern is inappropriate as described above, if a new random pattern is required, the standard deviation s or the threshold value T is changed in step 608 of FIG. 6 described above. Thus, a process for generating a new binary image f ′ (x, y) is performed.
[0126]
In the flow of FIG. 7, when all the evaluation determinations in Step 705 are Yes and the evaluation window is completed, the process proceeds to Step 709, where the evaluation is completed, and the generated binary image f (x, y) is the projection pattern. As appropriate.
[0127]
Next, a method for evaluating a random image pattern using a pattern feature value parameter by extracting a pattern of a partial region (window) from the binary image f (x, y) will be described with reference to FIG.
[0128]
In step 801 of FIG. 8, a binary image f (x, y) having a size of XSIZE × YSIZE is acquired. For example, the random pattern shown in the upper right of FIG.
[0129]
In step 802, a window size: Nx × Ny to be applied is set as the evaluation reference size. For example, an Nx × Ny window shown in the lower right of FIG. In step 803, coordinates xm and ym for setting the position of the window are set. These coordinates are set as x coordinate value: xm = Nx / 2 and y coordinate value: ym = Nn / 2.
[0130]
Next, in step 804, a window feature matrix R defined by (xm, ym) is obtained according to the following equation.
[0131]
[Expression 4]

[0132]
Next, in step 805, eigenvalues: λm1 and λm2 of the feature matrix R are obtained, and in step 806, the pattern suitability index obtained from the eigenvalues of the window feature matrix R defined by (xm, ym): λm1, λm2. The feature amount as a value: λ = min (λm1, λm2) is compared with a predetermined threshold value λT, and if the feature amount: λ is larger than the threshold value λT, the window has sufficient randomness. In step 807, it is determined whether or not there is a window area to be compared. If so, the process proceeds to step 809, where xm and ym are updated to obtain a new window area. The feature amount is calculated and compared with a threshold value.
[0133]
If the determination in step 806 is No, that is, if it is determined that the feature amount λ is not larger than the predetermined threshold λT, the window in the generated binary image includes a periodic component that is inappropriate for light projection. In step 808, it is determined that the light projection pattern is inappropriate. In step 810, the evaluation process ends.
[0134]
When it is determined that the projection pattern is inappropriate as described above, if a new random pattern is required, the standard deviation s or the threshold value T is changed in step 608 of FIG. 6 described above. Thus, a process for generating a new binary image f ′ (x, y) is performed.
[0135]
In the flow of FIG. 8, when all the evaluation determinations in step 806 are Yes and the evaluation window is completed, the process proceeds to step 810, the evaluation is completed, and the generated binary image f (x, y) is the projection pattern. As appropriate.
[0136]
Next, referring to FIG. 9, a method of extracting a plurality of partial region (window) patterns from the binary image f (x, y) and evaluating a random image pattern using a matching score between windows. explain. In the processing flow of FIG. 7 and FIG. 8 described above, the image characteristics in one partial area are evaluated. Here, it is determined whether there is a similar pattern between different partial areas, and the similarity is determined. When it is determined that there is a pattern, it is determined that the light projection pattern is inappropriate.
[0137]
In step 901 of FIG. 9, a binary image f (x, y) having a size of XSIZE × YSIZE is acquired.
[0138]
In step 902, the size of the window to be applied as the evaluation reference size: Nx × Ny is set. For example, it is an Nx × Ny window shown in the lower right of FIG. In this case, the windows to be compared are also set as windows of the same size. In step 903, coordinates xm and ym for setting the position of the window are set. These coordinates are set as x coordinate value: xm = Nx / 2 and y coordinate value: ym = Nn / 2.
[0139]
Next, in step 904, the inter-region correlation value between the patterns P1 (i, j) and P2 (i, j) of the two windows in the image is obtained as a pattern suitability index value according to the following equation.
[0140]
[Equation 5]

[0141]
Next, in step 905, the correlation value rm as the pattern suitability index value obtained according to the above equation is compared with a predetermined threshold value T3. If the correlation value rm is smaller than the threshold value T3, step 904 is executed. In step 906, it is determined whether or not there is a window area to be compared. In step 908, the two windows compared in step 908 are determined to be images having no inappropriate similar pattern. Then, xm and ym are updated, a new window area is set, and the pattern matching process in step 904 and subsequent steps is performed.
[0142]
When the determination in step 905 is No, that is, when it is determined that the correlation value rm is not smaller than the predetermined threshold value T3, it is determined that the two windows compared in step 904 have inappropriate similar patterns, and step 907 In step 909, the evaluation process is terminated.
[0143]
When it is determined that the projection pattern is inappropriate as described above, if a new random pattern is required, the standard deviation s or the threshold value T is changed in step 608 of FIG. 6 described above. Thus, a process for generating a new binary image f ′ (x, y) is performed.
[0144]
In the flow of FIG. 9, when all the evaluation determinations in step 905 are Yes and the evaluation window is completed, the process proceeds to step 909, where the evaluation is completed, and the generated binary image f (x, y) is the projection pattern. As appropriate.
[0145]
In the above, an example in which a pattern based on a binary image is projected has been described. However, an example of pattern generation processing and a pattern evaluation method in the case of projecting a light and shade pattern instead of a binary image will be described below.
[0146]
First, FIG. 10 shows a processing flow for generating a random pattern using a grayscale image. Each step of the flow of FIG. 10 will be described. First, in step 1001 and step 1002, the initial values of the coordinate values of the image area for generating the light and shade pattern are set as i = 0 and j = 0.
[0147]
Next, in step 1003, a uniform random number between 0 and 1 is generated and stored as an array a [n]. Further, in step 1004, a normal random number R having a standard deviation s and an average value am is generated based on the uniform random number array a [n] generated in step 1003, and corresponds to the set coordinates (i, j). The grayscale data is set as R = f (i, j). This is the gray value of the coordinates (i, j).
[0148]
In step 1005 and step 1006, the coordinates where the gray value is set are compared with the set size of the projection pattern image to determine the presence / absence of the gray value non-set area. If there is an unset area, the coordinates are updated in step 1007 and step 1008, respectively, and the processing after step 1003 is repeatedly executed to set the gray value of each coordinate based on a random number. When the gray values of all the coordinates included in the area of XSIZE × YSIZE are determined, the process proceeds to step 1009, and the process of generating the light projection pattern composed of the gray data is completed.
[0149]
Various methods can be applied as a random number generation method. The method shown in the flow of FIG. 10 is merely an example, and the random number generation method applied to the present invention is not limited to the method shown in FIG.
[0150]
Since the random pattern generated according to the flow of FIG. 10 has a gray value instead of a binary image pattern, it is necessary to evaluate the suitability of the pattern as a gray pattern image.
[0151]
FIG. 11 shows an evaluation process flow of a light projection pattern having a gray value. 11 extracts, for example, a partial area (window) pattern shown in the lower right of FIG. 11 from the gray image f (x, y) shown in the upper right of FIG. 11 to obtain a dispersion value, and evaluates the gray image pattern. It shows how to do.
[0152]
In step 1101 of FIG. 11, a grayscale image f (x, y) having a size of XSIZE × YSIZE is acquired. For example, the shading pattern shown in the upper right of FIG.
[0153]
In step 1102, a window size to be applied as the evaluation reference size: Nx × Ny is set. For example, it is an Nx × Ny window shown in the lower right of FIG. In step 1103, coordinates xm and ym for setting the position of the window are set. These coordinates are set as x coordinate value: xm = Nx / 2 and y coordinate value: ym = Ny / 2.
[0154]
Next, in step 1104, a dispersion value σm in the window defined by (xm, ym) is obtained according to the following equation.
[0155]
[Formula 6]

[0156]
Next, in step 1105, the variance value σm as the pattern suitability index value indicating the suitability as the window pattern defined by (xm, ym) is compared with a predetermined threshold value σT to perform dispersion. If the value σm is larger than the threshold σT, it is determined that the window is a gray image having sufficient randomness, and it is determined in step 1106 whether there is a window area to be compared. In step 1108, xm and ym are updated, and a new dispersion value of the window area is calculated and compared with a threshold value.
[0157]
If the determination in step 1105 is No, that is, if it is determined that the variance value σm is not greater than the predetermined threshold σT, it is determined that the window in the generated grayscale image is inappropriate for light projection, and step 1107 In step 1109, the evaluation process is terminated.
[0158]
Thus, when it is determined that the light projection pattern is inappropriate, if a new random pattern is necessary, the standard deviation s is changed in step 1004 of FIG. Grayscale image generation processing can be performed.
[0159]
In the flow of FIG. 11, when all the evaluation determinations in step 1105 are “Yes” and the evaluation window is completed, the process proceeds to step 1109, where the evaluation is completed, and the generated grayscale image f (x, y) is used as the projection pattern. Judge that it is appropriate.
[0160]
FIG. 12 is a diagram showing an example in which a distance image is obtained from two observed images by projecting the projection image pattern generated by the above-described method onto a target (plane). FIG. 12A shows a light projection image pattern composed of random point groups generated according to the above-described random pattern generation method of the present invention. FIGS. 12B and 12C are images observed by a camera 1 and a camera 2 (for example, a standard camera and a reference camera), respectively, by projecting the pattern onto a plain wall as a measurement target.
[0161]
FIG. 12D is a distance image of the plain wall that is the measurement target obtained as a result. This distance image uses a pattern in an arbitrary Nx × Ny window in the reference camera image, and is associated with an epipolar line in the reference camera image, that is, similar to a pattern in an arbitrary Nx × Ny window in the reference camera image. This is a distance image obtained based on the parallax data obtained by obtaining parallax data by determining in which part on the epipolar line of the reference camera image the pattern appears. In the distance image of FIG. 12 (d), uniform distance data is reflected on the plain wall that is the measurement target, and this result shows that the random image generated according to the random pattern generation method of the present invention is measured even when the measurement target is a plain wall. It shows that the distance to the camera is accurately measured by projecting a projection image pattern made up of point clouds.
[0162]
FIG. 13 is a conceptual diagram schematically showing an example of a distance image measurement result obtained from a stereo image observed by a reference camera and a reference camera by projecting an image pattern composed of the above-mentioned random point cloud onto a face. It is.
[0163]
FIG. 13A shows an image pattern for projection composed of random point groups generated according to the above-described random pattern generation method of the present invention. FIG. 13B is a schematic diagram in which the pattern is projected onto the face to be measured.
[0164]
FIGS. 13C and 13D are images obtained by projecting a projection image pattern onto the face to be measured and observed by the reference camera and the reference camera, respectively.
[0165]
FIG. 13E shows a distance image of the face to be measured as a result. This distance image is a conceptual diagram that is abstractly shown for explanation, and is different from an actual image, but when a projection image pattern composed of random point groups generated according to the random pattern generation method of the present invention is projected In the corresponding point marking process, the possibility of mismatching is reduced, and an accurate distance is required on almost all surfaces of the measurement target. That is, it is possible to obtain a highly accurate distance image in which shades corresponding to the distance from the reference camera to each part of the face to be measured are shown. By using the random pattern generation method of the present invention, mismatching occurs in the matching process between the reference camera image and the reference camera image even if the measurement object is an object whose feature is not clear, such as a human face. The possibility is reduced, and the distance image (or three-dimensional shape) can be generated with high accuracy.
[0166]
In the above-described embodiment, a configuration has been described in which two cameras are used to capture an image, and a distance image is obtained from two projection pattern images. However, a multi-baseline stereo system using three or more cameras is further described. It is also possible to apply this embodiment in (for example, disclosed in JP-A-10-289315). Also, the random pattern generation apparatus and method of the present invention can be applied to a case where stereo shooting is performed using a camera group in which each unit can be moved freely instead of a fixed camera unit group. is there.
[0167]
[Distance Image Generating Device—Embodiment 1]
Next, a distance image generation device and a distance image generation method to which the above-described random pattern generation device and method are applied will be described.
[0168]
FIG. 14 is a block diagram showing the configuration of the distance image generating apparatus according to the present invention. The distance image generation apparatus measures the three-dimensional shape of the measurement target by applying the stereo image method described above, and has a three-dimensional shape, for example, an angle with respect to the measurement target object such as a human face. Images are taken using two cameras arranged at different positions, a so-called reference camera and a reference camera, and the object to be measured is based on the stereo image method described in FIG. Measure the surface shape.
[0169]
The random pattern generation device of the present invention described above corresponds to the light projection pattern generation device 1211 in FIG. The random pattern generated by the light projection pattern generation device 1211 is irradiated by the light projection pattern irradiation unit 1212. This projection pattern is an aperiodic projection pattern based on a uniform random number or a regular random number.
[0170]
The light projection pattern evaluation unit 1210 evaluates at least one of the light projection pattern images photographed by the standard camera 201 and the reference camera 204. This image evaluation is performed by using the configuration of FIG. 5 described in the description of the random pattern generation device described above and the evaluation pattern in FIGS. 6 to 9 for the projection pattern image captured by the reference camera 201 or the reference camera 204. It is applied to and executed. Alternatively, a grayscale image can be used as the projection pattern, and generation and evaluation of the projection pattern may be executed according to the processes of FIGS. 10 and 11.
[0171]
That is, for example, when the light projection pattern evaluation unit 1210 evaluates the light projection pattern image captured by the reference camera 201 in FIG. 14 according to the evaluation flow in FIG. 6, the images from the frame memories 1a and 203 are displayed in FIG. The captured image p (i, j) or the binary image p ′ (i, j) obtained by processing the captured image p (i, j) with a predetermined threshold value Tp in accordance with step 605 of the evaluation flow of The change rate r is calculated as a pattern suitability index value indicating the suitability of the pattern, and a comparison with the threshold value in step 606 is executed to determine whether or not the light projection pattern is appropriate.
[0172]
When the projection pattern image photographed by the standard camera 201 or the reference camera 204 is evaluated according to the evaluation flow of FIG. 7 described above, the captured image p (i, j) or the captured image p is used. The following steps 702 to 709 shown in FIG. 7 are executed with the binary image p ′ (i, j) obtained by processing (i, j) with a predetermined threshold Tp as a binary image shown in step 701. Then, the change rate rm of each window is calculated as a pattern suitability index value indicating the suitability of the pattern, and evaluation is performed.
[0173]
When the projection pattern image photographed by the standard camera 201 or the reference camera 204 is evaluated according to the evaluation flow of FIG. 8 described above, the captured image p (i, j) or the captured image p is used. The following steps 802 to 810 shown in FIG. 8 are executed with the binary image p ′ (i, j) obtained by processing (i, j) with a predetermined threshold Tp as a binary image shown in step 801. Thus, the eigenvalue of the feature matrix R of each window is obtained as a pattern suitability index value indicating the suitability of the pattern, and this is compared with a threshold value for evaluation.
[0174]
When the projection pattern image photographed by the standard camera 201 or the reference camera 204 is evaluated according to the evaluation flow of FIG. 9 described above, the captured image p (i, j) or the captured image p is used. The following steps 902 to 909 shown in FIG. 9 are executed with the binary image p ′ (i, j) obtained by processing (i, j) with a predetermined threshold Tp as a binary image shown in step 901. Then, a matching rate between different windows is obtained, and this is evaluated by comparing it with a threshold value as a pattern suitability index value indicating the suitability of the pattern.
[0175]
In addition to the above, evaluation using the dispersion value of the pattern image (see FIG. 11) is also possible. Further, when a grayscale image is used as the projection pattern, the projection is performed according to the processing of FIGS. A light pattern is generated and an evaluation process is executed.
[0176]
The light projection pattern evaluation unit 1210 may be configured to evaluate either the captured image of the standard camera 201 or the captured image of the reference camera 202, or may be configured to evaluate both images.
[0177]
If it is determined by the above-described various evaluation methods that the projection pattern is inappropriate, the projection pattern generation unit 1211 in FIG. 14 stops using the pattern and selects or generates a new projection pattern. And irradiate the object to be measured.
[0178]
The projection pattern generation device 1211 has a plurality of projection pattern slides in which different non-periodic patterns are formed in advance on a slide, etc., and sequentially selects and irradiates a projection pattern from the plurality of projection patterns, and evaluates the projection pattern. What is recognized as an appropriate projection pattern in the means 1210 is selected as a final projection pattern, that is, a pattern for generating a distance image. Alternatively, the light projection pattern irradiating unit 1212 may have a transmissive liquid crystal display element, and the light projection pattern generation device 1211 may control the transmissive liquid crystal display element to dynamically generate and project a random pattern. The dynamic random pattern generation method will be described in detail with reference to FIG.
[0179]
If the projection pattern evaluation unit 1210 recognizes that the projection pattern is appropriate, the result is notified to the image association unit 205, and the projection pattern is photographed by the reference camera 201 and the reference camera 202 together with the measurement target in the projection pattern. Then, based on the images stored in the frame memories 203 and 204, the inter-image association unit 205 performs association processing between the projection pattern images. That is, the pixel image of the reference image captured by the reference camera 202 is associated with each pixel of the image captured by the standard camera 201, and the parallax data of the reference image with respect to the standard image is output to the distance image generation unit 206. The generation unit 206 executes a distance image generation process.
[0180]
In the distance image generation apparatus of the present invention, the light projection pattern used when the distance image is generated is generated or selected by the light projection pattern generation means 1211 and is recognized as appropriate by the light projection pattern evaluation means 1210. Therefore, the occurrence of mismatching is suppressed in the image association process, the image association process in the image association unit 205, and the distance image generation process in the distance image generation unit 206. Is performed with high accuracy.
[0181]
The processing in each block having the configuration shown in FIG. 14 and inter-block data transfer control are controlled by a control means (not shown) and stored in a predetermined memory, for example, a semiconductor memory such as a RAM or a ROM, a magnetic disk, an optical disk, or the like. It can be controlled by a control program recorded on the medium. In addition, the control mode can be changed by a user inputting a command, a parameter, or the like using an input unit (not shown).
[0182]
Next, a configuration block diagram in which the projection pattern irradiation unit 1212 has a transmissive liquid crystal display element, and the projection pattern generation device 1211 dynamically generates a random pattern by controlling the transmissive liquid crystal display element and projects it. As shown in FIG.
[0183]
As shown in FIG. 15, the light projection pattern irradiating means 1212 has a transmissive liquid crystal display element 1320, and the measurement target 200 is irradiated with light from the light source 1321 through the transmissive liquid crystal display element 1320.
[0184]
The projection pattern generation unit 1211 includes a liquid crystal control unit 1307, and the liquid crystal control unit 1307 executes control for forming various non-periodic patterns on the transmissive liquid crystal display element 1320. The light projection pattern generation unit 1211 has the same configuration as the configuration units 501 to 504, 507, and 508 of FIG. 5 used in the description of the random pattern generation device, and generates, for example, a binary uniform random number. Uniform random number generating means 1301, normal random number generating means 1302 for generating a normal random number Rk based on a preset standard deviation s and an average value am based on the uniform random number generated in the uniform random number generating means 1301; An image generation unit 1303 that generates an image rd_im [k] based on the normal random number Rk generated by the normal random number generation unit 1302, performs a binarization process on the image rd_im [k] using a predetermined threshold T, and outputs a binary image Binary image generation means 1304 for generating f (i, j) is included.
[0185]
The liquid crystal control unit 1307 controls the transmissive liquid crystal display element 1320 so as to form a pattern corresponding to the binary image generated by the binary image generation unit 1304.
[0186]
The standard deviation s can be set to various values in the standard deviation setting means 1305, and various normal random numbers can be generated in the normal random number generation means 1302 based on the set standard deviation. Various values can be set in the means 1306, and various binary images are generated in the binary image generation means 1304 based on these various setting values.
[0187]
On the other hand, the projection pattern evaluation unit 1210 receives the captured image of the standard camera 201 and the captured image of the reference camera 202 via the frame memories 1a and 203 and the frame memories 1a and 204, and the captured image change rate detection unit 1308. The pattern of the change rate r of the captured image p (i, j) or the binary image p ′ (i, j) obtained by processing the captured image p (i, j) with a predetermined threshold value Tp in FIG. It is calculated as a pattern suitability index value indicating the suitability. Further, the change rate determination unit 1309 performs a comparison process with a predetermined threshold value T1 to determine whether the light projection pattern is appropriate.
[0188]
The determination result in the change rate determination unit 1309 is fed back to the projection pattern generation unit 1211. When the comparison result is determined to be an inappropriate projection pattern, the threshold value T setting unit 1306 of the projection pattern generation unit 1211 is used. Alternatively, a new threshold value T or standard deviation s is set in the standard deviation s setting means 1305, and a new binary image is generated. A pattern based on the new binary image is formed on the transmissive liquid crystal display element 1320 in the projection pattern irradiation unit 1212 under the control of the liquid crystal control unit 1307.
[0189]
When it is determined that the comparison result in the change rate determination unit 1309 is an appropriate light projection pattern, although not shown in FIG. 15, the image association unit 205 is notified as described in FIG. The association process based on the image data stored in the frame memory is started.
[0190]
In FIG. 15, the light projection pattern evaluation unit 1210 employs the image evaluation configuration of FIGS. 5 and 6 in the description of the random pattern generation device described above, but is not limited to this evaluation configuration. Any one of the evaluation based on the change rate rm of each window described in the above, the evaluation based on the eigenvalue of the feature matrix of each window described in FIG. 8, and the evaluation configuration based on the matching rate between the two windows described in FIG. It is also possible.
[0191]
FIG. 16 shows a light projection pattern determination processing flow in the distance image generation apparatus of the present invention. Hereinafter, the processing of each step of FIG. 16 will be described with reference to FIGS. 14 and 15.
[0192]
Step 1401 is a step of setting a light projection pattern to be projected onto the measurement target 200. When the light projection pattern generation unit 1211 has a slide in which a plurality of non-periodic patterns are formed in advance, one slide is selected as the light projection pattern. When the light projection pattern generation unit 1211 includes the transmissive liquid crystal display element 1320, a pattern based on the binary image generated by the binary image generation unit 1304 shown in FIG. Control is performed by liquid crystal control means 1307 so as to form 1320.
[0193]
Step 1402 is a step of irradiating the projection pattern on the measuring object 200 and photographing using the standard camera 201 and the reference camera 202. Images taken by each camera are stored in the frame memories 1a and 203 and the frame memories 2a and 204.
[0194]
Step 1403 is a projection pattern evaluation process step based on the photographed image. The light projection pattern evaluation unit 1210 executes a light projection pattern evaluation process based on the photographed image. These evaluation processes are performed by applying the above-described evaluation process configuration shown in FIGS.
[0195]
Step 1404 is a step of determining whether or not there is an inappropriate part as the light projection pattern in the evaluation process in the light projection pattern evaluation unit 1210. If there is no inappropriate part, the process proceeds to Step 1405, where The association unit 205 and the distance image generation unit execute an image correlation process and a distance image generation process based on the captured image.
[0196]
If the determination in step 1404 is No, that is, if it is determined that there is an inappropriate portion as the light projection pattern in the evaluation process in the light projection pattern evaluation unit 1210, the process proceeds to step 1406 to generate or select a new light projection pattern. Processing is performed, and further, processing after step 1401, that is, irradiation of a new projection pattern, photographing, and evaluation processing are executed.
[0197]
An actual image acquisition example in the distance image generation apparatus of the present invention is shown in FIG. In FIG. 17, a pattern is preliminarily attached to the measurement target. When the measurement target with this pattern is irradiated with a non-periodic projection pattern, the result of the measurement overlaps the original pattern of the measurement target and the projection pattern. This shows an example in which the association processing becomes difficult due to the occurrence of the above.
[0198]
FIG. 17A shows a pattern attached to the measurement target, and FIG. 17B shows the first light projection pattern. Although the light projection pattern itself is aperiodic, this aperiodic pattern is projected onto the measurement object (c), and when this is photographed, the original pattern of the measurement object and the light projection pattern as shown in (d) As a result, the matching process becomes difficult. The projection pattern evaluation unit shown in FIG. 14 or 13 performs the evaluation of the image shown in FIG. 17D by the methods described with reference to FIGS.
[0199]
Since the evaluation of the image shown in FIG. 17D is an evaluation that the image is inappropriate as the light projection pattern, the light projection pattern generation unit 1211 generates a new light projection pattern (e) to measure the measurement target. (F) to obtain the result (g). Evaluation processing is also performed on this image (g), and when it is determined that the result of the evaluation is appropriate, association processing and distance image generation processing are started.
[0200]
As described above, the distance image generation apparatus of the present invention evaluates and examines the projection pattern based on the captured image, and the distance image of the appropriate projection pattern irradiation image in which the occurrence of mismatching is suppressed. Since it can be generated, a more accurate distance image can be obtained.
[0201]
In addition, an appropriate light projection pattern is selected before performing a corresponding point marking process and a distance image generation process for an inappropriate light projection pattern, that is, a light projection pattern image in which mismatching frequently occurs in the association process. Therefore, it is possible to obtain a highly accurate distance image by a single process without the necessity of repeatedly performing complicated corresponding point marking processing and distance image generation processing.
[0202]
In the above-described embodiment, a configuration has been described in which two cameras are used to capture an image, and a distance image is obtained from two projection pattern images. However, a multi-baseline stereo system using three or more cameras is further described. The present embodiment can also be applied in (for example, disclosed in JP-A-10-289315), and the above-described evaluation may be performed on one or a plurality of images captured by each camera. .
[0203]
In addition, the configuration of the present invention can be applied to a case where stereo shooting is performed using a camera group in which each unit can be moved freely instead of a fixed camera device group. Evaluation of a captured image can be easily executed, and it becomes easy to set a camera at an arbitrary position.
[0204]
[Distance Image Generation Device—Embodiment 2]
The configuration of the above-described first embodiment of the distance image generation device is a configuration in which the evaluation of the acquired light projection pattern image is executed before the distance image generation, but the distance image is actually generated and the presence / absence of the mismatch region is determined. A configuration for generating a new light projection pattern based on the determination will be described as a second embodiment.
[0205]
The distance image is image data in which, for example, a distance value between the reference camera and the measurement target is reflected as a gray value on each pixel of the captured image, and is used for generating three-dimensional graphic data. In the first embodiment described above, the suitability of the projection pattern is determined before the distance image generation step, but it is guaranteed that there is no possibility that a mismatching region is detected in the actual distance image. Therefore, in the second embodiment, when a mismatching area is detected after the distance image is generated, a new light projection pattern is irradiated to obtain a highly accurate distance image in which the mismatching area is reduced. provide. Note that the mismatching region is a pixel region that does not indicate an accurate distance in the distance image, and occurs when the inter-image association processing is not accurately executed.
[0206]
The configuration of Example 2 is shown in FIG. The difference from the first embodiment is a mismatching region detection unit 1601. The mismatching region detection unit 1601 acquires a distance image from the image memory 207 that stores the distance image generated by the distance image generation unit 206, and makes a mistake. Detect matching regions.
[0207]
When a mismatching area is detected in the distance image by the mismatching area detection unit 1601, the projection pattern generation unit 1602 is notified, and a new projection pattern is generated or selected, and the projection pattern irradiation unit 1603 A new projection pattern is irradiated. The configuration of the light projection pattern irradiation unit 1603 is the same as that of the first embodiment, and a description thereof is omitted.
[0208]
The light projection pattern generation unit 1602 has a configuration for generating a pattern to be projected onto the measurement target, and can generate a plurality of different aperiodic light projection patterns using random numbers. Furthermore, it has the same evaluation configuration as the light projection pattern evaluation means in the first embodiment, and performs determination based on the pattern suitability index value indicating the suitability of the generated light projection pattern as a pattern, and only those having good judgment. Set as projection pattern.
[0209]
That is, the pattern suitability index value indicating the suitability as the pattern of the generated light projection pattern, for example, the above-described change rate, feature amount, variance value, etc. is calculated, and the pattern suitability index value indicating the suitability as the calculated pattern Is compared with a predetermined threshold value to determine the suitability as the distance image generation light projection pattern, and only the pattern determined to be suitable is set as the light projection pattern. Therefore, the projection pattern generated by the projection pattern generation unit 1602 and irradiated by the projection pattern irradiation unit 1603 is basically a projection pattern that hardly causes periodicity.
[0210]
The configuration of the second embodiment will be described below with a focus on the processing of the mismatching area detection unit 1601. FIG. 19 shows a processing flow of the distance image generating apparatus of this embodiment.
[0211]
Step 1701 is a step of setting a light projection pattern to be projected onto the measurement target 200. When the light projection pattern generation unit 1602 has a slide in which a plurality of non-periodic patterns are formed in advance, one slide is selected as the light projection pattern. When the light projection pattern generation unit 1602 has the transmissive liquid crystal display element described with reference to FIG. 15, a pattern based on the binary image generated by the light projection pattern generation unit 1602 with the binary image generation unit 1304 shown in FIG. Control is performed by the liquid crystal control means 1307 so as to form the transmissive liquid crystal display element 1320.
[0212]
Step 1702 is a distance image generation processing step based on the projection pattern image, and is based on images taken by the base camera 201 and the reference camera 202, that is, images stored in the frame memories 1a and 203 and the frame memories 2a and 204. This is executed by the image correlation unit 205 and the distance image generation unit 206.
[0213]
Step 1703 is a mismatching area detection process based on the generated distance image, and is executed by the mismatching area detection means 1601. Mismatch region detection processing will be described with reference to FIGS. 20 and 21. FIG.
[0214]
FIG. 20 shows an outline of the mismatching detection process. The distance image labeling process is executed in step 1801 to obtain a labeling image 1802. Based on the labeling image 1802, the mismatching area is determined in step 1803. To detect.
[0215]
FIG. 21 shows a detailed processing flow of the labeling process in step 1801 shown in FIG. First, in step 2101 of FIG. 21, the mismatching area detection unit 1601 shown in FIG. 18 reads the distance image D1 from the image memory 207.
[0216]
In step 2102, the mismatching area detection unit 1601 initializes the number k of unlabeled pixels as an initialization process for setting coordinates for starting the labeling process, and the mismatching area detection unit 1601 stores the built-in coordinate memory. The buffers x [k] and y [k] are initialized.
[0217]
In step 2103, the mismatching region detection means 1601 sets an arbitrary position coordinate of a portion corresponding to the measurement target (subject) in the distance image D1 as the coordinate (i, j) of the target pixel.
[0218]
In step 2104, the mismatching region detection means 1601 extracts the pixel value g (i, j) of the target pixel of the distance image D1. The pixel value g (i, j) indicates the distance value of the pixel (i, j).
[0219]
Next, in step 2105, the mismatching region detection means 1601 determines the pixel value g (i + m, j + n) (where m, n = −) of one unprocessed pixel among the pixels adjacent to the target pixel (i, j). 1, 0, 1) is extracted.
[0220]
In step 2106, the mismatching area detection unit 1601 determines whether the pixel value g (i, j) of the pixel of interest (i, j) is the label value L0, and further the pixel value of the pixel of interest. It is determined whether or not the absolute value of the difference between g (i, j) and the pixel value g (i + m, j + n) of an adjacent pixel is smaller than a predetermined threshold value.
[0221]
The continuity between the pixel (i, j) and the adjacent pixel (i + m, j + n) is determined by comparison with this threshold value. That is, if the pixel value of the pixel (i, j) and the adjacent pixel (i + m, j + n) and the difference between g (i, j) and g (i + m, j + n) are smaller than the threshold value, the two pixels are in a continuous region. If it is greater than the threshold, it is determined that the region is a discontinuous region.
[0222]
It is determined that the pixel value g (i, j) of the pixel of interest is not L0, and the pixel value g (i + m.j + n) of the adjacent pixel is continuous with respect to the pixel value g (i, j). If YES, go to step 2107.
[0223]
In step 2107, the mismatching area detection unit 1601 increments the number k of unlabeled pixels by 1, stores the x coordinate i + m of the adjacent pixel in the coordinate storage buffer x [k], and stores the y coordinate j + n as the coordinate. Stored in the buffer y [k].
[0224]
In step 2108, the mismatching region detection means 1601 determines whether or not the processing of steps 2105 and 2106 has been performed on all eight pixels adjacent to the pixel of interest, and steps 2105 and 2106 are performed on the adjacent pixels. If it is determined that there is something that has not been processed, the process returns to step 2105 and the subsequent processing is repeated. Thereafter, if it is determined in step 2108 that the processing of steps 2105 and 2106 has been performed on all eight adjacent pixels, the process proceeds to step 2109.
[0225]
In step 2109, the mismatching area detection unit 1601 replaces the pixel value g (i, j) of the target pixel with the label value L0. Here, the label value L0 is given to the pixel of the part corresponding to the measurement object (subject) of the distance image D1.
[0226]
In step 2110, the mismatching region detection unit 1601 determines whether or not the number of unlabeled pixels k is 0. If it is determined that the number of unlabeled pixels k is not 0, the process proceeds to step 2111.
[0227]
In step 2111, the mismatching area detection unit 1601 sets the value stored in the coordinate storage buffer x [k] as the x coordinate of the target pixel, and uses the value stored in the coordinate storage buffer y [k] as the target. The y coordinate of the pixel. Thereby, a pixel adjacent to the previous pixel of interest and having a continuous pixel value is set as a new pixel of interest. Thereafter, the mismatching region detection unit 1601 increments the number k of unlabeled pixels by 1 and returns to Step 2104.
[0228]
After that, in step 2110, the processing from step 2104 is repeated until it is determined that the number of unprocessed pixels k that are labeled k is 0. If it is determined that the number of unprocessed pixels k that are labeled k is 0, move on. In step 2112, the mismatching region detection unit 1601 replaces the pixel value of the pixel of the distance image D1 whose pixel value is not replaced with the label value L0 with the label value L1 indicating that it is a mismatching region.
[0229]
When the labeling process as described above is completed, the same label is assigned to the continuous region in which the distance data has a difference equal to or smaller than a predetermined threshold in the distance image D1, and for example, a labeling image 1802 in FIG. 20 is obtained.
[0230]
The mismatching region detection means 1601 determines whether or not there is a small isolated region that is surrounded by pixels to which the same label is assigned and that has a label different from the surroundings. In the labeling image 1802 of FIG. 20, regions B, C, and D satisfy these conditions, and it is determined that these regions B, C, and D are mismatching regions (step 1803).
[0231]
Returning to FIG. 19, the description will be further continued. If no mismatched area is detected as a result of the mismatched area detection process described above, the determination in step 1704 is No, and the process proceeds to step 1705, where the generated distance image is output.
[0232]
If a mismatching area is detected, the process proceeds to step 1706, where a new light projection pattern is generated or selected, and further, the processing after step 1701, that is, a distance image generation process based on the new light projection pattern is performed. A mismatching area detection process is executed. The process for generating a new light projection pattern is the same as that in the first embodiment, and a description thereof will be omitted.
[0233]
Note that step 1704 in FIG. 19 is configured to change the process depending on whether or not a mismatching region is detected, but only when a certain threshold value is set and the mismatching region exceeds the set threshold value or more. You may comprise so that the distance image generation process based on a new light projection pattern may be performed.
[0234]
According to the distance image generation apparatus and method of the present embodiment, the configuration is such that a mismatching region based on the generated distance image is detected and a new light projection pattern is set, and the light projection pattern generation means generates The pattern used is only a pattern that is determined to have good suitability as a distance image generation light projection pattern based on a pattern suitability index value indicating suitability as various patterns such as change rate. Therefore, it is possible to generate an appropriate distance image, and it is possible to output a highly accurate distance image from which the mismatching region has been eliminated.
[0235]
The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present invention. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. Further, what is configured by appropriately combining the above-described embodiments is also included in the scope of the present invention, and in order to determine the gist of the present invention, the column of the claims described at the beginning is referred to. Should.
[0236]
【The invention's effect】
As described above in detail, according to the present invention, the randomness pattern for projection is adjusted based on the pattern or the captured image, and the suitability degree as a pattern such as a pattern change rate, a feature amount, a variance value, or pattern matching is obtained. Since it is configured to obtain an appropriate projection pattern by evaluating with the pattern suitability index value shown, it is possible to acquire an appropriate projection pattern image for various measurement objects, and a highly accurate distance An image is obtained.
[0237]
Furthermore, according to the present invention, it is possible to evaluate and examine the projection pattern based on the pattern or the captured image, and to generate a distance image for an appropriate projection pattern irradiation image in which occurrence of mismatching is suppressed. As a result, a more accurate range image can be obtained, and an appropriate projection pattern can be selected before performing the corresponding point marking process and the range image generation process for the projection pattern image that often causes mismatching. Therefore, it is possible to obtain a highly accurate distance image by a single process without the necessity of repeatedly performing complicated corresponding point marking processing and distance image generation processing.
[0238]
Furthermore, according to the present invention, a pattern suitability degree indicating a suitability degree as various patterns such as a change rate is set by detecting a mismatching region based on the generated distance image and setting a new projection pattern. Since it is configured to use only a pattern that is determined to be suitable as a projection pattern for generating a distance image based on the index value, a highly accurate distance image from which the mismatching area is eliminated can be output. Is possible.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a stereo image method applicable as a three-dimensional information acquisition configuration used in the present invention.
FIG. 2 is a block diagram showing a configuration of an embodiment of a distance image generating apparatus to which the random pattern generating apparatus of the present invention is applied.
FIG. 3 is a diagram showing an example of an aperiodic light projection pattern used in the light projection pattern generation and irradiation means in the present invention.
FIG. 4 is a diagram illustrating a configuration example of a light projection pattern generation and irradiation unit in the present invention.
FIG. 5 is a configuration block diagram of a random pattern generation device of the present invention.
FIG. 6 is a diagram for explaining a projection pattern generation process flow of the random pattern generation apparatus of the present invention.
FIG. 7 is a diagram for explaining a projection pattern evaluation processing flow (No. 1) in the random pattern generation device of the present invention.
FIG. 8 is a diagram for explaining a projection pattern evaluation processing flow (No. 2) in the random pattern generation device of the present invention.
FIG. 9 is a diagram for explaining a projection pattern evaluation process flow (part 3) in the random pattern generation device of the present invention;
FIG. 10 is a diagram for explaining a light projection pattern generation processing flow of the random pattern generation device of the present invention.
FIG. 11 is a diagram for explaining a light projection pattern evaluation process flow in the random pattern generation apparatus of the present invention.
FIG. 12 is a diagram showing a first example of distance image generation using a light projection pattern generated by the random pattern generation device of the present invention.
FIG. 13 is a diagram showing a distance image generation example (No. 2) using a light projection pattern generated by the random pattern generation device of the present invention.
FIG. 14 is a block diagram showing a configuration of an embodiment (Example 1) of a distance image generating apparatus according to the present invention.
FIG. 15 is a diagram showing a detailed configuration of a light projection pattern generation unit and a light projection pattern evaluation unit of the distance image generation apparatus of the present invention.
FIG. 16 is a flowchart of distance image generation processing in the distance image generation apparatus of the present invention.
FIG. 17 is a diagram illustrating a processing process of a light projection pattern generation unit and a light projection pattern evaluation unit in the distance image generation apparatus of the present invention.
FIG. 18 is a block diagram showing a configuration of an embodiment (embodiment 2) of a distance image generating apparatus of the present invention.
FIG. 19 is a diagram showing a processing flow of the distance image generating apparatus of the present invention.
FIG. 20 is a diagram showing a detection process flow of a mismatching region in the distance image generation device of the present invention.
FIG. 21 is a diagram showing a processing flow of a labeling process based on a distance image in the distance image generating apparatus of the present invention.
[Explanation of symbols]
201 ... reference camera
202 ... Reference camera
203, 204 ... frame memory
205 ... Image association means
206: Distance image generating means
207 ... Image memory
208: Output means
220 ... Projection pattern generation and irradiation means
401: Control means
402 ... Camera 1 (reference camera)
403 ... Camera 2 (reference camera)
404: Image memory
501: Uniform random number generation means
502: Regular random number generation means
503 ... Image generation means
504 ... Binary image generation means
505 ... Change rate detection means
506 ... Change rate determination means
507 ... Standard deviation setting means
508 ... Threshold setting means
1210 ... Light projection pattern evaluation means
1211 ... Light projection pattern generation means
1212: Light projection pattern irradiation means
1301 ... Uniform random number generation means
1302: Regular random number generating means
1303 ... Image generation means
1304: Binary image generation means
1305: Standard deviation setting means
1306: Threshold setting means
1307: Liquid crystal control means
1308 ... Photographed image change rate detection means
1309: Change rate determination means
1601 ... Mismatching region detection means
1602 ... Light projection pattern generation means
1603 ... Light projection pattern irradiation means

Claims (13)

In a random pattern generation device that generates a projection pattern to be projected onto a measurement object in measurement of a three-dimensional shape according to a stereo image method ,
A projection pattern generating means capable of generating a plurality of different aperiodic projection patterns using random numbers;
A projection pattern evaluation unit that performs evaluation of the projection pattern generated in the projection pattern generation unit;
The light projection pattern evaluation unit calculates a pattern suitability index value indicating the suitability of the light projection pattern generated by the light projection pattern generation unit based on the generated light projection pattern, and calculates the pattern suitability by comparing with a predetermined threshold index value, generating projection patterns have a determining configuration whether with suitability as projection pattern for three-dimensional shape measurement,
The light projection pattern evaluation means includes:
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
The value of any one of (1) to (4) above is calculated as the pattern suitability index value, and the generated light projection pattern is projected for three-dimensional shape measurement by comparing the calculated value with a predetermined threshold value. A random pattern generation device having a configuration for determining whether or not a pattern has suitability .   The projection pattern generation unit has a configuration for generating a different aperiodic projection pattern by changing a standard deviation s that defines a normal random number, and the generated projection pattern is the projection pattern evaluation unit. When it is determined that the projection pattern is not suitable as a projection pattern for measuring a three-dimensional shape, a new aperiodic projection pattern in which the standard deviation s is changed is generated. Item 2. The random pattern generation device according to Item 1.   The light projection pattern generation means has a configuration for generating a binary image obtained by binarizing an image made of normal random numbers using a threshold value T, and changing the threshold value T allows different aperiodic light projection patterns to be generated. The threshold T is changed when it is determined by the projection pattern evaluation means that the generated projection pattern is not suitable as a projection pattern for three-dimensional shape measurement. The random pattern generation device according to claim 1, wherein the random pattern generation device has a configuration for generating a new aperiodic projection pattern.   The projection pattern generation unit has a configuration for generating a grayscale image based on normal random numbers, and the generated projection pattern has suitability as a projection pattern for three-dimensional shape measurement in the projection pattern evaluation unit. 2. The random pattern generation device according to claim 1, further comprising a configuration for generating a new aperiodic projection pattern in which a parameter defining a normal random number is changed when it is determined that there is no normal random number.   The projection pattern evaluation unit is configured to extract at least one or more windows that are partial areas from the projection pattern generated by the projection pattern generation unit and calculate the pattern suitability index value. The random pattern generation device according to claim 1. In a random pattern generation method for generating a projection pattern to be projected onto a measurement object in measurement of a three-dimensional shape according to a stereo image method ,
(A) a projection pattern generation step for generating a non-periodic projection pattern using random numbers;
(B) an evaluation step for evaluating the projection pattern generated in the projection pattern generation step;
(C) As a pattern suitability index value indicating the suitability as a pattern of the projection pattern generated in the projection pattern generation step ,
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
A pattern suitability index value calculating step for calculating any one of the values (1) to (4) based on the generated light projection pattern;
(D) an aptitude determination step for determining whether or not the generated light projection pattern has suitability as a three-dimensional shape measurement light projection pattern by comparing the calculated pattern suitability index value with a predetermined threshold;
An evaluation step including:
Until it is determined that there is appropriateness in the suitability determining step, different projection patterns are generated in step (a), and steps (a) to (d) are repeatedly executed.
A random pattern generation method characterized in that the projection pattern determined to be appropriate in the suitability determination step in the evaluation step is used as a three-dimensional shape measurement projection pattern. In a distance image generation device that generates a distance image by measuring a three-dimensional shape of the measurement object using images taken from different viewpoints,
A projection pattern generating means capable of generating a plurality of different aperiodic projection patterns using random numbers;
A reference camera for capturing a projection pattern image of the measurement target;
A reference camera that captures a projection pattern image at a different viewpoint from the reference camera;
Evaluate the pattern suitability of at least one of the projection pattern images captured by the reference camera or the reference camera, and evaluate whether or not the projection pattern image for the distance image generation has suitability. It is a light projection pattern evaluation means , as a pattern suitability index value indicating the suitability as a pattern of the light projection pattern,
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
By calculating one of the above values (1) to (4) and comparing the calculated pattern suitability index value with a predetermined threshold value, the generated light projection pattern becomes a three-dimensional shape measurement light projection pattern. A projection pattern evaluation means for determining whether or not it has aptitude,
An image-to-image association unit that detects correspondence between the projection pattern images captured by the reference camera and the reference camera and outputs parallax data;
Distance image generating means for generating a distance image based on the parallax data output by the image correlation means;
When the projection pattern image obtained based on the generated projection pattern is determined not to be suitable as the projection pattern image for distance image generation by the projection pattern evaluation unit, Generate a periodic flood pattern,
The inter-image association unit and the distance image generation unit determine that a projection pattern image obtained based on the generated projection pattern is appropriate as a projection image for distance image generation by the projection pattern evaluation unit. A distance image generation apparatus characterized in that the processing is executed on the condition of the above. The projection pattern generating means has a configuration for generating a different aperiodic projection pattern by changing a standard deviation s defining a normal random number, and a projection pattern obtained based on the generated projection pattern When the light projection pattern evaluation unit determines that the image is suitable as a distance image generation light projection pattern, a new aperiodic light projection pattern in which the standard deviation s is changed is generated. The distance image generating apparatus according to claim 7 . The light projection pattern generation unit has a configuration for generating a binary image obtained by binarizing an image made of normal random numbers using a threshold value T, and changing the threshold value T allows different aperiodic light projection patterns to be generated. The projection pattern image obtained based on the generated projection pattern is determined by the projection pattern evaluation means as having no suitability as the projection pattern for distance image generation. The distance image generating apparatus according to claim 7 , wherein the distance image generating apparatus has a configuration for generating a new aperiodic projection pattern in which the threshold value T is changed. In a distance image generation device that generates a distance image by measuring a three-dimensional shape of the measurement object using images taken from different viewpoints,
This is a projection pattern generation means for generating a pattern to be projected onto a measurement target, and can generate a plurality of different aperiodic projection patterns using random numbers, and also shows a pattern suitability indicating the suitability of the pattern The index value is calculated based on the generated light projection pattern, and the suitability as the distance image generation light projection pattern is determined by comparing the calculated pattern suitability index value with a predetermined threshold value. A projection pattern generating means for setting only a pattern as a projection pattern;
A reference camera for capturing a projection pattern image of the measurement target;
A reference camera that captures a projection pattern image at a different viewpoint from the reference camera;
An image-to-image association unit that detects correspondence between the projection pattern images captured by the reference camera and the reference camera and outputs parallax data;
A distance image generating means for generating a distance image based on the parallax data output by the inter-image association means;
A mismatching area detecting means for detecting a mismatching area in the distance image generated by the distance image generating means;
Have
The distance image having a configuration in which the projection pattern generation unit generates a new aperiodic projection pattern when the mismatching region detection unit detects a mismatching region equal to or greater than a predetermined threshold. Generator. In a distance image generation method for generating a distance image by measuring a three-dimensional shape of the measurement object using images taken from different viewpoints of the measurement object,
A projection pattern generation step for generating a non-periodic projection pattern using random numbers;
An image capturing step of capturing the light projection pattern image of the measurement object at a synchronized timing from different viewpoints by applying a standard camera and a reference camera ;
Evaluate the pattern suitability of at least one of the projection pattern images captured by the reference camera or the reference camera, and evaluate whether or not the projection pattern image for the distance image generation has suitability. As a pattern suitability index value indicating the suitability as a pattern of the light projection pattern in the light projection pattern evaluation step ,
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
By calculating one of the above values (1) to (4) and comparing the calculated pattern suitability index value with a predetermined threshold value, the generated light projection pattern becomes a three-dimensional shape measurement light projection pattern. A projection pattern evaluation step for determining whether or not it has aptitude ; and
An image-corresponding step for detecting a correspondence between each projection pattern image captured by the reference camera and the reference camera and outputting parallax data;
A distance image generation step for generating a distance image based on the parallax data output by the inter-image association means;
In the projection pattern generation step, when it is determined that the projection pattern image obtained based on the generated projection pattern is not suitable as the projection pattern image for distance image generation in the projection pattern evaluation step, Repeatedly generate a periodic flood pattern,
In the association step between images and the distance image generation step, it is determined that the projection pattern image obtained based on the generated projection pattern is appropriate as the projection image pattern for distance image generation in the projection pattern evaluation step. A distance image generation method characterized in that processing is executed on the condition. In a distance image generation method for generating a distance image by measuring a three-dimensional shape of the measurement object using images taken from different viewpoints of the measurement object,
A non-periodic projection pattern using random numbers is generated, a pattern suitability index value indicating suitability as a pattern is calculated based on the generated projection pattern, and the calculated pattern suitability index value is a predetermined threshold value A light projection pattern generation step for determining suitability as a light projection pattern for distance image generation by comparing with, and setting only a pattern determined as suitable as a light projection pattern;
An image capturing step of capturing the light projection pattern image of the measurement object at a timing synchronized from different viewpoints by applying a standard camera and a reference camera ;
An image-to-image association step of detecting correspondence between each projection pattern image captured by the reference camera and the reference camera and outputting parallax data;
A distance image generation step for generating a distance image based on the parallax data output in the inter-image association step;
A mismatching region detection step for detecting a mismatching region in the distance image based on the projection pattern image generated in the distance image generation step;
Have
The light projection pattern generation step includes a new aperiodic light projection when a mismatching area equal to or greater than a predetermined threshold is detected in the mismatching area detection step in the distance image obtained based on the generated light projection pattern. A distance image generation method characterized by generating a pattern. A computer program for executing a projection pattern generation process used for executing a three-dimensional shape measurement for measuring a three-dimensional shape of the measurement target using images obtained by photographing the measurement target from different viewpoints on a computer system. A program providing medium provided tangibly, the computer program comprising:
(A) a projection pattern generation step for generating a non-periodic projection pattern using random numbers;
(B) an evaluation step for evaluating the projection pattern generated in the projection pattern generation step;
(C) As a pattern suitability index value indicating the suitability as a pattern of the projection pattern generated in the projection pattern generation step ,
(1) a change rate indicating a periodic change in pixel values constituting a light projection pattern;
(2) a feature amount calculated from an eigenvalue of a feature matrix based on a pixel value of an image constituting a projection pattern;
(3) An inter-region correlation value indicating the similarity between different partial regions constituting the projection pattern,
(4) The dispersion value of the image constituting the projection pattern,
A pattern suitability index value calculating step for calculating any one of the values (1) to (4) based on the generated light projection pattern;
(D) an aptitude determination step for determining whether or not the generated light projection pattern has suitability as a three-dimensional shape measurement light projection pattern by comparing the calculated pattern suitability index value with a predetermined threshold;
An evaluation step including:
Until it is determined that there is appropriateness in the suitability determining step, different projection patterns are generated in step (a), and steps (a) to (d) are repeatedly executed.
And determining the projection pattern determined to be appropriate in the suitability determination step in the evaluation step as a projection pattern for three-dimensional shape measurement.

Images