JP4131097B2 - Image processing method, image processing apparatus, and image processing program - Google Patents

Description

【００５０】
なお、本発明のより望ましい形態は、上記式（ア）に従って、厳密な相関値計算を行う方法であるが、本実施の形態では、照明の変動パターンが点滅からなるバイナリ符号であることや、自然界に検出対象と相関度の高い急激なドットデータ変動の頻度が比較的少ないことから、上記式（ア）のような厳密な相関値計算を用いなくても、簡易的な相関度算出処理でも可能である。ちなみに、次式（イ）による方法は、検出スレッシュ設定が厳密な相関値計算よりやや煩雑になるが、通常の相関値計算にあるような、二乗や平方根などのコストのかかる演算を回避しつつ、検出性能を確保することが可能である。以後、この式による値も相関値と呼ぶ。値域は８ｂｉｔ輝度信号であれば、０〜２５５を取る。
【００５１】
相関値＝Σ１／Ｃ１−Σ０／Ｃ０ ・・・・（イ）
【００５２】
但し、Σ１はパターンの論理１に対応する輝度レベルの総和（すなわち、論理１シンボルの総和値）、Ｃ１はパターンの論理１の個数（すなわち、論理１シンボルの個数）、Σ０はパターンの論理０に対応する輝度レベルの総和（すなわち、論理０シンボルの総和値）、Ｃ０はパターンの論理０の個数（すなわち、論理０シンボルの個数）である。
【００５３】
今、パターン１は「１０００１０１」のパターンシンボルを持ち、パターン２は「１１１０１００」のパターンシンボルを持つから、これらパターンごとの式（イ）は、次式（イ−１、イ−２）のようになる。なお、以下において、相関値１はパターン１の相関値、相関値２はパターン２の相関値である。
【００５４】

【００５５】
図８において、ｔ０〜ｔ６のそれぞれで、外乱光の輝度レベルは“１３０”、“２４０”、“２００”、“１９０”、“２２０”、“１０８”、“１１９”となっており、第１光源領域の輝度レベルは“２００”、“６５”、“６０”、“６３”、“２０８”、“５８”、“１９８”となっている。同様に、第２光源領域の輝度レベルは“１２５”、“１２６”、“１２４”、“２９”、“１２４”、“２８”、“３０”、第３光源領域の輝度レベルは“１６５”、“１５５”、“１５２”、“７０”、“１５４”、“８０”、“７１”となっている。
【００５６】
以下、これらの実際値（輝度レベル）を、パターン１及びパターン２のそれぞれについて、上式（イ−１、イ−２）に当てはめた場合の、相関値１及び相関値２の算出方法を例示する。
【００５７】
はじめに、パターン１の相関値（相関値１）を算出する。すなわち、上式（イ−１）を用いて、外乱光、第１〜第３光源領域の各相関値（相関値１）を算出する。パターン１の論理１シンボルはｔ０、ｔ４、ｔ６の位置にあり、その個数（Ｃ１）は“３”である。また、パターン１の論理０シンボルはｔ１、ｔ２、ｔ３、ｔ５の位置にあり、その個数（Ｃ０）は“４”である。
【００５８】
したがって、パターン１の外乱光のΣ１は、ｔ０＋ｔ４＋ｔ６＝“１３０”＋“２２０”＋“１１９”＝“４６９”となり、Σ０は、ｔ１＋ｔ２＋ｔ３＋ｔ５＝“２４０”＋“２００”＋“１９０”＋“１０８”＝“７３８”となり、Ｃ１＝“３”、Ｃ０＝“４”であるから、パターン１の外乱光の相関値１は、上式（イ−１）より、“４６９”／“３”−“７３８”／“４”＝“−２８．１７”となる。
【００５９】
また、パターン１の第１光源領域のΣ１は、ｔ０＋ｔ４＋ｔ６＝“２００”＋“２０８”＋“１９８”＝“６０６”となり、Σ０は、ｔ１＋ｔ２＋ｔ３＋ｔ５＝“６５”＋“６０”＋“６３”＋“５８”＝“２４６”となり、Ｃ１＝“３”、Ｃ０＝“４”であるから、パターン１の第１光源領域の相関値１は、上式（イ−１）より、“６０６”／“３”−“２４６”／“４”＝“１４０．５０”となる。
【００６０】
また、パターン１の第２光源領域のΣ１は、ｔ０＋ｔ４＋ｔ６＝“１２５”＋“１２４”＋“３０”＝“２７９”となり、Σ０は、ｔ１＋ｔ２＋ｔ３＋ｔ５＝“１２６”＋“１２４”＋“２９”＋“２８”＝“３０７”となり、Ｃ１＝“３”、Ｃ０＝“４”であるから、パターン１の第２光源領域の相関値１は、上式（イ−１）より、“２７９”／“３”−“３０７”／“４”＝“１６．２５”となる。
【００６１】
また、パターン１の第３光源領域のΣ１は、ｔ０＋ｔ４＋ｔ６＝“１６５”＋“１５４”＋“７１”＝“３９０”となり、Σ０は、ｔ１＋ｔ２＋ｔ３＋ｔ５＝“１５５”＋“１５２”＋“７０”＋“８０”＝“４５７”となり、Ｃ１＝“３”、Ｃ０＝“４”であるから、パターン１の第３光源領域の相関値１は、上式（イ−１）より、“３９０”／“３”−“４５７”／“４”＝“１５．７５”となる。
【００６２】
次に、パターン２の相関値（相関値２）を算出する。すなわち、上式（イ−２）を用いて、外乱光、第１〜第３光源領域の各相関値（相関値２）を算出する。パターン２の論理１シンボルはｔ０、ｔ１、ｔ２、ｔ４の位置にあり、その個数（Ｃ１）は“４”である。また、パターン２の論理０シンボルはｔ３、ｔ５、ｔ６の位置にあり、その個数（Ｃ０）は“３”である。
【００６３】
したがって、パターン２の外乱光のΣ１は、ｔ０＋ｔ１＋ｔ２＋ｔ４＝“１３０”＋“２４０”＋“２００”＋“２２０”＝“７９０”となり、Σ０は、ｔ３＋ｔ５＋ｔ６＝“１９０”＋“１０８”＋“１１９”＝“４１７”となり、Ｃ１＝“４”、Ｃ０＝“３”であるから、パターン２の外乱光の相関値２は、上式（イ−２）より、“７９０”／“４”−“４１７”／“３”＝“５８，５０”となる。
【００６４】
また、パターン２の第１光源領域のΣ１は、ｔ０＋ｔ１＋ｔ２＋ｔ４＝“２００”＋“６５”＋“６０”＋“２０８”＝“５３３”となり、Σ０は、ｔ３＋ｔ５＋ｔ６＝“６３”＋“５８”＋“１９８”＝“３１９”となり、Ｃ１＝“４”、Ｃ０＝“３”であるから、パターン２の第１光源領域の相関値２は、上式（イ−２）より、“５３３”／“４”−“３１９”／“３”＝“２６．９２”となる。
【００６５】
また、パターン２の第２光源領域のΣ１は、ｔ０＋ｔ１＋ｔ２＋ｔ４＝“１２５”＋“１２６”＋“１２４”＋“１２４”＝“４９９”となり、Σ０は、ｔ３＋ｔ５＋ｔ６＝“２９”＋“２８”＋“３０”＝“８７”となり、Ｃ１＝“４”、Ｃ０＝“３”であるから、パターン２の第２光源領域の相関値２は、上式（イ−２）より、“４９９”／“４”−“８７”／“３”＝“９５．７５”となる。
【００６６】
また、パターン２の第３光源領域のΣ１は、ｔ０＋ｔ１＋ｔ２＋ｔ４＝“１６５”＋“１５５”＋“１５２”＋“１５４”＝“６２６”となり、Σ０は、ｔ３＋ｔ５＋ｔ６＝“７０”＋“８０”＋“７１”＝“２２１”となり、Ｃ１＝“４”、Ｃ０＝“３”であるから、パターン２の第３光源領域の相関値２は、上式（イ−２）より、“６２６”／“４”−“２２１”／“３”＝“８２．８３”となる。
【００６７】
以上の計算によって得られた各相関値（相関値１、相関値２）を比較すると、パターン１については輝点２ａ（第１光源領域）の相関値１（“１４０．５０”）が最大であり、パターン２については照射光領域３ｂ（第２光源領域）の相関値２（“９５．７５”）が最大であり、照射光領域４ｂ（第３光源領域）の相関値２（“８２．８３”）がその次につけている。
【００６８】
したがって、相関値判定用のスレッシュレベルを、例えば、“８０．００”とすると、パターン１については、その最大値を示した輝点２ａ（第１光源領域）の相関値１（“１４０．５０”）だけが抽出されることとなり、また、パターン２については、その最大値を示した照射光領域３ｂ（第２光源領域）の相関値２（“９５．７５”）と、次位の大きさを示した照射光領域４ｂ（第３光源領域）の相関値２（“８２．８３”）が抽出されることとなる。
【００６９】
その結果、外乱光が重畳されている場合であっても、当該外乱光に影響されることなく、輝点２ａや照射光領域３ｂ及び４ｂの検出を支障なく行うことができ、しかも、外乱光の最大輝度レベルや輝度変化幅が輝点２ａや照射光領域３ｂ及び４ｂのそれよりも大きい場合（図８のリスト図は敢えてそのようなレベル関係にしてある。）であっても、上記の検出動作、すなわち、輝点２ａや照射光領域３ｂ及び４ｂの検出を何ら支障なく行うことができる。
【００７０】
図９は、上記第１の実施の形態における画像処理装置１の処理フローを示す図である。同図において、まず、第２光源３や第３光源４（要すれば第１光源２にも）に所要の変動パターンのセット（設定）、当該変動パターンとの処理対応リストのセット（設定）、及び、各光源との同期合わせを行ったり等の初期化を実行する（ステップＳ１１）。次に、カメラ６からの画像信号取り込み（フレームキャプチャ）を待つ（ステップＳ１２）。画像取り込みが行われると、キャプチャフレームバッファ１２５にキャプチャされた画像信号に基づく画像データを、そのときのタイムスロットカウンタ部１８の値（Ｎ）に対応したバッファ（例えば、Ｎ＝０であればリングバッファ１２３のバッファ０）にコピーする（ステップＳ１３）。
【００７１】
次に、モジュロ演算を行って（タイムスロットは、変動パターンが、１１１０１００の長さＬ＝７ビットパターンで表される場合、Ｎ＝０〜６の値を周期的に取る必要があるため）、タイムスロットカウンタ部１８の値を更新し（ステップＳ１４）、各検出変動パターンごとに相関値画像算出や判定２値が像算出等の処理を行い（ステップＳ１５）、パターン１の変動領域が検出された場合（ステップＳ１６）は、変動パターン／処理対応リスト格納部１５１からパターン１用処理指示を取り出し、処理を実行して、結果出力画像バッファ１５２に書き込む（ステップＳ１７）。
【００７２】
変動パターン／処理対応リスト格納部１５１の格納内容が図１２のようなものである場合、上記処理により、変動パタン１により照射された領域（または直接発光領域）の左上座標に、格納されている合成用ＣＧ画像等が書き出される。図６に示した、光源と符号パターンの組み合わせの場合、変動パターン１を設定した第１光源の座標に合成用のＣＧ画像（アニメキャラクタや、小道具の画像その他）が上書きされる。また、パターン２の変動領域が検出された場合（ステップＳ１８）は、同様に、変動パターン／処理対応リスト格納部１５１からパターン２用処理指示を取り出し、処理を実行して、結果出力画像バッファ１５２に書き込む（ステップＳ１９）。
【００７３】
設定および、実使用の様子が、図１２、図６の場合、変動パターン２を設定された、第２、第３光源で照射された、中央の立姿人物の上半身（４ｂ）、および、左下のボール（３ｂ）のみが、切り出されることになる。なお、処理内容の指定により、立ち姿人物の切り出しを行う際、変動パターン１で行う、ＣＧ合成のほうが表示優先度が高いため、このパターン２で対応処理を行っても立ち姿人物の差し出した手が、パターン１で書きこみ合成した、ＣＧ等の上になってしまうことはない。
【００７４】
そして、変動パターン／処理対応リスト格納部１５１から、非検出領域対象処理を実行し（ステップＳ２０）、図６の設定例の場合は、非検出領域に対しては白となるので、図１２、図６の例では、白いバックの中でボールと立姿人物の上半身とが切り出され、あたかもその人物の右手に、合成したＣＧ画像の物体（熊のぬいぐるみ）を持っているような結果出力が得られる。
【００７５】
なお、図４の変動パターン処理結果リスト１５１により、まったく同一の変動照明を用いながら、処理結果はまったく違うものを得ることも可能である。
【００７６】
後は、この結果画像を出力（ステップＳ２１）した後、再びフレームキャプチャ（ステップＳ１２）以降を繰り返す。
【００７７】
図１０は、上記のステップＳ１５の具体的フローであり、このフローでは、変動パターン１の処理と変動パターン２の処理を同時並行的に実行する。すなわち、まず、リングバッファ１２３の各バッファ０〜ｎに含まれる画像信号の中で変動パターンの“１”に相当する部分の加算結果を求め、その加算画像数で割った画像を、正画像バッファ１２１ｃ（変動パターン２の処理フローでは正画像バッファ１２２ｃ）に格納する（ステップＳ１５ａ）。この処理の目的は、格納したバッファの画像系列の中で、輝度が高くなっているべきタイミング（＝照明点灯）での画像の輝度平均を算出することである。
【００７８】
次に、リングバッファ１２３の各バッファ０〜ｎに含まれる画像信号の中で変動パターンの“０”に相当する部分の加算結果を求め、その加算画像数で割った画像を、負画像バッファ１２１ｄ（変動パターン２の処理フローでは正画像バッファ１２２ｄ）に格納する（ステップＳ１５ｂ）。この処理の目的は、輝度が低くなっているべきタイミング（＝照明消灯）での各画像の輝度平均を算出することである。
【００７９】
次に、正画像バッファの格納画像と負画像バッファの格納画像との差分をとり、その差分画像を相関値画像バッファ１２１ａ（変動パターン２の処理フローでは相関値画像バッファ１２２ａ）に格納する（ステップＳ１５ｃ）。 そして、最後に、照明強度／拡散特性、検出物体の反射係数、背景環境の空間的、時間的変動度によって定める一定の値により、相関値画像バッファ１２１ａ（変動パターン２の処理フローでは正画像バッファ１２２ａ）に格納する（ステップＳ１５ｄ）。
【００８０】
図１１は、図１０動作フローの処理の動作概念図である。中央に示した一連の画像バッファを参照することで、変動パターン１、２それぞれの検出処理では、点灯タイミングに対応するビット「１」および消灯に対応するビット「０」に対応する画像群が加算・平均されそれらの差分画像を、該当パターンの相関画像として得られる様子を示している。
【００８１】
以上のことから、本実施の形態によれば、撮影場所を問わずに、撮影画像中の所望領域を切り出したり、撮影画像中の所望位置に他の画像を張り付けたりする等の加工処理を行うことができる汎用性に優れた画像処理方法及び画像処理装置を提供することができるうえ、とりわけ外乱光に晒される撮影現場であっても、その外乱光の影響を受けることなく、撮影範囲５の画像信号の中から輝点２ｂや照射光領域３ｂ及び４ｂの検出を行うことができ、所望領域の画像オブジェクト切り出しや、所望位置への画像オブジェクトの張り付け等を行うことができるという格別有益な効果が得られる。
【００８２】
また、輝点２ｂや照射光領域３ｂ及び４ｂの変動パターン（明滅パターン）と画像加工処理の内容とを対応付けしたデータ（変動パターン／処理対応リスト）を変動パターン／処理対応リスト格納部１５１に格納し、実際に検出された輝点２ｂや照射光領域３ｂ及び４ｂの変動パターンと変動パターン／処理対応リストとを照合して、そのリストから、実行すべき画像加工処理の種別（画像の張り付け又は切り出し）を取り出すようにしているため、いちいち、撮像側（カメラ６側）で画像加工の種別を指定する必要がなく、操作性の簡素化を図ることができる。
【００８３】
＜第２の実施の形態＞
次に、本発明の第２の実施の形態について説明する。
まず、前記第１の実施の形態では、式（イ）で示した簡略的な相関値評価式を用いることにより、検出すべきパターンについて、他の画像領域の輝度等の変動(＝外乱)の影響を受けることなく、目的とする信号領域だけを検出することができるとしている。
【００８４】
しかしながら、前記第１の実施の形態における簡略的な相関値評価式（式（イ））は、より巌しい環境条件の場合、たとえば、外乱光の変動成分に検出パターン変動の１タイムスロットにきわめて近い瞬間的変動が多発するような画像領域が含まれる場合において、「変動の大きな外乱信号に影響されない、したがって、誤検出しない」という性能を完全に満たすことが困難であるという欠点が予想される。
【００８５】
このような厳しい環境条件を想定した場合、前記第１の実施の形態では、▲１▼パターンのビット数を増やす、または、▲２▼照射や発光光源の輝度を上げて、より大きな変動を与えるようにし、図１０のステップＳ１５ｄにおける２値化画像取得のための判定スレッショルドを上げる(＝判定条件を厳しくする)という、いずれかの対策をとる必要があるが、▲１▼の場合、その対策は処理においてオーバヘッドの増加を招く欠点があり、また、▲２▼の場合、その対策は検出感度が低下する欠点がある。
【００８６】
そこで、第２の実施の形態では、相関値評価式に正規化処理を加えることにより、より簡便で、しかも、上記の厳しい環境条件を想定した場合においても、良好な特性の相関値取得を可能とすることができるように改良した場合について詳述するものである。
なお、第２の実施の形態のブロツク構成図等は、前記第１の実施の形態と同一であるため図示等を省略し、また、処理フロー等についても相関値に係わる部分が第２の実施の形態に特有の処理になっている以外はすべて同一であるから、これも図示等を省略する。
【００８７】
第２の実施の形態のポイントは、前記第１の実施の形態で示した簡略的な相関値評価式（式（イ））の代わりに、次式（ウ）を使用することにある。
【００８８】
相関値＝（〔前式（イ）〕／ΔＷ）×ｃｏｎｓｔ ・・・・（ウ）
ΔＷ＝ｍａｘ（t０・・・・t６）−ｍｉｎ（t０・・・・t６） ・・・・（エ）
【００８９】
但し、ｍａｘ（引数）は引数で与えられたデータの最大値を返すｍａｘ関数、ｍｉｎ（引数）は引数で与えられたデータの最小値を返すｍｉｎ関数である。なお、式（ウ）において、ｃｏｎｓｔは乗算係数（たとえば、ｃｏｎｓｔ＝“２５５”）である。ｃｏｎｓｔを乗じているのは、最大０〜１の値を扱いやすいドット信号輝度範囲０〜２５５の値域に変換するためのである。ｃｏｎｓｔの値は、対象とする値域によっては“２５５”以外になることはいうまでもない。
【００９０】
前記第１の実施の形態と同様に、パターン１のパターンシンボルは「１０００１０１」であり、パターン２のパターンシンボルは「１１１０１００」であるものとする。これらパターンごとの式（ウ）は、次式（ウ−１、ウ−２）のようになる。なお、以下において、相関値１はパターン１の相関値、相関値２はパターン２の相関値である。
【００９１】

【００９２】
図１３において、ｔ０〜ｔ６のそれぞれで、第１光源領域の輝度レベルは“２００”、“６５”、“６０”、“６３”、“２０８”、“５８”、“１９８”となっており、同様に、第２光源領域の輝度レベルは“１２５”、“１２６”、“１２４”、“２９”、“１２４”、“２８”、“３０”、第３光源領域の輝度レベルは“１６５”、“１５５”、“１５２”、“７０”、“１５４”、“８０”、“７１”となっている。
【００９３】
これら第１〜第３光源領域の輝度レベル値の例は、前記第１の実施の形態（図８）と同一であるが、外乱光の輝度レベルについては、上記の厳しい環境条件を想定した、たとえば、“２５０”、“２４０”、“２５５”、“１００”、“３０”、“８０”、“１１９”となっている点で相違する。なお、第２光源領域周縁とは、第２光源領域（立姿人物５ｃへの照射領域）のエッジ部分のことをいう。これは、このエッジ部分における第１の実施の形態の不都合を説明するために、敢えて追加したものである。
【００９４】
ここで、図１３の各代表座標点の輝度レベルの時間的変化を図１４のとおりとする。図１４において、ハッチング付き四角記号のグラフは輝点２ａ（第１光源領域）の輝度レベル変化を表し、三角記号のグラフは照射光領域３ｂ（第２光源領域；立姿人物５ｃへの照射領域）の輝度レベル変化を表し、×記号のグラフは照射光領域４ｂ（第３光源領域；球状物体５ａへの照射領域）の輝度レベル変化を表し、丸記号のグラフは照射光源領域４ｂのエッジ（第３光源領域周縁；立姿人物５ｃへの照射領域エッジ）の輝度レベル変化を表し、黒菱形記号のグラフは外乱光（テレビ５ｂのフリッカ点灯）の輝度レベル変化を表している。なお、立姿人物５ｃは黒色の服を着ているものとする。すなわち、第３光源領域周縁は、黒色服のエッジを表している。
【００９５】
図１３の例に、前記第１の実施の形態における式（イ）を適用した場合、以下に詳述するように、外乱光を確実に排除できず、さらに、第２光源領域周縁も抽出できないという不都合を生じるが、第２の実施の形態における式（ウ）を適用した場合、このような不都合を生じることがない。
【００９６】
以下、図１３の例に、前記第１の実施の形態における式（イ）を適用した場合の相関値を、パターン１について「相関値１Ａ」、パターン２について「相関値２Ａ」ということとし、さらに、第２の実施の形態の式（ウ）を適用した場合の相関値を、パターン１については「相関値１Ｂ」といい、パターン２について「相関値２Ｂ」ということとして、それぞれの相関値（相関値１Ａ、２Ａ、１Ｂ、２Ｂ）の算出方法を例示する。
【００９７】
同図において、相関値１Ａは式（イ）を適用して算出したパターン１の相関値、相関値２Ａは式（イ）を適用して算出したパターン２の相関値であり、これらの相関値１Ａ、２Ａは、前記第１の実施の形態における相関値に相当するものである。また、相関値１Ｂは式（ウ）を適用して算出したパターン１の相関値、相関値２Ｂは式（ウ）を適用して算出したパターン２の相関値であり、これらの相関値１Ｂ、２Ｂは第２の実施の形態に特有のものである。
【００９８】
また、相関値１Ａ及び相関値２Ａの値は、外乱光と第２光源領域周縁を除き、前記第１の実施の形態における計算結果（図８参照）と同一である。すなわち、第１光源領域〜第３光源領域の相関値１Ａはそれぞれ“１４０．５０”、“１６．２５”、“１５．７５”、相関値２Ａはそれぞれ“２６．９２”、“９５．７５”、“８２．８３”であり、これらの値は、前記第１の実施の形態における計算結果（図８参照）と同一である。
【００９９】
前記第１の実施の形態と相違する外乱光と第２光源領域周縁のそれぞれの相関値１Ａ及び相関値２Ａは、式（イ）を適用して、次のようにして求められる。すなわち、図１３において、ｔ０〜ｔ６のそれぞれで、外乱光の輝度レベルは“２５０”、“２４０”、“２５５”、“１００”、“３０”、“８０”、“１１９” であり、また、第２光源領域周縁の輝度レベルは“７０”、“６９”、“６９”、“２０”、“６８”、“２１”、“２０” である。
【０１００】
パターン１の論理１シンボルはｔ０、ｔ４、ｔ６の位置にあり、その個数（Ｃ１）は“３”である。また、パターン１の論理０シンボルはｔ１、ｔ２、ｔ３、ｔ５の位置にあり、その個数（Ｃ０）は“４”である。したがって、パターン１の外乱光のΣ１は、ｔ０＋ｔ４＋ｔ６＝“２５０”＋“３０”＋“１１９”＝“３９９”となり、Σ０は、ｔ１＋ｔ２＋ｔ３＋ｔ５＝“２４０”＋“２５５”＋“１００”＋“８０”＝“６７５”となり、Ｃ１＝“３”、Ｃ０＝“４”であるから、パターン１の外乱光の相関値１Ａは、上式（イ−１）より、“３９９”／“３”−“６７５”／“４”＝“−３５．７５”となる。
【０１０１】
また、パターン１の第２光源領域周縁のΣ１は、ｔ０＋ｔ４＋ｔ６＝“７０”＋“６８”＋“２０”＝“１５８”となり、Σ０は、ｔ１＋ｔ２＋ｔ３＋ｔ５＝“６９”＋“６９”＋“２０”＋“２１”＝“１７９”となり、Ｃ１＝“３”、Ｃ０＝“４”であるから、パターン１の第２光源領域周縁の相関値１Ａは、上式（イ−１）より、“１５８”／“３”−“１７９”／“４”＝“７．９２”となる。
【０１０２】
パターン２の論理１シンボルはｔ０、ｔ１、ｔ２、ｔ４の位置にあり、その個数（Ｃ１）は“４”である。また、パターン２の論理０シンボルはｔ３、ｔ５、ｔ６の位置にあり、その個数（Ｃ０）は“３”である。したがって、パターン２の外乱光のΣ１は、ｔ０＋ｔ１＋ｔ２＋ｔ４＝“２５０”＋“２４０”＋“２５５”＋“３０”＝“７７５”となり、Σ０は、ｔ３＋ｔ５＋ｔ６＝“１００”＋“８０”＋“１１９”＝“２９９”となり、Ｃ１＝“４”、Ｃ０＝“３”であるから、パターン２の外乱光の相関値２Ａは、上式（イ−２）より、“７７５”／“４”−“２９９”／“３”＝“９４．０８”となる。
【０１０３】
また、パターン２の第２光源領域周縁のΣ１は、ｔ０＋ｔ１＋ｔ２＋ｔ４＝“７０”＋“６９”＋“６９”＋“６８”＝“２７６”となり、Σ０は、ｔ３＋ｔ５＋ｔ６＝“２０”＋“２１”＋“２０”＝“６１”となり、Ｃ１＝“４”、Ｃ０＝“３”であるから、パターン２の第２光源領域周縁の相関値２Ａは、上式（イ−２）より、“２７６”／“４”−“６１”／“３”＝“４８．６７”となる。
【０１０４】
次に、図１３の例に、第２の実施の形態における式（ウ）を適用して、パターン１の「相関値１Ｂ」とパターン２の「相関値２Ｂ」をすべて算出する。
まず、式（ウ）のΔＷ項を計算する。図１３において、ｔ０〜ｔ６のうち最大値を持つシンボルに着目すると、外乱光はｔ２の“２５５”、第１光源領域はｔ４の“２０８”、第２光源領域はｔ１の“１２６”、第３光源領域はｔ０の“１６５”、第２光源領域周縁はｔ０の“７０”である。また、ｔ０〜ｔ６のうち最小値を持つシンボルに着目すると、外乱光はｔ４の“３０”、第１光源領域はｔ５の“５８”、第２光源領域はｔ５の“２８”、第３光源領域はｔ３の“７０”、第２光源領域周縁はｔ３の“２０”である。
【０１０５】
したがって、外乱光のΔＷをΔＷ０、第１光源領域のΔＷをΔＷ１、第２光源領域のΔＷをΔＷ２、第３光源領域のΔＷをΔＷ３、及び、第２光源領域周縁のΔＷをΔＷ４と呼ぶことにすると、式（エ）より、ΔＷ０＝“２５５”−“３０”＝“２２５”、ΔＷ１＝“２０８”−“５８”＝“１５０”、ΔＷ２＝“１２６”−“２８”＝“９８”、ΔＷ３＝“１６５”−“７０”＝“９５”、ΔＷ４＝“７０”−“２０”＝“５０”となる。
【０１０６】
＜相関値１Ｂ＞
図１３の例におけるパターン１の相関値（相関値１Ｂ）は、式（ウ−１）より、相関値１ＡをΔＷで除算し、それにｃｏｎｓｔ＝“２５５”を乗じた値で与えられる。したがって、外乱光の相関値１Ａは“−３５．７５”、ΔＷ０は“２２５”であるから、“−４０．５２”が得られる。
【０１０７】
同様に、第１光源領域の相関値１Ａは“１４０．５０”、ΔＷ１は“１５０”であるから、“２３８．８５”が得られ、第２光源領域の相関値１Ａは“１６．２５”、ΔＷ２は“９８”であるから、“４２．２８”が得られる。また、第３光源領域の相関値１Ａは“１５．７５”、ΔＷ３は“９５”であるから、“４２．２８”が得られ、第２光源領域周縁の相関値１Ａは“７．９２”、ΔＷ４は“５０”であるから、“４０．３９”が得られる。
【０１０８】
＜相関値２Ｂ＞
図１３の例におけるパターン２の相関値（相関値２Ｂ）は、式（ウ−２）より、相関値２ＡをΔＷで除算し、それにｃｏｎｓｔ＝“２５５”を乗じた値で与えられる。したがって、外乱光の相関値２Ａは“９４．０８”、ΔＷ０は“２２５”であるから、“１０６．６３”が得られる。
【０１０９】
同様に、第１光源領域の相関値２Ａは“２６．９２”、ΔＷ１は“１５０”であるから、“４５．７６”が得られ、第２光源領域の相関値２Ａは“９５．７５”、ΔＷ２は“９８”であるから、“２４９．１５”が得られる。また、第３光源領域の相関値２Ａは“８２．８３”、ΔＷ３は“９５”であるから、“２２２．３４”が得られ、第２光源領域周縁の相関値２Ａは“４８．６７”、ΔＷ４は“５０”であるから、“２４８．２２”が得られる。
【０１１０】
＜各相関値の評価＞
さて、以上の計算により、図１３におけるすべての相関値（相関値１Ａ、２Ａ、１Ｂ、２Ｂ）が明らかになった。このうちの相関値１Ａ、２Ａについては、式（イ）を適用して求めたもの、すなわち、前記第１の実施の形態によるものである。
【０１１１】
不都合の実例は、図１３において、外乱光の相関値２Ａ（“９４．０８”）と第２光源領域周縁の相関値２Ａ（“４８．６７”）に見ることができる。
すなわち、外乱光の相関値２Ａは“９４．０８”という大きな値であるため、しきい値（たとえば、“８０．００”）を上回ってしまい分離することができないという不都合がある。また、第２光源領域周縁の相関値２Ａは“４８．６７”という小さな値であるため、しきい値（たとえば、“８０．００”）を下回ってしまい、抽出することができないという不都合がある。
【０１１２】
これに対して、式（ウ）を適用して求めた相関値、すなわち、第２の実施の形態によって求めた相関値１Ｂ及び２Ｂは、たとえば、外乱光の相関値２Ｂは“１０６．６３”、第２光源領域周縁の相関値２Ｂは“２４８．２２”であるため、適当なしきい値（たとえば、“２００．００”）を使用することにより、外乱光の相関値２Ｂ（“１０６．６３”）を除外することができ、且つ、第２光源領域周縁の相関値２Ｂ（“２４８．２２”）を抽出することができ、前記第１の実施の形態の不都合を解消することができる。
【０１１３】
以上、説明したとおり、第２の実施の形態にあっては、より巌しい環境条件の場合、たとえば、外乱の変動成分に検出パターン変動の１タイムスロットにきわめて近い瞬間的変動が多発するような画像領域が含まれる場合において、「変動の大きな外乱信号に影響されない、したがって、誤検出しない」という性能を完全に満たすことができるという格別のメリットが得られる。。
【０１１４】
なお、第２の実施の形態の処理は、カメラノイズなどによるフレームごとのサンプリングレベル変動のような、微小な変動列が正規化処理により大きい値が乗じられる可能性があり、検出対象外となるべき点が、偶然大きな相関値を持ってしまうことがある。しかしながら、このような検出対象外の点は孤立して出現する特性があるので、対処方法として、図１０のステップＳ１５ｄの２値化の前に、メディアンフィルタを追加することで解消することができる。
【０１１５】
また、カメラ特性が悪い場合、あらかじめ正規化する前に、ある程度以下の第１方式相関値(たとえば、０〜２５５の値域において５以下)は、切リ捨てる処理を追加してもよい。
【０１１６】
また、前記第１の実施の形態では、信号強度はそれ自体が２値化判定スレッシュを決める重要な要因であり、たとえば、黒っぼいものについては、それに応じて（２値化判定スレッシュを）下げるなどの使用条件による適切な設定などが必要であった。しかしながら、第２の実施の形態によれば、判定スレッシュは変動の大きさによらず同一パターンであることを表すので、環境条件による得られるΔＷの大小に影響されないという利点を備えるので、正規化された２値化判定スレッショルドを設定することができる。
【０１１７】
このように、第２の実施の形態によれば、相関係数計算によって、より厳密に「パターンの類似度」だけを評価することができる。したがって、より、変動パターンに相似した外乱があっても、また、変動の小さい第２光源領域周辺のようなドットの値変動列であっても、良好な相関値評価を行うことができる。
【０１１８】
なお、以上の実施の形態では、複数フレームを使って画像を処理するため、検出対象が動いた場合にエッジ部分がずれることがあり、結果として、検出できるのは各フレームで重なっている領域のみになるという問題があるが、撮像素子のフレームレートを上げる等により、動きに伴う検出誤差を許容範囲に収めることが可能になる。
【０１１９】
また、検出対象が動く場合は、複数枚のフレームバッファ間の動きベクトルデータを算出し、フレーム間の動きベクトルを参照し、対応点群のセットを取り出すことで行うなどするとよい。
【０１２０】
以上、説明したとおり、第２の実施の形態によれば、前記第１の実施の形態と比較して、より近似したパターンの外乱、また、より弱い変動の目的パターンも良好に切り出すことができる。また、式（ア）のような厳密な相関値に比べて、遙かに処理負担を引き下げることができ、且つ、信号の有無を判定する処理において、使用条件や環境条件の影響を受けにくいものとすることができる。
【０１２１】
【発明の効果】
本発明によれば、時間軸方向に並ぶ複数の画像信号に、所定のパターンで輝度が変動する発光領域を含ませておくだけで、当該画像内の発光領域に任意の画像を張り付けることができ、又は、当該領域内の画像を切り出すことができる。したがって、画像中の所望領域を切り出したり、画像中の所望位置に他の画像を張り付けたりする等の加工処理を行うことができる。
【図面の簡単な説明】
【図１】実施の形態の原理ブロック構成図である。
【図２】自発光型光源と外部照射型光源の具体例である。
【図３】撮影範囲５に四つの被写体が含まれる場合の使用例を示す図である。
【図４】画像処理装置１及び第１光源２〜第３光源４の具体的構成を示す図である。
【図５】画像処理装置１と各光源の間の通信構成図である。
【図６】リングバッファ１２３の画像格納状態を示す図である。
【図７】輝度レベルの時間的変化を示す図である。
【図８】各フレーム時間ｔ０〜ｔ６ごとの輝度レベル実際値及びこれらの実際値を適用して得られた相関値を示すリスト図である。
【図９】画像処理装置１の処理フローを示す図である。
【図１０】図９のステップＳ１５の具体的フローを示す図である。
【図１１】図１０のフローの動作概念図である。
【図１２】変動パターン／処理対応リストの一例を示す図である。
【図１３】各フレーム時間ｔ０〜ｔ６ごとの輝度レベル実際値及びこれらの実際値を適用して得られた相関値を示すリスト図である。
【図１４】輝度レベルの時間的変化を示す図である。
【符号の説明】
Ｇａ 球状物体像
Ｇｂ 画像オブジェクト
Ｇｃ 上半身像
１ 画像処理装置
２ 第１光源（自発光型光源）
２ａ 輝点
３ 第２光源（外部照射型光源）
３ｂ 照射光領域
４ 第３光源（外部照射型光源）
４ｂ 照射光領域
５ 撮影範囲
５ａ 球状物体
５ｂ テレビ
５ｃ 立姿人物
６ カメラ
１１ 光源制御部
１２ 画像検出部
１３ 検出領域形状認識処理部
１４ オブジェクト符号化処理部
１５ 効果付け処理装置
１６ 表示装置
１７ ＣＰＵ
１８ タイムスロットカウンタ部
４１ 同期トリガ信号取出部
４３ 変動パターンレジスタ
１１１ 符号クロック部
１１２ 同期トリガ信号出力部
１２１ａ 相関値画像バッファ
１２１ｅ 変動パターンレジスタ
１２２ａ 相関値画像バッファ
１２２ｅ 変動パターンレジスタ
１２３ リングバッファ
１２５ キャプチャフレームバッファ[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to an image processing method., Image processing apparatus, and image processing programFor more details,In the imageCut out the desired area,In the imageImage processing method applicable when performing processing such as pasting another image at a desired position, Image processing apparatus, and image processing programAbout.
[0002]
[Prior art]
In recent years, along with the popularization of digital image technology and photographing apparatuses using the same, it has become possible to easily generate digital image data of various subjects. Such image data can be processed from the characteristics of the digital signal (especially ease of processing), for example, to cut out a desired area in the captured image, or to paste another image at a desired position in the captured image. Therefore, conventionally, the following image processing has been performed.
[0003]
(1) Method using a dedicated processing tool
This is an image processing method in which target image data is read using a tool called an image editor or the like, and a desired area is artificially specified and cut out. The clipped image is saved as an alias image object. In addition, when pasting another image at a desired position in the photographed image, the pasting position is artificially designated in the same manner, and the desired pasted image is prepared from the pasted images prepared in advance at the designated position. Follow the steps of selecting and pasting. The disadvantages of this method are that it requires a dedicated processing tool and a processing device for image processing (such as a personal computer), so that real-time processing cannot be performed at the shooting site. There is a fatal problem.
[0004]
(2) Color separation method
This is a technique for separating an image signal of a subject having a specific background color. Sometimes called chroma key separation. Generally, a blue portion is removed from an image signal of a subject photographed with a blue background, and only the subject image is extracted.
[0005]
[Problems to be solved by the invention]
By the way, the method (2) has the merit that the disadvantage of the above (1) can be eliminated because the above-described color separation processing function is installed in the photographing apparatus, so that real-time performance can be obtained at the photographing site. However, because it requires a specific background color, it can only be used in specific places where the environment is prepared, such as a studio, and can be used universally regardless of the shooting location. There was a problem that it was not possible.
[0006]
  The problem to be solved by the present invention is:In the imageCut out the desired area,In the imageA versatile image processing method capable of performing processing such as pasting another image at a desired position., Image processing apparatus, and image processing programIs to provide.
[0007]
[Means for Solving the Problems]
  The invention according to claim 1 is an image processing method for processing a plurality of image signals arranged in the time axis direction, including a light emitting region whose luminance varies in a predetermined pattern,A first step of binarizing the plurality of image signals in a time axis direction to form a pattern symbol into a logical 1 symbol and a logical 0 symbol; a logical 1 symbol converted into a pattern symbol in the first step; and A pattern matching is performed based on a second step of calculating a total value of each of the logical 0 symbols, and a difference value between the total value of the logical 1 symbols calculated in the second step and the total value of the logical 0 symbols. And a third step of extracting an image of the light emitting area, and a fourth step of pasting an arbitrary image at the position of the image extracted in the third step or cutting out the extracted image ,It is characterized by including.
[0009]
  The invention according to claim 2In the first aspect of the present invention, the method further includes a fifth step of obtaining a difference value between the maximum symbol and the minimum symbol in each logical symbol converted into a pattern symbol in the first step, and the third step includes The difference between the total value of the logical 1 symbols determined in the second step and the total value of the logical 0 symbols is the difference value between the maximum symbol and the minimum symbol determined in the fifth step. Perform pattern matching based on the normalized result and extract the image of the light emitting areaIt is characterized by that.
[0012]
  According to a third aspect of the present invention, a plurality of image signals arranged in the time axis direction including a light emitting region whose luminance varies with a predetermined pattern are processed.In the image processing apparatus, a first means for binarizing the plurality of image signals in the time axis direction to form a pattern symbol into a logical 1 symbol and a logical 0 symbol, and a logical 1 pattern symbolized by the first means A second means for obtaining a sum value of each of the symbols and logic 0 symbols, and a difference value between the sum value of the logic 1 symbols and the sum value of the logic 0 symbols obtained by the second means. A third means for performing pattern matching and extracting an image of the light emitting area; and a fourth means for pasting an arbitrary image at the position of the image extracted by the third means or for cutting out the extracted image. MeansIt is characterized by that.
[0013]
  The invention described in claim 4The invention according to claim 3 further includes fifth means for obtaining a difference value between the maximum symbol and the minimum symbol in each logical symbol converted into a pattern symbol by the first means, and the third means comprises: The difference between the total value of the logical 1 symbols obtained by the second means and the total value of the logical 0 symbols is normalized by the difference value between the maximum symbol and the minimum symbol obtained by the fifth means. Perform pattern matching based on the result and extract the image of the light emitting areaIt is characterized by that. The invention according to claim 5A computer that performs image processing on a plurality of image signals arranged in the time axis direction, including a light emitting region whose luminance varies in a predetermined pattern, is binarized in the time axis direction to perform logic processing. First means for pattern symbolizing into 1 symbol and logic 0 symbol, logic 1 symbol converted into a pattern symbol by the first means, and second means for obtaining the total value of each of the logic 0 symbols, A third means for extracting an image of the light emitting area by performing pattern matching on the basis of a difference value between the total value of the logical 1 symbols obtained by the means of 2 and the total value of the logical 0 symbols; A fourth means for pasting an arbitrary image at the position of the image extracted by the above, or for cutting out the extracted image;It is made to function as.
[0014]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<First Embodiment>
FIG. 1 is a block diagram illustrating the principle of the first embodiment. In this figure, 1 is an image processing device, 2 is a first light source, 3 is a second light source, 4 is a third light source, 5 is a photographing range, 6 is a photographing device (hereinafter referred to as “camera”), and 11 is light source control. , 12 is an image detection unit, 13 is a detection region shape recognition processing unit, and 14 is an object encoding processing unit.
[0015]
The shooting range 5 is a shooting range of the camera 6, and the camera 6 takes images of the background and subjects (or a group of subjects consisting of a plurality of subjects) existing in the shooting range 5, and takes the shot images for a predetermined frame period ( For example, it is generated and output at several tens of frames per second).
[0016]
Three representative light sources (the first light source 2, the second light source 3, and the third light source 4) repeatedly change the luminance (for example, blinking; hereinafter represented by blinking) in a predetermined pattern or a controllable pattern. These are a “self-luminous type” that emits light within the photographing range 5 and a light beam from the outside of the photographing range 5 toward an arbitrary part or an arbitrary region of the subject located inside the photographing range 5. It can be divided into “external irradiation type” for irradiation. Hereinafter, for convenience, the first light source 2 will be described as a self-luminous type, and the second and third light sources 3 and 4 will be described as an external irradiation type.
[0017]
Although it does not specifically limit, FIG. 2 is a specific example of a self-light-emitting light source and an external illumination light source. In this figure, a point light source 35 such as an LED (Light-Emitting Diode) or a small light bulb is a kind of self-luminous light source that emits light, and an EL (Electro Luminescent) panel, a CRT (Cathode Ray Tube), or a reflection. A surface light source 36 of a control type (reflector, printing body mechanical control or LCD: Liquid Crystal Display) is also a kind of self-luminous light source that emits light. On the other hand, the laser pointer 37 that irradiates the beam is a kind of external illumination light source, and the flashlight 38 is also a kind of external illumination light source because it can emit visible light.
[0018]
Therefore, the illustrated point light source 35 or the surface light source 36 can be used for the first light source 2, and the illustrated laser pointer 37 or the flashlight 38 can be used for the second and third light sources 3 and 4. Can do. If the laser pointer 37 and the flashlight 38 are used inside the photographing range 5 and the light beam is irradiated toward the camera 6, the laser pointer 37 and the flashlight 38 are also self-luminous light sources. Since these are one type, these can also be used for the first light source 2.
[0019]
As described above, the camera 6 captures an image of a subject (or a subject group made up of a plurality of subjects) in the photographing range 5, and generates and outputs the photographed image at a predetermined frame period (for example, several tens of frames per second). When the first light source 2 to the third light source 4 are blinking in a predetermined pattern, the camera 6 includes the pattern information of the blinking light by the first light source 2 to the third light source 4 in addition to the subject image information. A frame image to be generated is generated and output.
[0020]
The light source control unit 11 and the image detection unit 12 control the drive timing with each other, and control the pattern cycle when all or some of the blinking patterns of the first light source 2 to the third light source 4 are externally controllable. The frame image signal from the camera 6 is received, and the light emission position of the self-luminous light source and the external illumination light source are based on the variation pattern of the partial signal level (typically luminance level) in the image signal. The irradiation position / irradiation area is specified, and the specific information and the image signal are output to the image utilization apparatus (detection area shape recognition processing unit 13 in FIG. 1).
[0021]
The usage of the image processing apparatus 1 in the basic block configuration shown in FIG. 1 is as follows. That is, the first light source 2 that is a self-luminous light source is placed at an arbitrary position of a subject (or subject group) existing in the shooting range 5 of the camera 6 and / or in an arbitrary region of the subject (or subject group). The light is emitted from the second light source 3 and the third light source 4 (or one of them), which are external illumination light sources.
[0022]
The place where the first light source 2 is placed and the irradiation range of light from the second light source 3 and the third light source 4 depend on the functions of the detection area shape recognition processing unit 13 of the image processing apparatus 1. For example, if a function for cutting out a desired area in a captured image is provided, the irradiation range of light from the second light source 3 and the third light source 4 can be adjusted to the desired cut-out area, and a desired area in the captured image can be obtained. If the function of attaching another image to the position is provided, the place where the first light source 2 is placed can be adjusted to the image attaching position.
[0023]
FIG. 3 is a diagram illustrating an example of use in the case where four subjects are included in the shooting range 5. The illustrated subject images are a spherical object 5a placed on the floor, a television 5b, a standing figure 5c with one arm extended horizontally, and a vertically long box-like object 5d. These subject examples are for convenience of explanation. Only. In FIG. 3, a bright spot 2a is drawn on the horizontal arm tip of the standing figure 5c. This bright spot 2a is caused by light emission of the first light source 2 which is a self-luminous light source. That is, it is assumed that, for example, the point light source 35 or the surface light source 36 of FIG. Further, different light is irradiated to the entire spherical object 5a and the upper body portion of the standing figure 5c, and these irradiation light regions 3b and 4b are a second light source 3 and a third light source which are external irradiation light sources. This is due to the emission of 4. That is, it is assumed that, for example, the irradiation light from the laser pointer 37 or the flashlight 38 in FIG. 2 is applied to the entire spherical object 5a and the upper half of the standing figure 5c.
[0024]
The camera 6 captures an image of a subject (or subject group consisting of a plurality of subjects) in the photographing range 5, and generates and outputs the photographed image at a predetermined frame period (for example, several tens of frames per second). Includes object images of the spherical object 5a, the television 5b, the standing figure 5c, and the vertically long box-like object 5d. Of the subject image, the luminance of the entire spherical object 5a varies in a predetermined pattern depending on the irradiation light region 3b, and the luminance of the upper body image portion of the standing figure 5c varies in a predetermined pattern depending on the irradiation light region 4b. Further, the luminance of the standing figure 5c near the horizontal arm tip varies in a predetermined pattern due to the bright spot 2a.
[0025]
The image detection unit 12 captures an image signal from the camera 6 and is self-luminous based on a variation pattern of a partial signal level (typically a luminance level) in the image signal that is blinked and controlled by the light source control unit 11. The light emission position of the light source (first light source 2) (the position of the bright spot 2a) and the irradiation position / irradiation region (the positions of the irradiation light regions 3b and 4b) and the external irradiation type light sources (second light source 3 and third light source 4) The area range) is specified, and the specified information and the image signal are output to the subsequent stage.
[0026]
Here, the main function of the image processing apparatus 1 is, for example, one having a function of cutting out a desired area in a captured image, or one having a function of attaching another image to a desired position in a captured image (or its) It has both functions), and can receive an output image signal and specific information and cut out an image from the image signal based on the specific information. In FIG. 3, the effect processing device 15 and the display device 16 receive the image signal output from the image detection unit 12 and specific information, and based on the specific information, put another image at a desired position in the captured image. The process of pasting and displaying it can be performed.
[0027]
The image obtained by these processes will be described by taking the imaging range 5 shown in FIG. 3 as an example. First, in the image clipping process, partial images included in the irradiated light areas 3b and 4b in the captured image are clipped. Therefore, in the example of FIG. 3, the entire image of the spherical object 5a and the upper body image of the standing figure 5c are cut out, and can be converted into data as individually usable image objects by the object encoding unit 14 of FIG. . On the other hand, in the image pasting process, an image object prepared in advance is pasted to the position of the bright spot 2a in the captured image (pixel coordinates in the capturing area 5), so that, for example, the effect processing apparatus 15 in FIG. The image object can be pasted at the position of the bright spot 2a and output to the display device 16 for display.
[0028]
The image object in the pasting process may be any image in principle, but from the viewpoint of practical use, an image object that can give a desired effect to the original image is preferable. For example, if an interesting image object such as “Stuffed Bear” (see FIG. 12) can be pasted, a virtual effect such as a stuffed toy can be given to the standing figure 5c.
[0029]
Incidentally, an image G shown in the lower right of FIG. 3 is an example of a processed image output to the display device 16, and the effect processing device 15 has both an image clipping function and an image pasting function. The case is assumed. The image G includes a spherical object image Ga cut out by the irradiation light region 3 b from the second light source 3, an upper body image Gc cut out by the irradiation light region 4 b from the third light source 4, and the first light source 2. The image object Gb is attached to the position of the bright spot 2a.
[0030]
Since the spherical object image Ga and the upper body image Gc are image objects that can be used independently, for example, the object-based encoding processing in today's advanced multimedia standards such as MPEG-4 and MPEG-7 is also flexible. Furthermore, since any image object can be pasted, it is possible to easily generate a processed image pasted with a bear doll, for example, as shown in FIG.
[0031]
FIG. 4 is a diagram showing a specific configuration of the image processing apparatus 1, the second light source 3, and the third light source 4 of FIG. 1 (or FIG. 3). It should be noted that the shooting range 5 and the subjects (spherical object 5a, television 5b, standing figure 5c, and vertically long box-like object 5d) included in the shooting range, irradiation light regions (irradiation light regions 3b and 4b), and bright spots (bright spots 2a ) Represents the same reference numerals as those in FIG. Furthermore, although the first light source is not shown in FIG. 4, the bright spot 2a of the horizontal arm tip of the standing figure 5c is due to the light emission of the first light source, and the standing figure 5c is the first light source in FIG. Suppose you hold the same thing as 2.
[0032]
4, the image processing apparatus 1 includes a CPU 17, a variation pattern / processing correspondence list storage unit 151, a result output image buffer 152, a code clock unit 111, a time slot counter unit 18, a first correlation processing unit 121, and a second correlation process. Unit 122, ring buffer 123, digitizing unit 124, capture frame buffer 125, synchronization trigger signal output unit 112, display device 161, and the like. The third light source 4 includes a synchronization trigger signal extraction unit 41, a light emitting unit 42, a variation pattern register 43, a bit clock unit 44, a sign clock unit 45, and the like. The second light source 3 has the same configuration as the third light source 4. With elements.
[0033]
<Description of each component of the image processing apparatus 1>
The CPU 17 controls the overall operation of the image processing apparatus 1 by program control and performs remote control of blinking patterns such as the second light source 3 and the third light source 4. The variation pattern / process correspondence list storage unit 151 has a function of associating and listing the processing contents of the image area correlated with the pattern for each variation pattern (flashing pattern of each light source 2 to 4). This functions in order to assign different image processing effects to the respective image regions in the present embodiment in which different variation patterns can be simultaneously discriminated and detected. The result output image buffer 152 stores a result obtained by integrating the results of the CPU 17 detecting each variation pattern and performing all the processes (cutout, erasure, replacement, etc.) on each image region based on the detection result.
[0034]
The code clock unit 111 generates a clock that determines the overall phase of the variation pattern. This clock designates the first timing of each variation pattern series. The timing of the code clock of the image processing apparatus 1 is slightly delayed by t time as shown in FIGS. 5B and 5C compared to the blinking timing of the blinking patterns of the light source devices 2 to 4. . Since the blinking transition of the illumination must be completed before the image capturing starts, the value of t is, for example, about 1.5 ms when the unstable time of the light source is 1 ms. The time slot counter unit 18 is a clock counter that designates the variation timing within the variation pattern. The camera 6 and the image processing apparatus 1 are configured to be synchronized in the time slots of each illumination, and the cycle is equal to the frame rate of the camera 6 as shown in FIG. Is delayed by t time compared to the light source device.
[0035]
The first correlation processing unit 121 includes a correlation value image buffer 121a, a binary image buffer 121b, a positive image buffer 121c, a negative image buffer 121d, and a fluctuation pattern register 121e, and functions as a function of capturing captured images in time series. The blinking pattern process indicated by the variation pattern register is processed to obtain a binary image in which the presence or absence of correlation in the image is determined. The second correlation processing unit 122 includes a correlation value image buffer 122a, a binary image buffer 122b, a positive image buffer 122c, a negative image buffer 122d, and a variation pattern register 122e, and a first correlation with respect to the second detected variation pattern. The same processing as the processing unit 121 is performed. The ring buffer 123 is an image memory that stores the past n images in order to obtain the degree of correlation for each time variation pattern to be detected with respect to time-series processing of captured images. If it is desired to detect a luminance fluctuation region where the light source fluctuation pattern matches a predetermined pattern symbol (for example, “1110100”), the number of images stored in the ring buffer 123 is the number of symbols of the pattern symbol. This is n sheets corresponding to n (here, n = 7).
[0036]
Note that the longer the pattern length (that is, the greater the number of symbols n), the more drastically the probability of coincidental coincidence with the natural luminance fluctuation (= false detection probability) can be drastically reduced. is there. However, if there is a very large brightness fluctuation in the background or surroundings, or if an object with a large instantaneous displacement is to be imaged, stronger illumination (and a higher detection threshold level) or a longer series of fluctuation patterns Need. In particular, when there is flickering illumination or when there is no object moving at high speed, irradiation with brightness variation illumination including blinking of 5 to 10 bits (number of symbols n = 5 to 10) has a reflection similar to that of human skin. Image detection is possible if the object can give a fluctuation of about 5% or more in terms of fluctuations in luminance.
[0037]
The digitizing unit 124 converts the image signal captured from the camera 6 into digital image data. The capture frame buffer 125 buffers image data for one frame. The synchronization trigger signal output unit 112 outputs a synchronization signal for light emission synchronization to the second light source 3 and the third light source 4. The display device 161 displays the image output as a result.
[0038]
<Description of each component of the third light source 4>
The synchronization trigger signal extraction unit 41 takes in the synchronization signal output from the image processing apparatus 1 and outputs it to the code clock unit 45. The light emitting unit 42 emits light with a blinking pattern corresponding to the variation pattern set in the variation pattern register 43, and irradiates the light to an arbitrary region of the photographing range 5 (in the figure, the upper body portion of a standing figure). The variation pattern register 43 holds the light emission pattern data of the light emitting unit 42. The bit clock unit 44 designates the ON / OFF timing of one time slot (one bit of the fluctuation pattern code) of the fluctuation pattern register. The code clock unit 45 designates the output timing of the first bit of the variation pattern register series.
[0039]
In the second light source 3 and the third light source 4 shown in FIG. 4, the flashing pattern of the light emission can be controlled from the outside, and the image processing apparatus 1 considers disturbances etc. from several prepared fluctuation patterns. Then, an appropriate pattern is selected, and the selected pattern is set in the second light source 3 or the third light source 4. For this pattern setting, the image processing apparatus 1 and each of the light sources 2 to 4 are configured to be able to communicate with each other. For example, as shown in FIG. The two light sources 3 and the third light source 4 are wired-connected by a connector or the like, or wirelessly connected by a radio or the like. Alternatively, as shown in FIG. 4, one light source (third light source 4 in FIG. 4) is used as a relay, and indirectly between the image processing apparatus 1 and another light source (second light source 3 in FIG. 4). It may be connected.
[0040]
For the sake of convenience, the variation pattern of the bright spot 2a is referred to as “variation pattern 1”, and another variation pattern different from the variation pattern 1 (hereinafter referred to as “variation pattern 2”) from the image processing apparatus 1 to the second light source 3 ”) Is set. In order to simplify the explanation, the variation pattern 1 is repeated “1000101”, the variation pattern 2 is repeated “1110100”, and the blinking (on / off) of the light corresponds to “1” and “0” of the pattern. Then, the bright spot 2a repeats “on → off → off → off → on → off → on”, and the two irradiation light regions 3b and 4b are “on → on → on → off → on → off → "OFF" will be repeated.
[0041]
These light emission patterns are photographed by the camera 6 together with the respective subject images in the photographing range 5 and output to the image processing apparatus 1 at a predetermined frame period. At the same time, an image signal for each frame digitized by the digitizing unit 124 is obtained. After one screen is developed in the capture frame buffer 125, the developed screen is sent to the ring buffer 123 and sequentially stored in the buffer 0 to the buffer n.
[0042]
FIG. 5B is a comparison diagram of timing on the imaging side (camera 6 side) and timing on the light source side (for example, the second light source 3 and the third light source 4). In the imaging side, a predetermined imaging side frame clock is used. The frame image is captured (photographed) by the synchronized imaging-side frame sampling clock, while the light source-side code output bit clock synchronized with the light-source-side code frame clock has the above-described variation pattern (“1110100”). The light emitting unit 42 is blinked. The rise and fall timing of the light source output signal is advanced by a certain time t with respect to the imaging side code frame clock, and this time t is the operation transition time of the light source output (left-down hatching in FIG. 5C). Reference) and a time considering the data sample delay on the imaging side.
[0043]
FIG. 6 is a diagram showing an image storage state of the ring buffer 123. Here, the number of buffers (n) of the ring buffer 123 is set to 7 for convenience, and each buffer is represented by reference numerals B0 to B6. Symbols t0 to t6 attached to the left side of B0 to B6 are frame times. When a frame image at time t0 is stored in B0, a frame image at the next time t1 is stored in B1, and at the next time t2. When the frame image is stored in B2 and this is repeated and the frame image at time t6 is stored in B6, a unique variation pattern is extracted using these seven frame images, and then again at the next time. Store the frame in the first B0.
[0044]
In FIG. 6, if the stored images of B0 to B6 are as shown in the figure, the entire image of the spherical object 5a and the upper body image of the standing figure 5c included in the stored image of B0 are brightened by the irradiation light regions 3b and 4b, respectively. In addition, the vicinity of the horizontal arm of the standing figure 5c is also brightened by the bright spot 2a. Similarly, regarding B1 to B6, the overall image of the spherical object 5a and the upper body image of the standing figure 5c are also brightened by the irradiation light regions 3b and 4b in B1, B2 and B4, respectively, while B3, B5 and B6 Then, since there is no irradiation light area | region 3b and 4b, all are dark. Moreover, the bright spot 2a exists also in B4 and B6, and does not exist in B1, B2, B3, and B5.
[0045]
Accordingly, the seven frame images shown in the figure are periodic patterns of “on → on → on → off → on → off → off” for the whole image of the spherical object 5a and the upper body image of the standing figure 5c, that is, “1110100”. In the vicinity of the horizontal arm tip of the standing figure 5c, “ON → OFF → OFF → OFF → ON → OFF → ON”, that is, “1000101” It becomes a specific image portion blinking with the fluctuation pattern 1. Even if such a specific image portion includes disturbance light having a low correlation degree with respect to the fluctuation pattern, for example, disturbance light due to luminance flicker of the television 5b in the shooting range 5, for the following reason, By distinguishing (discriminating) both, it is possible to cut out or paste a specific image portion without any trouble.
[0046]
The reason can be considered as follows. That is, for the irradiation light areas 3b and 4b and the bright spot 2a in the stored images B0 to B6, their representative coordinate points (for example, the coordinate point at the center of the area) are selected, and the brightness level of each representative coordinate point is selected. The temporal change is as shown in FIG. In FIG. 7, a hatched square symbol graph represents a change in luminance level of the bright spot 2a (first light source region), and a triangular symbol graph represents an irradiation light region 3b (second light source region; irradiation region to the standing figure 5c). The graph of the symbol x represents the luminance level change of the irradiation light region 4b (third light source region; the irradiation region of the spherical object 5a), and the graph of the black rhombus symbol represents disturbance light (flicker of the television 5b). Lighting level).
[0047]
Among these four graphs, the maximum luminance level and the luminance change width of the graph representing the luminance level change of the disturbance light are intentionally larger than those of the other graphs. This is to emphasize that the bright spot 2a and the irradiation light regions 3b and 4b can be detected without any trouble without being affected by disturbance light by applying the correlation value calculation method described below. .
[0048]
FIG. 8 shows the luminance level actual values for each frame time t0 to t6 in the four graphs of FIG. 7, and the correlation values (patterns) obtained by applying the calculation formula based on the following theory to these actual values. 2 is a list diagram showing correlation values of 1 and pattern 2. In this list diagram, the correlation value is obtained as follows. A formula for calculating the above-described series of values, that is, correlation values between vectors, is generally defined by correlation value = inner product / norm (as a value of 0 to 1). Therefore, the correlation value r is normalized so that the sign bit 0 corresponds to −1 to be 1,1,1, −1,1, −1, −1, and t0 to t6 are also normalized to have an average of 0, It is expressed as the following formula (A).
[0049]
[Expression 1]

[0050]
A more desirable form of the present invention is a method of performing a strict correlation value calculation according to the above formula (a), but in this embodiment, the illumination fluctuation pattern is a binary code consisting of blinking, Since the frequency of sudden dot data fluctuations that have a high degree of correlation with the detection target in the natural world is relatively small, even if a simple correlation degree calculation process such as the above formula (a) is not used, a simple correlation degree calculation process can be performed. Is possible. By the way, the method based on the following equation (A) makes the detection threshold setting a little more complicated than the exact correlation value calculation, but avoids costly operations such as square and square root as in the normal correlation value calculation. It is possible to ensure detection performance. Hereinafter, the value according to this equation is also referred to as a correlation value. The value range is 0 to 255 if it is an 8-bit luminance signal.
[0051]
Correlation value = Σ1 / C1-Σ0 / C0 (A)
[0052]
However, Σ1 is the sum of the luminance levels corresponding to the logic 1 of the pattern (ie, the sum of logic 1 symbols), C1 is the number of logic 1 of the pattern (ie, the number of logic 1 symbols), and Σ0 is the logic 0 of the pattern Is the sum of the luminance levels corresponding to (that is, the sum of logical 0 symbols), and C0 is the number of logical 0s in the pattern (ie, the number of logical 0 symbols).
[0053]
Since the pattern 1 has a pattern symbol of “1000101” and the pattern 2 has a pattern symbol of “1110100”, the expression (a) for each pattern is as shown in the following expressions (a-1, i-2): become. In the following, correlation value 1 is the correlation value of pattern 1, and correlation value 2 is the correlation value of pattern 2.
[0054]

[0055]
In FIG. 8, the disturbance light luminance levels are “130”, “240”, “200”, “190”, “220”, “108”, and “119” at t0 to t6, respectively. The luminance level of one light source region is “200”, “65”, “60”, “63”, “208”, “58”, “198”. Similarly, the luminance level of the second light source region is “125”, “126”, “124”, “29”, “124”, “28”, “30”, and the luminance level of the third light source region is “165”. , “155”, “152”, “70”, “154”, “80”, “71”.
[0056]
Hereinafter, a method of calculating correlation value 1 and correlation value 2 when these actual values (luminance levels) are applied to the above formulas (a-1, i-2) for each of pattern 1 and pattern 2 will be exemplified. To do.
[0057]
First, the correlation value of pattern 1 (correlation value 1) is calculated. That is, the correlation value (correlation value 1) of the disturbance light and the first to third light source regions is calculated using the above equation (A-1). The logic 1 symbol of pattern 1 is at the positions t0, t4, and t6, and the number (C1) thereof is “3”. The logic 0 symbol of pattern 1 is at the positions t1, t2, t3, and t5, and the number (C0) thereof is “4”.
[0058]
Therefore, Σ1 of disturbance light of pattern 1 is t0 + t4 + t6 = “130” + “220” + “119” = “469”, and Σ0 is t1 + t2 + t3 + t5 = “240” + “200” + “190” + “108”. Since “738”, C1 = “3”, and C0 = “4”, the correlation value 1 of the disturbance light of the pattern 1 is “469” / “3” − “from the above equation (A-1). 738 ”/“ 4 ”=“ − 28.17 ”.
[0059]
Also, Σ1 of the first light source region of pattern 1 is t0 + t4 + t6 = “200” + “208” + “198” = “606”, and Σ0 is t1 + t2 + t3 + t5 = “65” + “60” + “63” + “ 58 ”=“ 246 ”, and C1 =“ 3 ”and C0 =“ 4 ”. Therefore, the correlation value 1 of the first light source region of the pattern 1 is“ 606 ”/“ from the above equation (A-1). 3 "-" 246 "/" 4 "=" 140.50 ".
[0060]
Further, Σ1 of the second light source region of pattern 1 is t0 + t4 + t6 = “125” + “124” + “30” = “279”, and Σ0 is t1 + t2 + t3 + t5 = “126” + “124” + “29” + “ Since 28 ”=“ 307 ”, C1 =“ 3 ”, and C0 =“ 4 ”, the correlation value 1 of the second light source region of the pattern 1 is“ 279 ”/“ from the above equation (A-1). 3 "-" 307 "/" 4 "=" 16.25 ".
[0061]
Also, Σ1 of the third light source region of pattern 1 is t0 + t4 + t6 = “165” + “154” + “71” = “390”, and Σ0 is t1 + t2 + t3 + t5 = “155” + “152” + “70” + “ Since 80 ”=“ 457 ”, C1 =“ 3 ”, and C0 =“ 4 ”, the correlation value 1 of the third light source region of the pattern 1 is“ 390 ”/“ from the above equation (A-1). 3 "-" 457 "/" 4 "=" 15.75 ".
[0062]
Next, the correlation value of pattern 2 (correlation value 2) is calculated. That is, the correlation value (correlation value 2) of the disturbance light and the first to third light source regions is calculated using the above equation (A-2). The logic 1 symbol of pattern 2 is at the positions t0, t1, t2, and t4, and the number (C1) thereof is “4”. The logic 0 symbol of pattern 2 is at the positions t3, t5, and t6, and the number (C0) thereof is “3”.
[0063]
Therefore, Σ1 of disturbance light of pattern 2 is t0 + t1 + t2 + t4 = “130” + “240” + “200” + “220” = “790”, and Σ0 is t3 + t5 + t6 = “190” + “108” + “119”. = “417”, C1 = “4”, and C0 = “3”. Therefore, the correlation value 2 of the disturbance light of the pattern 2 is “790” / “4” − “ 417 "/" 3 "=" 58, 50 ".
[0064]
Further, Σ1 of the first light source region of pattern 2 is t0 + t1 + t2 + t4 = “200” + “65” + “60” + “208” = “533”, and Σ0 is t3 + t5 + t6 = “63” + “58” + “ Since 198 ”=“ 319 ”, C1 =“ 4 ”, and C0 =“ 3 ”, the correlation value 2 of the first light source region of the pattern 2 is“ 533 ”/“ from the above equation (A-2). 4 "-" 319 "/" 3 "=" 26.92 ".
[0065]
Also, Σ1 of the second light source region of pattern 2 is t0 + t1 + t2 + t4 = “125” + “126” + “124” + “124” = “499”, and Σ0 is t3 + t5 + t6 = “29” + “28” + “ Since 30 ”=“ 87 ”, C1 =“ 4 ”, and C0 =“ 3 ”, the correlation value 2 of the second light source region of the pattern 2 is“ 499 ”/“ from the above equation (A-2). 4 "-" 87 "/" 3 "=" 95.75 ".
[0066]
Further, Σ1 of the third light source region of pattern 2 is t0 + t1 + t2 + t4 = “165” + “155” + “152” + “154” = “626”, and Σ0 is t3 + t5 + t6 = “70” + “80” + “ 71 ”=“ 221 ”, C1 =“ 4 ”, and C0 =“ 3 ”. Therefore, the correlation value 2 of the third light source region of the pattern 2 is“ 626 ”/“ from the above equation (A-2). 4 "-" 221 "/" 3 "=" 82.83 ".
[0067]
When the correlation values (correlation value 1 and correlation value 2) obtained by the above calculation are compared, the correlation value 1 (“140.50”) of the bright spot 2a (first light source region) is maximum for pattern 1. Yes, for pattern 2, the correlation value 2 (“95.75”) of the irradiation light region 3b (second light source region) is the maximum, and the correlation value 2 (“82.75”) of the irradiation light region 4b (third light source region). 83 ") is the next.
[0068]
Therefore, if the threshold level for correlation value determination is, for example, “80.00”, for pattern 1, the correlation value 1 (“140.50” of the bright spot 2a (first light source region) indicating the maximum value is obtained. ”) Is extracted, and for the pattern 2, the correlation value 2 (“ 95.75 ”) of the irradiation light region 3b (second light source region) indicating the maximum value and the next higher magnitude are obtained. The correlation value 2 (“82.83”) of the irradiation light region 4b (third light source region) indicating the above is extracted.
[0069]
As a result, even when disturbance light is superimposed, the detection of the bright spot 2a and the irradiation light regions 3b and 4b can be performed without hindrance without being affected by the disturbance light, and the disturbance light Even when the maximum luminance level and the luminance change width are larger than those of the bright spot 2a and the irradiation light regions 3b and 4b (the list in FIG. 8 dares to have such a level relationship). The detection operation, that is, the detection of the bright spot 2a and the irradiation light regions 3b and 4b can be performed without any trouble.
[0070]
FIG. 9 is a diagram showing a processing flow of the image processing apparatus 1 in the first embodiment. In the figure, first, a required variation pattern set (setting) for the second light source 3 and the third light source 4 (and also the first light source 2 if necessary), and a processing correspondence list set with the variation pattern (setting). And initialization such as synchronization with each light source is performed (step S11). Next, it waits for image signal capture (frame capture) from the camera 6 (step S12). When the image is captured, the image data based on the image signal captured by the capture frame buffer 125 is converted into a buffer corresponding to the value (N) of the time slot counter unit 18 at that time (for example, if N = 0, a ring Copy to buffer 0) of buffer 123 (step S13).
[0071]
Next, modulo operation is performed (the time slot needs to periodically take a value of N = 0 to 6 when the variation pattern is represented by a length L = 7 bit pattern of 1110100) The value of the time slot counter unit 18 is updated (step S14), the correlation value image calculation and the determination binary value are performed for each detected variation pattern, and the image calculation is performed (step S15), and the variation region of the pattern 1 is detected. If this is the case (step S16), the pattern 1 processing instruction is taken out from the variation pattern / processing correspondence list storage 151, executed, and written in the result output image buffer 152 (step S17).
[0072]
When the stored contents of the variation pattern / process correspondence list storage unit 151 are as shown in FIG. 12, the above processing is stored in the upper left coordinates of the region irradiated with the variation pattern 1 (or the direct light emission region). A composite CG image or the like is written out. In the case of the combination of the light source and the code pattern shown in FIG. 6, a CG image for synthesis (an animation character, an image of a prop, etc.) is overwritten on the coordinates of the first light source set with the variation pattern 1. When the variation area of pattern 2 is detected (step S18), the pattern 2 processing instruction is similarly extracted from the variation pattern / processing correspondence list storage unit 151, the process is executed, and the result output image buffer 152 is processed. (Step S19).
[0073]
In the case of setting and actual use in FIG. 12 and FIG. 6, the upper body (4b) of the central standing figure irradiated with the second and third light sources set with the variation pattern 2 and the lower left Only the ball (3b) will be cut out. It should be noted that when a standing figure is cut out by specifying the processing contents, the display priority is higher in the CG synthesis performed in the variation pattern 1, and thus the hand that the standing figure is presented even if the corresponding process is performed in this pattern 2. CG and the like that are written and synthesized with pattern 1 will not become.
[0074]
Then, the non-detection area target process is executed from the variation pattern / process correspondence list storage unit 151 (step S20). In the case of the setting example in FIG. In the example of FIG. 6, the ball and the upper body of the standing figure are cut out in a white background, and a result output is obtained as if the synthesized CG image object (the bear stuffed animal) is in the right hand of the person. It is done.
[0075]
Note that it is possible to obtain completely different processing results from the variation pattern processing result list 151 of FIG. 4 while using exactly the same variation illumination.
[0076]
Thereafter, after outputting the result image (step S21), the frame capture (step S12) and the subsequent steps are repeated again.
[0077]
FIG. 10 is a specific flow of the above step S15. In this flow, the process of the fluctuation pattern 1 and the process of the fluctuation pattern 2 are executed in parallel. That is, first, an addition result of a portion corresponding to “1” of the fluctuation pattern in the image signal included in each of the buffers 0 to n of the ring buffer 123 is obtained, and an image divided by the number of added images is obtained as a normal image buffer. 121c (the normal image buffer 122c in the processing flow of variation pattern 2) is stored (step S15a). The purpose of this processing is to calculate the average luminance of the image at the timing when the luminance should be high (= illumination lighting) in the stored buffer image series.
[0078]
Next, an addition result of a portion corresponding to “0” of the variation pattern in the image signals included in each of the buffers 0 to n of the ring buffer 123 is obtained, and an image divided by the number of added images is obtained as a negative image buffer 121d. (In the processing flow of the variation pattern 2, it is stored in the normal image buffer 122d) (step S15b). The purpose of this process is to calculate the average luminance of each image at the timing when the luminance should be low (= illumination off).
[0079]
Next, the difference between the stored image in the positive image buffer and the stored image in the negative image buffer is taken, and the difference image is stored in the correlation value image buffer 121a (correlation value image buffer 122a in the process flow of the variation pattern 2) (step). S15c). Finally, the correlation value image buffer 121a (the normal image buffer in the processing flow of the fluctuation pattern 2) is determined by a fixed value determined by the illumination intensity / diffusion characteristic, the reflection coefficient of the detected object, and the spatial and temporal variability of the background environment. 122a) (step S15d).
[0080]
FIG. 11 is an operation conceptual diagram of the processing of the operation flow of FIG. By referring to the series of image buffers shown in the center, in the detection processes of the variation patterns 1 and 2, the image group corresponding to the bit “1” corresponding to the lighting timing and the bit “0” corresponding to the lighting off is added. The averaged difference image is shown as a correlation image of the corresponding pattern.
[0081]
From the above, according to the present embodiment, processing such as cutting out a desired area in a captured image or pasting another image at a desired position in the captured image is performed regardless of the shooting location. An image processing method and an image processing apparatus excellent in versatility that can be provided, and in particular, even in a shooting site that is exposed to ambient light, without being affected by the ambient light, It is possible to detect the bright spot 2b and the irradiation light areas 3b and 4b from the image signal, and it is possible to perform a particularly useful effect that it is possible to cut out an image object in a desired area, paste an image object at a desired position, and the like. Is obtained.
[0082]
Further, data (variation pattern / processing correspondence list) in which the variation pattern (blinking pattern) of the bright spot 2b and the irradiation light regions 3b and 4b is associated with the content of the image processing is stored in the variation pattern / processing correspondence list storage unit 151. The stored variation pattern of the bright spot 2b and the irradiation light areas 3b and 4b actually detected is compared with the variation pattern / processing correspondence list, and the type of image processing to be executed (image pasting) is checked from the list. In addition, since it is not necessary to designate the type of image processing on the imaging side (camera 6 side), the operability can be simplified.
[0083]
<Second Embodiment>
Next, a second embodiment of the present invention will be described.
First, in the first embodiment, by using the simple correlation value evaluation formula shown in the formula (A), the variation (= disturbance) of the brightness of other image regions is detected for the pattern to be detected. Only the target signal region can be detected without being affected.
[0084]
However, the simple correlation value evaluation formula (formula (A)) in the first embodiment is extremely effective in one time slot of detection pattern fluctuation in disturbance light fluctuation components, for example, in the case of more severe environmental conditions. In the case of including an image region in which near instantaneous fluctuations occur frequently, it is expected that it is difficult to completely satisfy the performance of “not affected by disturbance signals having large fluctuations, and therefore not erroneously detected”. .
[0085]
Assuming such severe environmental conditions, in the first embodiment, (1) the number of bits of the pattern is increased, or (2) the brightness of the illumination or light emitting light source is increased to give larger fluctuations. Thus, it is necessary to take one of the measures of raising the determination threshold for acquiring the binarized image in step S15d in FIG. 10 (= stricter the determination condition). Has the disadvantage of increasing the overhead in processing, and in the case of (2), the countermeasure has the disadvantage of lowering the detection sensitivity.
[0086]
Therefore, in the second embodiment, by adding a normalization process to the correlation value evaluation formula, it is easier to obtain a correlation value with good characteristics even when the above severe environmental conditions are assumed. The case where it is improved so that
The block configuration diagram and the like of the second embodiment are the same as those of the first embodiment, so illustrations thereof are omitted, and the processing flow and the like are related to the correlation value in the second embodiment. Since the processing is the same except for the processing peculiar to the form of FIG.
[0087]
The point of the second embodiment is to use the following equation (c) instead of the simple correlation value evaluation equation (equation (a)) shown in the first embodiment.
[0088]
Correlation value = ([formula (A)] / ΔW) × const (C)
ΔW = max (t0... T6) −min (t0... T6) (D)
[0089]
However, max (argument) is a max function that returns the maximum value of the data given by the argument, and min (argument) is a min function that returns the minimum value of the data given by the argument. In equation (c), const is a multiplication coefficient (for example, const = “255”). const is multiplied to convert the maximum value of 0 to 1 into an easy-to-handle dot signal luminance range of 0 to 255. It goes without saying that the value of const is other than “255” depending on the target range.
[0090]
As in the first embodiment, the pattern symbol of pattern 1 is “1000101”, and the pattern symbol of pattern 2 is “1110100”. The formula (c) for each pattern is as the following formulas (c-1, c-2). In the following, correlation value 1 is the correlation value of pattern 1, and correlation value 2 is the correlation value of pattern 2.
[0091]

[0092]
In FIG. 13, the luminance levels of the first light source region are “200”, “65”, “60”, “63”, “208”, “58”, “198” at t0 to t6, respectively. Similarly, the luminance level of the second light source region is “125”, “126”, “124”, “29”, “124”, “28”, “30”, and the luminance level of the third light source region is “165”. ”,“ 155 ”,“ 152 ”,“ 70 ”,“ 154 ”,“ 80 ”,“ 71 ”.
[0093]
Examples of the brightness level values of the first to third light source regions are the same as those in the first embodiment (FIG. 8), but the brightness level of the disturbance light is assumed to be severe environmental conditions described above. For example, “250”, “240”, “255”, “100”, “30”, “80”, “119” are different. Note that the peripheral edge of the second light source region refers to an edge portion of the second light source region (an irradiation region for the standing figure 5c). This is intentionally added to explain the inconvenience of the first embodiment at this edge portion.
[0094]
Here, the temporal change in the luminance level of each representative coordinate point in FIG. 13 is as shown in FIG. In FIG. 14, a hatched square symbol graph represents a change in luminance level of the bright spot 2a (first light source region), and a triangular symbol graph represents an irradiation light region 3b (second light source region; irradiation region to the standing figure 5c). X represents a change in luminance level of the irradiation light region 4b (third light source region; irradiation region to the spherical object 5a), and a circle symbol represents an edge (first of the irradiation light source region 4b). 3 light source region periphery; irradiation region edge to standing figure 5c) represents a luminance level change, and a black rhombus symbol graph represents a luminance level change of disturbance light (flicker lighting of TV 5b). It is assumed that the standing figure 5c is wearing black clothes. That is, the periphery of the third light source region represents the edge of the black clothes.
[0095]
When the formula (A) in the first embodiment is applied to the example of FIG. 13, disturbance light cannot be reliably excluded and the periphery of the second light source region cannot be extracted as will be described in detail below. However, when the formula (c) in the second embodiment is applied, such a problem does not occur.
[0096]
Hereinafter, in the example of FIG. 13, the correlation value when the formula (a) in the first embodiment is applied is referred to as “correlation value 1A” for pattern 1 and “correlation value 2A” for pattern 2. Furthermore, the correlation value when the formula (c) of the second embodiment is applied is referred to as “correlation value 1B” for pattern 1 and “correlation value 2B” for pattern 2, and the respective correlation values. The calculation method of (correlation value 1A, 2A, 1B, 2B) is illustrated.
[0097]
In the figure, the correlation value 1A is the correlation value of the pattern 1 calculated by applying the formula (A), the correlation value 2A is the correlation value of the pattern 2 calculated by applying the formula (A), and these correlation values 1A and 2A correspond to the correlation values in the first embodiment. Further, the correlation value 1B is the correlation value of the pattern 1 calculated by applying the formula (c), the correlation value 2B is the correlation value of the pattern 2 calculated by applying the formula (c), and these correlation values 1B, 2B is unique to the second embodiment.
[0098]
Further, the correlation value 1A and the correlation value 2A are the same as the calculation results (see FIG. 8) in the first embodiment, except for disturbance light and the second light source region periphery. That is, the correlation values 1A of the first light source region to the third light source region are “140.50”, “16.25”, “15.75”, respectively, and the correlation values 2A are “26.92”, “95.75”, respectively. "82.83", and these values are the same as the calculation results (see FIG. 8) in the first embodiment.
[0099]
The correlation value 1A and the correlation value 2A of the disturbance light and the second light source region periphery, which are different from those of the first embodiment, are obtained as follows by applying the formula (A). That is, in FIG. 13, the luminance level of disturbance light is “250”, “240”, “255”, “100”, “30”, “80”, “119” at t0 to t6, respectively. The luminance levels around the second light source region are “70”, “69”, “69”, “20”, “68”, “21”, “20”.
[0100]
The logic 1 symbol of pattern 1 is at the positions t0, t4, and t6, and the number (C1) thereof is “3”. The logic 0 symbol of pattern 1 is at the positions t1, t2, t3, and t5, and the number (C0) thereof is “4”. Therefore, Σ1 of disturbance light of pattern 1 is t0 + t4 + t6 = “250” + “30” + “119” = “399”, and Σ0 is t1 + t2 + t3 + t5 = “240” + “255” + “100” + “80” = “675”, C1 = “3”, and C0 = “4”. Therefore, the disturbance light correlation value 1A of the pattern 1 is “399” / “3” − “ 675 ”/“ 4 ”=“ − 35.75 ”.
[0101]
Further, Σ1 at the periphery of the second light source region of pattern 1 is t0 + t4 + t6 = “70” + “68” + “20” = “158”, and Σ0 is t1 + t2 + t3 + t5 = “69” + “69” + “20” + Since “21” = “179”, C1 = “3”, and C0 = “4”, the correlation value 1A around the second light source region of the pattern 1 is “158” from the above equation (A-1). /“3”−“179”/“4”=“7.92 ”.
[0102]
The logic 1 symbol of pattern 2 is at the positions t0, t1, t2, and t4, and the number (C1) thereof is “4”. The logic 0 symbol of pattern 2 is at the positions t3, t5, and t6, and the number (C0) thereof is “3”. Therefore, Σ1 of disturbance light of pattern 2 is t0 + t1 + t2 + t4 = “250” + “240” + “255” + “30” = “775”, and Σ0 is t3 + t5 + t6 = “100” + “80” + “119”. = “299”, C1 = “4”, and C0 = “3”. Therefore, the correlation value 2A of the disturbance light of the pattern 2 is “775” / “4” − “from the above equation (A-2). 299 ”/“ 3 ”=“ 94.08 ”.
[0103]
Further, Σ1 at the periphery of the second light source region of pattern 2 is t0 + t1 + t2 + t4 = “70” + “69” + “69” + “68” = “276”, and Σ0 is t3 + t5 + t6 = “20” + “21” + Since “20” = “61”, C1 = “4”, and C0 = “3”, the correlation value 2A of the second light source region periphery of the pattern 2 is “276” from the above equation (A-2). /“4”−“61”/“3”=“48.67 ”.
[0104]
Next, formula (c) in the second embodiment is applied to the example of FIG. 13 to calculate all of “correlation value 1B” of pattern 1 and “correlation value 2B” of pattern 2.
First, the ΔW term of the formula (c) is calculated. In FIG. 13, focusing on the symbol having the maximum value among t0 to t6, the disturbance light is “255” at t2, the first light source region is “208” at t4, the second light source region is “126” at t1, The three light source regions are “165” at t0, and the periphery of the second light source region is “70” at t0. Focusing on the symbol having the minimum value among t0 to t6, the disturbance light is “30” at t4, the first light source region is “58” at t5, the second light source region is “28” at t5, and the third light source. The area is “70” at t3, and the periphery of the second light source area is “20” at t3.
[0105]
Therefore, the disturbance light ΔW is called ΔW0, the first light source region ΔW is called ΔW1, the second light source region ΔW is called ΔW2, the third light source region ΔW is called ΔW3, and the second light source region periphery ΔW is called ΔW4. Then, from equation (D), ΔW0 = “255” − “30” = “225”, ΔW1 = “208” − “58” = “150”, ΔW2 = “126” − “28” = “98” ΔW3 = “165” − “70” = “95” and ΔW4 = “70” − “20” = “50”.
[0106]
<Correlation value 1B>
The correlation value (correlation value 1B) of the pattern 1 in the example of FIG. 13 is given by a value obtained by dividing the correlation value 1A by ΔW and multiplying by const = “255” according to the equation (c-1). Accordingly, since the correlation value 1A of the disturbance light is “−35.75” and ΔW0 is “225”, “−40.52” is obtained.
[0107]
Similarly, since the correlation value 1A of the first light source region is “140.50” and ΔW1 is “150”, “238.85” is obtained, and the correlation value 1A of the second light source region is “16.25”. , ΔW2 is “98”, so “42.28” is obtained. Further, since the correlation value 1A of the third light source region is “15.75” and ΔW3 is “95”, “42.28” is obtained, and the correlation value 1A of the periphery of the second light source region is “7.92”. , ΔW4 is “50”, so “40.39” is obtained.
[0108]
<Correlation value 2B>
The correlation value (correlation value 2B) of the pattern 2 in the example of FIG. 13 is given by a value obtained by dividing the correlation value 2A by ΔW and multiplying by const = “255” from the equation (c-2). Accordingly, the correlation value 2A of disturbance light is “94.08” and ΔW0 is “225”, so “106.63” is obtained.
[0109]
Similarly, since the correlation value 2A of the first light source region is “26.92” and ΔW1 is “150”, “45.76” is obtained, and the correlation value 2A of the second light source region is “95.75”. , ΔW2 is “98”, so “249.15” is obtained. Further, since the correlation value 2A of the third light source region is “82.83” and ΔW3 is “95”, “222.34” is obtained, and the correlation value 2A of the periphery of the second light source region is “48.67”. , ΔW4 is “50”, so “248.22” is obtained.
[0110]
<Evaluation of each correlation value>
By the above calculation, all the correlation values (correlation values 1A, 2A, 1B, 2B) in FIG. 13 are clarified. Of these, the correlation values 1A and 2A are obtained by applying the formula (A), that is, according to the first embodiment.
[0111]
An inconvenient example can be seen in FIG. 13 in the correlation value 2A (“94.08”) of disturbance light and the correlation value 2A (“48.67”) of the periphery of the second light source region.
In other words, since the disturbance light correlation value 2A is a large value of “94.08”, it exceeds the threshold value (for example, “80.00”) and cannot be separated. Further, since the correlation value 2A at the periphery of the second light source region is a small value of “48.67”, it is below a threshold value (for example, “80.00”), and there is a disadvantage that extraction cannot be performed. .
[0112]
On the other hand, the correlation value obtained by applying the formula (c), that is, the correlation values 1B and 2B obtained by the second embodiment, for example, the correlation value 2B of disturbance light is “106.63”. Since the correlation value 2B at the periphery of the second light source region is “248.22”, by using an appropriate threshold value (for example, “200.00”), the correlation value 2B of disturbance light (“106.63”) is used. ”) Can be excluded, and the correlation value 2B (“ 248.22 ”) around the second light source region can be extracted, thereby eliminating the disadvantages of the first embodiment.
[0113]
As described above, in the second embodiment, in the case of more severe environmental conditions, for example, instantaneous fluctuations very close to one time slot of the detection pattern fluctuation frequently occur in the fluctuation component of the disturbance. In the case where the image region is included, a special merit is obtained that the performance of “not affected by a disturbance signal having a large variation and therefore not erroneously detected” can be completely satisfied. .
[0114]
Note that the processing of the second embodiment is not subject to detection because there is a possibility that a minute variation sequence such as a sampling level variation for each frame due to camera noise or the like may be multiplied by a larger value in the normalization processing. A power point may have a large correlation value by chance. However, since such a point that is not a detection target has a characteristic of appearing in isolation, it can be solved by adding a median filter before binarization in step S15d in FIG. 10 as a countermeasure. .
[0115]
In addition, when the camera characteristics are poor, before normalization in advance, a process of truncating the first method correlation value (for example, 5 or less in the range of 0 to 255) below to some extent may be added.
[0116]
In the first embodiment, the signal strength itself is an important factor for determining the binarization determination threshold. For example, for a black object, the binarization determination threshold is set accordingly. ) Appropriate settings were required depending on the usage conditions such as lowering. However, according to the second embodiment, since the determination threshold represents the same pattern regardless of the magnitude of the fluctuation, it has an advantage that it is not affected by the magnitude of ΔW obtained by the environmental condition. The binarization determination threshold that has been set can be set.
[0117]
Thus, according to the second embodiment, only the “pattern similarity” can be more strictly evaluated by the correlation coefficient calculation. Therefore, even if there is a disturbance similar to the variation pattern, or even in the dot value variation sequence around the second light source region where the variation is small, it is possible to perform a favorable correlation value evaluation.
[0118]
In the above embodiment, since the image is processed using a plurality of frames, the edge portion may be shifted when the detection target moves, and as a result, only the overlapping area in each frame can be detected. However, it is possible to keep the detection error associated with the movement within an allowable range by increasing the frame rate of the image sensor.
[0119]
Further, when the detection target moves, it is preferable to calculate motion vector data between a plurality of frame buffers, refer to the motion vectors between frames, and extract a set of corresponding points.
[0120]
As described above, according to the second embodiment, compared to the first embodiment, a disturbance of a more approximate pattern and a target pattern having a weaker fluctuation can be well cut out. . Compared to the exact correlation value as in equation (a), the processing load can be greatly reduced, and the process for determining the presence or absence of a signal is less susceptible to the influence of use conditions and environmental conditions. It can be.
[0121]
【The invention's effect】
  According to the present invention, it is possible to attach an arbitrary image to a light emitting area in the image only by including a light emitting area whose luminance varies in a predetermined pattern in a plurality of image signals arranged in the time axis direction. Or an image in the region can be cut out. Therefore,In the imageCut out the desired area ofIn the imageProcessing such as pasting other images to the desired position.
[Brief description of the drawings]
FIG. 1 is a principle block diagram of an embodiment.
FIG. 2 is a specific example of a self-luminous light source and an external illumination light source.
FIG. 3 is a diagram illustrating a usage example when four shooting subjects are included in a shooting range 5;
FIG. 4 is a diagram showing a specific configuration of the image processing apparatus 1 and a first light source 2 to a third light source 4;
FIG. 5 is a communication configuration diagram between the image processing apparatus 1 and each light source.
6 is a diagram showing an image storage state of the ring buffer 123. FIG.
FIG. 7 is a diagram showing temporal changes in luminance levels.
FIG. 8 is a list diagram showing luminance level actual values for each frame time t0 to t6 and correlation values obtained by applying these actual values.
9 is a diagram showing a processing flow of the image processing apparatus 1. FIG.
FIG. 10 is a diagram showing a specific flow of step S15 of FIG.
11 is an operation concept diagram of the flow of FIG.
FIG. 12 is a diagram illustrating an example of a variation pattern / processing correspondence list;
FIG. 13 is a list diagram showing luminance level actual values for each frame time t0 to t6 and correlation values obtained by applying these actual values.
FIG. 14 is a diagram illustrating a temporal change in luminance level.
[Explanation of symbols]
Ga spherical object image
Gb image object
Gc Upper body image
1 Image processing device
2 First light source (Self-emitting light source)
2a bright spot
3 Second light source (external illumination light source)
3b Irradiation light area
4 Third light source (external illumination light source)
4b Irradiation light area
5 Shooting range
5a Spherical object
5b TV
5c standing figure
6 Camera
11 Light source controller
12 Image detector
13 Detection region shape recognition processing unit
14 Object coding processor
15 Effect processing equipment
16 Display device
17 CPU
18 Time slot counter
41 Synchronous trigger signal extractor
43 Fluctuation pattern register
111 Code clock
112 Synchronous trigger signal output section
121a Correlation value image buffer
121e Fluctuation pattern register
122a Correlation value image buffer
122e Fluctuation pattern register
123 Ring buffer
125 capture frame buffer

Claims (5)

An image processing method for processing a plurality of image signals arranged in a time axis direction, including a light emitting region whose luminance varies with a predetermined pattern,
A first step of patterning the symbols into logical one symbol and the logic 0 symbol by binarizing the plurality of image signals in the time axis direction,
The first patterned symbolized logical one symbol in step, and a second step of obtaining the respective sum values of logic zero symbol,
A third step of extracting an image of the light emitting area by performing pattern matching based on a difference value between the total value of the logical 1 symbols obtained in the second step and the total value of the logical 0 symbols;
An image processing method comprising: a fourth step of pasting an arbitrary image at the position of the image extracted in the third step , or cutting out the extracted image. And further including a fifth step of obtaining a difference value between the maximum symbol and the minimum symbol in each of the logical symbols converted into pattern symbols in the first step ,
In the third step, the difference between the total value of the logical 1 symbols determined in the second step and the total value of the logical 0 symbols is calculated by calculating the difference between the maximum symbol and the minimum value determined in the fifth step. The image processing method according to claim 1, wherein pattern matching is performed based on a result normalized with a difference value from a symbol to extract an image of the light emitting area. In an image processing apparatus for processing a plurality of image signals arranged in the time axis direction, including a light emitting region whose luminance varies with a predetermined pattern,
First means for binarizing the plurality of image signals in a time axis direction to form pattern symbols into logical 1 symbols and logical 0 symbols;
A second means for obtaining a total value of each of the logical 1 symbol and the logical 0 symbol that are pattern-symbolized by the first means;
Third means for performing pattern matching on the basis of a difference value between the total value of the logical 1 symbols obtained by the second means and the total value of the logical 0 symbols, and extracting an image of the light emitting area;
A fourth means for pasting an arbitrary image at the position of the image extracted by the third means, or for cutting out the extracted image;
An image processing apparatus comprising: A fifth means for obtaining a difference value between the maximum symbol and the minimum symbol in each of the logical symbols converted into pattern symbols by the first means;
The third means calculates the difference between the sum value of the logical 1 symbols obtained by the second means and the sum value of the logical 0 symbols, and determines the maximum symbol and the minimum symbol obtained by the fifth means. The image processing apparatus according to claim 3, wherein pattern matching is performed based on a result obtained by normalizing with a difference value between the two to extract an image of the light emitting area. A computer that performs image processing on a plurality of image signals arranged in the time axis direction, including a light-emitting region whose luminance varies in a predetermined pattern,
First means for binarizing the plurality of image signals in a time axis direction to form a pattern symbol into a logical 1 symbol and a logical 0 symbol;
A second means for obtaining a total value of each of the logical 1 symbol and the logical 0 symbol that are pattern-symbolized by the first means;
Third means for extracting an image of the light emitting region by performing pattern matching based on a difference value between the total value of the logical 1 symbols and the total value of the logical 0 symbols obtained by the second means;
Fourth means for pasting an arbitrary image at the position of the image extracted by the third means, or for cutting out the extracted image
An image processing program that functions as an image processing program.

Images