JP2004109246A - Projection system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a projection system for generating a proper image in accordance with the condition of a projection surface. <P>SOLUTION: In the projection system, an image is obtained by a camera section 20 picking up the projection surface on which a pattern whose luminance changes stepwise and with regular intervals is projected by a projector section 10. Then, in a camera controlling section 200, a correction value for correcting the gradation property of the obtained image is set so that the gradual change amount of the brightness of the area where the pattern is projected is of the regular intervals. Then, in a γ correction circuit 218, the gradation property of pickup image obtained by the camera section 20 is corrected based on the correction value set at the camera controlling section 200, and an element drive controlling function 101 corrects the gradation property of a projected image projected from the projector section 10 on the projection surface. <P>COPYRIGHT: (C)2004,JPO

Description

【０１５８】
ここでは、Ｘｂ，Ｘｃ、Ｍｂ，Ｍｃも既知であるため、式（１）より撮像面上の水平方向（Ｘ方向）の画素位置Ｘｉにおける縮小倍率Ｍｉを求めることができる。例えば、Ｍｃ＝０．９、Ｍｂ＝１．１、Ｘｂ＝１、Ｘｃ＝−１である場合、Ｘｉ＝０．５とすると、式（１）より、縮小倍率Ｍｉ＝０．８６３５と算出される。
【０１５９】
このように、補正値算出機能３０２では、撮像素子２１３の撮像面上のＸ方向の全ての画素について局所的な縮小倍率Ｍｉを算出する。そして、補正値算出機能３０２によって算出された歪み補正値に基づいて、画像補正機能３０３によって、撮影画像の歪みが補正される。具体的には、画素位置Ｘｉの同じ垂直ラインにおける画素を縮小倍率Ｍｉで縮小処理を行う。縮小処理としては、例えば、ニアエストネイバー法、バイリニア法、バイキュービック法等、一般的な画像処理によって実施することができる。即ち、画像補正機能３０３が、評価値演算機能２０４で算出された評価値に基づいて、カメラ部２０によってホワイトボードＷを撮影することにより得られる撮影画像について、歪みを補正する手段として機能する。
【０１６０】
このような撮影画像の歪み補正によって、全体的に画像中に表されるものは小さくなるが、台形歪みのない撮影画像を得ることができ、ホワイトボードＷへの書き込み等を見易い状態で保存や印刷等することができる。
【０１６１】
また、プロジェクター制御部１００の補正値算出機能１０２は、距離算出機能２０６において算出した撮影倍率比Ｌ２／Ｌ１，Ｌ２／Ｌ３に基づいて、投影画像の台形歪みを補正するための補正値を算出することもできる。これは、プロジェクター部１０とカメラ部２０とは一体となっているため、カメラ部２０の撮像素子２１３からホワイトボードＷまでの距離と、プロジェクター部１０の表示素子１２３からホワイトボードまでの距離とが、ほぼ一致することを利用するものである。そして、ここでは、従来のプロジェクターに広く採用されている台形歪み補正と同様な方法によって補正値を算出することができる。
【０１６２】
例えば、図１１に示すような投影画像を、図１２に示すような投影画像とするためには、図１１に示す投影画像のうち＋Ｘ方向の大きく投影される部分が縮小されるように投影される必要性がある。補正値算出機能１０２では、歪み補正値算出機能３０２において撮像素子２１３の撮像面上のＸ方向の全ての画素について縮小倍率Ｍｉを算出したのと同様の方法で、フレームメモリ１１２に記憶される画像データについて、Ｘ方向の全ての画素について縮小倍率Ｍｉ’を算出することができる。ここでは、プロジェクター制御部１００において、この縮小倍率Ｍｉ’が歪み補正値として設定される。
【０１６３】
その後は、プロジェクター制御部１００において設定された歪み補正値に基づいて、スケーラー１１３によって、投影画像の歪みが補正される。例えば、画素位置Ｘｉ’の同じ垂直ラインにおける画素を縮小倍率Ｍｉ’で縮小処理を行う。即ち、スケーラー１１３が、評価値演算機能２０４で算出された評価値に基づいて、プロジェクター部１０によってホワイトボードＷに投影される投影画像について、歪みを補正する手段として機能する。
【０１６４】
なお、ここで撮影画像および投影画像の歪みを正しく補正するためには、図１８に示すように、ホワイトボードＷ上の３点の距離を求める必要性がある。つまり、プロジェクター部１０から投影する焦点検出用のパターンには、全ての部分が直線状に並ばない３箇所の焦点検出用の部分が必要である。また、投影レンズ１０Ｒや撮影レンズ２０Ｒの各種収差等の誤差要因を勘案すると、図１５に示すように、焦点検出用パターンには、中心と４隅の計５箇所の焦点検出用の部分が存在する方が好ましい。
【０１６５】
また、ここでは、コントラスト方式のオートフォーカス動作において実施されているものと同様に、焦点検出用の領域Ａ１’〜Ａ５’内の各隣接画素間に関するコントラスト値の総和として評価値を求めることによって距離を測定する。よって、焦点検出用の部分に描かれる模様は、撮影レンズ２０Ｒの駆動によって評価値が変化するものであれば、点や線等何でも良い。しかし、より正しく距離を測定して、歪みをより良く補正するためには、焦点検出用の部分に描かれる模様は、撮影レンズ２０Ｒの駆動によって評価値が変化し易いものが好ましい。なお、撮影レンズ２０Ｒの駆動によって評価値が変化し易い模様としては、例えば、図１５に示す線が多く並んだような模様が挙げられる。
【０１６６】
なお、以上では、図１０に示すように、ホワイトボードＷがＺ軸を回転軸として、ＸＺ平面に対して角度θだけ回動させた状態を例にとって説明したが、Ｘ軸を回転軸としてＸＺ平面に対して角度θだけ回動させた状態であっても、同様にして、撮影画像および投影画像の歪みを補正することができる。よって、ここでは、縦および横方向の台形歪み補正の両方ともを実施することができる。
【０１６７】
また、ホワイトボードＷをＺ軸を回転軸として、ＸＹ平面に対して角度θだけ回動させた上に、Ｘ軸を回転軸として回動させた状態であっても、距離算出機能２０６において、カメラ付きプロジェクター２からホワイトボードＷに投影される部分Ａ１〜Ａ５までの距離をそれぞれ算出することができ、その値に基づいて、撮影画像および投影画像の歪みを補正することができる。例えば、ホワイトボードＷに投影される部分Ａ１〜Ａ５までの距離が分かれば、内挿および外挿によって、カメラ付きプロジェクター２から投影画像上の任意の点までの距離を推定することができる。そして、その距離に基づいて、撮影画像の画素毎に縦方向および横方向の縮小倍率を求めることによって、撮影画像および投影画像の歪みを補正することができる。よって、ここでは、プロジェクター部１０やカメラ部２０と、ホワイトボードＷとの相対的な位置関係等から生じる投影画像や撮影画像のあらゆる方向への歪みを補正することができる。
【０１６８】
なお、上述したような投影画像に対する歪み補正値の設定動作は、投影システム１の使用開始時、および、使用開始時から一定時間経過する毎に行われるようにしても良い。このような構成とすることで、途中でホワイトボードＷが動いた場合にも対応することができる。また、上述したように、撮影画像に対する歪み補正値は、撮影が行われる毎に行われるため、途中でホワイトボードＷが動いた場合にも対応することができる。なお、ここでは、焦点検出用パターンを投影するタイミングと、撮影するタイミングとが同期しており、さらに、焦点検出用パターンを投影する時間はごく短時間であるため、投影画像を見ている人に気づかれず、投影画像が見づらくなることがない。
【０１６９】
以上のように、ここでは、パソコン３０において、投影プログラムが実行されているときに、ユーザーがパソコン３０のキーボード等を種々操作すると、焦点検出用の部分（部分Ａ１〜Ａ５）を複数含むパターン（焦点検出用パターンＡＦＰ）をホワイトボードＷに投影する。ホワイトボードＷに投影されたパターンを撮影して入力された画像における複数の焦点検出用の部分に対応する領域（領域Ａ１’〜Ａ５’）について、それぞれ撮影レンズ２０Ｒの合焦状態に関する評価値を求める。さらに、その評価値に基づいて、ホワイトボードＷに投影される投影画像、および、ホワイトボードＷを撮影することにより得られる撮影画像のうちの少なくとも一方の画像について、歪みを補正する。その結果、プロジェクター部１０やカメラ部２０と、ホワイトボードＷとの相対的な位置関係等から生じる投影画像や撮影画像の歪みを補正することができる。
【０１７０】
また、ここでは、１つのパターン（焦点検出用パターンＡＦＰ）を投影して、そのパターンを撮影して入力された画像に基づいて、投影画像と撮影画像の両方の歪みの補正用の補正値を算出することができるため、効率良く、投影画像と撮影画像の両方の歪み補正用の補正値を算出することができる。
【０１７１】
＜（１−３−５）　会議内容の記録＞
図１９は、ホワイトボードＷ上の状態を例示する模式図である。図１９に示すように、枠ＰＳ２内は投影画像が投影される投影領域であり、その投影画像に合わせて、マーカーＭＫによる書き込みＭＲが行われている。
【０１７２】
そして、パソコン３０において、投影プログラムが実行されて、図１９に示すような投影画像が投影されているときに、ユーザーがパソコン３０のキーボード等を種々操作すると、ホワイトボードＷ上に投影されている投影画像および書き込みＭＲ等を併せて記録する記録動作が実行される。
【０１７３】
記録動作が開始すると、ＰＣ制御部３００からカメラ制御部２００に記録用の撮影を実施するための撮影信号が伝送される。そして、カメラ制御部２００は、プロジェクター部１０にコマンドを送信し、垂直同期信号に同期して次のフィールドから、パソコン３０からのアナログＲＧＢ信号の入力に拘わらず、フレームメモリ１１２に記憶されるデジタル画像データが、略均一な単一色である全面白色となるように設定する。したがって、次の垂直同期信号に同期して、プロジェクター部１０が、ホワイトボードＷに、単一色の光を投影する。
【０１７４】
これに対して、カメラ制御部２００が、撮像素子駆動回路２１４に撮影のトリガー信号を出し、次の垂直同期信号に同期して、カメラ部２０が、単一色である全面白色の光が投影されたホワイトボードＷを撮影することによって画像を取得する。このとき、カメラ制御部２００の制御の下で、露光量が撮像素子２１３の許容量を超過して、撮像素子２１３の出力が飽和しないように、レンズ駆動回路２１２が撮影レンズ２０Ｒの絞り値を所定の絞り値に設定する。なお、ここでは、シャッタースピードが垂直同期信号によって決まり、プロジェクター部１０からの発光は全面白色の光であり、発光量も決まるため、撮像素子２１３の出力が飽和しないように、所定の絞り値を容易に設定することができる。
【０１７５】
図２０は、ホワイトボード上の書き込みを撮影した撮影画像ＰＧを例示する模式図である。図２０に示すように、ホワイトボードＷ上の枠ＰＳ３内に全面白色の光が照射され、書き込みＭＲを鮮明に撮影することができる。
【０１７６】
図２０に示すような撮影画像ＰＧは、カメラ制御部２００の制御の下で、ＰＣ制御部３００の画像補正機能３０３に伝送される。
【０１７７】
次に、画像補正機能３０３では、投影するための投影原画像と、撮影画像ＰＧに係る画像とをデータ合成する際に、ずれのない合成画像を生成するため、上述したように、撮影画像ＰＧの歪み補正を行う。その後、撮影画像ＰＧのうち投影領域に対応する領域を、撮影画像の全体を占めるように、投影領域の抽出・拡大処理を行う。例えば、図２０に示すように、ホワイトボードＷ上の投影領域には略均一な全面白色の光が照射され、その周辺は投影領域よりも十分暗いため、輝度変化に基づいて、撮影画像ＰＧのうち、投影領域とその周辺部との境界（枠ＰＳ３）を容易に検出することができる。よって、画像補正機能３０３では、枠ＰＳ３内の画像を抽出し、その抽出した画像を撮影画像の全体となるように拡大処理を施す。
【０１７８】
領域抽出機能３０４は、画像補正機能３０３によって歪み補正、および投影領域の抽出・拡大処理が施された撮影画像から、マーカーＭＫ等を使用してマニュアルでホワイトボードＷに直接書き込まれる等して直接に表示された視覚要素（文字、図形、色彩等）に対応する領域を抽出する。例えば、図２０に示すように、ホワイトボードＷ上の投影領域内には全面白色の光が照射される場合は、書き込みＭＲと、その周辺とは色が全く異なるため、撮影画像ＰＧから書き込みＭＲに対応する領域を容易に抽出することができる。即ち、領域抽出機能３０４が、カメラ部２０によって取得される画像について、ホワイトボードＷ上に直接に付された視覚要素に対応する領域を抽出する手段として機能する。
【０１７９】
なお、ここでは、書き込みＭＲに対応する領域を抽出する場合を例にとって説明したが、ホワイトボードＷに紙の資料や写真等を直接貼付して視覚要素を与えた際にも同様な方法によって、撮影画像から資料や写真等に対応する領域を抽出することができる。
【０１８０】
図２１は、ホワイトボードＷ上に投影するための投影原画像ＰＰを例示する模式図である。なお、図２１では、パソコン３０の記憶部３２に記憶される投影原画像ＰＰを示しており、図１９に示すホワイトボードＷ上に投影された投影画像に対応する投影原画像である。
【０１８１】
次に、画像合成機能３０５は、領域抽出機能３０４によって抽出された領域と、プロジェクター部１０によって投影原画像ＰＰとをデジタル情報としてデータ合成して画像データ（合成画像）を生成し、記憶部３２に記憶する。図２２は、投影原画像ＰＰと書き込みＭＲとをデータ合成した画像（合成画像）ＣＰを例示する模式図である。図２２に示す合成画像ＣＰは、図１９に示すホワイトボードＷ上の状態を反映した画像となっている。即ち、画像合成機能３０５が、領域抽出機能３０４によって抽出した領域と、プロジェクター部１０によってホワイトボードＷに投影するための投影原画像とをデータ合成する手段として機能する。
【０１８２】
したがって、ここでは、単一色の光をホワイトボードＷに照射して、ホワイトボードＷを撮影して画像を取得する。そして、取得される画像から、ホワイトボードＷ上に直接書き込まれたり、添付される等して直接に付された視覚要素に対応する領域を抽出する。そして抽出した領域とホワイトボードＷに投影するための投影原画像とデータ合成する。その結果、ホワイトボードＷに投影するための投影原画像と、ホワイトボードＷ上に直接書き込まれた文字や線や添付された資料等とをデータ合成した高品質な画像を作成することができる。即ち、ホワイトボードＷに投影される情報と、ホワイトボードＷに直接付加される視覚要素の情報とを併せた高品質の画像を取得可能な投影システムを提供することができる。
【０１８３】
以上、第１実施形態に係る投影システム１においては、外光や、ホワイトボードＷの傾き等を勘案して、投影画像や撮影画像のホワイトバランスや階調特性や歪み等を補正する。また、ホワイトボードＷに投影される情報と、ホワイトボードＷに直接付加される情報とを併せた高品質の画像を取得する。よって、ホワイトボードＷ（投影面）の状況に応じた適正な画像を投影および撮影可能であるとともに、ホワイトボードＷ（投影面）の状況に応じた適正な画像を生成可能な投影システムを提供することができる。
【０１８４】
＜（２）　第２実施形態＞
＜（２−１）　撮影画像の歪み補正＞
上述したように、ホワイトボードＷを斜め撮影した際に、ホワイトボードＷの正面から撮影したような撮影画像を得るためには、撮影画像の歪み補正をしなければならない。また、ホワイトボードＷに対して斜め投影している際に、ホワイトボードＷへの書き込み等を撮影して得られた撮影画像と、投影するための投影原画像とをデータ合成するためには、投影原画像と整合するように撮影画像の歪み補正をしなければならない。
【０１８５】
このような撮影画像の歪み補正を行うために、第１実施形態に係る投影システム１では、焦点検出用パターンＡＦＰを投影し、そのパターンを撮影して入力された画像に基づいて撮影画像の歪みを補正したが、第２実施形態に係る投影システム１Ａでは、撮影画像の歪み補正を行うために、第１実施形態とは異なるパターンをホワイトボードＷに投影する。
【０１８６】
したがって、第２実施形態に係る投影システム１Ａでは、カメラ制御部２００およびＰＣ制御部３００の機能が第１実施形態に係る投影システム１と異なる。具体的には、図２３に示すように、第２実施形態に係るカメラ制御部２００Ａには、第１実施形態に係る各種機能に加えて、対応点抽出機能２０７が付加されている。また、図２に示すように、第２実施形態に係るＰＣ制御部３００Ａでは、補正値算出機能３０２Ａおよび画像補正機能３０３Ａの機能が、第１実施形態に係る補正値算出機能３０２および画像補正機能３０３と異なる。
【０１８７】
以下、図２および図２３等を適宜参照しつつ、第１実施形態に係る投影システム１と異なる部分のみについて説明し、その他の部分については、第１実施形態に係る投影システム１と同様となるため、同様な符号を付して説明を省略する。
【０１８８】
ここでは、上述した図１０に示した場合と同様に、投影レンズ１０Ｒおよび撮影レンズ２０Ｒの光軸がほぼＹ軸と一致し、ホワイトボードＷの投影面は、ＸＺへ平面から角度θだけ傾いている場合を例に挙げて説明する。
【０１８９】
パソコン３０において、投影プログラムが実行されているときに、ユーザーがパソコン３０のキーボード等を種々操作すると、撮影画像の歪み補正動作が開始して、カメラ部２０がホワイトボードＷを撮影し、取得された撮影画像の歪みを補正したりする。
【０１９０】
撮影画像の歪み補正動作が開始すると、ＰＣ制御部３００からカメラ制御部２００Ａに撮影画像の歪み補正用の撮影を実施すべく撮影信号が伝送される。そして、カメラ制御部２００Ａはプロジェクター部１０にコマンドを送信し、垂直同期信号に同期して次のフィールドから、パソコン３０からのアナログＲＧＢ信号の入力に拘わらず、フレームメモリ１１２に記憶されるデジタル画像データが、ＲＯＭ１０ｂに格納される歪みを補正するためのパターン（方眼パターン）となるように設定する。したがって、次の垂直同期信号に同期して、プロジェクター部１０が、ホワイトボードＷに、方眼パターンを投影する。
【０１９１】
図２４は、プロジェクター部１０からホワイトボードＷに投影される方眼パターンＧＰＳを例示する模式図である。方眼パターンＧＰＳは、輪郭部分が白色の枠ＧＰＦと、縦横に伸びる白色の線分からなる方眼状のパターンＧＰとから構成されている。なお、方眼パターンＧＰＳでは、縦横に伸びる白色の線分が交差する点（例えば、点Ａ、点Ｂ、点Ｃ）が、方眼パターンＧＰＳ上の位置を特定するための特異点（他の点と識別可能な幾何学的特徴を有する点）となっている。即ち、プロジェクター部１０が、ホワイトボードＷに、複数の特異点を含む所定の方眼パターンを投影する手段として機能する。
【０１９２】
これに対して、カメラ制御部２００が、撮像素子駆動回路２１４に撮影のトリガー信号を出し、次の垂直同期信号に同期して、カメラ部２０が、ホワイトボードＷに投影された所定の方眼パターンＧＰＳを撮影することによって、画像を取得する。このとき、カメラ制御部２００の制御の下で、露光量が撮像素子２１３の許容量を超過して、撮像素子２１３の出力が飽和しないように、レンズ駆動回路２１２が撮影レンズ２０Ｒの絞り値を所定の絞り値に設定する。
【０１９３】
なお、ここでは、シャッタースピードが垂直同期信号によって決まり、プロジェクター部１０からは方眼パターンＧＰＳが投影され、発光量も決まるため、撮像素子２１３の出力が飽和しないように、所定の絞り値を容易に設定することができる。
【０１９４】
次に、プロジェクター部１０から方眼パターンＧＰＳをホワイトボードＷに投影して撮影を行った際に、Ａ／Ｄ変換器２１６から図２３に示すカメラ制御部２００の対応点抽出機能２０７に出力される。
【０１９５】
図２５は、ホワイトボードＷに投影された方眼パターンＧＰＳをカメラ部２０で撮影して取得される画像Ｇ２を例示する模式図である。図２５では、枠ＧＰＦ’が、プロジェクター部１０から投影される方眼パターンＧＰＳの枠ＧＰＦに対応し、方眼状のパターンＧＰ’が方眼状のパターンＧＰに対応する。そして、ここでは、撮影レンズ２０Ｒの光軸とホワイトボードＷとは垂直の関係にないため、図２５に示すように、方眼状のパターンＧＰ’に台形歪みが発生している。
【０１９６】
対応点抽出機能２０７では、輝度の変化を検出することによって、図２５に示す画像Ｇ２から、枠ＧＰＦ’外の領域を輝度による判別で排除する。次に、残された方眼状のパターンＧＰ’の縦方向と横方向への輝度変化を見ることにより、方眼状のパターンＧＰ’の縦および横から何本目の線が何処にあるのかを検出することができる。よって、対応点抽出機能２０７は、図２５に示すＧ２における、方眼状のパターンＧＰ’の線分が交差する点の位置を特定することができる。その結果、例えば、図２４における点Ａ、点Ｂ、点Ｃに対応する点として、図２５に示す画像Ｇ２における点Ａ’、点Ｂ’、点Ｃ’をそれぞれ特定し、抽出することができる。即ち、対応点抽出機能２０７が、画像Ｇ２について、プロジェクター部１０から投影される方眼パターンＧＰＳに含まれる複数の特異点にそれぞれ対応する対応点を抽出する手段として機能する。
【０１９７】
次に、図２に示すＰＣ制御部３００Ａの補正値算出機能３０２Ａにおいて、図２４に示す方眼パターンＧＰＳにおける複数の特異点（例えば、点Ａ、点Ｂ、点Ｃ）と、図２５に示す画像Ｇ２における対応点（点Ａ’、点Ｂ’、点Ｃ’）との位置関係から撮影画像の歪みを補正するための補正値を算出する。即ち、補正値算出機能３０２Ａが、複数の特異点と、その対応点との位置関係から補正値を算出する手段として機能する。
【０１９８】
ここで、図２６および図２７を参照しつつ、補正値算出機能３０２Ａにおける補正値を算出するための原理を簡単に説明する。
【０１９９】
図２６および図２７は、補正値を算出するための原理を説明するための図である。図２６では、プロジェクターの光軸をｚ軸とし、光源を原点としてｘｙｚ直交座標系を設定した場合に、ライトバルブ（例えば、表示素子１２３）における画素と、投影面（ホワイトボードＷ）上に投影される画像との位置関係を模式的に示す。また、図２７では、カメラの光軸をｚ’軸としたｘ’ｙ’ｚ’直交座標系を設定した場合において、投影面（ホワイトボードＷ）上の位置と、撮像素子２１３における画素との位置関係を模式的に示している。
【０２００】
図２６に示すように、プロジェクター部１０から発せられるある光線（第１の光線）が、下式（２）の関係を有する場合、ライトバルブである第１の投影面Ｐ１をｚ＝１で表される平面とすると、第１の光線と第１の投影面Ｐ１との交点の座標は点Ｍ（ｐ，ｑ，１）となる。
【０２０１】
ｘ／ｐ＝ｙ／ｑ＝ｚ・・・（２）
【０２０２】
次に、この状態で、画像として第２の投影面Ｐ２（例えば、ホワイトボードＷ）に投影されることを前提に発せられる光線群と、第２の投影面Ｐ２との交点Ｍ’を求める。ここで、第２の投影面Ｐ２を下式（３）で表すと、第１の光線と第２の投影面Ｐ２との交点Ｍ’の座標は次式（４）〜（６）で表される。
【０２０３】
ｚ＝ａｘ＋ｂｙ＋ｃ・・・（３）
ｘ＝ｃｐ／（１−ａｐ−ｂｑ）・・・（４）
ｙ＝ｃｑ／（１−ａｐ−ｂｑ）・・・（５）
ｚ＝ｃ／（１−ａｐ−ｂｑ）・・・（６）
【０２０４】
つまり、ここでは、第１の投影面Ｐ１の画像を第２の投影面Ｐ２に投影することで、３次元空間において、画像の一部を示す点Ｍが点Ｍ’に変換されたことになる。
【０２０５】
そして、図２７に示すように、カメラの撮像素子に照射されるある光線（第２の光線）が、下式（７）の関係を有する場合、撮像素子の撮像面となる第３の投影面Ｐ３をｚ’＝１で表される平面とすると、第２の光線と第３の投影面Ｐ３との交点の座標は点Ｎ（ｓ，ｔ，１）となる。
【０２０６】
ｘ’／ｓ＝ｙ’／ｔ＝ｚ’・・・（７）
【０２０７】
この状態で、第３の投影面Ｐ３に画像を結像するために照射される光線群と、第３の投影面Ｐ３とは異なる第４の投影面Ｐ４（例えば、ホワイトボードＷ）との交点Ｎ’を求める。ここで、第４の投影面Ｐ４を下式（８）で表すと、第２の光線と第４の投影面Ｐ２との交点Ｎ’の座標は次式（９）〜（１１）で表される。
【０２０８】
ｚ’＝ｄｘ’＋ｅｙ’＋ｆ・・・（８）
ｘ’＝ｆｓ／（１−ｄｓ−ｅｔ）・・・（９）
ｙ’＝ｆｔ／（１−ｄｓ−ｅｔ）・・・（１０）
ｚ’＝ｆ／（１−ｄｓ−ｅｔ）・・・（１１）
【０２０９】
つまり、ここでは、第４の投影面Ｐ４の画像を第３の投影面Ｐ３に投影することで、３次元空間において、画像の一部を示す点Ｎ’が点Ｎに変換されたことになる。
【０２１０】
そして、プロジェクターによる投影画像を、プロジェクターとは異なる位置と姿勢を有するカメラで撮影した場合には、３次元空間において、投影画像を構成するある点の座標は、図２６に示す投影過程における点Ｍから点Ｍ’への座標変換と、プロジェクターについてのｘｙｚ直交座標系からカメラについてのｘ’ｙ’ｚ’直交座標系への座標変換と、図２７に示す撮影過程における点Ｎ’から点Ｎへの座標変換を受けることとなる。また、ここで、ｘｙｚ直交座標系からｘ’ｙ’ｚ’直交座標系への座標変換を示す係数（座標変換係数）をｒ_１１，ｒ_１２，ｒ_１３，ｒ_２１，ｒ_２２，ｒ_２３，ｒ_３１，ｒ_３２，ｒ_３３として、下式（１２）〜（１４）の関係が成立するものとして説明を続ける。
【０２１１】
ｘ’＝ｒ_１１ｘ＋ｒ_１２ｙ＋ｒ_１３ｚ・・・（１２）
ｙ’＝ｒ_２１ｘ＋ｒ_２２ｙ＋ｒ_２３ｚ・・・（１３）
ｚ’＝ｒ_３１ｘ＋ｒ_３２ｙ＋ｒ_３３ｚ・・・（１４）
【０２１２】
ここで、式（１２）〜（１４）に、式（４）〜（６）および式（９）〜（１１）を代入すると、下式（１５）〜（１７）の関係式が求まる。
【０２１３】
【数２】

【０２１４】
式（１５）〜（１７）の３つの式には、ａ〜ｆ、ｒ_１１〜ｒ_３３の１５個の未知である係数が含まれているが、以下の手順で整理すると未知数を９個まで減らすことができる。
【０２１５】
例えば、第２の投影面と第４の投影面は同一であるため、一方の投影面の式が決まれば、他方の投影面の式は、ｘｙｚ直交座標系からｘ’ｙ’ｚ’直交座標系への座標変換係数を用いて表すことができる。つまり、未知数ａ，ｂ，ｃまたは未知数ｄ，ｅ，ｆのうち一方を無くすことができ、未知数が３個減る。また、２つの座標系（ｘｙｚ直交座標系とｘ’ｙ’ｚ’直交座標系）の間で、方位関係が傾いているのみで、拡大縮小を伴わないことを条件とすると、下式（１８）〜（２０）の関係を用いて、未知数である９個の座標変換係数ｒ_１１〜ｒ_３３を６個の未知数に減らすことができる。
【０２１６】
ｒ_１１ ^２＋ｒ_１２ ^２＋ｒ_１３ ^２＝１・・・（１８）
ｒ_２１ ^２＋ｒ_２２ ^２＋ｒ_２３ ^２＝１・・・（１９）
ｒ_３１ ^２＋ｒ_３２ ^２＋ｒ_３３ ^２＝１・・・（２０）
【０２１７】
このように、式（１５）〜（１７）の３つの式における未知数は９個とすることができるため、第１の投影面Ｐ１上での点Ｍ（ｐ，ｑ，１）と第３の投影面Ｐ３上での点Ｎ（ｓ，ｔ，１）とを対応づける組合せが３組あれば、その３組の座標を式（１５）〜（１７）に代入すると、９つの式が得られ、残りの９個の未知数を求めることができる。よって、第１の投影面Ｐ１上の任意の点と、その点に対応する第３の投影面Ｐ３上の点とを相互に変換することができる式（変換式）を導出することができる。
【０２１８】
したがって、例えば、補正値算出機能３０２Ａにおいて、図２４に示す方眼パターンＧＰＳにおける３点の特異点（例えば、点Ａ、点Ｂ、点Ｃ）と、図２５に示す画像Ｇ２における対応点（点Ａ’、点Ｂ’、点Ｃ’）との位置関係から、変換式を求めることができる。なお、ここでは、この変換式が撮影画像の歪みを補正するための補正値にあたる。
【０２１９】
そして、画像補正機能３０３Ａが、補正値算出機能３０２Ａで算出された補正値に基づいて、カメラ部２０によってホワイトボードＷを撮影することにより得られる撮影画像の歪みを補正する。この結果、ホワイトボードＷに投影される画像に合わせて書き込まれる文字や資料等をカメラ部２０で撮影して画像を取得し、その画像を、投影画像と整合するように座標変換して画像を生成することができる。
【０２２０】
ところで、プロジェクター部１０において、投影画像の歪み補正をしている場合には、第１の投影面Ｐ１における画像は、投影画像の歪み補正値に基づいて、投影原画像を変形させたものとなっている。しかし、第１の投影面Ｐ１における画像と、投影原画像との位置関係は、投影画像の歪み補正値によって規定されるため、カメラ部２０で撮影した画像を、投影原画像と整合するように座標変換をすることは容易に可能である。なお、このような場合は、ＰＣ制御部３００がプロジェクター制御部１００から投影画像に対する歪み補正値を取得することによって、変換式を、投影画像に対する歪み補正値を加味した式とし、座標変換を実現する。
【０２２１】
したがって、プロジェクター部１０において、投影画像の歪み補正をしている場合には、ホワイトボードＷに投影される画像は、ホワイトボードＷの正面から見ると、パソコン３０の記憶部３２に記憶される投影原画像をそのまま拡大したものとなっている。よって、カメラ部２０で撮影した画像を、投影原画像と整合するように座標変換をすると、ホワイトボードＷ上の書き込み等をホワイトボードＷの正面から撮影して取得された画像のように補正することができる。
【０２２２】
なお、上述したように、ここでは、投影するための画像における３点の特異点と、３点の特異点に対応する撮影画像における対応点との位置関係から、撮影画像の歪みを補正するための補正値を求めることができる。そして、この３点の特異点は直線状に並ばないことが必要である。しかし、投影レンズ１０Ｒや撮影レンズ２０Ｒの各種収差等の誤差要因を勘案すると、すべての画像の中心と４隅の計５箇所における特異点と、その対応点との位置関係から、撮影画像の歪み補正をするための補正値を求める方が好ましい。
【０２２３】
また、上述したように、撮影画像に対する歪み補正値は、撮影が行われる毎に行われるため、途中でホワイトボードＷが動いた場合にも対応することができる。なお、ここでは、方眼パターンＧＰＳを投影するタイミングと、撮影するタイミングとが同期しており、さらに、方眼パターンＧＰＳを投影する時間はごく短時間であるため、投影画像を見ている人に気づかれず、投影画像が見づらくなることがない。
【０２２４】
以上のように、第２実施形態に係る投影システム１Ａでは、複数の特異点（例えば、点Ａ，Ｂ，Ｃ）を含む所定の方眼パターンＧＰＳをホワイトボードＷに投影する。そして、その投影された方眼パターンＧＰＳを撮影して得られた画像Ｇ２から、複数の特異点にそれぞれ対応する対応点を抽出する。さらに、方眼パターンＧＰＳにおける複数の特異点と、対応点との位置関係とから補正値を算出する。その補正値に基づいて、ホワイトボードＷを撮影することにより得られる撮影画像の歪みを補正する。その結果、ホワイトボードＷ（投影面）とカメラ部２０との相対的な位置関係等から生じる撮影画像の歪みを補正することができる。
【０２２５】
また、ここでは、ホワイトボードＷを撮影することによって得られる撮影画像を、投影するための投影原画像と整合するように補正することができる。よって、上述したように、ホワイトボードＷ上に直接書き込まれたり、添付される等して直接に付された視覚要素に対応する領域を撮影画像から抽出して、ホワイトボードＷに投影するための投影原画像とデータ合成する際に、ずれのない合成画像を生成することができる。
【０２２６】
＜（３）　第３実施形態＞
第３実施形態では、複数の会議室のパソコンをネットワークを介して接続し、同じ画像を各会議室において拡大投影しながら、遠隔地同士で会議を協同して進行させる電子会議システムについて示している。
【０２２７】
＜（３−１）　電子会議システムの内部構成＞
図２８は、電子会議システムの内部構成を示すブロック図である。図２８に示すように、電子会議システムでは、２つの会議室ＣＲ１，ＣＲ２にそれぞれ、ホワイトボードＷ１，Ｗ２に画像を投影するように、第１実施形態に係る投影システム１とほぼ同様な投影システム１Ｂが設置されている。両会議室ＣＲ１，ＣＲ２に設置される投影システム１Ｂの内部構成は、全く同様であるため、同様な部分については同様な符号を付しており、また、投影システム１Ｂは投影システム１に異なる機能が若干付加されたものであり、以下、主に、投影システム１と異なる点について説明する。なお、投影システム１Ｂの動作の制御は、パソコン３０内に記憶されるプログラムがＰＣ制御部３００において実行されることによって実現されているものとする。
【０２２８】
投影システム１Ｂのパソコン３０にはスピーカＳＰが設けられるとともに、マイクＭＣが接続され、両会議室ＣＲ１，ＣＲ２内の音声を入力して音声データを取得することができる。また、投影システム１Ｂは、それぞれパソコン３０がネットワーク４００に接続され、両会議室ＣＲ１，ＣＲ２に設置される投影システム１Ｂは相互にデータを送受信することができる。
【０２２９】
そして、パソコン３０の記憶部３２に記憶される画像データ（投影原画像）を送受信することによって、両会議室ＣＲ１，ＣＲ２において同様な投影画像をホワイトボードＷに投影することができる。例えば、会議室ＣＲ１のパソコン３０に記憶される画像データ（投影原画像）を会議室ＣＲ２のパソコン３０に送信することによって、会議室ＣＲ１のパソコンに記憶される画像データに基づいて、両会議室ＣＲ１，ＣＲ２において同様な投影画像を投影することができる。
【０２３０】
また、両会議室ＣＲ１，ＣＲ２に設置される投影システム１Ｂは、パソコン３０がネットワーク４００を介して相互にデータを送受信することができるため、マイクＭＣを介して取得される音声データを相互に送受信して、スピーカＳＰから音声を出力することができる。
【０２３１】
ところで、上述した投影システム１では、会議内容の記録動作において、画像合成機能３０５が、領域抽出機能３０４によって抽出された領域と、プロジェクター部１０によって投影するための投影原画像とをデータ合成して合成画像データを生成し、記憶部３２に記憶した。これに対して、投影システム１Ｂでは、投影システム１における会議内容の記録動作と同様に、カメラ部２０によってホワイトボードＷ１，Ｗ２の撮影を行い、その画像に基づいて、領域抽出機能３０４によって抽出した領域についての画像データをネットワーク４００を介して他方の投影システム１Ｂに送信する。そして、他方の投影システム１Ｂにおいて、投影原画像とデータ合成して合成画像データを生成し、ホワイトボードＷに投影する。なお、投影システム１Ｂにおける投影画像と撮影画像との合成処理のその他の動作については、投影システム１と同様となる。
【０２３２】
ここで、「ネットワーク」とは、データ通信を行う通信回線網であり、具体的には、インターネット、ＬＡＮ、ＷＡＮ、ＣＡＴＶなどの、電気通信回線（光通信回線を含む）により構成される各種の通信回線網である。ネットワークに対する接続形態は、専用回線などを利用した常時接続であってもよいし、アナログ回線あるいはデジタル回線（ＩＳＤＮ）などの電話回線を利用したダイアルアップ接続などの一時的な接続のいずれであってもよい。また、その伝送方式は、無線方式および有線方式のいずれであってもよい。
【０２３３】
＜（３−２）　電子会議システムの使用例＞
図２９は、電子会議システムの使用例を示す模式図である。図２９に示すように、複数の会議室ＣＲ１，ＣＲ２には、それぞれ、投影システム１Ｂが設置され、投影システム１に備えられたパソコン３０が、ネットワーク等を介して、データを送受信し合うことによって、ホワイトボードＷ１，Ｗ２上に直接書き込まれた内容等も反映させながら、パソコン３０に表示される同一の画像を、ホワイトボードＷ１，Ｗ２に投影している。また、パソコン３０が図示を省略するマイクＭＣによって音声データを取得し、ネットワーク４００を介して音声データを送信受信することによって、一方の会議室における音声を他方の会議室において、パソコン３０に設けられたスピーカＳＰから出力することができる。
【０２３４】
そして、例えば、図２９に示すように、会議室ＣＲ１の説明者ＰＭ１は、ホワイトボードＷ上に投影された画像中の重要な点に赤のマーカーでアンダーラインＭＲ１を付し、会議の参加者に説明している。一方、会議室ＣＲ２の説明者ＰＭ２は、ホワイトボードＷ上に投影された画像中の重要な点に丸印ＭＲ２を付すとともに、「４０万画素超」という文字ＭＲ３を書き込んで、会議の参加者に説明している。
【０２３５】
このとき、両会議室ＣＲ１，ＣＲ２においては、もう一方の会議室で書き込まれた内容等を反映させるため、一定間隔毎に、カメラ部２０がホワイトボードＷ上の書き込み等を撮影し、その画像に基づいて、領域抽出機能３０４によって抽出した領域についての画像データをネットワーク４００を介して他方の投影システム１Ｂに送信する。そして、他方の投影システム１Ｂにおいて、投影原画像とデータ合成して合成画像データを生成し、ホワイトボードＷに投影する。
【０２３６】
その結果、図２９に示すように、一方の会議室においてホワイトボードＷに書き込まれたり、貼付された資料等の内容を他方の会議室のホワイトボードＷに反映させることができる。
【０２３７】
なお、一方の会議室において、ホワイトボードに投影した投影画像に合わせてマーカー等で書き込みを行った場合に、そのままカメラ部２０で撮影してその画像を他方の会議室にネットワーク４００を介して伝達し、その画像に基づいて、他方の会議室において、ホワイトボードＷに画像を投影しても、一方の会議室においてホワイトボードＷに書き込まれたり、貼付された資料等の内容を他方の会議室のホワイトボードＷに反映させることができる。しかし、そのままカメラ部２０で撮影すると、投影した画像とマーカーの書き込みとが重なって撮影されるため、読みにくい画像になったり、ホワイトボードＷに投影した画像をカメラで撮影した場合は、投影画像に対応する部分の画質が劣化してしまい、他方の会議室では、画像の作成者が意図した内容を会議に参加した人々に対して正確に伝達することができない。
【０２３８】
このような問題に対して、本実施形態に係る投影システム１Ｂでは、単一色（全面白色）の光をホワイトボードＷに照射して、ホワイトボードＷを撮影することによって画像を取得する。そして、取得された画像から、ホワイトボードＷ上に直接書き込まれたり、添付される等して直接に付された視覚要素に対応する領域を抽出して、ホワイトボードＷに投影するための投影原画像とデータ合成する。その結果、ホワイトボードＷに投影するための投影画像と、ホワイトボードＷ上に直接書き込まれた文字や線や添付された資料等の視覚要素とをデータ合成した高品質な画像を作成することができる。
【０２３９】
なお、上述したように、カメラ部２０による撮影時に、全面白色等の単一色の光をプロジェクター部１０から照射する。よって、全面白色等の単一色の光をホワイトボードＷに投影する時間が長いと、会議の参加者に違和感を与えたり、画像が見づらくなったりする。しかし、本実施形態に係る投影システム１Ｂにおいては、単一色の光を照射するタイミングと、撮影するタイミングとが同期しており、さらに、単一色の光を投影する時間は、１フレーム以内というごく短時間であるため、会議の参加者に違和感を与えたり、画像が見づらくなったりすることがない。
【０２４０】
また、ホワイトボードＷへの書き込み等の撮影は、ユーザーがパソコン３０のキーボード等を種々操作することによって行うことができるが、電子会議では、時間的なずれの低減が要求されるため、ホワイトボードＷ上の書き込み等を次々と撮影し、画像データをネットワーク４００を介して伝送する場合もある。このような場合は、ユーザーがパソコン３０において撮影間隔を短く設定し、撮影動作を繰り返し行って、書き込み等に係る撮影画像と投影原画像のデータ合成を短い周期で行うことが好ましい。
【０２４１】
ところで、ここでは、ホワイトボードＷ上の書き込みを撮影する際に、撮影画像の歪み補正をするか否かは、プロジェクター部１０から投影される投影画像の歪み補正を行っているか否かに従う。
【０２４２】
例えば、投影画像の歪み補正を行っていない場合は、会議の説明者は特に台形歪み等を問題とせず、投影画像に合わせて、ホワイトボードＷに書き込み等をしたものと考えられる。よって、この場合は、撮影画像に対する歪み補正は行わない。なお、投影画像の歪み補正を行っているか否かは、プロジェクター制御部１００とカメラ制御部２００間のデータの交信等によって情報を取得し、撮影画像に対する歪みを行うか否かについて、ＰＣ制御部３００等によって制御および管理することができる。一方、投影画像の歪み補正を行っている場合は、上述した撮影画像の歪み補正を必ず行い、他方の会議室の投影システムに画像データを伝送する。
【０２４３】
以上のように、第３実施形態に係る電子会議システムでは、一方の投影システム１Ｂにおいて、単一色（全面白色）の光をホワイトボードＷに照射して、ホワイトボードＷを撮影することによって画像を取得する。そして、取得された画像から、ホワイトボードＷ上に直接書き込まれたり、添付される等して直接に付された視覚要素に対応する領域を抽出して、他方の投影システムに伝送する。そして、他方の投影システムにおいて、抽出された領域と、ホワイトボードＷ上に投影するための投影原画像とをデータ合成して投影する。その結果、遠隔地等場所を違える会議室同士で、意志疎通を十分図ることができる。また、ホワイトボードＷに投影される情報と、ホワイトボードＷに直接付加される視覚要素の情報とを併せた高品質の画像を投影可能な投影システムを提供することができる。
【０２４４】
＜（４）　変形例＞
以上、この発明の実施形態について説明したが、この発明は上記説明した内容のものに限定されるものではない。
【０２４５】
◎例えば、上述した実施形態では、スケーラー１１３等は専用の電子回路となっているが、プロジェクター制御部１００内のＣＰＵにおいてプログラムを実行することにより、その機能を実現するようにしても良い。
【０２４６】
◎また、上述した実施形態では、プロジェクター制御部１００、カメラ制御部２００、およびＰＣ制御部３００を別々に設けているが、それらの機能のうちの一部、または全ての機能を、１箇所のＣＰＵにおいてプログラムを実行することによって実現するようにしても良い。
【０２４７】
◎また、投影プログラムが実行されているときに、ユーザーがパソコン３０のキーボード等を種々操作することによって、投影画像のＷＢ補正、撮影画像のＷＢ補正、投影画像の階調特性の補正、撮影画像の階調特性の補正、投影画像の歪み補正、および撮影画像の歪み補正のうちの全ての動作を実行しなかったり、いずれか１つの動作を実行したり、種々の組合せの動作を実行したり、全ての動作を実行するようにしても良い。
【０２４８】
◎また、上述した第２実施形態では、第２および第４等の投影面が平面の場合における撮影画像の歪み補正について例示したが、第２および第４の投影面が曲面である場合にも、曲面の式を規定することによって、第２実施形態と同様な手法によって、第１の投影面と第３の投影面との間の座標変換係数を求めることができる。したがって、プロジェクター部１０から投影画像が投影される投影面が曲面の場合にも撮影画像の歪み補正を実施可能である。
【０２４９】
◎また、上述した実施形態では、ＷＢ補正値を求めるために、投影領域に全面無彩色の光を投影したが、投影領域の一部に無彩色の光を投影して、その部分に係る画像データに基づいてＷＢ補正値を求めるようにしても良い。
【０２５０】
◎また、上述した実施形態では、プロジェクター制御部１００、カメラ制御部２００，２００Ａ、ＰＣ制御部３００，３００Ａの間におけるデータの送受信は、ＵＳＢＩ／Ｆ３４とＵＳＢＩ／Ｆ２３０の間、およびＵＳＢＩ／Ｆ２３０とＵＳＢＩ／Ｆ１３０の間をそれぞれケーブルによって接続することによって実現されていたが、接続の組合せや方式はこれに限られず、種々の組合せや方式で接続しても良い。
【０２５１】
◎また、上述した実施形態では、投影画像の歪み補正を自動で行う点を特徴とする動作以外の動作においては、投影画像の歪み補正の設定はユーザーが手動で行っても良い。
【０２５２】
◎また、上述した実施形態において例示した投影システム１，１Ａ，１Ｂは、上述したように撮影画像の歪み補正を行うため、ホワイトボードＷ等の被写体に係る撮影画像の歪み補正を行う歪み補正装置として見ることもできる。
【０２５３】
なお、上述した具体的実施形態には以下の構成を有する発明が含まれている。
【０２５４】
（１）被写体に複数の焦点検出用の部分を含むパターンを投影する投影手段と、前記投影面に投影された前記パターンを撮影することによって、複数の画素で構成される前記パターンに係る画像を入力する撮影手段と、前記撮影手段における撮影レンズの合焦制御を行う合焦制御手段と、前記画像における前記複数の焦点検出用の部分に対応する領域について、それぞれ前記撮影レンズの合焦状態に関する評価値を算出する評価値算出手段と、前記評価値に基づいて、前記撮影手段によって前記被写体を撮影することにより得られる撮影画像について、歪みを補正する歪み補正手段とを備えることを特徴とする歪み補正装置。
【０２５５】
（１）の発明によれば、焦点検出用の領域を複数含むパターンを投影し、そのパターンを撮影して入力された画像における、複数の焦点検出用の部分に対応する領域について、それぞれ撮影レンズの合焦状態に関する評価値を求め、その評価値に基づいて、被写体を撮影することにより得られる撮影画像の歪みを補正するような構成とすることによって、撮影手段と、被写体との相対的な位置関係等から生じる撮影画像の歪みを補正することができる。
【０２５６】
（２）被写体に複数の特異点を含む所定のパターンを投影する投影手段と、前記投影面に投影された前記所定のパターンを撮影することによって、画像を取得する撮影手段と、前記画像について、前記複数の特異点にそれぞれ対応する対応点を抽出する抽出手段と、前記複数の特異点と、前記対応点との位置関係から補正値を算出する算出手段と、前記補正値に基づいて、前記撮影手段によって前記被写体を撮影することにより得られる撮影画像の歪みを補正する歪み補正手段とを備えることを特徴とする歪み補正装置。
【０２５７】
（２）の発明によれば、複数の特異点を含む所定のパターンを被写体に投影し、その投影された所定のパターンを撮影して得られた画像から、複数の特異点にそれぞれ対応する対応点を抽出し、所定のパターンにおける複数の特異点と、対応点との位置関係から補正値を算出し、その補正値に基づいて、被写体を撮影することにより得られる撮影画像の歪みを補正するような構成とすることによって、被写体と撮影手段との相対的な位置関係等から生じる撮影画像の歪みを補正することができる。
【０２５８】
（３）投影システムに含まれるコンピュータによって実行されることにより、前記投影システムを、請求項１から請求項５、（１）および（２）のうちいずれかに記載の投影システムとして機能させるプログラム。
【０２５９】
（３）の発明によれば、請求項１から請求項５、（１）および（２）のいずれかに記載の発明と同様な効果を得ることができる。
【０２６０】
【発明の効果】
以上説明したように、請求項１から請求項５の発明によれば、投影面の状況に応じた適正な画像処理が可能な投影システムを提供することができる。
【０２６１】
また、請求項１から請求項３の発明によれば、投影面の状況に応じた適正な画像を投影可能な投影システムを提供することができる。
【０２６２】
特に、請求項１の発明によれば、輝度が段階的に変化するパターンを投影面に投影し、そのパターンを撮影して得られた画像に基づいて、投影面に投影される投影画像や、投影面等を撮影することによって得られる撮影画像の階調特性を補正するような構成とすることによって、外光の影響や投影面の反射率等により投影画像や撮影画像の階調特性が変化するのを補正することができる。
【０２６３】
また、請求項２の発明によれば、無彩色の光を投影した投影面を撮影して得られた画像に基づいて、投影面に投影される投影画像や、投影面等を撮影することによって得られる撮影画像のホワイトバランスを補正するような構成とすることによって、投影面の色や、外光の色や、投影手段の光源の色等の影響により、投影画像や撮影画像の色調が変化するのを補正することができる。
【０２６４】
また、請求項３の発明によれば、焦点検出用の部分を複数含むパターンを投影し、そのパターンを撮影して入力された画像における複数の焦点検出用の部分に対応する領域について、それぞれ撮影レンズの合焦状態に関する評価値を求め、その評価値に基づいて、投影面に投影される投影画像や、投影面を撮影することにより得られる撮影画像の歪みを補正するような構成とすることによって、投影手段や撮影手段と、投影面との相対的な位置関係等から生じる投影画像や撮影画像の歪みを補正することができる。
【０２６５】
また、請求項４の発明によれば、複数の特異点を含む所定のパターンを投影面に投影し、その投影された所定のパターンを撮影して得られた画像から、複数の特異点にそれぞれ対応する対応点を抽出し、所定のパターンにおける複数の特異点と、対応点との位置関係から補正値を算出し、その補正値に基づいて、投影面を撮影することにより得られる撮影画像の歪みを補正するような構成とすることによって、投影面と撮影手段との相対的な位置関係等から生じる撮影画像の歪みを補正することができる。
【０２６６】
また、請求項５の発明によれば、単一色の光を投影面に投影して投影面を撮影することによって取得される画像について、投影面に直接書き込まれたり、添付される等して直接に付された視覚要素に対応する領域を抽出して、投影面に投影するための画像とデータ合成するような構成とすることによって、投影面に投影される画像と、投影面に直接書き込まれた文字や線や添付された資料等の視覚要素とを合成した高品質な画像を作成することができる。即ち、投影面に投影される情報と、投影面に直接付加される情報とを併せた高品質の画像を取得可能な投影システムを提供することができる。
【図面の簡単な説明】
【図１】第１実施形態に係る投影システムの使用例を示す模式図である。
【図２】第１実施形態に係る投影システムの内部構成を示すブロック図である。
【図３】第１実施形態に係るカメラ制御部の機能を説明するためのブロック図である。
【図４】ホワイトバランス補正値の設定方法を説明するための図である。
【図５】階調特性補正用の撮影時に取得される画像を例示する模式図である。
【図６】グレースケールチャートに係る理想的な明るさの比を例示する模式図である。
【図７】外光の影響がある場合の明るさの比を例示する模式図である。
【図８】階調特性補正用の補正値の算出方法を説明するための図である。
【図９】外光の影響がある場合の階調変換特性を例示する図である。
【図１０】ホワイトボードに対して斜めの方向から投影および撮影する場合を例示する模式図である。
【図１１】ホワイトボードに対して斜め投影される画像を例示する模式図である。
【図１２】歪み補正後の投影画像を示す模式図である。
【図１３】斜め撮影時に取得される撮影画像を例示する模式図である。
【図１４】歪み補正後の撮影画像を示す模式図である。
【図１５】ホワイトボードに投影される焦点検出用パターンを例示する図である。
【図１６】ホワイトボードに投影された焦点検出用パターンを撮影することによって取得される撮影画像を例示する模式図である。
【図１７】評価値と撮影レンズの位置との関係を表す曲線を例示する模式図である。
【図１８】撮影画像に対する歪み補正用の補正値の算出方法を説明するための図である。
【図１９】ホワイトボード上の状態を例示する模式図である。
【図２０】ホワイトボードの書き込みを撮影した撮影画像を例示する模式図である。
【図２１】ホワイトボードに投影するための投影原画像を例示する模式図である。
【図２２】投影原画像と書き込みとをデータ合成した合成画像を例示する模式図である。
【図２３】第２実施形態に係るカメラ制御部の機能を説明するためのブロック図である。
【図２４】ホワイトボードに投影される方眼パターンを例示する模式図である。
【図２５】ホワイトボードに投影された方眼パターンを撮影して得られた画面を例示する模式図である。
【図２６】撮影画像の歪み補正用を算出するための原理を説明するための図である。
【図２７】撮影画像の歪み補正用を算出するための原理を説明するための図である。
【図２８】電子会議システムの内部構成を示すブロック図である。
【図２９】電子会議システムの使用例を示す模式図である。
【図３０】会議における従来の投影システムの使用例を示す模式図である。
【符号の説明】
１，１Ａ，１Ｂ　投影システム
２　カメラ付きプロジェクター
１０　プロジェクター部（投影手段）
２０　カメラ部（撮影手段）
３０　パソコン
１００　プロジェクター制御部
１０１　素子駆動制御機能（ホワイトバランス補正手段、階調補正手段）
１１３　スケーラー（歪み補正手段）
２００，２００Ａ　カメラ制御部
２０４　評価値演算機能（評価値算出手段）
２０５　合焦制御機能（合焦制御手段）
２０７　対応点抽出機能（抽出手段）
２１０　撮影機能部（撮影手段）
２１７　ＷＢ回路（ホワイトバランス補正手段）
２１８　γ補正回路（階調補正手段）
３００　ＰＣ制御部
３０２Ａ　補正値算出機能（算出手段）
３０３Ａ　画像補正機能（歪み補正手段）
３０４　領域抽出機能（領域抽出手段）
３０５　画像合成機能（合成手段）[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a technology of a projection system that projects an image on a projection surface.
[0002]
[Prior art]
An image displayed on a screen of a personal computer (hereinafter, abbreviated as “PC”) is projected by a projection device onto a projection surface such as a whiteboard to hold a meeting, for example, for use in meetings (for example, Non-Patent Documents). 1).
[0003]
FIG. 30 is a schematic diagram illustrating an example of use of the conventional projection system 500 in a conference. FIG. 30 is a top view of the conference room CR10 as viewed from above.
[0004]
As shown in FIG. 30, the conventional projection system 500 mainly includes a projection device 510 and a personal computer 520, and is installed on the conference desk CD10. Then, projection device 510 and personal computer 520 are connected by a cable, and projection device 510 enlarges and projects an image similar to the image displayed on the screen of personal computer 520 onto whiteboard W10.
[0005]
Here, since the presenter PM10 and the like can proceed with the conference while appropriately pointing to the image projected on the whiteboard W10, the contents of the conference and the like can be easily communicated to the people M11 to M15 who participated in the conference. Can be. Therefore, there is no need to distribute meeting materials, and the cost of creating materials can be significantly reduced.
[0006]
In such a meeting, it is common practice that the presenter PM10 or the like manually writes on the whiteboard W10 with the marker MK or the like in accordance with the image enlarged and projected on the whiteboard W10. Then, in order to record the content of the conference, the content written on the whiteboard W10 may be photographed by a camera (for example, see Non-Patent Document 2).
[0007]
Further, although not shown, the personal computers 520 in a plurality of conference rooms are connected via a network, and in a different conference room, a similar image is enlarged and projected onto the whiteboard W10 by the projection device 510, and a conference is held between remote locations. There is also a service of an electronic conferencing system that proceeds in cooperation.
[0008]
Prior art documents relating to such technology include the following.
[0009]
[Patent Document 1]
JP-A-9-289600
[Patent Document 2]
JP-A-9-289611
[Patent Document 3]
JP-A-2002-57879
[Patent Document 4]
Japanese Patent No. 2632520
[Patent Document 5]
Registered Utility Model No. 4-18308
[Non-patent document 1]
"DATA PROJECTOR (TLP-X20DJ / TLP-X21DJ) Catalog", Toshiba Digital Media Network Company, March 2002
[Non-patent document 2]
"New Release of White and White Sheet for Marker Pen Corresponding to Marker Pen A / B Sheet" [online], June 4, 1999, Fuji Photo Film Co., Ltd., [August 6, 2002] Day search], Internet <URL: http://www.fujifilm.co.jp/news_r/nrj458.html>.
[0010]
[Problems to be solved by the invention]
However, the color tone between the image displayed on the screen of the personal computer 520 and the image projected on the projection surface is affected by the color and reflectance of the projection surface, external light, and the deterioration of the light source of the projection device. And differences in tone characteristics and the like may occur. Further, when an image is projected by a projection device from a direction oblique to the projection surface, an image projected on the projection surface may be distorted due to a relative positional relationship between the projection surface and the projection device. In these cases, there arises a problem that the content intended by the creator of the image cannot be accurately transmitted to the people who participated in the conference.
[0011]
Furthermore, in the above-described electronic conferencing system in which a conference is cooperatively proceeded between remote locations, the contents written on the projection plane are not reflected on the projection planes of the other conference rooms. There is also a problem that it becomes difficult to understand the contents of the information.
[0012]
In addition, when the meeting materials are not distributed, the enlarged and projected images are combined with the contents written on the projection surface and the like to record as meeting contents. There is also a problem that it is difficult to take a clean image or the like.
[0013]
Further, when recording the contents written on the projection surface to record the contents of the meeting, the relative position between the projection surface and the camera may be required, such as when it is necessary to take an image from a direction oblique to the projection surface. Depending on the positional relationship, the captured image may be distorted. In order to address this problem, it is conceivable to apply the following conventional technology regarding cameras. For example, in the cameras disclosed in Patent Literatures 1 and 2, a user measures the angle between the longitudinal direction of the camera and the whiteboard with a measure, and inputs the angle to the camera to correct the trapezoidal distortion of the captured image. it can. However, this method has a problem that a complicated operation is required and the accuracy of correction is low. Further, the camera disclosed in Patent Document 3 can calculate a correction value for correcting distortion by extrapolating at least two pairs of opposed line segments from a character or figure written on a projection surface. However, this method is not effective unless writing or the like capable of extrapolating at least two sets of line segments, and the use case is limited.
[0014]
The present invention has been made in view of the above problems, and has as its basic object to provide a projection system capable of performing appropriate image processing according to the state of a projection surface.
[0015]
And there are the following specific purposes corresponding to this basic purpose.
[0016]
A first specific object is to provide a projection system that can “generate” an appropriate image according to the condition of the projection plane.
[0017]
A second specific object is to provide a projection system capable of “projecting” an appropriate image according to the condition of the projection plane.
[0018]
A third specific object is to provide a projection system capable of “taking” an appropriate image according to the state of the projection plane.
[0019]
Another object of the present invention is to provide a projection system capable of acquiring a high-quality image combining information projected on a projection plane and information directly added to the projection plane.
[0020]
[Means for Solving the Problems]
In order to solve the above-mentioned problem, an invention according to claim 1 is a projection system, comprising: a projection unit configured to project a pattern whose luminance changes stepwise on a projection surface; and the pattern projected onto the projection surface. A photographing unit for acquiring an image by photographing the image, a projection image projected on the projection surface by the projection unit based on the image acquired by the photographing unit, and the projection surface by the photographing unit. The image processing apparatus further includes a gradation correcting unit that corrects a gradation characteristic of at least one of the captured images obtained by capturing the image.
[0021]
The invention according to claim 2 is a projection system, wherein a projection unit that projects achromatic light on a projection surface, and an image is obtained by photographing the projection surface on which the achromatic light is projected. A projection image projected on the projection surface by the projection unit based on the image acquired by the photography unit, and a captured image obtained by photographing the projection surface by the photography unit And white balance correction means for correcting the white balance of at least one of the images.
[0022]
The invention according to claim 3 is a projection system, comprising: a projection unit configured to project a pattern including a plurality of focus detection portions on a projection plane; and photographing the pattern projected on the projection plane. A photographing unit that inputs an image of the pattern composed of a plurality of pixels; a focus control unit that performs focusing control of a photographing lens in the photographing unit; and a plurality of focus detection portions in the image. Evaluation value calculation means for calculating an evaluation value for the in-focus state of the photographing lens for each corresponding area; a projection image projected on the projection surface by the projection means based on the evaluation value; Means for correcting distortion for at least one of the captured images obtained by capturing the projection plane by means. And butterflies.
[0023]
Further, the invention according to claim 4 is a projection system, wherein a projection means for projecting a predetermined pattern including a plurality of singular points on a projection surface, and photographing the predetermined pattern projected on the projection surface. By means of, a photographing unit for acquiring an image, an extracting unit for extracting corresponding points respectively corresponding to the plurality of singularities, and a correction value based on a positional relationship between the plurality of singularities and the corresponding points. It is characterized by comprising a calculating means for calculating, and a distortion correcting means for correcting distortion of a photographed image obtained by photographing the projection plane by the photographing means based on the correction value.
[0024]
Further, the invention according to claim 5 is a projection system, wherein a projection means for projecting a single color light on a projection surface, and an image is obtained by photographing the projection surface on which the single color light is projected. Imaging means to perform, for the image obtained by the imaging means, an area extraction means for extracting an area corresponding to a visual element directly attached to the projection plane, and the area extracted by the area extraction means, A synthesizing unit for synthesizing data with an image to be projected onto the projection plane by the projection unit.
[0025]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0026]
<(1) First embodiment>
<(1-1) Overview of projection system>
FIG. 1 is a schematic diagram showing an example of use of a projection system 1 according to the first embodiment of the present invention. FIG. 1 is a top view of the conference room CR1 as viewed from above. In FIG. 1 and subsequent drawings, an XYZ orthogonal coordinate system is added as necessary to clarify the azimuth relationship.
[0027]
As shown in FIG. 1, a projection system 1 according to a first embodiment of the present invention mainly includes a projector 2 with a camera (digital camera) and a personal computer (hereinafter abbreviated as “PC”) 30. Is installed on the conference desk CD. Then, the camera-equipped projector 2 and the personal computer 30 are connected by a cable CB, and the camera-equipped projector 2 enlarges and projects an image similar to the image displayed on the screen of the personal computer 30 onto a whiteboard W as a projection surface. .
[0028]
Here, the presenter PM or the like appropriately proceeds with the meeting while pointing to an image (projected image) enlarged and projected on the whiteboard W or writing the whiteboard W with the marker MK or the like. Therefore, since the contents of the conference and the like can be easily communicated to the people M1 to M5 who have participated in the conference, there is no need to distribute the materials for the conference in particular, and the cost for preparing the materials can be greatly reduced. .
[0029]
The camera-equipped projector 2 mainly includes a projector unit 10 and a camera unit 20, and the camera unit 20 is fixed to a housing of the projector unit 10. For example, the projection lens 10R of the projector unit 10 and the photographing lens 20R of the camera unit 20 are arranged as close as possible, and the projection lens 10R and the photographing lens 20R are arranged so that the optical axis and the angle of view are close to each other. Is set.
[0030]
In FIG. 1, a solid line 10L schematically showing the angle of view of the projection lens 10R and a dotted line 20L schematically showing the angle of view of the photographing lens 20R are shown. As shown in FIG. 1, here, the entire area of the projection image projected on the whiteboard W (hereinafter, referred to as “projection area”) is set so as to fall within a range that can be photographed by the camera unit 20.
[0031]
<(1-2) Internal configuration of projection system>
FIG. 2 is a block diagram illustrating an internal configuration of the projection system 1. As described above, the projection system 1 mainly includes the projector unit 10, the camera unit 20, and the personal computer 30. Hereinafter, the three parts will be described in order.
[0032]
<(1-2-1) Internal configuration of projector unit>
In the present embodiment, a one-chip system using one general DMD (Digital Micromirror Device) (registered trademark) as a display element 123 described later will be described as an example.
[0033]
As shown in FIG. 2, the projector unit 10 mainly includes a projector control unit 100, an image generation function unit 110, a projection function unit 120, and a USB I / F 34, like a general projector.
[0034]
The projection function unit 120 includes a light source 121, a color wheel 122 for color separation, a display element 123, and a projection lens 10R.
[0035]
The light source 121 is a light source that generates white light, and the projection lens 10R is a general projection lens.
[0036]
The color wheel for color separation 122 has three primary color filters of RGB and rotates in synchronization with the driving of the display element 123. The white light emitted from the light source 121 is sequentially separated into R (red) light, G (green) light, and B (blue) light in a time-division manner, and the color-separated light is displayed. The light is emitted to the display element surface of the element 123.
[0037]
The display element 123 is configured by one DMD as described above. The DMD is configured by a minute mirror for each pixel, and the angle of the minute mirror is changed based on control by an electric signal emitted from the projector control unit 100. When the electric signal is emitted from the projector control unit 100, the angle of the minute mirror is set such that the light beam emitted from the color separation color wheel 122 is reflected by the minute mirror and passes through the projection lens 10R. Is configured. On the other hand, in a state where the electric signal is not emitted from the projector control unit 100, the angle of the minute mirror is set so that the light beam emitted from the color separation color wheel 122 is reflected by the minute mirror and does not pass through the projection lens 10R. Is configured.
[0038]
The image generation function unit 110 generates image data for projection based on the analog RGB signals input from the personal computer 30. Here, an example will be described in which the analog RGB signals input from the personal computer 30 are all in a moving image format so as to be compatible with moving images.
[0039]
The image generation function unit 110 includes an analog RGB decoder 111, a frame memory 112, a scaler 113, and an image memory 114.
[0040]
The analog RGB decoder 111 generates digital image data suitable for driving the projection function unit 120 from the analog RGB signals input from the personal computer 30. In other words, digital image data suitable for driving the color separation color wheel 122 and the display element 123 is generated. Although not shown, the analog RGB decoder 111 converts the analog image signal into a digital image signal (for example, an 8-bit digital image signal) and a PLL circuit that generates a sampling pulse for sampling the analog signal. And an A / D converter.
[0041]
The analog RGB decoder 111 separates a horizontal synchronizing signal and a vertical synchronizing signal from an analog RGB signal input from the personal computer 30. Then, the PLL circuit generates sampling pulses such as the horizontal synchronization signal and the vertical synchronization signal at three times the speed of the separated horizontal synchronization signal and the vertical synchronization signal indicating the start of one frame.
[0042]
For example, assuming that the number of pixels of a display element 123 described later corresponds to the S-VGA standard (800 × 600 pixels), the PLL circuit generates a horizontal synchronization signal and then generates the next horizontal synchronization signal. By then, a pulse for sampling data for 800 pixels is generated. In addition, sampling is performed so as to synchronize the vertical synchronization signal generated from the PLL circuit, the RGB switching timing of the color separation color wheel 122, and the vertical scanning of the display element 123, and digital image data is generated.
[0043]
More specifically, when the timing to emit R light from the color separation color wheel 122 to the display element 123 in synchronization with the vertical synchronization signal generated from the PLL circuit, the input of the A / D converter becomes analog. The signal is switched to the R signal, and sampling is started. When the sampling of 600 vertical lines is completed, the timing for irradiating the display element 123 with the G light from the color separation color wheel 122 is synchronized with the next vertical synchronization signal, and the input of the A / D converter is performed. Is switched to the G signal of the analog signals, and sampling is started. Further, when the sampling of 600 lines in the vertical direction is completed, the timing for irradiating the B light from the color separation color wheel 122 to the display element 123 is synchronized with the next vertical synchronization signal, and the input of the A / D converter is performed. Is switched to the G signal of the analog signals, and sampling is started.
[0044]
Thereafter, when the sampling of the vertical 600 lines is completed, the same sampling operation is further repeated, and digital image data suitable for driving the projection function unit 120 is generated from the analog RGB signals input from the personal computer 30. The digital image data generated as described above is stored in the frame memory 112 as needed.
[0045]
The scaler 113 corrects trapezoidal distortion or the like that occurs when the optical axis of the projection lens 10R is not perpendicular to the whiteboard W that is the projection plane. In a conventional projector, the scaler scans the pixel data stored in the frame memory 112 by, for example, measuring and inputting the relative angle between the lens surface of the projection lens and the projection surface with the amount of eye. Enlargement and reduction calculations were performed for each line. On the other hand, the scaler 113 according to the first embodiment of the present invention converts digital image data stored in the frame memory 112 based on a correction value for distortion correction calculated by the correction value calculation function 102 described later. The enlargement / reduction operation is performed for each scanning line or each pixel. Note that the number of pixels of the image data output from the scaler 113 corresponds to the number of pixels of the display element 123. Calculation of a correction value for distortion correction and distortion correction will be described later in detail.
[0046]
The image memory 114 is configured by a volatile or non-volatile storage medium, and stores digital image data on which scaling operation has been performed by the scaler 113. Then, the projector control unit 100 controls the driving of the projection function unit 120 based on the digital image data stored in the image memory 114, thereby synchronizing with the vertical synchronization signal, the R field, the G field, The field B is repeatedly optically projected. Therefore, the projector unit 10 functions as a unit that projects an image on the whiteboard W (projection plane).
[0047]
Here, an electric signal from the projector control unit 100 to the display element 123 is generated for a period corresponding to the digital value stored in the image memory 114 for each pixel. That is, the reflection angle of the minute mirror of the display element 123 is changed for each pixel during a period corresponding to the digital value stored in the image memory 114. Note that the relationship between the digital value for each pixel and the generation time of the electric signal from the projector control unit 100 to the display element 123 is determined by using an LUT (conversion table) stored in the projector control unit 100 for each of R, G, and B. ). The LUT can be changed in the projector control unit 100, and the white balance (hereinafter abbreviated as “WB”) and the gradation of the projected image are adjusted according to correction values for gradation characteristic correction and white balance correction, which will be described later. The characteristics can be set arbitrarily.
[0048]
The projector control unit 100 has a microcomputer (CPU), a RAM 10a, and a ROM 10b, and controls the operation of each of the above-described members of the projector unit 10 organically to control the operation of the projector unit 10 overall. Note that each function of the projector control unit 100 is realized by a program stored in the ROM 10b in the CPU.
[0049]
Further, the projector control unit 100 has an element drive control function 101 for generating an electric signal in order to change a reflection angle of a minute mirror of the display element 123, and further includes a correction value for correcting distortion of a projected image. Is provided. The correction value calculation function 102 calculates a correction value for correcting distortion of a projected image based on a photographing magnification ratio calculated by a camera control unit 200 described later.
[0050]
The USB I / F 130 is a communication interface conforming to the USB standard for enabling communication between the projector control unit 100 and the camera unit 20.
[0051]
<(1-2-2) Internal Configuration of Camera Unit>
As shown in FIG. 2, the camera unit 20 mainly includes a camera control unit 200, a photographing function unit 210, an image memory 220, and a USB I / F 230.
[0052]
The photographing function unit 210 performs image signal processing in order to acquire an image, and includes a photographing lens 20R, a lens driving circuit 212, an image sensor 213, an image sensor driving circuit 214, a signal processing circuit 215, an A / D converter. 216, a WB circuit 217, a gamma correction circuit 218, and a color conversion / color correction circuit 219.
[0053]
The image sensor 213 is a general CCD, and converts an optical image of a subject formed by the photographing lens 20R into image signals of R, G, and B color components (from a signal train of pixel signals received by each pixel). ) And output. The image sensor driving circuit 214 generates various timing pulses for controlling driving of the image sensor 213.
[0054]
The exposure control in the photographing function unit 210 is performed by adjusting the aperture of the photographing lens 20R by the lens driving circuit 212 and the exposure amount of the image sensor 213, that is, the charge accumulation time of the image sensor 213 corresponding to the shutter speed. Here, when an appropriate shutter speed cannot be set when the subject luminance is low, the improper exposure due to insufficient exposure is corrected by adjusting the level of the image signal output from the image sensor 213. That is, when the luminance is low, exposure control is performed by combining the aperture, the shutter speed, and the gain adjustment of the signal processing circuit 215.
[0055]
The image sensor drive circuit 214 generates a drive control signal for the image sensor 213 based on a reference clock transmitted from the camera control unit 200. The image sensor drive circuit 214 outputs clock signals such as integration start / end (exposure start / end) timing signals and readout control signals (horizontal synchronization signal, vertical synchronization signal, transfer signal, etc.) of the light receiving signal of each pixel. Generate and output to the image sensor 213.
[0056]
The signal processing circuit 215 performs predetermined analog signal processing on an image signal (analog signal) output from the image sensor 213. The signal processing circuit 215 has a CDS (correlated double sampling) circuit and an AGC (auto gain control) circuit, performs noise reduction processing of an image signal by the CDS circuit, and adjusts the gain by the AGC circuit. Adjust the signal level.
[0057]
The A / D converter 216 converts each pixel signal of the image signal output from the signal processing circuit 215 into a digital signal. The A / D converter 216 converts each pixel signal (analog signal) into a digital image signal (for example, an 8-bit digital image signal) based on an A / D conversion clock input from a timing generation circuit (not shown). Convert to
[0058]
A WB (white balance) circuit 217 converts the pixel values of the pixel data of each of the R, G, and B color components output from the A / D converter 216, thereby converting the image data (photographed image) into a WB image. The correction is performed. For example, the WB circuit 217 adjusts the pixel values of the R, G, and B color components based on the color component values (RGB values of all pixels) of the image data. This WB correction will be further described later.
[0059]
The γ correction circuit 218 corrects the gradation characteristics of the image data (captured image) input from the WB circuit 217. This gradation correction will be further described later.
[0060]
The color conversion / color correction circuit 219 converts the color space of the image data input from the γ correction circuit 218 into, for example, a YCrCb system, and performs color correction for improving color reproducibility.
[0061]
The image memory 220 is a memory that temporarily stores image data (captured image) obtained by the image sensor 213 and subjected to the above-described image processing. The image memory 220 has a storage capacity of at least several frames. That is, the image memory 220 has at least a storage capacity for several frames of pixel data corresponding to the number of pixels of the image sensor 213, and each pixel data is stored at a corresponding pixel position.
[0062]
The USB I / F 230 is a communication interface conforming to the USB standard for enabling communication with the projector unit 10 and the personal computer 30. The USB I / F 230 is connected to the USB I / F 130 of the projector unit 10 via a cable, and is connected to the projector control unit 100 and the camera control unit. Data transmission / reception with the unit 200 is possible.
[0063]
The camera control unit 200 has a microcomputer (CPU), a RAM 20a, and a ROM 20b, and controls the driving of each member of the camera unit 20 organically to control the operation of the camera unit 20 overall. Each function of the camera control unit 200 is realized by the CPU executing a program stored in the ROM 20b.
[0064]
In addition, the camera control unit 200 performs an image (projected image) optically projected on the whiteboard W by the projector unit 10 and an image (captured image) obtained by capturing the whiteboard W by the camera unit 20. It has various functions for correcting gradation characteristics, WB and distortion. Further, the camera control unit 200 also has a function of photographing the writing on the whiteboard W and recording the contents of the conference.
[0065]
FIG. 3 is a block diagram for explaining various functions of the camera control unit 200. FIG. 3 specifically illustrates functions realized by the CPU.
[0066]
As shown in FIG. 3, a camera control unit 200 includes a WB control function 201 for controlling WB correction for a projected image and a captured image, and a tone control function for correcting a tone characteristic for a projected image and a captured image. 202 is provided. In addition, the camera control unit 200 includes an AF area extraction function 203, an evaluation value calculation function 204, a focus control function 205, and a distance calculation function 206 for correcting distortion in the projected image and the captured image.
[0067]
Note that the various functions shown in FIG. 3 will be described in detail later when white balance (WB) correction, gradation characteristic correction, image distortion correction, and recording of conference contents are specifically described. I do.
[0068]
<(1-2-3) @ Internal configuration of personal computer>
As shown in FIG. 2, the personal computer 30 includes a storage unit 32, an output I / F 33, a USB I / F 34, an LCD 35, and a PC control unit 300.
[0069]
The storage unit 32 is composed of, for example, a hard disk or the like, and stores programs and image data described later. The image data stored in the storage unit 32 is image data that is a source of a projected image projected from the projector unit 10 onto the whiteboard W. In the following, the image data on which the projected image is based is referred to as a “projected original image”.
[0070]
The output I / F 33 is an interface for transmitting data to the projector unit 10. Specifically, an analog RGB signal output from the PC control unit 300 is transmitted to the analog RGB decoder 111 of the projector unit 10.
[0071]
The USB I / F 34 is a communication interface conforming to the USB standard for enabling communication with the camera unit 20. The USB I / F 34 is connected to a USB I / F 230 of the camera unit 20 by a cable, and is connected to the camera control unit 200 and the PC control unit 300. It is possible to transmit and receive data between.
[0072]
The LCD 35 is a display device for displaying an image based on image data or the like stored in the storage unit 32.
[0073]
The PC control unit 300 has a microcomputer (CPU), a RAM 30a, and a ROM 30b, and is a unit that integrally controls each unit of the personal computer 30. Then, a program stored in the storage unit 32 is loaded into the RAM 30a of the PC control unit 300 and executed by the CPU, thereby realizing various functions of the PC control unit 300. The functions of the PC control unit 300 include, for example, an analog RGB encoder function 301, a correction value calculation function 302, an image correction function 303, an area extraction function 304, and an image synthesis function 305.
[0074]
Further, the PC control unit 300 executes a program (projection program) for controlling the operation of the camera-equipped projector 2 stored in the storage unit 32, so that the camera-equipped projector 2 and the personal computer 30 operate as the projection system 1. Execute. That is, in the projection system 1, the projector unit 10 and the camera unit 20 are controlled by the personal computer 30 via the USBs 34, 130, and 230.
[0075]
For example, when the user operates a keyboard or the like (not shown) provided in the personal computer 30, the PC control unit 300 executes a projection program for controlling the operation of the projector with camera 2. When the user performs various operations on the keyboard and the like, operations such as WB correction, gradation characteristic correction, image distortion correction, and recording of meeting contents, which are characteristic parts of the present invention, are executed. Note that these operations will be described later in detail.
[0076]
When the user operates a keyboard or the like of the personal computer 30 to start projection of an image by the projector unit 10, the PC control unit 300 reads an original projection image from the storage unit 32. Then, an analog RGB signal is generated by the analog RGB encoder function 301 and transmitted to the projector unit 10 via the output I / F 33. Further, an image (projected original image) having the same contents is displayed on the LCD 35 in synchronization with the projection image projected on the whiteboard W by the projector unit 10.
[0077]
Further, the PC control unit 300 acquires image data (captured image) from the camera unit 20 via the USB I / F 34 by a user's operation of a keyboard or the like. Note that this photographed image is an image obtained by photographing the whiteboard W.
[0078]
The correction value calculation function 302 calculates a correction value for correcting a distortion of a captured image based on a shooting magnification ratio calculated by a camera control unit 200 described later. The image correction function 303 corrects the distortion of the captured image based on the distortion correction value calculated by the correction value calculation function 302. Further, the range that can be photographed by the camera unit 20 includes all the projection area where the image is projected on the whiteboard W, and also includes the surrounding area. The extraction and enlargement processing of the projection area is performed so that the area corresponding to the projection area occupies the entire captured image. The calculation of the correction value in the correction value calculation function 302, the distortion correction of the captured image in the image correction function 303, and the extraction and enlargement processing of the projection area will be described later in detail when describing the operation.
[0079]
By the way, when the PC control unit 300 obtains image data from the camera unit 20, the content of the meeting is simply determined by capturing the whiteboard W to obtain a captured image, or by capturing the whiteboard W and capturing it together with the projected image. May be recorded. If a captured image is simply obtained, the storage unit 32 stores the captured image on which the distortion correction of the captured image has been performed by the image correction function 303. On the other hand, if the contents of the meeting are recorded together with the projected image, the region extraction function 304 performs a distortion correction of the captured image by the image correction function 303, and a region extraction function 304 from the captured image subjected to the extraction / enlargement processing of the projection region. An area corresponding to a portion to which a color or the like is directly added by, for example, manually writing directly on the whiteboard W using the marker MK or the like is extracted. That is, the region extracting function 304 functions as a unit that extracts a region corresponding to a visual element directly attached to the whiteboard W (projection plane) from an image acquired by the camera unit 20.
[0080]
Then, the image synthesis function 305 performs data synthesis (specifically, as digital data) of the area extracted by the area extraction function 304 and a projection original image to be projected on the whiteboard W (projection plane) by the projector unit 10. (Compositing) to generate image data, and store it in the storage unit 32.
[0081]
The processing in the area extracting function 304 and the image synthesizing function 305 will be described later in detail when describing the operation.
[0082]
<(1-3) Description of operation of projection system>
<(1-3-1) WB correction of projected image>
In the personal computer 30, when the user performs various operations on the keyboard and the like of the personal computer 30 while the projection program is being executed, the operation of WB correction of the projected image is executed.
[0083]
Hereinafter, the operation of WB correction of a projected image will be described with reference to FIGS.
[0084]
When the operation of the WB correction of the projected image starts, a photographing signal is transmitted from the PC control unit 300 to the camera control unit 200 to perform photographing for the WB correction of the projected image. Then, the camera control unit 200 transmits a command to the projector unit 10 and synchronizes with the vertical synchronizing signal in the next field, regardless of the input of the analog RGB signal from the personal computer 30. The image data is set so as to be gray, which is an achromatic color, and to have a spatially uniform image level. That is, in synchronization with the next vertical synchronizing signal, the projector unit 10 projects achromatic gray light on the entire surface of the whiteboard W as the projection surface. For example, when the digital image data is 8-bit data, the entire gray light can be generated by setting each pixel value of each color of RGB to be 100/255 gradation.
[0085]
The achromatic color light projected from the projector unit 10 is not limited to the whole gray color in which each pixel value of each color of RGB is set to be 100/255 tones, but other colors. May be set as follows. However, from a human visual point of view, it is preferable that the light be an achromatic color light having a middle gradation such that each pixel value of each color of RGB is 100/255 gradation.
[0086]
On the other hand, the camera control unit 200 issues a trigger signal for photographing to the image sensor driving circuit 214, and the camera unit 20 projects achromatic gray light in the entire surface in synchronization with the next vertical synchronization signal. A digital image is obtained by photographing the whiteboard W (projection plane). At this time, under the control of the camera control unit 200, the lens drive circuit 212 adjusts the aperture value of the photographing lens 20R so that the exposure amount exceeds the allowable amount of the image sensor 213 and the output of the image sensor 213 does not saturate. Set to a predetermined aperture value. Here, since the shutter speed is determined by the vertical synchronizing signal, the entire gray light is projected from the projector unit 10 and the light emission amount is also determined, the predetermined aperture value is easily adjusted so that the output of the image sensor 213 is not saturated. Can be set to
[0087]
The WB control function 201 of the camera control unit 200 shown in FIG. 3 outputs the light from the A / D converter 216 when the projector unit 10 projects substantially uniform light of gray on the entire surface onto the whiteboard W for shooting. To obtain a digital image signal. Then, based on the obtained digital image signal, the ratio of R / G and B / G pixel values is calculated for each pixel, and the average value (Rs / Gs, Bs / Gs) for all pixels is calculated. It is determined (gr, gb). In order to increase the efficiency of the calculation, the image may be divided into a plurality of blocks having a predetermined area, and gr and gb may be obtained only from the pixels representative of each block. May be used to determine gr and gb.
[0088]
Next, the WB control function 201 sets a WB correction value for WB correction on the projection image based on the calculated (gr, gb).
[0089]
FIG. 4 is a diagram for explaining a method of setting a WB correction value in the WB control function 201. In FIG. 4, a point 510 is located at coordinates (gr, gb) in the XX-YY coordinate system. As the value of the XX coordinate of the coordinates (gr, gb) is larger and the value of the YY coordinate is smaller, Rs / Gs becomes larger than Bs / Gs, so that the original image is reddish. Conversely, as the value of the XX coordinate is smaller and the value of the YY coordinate is larger, Rs / Gs is smaller than Bs / Gs, so that the original image is bluish. Therefore, a point near the straight line Le extending at an angle of 45 ° from the origin is a point corresponding to an image in which WB is adjusted.
[0090]
As shown in FIG. 4, a predetermined region Ra is set in the XX-YY coordinate system according to a preset WB setting, and is affected by external light, the color of the projection surface, the deterioration of the light source 121, and the like. As shown in a point 510, when the coordinates (gr, gb) are not within the predetermined area Ra, the WB is changed so that the position of the point 510 is converted to the closest position (point 511) in the area Ra. Set the correction value. When the point 510 is included in the region Ra, the position of the point 510 is not converted, so that the WB correction value is not set. If the coordinates (gr, gb) are not within a certain range, it is regarded as abnormal data and no WB correction value is set.
[0091]
The WB correction value for the projection image set in the WB control function 201 is transmitted to the projector control unit 100. Then, the projector control unit 100 changes the LUT (conversion table) according to the received WB correction value, and changes the display color temperature of the image projected from the projector unit 10 by the element drive control function 101. That is, here, the element drive control function 101 is projected on the whiteboard W by the projector unit 10 based on the image obtained by photographing the whiteboard W on which the entire gray light is projected by the projector unit 10. Function as a means for correcting the white balance of the projected image.
[0092]
<(1-3-2) WB correction of captured image>
When the user performs various operations on the keyboard and the like of the personal computer 30 while the projection program is being executed on the personal computer 30, the WB correction operation of the captured image is executed.
[0093]
In the above-described WB correction operation on the projected image, the entire gray light is projected from the projector unit 10 onto the whiteboard W during the WB correction shooting. In the WB correction operation on the captured image, for example, the WB correction is performed. At the time of shooting, the projector unit 10 projects substantially uniform white light over the entire surface of the white board W as achromatic light. That is, in synchronization with the image capturing by the camera unit 20, the projector unit 10 projects achromatic white light on the entire whiteboard W, which is a projection surface. For example, when the digital image data is 8-bit data, the entire white light can be generated by setting each pixel value of each color of RGB to have the maximum luminance, that is, 255/255 gradations.
[0094]
Note that the color of the achromatic color light projected from the projector unit 10 is not limited to a white color in which the pixel values of each of the RGB colors are set to have 255/255 gradations. The color may be the same as the color of the light that is projected as illumination light for photographing from the projector unit 10 when the directly described or pasted material or the like is photographed by the camera unit 20.
[0095]
Until the setting of the WB correction value for the captured image, the light projected from the projector unit 10 to the whiteboard W during the shooting for the WB correction is different from the setting of the WB correction value for the projection image only. A function 201 sets a WB correction value for a captured image in the same manner as setting a WB correction value for a projection image.
[0096]
When the WB correction value is set, the set values of the R, G, and B gains in the WB circuit 217 are obtained. Then, the WB correction is performed on the captured image obtained by the camera unit 20 in the WB circuit 217 based on the obtained gain setting value. That is, here, the WB circuit 217 controls the whiteboard W by the camera unit 20 based on an image obtained by shooting the whiteboard W on which achromatic white light is projected by the projector unit 10. It functions as a means for correcting a white balance of a photographed image obtained by photographing.
[0097]
As described above, here, when the user operates the keyboard or the like of the personal computer 30 variously while the projection program is being executed on the personal computer 30, the whiteboard W (projection surface) on which the achromatic light is projected is photographed. To get an image. Then, based on the obtained image, the WB is corrected for at least one of a projection image projected on the whiteboard W and a captured image obtained by capturing the whiteboard W. As a result, it is possible to correct a change in the color tone of the projected image or the captured image due to the influence of the unique color of the surface of the whiteboard W, the color of the external light, the color of the light source 121, and the like.
[0098]
Note that the above-described WB correction operation may be performed at the start of use of the projection system 1 and every time a predetermined time elapses from the start of use. With such a configuration, it is possible to cope with a case where the influence of external light changes over time. Here, the timing for projecting the achromatic light and the timing for shooting are synchronized, and the time for projecting the achromatic light is very short. , And the projected image is not difficult to see.
[0099]
<(1-3-3) Correction of gradation characteristics>
In the personal computer 30, when the user executes various operations on the keyboard or the like of the personal computer 30 while the projection program is being executed, the operation of correcting the gradation characteristics of the projected image or the photographed image is executed.
[0100]
Hereinafter, the operation of correcting the gradation characteristic will be described with reference to FIGS.
[0101]
When the operation of correcting the gradation characteristic is started, a photographing signal is transmitted from the PC control unit 300 to the camera control unit 200 to perform photographing for gradation characteristic correction. Then, the camera control unit 200 transmits a command to the projector unit 10 and synchronizes with the vertical synchronization signal in the next field, regardless of the input of the analog RGB signal from the personal computer 30, regardless of the digital image stored in the frame memory 112. The data is set so as to be a gray scale chart which is a pattern in which the gradation stored in the ROM 10b changes stepwise. Therefore, in synchronization with the next vertical synchronizing signal, the projector unit 10 projects a pattern (gray scale chart) whose luminance changes stepwise on the whiteboard W. A specific example of the gray scale chart will be described later.
[0102]
In response to this, the camera control unit 200 issues a trigger signal for shooting to the image sensor driving circuit 214, and in synchronization with the next vertical synchronization signal, the camera unit 20 causes the brightness projected on the whiteboard W to change stepwise. An image is obtained by photographing a pattern that changes gradually (gradual gradation distribution pattern). At this time, under the control of the camera control unit 200, the lens drive circuit 212 adjusts the aperture value of the photographing lens 20R so that the exposure amount exceeds the allowable amount of the image sensor 213 and the output of the image sensor 213 does not saturate. Set to a predetermined aperture value. Here, since the shutter speed is determined by the vertical synchronization signal, the gray scale chart is projected from the projector unit 10 and the light emission amount is also determined, the predetermined aperture value is easily adjusted so that the output of the image sensor 213 is not saturated. Can be set.
[0103]
Next, in the gradation control function 202 of the camera control unit 200 shown in FIG. 3, when the gray scale chart is projected from the projector unit 10 onto the whiteboard W and the image is captured, the gray scale chart is output from the A / D converter 216. A correction value for gradation characteristic correction is calculated based on the digital image signal.
[0104]
Here, a method of calculating a correction value for gradation characteristic correction in the gradation control function 202 will be described.
[0105]
FIG. 5 is a schematic diagram illustrating an image G1 based on a digital image signal output from the A / D converter 216 at the time of photographing for gradation characteristic correction. In FIG. 5, the inside of the frame PS is a portion corresponding to the gray scale chart projected from the projector unit 10.
[0106]
The gray scale chart has achromatic regions S1 to S7 in which luminance changes at regular intervals and in steps. Here, the region S1 is white and the region S7 is black, and the brightness is set so as to gradually decrease in the order of the regions S1 to S7. Further, the gray scale chart is set so that portions other than the regions S1 to S7 have an achromatic color having the same luminance as that of the region S4.
[0107]
In addition, as shown in FIG. 5, the projection area projected on the whiteboard W by the projector unit 10 does not completely match the area on the whiteboard W captured by the camera unit 20. Therefore, the gradation control function 202 uses the image data on the broken line SL1 extending to the left and right at the center of the image G1, which is considered to surely include the regions S1 to S7 of the gray scale chart. If the number of pixels of the image sensor 213 is 1024 × 768, left and right data that do not correspond to the regions S1 to S7 are excluded from the data of 1024 pixels on the broken line SL1. For example, looking at the change in the luminance from the −X direction to the + X direction on the broken line SL1, the luminance (pixel value) increases significantly at the region S1 and significantly increases at the position exiting the region S7 (pixel value). ), The exclusion of left and right data can be easily achieved by detecting changes in brightness.
[0108]
Thereafter, the gradation control function 202 obtains an average value of the pixel values of each of the regions S1 to S7 from the pixel values of the obtained image data and the positional relationship between the regions S1 to S7. Then, the pixel value of the brightest area S1 of the gray scale chart is set to 1, and the pixel values of the areas S2 to S7 are converted based on the pixel value. That is, here, the ratio of the brightness of the regions S2 to S7 based on the brightness of the region S1 is obtained.
[0109]
FIG. 6 is a schematic diagram illustrating the ideal brightness ratio of the regions S1 to S7. In FIG. 6, the horizontal axis indicates the position in the X direction in the image G1, and the vertical axis indicates the brightness ratio. In addition, the brightness ratios corresponding to the regions S1 to S7 are shown as Step1 to Step7. As shown in FIG. 6, in an ideal case where there is no influence of external light, assuming that the brightness ratio of Step 1 is 1, the brightness ratio of Step 7 is 0.05.
[0110]
FIG. 7 is a schematic view illustrating a brightness ratio when there is an influence of external light. In FIG. 7, the solid line indicates the brightness ratio when there is the influence of external light, and the broken line indicates the ideal brightness ratio shown in FIG. 6 for reference.
[0111]
As shown in FIG. 7, when there is an influence of external light, the difference in brightness between Step 5 and Step 7 is significantly reduced. In such a case, a portion of the projection image projected on the whiteboard W having a low luminance has a blackout condition or the like, and an intended image cannot be projected. In order to solve such a problem, here, the gradation control function 202 calculates a correction value for gradation characteristic correction to correct the gradation characteristic.
[0112]
FIG. 8 is a diagram for explaining a method of calculating a correction value for gradation characteristic correction. In FIG. 8, the solid line shows the ideally corrected brightness ratio when there is an influence of external light shown in FIG. 7, and the broken line shows the ideal brightness ratio shown in FIG. Is shown for reference.
[0113]
As illustrated in FIG. 8, the gradation control function 202 determines that the brightness ratios of Step 1 to Step 7 are equally spaced and based on the brightness ratio of Step 1 and Step 7 when there is the influence of external light illustrated in FIG. 7. The brightness ratio is calculated so as to decrease in order. The calculated brightness ratio is obtained by ideally correcting the brightness ratio when there is an influence of external light shown in FIG. Note that the gradation control function 202 is configured such that the brightness ratio of Step 1 to Step 7 decreases at regular intervals and in order, but the present invention is not limited to this, and correction may be made so that specific brightness is emphasized. good.
[0114]
As shown in FIG. 8, the difference in brightness between Step 1 to Step 7 is smaller than the difference between ideal brightness between Step 1 and Step 7 shown in FIG. 6, but the difference between the brightness between Step 5 and Step 7 is smaller. The problem that the difference is significantly reduced has been solved. Therefore, the tone control function 202 calculates a tone conversion characteristic for ideally correcting the brightness ratio shown in FIG. 7 to the brightness ratio shown in FIG. 8 from the brightness ratio under the influence of external light. I do.
[0115]
FIG. 9 is a diagram illustrating a gradation conversion characteristic when there is an influence of external light. In FIG. 9, the horizontal axis indicates the gradation characteristics before the conversion, the vertical axis indicates the gradation characteristics after the conversion, and the curve 9C indicates the relationship between the gradation characteristics before and after the conversion (the gradation conversion). Characteristic). This gradation conversion characteristic is a correction value for gradation characteristic correction for ideally correcting the brightness ratio under the influence of external light shown in FIG. 7 to the brightness ratio shown in FIG. It becomes.
[0116]
The camera control unit 200 transmits the correction value (tone conversion characteristic) for the tone characteristic correction calculated by the tone control function 202 to the projector control unit 100 via the USB I / Fs 130 and 230. Then, the projector control unit 100 corrects the LUT (conversion table) stored in the ROM 10b according to the correction value for correcting the gradation characteristic. After the LUT is corrected, a projection image considering the influence of external light and the like is projected on the whiteboard W under the control of the element drive control function 101.
[0117]
Therefore, here, the element drive control function 101 performs the projector unit based on an image acquired by photographing the whiteboard W onto which the pattern in which the luminance changes stepwise by the projector unit 10 is captured by the camera unit 20. 10 functions as means for correcting the gradation characteristics of the projection image projected on the whiteboard W.
[0118]
Further, as described above, even when writing on a whiteboard W, or attaching a document or a photograph when photographing, the gradation characteristics of the photographed image may be affected by the influence of external light, the reflectance of the whiteboard W, and the like. May change. Therefore, here, in order to suppress such a change in the gradation characteristic of the captured image, the gradation control function 202 corrects the gradation characteristic correction value (gradation conversion characteristic) calculated by the above method. Is used. Specifically, the correction value calculated by the gradation control function 202 is set as a correction value in the γ correction circuit 218 in the projector control unit 100. When the correction value in the γ correction circuit 218 is set as described above, thereafter, the gradation characteristics of the captured image can be corrected in consideration of the influence of external light and the like.
[0119]
Therefore, here, the γ correction circuit 218 performs the camera unit 20 based on an image obtained by photographing the whiteboard W onto which the pattern in which the luminance changes stepwise by the projector unit 10 is captured by the camera unit 20. Functions as a means for correcting a gradation characteristic of a photographed image obtained by photographing the whiteboard W.
[0120]
As described above, here, the pattern in which the luminance changes stepwise is projected on the whiteboard W (projection surface), and the pattern is photographed to acquire an image. Then, based on the acquired image, the gradation of at least one of a projected image projected on the whiteboard W and a photographed image obtained by photographing the whiteboard W (projection surface) or the like is obtained. Correct the characteristics. As a result, it is possible to correct a change in the gradation characteristic of the projection image or the captured image due to the influence of external light, the optical reflectance on the surface of the whiteboard W, and the like.
[0121]
Note that the above-described calculation of the correction value for gradation characteristic correction may be performed at the start of use of the projection system 1 and every time a predetermined time elapses from the start of use. With such a configuration, it is possible to cope with a case where the influence of external light or the like changes over time. And here, the timing of projecting the grayscale chart and the timing of shooting are synchronized, and the time of projecting the grayscale chart is very short, so that a person who is watching the projected image notices. And the projected image is not difficult to see.
[0122]
Further, in the above-described correction of the gradation characteristics, the gray scale chart is projected on the white board W. However, the present invention is not limited to this. For example, three patterns in which the luminance of the three colors of RGB changes stepwise are used. The three patterns may be separately projected, the three patterns may be separately photographed, and the correction values for the gradation characteristic correction may be calculated for each of the three colors RGB. However, when the pattern in which the luminance changes stepwise is set to a pattern in which the achromatic luminance changes stepwise as in a gray scale chart, the correction value can be calculated only by photographing one pattern. It is possible to efficiently correct the gradation characteristics of a projected image or a captured image.
[0123]
As described above, a correction value for WB correction can be calculated and set based on an image obtained by photographing a gray scale chart. With such a configuration, WB correction and correction of gradation characteristics can be efficiently performed.
[0124]
<(1-3-4) Image distortion correction>
FIG. 10 is a schematic view exemplifying a case of projecting and photographing from an oblique direction with respect to the whiteboard W. FIG. 10 is a view of the whiteboard W and the projector with camera 2 as viewed from directly above. In FIG. 10, a solid line 10L schematically showing the angle of view of the projection lens 10R and the angle of view of the photographing lens 20R. A dotted line 20L is schematically shown. In the state shown in FIG. 10, the optical axes of the projection lens 10R and the photographing lens 20R substantially coincide with the Y axis, and the whiteboard W is rotated about the Z axis by an angle θ with respect to the XZ plane. Placed in the position.
[0125]
Hereinafter, distortion correction of a captured image and a projected image will be described by taking as an example a case where a whiteboard W and a projector with a camera 2 are arranged as shown in FIG.
[0126]
FIG. 11 shows a state in which the projection image Sm1 projected on the whiteboard W is projected from the projector unit 10 without correcting distortion of the projection image described later, in front of the whiteboard W (in the direction normal to the projection plane of the whiteboard W). FIG. At this time, as shown in FIG. 11, a trapezoidal distortion occurs in the projection image Sm1. That is, here, when performing oblique projection in which an image is projected from a diagonal direction on the whiteboard W due to the relative positional relationship between the whiteboard W and the projector unit 10, the projected image on the whiteboard W is Distorted.
[0127]
In such a case, there arises a problem that the content intended by the creator of the image cannot be accurately transmitted to the people who participated in the conference. Therefore, here, it is necessary to correct the trapezoidal distortion caused by the oblique projection as shown in FIG. 11 to obtain a projection image Sm1 ′ as projected from the front of the whiteboard W as shown in FIG. .
[0128]
FIG. 13 is a schematic diagram illustrating a photographed image obtained when the camera unit 20 performs oblique photographing in which the whiteboard W is photographed in an oblique direction without performing distortion correction of a photographed image described later. . Although the size of each character “ABCD” shown in FIG. 13 is substantially the same, trapezoidal distortion occurs in the photographed image by oblique photographing, and the right end dimension is smaller than the left end dimension.
[0129]
In such a case, when data and the like are difficult to read, or when a photographed image obtained by photographing writing on a whiteboard W described later and a projection original image for projection are combined with data, A deviation from the original projection image occurs. Therefore, here, as shown in FIG. 13, it is necessary to correct a trapezoidal distortion or the like caused by oblique photographing, and to obtain an image photographed from the front of the whiteboard W as shown in FIG.
[0130]
Hereinafter, the operation of correcting the distortion of the projected image and the captured image caused by the relative positional relationship between the camera-equipped projector 2 and the whiteboard W will be described with reference to FIGS.
[0131]
When the user performs various operations on the keyboard or the like of the personal computer 30 while the projection program is being executed in the personal computer 30, a distortion correction operation of the projection image is executed, or a distortion correction operation of the photographed image starts, and the camera unit is started. 20 photographs the whiteboard W and corrects the distortion of the acquired photographed image.
[0132]
When the distortion correction operation is started, an imaging signal is transmitted from the PC control unit 300 to the camera control unit 200 in order to execute imaging for distortion correction. Then, the camera control unit 200 transmits a command to the projector unit 10 and synchronizes with the vertical synchronization signal from the next field regardless of the input of the analog RGB signal from the personal computer 30, regardless of the input of the analog RGB signal. The image data is set so as to be a pattern (focus detection pattern) capable of detecting the distance between each position of the whiteboard W and the camera-equipped projector 2. Therefore, the projector unit 10 projects a pattern for focus detection on the whiteboard W in synchronization with the next vertical synchronization signal.
[0133]
FIG. 15 is a diagram illustrating a focus detection pattern AFP projected from the projector unit 10 onto the whiteboard W. For convenience of illustration, FIG. 15 is drawn with black and white inverted. However, the focus detection pattern AFP includes a bright white frame PF that is a rectangular outline portion, a plurality of focus detection portions A1 to A5, and black. Background BG. Each of the portions A1 to A5 is a striped region in which a plurality of white lines are arranged, and is distributed and arranged substantially at the center and four corners of the focus detection pattern AFP.
[0134]
In response to this, the camera control unit 200 issues a trigger signal for photographing to the image sensor driving circuit 214, and the camera unit 20 synchronizes with the next vertical synchronization signal to cause the camera unit 20 to detect the focus projected on the whiteboard W. By capturing a pattern, an image related to a focus detection pattern composed of a plurality of pixels is input. At this time, under the control of the camera control unit 200, the aperture of the photographing lens 20R is set close to the opening so that focus detection becomes easy. Therefore, the brightness of the focus detection pattern AFP projected from the projector unit 10 is set so that the exposure amount exceeds the allowable amount of the image sensor 213 and the output of the image sensor 213 does not saturate.
[0135]
Here, since the shutter speed is determined by the vertical synchronization signal and the aperture of the photographing lens 20R is almost opened, the brightness of the focus detection pattern AFP is easily set so that the output of the image sensor 213 does not saturate. be able to.
[0136]
Next, a digital image signal is output from the A / D converter 216 to the camera control unit 200 when the focus detection pattern AFP is projected from the projector unit 10 onto the whiteboard W and shooting is performed.
[0137]
FIG. 16 is a schematic diagram illustrating an image G3 obtained by photographing the whiteboard W onto which the focus detection pattern AFP is projected by the camera unit 20. In FIG. 16, the inside of the frame PF ′ is an area corresponding to the focus detection pattern AFP projected from the projector unit 10. As shown in FIG. 16, areas A1 'to A5' surrounded by dotted lines are areas for focus detection corresponding to portions A1 to A5 for focus detection. A portion where the focus detection pattern AFP is not projected from the projector section 10 is a dark area.
[0138]
Therefore, the AF area extraction function 203 shown in FIG. 3 extracts the outer edge of the bright white frame PF ′ by a change in color or gradation, etc., so that the boundary pixel of the area where the focus detection pattern AFP is projected is displayed. Can be detected. Then, the AF area extraction function 203 specifies the positions of the areas A1 'to A5' for focus detection, and extracts the image data of the areas A1 'to A5'. The positions of the regions A1 'to A5' for focus detection can be easily specified by, for example, a positional relationship with the frame PF 'or a change in luminance.
[0139]
The evaluation value calculation function 204 captures the image data of the areas A1 'to A5' from the AF area extraction function 203, and evaluates the contrast of each area. That is, the evaluation value calculation function 204 obtains the evaluation value as the sum of the contrast values between adjacent pixels in the areas A1 'to A5', similarly to the one performed in the contrast type autofocus operation.
[0140]
Then, the focus control function 205 sends a control signal to the lens drive circuit 212 to perform shooting several to several tens of times while gradually changing the focal position of the shooting lens 20R. As described above, the evaluation value calculation function At 204, the evaluation values for the areas A1 'to A5' are calculated. That is, the focus control function 205 functions as a unit that controls the focus of the photographing lens 20 </ b> R in the camera unit 20, and the evaluation value calculation function 204 performs the plurality of focus detections in the image G <b> 3 related to the focus detection pattern AFP. Each of the areas A1 ′ to A5 ′ corresponding to the parts A1 to A5 functions as a unit that calculates an evaluation value regarding the focusing state of the photographing lens 20R.
[0141]
FIG. 17 is a schematic diagram illustrating curves C14, C3, and C25 representing the relationship between the evaluation values calculated for the regions A1 'to A5' and the position of the imaging lens 20R in the optical axis direction. In FIG. 17, the horizontal axis indicates the lens position X and the vertical axis indicates the evaluation value C, and the curves C14, C3, and C25 are the evaluation values in the areas A1 'and A4', the area A3 ', and the areas A2' and A5 ', respectively. The relationship between C and the lens position X is shown.
[0142]
As shown in FIG. 17, the distance calculation function 206 determines the lens position X = X1 at which the evaluation value of the curve C14 becomes the maximum and the lens position at which the evaluation value of the curve C3 becomes the maximum in order from the retraction side to the retraction side. X = X2, the lens position X = X3 where the evaluation value of the curve C25 is maximum is detected. Then, the distance calculation function 206 calculates distances from the camera-equipped projector 2 to the portions A1 to A5 projected on the whiteboard W by calculating focal positions corresponding to the lens positions X = X1, X2, and X3. I do. Here, the distance to the parts A1 and A4 is calculated as L1, the distance to the part A3 is calculated as L2, and the distance to the parts A2 and A5 is calculated as L3.
[0143]
Here, when distortion correction of a captured image is performed, the camera unit 20 acquires a captured image to be subjected to distortion correction.
[0144]
Specifically, a photographing signal for photographing the whiteboard W is transmitted from the PC control unit 300 to the camera control unit 200. Then, the camera control unit 200 transmits a command to the projector unit 10, and in synchronization with the vertical synchronization signal, the next field stores the digital image stored in the frame memory 112 regardless of the input of the analog RGB signal from the personal computer 30. The data is set so as to be a bright white image. Accordingly, bright light of white color is projected on the entire whiteboard W.
[0145]
On the other hand, the camera control unit 200 outputs a trigger signal for shooting to the image sensor driving circuit 214, and the camera unit 20 synchronizes with the next vertical synchronizing signal to project a white board on which white bright light is projected. W is photographed. At this time, the lens drive circuit 212 sets the aperture value of the photographing lens 20R to a predetermined aperture value so that the exposure amount exceeds the allowable amount of the image sensor 213 and the output of the image sensor 213 does not saturate. Here, since the shutter speed is determined by the vertical synchronization signal, the light emission from the projector unit 10 is entirely white light, and the light emission amount is also determined, the predetermined aperture value is easily set so that the output of the image sensor 213 does not saturate. Can be set to
[0146]
The photographed image photographed and acquired by the camera unit 20 is subjected to processing of each unit of the photographing function unit 210, and is transferred to the PC control unit 300 under the control of the camera control unit 200.
[0147]
Here, a method of calculating a correction value for distortion correction will be briefly described.
[0148]
FIG. 18 is a diagram for explaining a method of calculating a correction value for distortion correction. FIG. 18 is a schematic diagram illustrating a positional relationship between the whiteboard W, the photographing lens 20R, and the image sensor 213 in a plan view. In FIG. 18, points P, F, and G are points on the whiteboard W, the point P corresponds to the position of the center of the portion A3 projected on the whiteboard W, and the point F is on the whiteboard W. The point G corresponds to the center position of the projected portions A1 and A4, and the point G corresponds to the center position of the projected portions A2 and A5 on the whiteboard W.
[0149]
In FIG. 18, L is the optical axis of the photographing lens 20R. Point A is the intersection of the imaging plane of the imaging element 213 and the optical axis L. N0, N1, and N2 are line segments parallel to the imaging surface of the imaging device 213 passing through the points P, F, and G on the whiteboard W, respectively, and N3 is a subject passing through a point A on the imaging surface of the imaging device 213. This is a line segment parallel to the plane (line segment FG). N5 is a line segment indicating the lens surface of the photographing lens 20R. Point O is the intersection of the lens surface of the taking lens 20R and the optical axis L. Point B is the intersection of the extension of the line segment OF and the imaging surface of the imaging element 213. Point C is the intersection of the extension of the line segment OG and the imaging surface of the imaging element 213. A point Q is an intersection between the line segment N1 and the optical axis L, a point R is an intersection between the line segment N2 and the optical axis L, and points D and E are an extension of the line segment N0 and the line segment BF and a line segment GC, respectively. Is the intersection with Points B 'and C' are the intersections of the line segment N3 and the extension lines of the line segment FB and the line segment GC, respectively.
[0150]
As shown in FIG. 18, the light image emitted from the FG on the whiteboard W forms an image on the imaging surface of the image sensor 213. However, since the imaging surface and the whiteboard W are inclined by the inclination angle θ, The light image BC formed on the surface is equivalent to the light image emitted from the DE projected on the imaging surface.
[0151]
The imaging magnifications at points A, B, and C on the imaging surface of the imaging element 213 are respectively Ma, Mb, and Mc, and the object distances are L2 (= OP), L1 (= OQ), and L3 (= OR). Then, since Mb = Ma · OP / OQ = Ma · L2 / L1 and Mc = Ma · OP / OR = Ma · L2 / L3, Mb> Ma> Mc, and the light image formed on the imaging surface is 13, an image distorted in a trapezoidal shape as shown in FIG. 13, and only the point A (intersection between the optical axis L and the imaging plane) is completely in focus in the optical image BC.
[0152]
The photographing magnification M is a constant determined by the photographing lens 20R, and is obtained by K (proportional constant) × f (focal length). Therefore, if the distances L1, L2, and L3 at the points P, F, and G on the whiteboard W are measured, the photographing magnification ratio for the point A can be obtained from L2 / L1 and L2 / L3, and this information is used. Then, a correction value for distortion correction can be calculated. Here, strictly speaking, L2 / L1 and L2 / L3 are different, but even if a correction value is calculated using any one of the values as a representative, the error is extremely small. The correction value may be calculated using the average value of L2 / L1 and L2 / L3.
[0153]
By the way, the distortion correction of the photographed image can be performed by the method disclosed in Patent Document 1, but in the present embodiment, the distortion correction is performed by the image correction function 303 of the personal computer 30.
[0154]
The distance calculation function 206 calculates the photographing magnification ratios L2 / L1, L2 / L3, and transfers them to the correction value calculation function 302 of the PC control unit 300. Then, the correction value calculation function 302 calculates a distortion correction value for correcting the distortion of the captured image based on the imaging magnification ratios L2 / L1 and L2 / L3.
[0155]
The correction value calculation function 302 calculates and sets the reduction magnification of each part of the image so that a high-magnification portion of the captured image can be matched with a low-magnification portion. Here, this reduction magnification is a distortion correction value.
[0156]
The reduction magnification Mb ′ at point B in FIG. 18 is Mc / Mb. Here, the reduction magnification at the pixel position Xi in the horizontal direction (X direction) on the imaging surface of the imaging element 213 is Mi, and points B and C Assuming that the pixel positions in the horizontal direction (X direction) on the imaging surface are Xb and Xc, the following equation (1) holds.
[0157]
(Equation 1)

[0158]
Here, since Xb, Xc, Mb, and Mc are also known, the reduction magnification Mi at the pixel position Xi in the horizontal direction (X direction) on the imaging surface can be obtained from Expression (1). For example, when Mc = 0.9, Mb = 1.1, Xb = 1, and Xc = -1, if Xi = 0.5, the reduction magnification Mi = 0.8635 is calculated from the equation (1). You.
[0159]
As described above, the correction value calculation function 302 calculates the local reduction magnification Mi for all pixels in the X direction on the imaging surface of the imaging element 213. Then, based on the distortion correction value calculated by the correction value calculation function 302, the distortion of the captured image is corrected by the image correction function 303. Specifically, a pixel on the same vertical line at the pixel position Xi is subjected to a reduction process at a reduction ratio Mi. The reduction processing can be performed by general image processing such as a nearest neighbor method, a bilinear method, and a bicubic method. That is, the image correction function 303 functions as a unit that corrects distortion of a captured image obtained by capturing the whiteboard W by the camera unit 20 based on the evaluation value calculated by the evaluation value calculation function 204.
[0160]
By such a distortion correction of the photographed image, what is entirely represented in the image is reduced, but a photographed image without trapezoidal distortion can be obtained, and writing or the like on the whiteboard W can be stored in an easily viewable state. Printing can be performed.
[0161]
The correction value calculation function 102 of the projector control unit 100 calculates a correction value for correcting the trapezoidal distortion of the projection image based on the photographing magnification ratios L2 / L1 and L2 / L3 calculated by the distance calculation function 206. You can also. Since the projector unit 10 and the camera unit 20 are integrated, the distance from the image sensor 213 of the camera unit 20 to the whiteboard W and the distance from the display element 123 of the projector unit 10 to the whiteboard are different. , Which almost match. Here, the correction value can be calculated by the same method as the trapezoidal distortion correction widely used in the conventional projector.
[0162]
For example, in order to make the projection image as shown in FIG. 11 into the projection image as shown in FIG. 12, the projection image shown in FIG. Need to be In the correction value calculation function 102, the image stored in the frame memory 112 is calculated in the same manner as the distortion correction value calculation function 302 calculates the reduction ratio Mi for all pixels in the X direction on the imaging surface of the imaging element 213. With respect to the data, the reduction ratio Mi ′ can be calculated for all the pixels in the X direction. Here, in the projector control unit 100, the reduction magnification Mi 'is set as a distortion correction value.
[0163]
After that, the distortion of the projected image is corrected by the scaler 113 based on the distortion correction value set in the projector control unit 100. For example, a pixel on the same vertical line at the pixel position Xi 'is subjected to reduction processing at a reduction ratio Mi'. That is, the scaler 113 functions as a unit that corrects distortion of a projection image projected on the whiteboard W by the projector unit 10 based on the evaluation value calculated by the evaluation value calculation function 204.
[0164]
Here, in order to correctly correct the distortion of the photographed image and the projected image, it is necessary to obtain the distances of three points on the whiteboard W as shown in FIG. That is, the focus detection pattern projected from the projector unit 10 needs three focus detection portions in which not all portions are arranged in a straight line. Considering error factors such as various aberrations of the projection lens 10R and the photographing lens 20R, as shown in FIG. 15, the focus detection pattern includes a total of five focus detection portions at the center and four corners. Is preferred.
[0165]
In addition, here, as in the case of the contrast type autofocus operation, the evaluation value is obtained by calculating the evaluation value as the sum of contrast values between adjacent pixels in the focus detection areas A1 'to A5'. Is measured. Therefore, the pattern drawn on the focus detection portion may be any point or line as long as the evaluation value is changed by driving the photographing lens 20R. However, in order to more accurately measure the distance and better correct the distortion, it is preferable that the pattern drawn on the focus detection portion be such that the evaluation value is easily changed by driving the photographing lens 20R. Note that a pattern whose evaluation value is likely to change by driving the photographing lens 20R is, for example, a pattern in which many lines are arranged as shown in FIG.
[0166]
In the above description, as shown in FIG. 10, the whiteboard W has been described as an example in which the white board W is rotated by an angle θ with respect to the XZ plane with the Z axis as the rotation axis. Even in the state of being rotated by the angle θ with respect to the plane, the distortion of the captured image and the projection image can be similarly corrected. Therefore, here, both the vertical and horizontal trapezoidal distortion correction can be performed.
[0167]
Further, even when the whiteboard W is rotated by the angle θ with respect to the XY plane with the Z axis as the rotation axis, the distance calculation function The distances from the camera-equipped projector 2 to the portions A1 to A5 projected on the whiteboard W can be calculated, and the distortions of the captured image and the projected image can be corrected based on the calculated values. For example, if the distance to the portions A1 to A5 projected on the whiteboard W is known, the distance from the camera-equipped projector 2 to an arbitrary point on the projection image can be estimated by interpolation and extrapolation. Then, the distortion of the photographed image and the projected image can be corrected by obtaining the reduction ratio in the vertical direction and the horizontal direction for each pixel of the photographed image based on the distance. Therefore, here, it is possible to correct distortion in all directions of a projected image or a captured image, which is caused by a relative positional relationship between the projector unit 10 or the camera unit 20 and the whiteboard W.
[0168]
Note that the operation of setting the distortion correction value for the projection image as described above may be performed at the start of use of the projection system 1 and every time a predetermined time elapses from the start of use. By adopting such a configuration, it is possible to cope with a case where the whiteboard W moves halfway. Further, as described above, since the distortion correction value for the captured image is performed every time the image capturing is performed, it is possible to cope with a case where the whiteboard W is moved in the middle. Here, the timing of projecting the focus detection pattern and the timing of shooting are synchronized, and the time of projecting the focus detection pattern is very short. , And the projected image is not difficult to see.
[0169]
As described above, here, when the user performs various operations on the keyboard and the like of the personal computer 30 while the projection program is being executed on the personal computer 30, a pattern including a plurality of focus detection portions (portions A1 to A5) ( The focus detection pattern AFP) is projected on the whiteboard W. For the areas (areas A1 ′ to A5 ′) corresponding to a plurality of focus detection parts in the image input by capturing the pattern projected on the whiteboard W, the evaluation values related to the in-focus state of the photographing lens 20R are calculated. Ask. Further, based on the evaluation value, distortion is corrected for at least one of a projection image projected on the whiteboard W and a photographed image obtained by photographing the whiteboard W. As a result, it is possible to correct a distortion of a projected image or a captured image caused by a relative positional relationship between the projector unit 10 or the camera unit 20 and the whiteboard W.
[0170]
Further, here, one pattern (focus detection pattern AFP) is projected, and the correction value for correcting the distortion of both the projected image and the captured image is calculated based on the image captured and input. Since the calculation can be performed, it is possible to efficiently calculate the correction value for correcting the distortion of both the projected image and the captured image.
[0171]
<(1-3-5) Record of meeting contents>
FIG. 19 is a schematic view illustrating a state on the whiteboard W. As shown in FIG. 19, the inside of the frame PS2 is a projection area where the projection image is projected, and the writing MR using the marker MK is performed in accordance with the projection image.
[0172]
Then, when the projection program is executed on the personal computer 30 and a projection image as shown in FIG. 19 is projected, when the user variously operates a keyboard or the like of the personal computer 30, the projection image is projected on the whiteboard W. A recording operation is performed to record the projection image and the write MR together.
[0173]
When the recording operation is started, a photographing signal for performing photographing for recording is transmitted from the PC control unit 300 to the camera control unit 200. Then, the camera control unit 200 transmits a command to the projector unit 10 and synchronizes with the vertical synchronization signal from the next field regardless of the input of the analog RGB signal from the personal computer 30, regardless of the input of the analog RGB signal. The image data is set such that the entire surface is white, which is a substantially uniform single color. Therefore, in synchronization with the next vertical synchronizing signal, the projector unit 10 projects light of a single color on the whiteboard W.
[0174]
On the other hand, the camera control unit 200 issues a trigger signal for photographing to the image sensor drive circuit 214, and the camera unit 20 projects a single color of entirely white light in synchronization with the next vertical synchronization signal. An image is obtained by photographing the whiteboard W. At this time, under the control of the camera control unit 200, the lens drive circuit 212 adjusts the aperture value of the photographing lens 20R so that the exposure amount exceeds the allowable amount of the image sensor 213 and the output of the image sensor 213 does not saturate. Set to a predetermined aperture value. Note that, here, the shutter speed is determined by the vertical synchronization signal, and the light emission from the projector unit 10 is entirely white light, and the light emission amount is also determined. Therefore, the predetermined aperture value is set so that the output of the image sensor 213 does not saturate. It can be easily set.
[0175]
FIG. 20 is a schematic diagram illustrating a captured image PG obtained by capturing writing on a whiteboard. As shown in FIG. 20, the entire surface of the frame PS3 on the whiteboard W is illuminated with white light, so that the written MR can be clearly photographed.
[0176]
The captured image PG as shown in FIG. 20 is transmitted to the image correction function 303 of the PC control section 300 under the control of the camera control section 200.
[0177]
Next, in the image correction function 303, when data of the projection original image to be projected and the image related to the captured image PG are combined to generate a composite image with no shift, as described above, the captured image PG Is performed. After that, the projection area is extracted and enlarged so that the area corresponding to the projection area in the captured image PG occupies the entire captured image. For example, as shown in FIG. 20, a substantially uniform white light is applied to the entire projection area on the whiteboard W, and the surrounding area is sufficiently darker than the projection area. Among them, the boundary (frame PS3) between the projection area and its peripheral part can be easily detected. Therefore, the image correction function 303 extracts an image in the frame PS3 and performs enlargement processing on the extracted image so as to be the entire captured image.
[0178]
The region extraction function 304 is directly written on the whiteboard W manually using a marker MK or the like from a captured image on which distortion correction has been performed by the image correction function 303 and the projection region has been extracted and enlarged. The area corresponding to the visual element (characters, graphics, colors, etc.) displayed in is extracted. For example, as shown in FIG. 20, when the projection area on the whiteboard W is entirely illuminated with white light, the writing MR and the surrounding area have completely different colors. Can be easily extracted. That is, the region extracting function 304 functions as a unit that extracts a region corresponding to a visual element directly attached on the whiteboard W from an image acquired by the camera unit 20.
[0179]
Here, the case where the region corresponding to the writing MR is extracted has been described as an example. However, when a visual material is given by directly pasting a paper material or a photograph on the whiteboard W, a similar method is used. It is possible to extract an area corresponding to a material, a photograph, or the like from a captured image.
[0180]
FIG. 21 is a schematic view illustrating a projection original image PP for projecting on the whiteboard W. Note that FIG. 21 shows the projection original image PP stored in the storage unit 32 of the personal computer 30 and is a projection original image corresponding to the projection image projected on the whiteboard W shown in FIG.
[0181]
Next, the image synthesizing function 305 generates image data (synthesized image) by synthesizing the area extracted by the area extracting function 304 and the projection original image PP as digital information by the projector unit 10 to generate image data (synthesized image). To memorize. FIG. 22 is a schematic diagram illustrating an image (synthesized image) CP obtained by synthesizing data of the projection original image PP and the writing MR. The composite image CP shown in FIG. 22 is an image reflecting the state on the whiteboard W shown in FIG. That is, the image combining function 305 functions as a unit that combines data of the region extracted by the region extracting function 304 and a projection original image to be projected on the whiteboard W by the projector unit 10.
[0182]
Therefore, here, the white board W is irradiated with light of a single color, and the white board W is photographed to acquire an image. Then, from the acquired image, an area corresponding to a visual element directly written, for example, attached or attached to the whiteboard W is extracted. Then, data is synthesized with the extracted region and a projection original image to be projected on the whiteboard W. As a result, it is possible to create a high-quality image in which a projection original image to be projected on the whiteboard W and data such as characters, lines, and attached materials directly written on the whiteboard W are combined. That is, it is possible to provide a projection system capable of acquiring a high-quality image combining information projected on the whiteboard W and information on visual elements directly added to the whiteboard W.
[0183]
As described above, in the projection system 1 according to the first embodiment, the white balance, the gradation characteristics, the distortion, and the like of the projection image and the captured image are corrected in consideration of the external light, the inclination of the whiteboard W, and the like. Further, a high-quality image combining information projected on the whiteboard W and information directly added to the whiteboard W is acquired. Therefore, a projection system capable of projecting and photographing an appropriate image according to the state of the whiteboard W (projection plane) and generating an appropriate image according to the state of the whiteboard W (projection plane) is provided. be able to.
[0184]
<(2) Second embodiment>
<(2-1) Distortion correction of captured image>
As described above, when obliquely photographing the whiteboard W, in order to obtain a photographed image as if photographed from the front of the whiteboard W, the distortion of the photographed image must be corrected. In addition, when obliquely projecting onto the whiteboard W, in order to combine data of a captured image obtained by capturing a writing or the like on the whiteboard W and a projection original image for projection, The distortion of the photographed image must be corrected so as to match the original projected image.
[0185]
In order to perform such distortion correction of the captured image, the projection system 1 according to the first embodiment projects the focus detection pattern AFP, captures the pattern, and corrects the distortion of the captured image based on the input image. However, in the projection system 1A according to the second embodiment, a pattern different from that of the first embodiment is projected on the whiteboard W in order to correct distortion of a captured image.
[0186]
Therefore, in the projection system 1A according to the second embodiment, the functions of the camera control unit 200 and the PC control unit 300 are different from those of the projection system 1 according to the first embodiment. Specifically, as shown in FIG. 23, the camera control unit 200A according to the second embodiment is provided with a corresponding point extraction function 207 in addition to the various functions according to the first embodiment. As shown in FIG. 2, in the PC control unit 300A according to the second embodiment, the functions of the correction value calculation function 302A and the image correction function 303A are the same as the correction value calculation function 302 and the image correction function according to the first embodiment. 303.
[0187]
Hereinafter, only portions different from the projection system 1 according to the first embodiment will be described with reference to FIGS. 2 and 23 as appropriate, and the other portions will be the same as those of the projection system 1 according to the first embodiment. Therefore, the same reference numerals are given and the description is omitted.
[0188]
Here, as in the case shown in FIG. 10 described above, the optical axes of the projection lens 10R and the photographing lens 20R substantially coincide with the Y axis, and the projection surface of the whiteboard W is inclined from the plane to XZ by an angle θ. An example will be described.
[0189]
When the user performs various operations on the keyboard or the like of the personal computer 30 while the projection program is being executed in the personal computer 30, the distortion correction operation of the captured image starts, and the camera unit 20 captures the whiteboard W and acquires the whiteboard W. Or correct the distortion of the captured image.
[0190]
When the distortion correction operation of the photographed image is started, a photographing signal is transmitted from the PC control unit 300 to the camera control unit 200A to perform photographing for distortion correction of the photographed image. Then, the camera control unit 200A transmits a command to the projector unit 10 and synchronizes with the vertical synchronizing signal from the next field, regardless of the input of the analog RGB signal from the personal computer 30, regardless of the digital image stored in the frame memory 112. The data is set to be a pattern (square pattern) for correcting distortion stored in the ROM 10b. Therefore, the projector unit 10 projects the grid pattern on the whiteboard W in synchronization with the next vertical synchronization signal.
[0191]
FIG. 24 is a schematic view illustrating a grid pattern GPS projected from the projector unit 10 onto the whiteboard W. The grid pattern GPS is composed of a frame GPF having a white outline portion and a grid pattern GP composed of white line segments extending vertically and horizontally. Note that, in the grid pattern GPS, points where white lines extending in the vertical and horizontal directions intersect (for example, points A, B, and C) are singular points (specific points and other points) for specifying a position on the grid pattern GPS. Points with identifiable geometric features). That is, the projector unit 10 functions as a unit that projects a predetermined grid pattern including a plurality of singular points on the whiteboard W.
[0192]
In response to this, the camera control unit 200 outputs a trigger signal for shooting to the image sensor driving circuit 214, and in synchronization with the next vertical synchronizing signal, the camera unit 20 sets the predetermined grid pattern projected on the whiteboard W An image is acquired by photographing a GPS. At this time, under the control of the camera control unit 200, the lens drive circuit 212 adjusts the aperture value of the photographing lens 20R so that the exposure amount exceeds the allowable amount of the image sensor 213 and the output of the image sensor 213 does not saturate. Set to a predetermined aperture value.
[0193]
Here, since the shutter speed is determined by the vertical synchronization signal and the grid pattern GPS is projected from the projector unit 10 and the light emission amount is also determined, the predetermined aperture value is easily set so that the output of the image sensor 213 is not saturated. Can be set.
[0194]
Next, when the grid pattern GPS is projected from the projector unit 10 onto the whiteboard W and the image is captured, the image is output from the A / D converter 216 to the corresponding point extraction function 207 of the camera control unit 200 shown in FIG. .
[0195]
FIG. 25 is a schematic diagram illustrating an image G2 obtained by photographing the grid pattern GPS projected on the whiteboard W with the camera unit 20. In FIG. 25, the frame GPF 'corresponds to the frame GPF of the grid pattern GPS projected from the projector unit 10, and the grid pattern GP' corresponds to the grid pattern GP. Here, since the optical axis of the photographing lens 20R and the whiteboard W are not in a vertical relationship, trapezoidal distortion occurs in the grid pattern GP 'as shown in FIG.
[0196]
The corresponding point extraction function 207 detects a change in luminance, and excludes an area outside the frame GPF 'from the image G2 shown in FIG. Next, by observing the luminance changes in the vertical and horizontal directions of the remaining grid pattern GP ′, it is possible to detect where the vertical and horizontal lines of the grid pattern GP ′ are located. be able to. Therefore, the corresponding point extraction function 207 can specify the position of the point where the line segment of the grid pattern GP 'intersects in G2 shown in FIG. As a result, for example, points A ′, B ′, and C ′ in the image G2 illustrated in FIG. 25 can be specified and extracted as points corresponding to the points A, B, and C in FIG. . That is, the corresponding point extraction function 207 functions as a unit that extracts corresponding points respectively corresponding to a plurality of singular points included in the grid pattern GPS projected from the projector unit 10 for the image G2.
[0197]
Next, in the correction value calculation function 302A of the PC control unit 300A shown in FIG. 2, a plurality of singular points (for example, points A, B, and C) in the grid pattern GPS shown in FIG. A correction value for correcting the distortion of the captured image is calculated from the positional relationship with the corresponding points (point A ′, point B ′, point C ′) in G2. That is, the correction value calculation function 302A functions as a unit that calculates a correction value from the positional relationship between a plurality of singular points and their corresponding points.
[0198]
Here, the principle for calculating the correction value in the correction value calculation function 302A will be briefly described with reference to FIGS. 26 and 27.
[0199]
FIG. 26 and FIG. 27 are diagrams for explaining the principle for calculating the correction value. In FIG. 26, when the xyz orthogonal coordinate system is set with the optical axis of the projector as the z-axis and the light source as the origin, the pixel on the light valve (for example, the display element 123) and the projection on the projection surface (whiteboard W) Schematically shows the positional relationship with the image to be performed. In FIG. 27, when an x′y′z ′ orthogonal coordinate system with the optical axis of the camera as the z ′ axis is set, the position on the projection plane (whiteboard W) and the pixel on the image sensor 213 are different. The positional relationship is schematically shown.
[0200]
As shown in FIG. 26, when a certain light ray (first light ray) emitted from the projector unit 10 has the relationship of the following equation (2), the first projection plane P1 which is a light valve is represented by z = 1. Assuming that the plane is a coordinate plane, the coordinates of the intersection of the first ray and the first projection plane P1 are the point M (p, q, 1).
[0201]
x / p = y / q = z (2)
[0202]
Next, in this state, an intersection M 'between the light beam group that is emitted on the assumption that the image is projected on the second projection plane P2 (for example, the whiteboard W) and the second projection plane P2 is obtained. Here, when the second projection plane P2 is expressed by the following equation (3), the coordinates of the intersection M ′ between the first light ray and the second projection plane P2 are expressed by the following equations (4) to (6). You.
[0203]
z = ax + by + c (3)
x = cp / (1-ap-bq) (4)
y = cq / (1-ap-bq) (5)
z = c / (1-ap-bq) (6)
[0204]
That is, here, by projecting the image of the first projection plane P1 onto the second projection plane P2, the point M indicating a part of the image is converted into the point M ′ in the three-dimensional space. .
[0205]
Then, as shown in FIG. 27, when a certain light ray (second light ray) irradiated on the image pickup device of the camera has the relationship of the following expression (7), a third projection surface serving as an image pickup surface of the image pickup device Assuming that P3 is a plane represented by z ′ = 1, the coordinates of the intersection of the second light ray and the third projection plane P3 are points N (s, t, 1).
[0206]
x '/ s = y' / t = z '(7)
[0207]
In this state, an intersection of a light beam group irradiated to form an image on the third projection plane P3 and a fourth projection plane P4 (for example, a whiteboard W) different from the third projection plane P3. Find N '. Here, if the fourth projection plane P4 is represented by the following equation (8), the coordinates of the intersection N ′ between the second light ray and the fourth projection plane P2 are represented by the following equations (9) to (11). You.
[0208]
z '= dx' + ey '+ f (8)
x '= fs / (1-ds-et) (9)
y '= ft / (1-ds-et) (10)
z '= f / (1-ds-et) (11)
[0209]
That is, here, by projecting the image of the fourth projection plane P4 onto the third projection plane P3, the point N ′ indicating a part of the image is converted into the point N in the three-dimensional space. .
[0210]
When the image projected by the projector is captured by a camera having a position and orientation different from that of the projector, the coordinates of a point forming the projected image in the three-dimensional space are represented by a point M in the projection process shown in FIG. From the point N 'to the point N in the photographing process shown in FIG. 27, the coordinate conversion from the point x' to the point M ', the coordinate conversion from the xyz rectangular coordinate system for the projector to the x'y'z' rectangular coordinate system for the camera. Will be subjected to the coordinate transformation. Here, a coefficient (coordinate conversion coefficient) indicating a coordinate conversion from the xyz rectangular coordinate system to the x'y'z 'rectangular coordinate system is represented by r.₁₁, R₁₂, R_Thirteen, R₂₁, R₂₂, R₂₃, R₃₁, R₃₂, R₃₃The description will be continued assuming that the following equations (12) to (14) hold.
[0211]
x '= r₁₁x + r₁₂y + r_Thirteenz ... (12)
y '= r₂₁x + r₂₂y + r₂₃z ... (13)
z '= r₃₁x + r₃₂y + r₃₃z (14)
[0212]
Here, by substituting the equations (4) to (6) and the equations (9) to (11) into the equations (12) to (14), the following equations (15) to (17) are obtained.
[0213]
(Equation 2)

[0214]
Equations (15) to (17) include a to f, r₁₁~ R₃₃Although the 15 unknown coefficients are included, the number of unknowns can be reduced to 9 by rearranging in the following procedure.
[0215]
For example, since the second projection plane and the fourth projection plane are the same, if the equation of one projection plane is determined, the equation of the other projection plane is calculated from the xyz orthogonal coordinate system to the x′y′z ′ orthogonal coordinate. It can be expressed using a coordinate conversion coefficient for the system. That is, one of the unknowns a, b, and c or the unknowns d, e, and f can be eliminated, and the number of unknowns is reduced by three. Further, if only the azimuth relationship is inclined between two coordinate systems (xyz rectangular coordinate system and x'y'z 'rectangular coordinate system) and no enlargement / reduction is involved, the following equation (18) is obtained. ) To (20), nine unknown coordinate transformation coefficients r₁₁~ R₃₃Can be reduced to six unknowns.
[0216]
r₁₁ ²+ R₁₂ ²+ R_Thirteen ²= 1 ... (18)
r₂₁ ²+ R₂₂ ²+ R₂₃ ²= 1 ... (19)
r₃₁ ²+ R₃₂ ²+ R₃₃ ²= 1 ... (20)
[0219]
As described above, since the number of unknowns in the three equations (15) to (17) can be nine, the point M (p, q, 1) on the first projection plane P1 and the third If there are three sets of combinations for associating points N (s, t, 1) on the projection plane P3, substituting the three sets of coordinates into equations (15) to (17) yields nine equations. , The remaining nine unknowns can be obtained. Therefore, it is possible to derive an equation (conversion equation) that can mutually convert an arbitrary point on the first projection plane P1 and a point on the third projection plane P3 corresponding to the point.
[0218]
Therefore, for example, in the correction value calculation function 302A, three singular points (for example, point A, point B, and point C) in the grid pattern GPS illustrated in FIG. 24 and corresponding points (point A) in the image G2 illustrated in FIG. , Point B ′, point C ′), a conversion equation can be obtained. Here, this conversion formula corresponds to a correction value for correcting the distortion of the captured image.
[0219]
Then, the image correction function 303A corrects the distortion of the captured image obtained by capturing the whiteboard W by the camera unit 20 based on the correction value calculated by the correction value calculation function 302A. As a result, characters and materials written in accordance with the image projected on the whiteboard W are captured by the camera unit 20 to obtain an image, and the image is subjected to coordinate transformation so as to match the projected image, and the image is transformed. Can be generated.
[0220]
By the way, when the projector section 10 corrects the distortion of the projection image, the image on the first projection plane P1 is obtained by deforming the original projection image based on the distortion correction value of the projection image. ing. However, since the positional relationship between the image on the first projection plane P1 and the original projection image is defined by the distortion correction value of the projection image, the image captured by the camera unit 20 is matched with the original projection image. It is easily possible to perform coordinate transformation. In such a case, the PC control unit 300 obtains a distortion correction value for the projection image from the projector control unit 100, and the coordinate conversion is realized by using the conversion formula as an expression taking into account the distortion correction value for the projection image. I do.
[0221]
Therefore, when distortion of the projection image is corrected in the projector unit 10, the image projected on the whiteboard W, when viewed from the front of the whiteboard W, is stored in the storage unit 32 of the personal computer 30. The original image is enlarged as it is. Therefore, when the image captured by the camera unit 20 is subjected to coordinate transformation so as to match the original projection image, writing on the whiteboard W is corrected like an image obtained by capturing from the front of the whiteboard W. be able to.
[0222]
Note that, as described above, in order to correct the distortion of the captured image from the positional relationship between the three singular points in the image to be projected and the corresponding points in the captured image corresponding to the three singular points. Can be obtained. Then, it is necessary that these three singular points are not arranged in a straight line. However, when error factors such as various aberrations of the projection lens 10R and the photographing lens 20R are taken into consideration, the distortion of the photographed image is determined due to the positional relationship between the singular points at the center and all four corners of all the images and the corresponding points. It is preferable to obtain a correction value for performing the correction.
[0223]
Further, as described above, since the distortion correction value for the captured image is performed every time the image capturing is performed, it is possible to cope with a case where the whiteboard W is moved in the middle. In this case, the timing for projecting the grid pattern GPS and the timing for capturing are synchronized, and the time for projecting the grid pattern GPS is very short, so that a person watching the projected image notices. And the projected image is not difficult to see.
[0224]
As described above, in the projection system 1A according to the second embodiment, a predetermined grid pattern GPS including a plurality of singular points (for example, points A, B, and C) is projected on the whiteboard W. Then, corresponding points respectively corresponding to a plurality of singular points are extracted from an image G2 obtained by photographing the projected grid pattern GPS. Further, a correction value is calculated from a plurality of singular points in the grid pattern GPS and a positional relationship between the corresponding points. Based on the correction value, distortion of a captured image obtained by capturing the whiteboard W is corrected. As a result, it is possible to correct the distortion of the captured image caused by the relative positional relationship between the whiteboard W (projection plane) and the camera unit 20 and the like.
[0225]
Further, here, the captured image obtained by capturing the whiteboard W can be corrected so as to match the original image to be projected. Therefore, as described above, an area corresponding to a visual element directly written on or attached to the whiteboard W is extracted from the captured image and projected onto the whiteboard W, as described above. When synthesizing data with the projection original image, it is possible to generate a synthesized image without deviation.
[0226]
<(3) Third Embodiment>
In the third embodiment, an electronic conferencing system in which personal computers in a plurality of conference rooms are connected via a network, and the same image is enlarged and projected in each conference room and a conference is performed in a cooperative manner between remote locations is shown. .
[0227]
<(3-1) Internal configuration of the electronic conference system>
FIG. 28 is a block diagram showing an internal configuration of the electronic conference system. As shown in FIG. 28, in the electronic conference system, a projection system substantially similar to the projection system 1 according to the first embodiment is configured to project an image on the whiteboards W1 and W2 in the two conference rooms CR1 and CR2, respectively. 1B is installed. Since the internal configurations of the projection systems 1B installed in both the conference rooms CR1 and CR2 are completely the same, the same parts are denoted by the same reference numerals, and the projection system 1B has different functions from the projection system 1. Are added a little, and different points from the projection system 1 will be mainly described below. The operation of the projection system 1B is controlled by executing a program stored in the personal computer 30 in the PC control unit 300.
[0228]
The personal computer 30 of the projection system 1B is provided with a speaker SP and a microphone MC, and can input audio in both conference rooms CR1 and CR2 to acquire audio data. In the projection system 1B, the personal computer 30 is connected to the network 400, and the projection systems 1B installed in the conference rooms CR1 and CR2 can mutually transmit and receive data.
[0229]
Then, by transmitting and receiving the image data (projected original image) stored in the storage unit 32 of the personal computer 30, a similar projected image can be projected on the whiteboard W in both the conference rooms CR1 and CR2. For example, by transmitting image data (projected original image) stored in the personal computer 30 of the conference room CR1 to the personal computer 30 of the conference room CR2, the two conference rooms are stored based on the image data stored in the personal computer of the conference room CR1. Similar projection images can be projected in CR1 and CR2.
[0230]
The projection systems 1B installed in both conference rooms CR1 and CR2 can transmit and receive audio data obtained through the microphone MC to and from the personal computer 30 because the personal computer 30 can transmit and receive data to and from each other via the network 400. Thus, sound can be output from the speaker SP.
[0231]
By the way, in the projection system 1 described above, in the recording operation of the meeting contents, the image synthesizing function 305 performs data synthesis of the area extracted by the area extracting function 304 and the original projection image to be projected by the projector unit 10. The composite image data was generated and stored in the storage unit 32. On the other hand, in the projection system 1B, as in the recording operation of the meeting contents in the projection system 1, the whiteboards W1 and W2 are photographed by the camera unit 20 and extracted by the region extraction function 304 based on the images. The image data for the area is transmitted to the other projection system 1B via the network 400. Then, the other projection system 1 </ b> B combines the data with the original projection image to generate composite image data, and projects the composite image data on the whiteboard W. The other operations of the combining process of the projection image and the captured image in the projection system 1B are the same as those in the projection system 1.
[0232]
Here, the “network” is a communication line network for performing data communication, and specifically, various types of electric communication lines (including optical communication lines) such as the Internet, a LAN, a WAN, and a CATV. It is a communication network. The connection form to the network may be a permanent connection using a dedicated line or a temporary connection such as a dial-up connection using a telephone line such as an analog line or a digital line (ISDN). Good. The transmission method may be any of a wireless method and a wired method.
[0233]
<(3-2) Example of using electronic conference system>
FIG. 29 is a schematic diagram showing a usage example of the electronic conference system. As shown in FIG. 29, a projection system 1B is installed in each of the plurality of conference rooms CR1 and CR2, and a personal computer 30 provided in the projection system 1 transmits and receives data via a network or the like. The same image displayed on the personal computer 30 is projected onto the whiteboards W1 and W2 while reflecting the contents written directly on the whiteboards W1 and W2. Also, the personal computer 30 obtains audio data with a microphone MC (not shown) and transmits and receives the audio data via the network 400, so that the audio in one conference room is provided in the personal computer 30 in the other conference room. Output from the speaker SP.
[0234]
Then, for example, as shown in FIG. 29, the presenter PM1 of the conference room CR1 attaches an underline MR1 to important points in the image projected on the whiteboard W with a red marker, and the participants of the conference Has been described. On the other hand, the explainer PM2 of the conference room CR2 attaches a circle MR2 to an important point in the image projected on the whiteboard W and writes a character MR3 of “exceeding 400,000 pixels”, and Has been described.
[0235]
At this time, in each of the conference rooms CR1 and CR2, the camera unit 20 captures images and the like on the whiteboard W at regular intervals in order to reflect the contents and the like written in the other conference room. , The image data of the area extracted by the area extracting function 304 is transmitted to the other projection system 1B via the network 400. Then, the other projection system 1 </ b> B combines the data with the original projection image to generate composite image data, and projects the composite image data on the whiteboard W.
[0236]
As a result, as shown in FIG. 29, the contents of materials and the like written on or attached to the whiteboard W in one conference room can be reflected on the whiteboard W in the other conference room.
[0237]
When writing is performed with a marker or the like in accordance with the projected image projected on the whiteboard in one conference room, the image is directly taken by the camera unit 20 and transmitted to the other conference room via the network 400. Then, even if the image is projected on the whiteboard W in the other conference room based on the image, the contents of the materials and the like written or affixed to the whiteboard W in the one conference room are copied to the other conference room. On the whiteboard W. However, if the image is taken by the camera unit 20 as it is, the projected image and the writing of the marker overlap and the image is taken, so that the image becomes difficult to read, or when the image projected on the whiteboard W is taken by the camera, the projected image In the other conference room, the content intended by the creator of the image cannot be accurately transmitted to the people who participated in the conference.
[0238]
In order to solve such a problem, the projection system 1B according to the present embodiment irradiates the whiteboard W with light of a single color (entirely white) to capture an image by photographing the whiteboard W. Then, a projection source for extracting a region corresponding to a visual element directly written, attached, or the like directly onto the whiteboard W from the acquired image and projecting the region on the whiteboard W is extracted. Synthesize data with the image. As a result, it is possible to create a high-quality image in which a projection image to be projected on the whiteboard W and visual elements such as characters, lines, and attached materials directly written on the whiteboard W are synthesized. it can.
[0239]
As described above, the projector unit 10 irradiates light of a single color such as white on the entire surface at the time of shooting by the camera unit 20. Therefore, if the time of projecting the light of a single color such as white on the whiteboard W is long, the participants of the conference may feel uncomfortable or the image may be difficult to see. However, in the projection system 1B according to the present embodiment, the timing of irradiating the light of a single color and the timing of capturing are synchronized, and the time of projecting the light of a single color is as short as one frame or less. Since the time is short, the participants of the conference do not feel uncomfortable, and the image is not difficult to see.
[0240]
In addition, the user can perform various operations such as writing on the whiteboard W by variously operating the keyboard and the like of the personal computer 30. However, in an electronic conference, a reduction in time lag is required. In some cases, writing on the W or the like is photographed one after another, and image data is transmitted via the network 400. In such a case, it is preferable that the user sets a short photographing interval on the personal computer 30 and repeats the photographing operation so that the data synthesis of the photographed image and the original projection image related to writing or the like is performed in a short cycle.
[0241]
By the way, here, whether or not to correct the distortion of the photographed image when photographing the writing on the whiteboard W depends on whether or not the distortion of the projection image projected from the projector unit 10 is being performed.
[0242]
For example, when the distortion correction of the projection image is not performed, it is considered that the presenter of the conference wrote the whiteboard W or the like in accordance with the projection image without particularly considering the trapezoidal distortion or the like. Therefore, in this case, no distortion correction is performed on the captured image. Whether the distortion correction of the projection image is performed or not is determined by acquiring information by data exchange between the projector control unit 100 and the camera control unit 200 and performing the distortion on the captured image by the PC control unit. 300 and the like. On the other hand, when the distortion correction of the projection image is performed, the distortion correction of the captured image described above is always performed, and the image data is transmitted to the projection system of the other conference room.
[0243]
As described above, in the electronic conference system according to the third embodiment, one projection system 1B irradiates whiteboard W with light of a single color (entirely white) to capture an image by photographing whiteboard W. get. Then, from the acquired image, an area corresponding to the visual element directly written or attached to the whiteboard W is extracted and transmitted to the other projection system. Then, in the other projection system, the extracted area is combined with a projection original image to be projected on the whiteboard W, and projected. As a result, communication can be sufficiently achieved between conference rooms located in different places such as remote places. Further, it is possible to provide a projection system capable of projecting a high-quality image in which information projected on the whiteboard W and information of a visual element directly added to the whiteboard W are combined.
[0244]
<(4) Modification>
The embodiments of the present invention have been described above, but the present invention is not limited to the above-described contents.
[0245]
For example, in the above-described embodiment, the scaler 113 and the like are dedicated electronic circuits. However, the functions may be realized by executing a program in the CPU in the projector control unit 100.
[0246]
Also, in the above-described embodiment, the projector control unit 100, the camera control unit 200, and the PC control unit 300 are separately provided, but some or all of these functions are provided in one place. It may be realized by executing a program in the CPU.
[0247]
◎ Further, when the projection program is being executed, the user operates the keyboard of the personal computer 30 in various ways, so that the WB correction of the projected image, the WB correction of the captured image, the correction of the gradation characteristics of the projected image, the Of all of the gradation characteristic correction, distortion correction of the projected image, and distortion correction of the photographed image are not performed, any one of the operations is performed, or various combinations of operations are performed. , All the operations may be executed.
[0248]
Also, in the above-described second embodiment, the distortion correction of the captured image in the case where the second and fourth projection planes are planes has been described as an example. However, even when the second and fourth projection planes are curved planes, By defining the equation of the curved surface, a coordinate conversion coefficient between the first projection surface and the third projection surface can be obtained by the same method as in the second embodiment. Therefore, even when the projection surface onto which the projection image is projected from the projector unit 10 is a curved surface, distortion correction of the captured image can be performed.
[0249]
Also, in the above-described embodiment, in order to obtain a WB correction value, achromatic light is projected on the entire projection area, but achromatic light is projected on a part of the projection area, and an image related to the part is projected. The WB correction value may be obtained based on the data.
[0250]
In the above-described embodiment, data transmission and reception between the projector control unit 100, the camera control units 200 and 200A, and the PC control units 300 and 300A are performed between the USB I / F 34 and the USB I / F 230, and between the USB I / F 230 Although the USB I / Fs 130 are realized by connecting each with a cable, the connection combinations and methods are not limited to these, and the connections may be made in various combinations and methods.
[0251]
In the above-described embodiment, the user may manually set the distortion correction of the projection image in an operation other than the operation characterized in that the distortion correction of the projection image is performed automatically.
[0252]
In addition, the projection systems 1, 1A, and 1B exemplified in the above-described embodiment perform distortion correction of a captured image as described above, and therefore, a distortion correction device that performs distortion correction of a captured image of a subject such as a whiteboard W. Can also be seen as
[0253]
The specific embodiments described above include inventions having the following configurations.
[0254]
(1) Projection means for projecting a pattern including a plurality of focus detection portions onto a subject, and photographing the pattern projected on the projection surface to form an image related to the pattern including a plurality of pixels. The photographing means to be input, the focus control means for controlling the focusing of the photographing lens in the photographing means, and the in-focus state of the photographing lens with respect to regions corresponding to the plurality of focus detection portions in the image. An evaluation value calculation unit configured to calculate an evaluation value; and a distortion correction unit configured to correct a distortion of a captured image obtained by capturing the subject by the imaging unit based on the evaluation value. Distortion correction device.
[0255]
According to the invention of (1), a pattern including a plurality of regions for focus detection is projected, and the imaging lens is used for each of regions corresponding to a plurality of portions for focus detection in an image input by capturing the pattern. An evaluation value relating to the in-focus state is obtained, and distortion of a captured image obtained by imaging the subject is corrected based on the evaluation value. It is possible to correct distortion of a captured image caused by a positional relationship or the like.
[0256]
(2) projecting means for projecting a predetermined pattern including a plurality of singular points on a subject; photographing means for acquiring an image by photographing the predetermined pattern projected on the projection surface; Extracting means for extracting corresponding points respectively corresponding to the plurality of singular points, calculating means for calculating a correction value from a positional relationship between the plurality of singular points and the corresponding point, and, based on the correction value, A distortion correction unit that corrects a distortion of a captured image obtained by capturing the subject with a capturing unit.
[0257]
According to the invention of (2), a predetermined pattern including a plurality of singular points is projected on a subject, and a corresponding image corresponding to the plurality of singular points is obtained from an image obtained by photographing the projected predetermined pattern. A point is extracted, a correction value is calculated from a positional relationship between a plurality of singular points in a predetermined pattern and a corresponding point, and distortion of a captured image obtained by capturing an object is corrected based on the correction value. With such a configuration, it is possible to correct the distortion of the captured image caused by the relative positional relationship between the subject and the imaging unit.
[0258]
(3) A program which, when executed by a computer included in the projection system, causes the projection system to function as the projection system according to any one of claims 1 to 5, (1) and (2).
[0259]
According to the invention of (3), the same effect as the invention described in any one of claims 1 to 5, (1) and (2) can be obtained.
[0260]
【The invention's effect】
As described above, according to the first to fifth aspects of the present invention, it is possible to provide a projection system capable of performing appropriate image processing according to the state of the projection plane.
[0261]
Further, according to the first to third aspects of the present invention, it is possible to provide a projection system capable of projecting an appropriate image according to the situation of the projection plane.
[0262]
In particular, according to the first aspect of the present invention, a projected image projected onto a projection surface based on an image obtained by projecting a pattern whose luminance changes stepwise on a projection surface and photographing the pattern, By adopting a configuration that corrects the gradation characteristics of the photographed image obtained by photographing the projection surface, etc., the gradation characteristics of the projection image and the photographed image change due to the influence of external light, the reflectance of the projection surface, and the like. Can be corrected.
[0263]
According to the second aspect of the present invention, a projection image projected on the projection surface, a projection surface, or the like is photographed based on an image obtained by photographing the projection surface on which achromatic light is projected. By adopting a configuration that corrects the white balance of the obtained captured image, the color tone of the projected image or the captured image changes due to the influence of the color of the projection surface, the color of the external light, the color of the light source of the projection unit, and the like. Can be corrected.
[0264]
According to the third aspect of the present invention, a pattern including a plurality of focus detection portions is projected, and the pattern is photographed, and an image corresponding to the plurality of focus detection portions in an input image is captured. An evaluation value relating to the in-focus state of the lens is obtained, and based on the evaluation value, distortion of a projection image projected on the projection surface or a captured image obtained by photographing the projection surface is corrected. This makes it possible to correct distortion of a projected image or a captured image caused by a relative positional relationship between the projection unit or the imaging unit and the projection surface.
[0265]
According to the invention of claim 4, a predetermined pattern including a plurality of singular points is projected on a projection surface, and an image obtained by photographing the projected predetermined pattern is converted into a plurality of singular points. A corresponding corresponding point is extracted, a plurality of singular points in a predetermined pattern, and a correction value is calculated from a positional relationship between the corresponding point, and based on the correction value, a photographed image obtained by photographing the projection plane is obtained. By adopting a configuration that corrects the distortion, it is possible to correct the distortion of the captured image caused by the relative positional relationship between the projection plane and the imaging unit.
[0266]
According to the fifth aspect of the present invention, an image obtained by projecting a single color light on the projection surface and photographing the projection surface is directly written on the projection surface or directly attached thereto. By extracting a region corresponding to the visual element attached to the image and combining the data with an image to be projected on the projection surface, the image projected on the projection surface and the image directly written on the projection surface are written. It is possible to create a high-quality image by combining characters, lines, and visual elements such as attached materials. That is, it is possible to provide a projection system capable of acquiring a high-quality image combining information projected on the projection surface and information directly added to the projection surface.
[Brief description of the drawings]
FIG. 1 is a schematic diagram illustrating an example of use of a projection system according to a first embodiment.
FIG. 2 is a block diagram illustrating an internal configuration of the projection system according to the first embodiment.
FIG. 3 is a block diagram for explaining functions of a camera control unit according to the first embodiment.
FIG. 4 is a diagram for explaining a method of setting a white balance correction value.
FIG. 5 is a schematic view exemplifying an image acquired at the time of photographing for gradation characteristic correction.
FIG. 6 is a schematic diagram illustrating an ideal brightness ratio according to the gray scale chart.
FIG. 7 is a schematic diagram illustrating a brightness ratio when there is an influence of external light;
FIG. 8 is a diagram for explaining a method of calculating a correction value for gradation characteristic correction.
FIG. 9 is a diagram exemplifying gradation conversion characteristics when there is an influence of external light;
FIG. 10 is a schematic view exemplifying a case of projecting and photographing from an oblique direction with respect to a whiteboard.
FIG. 11 is a schematic view illustrating an image obliquely projected on a whiteboard.
FIG. 12 is a schematic diagram showing a projected image after distortion correction.
FIG. 13 is a schematic view illustrating a photographed image acquired at the time of oblique photographing.
FIG. 14 is a schematic diagram illustrating a captured image after distortion correction.
FIG. 15 is a diagram illustrating a focus detection pattern projected on a whiteboard.
FIG. 16 is a schematic view illustrating a captured image obtained by capturing a focus detection pattern projected on a whiteboard.
FIG. 17 is a schematic diagram illustrating a curve representing a relationship between an evaluation value and a position of a photographing lens.
FIG. 18 is a diagram illustrating a method of calculating a correction value for correcting a distortion of a captured image.
FIG. 19 is a schematic view illustrating a state on a whiteboard.
FIG. 20 is a schematic diagram illustrating a captured image of a whiteboard written;
FIG. 21 is a schematic diagram illustrating a projection original image for projecting on a whiteboard.
FIG. 22 is a schematic view illustrating a combined image obtained by combining data of a projection original image and writing.
FIG. 23 is a block diagram for explaining functions of a camera control unit according to the second embodiment.
FIG. 24 is a schematic view illustrating a grid pattern projected on a whiteboard.
FIG. 25 is a schematic view illustrating a screen obtained by photographing a grid pattern projected on a whiteboard;
FIG. 26 is a diagram for explaining a principle for calculating a distortion correction of a captured image.
FIG. 27 is a diagram for explaining a principle for calculating a distortion correction of a captured image.
FIG. 28 is a block diagram showing an internal configuration of the electronic conference system.
FIG. 29 is a schematic diagram showing a usage example of the electronic conference system.
FIG. 30 is a schematic diagram showing an example of use of a conventional projection system in a conference.
[Explanation of symbols]
1,1A, 1B projection system
2 Projector with camera
10 Projector unit (projection means)
20 Camera unit (photographing means)
30 PC
100 Projector control unit
101 element drive control function (white balance correction means, gradation correction means)
113 scaler (distortion correction means)
200, 200A Camera control unit
204 Evaluation value calculation function (evaluation value calculation means)
205 Focus control function (focus control means)
207 Corresponding point extraction function (extraction means)
210 photography function unit (photography means)
217 WB circuit (white balance correction means)
218 γ correction circuit (gradation correction means)
300 PC control unit
302A Correction value calculation function (calculation means)
303A Image correction function (distortion correction means)
304 area extraction function (area extraction means)
305 Image composition function (composition means)

Claims (5)

A projection system,
Projection means for projecting a pattern whose luminance changes stepwise on a projection surface,
A photographing unit that acquires an image by photographing the pattern projected on the projection surface,
Based on the image acquired by the photographing unit, a projection image projected on the projection surface by the projection unit, and at least one of a photographed image obtained by photographing the projection surface by the photographing unit Gradation correction means for correcting gradation characteristics of an image;
A projection system comprising: A projection system,
Projecting means for projecting achromatic light on the projection surface;
A photographing unit that acquires an image by photographing the projection surface on which the achromatic light is projected,
Based on the image acquired by the photographing unit, a projection image projected on the projection surface by the projection unit, and at least one of a photographed image obtained by photographing the projection surface by the photographing unit White balance correction means for correcting the white balance of the image;
A projection system comprising: A projection system,
Projection means for projecting a pattern including a plurality of focus detection portions on a projection surface,
By photographing the pattern projected on the projection surface, photographing means for inputting an image of the pattern composed of a plurality of pixels,
Focusing control means for performing focusing control of a photographing lens in the photographing means;
An evaluation value calculation unit configured to calculate an evaluation value regarding an in-focus state of the photographing lens for an area corresponding to the plurality of focus detection portions in the image,
Based on the evaluation value, distortion is corrected for at least one of a projection image projected on the projection surface by the projection unit and a photographed image obtained by photographing the projection surface by the photographing unit. And a distortion correction unit. A projection system,
Projection means for projecting a predetermined pattern including a plurality of singularities on a projection surface,
By photographing the predetermined pattern projected on the projection surface, a photographing means for acquiring an image,
For the image, extraction means for extracting corresponding points respectively corresponding to the plurality of singularities,
Calculation means for calculating a correction value from the positional relationship between the plurality of singular points and the corresponding point,
Distortion correction means for correcting distortion of a captured image obtained by imaging the projection surface by the imaging means based on the correction value;
A projection system comprising: A projection system,
Projecting means for projecting a single color light onto a projection surface;
A photographing unit that acquires an image by photographing the projection surface on which the single color light is projected,
For the image obtained by the photographing unit, an area extracting unit that extracts an area corresponding to a visual element directly attached to the projection plane,
Combining means for combining the area extracted by the area extracting means and an image to be projected onto the projection surface by the projecting means;
A projection system comprising:

Images