JP3702243B2 - Stereoscopic image processing method and apparatus - Google Patents

Description

変換を図３４で説明する。まず、Ｘ≧０かつＺ≧０の範囲を考える。点（Ｘ０，Ｙ０，Ｚ０）が縮小処理により点（Ｘ０，Ｙ０，Ｚ１）に移動したとき、縮小率Ｓｚは、

である。Ｃの座標は（Ｘ０，Ｙ０，０）でＤの座標は（Ｘ０，Ｙ０，Ｂ）である。
【０１０１】
Ｅは直線と平面の交点であり、座標を（Ｘ０，Ｙ０，Ｚ２）とすると、Ｚ２は下のように求めることができる。
【０１０２】
Ｚ＝Ｂ−Ｘ×Ｂ／Ａ （平面）
Ｘ＝Ｘ０，Ｙ＝Ｙ０ （直線）
Ｚ２＝Ｂ−Ｘ０×Ｂ／Ａ
したがって、

Ｘに対して一般に、
Ｓｚ＝１−Ｘ／Ａ
となる。ＸとＺの他の範囲についても同様の計算を行うと、以下の結果が得られ、変換が検証できる。
【０１０３】
Ｘ≧０のとき、Ｓｚ＝１−Ｘ／Ａ
Ｘ＜０のとき、Ｓｚ＝１＋Ｘ／Ａ
図３５は歪変換の別の例を示す。より厳密にはカメラから放射状に撮影が行われることを考慮し、Ｘ軸、Ｙ軸方向の縮小処理も組み合わせている。ここでは、ふたつのカメラの中心をカメラ位置の代表として変換を行う。変換式は以下の通りである。
【０１０４】
【数２】

図３６はこの変換を検証する。ここでも、Ｘ≧０かつＺ≧０の範囲を考える。点（Ｘ０，Ｙ０，Ｚ０）が縮小処理により点（Ｘ１，Ｙ１，Ｚ１）に移動したとき、縮小率Ｓｘ、Ｓｙ、Ｓｚは、

となる。Ｅは平面と直線の交点なので、前述と同様Ｓｘ、Ｓｙ、Ｓｚを求めることができる。
【０１０５】
なお、以上のように変換後の空間を平面の集合で表すと、面同士の接線を境に処理が変化し、場合により違和感が生じうる。その場合は曲面で接続するか、曲面だけで空間を構成してもよい。計算は曲面と直線の交点Ｅを求めるものに変わるだけである。
【０１０６】
また、以上の例では、縮小率は同一直線ＣＤ上では同じとなるが、重み付けを行ってもよい。例えばＳｘ、Ｓｙ、Ｓｚに、カメラからの距離Ｌに対する重み付け関数Ｇ（Ｌ）をかければよい。
【０１０７】
図３７から図４０までは、第３の立体画像処理装置１００の歪処理部１７４による処理とその原理を示す。
図３７は第３の立体画像処理装置１００に入力されたデプス情報付画像のデプスマップを示し、ここではデプスの範囲がＫ１〜Ｋ２の値をもつとする。ここでは近置のデプスを正、遠置のデプスを負で表す。
【０１０８】
図３８は元のデプス範囲２４０と変換後のデプス範囲２４２の関係を示す。デプスは画像周辺部に行くにしたがって一定値に近づく。歪処理部１７４はこの補正にしたがうようデプスマップを変換する。垂直方向に視差をもたせる場合も同様である。この変換もＺ方向の縮小のみなので、以下の式で表現できる。
【０１０９】
【数３】

なお、ＳｚはＸの値により場合分けされ、
Ｘ≧０のとき、Ｓｚ＝１−２Ｘ／Ｌ
Ｘ＜０のとき、Ｓｚ＝１＋２Ｘ／Ｌ
となる。以上の変換により、図３９に示す新たな要素をもつ新たなデプスマップが生成される。
【０１１０】
図４０はデプスマップに対する別の歪変換の原理を示す。空間は、より厳密にはユーザ１０から放射状に観察されるので、Ｘ軸、Ｙ軸方向の縮小処理も組み合わせている。ここでは、眼間中心を観察位置としている。具体的な処理は図３６の場合と同じ式になる。なお、もともとのデプスマップはＺ値しか持たないが、この計算を行う場合はＸ値とＹ値も保持することになる。Ｚ値はＸ方向あるいはＹ方向の画素シフト量に変換されるが、Ｘ値とＹ値はそれらに対するオフセット値として保持すればよい。
【０１１１】
いずれにしても、歪処理部１７４で変換されたデプスマップともとの画像は二次元画像生成部１７８へ入力され、ここで適正視差になるよう水平方向にシフトした合成処理が行われる。その詳細は後述する。
【０１１２】
図４１〜図５１は、第２の立体画像処理装置１００の位置シフト部１６０、およびその延長と把握できる第３の立体画像処理装置１００の二次元画像生成部１７８の処理を示す。
図４１は位置シフト部１６０によるふたつの視差画像の合成位置のシフト原理を示す。同図のごとく、初期状態では右眼画像Ｒと左眼画像Ｌの位置は一致している。しかし、同図上部のごとく左眼画像Ｌを相対的に右へシフトすると、近置点の視差は増加し、遠置点の視差は減少する。逆に、同図下部のごとく左眼画像Ｌを相対的に左へシフトすると、近置点の視差は減少し、遠置点の視差は増加する。
【０１１３】
以上が視差画像のシフトによる視差調整の本質である。画像のシフトは一方であってもよいし、両方を互いに逆方向にシフトしてもよい。またこの原理から、立体表示方式は、メガネ方式やメガネなし方式を問わず、視差を利用している全ての方式に適用できることがわかる。多視点映像や垂直方向の視差に対しても同様の処理が可能である。
【０１１４】
図４２はシフト処理を画素レベルで示す。左眼画像２００と右眼画像２０２には、ともに第１四角形２５０と第２四角形２５２が写っている。第１四角形２５０には近置視差がついており、その視差量を正数であらわすと、「６画素」となる。これに対し第２四角形２５２は遠置視差がついており、その視差量を負数であらわすと、「−６画素」となる。ここで、この視差量をそれぞれＦ２、Ｆ１とする。
【０１１５】
一方、ユーザが保有する表示装置の適正視差がＪ１〜Ｊ２であることが分かったとする。位置シフト部１６０は両画像の合成開始位置を互いに（Ｊ２−Ｆ２）画素シフトさせる。図４３はそのシフトの終了後の状態で、いま、Ｆ１＝−６、Ｆ２＝６であり、また、Ｊ１＝−５、Ｊ２＝４であったとすると、合成開始位置は互いに−２画素、つまり遠置方向に全体がシフトする方向にずらされることになる。最終的な視差量は図４３のごとく、Ｅ１＝−８、Ｅ２＝４となり、少なくとも近置方向に関して限界視差内に収まる。一般に遠置方向と比較して、近置方向の二重像の方が違和感が強いとされ、かつ被写体は近置方向に配置された状態で撮影されることが多いので、基本的に近置方向の視差を限界内に収めることが望ましい。以下に処理例を示す。
【０１１６】
１．近置点が限界視差外で、遠置点が限界視差内の場合は、近置点を限界視差点にシフトする。ただし、遠置点の視差が眼間距離に到達すれば処理をやめる。 ２．近置点が限界視差外で、遠置点が限界視差外の場合は、近置点を限界視差点にシフトする。ただし、遠置点の視差が眼間距離に到達すれば処理をやめる。 ３．近置点も遠置点も限界視差内の場合は処理しない。
４．近置点が限界視差内で、遠置点が限界視差外の場合は、遠置点を限界視差点にシフトするが、処理の途中で近置点が限界視差点に到達すれば処理をやめる。
【０１１７】
図４４は合成位置のシフトによる画像端の欠落を示す。ここでは左眼画像２００と右眼画像２０２のシフト量が１画素であり、左眼画像２００の右端と右眼画像２０２の左端にそれぞれ１画素幅の欠落部分２６０が生じる。画像端調整部１６８はこのとき、図４４のように画像端の画素列を複製して水平画素数を補償する。
【０１１８】
これ以外の方法として、欠落部分２６０は黒または白など特定の色で表示してもよいし、非表示にしてもよい。さらに初期画像のサイズと同じになるように切り出しや付け加え処理を施してもよい。また、予め初期画像のサイズを実際の表示サイズより大きくしておき、欠落部分２６０が表示に影響しないよう配慮してもよい。
【０１１９】
図４５は第２の立体画像処理装置１００による視差の手動調整の流れである。同図のごとく、まず視差画像として左右画像が人手で作成され（Ｓ１０）、これがネットワークその他のルートで配布される（Ｓ１２）。これを第２の立体画像処理装置１００が受け取り（Ｓ１４）、この図の例では、まずはそのままシフトなしの通常の状態で画像を合成して表示する（Ｓ１６）。すなわち、ここでは適正視差がまだ取得されていない場合や位置シフト部１６０を動作させていない場合を考えている。つづいて、立体表示された視差画像に対してユーザが立体感調整部１１２を介して視差の調整を指示し、これを位置シフト部１６０が「手動調整モード」で受け、画像合成位置を調整して表示する（Ｓ１８）。なお、Ｓ１０とＳ１２は画像クリエータの手続２７０、Ｓ１４以降は第２の立体画像処理装置１００の手続２７２である。また、図示しないが、このシフト量をヘッダに記録し、次回から参照して合成すれば、再調整の手間が省ける。
【０１２０】
図４６は第２の立体画像処理装置１００による自動調整の流れを示す。画像クリエータの手続２７０である、左右画像の生成（Ｓ３０）、画像配布（Ｓ３２）は図４５と同じである。また、第２の立体画像処理装置１００の手続２７２のうち、画像受取（Ｓ３４）も同様である。つぎに、視差量検出部１５０のマッチング部１５８によって視差画像間に予め付けられている視差、とくに最大視差を検出し（Ｓ３６）、一方、視差情報保持部１２０から適正視差、とくに限界視差を取得する（Ｓ３８）。この後、位置シフト部１６０が前述の処理によって限界視差を満たすよう画像の合成位置をシフトし（Ｓ４０）、視差書込部１６４、画像端調整部１６８、フォーマット変換部１１６による処理を経て立体表示される（Ｓ４２）。
【０１２１】
図４７は、第２の立体画像処理装置１００によるさらに別の自動調整の流れを示す。画像クリエータの手続２７０で左右画像の生成（Ｓ５０）した後、この時点で最大視差を検出して（Ｓ５２）視差画像のいずれかの視点画像のヘッダへ記録しておく（Ｓ５４）。この検出は対応点マッチングで実施してもよいが、クリエータが手作業で視差画像を生成したときは、その編集過程で当然に既知であるため、これを記録すればよい。この後、画像を配布する（Ｓ５６）。
【０１２２】
一方、第２の立体画像処理装置１００の手続２７２のうち、画像受取（Ｓ５８）は図４６と同様である。つぎに、視差量検出部１５０のヘッダ検査部１５６によって前述の最大視差をヘッダから読み出す（Ｓ６０）。一方、視差情報保持部１２０から限界視差を取得し（Ｓ６２）、以下の処理Ｓ６４、Ｓ６６は図４６の処理Ｓ４０、Ｓ４２とそれぞれ同じである。この方法によれば、最大視差を計算する必要がない。また、画像全体に適切な立体感を実現できる。さらに、シフト量はヘッダに記録できるため、原画像自体を損なうおそれがない。なお、図示しないが、図４６でも検出された最大視差をヘッダに記録すれば、つぎからは図４７の手続にしたがって処理することができる。
【０１２３】
なお、多眼式でも同様の処理が可能で、それぞれ隣り合う視点画像間の視差量に対して同様の処理を行えばよい。ただし、実際にはそれら複数の視点画像間の視差のうちの最大の視差によって全視点画像間の「最大視差」とみなし、合成位置のシフト量を決めてもよい。
【０１２４】
ヘッダ情報は多視点画像の少なくともひとつにあればよいとしたが、多視点画像が１枚の画像に合成されている場合はその画像のヘッダを利用すればよい。
【０１２５】
さらに、すでに合成済みの画像が配布される場合もあるが、その場合はいちど逆変換処理で画像を分離し、合成位置シフト量を計算して再合成するか、それと結果が同じになるよう画素の並べ替え処理を行えばよい。
【０１２６】
図４８〜図５１は、合成位置のシフトをデプス情報付画像について行う処理を示す。これは第３の立体画像処理装置１００の二次元画像生成部１７８にて行われる。
図４８、図４９はそれぞれデプス情報付画像を構成する平面画像２０４とデプスマップである。ここでは近置デプスを正、遠置デプスを負で表している。オブジェクトとして第１四角形２５０、第２四角形２５２、第３四角形２５４が存在し、第１四角形２５０はデプス「４」、第２四角形２５２は「２」、第３四角形２５４は「−４」である。第１四角形２５０は最近置点、第２四角形２５２は中間近置点、第３四角形２５４が最遠置点にある。
【０１２７】
二次元画像生成部１７８は、もとの平面画像２０４を基礎として、まず、各画素をデプスマップの値分だけシフトさせる処理を行い、他方の視点画像を生成する。基準を左眼画像とすると、もとの平面画像２０４はそのまま左眼画像となる。第１四角形２５０を左に４画素、第２四角形２５２を左に２画素、第３四角形２５４を右に４画素シフトし、図５０のごとく、右眼画像２０２が作成される。画像端調整部１６８は、オブジェクトの移動による画素情報の欠落部分２６０を視差が「０」である、背景と判断される近接画素によって埋める。
【０１２８】
つづいて二次元画像生成部１７８は、適正視差を満たすデプスを計算する。デプスの範囲をＫ１〜Ｋ２とし、各画素のデプス値をＧｘｙとすると、デプスマップは図３７においてＨｘｙをＧｘｙへ変更した形になる。また、ユーザが保有する表示装置の適正視差がＪ１〜Ｊ２であることが分かったとする。この場合、そのデプスマップにおいて、各画素のデプス値Ｇは以下のように変換され、新しいデプス値Ｆｘｙが得られる。
【０１２９】
Ｆｘｙ＝Ｊ１＋（Ｇｘｙ−Ｋ１）×（Ｊ２−Ｊ１）／（Ｋ２−Ｋ１）
前述の例では、Ｋ１＝−４、Ｋ２＝４であり、また、Ｊ１＝−３、Ｊ２＝２であったとすると、この変換式により、図４９のデプスマップは図５１のデプスマップへ変換される。すなわち、「４」は「２」へ、「２」は「１」へ、「−４」は「−３」へそれぞれ変換される。Ｋ１とＫ２の間の中間値は、Ｊ１とＪ２の間に変換される。例えば第２四角形２５２はＧｘｙ＝２で、Ｆｘｙ＝０．７５となる。Ｆｘｙが整数にならない場合は、四捨五入や近置視差が小さくなるような処理を施せばよい。
【０１３０】
なお、上述の変換式は線形変換の例であるが、さらにＧｘｙに対する重み付け関数Ｆ（Ｇｘｙ）をかけたり、その他いろいろな非線形変換も考えられる。また、もとの平面画像２０４から、オブジェクトを互いに逆方向にシフトして新たに左右画像を生成することもできる。多眼式の場合は同様の処理を複数回行い、多視点画像を生成すればよい。
【０１３１】
以上が実施の形態に係る立体画像処理装置１００の構成および動作である。 立体画像処理装置１００は装置として説明したが、これはハードウエアとソフトウエアの組合せでもよく、ソフトウエアのみでも構成できる。その場合、立体画像処理装置１００の任意の部分をライブラリ化して各種プログラムから呼びだし可能にすると利便性が高い。プログラマは立体表示の知識が必要となる部分のプログラミングをスキップできる。ユーザにとっては、ソフトウエアやコンテンツによらず、立体表示に関する操作、すなわちＧＵＩが共通になり、設定した情報は他のソフトウエアでも共有できるため再設定の手間が省ける。
【０１３２】
なお、立体表示に関する処理ではなく、情報を複数のプログラム間で共有するだけでも有用である。各種プログラムはその情報を参照して画像の状態を決定できる。共有される情報の例は、前述の立体画像処理装置１００の情報取得部１１８で取得される情報である。この情報を図示しない記録部または補正マップ保持部１４０などに保持しておけばよい。
【０１３３】
図５２〜図５４は、以上の立体画像処理装置１００をライブラリとして利用する一例を示す。図５２は立体表示ライブラリ３００の用途を示す。立体表示ライブラリ３００は複数のプログラムＡ３０２、プログラムＢ３０４、プログラムＣ３０６などから関数を呼び出す形で参照される。パラメータファイル３１８には前述の情報のほか、ユーザの適正視差などが格納されている。立体表示ライブラリ３００はＡＰＩ（アプリケーションプログラムインタフェイス）３１０を介して複数の装置Ａ３１２、装置Ｂ３１４、装置Ｃ３１６などで利用される。
【０１３４】
プログラムＡ３０２等の例として、ゲーム、いわゆるＷｅｂ３Ｄと呼ばれる三次元アプリケーション、三次元デスクトップ画面、三次元地図、二次元画像である視差画像のビュア、デプス情報付画像などのビュアが考えられる。ゲームの中でも、当然ながら座標の使い方が違うものがあるが、立体表示ライブラリ３００はそれにも対応できる。
【０１３５】
一方、装置Ａ３１２等の例として、２眼や多眼のパララックスバリア式、シャッタめがね方式、偏光めがね方式など、視差を利用する任意の立体表示装置である。
【０１３６】
図５３は三次元データソフト４０２に立体表示ライブラリ３００が組み込まれた例を示す。三次元データソフト４０２はプログラム本体４０４と、そのために適正視差を実現する立体表示ライブラリ３００と、撮影指示処理部４０６を備える。プログラム本体４０４はユーザインタフェイス４１０を介してユーザと連絡する。撮影指示処理部４０６は、ユーザの指示にしたがって、プログラム本体４０４の動作中の所定の場面を仮想的にカメラ撮影する。撮影された画像は画像記録装置４１２に記録される。また、立体表示装置４０８へ出力される。
【０１３７】
たとえば三次元データソフト４０２がゲームソフトであるとする。その場合、ユーザはゲーム中は立体表示ライブラリ３００によって適切な立体感を体験しながらゲームを実行できる。ゲーム中、ユーザが記録に残したい場合、たとえば、対戦型戦闘ゲームで完全勝利をおさめたとき、ユーザインタフェイス４１０を介して撮影指示処理部４０６へ指示を出し、その場面を記録する。その際、立体表示ライブラリ３００を利用し、後に立体表示装置４０８で再生したときに適正視差となるよう視差画像が生成され、これが画像記録装置４１２の電子アルバム等へ記録される。なお、記録を視差画像という二次元画像で行うことにより、プログラム本体４０４の有する三次元データ自体は流出せず、著作権保護の面にも配慮することができる。
【０１３８】
図５４は、図５３の三次元データソフト４０２をネットワーク利用型のシステム４３０へ組み込んだ例を示す。
ゲーム機４３２は図示しないネットワークを介し、サーバ４３６と、ユーザ端末４３４に接続される。ゲーム機４３２は、いわゆるアーケードゲーム用で、通信部４４２、三次元データソフト４０２およびゲームをローカルに表示する立体表示装置４４０を備える。三次元データソフト４０２は図５３のものである。三次元データソフト４０２から立体表示装置４４０へ表示される視差画像は立体表示装置４４０について予め最適設定されている。三次元データソフト４０２による視差の調整は、後述のように通信部４４２を介してユーザへ画像を送信する際に利用される。ここで利用する表示装置は、視差を調整して立体画像を生成する機能を備えていればよく、必ずしも立体表示ができる装置でなくてもよい。
【０１３９】
ユーザ端末４３４は、通信部４５４、立体画像を見るためのビュアプログラム４５２および立体画像をローカルに表示する任意のサイズおよびタイプの立体表示装置４５０を備える。ビュアプログラム４５２には立体画像処理装置１００が実装されている。
【０１４０】
サーバ４３６は、通信部４６０、ゲームに関連してユーザが仮想的に撮影した画像を記録する画像保持部４６２、およびユーザの適正視差情報、ユーザのメールアドレスその他の個人情報などをユーザと対応づけて記録するユーザ情報保持部４６４を備える。サーバ４３６は例えばゲームの公式サイトとして機能し、ゲーム実行中にユーザが気に入った場面や名勝負の動画または静止画を記録する。立体表示は動画、静止画のいずれでも可能である。
【０１４１】
以上の構成における画像撮影の一例は以下の要領で行われる。ユーザは予めユーザ端末４３４の立体表示装置４５０で立体表示を行い、立体画像処理装置１００の機能をもとに適正視差を取得し、これを通信部４５４を介してサーバ４３６へ通知し、ユーザ情報保持部４６４へ格納してもらう。この適正視差は、ユーザの保有する立体表示装置４５０のハードウエアに関係のない汎用的な記述になっている。
【０１４２】
ユーザは任意のタイミングでゲーム機４３２によってゲームをする。その間、立体表示装置４４０には最初に設定されていた視差、またはユーザが手動調整した視差による立体表示がなされる。ゲームのプレイ中、またはリプレイ中、ユーザが画像の記録を希望すると、ゲーム機４３２の三次元データソフト４０２に内蔵された立体表示ライブラリ３００が、ふたつの通信部４４２、４６０を介してサーバ４３６のユーザ情報保持部４６４からこのユーザの適正視差を取得し、それに合わせて視差画像を生成し、再びふたつの通信部４４２、４６０を介して画像保持部４６２へ仮想的に撮影された画像に関する視差画像を格納する。ユーザは自宅へ帰ったのち、この視差画像をユーザ端末４３４へダウンロードすれば、所望の立体感で立体表示ができる。その際も、ビュアプログラム４５２のもつ立体画像処理装置１００により、視差の手動調整は可能である。
【０１４３】
以上、この応用例によれば、本来、表示装置のハードウエアごと、ユーザごとに設定しなければならない立体感に関するプログラミングが立体画像処理装置１００および立体表示ライブラリ３００に集約されており、ゲームソフトのプログラマは立体表示に関する複雑な要件を一切気にする必要がない。これはゲームソフトに限らず、立体表示を利用する任意のソフトウエアについても同様であり、立体表示を利用するコンテンツやアプリケーションの開発の制約を解消する。したがって、これらの普及を飛躍的に促進することができる。
【０１４４】
とくに、もともと三次元のＣＧデータが存在するゲームその他のアプリケーションの場合、従来は的確な立体表示をコーディングすることが困難であることも大きな原因となって、せっかく三次元データをもちながら、それを立体表示に利用しなかったことも多い。実施の形態に係る立体画像処理装置１００または立体表示ライブラリ３００によれば、そうした弊害を除去でき、立体表示アプリケーションの充実化に寄与することができる。
【０１４５】
なお、図５４ではユーザの適正視差をサーバ４３６に登録したが、ユーザはその情報を記録したＩＣカードなどを持参してゲーム機４３２を利用してもよい。このカードに、このゲームに関する得点や気に入った画像を記録してもよい。
【０１４６】
以上、本発明を実施の形態をもとに説明した。この実施の形態は例示であり、それらの各構成要素や各処理プロセスの組合せにいろいろな変形例が可能なこと、またそうした変形例も本発明の範囲にあることは当業者に理解されるところである。以下、そうした例を挙げる。
【０１４７】
第１の立体画像処理装置１００は、三次元データを入力することにより、高い精度で処理が可能である。しかし、三次元データをいったんデプス情報付画像へ落とし、これに対して第３の立体画像処理装置１００を利用して視差画像を生成してもよい。場合により、そのほうが計算コストが低いことがある。同じように複数の視点画像を入力する際も、高精度の対応点マッチングを用いてデプスマップを作ることも可能で、このようにしてデプス情報付画像へ落とし、これに対して第３の立体画像処理装置１００を利用して視差画像を生成してもよい。
【０１４８】
第１の立体画像処理装置１００において、カメラ仮配置部１３０を立体画像処理装置１００の構成としたが、これは立体画像処理装置１００の前処理であってもよい。カメラの仮配置までは、適正視差に関係なく処理できるためである。同様に、第１、第２、第３の立体画像処理装置１００を構成する任意の処理ユニットを立体画像処理装置１００の外部へ出すことも可能であり、立体画像処理装置１００の構成の自由度の高さは当業者に理解されるところである。
【０１４９】
実施の形態では、視差の制御を水平方向に行う場合を説明したが、同様の処理は垂直方向についても実施できる。
【０１５０】
立体表示ライブラリ３００や立体画像処理装置１００の動作中は、文字データを拡大処理するユニットを設けてもよい。たとえば水平２視点による視差画像の場合、ユーザの目に見える画像の水平解像度は１／２になる。その結果、文字の可読性が低下しうるので、水平方向へ２倍に文字を引き延ばす処理が有効である。垂直方向にも視差がある場合、同様に垂直方向へ文字を引き延ばすことも有用である。
【０１５１】
立体表示ライブラリ３００や立体画像処理装置１００の動作中は、表示している画像に「３Ｄ」などの文字やマークを入れる「動作中表示部」を設けてもよい。その場合、ユーザは視差の調整が可能な画像であるか否かを知ることができる。
【０１５２】
立体表示／通常表示の切替ユニットを設けてもよい。このユニットはＧＵＩを含み、ユーザが所定のボタンをクリックすると、表示が立体表示から通常の二次元表示に切り替わり、その反対も可能な構成とすれば便利である。
【０１５３】
情報取得部１１８は必ずしもユーザ入力により情報を取得するのではなく、プラグアンドプレイなどの機能により自動的に取得できる情報があってもよい。
【０１５４】
実施の形態では、ＥとＡを導出する方法をとったが、これらを固定し、他のパラメータを導出する方法であってもよく、変数の指定は自由である。
【０１５５】
【発明の効果】
本発明によれば、以下のような効果がある。
１．人の生理に適合しやすい立体画像を生成または表示できる。
２．表示対象画像が変わってもユーザに適切な立体画像を生成または表示できる。
３．簡単な操作で立体表示の立体感を調整できる。
４．適切な立体表示が可能なコンテンツまたはアプリケーションづくりに際し、プログラマの負担を軽減できる。
５．立体表示を最適化しようとするユーザの手間が軽減される。
６．通常、プラグアンドプレイ機能の対象にならない立体感調整やヘッドトラッキング情報を容易に実現でき、後付のパララックスバリアのように、原理上プラグアンドプレイができない装置についても同様である。
【図面の簡単な説明】
【図１】 理想的な立体視ができているユーザ、スクリーン、再生オブジェクト１４の位置関係を示す図である。
【図２】 図１の状態を実現する撮影系の例を示す図である。
【図３】 図１の状態を実現する撮影系の別の例を示す図である。
【図４】 図１の状態を実現する撮影系の別の例を示す図である。
【図５】 第１の立体画像処理装置に利用するモデル座標系を示す図である。
【図６】 第１の立体画像処理装置に利用するワールド座標系を示す図である。
【図７】 第１の立体画像処理装置に利用するカメラ座標系を示す図である。
【図８】 第１の立体画像処理装置に利用するビューボリュームを示す図である。
【図９】 図８のボリュームの透視変換後の座標系を示す図である。
【図１０】 第１の立体画像処理装置に利用するスクリーン座標系を示す図である。
【図１１】 第１の立体画像処理装置の構成図である。
【図１２】 第２の立体画像処理装置の構成図である。
【図１３】 第３の立体画像処理装置の構成図である。
【図１４】 図１４（ａ）、図１４（ｂ）は、それぞれ、第１の立体画像処理装置の立体感調整部によって表示される左眼画像と右眼画像を示す図である。
【図１５】 第１の立体画像処理装置の立体感調整部によって表示される、異なる視差をもつ複数のオブジェクトを示す図である。
【図１６】 第１の立体画像処理装置の立体感調整部によって表示される、視差が変化するオブジェクトを示す図である。
【図１７】 適正視差が実現される際のカメラ画角、画像サイズ、視差の関係を示す図である。
【図１８】 図１７の状態を実現する撮影系の位置関係を示す図である。
【図１９】 図１７の状態を実現する撮影系の位置関係を示す図である。
【図２０】 多視点画像を適正視差で生成する際のカメラ配置を示す図である。
【図２１】 第１の立体画像処理装置の歪処理部が利用する視差補正マップを示す図である。
【図２２】 図２１の視差補正マップにしたがって視差画像を生成する際のカメラ視点を示す図である。
【図２３】 図２１の視差補正マップにしたがって視差画像を生成する際の別のカメラ視点を示す図である。
【図２４】 第１の立体画像処理装置の歪処理部が利用する視差補正マップを示す図である。
【図２５】 図２４の視差補正マップにしたがって視差画像を生成する際のカメラ視点を示す図である。
【図２６】 第１の立体画像処理装置の歪処理部が利用する距離感補正マップを示す図である。
【図２７】 図２６の距離感補正マップにしたがって視差画像を生成する際のカメラ視点を示す図である。
【図２８】 第１の立体画像処理装置の歪処理部が利用する別の距離感補正マップを示す図である。
【図２９】 図２８の距離感補正マップにしたがって視差画像を生成する際のカメラ視点を示す図である。
【図３０】 図３０（ａ）、図３０（ｂ）、図３０（ｃ）、図３０（ｄ）、図３０（ｅ）、図３０（ｆ）は、いずれも第１の立体画像処理装置の歪処理部が三次元空間に処理を施した結果得られる視差分布の上面図である。
【図３１】 第１の立体画像処理装置の歪処理部による処理の原理を示す図である。
【図３２】 図３１の処理を具体的に示す図である。
【図３３】 図３１の処理を具体的に示す図である。
【図３４】 図３１の処理を具体的に示す図である。
【図３５】 第１の立体画像処理装置の歪処理部による処理の別の例を示す図である。
【図３６】 図３５の処理を具体的に示す図である。
【図３７】 デプスマップを示す図である。
【図３８】 第３の立体画像処理装置の歪処理部による処理の例を示す図である。
【図３９】 第３の立体画像処理装置の歪処理部による処理によって生成されたデプスマップを示す図である。
【図４０】 第３の立体画像処理装置の歪処理部による処理の別の例を示す図である。
【図４１】 第２の立体画像処理装置の二次元画像生成部による処理の例を示す図である。
【図４２】 視差画像の例を示す図である。
【図４３】 第２の立体画像処理装置の二次元画像生成部によって合成位置がシフトされた視差画像を示す図である。
【図４４】 第２の立体画像処理装置の画像端調整部の処理を示す図である。
【図４５】 第２の立体画像処理装置の処理を示す図である。
【図４６】 第２の立体画像処理装置の別の処理を示す図である。
【図４７】 第２の立体画像処理装置の別の処理を示す図である。
【図４８】 デプスマップが付加された平面画像を示す図である。
【図４９】 デプスマップを示す図である。
【図５０】 第２の立体画像処理装置の二次元画像生成部でデプスマップをもとに視差画像を生成する様子を示す図である。
【図５１】 第２の立体画像処理装置の二次元画像生成部で補正されたデプスマップを示す図である。
【図５２】 実施の形態に係る立体画像処理装置をライブラリ化して利用する様子を示す図である。
【図５３】 立体表示ライブラリを三次元データソフトに組み込んだ構成図である。
【図５４】 立体表示ライブラリをネットワーク利用型のシステムで利用する様子を示す図である。
【符号の説明】
１０ ユーザ、 １２ スクリーン、 １４ 再生オブジェクト、 ２０ 現実のオブジェクト、 ２２，２４，２６，２８ カメラ、 ３０ 前方投影面、３２ 後方投影面、 １００ 立体画像処理装置、 １１２ 立体感調整部、１１４，１５２，１７０ 視差制御部、 １１６ フォーマット変換部、 １１８ 情報取得部、 １２２ 指示取得部、 １２４ 視差特定部、 １３２ カメラ配置決定部、 １３６，１７４ 歪処理部、 １４０，１７６ 補正マップ保持部、 １４２ 二次元画像生成部、 １５０ 視差量検出部、 １５６ ヘッダ検査部、 １５８ マッチング部、 １６０ 位置シフト部、 １６４ 視差書込部、 １６８ 画像端調整部、 １７８ 二次元画像生成部、 ２１０光軸交差面、 ３００ 立体表示ライブラリ、 ４０２ 三次元データソフト、 ４０６ 撮影指示処理部、 ４３０ ネットワーク利用型のシステム、 ４３２ ゲーム機、 ４３４ ユーザ端末、 ４３６ サーバ、 ４５２ ビュアプログラム。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a stereoscopic image processing technique, and more particularly to a method and apparatus for generating or displaying a stereoscopic image based on a parallax image.
[0002]
[Prior art]
In recent years, the lack of network infrastructure has been regarded as a problem. However, the transition to broadband has started, and rather, the number of types and the number of contents that effectively use a wide band are becoming conspicuous. Video has always been the most important means of expression, but most of the efforts so far have been related to improvements in display quality and data compression ratio, and compared to these, technical possibilities to expand the possibilities of expression itself. There is a feeling that such efforts are in the back.
[0003]
Under such circumstances, stereoscopic video display (hereinafter simply referred to as “stereoscopic display”) has been studied in various ways, and has been put into practical use in a limited market where theater applications and special display devices are used. In the future, R & D in this direction will accelerate with the aim of providing more realistic content, and the time will come when individual users can enjoy stereoscopic display at home.
[0004]
[Problems to be solved by the invention]
Under such circumstances, some problems have been pointed out for stereoscopic display. For example, it is difficult to optimize the parallax that causes a three-dimensional effect. Originally, it does not actually project a three-dimensional object, but rather casts the image by shifting it in pixel units with respect to the left and right eyes, and it is not easy to give the artificial three-dimensional effect a natural feeling. Absent.
[0005]
In addition, too much parallax may be a problem, and a viewer of a stereoscopic image (hereinafter also simply referred to as a user) may complain of mild discomfort. Of course, there are various factors such as not only the three-dimensional display but also the situation or sense of surroundings of the scene being displayed is not consistent with the situation. However, empirically, such a problem is easily observed when the parallax is too large, in other words, when the stereoscopic effect is too strong.
[0006]
The above is a human physiological story, but apart from that, there are technical factors that hinder the spread of stereoscopic video content and applications. Although stereoscopic vision is realized by parallax, even if the parallax is expressed by the shift amount of the pixels of the left and right images, the same stereoscopic video may or may not be properly stereoscopically viewed depending on the hardware of the display device. . If the parallax that expresses far away exceeds the interocular distance, stereoscopic viewing is theoretically impossible. Today, as the resolution and screen size of display devices diversify, such as PCs (personal computers), television receivers, and portable devices, the most suitable content for stereoscopic display is created by considering various hardware. Is more difficult, or it is more accurate that no methodology is provided for it.
[0007]
Also, given that methodology, it would be difficult to expect a general programmer to understand it and use it to create content and applications.
[0008]
The present invention has been made in view of such a background, and an object thereof is to generate or display a stereoscopic image that is easily adapted to human physiology. Another object is to generate or display a stereoscopic image suitable for the user even if the display target image or the display device changes. Yet another object is to adjust the stereoscopic effect with a simple operation when a stereoscopic display is being made. Yet another object is to reduce the burden on the programmer when creating contents or applications capable of appropriate stereoscopic display. Still another object is to provide a technology for realizing appropriate stereoscopic display as a business model.
[0009]
[Means for Solving the Problems]
The inventor's knowledge that forms the basis of the present invention is that the appropriate parallax is once separated from the hardware of the display device and the elements such as the distance between the user and the display device (hereinafter collectively referred to as “hardware”). It is in. That is, the general parallax expression is once described in a general form that does not depend on hardware by generalizing the expression of the appropriate parallax with the camera interval and the optical axis crossing position described later. “Hardware-independent” means that reading of hardware information specific to the display device is not necessary in principle, and if this general description is made, the parallax image is based on the appropriate parallax. If desired, the desired three-dimensional display is realized.
[0010]
Providing appropriate parallax for the acquisition of appropriate parallax and providing control for realizing the appropriate parallax in the library, so that general programmers can call this library and be aware of the principles and programming of complex stereoscopic vision. 3D display is realized.
[0011]
Of the various aspects of the present invention, the first group is based on a technique for obtaining an appropriate parallax based on a user response. This technique can be used for “initial setting” of parallax by the user, and once the appropriate parallax is acquired in the apparatus, the appropriate parallax is realized even when another image is displayed. However, this technique is not limited to the initial setting, but is also used for “manual adjustment” in which the user appropriately adjusts the parallax of the image being displayed. The following relates to the first group.
[0012]
The present invention relates to a stereoscopic image processing apparatus, an instruction acquisition unit that acquires a user's response to a stereoscopic image displayed based on a plurality of viewpoint images corresponding to different parallaxes, and based on the acquired response, A parallax identifying unit that identifies appropriate parallax for the user.
[0013]
The instruction acquisition unit is provided as, for example, a GUI (graphical user interface, the same applies hereinafter), and first displays while changing the parallax between viewpoint images. When the user feels the stereoscopic effect he likes, the user inputs that effect by operating a button or the like.
[0014]
A “stereoscopic image” is an image displayed with a stereoscopic effect, and the substance of the data is a “parallax image” in which a plurality of images are given parallax. A parallax image is generally a set of a plurality of two-dimensional images. Each image constituting the parallax image is a “viewpoint image” having a corresponding viewpoint. That is, a parallax image is composed of a plurality of viewpoint images, and when it is displayed, it is displayed as a stereoscopic image. The display of a stereoscopic image is also simply referred to as “stereoscopic display”.
[0015]
“Parallax” is a parameter for producing a three-dimensional effect, and various definitions are possible, but as an example, it can be expressed by a shift amount of a pixel representing the same point between viewpoint images. Hereinafter, in this specification, unless otherwise specified, the definition is followed.
[0016]
The range of the appropriate parallax may be specified. In this case, both ends of the range are referred to as “limit parallax”. The “specific parallax specification” may be performed with a maximum value allowable as a parallax of a nearby object described later.
[0017]
The stereoscopic image processing apparatus of the present invention may further include a parallax control unit that performs processing so that the specified appropriate parallax is realized even when another image is displayed. When another image is a stereoscopic image generated from three-dimensional data as a starting point, the parallax control unit may determine a plurality of viewpoints for generating the stereoscopic image according to the appropriate parallax. More specifically, the distance between a plurality of viewpoints and the intersection position of the optical axes for viewing the object from these viewpoints may be determined. An example of these processes is performed by a camera arrangement determination unit described later. If these processes are performed in real time, the optimum stereoscopic display is always realized.
[0018]
The parallax control unit may perform control so that proper parallax is achieved for a predetermined basic three-dimensional space to be displayed. An example of this processing is performed by a projection processing unit described later.
[0019]
The parallax control unit may perform control so that the appropriate parallax is realized with respect to the coordinates of the closest object and the coordinates of the farthest object in the three-dimensional space. An example of this processing is performed by a projection processing unit described later. The object may be static.
[0020]
“Neighboring” is the line of sight of a camera placed at each of a plurality of viewpoints, that is, before a plane (hereinafter referred to as “optical axis crossing position”) at an optical axis crossing position (hereinafter referred to as “optical axis crossing position”). Indicates a state in which a parallax is stereoscopically viewed. On the contrary, “distant” refers to a state in which a parallax is given so as to be stereoscopically viewed behind the optical axis crossing plane. The closer the parallax of the near object, the closer to the user is sensed, and the larger the parallax of the far object, the farther away from the user. In other words, unless otherwise specified, the parallax is defined as a non-negative value in which the positive and negative are not reversed in the near and far positions, and both the near and far parallaxes are zero at the optical axis crossing plane.
[0021]
Of the object or space to be displayed, the optical axis crossing plane coincides with the screen surface of the display device for a portion having no parallax. This is because the line of sight seen from both the left and right eyes reaches the same position in the screen plane, that is, intersects the pixels where no parallax is applied.
[0022]
When the another image is a plurality of two-dimensional images to which parallax has already been given, the parallax control unit may determine the horizontal shift amount of the plurality of two-dimensional images according to the appropriate parallax. In this aspect, the input for stereoscopic display is not generated with a high degree of freedom starting from three-dimensional data, but is already generated parallax images, and the parallax is fixed. In this case, it is not possible to return to the original three-dimensional space or the real space where the image was actually taken and change the camera position to redraw or re-shoot. For this reason, the parallax is adjusted by shifting the viewpoint images constituting the parallax image or the pixels included in the viewpoint images horizontally.
[0023]
When the another image is a planar image to which depth information is given (hereinafter also referred to as “image with depth information”), the parallax control unit may adjust the depth according to the appropriate parallax. An example of this processing is performed by a two-dimensional image generation unit of a third stereoscopic image processing apparatus described later.
[0024]
The stereoscopic image processing apparatus further includes a parallax holding unit that records appropriate parallax, and the parallax control unit is activated at a predetermined timing, for example, when the apparatus is started, or when the apparatus has a stereoscopic image processing function or a part thereof. For example, the appropriate parallax may be read and processing may be performed using the value as an initial value. That is, “startup” may have a hardware meaning or a software meaning. According to this aspect, once the user determines an appropriate parallax, automatic processing for adjusting the stereoscopic effect is realized thereafter. This is a function called “initial setting of appropriate parallax”.
[0025]
Another aspect of the present invention relates to a method for processing a stereoscopic image, the step of displaying a plurality of stereoscopic images with different parallax to a user, and specifying an appropriate parallax for the user based on a user response to the displayed stereoscopic image Steps.
[0026]
Still another aspect of the present invention also relates to a stereoscopic image processing method, which includes a step of acquiring appropriate parallax that depends on a user, and a step of processing an image before display so that the acquired appropriate parallax is realized. Here, “acquisition” may be a process of positively specifying or a process of reading from the parallax holding unit or the like.
[0027]
If each of these steps is implemented as a function of a library for stereoscopic display and the function of this library can be called as a function from multiple programs, the programmer must write the program in consideration of the hardware of the stereoscopic display device. Is effective.
[0028]
The second group of the present invention is based on a technique for adjusting parallax based on a user instruction. This technique can be used for “manual adjustment” of parallax by the user, and the user can change the stereoscopic effect of the displayed image as appropriate. However, this technique is not limited to manual adjustment, and can also be used to automatically adjust the parallax of the image by reading the above-described appropriate parallax when stereoscopically displaying a certain image. The difference from the automatic adjustment of the first group is that the automatic adjustment of the second group acts on a two-dimensional parallax image or an image with depth information. Use a group of technologies. The following relates to the second group.
[0029]
An aspect of the present invention relates to a stereoscopic image processing apparatus, an instruction acquisition unit that acquires a user instruction for a stereoscopic image displayed from a plurality of viewpoint images, and a parallax amount between the plurality of viewpoint images according to the acquired instruction And a parallax control unit that changes. An example of this processing is shown in FIG. 45 described later, and is a typical example of “manual adjustment”. If the user's instruction is provided by a simple GUI such as button operation, it is highly convenient.
[0030]
Another aspect of the present invention also relates to a stereoscopic image processing device, and relates to a parallax amount detection unit that detects a first parallax amount that occurs when a stereoscopic image is displayed from a plurality of viewpoint images, and the first parallax amount is allowed by a user. A parallax controller that changes the parallax amount between the plurality of viewpoint images so as to fall within a range of a second parallax amount that is a parallax amount. This is a typical example of “automatic adjustment”, and the above-described appropriate parallax can be used as the second parallax amount. An example of this processing is shown in FIG. 46 described later.
[0031]
The parallax amount detection unit detects the maximum value of the first parallax amount, and the parallax control unit changes the parallax amount between the viewpoint images so that the maximum value does not exceed the maximum value of the second parallax amount. Also good. In order to avoid excessive stereoscopic effect due to excessive parallax, the intent is to protect the maximum value of parallax, that is, limit parallax. The maximum value here may be considered as the maximum value on the near side.
[0032]
The parallax amount detection unit calculates the corresponding point matching between the plurality of viewpoint images to detect the first parallax amount, or the first parallax amount recorded in advance in any header of the plurality of viewpoint images. It may be detected. An example of these processes is shown in FIG. 47 described later.
[0033]
The parallax control unit may change the amount of parallax between the plurality of viewpoint images by shifting the synthesis position of the plurality of viewpoint images. This is common to FIGS. The shift of the composition position is a horizontal or vertical shift in units of pixels or the entire image. When the input is an image with depth information, the parallax control unit may change the parallax amount by adjusting the depth information.
[0034]
Another aspect of the present invention relates to a stereoscopic image processing method, a step of obtaining a user instruction for a stereoscopic image displayed based on a plurality of viewpoint images, and a parallax amount between the plurality of viewpoint images according to the instruction Changing.
[0035]
Another aspect of the present invention also relates to a stereoscopic image processing method, the step of detecting a first parallax amount that occurs when displaying a stereoscopic image from a plurality of viewpoint images, and the first parallax amount is an allowable parallax amount of a user And changing the amount of parallax between the plurality of viewpoint images so as to fall within the range of the second amount of parallax.
[0036]
These steps may be implemented as functions of a stereoscopic display library, and the functions of this library may be called as functions from a plurality of programs.
[0037]
The third group of the present invention is based on a technique for correcting parallax based on the position in the image. This “automatic correction” acts to reduce the user's uncomfortable feeling or refusal to stereoscopic display, and can be used in combination with the technologies of the first and second groups. In general, at the time of stereoscopic display, technical or physiological problems are pointed out, such as closer to the edge of the image, a plurality of viewpoint images are observed to be shifted and a sense of incongruity is likely to be produced. In the third group, this problem is reduced by reducing the parallax in a portion close to the image edge or adjusting the parallax so that the object moves from the near side to the far side. The following relates to the third group.
[0038]
An aspect of the present invention relates to a stereoscopic image processing apparatus, and maintains a parallax control unit that corrects parallax between a plurality of viewpoint images for displaying a stereoscopic image, and a correction map that the parallax control unit should refer to during the processing. The correction map is described so that the parallax is corrected based on the position in the viewpoint image. Examples of the correction map include a parallax correction map and a distance sense correction map.
[0039]
For example, the parallax control unit reduces the parallax at the peripheral portions of the plurality of viewpoint images, or changes the parallax so that the object is sensed further from the user. The parallax control unit may change the parallax by selectively performing processing on any of the plurality of viewpoint images.
[0040]
When multiple viewpoint images are generated from three-dimensional data, that is, when the viewpoint image can be generated by returning to the three-dimensional space, the parallax control unit controls the camera parameters to generate the parallax when generating the multiple viewpoint images. May be changed. Camera parameters include the distance between the left and right cameras, the angle at which an object is viewed from the camera, or the optical axis crossing position.
[0041]
Similarly, when a plurality of viewpoint images are generated from three-dimensional data, the parallax control unit may change the parallax by distorting the three-dimensional space itself in, for example, the world coordinate system when generating the plurality of viewpoint images. . On the other hand, when a plurality of viewpoint images are generated from an image with depth information, the parallax control unit may change the parallax by manipulating the depth information.
[0042]
Another aspect of the present invention relates to a stereoscopic image processing method, the step of acquiring a plurality of viewpoint images for displaying a stereoscopic image, and a parallax between the acquired viewpoint images based on positions in the viewpoint images. And changing steps. These steps may be implemented as functions of a stereoscopic display library, and the functions of this library may be called as functions from a plurality of programs.
[0043]
The fourth group of the present invention relates to a technology that provides the first to third groups and related functions as a software library, reduces the burden on programmers and users, and promotes the spread of stereoscopic image display applications. The following relates to the fourth group.
[0044]
An aspect of the present invention relates to a stereoscopic image processing method, which stores information related to stereoscopic image display on a memory, shares the stored information among a plurality of different programs, and any of these programs is a stereoscopic image. When displaying the image, the state of the image to be output is determined with reference to the stored information. An example of the state of the image is how much parallax is given to the parallax image.
[0045]
The “held information” may include information on any of the format of an image input to the stereoscopic image display device, the display order of viewpoint images, and the amount of parallax between viewpoint images. In addition to sharing the stored information, a process unique to stereoscopic image display may be shared by a plurality of programs. An example of “stereoscopic image display specific processing” is processing for determining stored information. Another example is a process related to a graphical user interface for determining an appropriate parallax, a display process for a parallax adjustment screen that supports realization of an appropriate parallax state, a process for detecting and tracking a user's head position, and a stereoscopic display device For example, a process of displaying an image for adjusting the image.
[0046]
Another aspect of the present invention relates to a stereoscopic image processing apparatus, and a stereoscopic effect adjusting unit that provides a user with a graphical user interface for adjusting the stereoscopic effect of a stereoscopic display image, and the result of adjusting the stereoscopic effect by the user. And a parallax control unit that generates a parallax image in a form that protects the limit parallax.
[0047]
The apparatus further includes an information detection unit that acquires information to be referred to in order to optimize the stereoscopic image display, and a conversion unit that converts the format of the parallax image generated by the parallax control unit according to the acquired information. May be included.
[0048]
The parallax control unit may control the camera parameters based on the three-dimensional data to generate a parallax image while protecting the limit parallax, or may control the depth of the image with depth information to generate the parallax image. Alternatively, the parallax image may be generated after determining the horizontal shift amounts of the plurality of two-dimensional images having parallax.
[0049]
The fifth group of the present invention relates to one application or business model using the above-described stereoscopic image processing technology or related technology. A fourth group of software libraries is available. The following relates to the fifth group.
[0050]
An aspect of the present invention relates to a stereoscopic image processing method, and converts an appropriate parallax for stereoscopic display of a parallax image into an expression format that does not depend on the display device hardware, and the appropriate parallax according to the expression format is changed between different display devices. Circulate.
[0051]
Another aspect of the present invention also relates to a stereoscopic image processing method, a step of reading an appropriate parallax of a user acquired by a first display device into a second display device, and a second display device according to the appropriate parallax. Adjusting the parallax between the parallax images, and outputting the adjusted parallax image from the second display device. For example, the first display device is a device normally used by the user, and the second display device is a device provided at another location.
[0052]
Another aspect of the present invention relates to a stereoscopic image processing device, which includes a first display device, a second display device, and a server connected via a network, and the first display device is acquired by the device. The user's appropriate disparity information is transmitted to the server, the server receives the appropriate disparity information, records it in association with the user, and when the user requests output of image data on the second display device, the device Reads out the appropriate parallax information of the user from the server, adjusts the parallax, and then outputs a parallax image.
[0053]
It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.
[0054]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 shows the positional relationship between the user 10, the screen 12, and the playback object 14 displayed in a three-dimensional manner. The distance between eyes of the user 10 is E, the distance between the user 10 and the screen 12 is D, and the width of the reproduction object 14 when displayed is W. Since the playback object 14 is stereoscopically displayed, it has pixels that are sensed closer to the screen 12, that is, pixels that are located closer to each other, and pixels that are sensed farther than the screen 12, that is, pixels that are located farther away. Pixels without parallax appear on the screen 12 because they appear exactly the same from both eyes on the screen 12
.
[0055]
FIG. 2 shows an imaging system for generating the ideal display of FIG. The distance between the two cameras 22 and 24 is E, and the distance from these to the optical axis crossing position when viewing the real object 20 (this is called the optical axis crossing distance) is D, and the same width as the screen 12 is set. If an object 20 with an actual angle of view and a width of W is actually photographed, parallax images can be obtained from the two cameras. If this is displayed on the screen 12 of FIG. 1, the ideal state of FIG. 1 is realized.
[0056]
FIG. 3 and FIG. 4 show a state where the positional relationship of FIG. 2 is A times (A <1) and B times (B> 1), respectively. The ideal state of FIG. 1 is also realized by the parallax images obtained with these positional relationships. In other words, the basic of ideal stereoscopic display starts from making W: D: E constant. This relationship also serves as a basis for adding parallax.
[0057]
FIGS. 5 to 10 show the outline of processing until stereoscopic display is performed based on the three-dimensional data of the object 20 in the embodiment.
FIG. 5 shows a model coordinate system, that is, a coordinate space that each three-dimensional object 20 has. In this space, coordinates when the object 20 is modeled are given. Usually, the origin is at the center of the object 20.
[0058]
FIG. 6 shows the world coordinate system. The world space is a wide space in which a scene is formed by arranging objects 20, floors, and walls. From the modeling in FIG. 5 to the determination of the world coordinate system in FIG. 6 can be recognized as “construction of three-dimensional data”.
[0059]
FIG. 7 shows the camera coordinate system. Conversion to the camera coordinate system is performed by setting the camera 22 at an arbitrary angle of view from an arbitrary position in the world coordinate system. Camera position, direction, and angle of view are camera parameters. In the case of stereoscopic display, since the parameters are determined for two cameras, the camera interval and the optical axis crossing position are also determined. Also, the origin is moved so that the midpoint of the two cameras is the origin.
[0060]
8 and 9 show a perspective coordinate system. First, as shown in FIG. 8, the space to be displayed is clipped by the front projection plane 30 and the rear projection plane 32. As will be described later, one feature of the embodiment is that the surface having the maximum disparity point is the front projection surface 30 and the surface having the maximum disparity point is the rear projection surface 32. After clipping, this view volume is converted into a rectangular parallelepiped as shown in FIG. The processing in FIGS. 8 and 9 is also referred to as projection processing.
[0061]
FIG. 10 shows the screen coordinate system. In the case of stereoscopic display, images from a plurality of cameras are converted into coordinate systems having screens, and a plurality of two-dimensional images, that is, parallax images are generated.
[0062]
11, 12, and 13 illustrate the configuration of the stereoscopic image processing apparatus 100 that is partially different. Hereinafter, for convenience, the stereoscopic image processing devices 100 are also referred to as first, second, and third stereoscopic image processing devices 100, respectively. These three-dimensional image processing apparatuses 100 can be integrated into the apparatus, but here, they are divided into three parts to avoid the complexity of the figure. The first stereoscopic image processing apparatus 100 is effective when the object and space to be drawn can be obtained from the three-dimensional data stage, and therefore the main input is the three-dimensional data. The second stereoscopic image processing apparatus 100 is effective for the parallax adjustment of a plurality of two-dimensional images to which parallax has already been given, that is, existing parallax images, and therefore inputs a two-dimensional parallax image. The third stereoscopic image processing apparatus 100 operates the depth information of the image with depth information to realize proper parallax, and therefore mainly inputs the image with depth information. These three inputs are collectively referred to as “original data”.
[0063]
In the case where the first to third stereoscopic image processing devices 100 are integrated and mounted, an “image format determination unit” is provided as a preprocessing unit thereof, and after determining three-dimensional data, a parallax image, and an image with depth information, It is good also as a structure which starts the optimal thing among the 1st-3rd three-dimensional image processing apparatuses 100. FIG.
[0064]
The first stereoscopic image processing apparatus 100 has functions of “initial setting” and “automatic adjustment” when setting a stereoscopic effect for stereoscopic display. When the user designates a range of his / her appropriate parallax for a stereoscopically displayed image, this is acquired by the system, and thereafter, when another stereoscopic image is displayed, a conversion process is performed in advance to realize this proper parallax. Is displayed. Therefore, the first stereoscopic image processing apparatus 100 allows the user to enjoy a stereoscopic display that suits him / herself after the setting procedure is performed only once in principle.
[0065]
The first stereoscopic image processing apparatus 100 further has a sub-function called “parallax correction” that artificially relaxes the parallax in the periphery of the image. As described above, as the image edge is approached, a shift between a plurality of viewpoint images is easily recognized as a “double image”. This is mainly due to mechanical errors such as parallax barrier and screen warpage of the display device. Therefore, at the peripheral part of the image, 1) reduce both the near parallax and the far parallax, 2) reduce the near parallax, and leave the far parallax as it is, 3) regardless of the near parallax and the far parallax. Various methods are implemented, such as shifting to a distant parallax. This “parallax correction” function also exists in the third stereoscopic image processing apparatus 100, but the processing differs depending on the difference in input data.
[0066]
The first stereoscopic image processing apparatus 100 includes a stereoscopic effect adjusting unit 112 that adjusts the stereoscopic effect based on a response from the user to a stereoscopically displayed image, and a parallax that stores the appropriate parallax specified by the stereoscopic effect adjusting unit 112. The information holding unit 120, the parallax control unit 114 that reads the appropriate parallax from the parallax information holding unit 120, generates the parallax image having the appropriate parallax from the original data, acquires the hardware information of the display device, and also displays the stereoscopic display An information acquisition unit 118 having a function of acquiring a method and a format conversion unit 116 that changes the format of the parallax image generated by the parallax control unit 114 based on the information acquired by the information acquisition unit 118 are included. Original data is simply called three-dimensional data, but strictly speaking, this is object and space data described in the world coordinate system.
[0067]
Examples of information acquired by the information acquisition unit 118 include the number of viewpoints for stereoscopic display, the method of a stereoscopic display device such as space division or time division, whether or not shutter glasses are used, and the viewpoint image in a multi-view type. There is a method of arrangement, whether there is an arrangement of viewpoint images in which parallax is reversed in the parallax image, a result of head tracking, and the like. Only the result of the head tracking is exceptionally inputted to the camera arrangement determining unit 132 via a route (not shown) and processed there.
[0068]
The above configuration can be realized in hardware by a CPU, memory, or other LSI of an arbitrary computer, and in software, it can be realized by a program having a GUI function, a parallax control function, or other functions. Here, functional blocks realized by the cooperation are depicted. Accordingly, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and the situation is the same for the subsequent configurations.
[0069]
The stereoscopic effect adjusting unit 112 includes an instruction acquiring unit 122 and a parallax specifying unit 124. The instruction acquisition unit 122 acquires the appropriate parallax range for the stereoscopically displayed image when the user specifies the range. Based on the range, the parallax specifying unit 124 specifies an appropriate parallax when the user uses the display device. The appropriate parallax is expressed in an expression format that does not depend on the hardware of the display device. By realizing the appropriate parallax, stereoscopic vision suitable for the user's physiology becomes possible.
[0070]
The parallax control unit 114 includes a camera temporary arrangement unit 130 that temporarily sets camera parameters, a camera arrangement determination unit 132 that corrects camera parameters temporarily set according to appropriate parallax, and a plurality of cameras when camera parameters are determined. An origin moving unit 134 that performs origin movement processing to set the middle point as the origin, a projection processing unit 138 that performs the above-described projection processing, and conversion processing to a screen coordinate system after projection processing to generate a parallax image A two-dimensional image generation unit 142. In addition, a distortion processing unit 136 that performs spatial distortion conversion (hereinafter also simply referred to as distortion conversion) is provided between the camera temporary arrangement unit 130 and the camera arrangement determination unit 132 in order to reduce parallax in the periphery of the image as necessary. ing. The distortion processing unit 136 reads out a correction map described later from the correction map holding unit 140 and uses it.
[0071]
If it is necessary to adjust the display device for stereoscopic display, a GUI (not shown) for that purpose may be added. With this GUI, processing such as determining the optimum display position by slightly shifting the entire displayed parallax image vertically and horizontally may be performed.
[0072]
The second stereoscopic image processing apparatus 100 in FIG. 12 receives a plurality of parallax images. This is also simply called an input image. The second stereoscopic image processing apparatus 100 reads the appropriate parallax previously acquired by the first stereoscopic image processing apparatus 100, adjusts the parallax of the input image, puts it within the range of the appropriate parallax, and outputs it. In that sense, the second stereoscopic image processing apparatus 100 has an “automatic adjustment” function of parallax. However, in addition to this, when the user wants to change the stereoscopic effect when the stereoscopic display is actually performed, a GUI function is provided, and a “manual adjustment” function for changing the parallax according to the user's instruction is also provided.
[0073]
The parallax of the already generated parallax image cannot normally be changed, but according to the second stereoscopic image processing apparatus 100, the level that can be sufficiently put into practical use by shifting the synthesis position of the viewpoint images constituting the parallax image You can change the stereoscopic effect. The second stereoscopic image processing apparatus 100 exhibits a good stereoscopic effect adjustment function even in a situation where input data cannot be traced back to three-dimensional data. Hereinafter, differences from the first stereoscopic image processing apparatus 100 will be mainly described.
[0074]
The stereoscopic effect adjusting unit 112 is used for manual adjustment. The instruction acquisition unit 122 realizes numerical input such as “+ n” and “−n” on a screen, for example, and the value is specified by the parallax specifying unit 124 as a parallax change amount. There are several possible relations between the numerical value and the specified stereoscopic effect. For example, “+ n” is an instruction to increase the stereoscopic effect, and “−n” is an instruction to decrease, and the larger the n, the larger the change amount to the stereoscopic effect. Also, “+ n” may be an instruction to move the object in the near direction, and “−n” may be an instruction to move the object in the far direction. As another method, the value of n may not be specified, and only the “+” and “−” buttons may be displayed, and the parallax may be changed each time the button is clicked.
[0075]
The second stereoscopic image processing apparatus 100 includes a parallax amount detection unit 150 and a parallax control unit 152. When the input image is a plurality of parallax images, the parallax amount detection unit 150 inspects the header area of the parallax images, and the parallax amount described in the form of the number of pixels, particularly the near maximum parallax pixel number and the far maximum parallax. If there is the number of pixels, this is acquired. If the amount of parallax is not described, the matching unit 158 identifies the amount of parallax by detecting a corresponding point between parallax images using a known method such as block matching. The matching unit 158 may perform processing only on an important region such as the center of the image, or may detect by focusing on the most important number of nearby parallax pixels. The detected amount of parallax is sent to the parallax control unit 152 in the form of the number of pixels.
[0076]
The position shift unit 160 of the parallax control unit 152 shifts the synthesis position of the viewpoint images constituting the parallax image in the horizontal direction so that the parallax amount between the viewpoint images becomes an appropriate parallax. The shift may be performed on any one of the viewpoint images. The position shift unit 160 also has another operation mode. When the user gives an instruction to increase or decrease the parallax via the stereoscopic effect adjusting unit 112, the image shift position is simply changed according to this instruction. In other words, the position shift unit 160 has two functions of an automatic adjustment function for appropriate parallax and a manual adjustment function by the user.
[0077]
The parallax writing unit 164 writes the parallax amount by the number of pixels in one of the header areas of the plurality of viewpoint images constituting the parallax image for the above-described parallax amount detection unit 150 or for another use. The image edge adjustment unit 168 fills in the missing pixels generated at the image edge due to the shift by the position shift unit 160.
[0078]
The third stereoscopic image processing apparatus 100 in FIG. 13 receives an image with depth information as an input. The third stereoscopic image processing apparatus 100 adjusts the depth so that proper parallax is realized. In addition, the above-described “parallax correction” function is provided. The distortion processing unit 174 of the parallax control unit 170 performs distortion conversion according to the correction map stored in the correction map holding unit 176 in the manner described later. The depth information and the image after distortion conversion are input to the two-dimensional image generation unit 178, where a parallax image is generated. The two-dimensional image generation unit 178 is different from the two-dimensional image generation unit 142 of the first stereoscopic image processing apparatus 100, and appropriate parallax is considered here. Since the image with depth information is also two-dimensional as an image, the two-dimensional image generation unit 178 has a function similar to that of the position shift unit 160 of the second stereoscopic image processing apparatus 100 inside, although not shown in the figure. Accordingly, the pixels in the image are shifted in the horizontal direction to generate a stereoscopic effect. At this time, an appropriate parallax is realized by a process described later.
[0079]
The processing operation and the principle of each part of each stereoscopic image processing apparatus 100 in the above configuration are as follows.
FIGS. 14A and 14B show the left eye image 200 and the right eye image 202 respectively displayed in the specific parallax specifying process by the stereoscopic effect adjusting unit 112 of the first stereoscopic image processing apparatus 100. . In each image, five black circles are displayed, and a larger parallax is placed closer to the top, and a larger parallax is placed farther down.
[0080]
FIG. 15 schematically shows a sense of distance sensed by the user 10 when these five black circles are displayed. The user 10 responds that the range of these five senses of distance is “appropriate”, and the instruction acquisition unit 122 acquires this response. In the figure, five black circles with different parallaxes are displayed simultaneously or sequentially, and the user 10 inputs whether or not the parallax is acceptable. On the other hand, in FIG. 16, the display itself is performed with a single black circle, but the parallax is continuously changed, and when the user 10 reaches the limit allowed in each of the far and near positions, a response is made. The response may be performed by using a known technique such as normal key operation, mouse operation, voice input, and the like.
[0081]
In either case of FIGS. 15 and 16, the instruction acquisition unit 122 can acquire the appropriate parallax as a range, and the near parallax and the far parallax limit parallax are determined. The near maximum parallax is a parallax corresponding to the proximity allowed for a point that appears closest to the user, and the far maximum parallax is a parallax corresponding to the distance allowed for a point that appears farthest from the user. However, in general, the near maximum parallax should be taken care of due to a user's physiological problem. Hereinafter, only the near maximum parallax may be referred to as a limit parallax.
[0082]
FIG. 17 shows the principle of actually adjusting the parallax between two viewpoints when a stereoscopically displayed image is taken out from three-dimensional data. First, the critical parallax determined by the user is converted into a prospective angle of the temporarily placed camera. By taking the angle expression, it becomes a general-purpose expression independent of the hardware of the display device. As shown in the figure, the limit parallax between the near and far positions can be expressed as M and N in terms of the number of pixels, and the angle of view θ of the camera corresponds to the number of horizontal pixels L on the display screen. The near maximum prospective angle φ and the far maximum prospective angle ψ, which are angles, are represented by θ, M, N, and L.
[0083]
tan (φ / 2) = M tan (θ / 2) / L
tan (ψ / 2) = Ntan (θ / 2) / L
Next, this information is applied to the extraction of the two viewpoint images in the three-dimensional space. As shown in FIG. 18, first, a basic expression space T (its depth is also expressed as T) is determined. Here, it is assumed that the basic expression space T is determined from restrictions on the arrangement of objects. Let S be the distance from the front projection plane 30 which is the front surface of the basic expression space T to the camera placement plane, that is, the viewpoint plane 208. T and S can be specified by the user. There are two viewpoints, and D is the distance from the viewpoint plane 208 of these optical axis intersecting planes 210. Let A be the distance between the optical axis crossing plane 210 and the front projection plane 30.
[0084]
Next, let P and Q be the near parallax and far limit parallax in the basic expression space T, respectively.
E: S = P: A
E: S + T = Q: TA
Is established. E is the distance between viewpoints. Now, the point G, which is a pixel with no parallax, is at a position where the optical axes K2 from both cameras intersect on the optical axis intersecting surface 210, and the optical axis intersecting surface 210 is the position of the screen surface. The light rays K1 that produce the near maximum parallax P intersect on the front projection plane 30, and the light rays K3 that produce the far maximum disparity Q intersect on the rear projection plane 32.
[0085]
P and Q use φ and ψ as shown in FIG.
P = 2 (S + A) tan (φ / 2)
Q = 2 (S + A) tan (ψ / 2)
As a result,
E = 2 (S + A) tan (θ / 2) · (SM + SN + TN) / (LT)
A = STM / (SM + SN + TN)
Is obtained. Since S and T are already known, A and E are automatically determined in this way. Therefore, the optical axis crossing distance D and the inter-camera distance E are automatically determined, and the camera parameters are determined. If the camera arrangement determining unit 132 determines the arrangement of the cameras according to these parameters, then the projection processing unit 138 and the two-dimensional image generation unit 142 perform the processing independently on the images from the respective cameras. A parallax image having parallax can be generated and output. As described above, E and A do not include hardware information, and an expression format independent of hardware is realized.
[0086]
Thereafter, appropriate parallax can be automatically realized if a camera is arranged so as to protect A or D and E even when another image is stereoscopically displayed. The entire process from specifying the appropriate parallax to ideal 3D display can be automated, so if this function is provided as a software library, programmers who create content and applications do not need to be aware of 3D display programming. . In addition, when L, M, and N are represented by the number of pixels, L indicates a display range, and thus it can be indicated by L whether the display is a full screen or a part of the screen. L is also a parameter independent of hardware.
[0087]
FIG. 20 shows a four-lens camera arrangement with four cameras 22, 24, 26 and 28. To be precise, the above-mentioned A and E should be determined so as to obtain an appropriate parallax between adjacent cameras, such as between the first camera 22 and the second camera 24. Even if A and E determined between the second camera 24 and the third camera 26 close to each other are used between other cameras, substantially the same effect can be obtained.
[0088]
Note that T is a restriction on the arrangement of objects, but may be determined by a program as the size of a basic three-dimensional space. In this case, the object can always be arranged only in the basic expression space T throughout the program, and for effective display, the object may sometimes be given a parallax so as to intentionally jump out of this space.
[0089]
As another example, T may be determined for the coordinates of the closest object and the most distant object in the three-dimensional space. If this is performed in real time, the basic expression space T is always obtained. Objects can be placed in As an exception to always putting an object in the basic expression space T, a short-time exception can be made if it is operated under the relaxed condition that “the average of the positions for a certain period of time only needs to be in the basic expression space T”. Furthermore, the object that defines the basic expression space T may be limited to a static object. In this case, an exceptional operation in which a dynamic object protrudes from the basic expression space T can be given. As yet another example, conversion may be performed in which a space in which objects are already arranged is reduced to the size of the width T of the basic expression space, or may be combined with the operations described above.
[0090]
Note that if the stereoscopic effect adjustment unit 112 of the first stereoscopic image processing apparatus 100 displays a double image easily as an image displayed to the user, the marginal parallax is determined to be small, and other images are displayed. The appearance frequency of double images can be reduced. As images that are likely to produce double images, those with contrasting colors and brightness between the object and the background are known, and such images can be used at the stage of identifying the critical parallax, that is, at the initial setting. .
[0091]
FIG. 21 to FIG. 36 show processing by the distortion processing unit 136 of the first stereoscopic image processing apparatus 100 and its principle.
FIG. 21 conceptually shows an example of the correction map stored in the correction map holding unit 140 of the first stereoscopic image processing apparatus 100. This map directly corrects the parallax, and the entire map directly corresponds to the parallax image, and the parallax becomes smaller as it goes to the peripheral part. FIG. 22 shows a change in parallax that occurs as a result of camera parameter manipulation by the camera placement determination unit 132 that has received a camera placement by the distortion processing unit 136 according to this correction map. “Normal parallax” is added when viewing the front direction from the left and right viewpoint positions of the two cameras, while “small parallax” is added when viewing the direction greatly deviating from the front. Actually, the camera arrangement determining unit 132 decreases the camera interval as it goes to the periphery.
[0092]
FIG. 23 shows another example in which the camera arrangement determining unit 132 changes the parallax by changing the camera arrangement in accordance with an instruction from the distortion processing unit 136. Here, among the two cameras, only the left camera is moved, and the parallax changes as “normal parallax” → “medium parallax” → “small parallax” toward the periphery of the image. This method has a lower calculation cost than FIG.
[0093]
FIG. 24 shows another example of the correction map. This map also changes the parallax, and the vicinity of the center of the image is not touched with the normal parallax, and the parallax is gradually reduced in the other parallax correction areas. FIG. 25 shows the camera position that the camera placement determination unit 132 changes according to this map. When the camera direction deviates greatly from the front, the position of the left camera is shifted to the right camera for the first time and “small parallax” is added.
[0094]
FIG. 26 conceptually shows another example of the correction map. This map corrects the sense of distance from the viewpoint to the object, and in order to realize this, the camera arrangement determining unit 132 adjusts the optical axis crossing distance of the two cameras. If the optical axis crossing distance is decreased as it goes to the periphery of the image, the object appears to be relatively recessed in the distant direction, so that the object is achieved particularly in the sense of reducing the near parallax. In order to reduce the optical axis crossing distance, the camera arrangement determining unit 132 may change the optical axis direction of the camera, and may change the direction of one of the cameras. FIG. 27 shows the change of the optical axis crossing position or the optical axis crossing surface 210 when a two-dimensional image is generated by the map of FIG. The optical axis crossing surface 210 is closer to the camera as it is closer to the image.
[0095]
FIG. 28 shows another correction map related to the sense of distance, and FIG. 29 shows how the camera arrangement determining unit 132 changes the optical axis crossing plane 210 in accordance with an instruction from the distortion processing unit 136 according to the map of FIG. In this example, the object is placed at the normal position without correction in the center area of the image, and the position of the object is corrected in the peripheral area of the image. For that purpose, in FIG. 29, the optical axis crossing surface 210 does not change near the center of the image, and the optical axis crossing surface 210 approaches the camera after a certain point. In FIG. 29, only the left camera is changed in direction.
[0096]
30A to 30F show another distortion conversion by the distortion processing unit 136. FIG. Unlike the previous examples, instead of changing the camera position, the camera coordinate system directly distorts the 3D space itself. 30A to 30F, the rectangular area is a top view of the original space, and the hatched area is a top view of the space after conversion. For example, the point U in the original space in FIG. This means that this point has been moved away. In FIG. 30 (a), the space is crushed in the direction of the arrow in the depth direction as it goes to the peripheral part, and the distance closer to a certain sense of distance, like the point W in FIG. I feel a sense. As a result, the sense of distance is uniform at the periphery of the image, and there are no objects that are particularly close to the image. This solves the double image problem and makes it easy to adapt to the user's physiology.
[0097]
30 (b), 30 (c), 30 (d), and 30 (e) all show modified examples of conversion that brings the sense of distance close to a constant value at the periphery of the image, and FIG. An example is shown in which all points are converted in the distant direction.
[0098]
FIG. 31 shows the principle for realizing the conversion of FIG. The rectangular parallelepiped space 228 includes a space where the projection processing of the first camera 22 and the second camera 24 is performed. The view volume of the first camera 22 is determined by the angle of view of the camera, the front projection plane 230, and the rear projection plane 232, and that of the second camera 24 is determined by the angle of view of the camera, the front projection plane 234, and the rear projection plane. 236. The distortion processing unit 136 performs distortion conversion on the rectangular parallelepiped space 228. The origin is the center of the rectangular parallelepiped space 228. In the case of the multi-lens type, the conversion principle is the same just by adding more cameras.
[0099]
FIG. 32 shows an example of distortion conversion, which employs reduction conversion in the Z direction. Actually, processing is performed on each object in the space. FIG. 33 expresses this conversion as a parallax correction map. The parallax is normal parallax, the parallax decreases as the absolute value of X increases, and X = ± A indicates no parallax. Here, since the reduction conversion is performed only in the Z direction, the conversion formula is as follows.
[0100]
[Expression 1]

The conversion will be described with reference to FIG. First, consider the range of X ≧ 0 and Z ≧ 0. When the point (X0, Y0, Z0) is moved to the point (X0, Y0, Z1) by the reduction process, the reduction ratio Sz is

It is. The coordinates of C are (X0, Y0, 0) and the coordinates of D are (X0, Y0, B).
[0101]
E is the intersection of the straight line and the plane. If the coordinates are (X0, Y0, Z2), Z2 can be obtained as follows.
[0102]
Z = B−X × B / A (plane)
X = X0, Y = Y0 (straight line)
Z2 = B−X0 × B / A
Therefore,

In general for X,
Sz = 1-X / A
It becomes. If the same calculation is performed for other ranges of X and Z, the following results are obtained and the conversion can be verified.
[0103]
When X ≧ 0, Sz = 1−X / A
When X <0, Sz = 1 + X / A
FIG. 35 shows another example of distortion conversion. Strictly speaking, taking into account that radiography is performed from the camera, reduction processing in the X-axis and Y-axis directions is also combined. Here, conversion is performed with the center of the two cameras as the representative of the camera position. The conversion formula is as follows.
[0104]
[Expression 2]

FIG. 36 verifies this conversion. Again, consider the range of X ≧ 0 and Z ≧ 0. When the point (X0, Y0, Z0) is moved to the point (X1, Y1, Z1) by the reduction process, the reduction ratios Sx, Sy, Sz are

It becomes. Since E is an intersection of a plane and a straight line, Sx, Sy, and Sz can be obtained as described above.
[0105]
If the converted space is represented by a set of planes as described above, the processing changes at the boundary of the tangents between the surfaces, and a sense of incongruity may occur in some cases. In that case, the space may be constituted by a curved surface or a curved surface alone. The calculation is merely changed to one for obtaining the intersection E between the curved surface and the straight line.
[0106]
In the above example, the reduction ratio is the same on the same straight line CD, but weighting may be performed. For example, a weighting function G (L) for the distance L from the camera may be applied to Sx, Sy, and Sz.
[0107]
FIGS. 37 to 40 show processing by the distortion processing unit 174 of the third stereoscopic image processing apparatus 100 and its principle.
FIG. 37 shows a depth map of an image with depth information input to the third stereoscopic image processing apparatus 100. Here, it is assumed that the depth range has values of K1 to K2. Here, the near depth is positive and the far depth is negative.
[0108]
FIG. 38 shows the relationship between the original depth range 240 and the converted depth range 242. The depth approaches a certain value as it goes to the periphery of the image. The distortion processing unit 174 converts the depth map according to this correction. The same applies to the case where parallax is provided in the vertical direction. Since this conversion is also only reduction in the Z direction, it can be expressed by the following equation.
[0109]
[Equation 3]

Sz is classified according to the value of X.
When X ≧ 0, Sz = 1−2X / L
When X <0, Sz = 1 + 2X / L
It becomes. By the above conversion, a new depth map having new elements shown in FIG. 39 is generated.
[0110]
FIG. 40 shows another distortion transformation principle for the depth map. More strictly speaking, the space is observed from the user 10 in a radial manner, so that the reduction processing in the X-axis and Y-axis directions is also combined. Here, the center between the eyes is the observation position. The specific processing is the same as in the case of FIG. The original depth map has only a Z value, but when this calculation is performed, the X value and the Y value are also held. The Z value is converted into a pixel shift amount in the X direction or the Y direction, but the X value and the Y value may be held as offset values for them.
[0111]
In any case, the original image with the depth map converted by the distortion processing unit 174 is input to the two-dimensional image generation unit 178, where a composition process shifted in the horizontal direction so as to obtain an appropriate parallax is performed. Details thereof will be described later.
[0112]
41 to 51 show the processing of the position shift unit 160 of the second stereoscopic image processing apparatus 100 and the two-dimensional image generation unit 178 of the third stereoscopic image processing apparatus 100 that can be grasped as an extension thereof.
FIG. 41 shows the principle of shifting the composite position of two parallax images by the position shift unit 160. As shown in the figure, the positions of the right eye image R and the left eye image L match in the initial state. However, when the left eye image L is relatively shifted to the right as shown in the upper part of the figure, the parallax at the near point increases and the parallax at the far point decreases. Conversely, when the left eye image L is shifted relatively to the left as shown in the lower part of the figure, the parallax at the near point decreases and the parallax at the far point increases.
[0113]
The above is the essence of parallax adjustment by shifting the parallax image. One image may be shifted, or both may be shifted in opposite directions. From this principle, it can be seen that the stereoscopic display method can be applied to all methods using parallax regardless of the glasses method or the glassesless method. Similar processing can be performed for multi-view images and vertical parallax.
[0114]
FIG. 42 shows the shift process at the pixel level. The left eye image 200 and the right eye image 202 both include a first square 250 and a second square 252. The first square 250 has a near parallax, and when the parallax amount is expressed as a positive number, it becomes “6 pixels”. On the other hand, the second square 252 has a distant parallax. When the parallax amount is expressed as a negative number, it becomes “−6 pixels”. Here, let this amount of parallax be F2 and F1, respectively.
[0115]
On the other hand, it is assumed that the appropriate parallax of the display device owned by the user is J1 to J2. The position shift unit 160 shifts the synthesis start position of both images by (J2-F2) pixels. FIG. 43 shows a state after the end of the shift. Now, assuming that F1 = −6 and F2 = 6, and J1 = −5 and J2 = 4, the composition start positions are −2 pixels, that is, It is shifted in the direction in which the whole shifts in the distant direction. As shown in FIG. 43, the final amount of parallax is E1 = −8 and E2 = 4, and at least falls within the limit parallax with respect to the close-up direction. In general, the double image in the close-up direction is more uncomfortable compared to the distant direction, and the subject is often photographed in the close-up direction. It is desirable to keep the directional parallax within limits. A processing example is shown below.
[0116]
1. When the near point is outside the limit parallax and the far point is within the limit parallax, the near point is shifted to the limit parallax point. However, the processing is stopped when the disparity parallax reaches the interocular distance. 2. When the near point is outside the limit parallax and the far point is outside the limit parallax, the near point is shifted to the limit parallax point. However, the processing is stopped when the disparity parallax reaches the interocular distance. 3. If neither the near point nor the far point is within the limit parallax, no processing is performed.
4). If the near point is within the limit parallax and the far point is outside the limit parallax, the far point is shifted to the limit parallax point, but if the near point reaches the limit parallax point during the process, the processing is stopped. .
[0117]
FIG. 44 shows the missing image edge due to the shift of the composite position. Here, the shift amount of the left eye image 200 and the right eye image 202 is one pixel, and a missing portion 260 having a width of one pixel occurs at the right end of the left eye image 200 and the left end of the right eye image 202, respectively. At this time, the image edge adjustment unit 168 compensates the number of horizontal pixels by duplicating the pixel line at the image edge as shown in FIG.
[0118]
As another method, the missing portion 260 may be displayed in a specific color such as black or white, or may be hidden. Further, clipping and adding processing may be performed so as to be the same as the size of the initial image. Also, it may be considered that the size of the initial image is made larger than the actual display size in advance so that the missing portion 260 does not affect the display.
[0119]
FIG. 45 is a flow of manual adjustment of parallax by the second stereoscopic image processing apparatus 100. As shown in the figure, first, left and right images are manually created as parallax images (S10), and distributed through a network or other route (S12). This is received by the second stereoscopic image processing apparatus 100 (S14), and in the example of this figure, first, the images are synthesized and displayed as they are in the normal state without shifting (S16). That is, here, the case where the appropriate parallax has not yet been acquired or the case where the position shift unit 160 is not operated is considered. Subsequently, the user instructs parallax adjustment via the stereoscopic effect adjustment unit 112 for the parallax image displayed stereoscopically, and the position shift unit 160 receives this in the “manual adjustment mode” to adjust the image synthesis position. Are displayed (S18). S10 and S12 are the procedure 270 for the image creator, and the procedure after S14 is the procedure 272 for the second stereoscopic image processing apparatus 100. Although not shown, if this shift amount is recorded in the header and is synthesized by referring to it from the next time, the labor of readjustment can be saved.
[0120]
FIG. 46 shows a flow of automatic adjustment by the second stereoscopic image processing apparatus 100. The left and right image generation (S30) and image distribution (S32), which are image creator procedures 270, are the same as those in FIG. Also, the image reception (S34) is the same in the procedure 272 of the second stereoscopic image processing apparatus 100. Next, the parallax preliminarily added between the parallax images, particularly the maximum parallax, is detected by the matching unit 158 of the parallax amount detection unit 150 (S36), while appropriate parallax, especially critical parallax is acquired from the parallax information holding unit 120. (S38). Thereafter, the position shift unit 160 shifts the composite position of the image so as to satisfy the limit parallax by the above-described processing (S40), and the stereoscopic display is performed through the processing by the parallax writing unit 164, the image edge adjustment unit 168, and the format conversion unit 116. (S42).
[0121]
FIG. 47 shows a flow of still another automatic adjustment by the second stereoscopic image processing apparatus 100. After the left and right images are generated in the image creator procedure 270 (S50), the maximum parallax is detected at this time (S52) and recorded in the header of one of the viewpoint images of the parallax image (S54). This detection may be performed by corresponding point matching. However, when the creator manually generates the parallax image, it is naturally known in the editing process, and therefore this may be recorded. Thereafter, the image is distributed (S56).
[0122]
On the other hand, in the procedure 272 of the second stereoscopic image processing apparatus 100, the image reception (S58) is the same as in FIG. Next, the above-described maximum parallax is read from the header by the header inspection unit 156 of the parallax amount detection unit 150 (S60). On the other hand, the limit parallax is acquired from the parallax information holding unit 120 (S62), and the following processes S64 and S66 are the same as the processes S40 and S42 of FIG. According to this method, there is no need to calculate the maximum parallax. In addition, it is possible to realize an appropriate stereoscopic effect for the entire image. Furthermore, since the shift amount can be recorded in the header, there is no possibility of damaging the original image itself. Although not shown, if the maximum parallax detected in FIG. 46 is recorded in the header, it can be processed in accordance with the procedure shown in FIG.
[0123]
Note that the same processing can be performed even in the multi-view method, and the same processing may be performed on the amount of parallax between adjacent viewpoint images. However, in practice, the shift amount of the synthesis position may be determined by regarding the maximum parallax among the parallaxes among the plurality of viewpoint images as the “maximum parallax” between all the viewpoint images.
[0124]
The header information may be in at least one of the multi-viewpoint images. However, when the multi-viewpoint image is combined into one image, the header of the image may be used.
[0125]
In some cases, images that have already been combined may be distributed. In such a case, the images are separated by inverse transform processing once, and the combined position shift amount is calculated and recombined, or the result is the same pixel. This sort processing may be performed.
[0126]
48 to 51 show processing for shifting the composite position for the image with depth information. This is performed by the two-dimensional image generation unit 178 of the third stereoscopic image processing apparatus 100.
FIG. 48 and FIG. 49 are a plane image 204 and a depth map that constitute an image with depth information, respectively. Here, the near depth is represented as positive and the far depth is represented as negative. There are a first quadrangle 250, a second quadrangle 252 and a third quadrangle 254 as objects, the first quadrangle 250 is depth “4”, the second quadrangle 252 is “2”, and the third quadrangle 254 is “−4”. . The first square 250 is the nearest placement point, the second square 252 is the intermediate near placement point, and the third square 254 is the farthest placement point.
[0127]
Based on the original planar image 204, the two-dimensional image generation unit 178 first performs a process of shifting each pixel by the value of the depth map to generate the other viewpoint image. If the reference is the left eye image, the original planar image 204 becomes the left eye image as it is. The first square 250 is shifted to the left by 4 pixels, the second square 252 is shifted to the left by 2 pixels, and the third square 254 is shifted to the right by 4 pixels, and the right eye image 202 is created as shown in FIG. The image edge adjustment unit 168 fills the pixel information missing portion 260 due to the movement of the object with the neighboring pixels determined to be the background whose parallax is “0”.
[0128]
Subsequently, the two-dimensional image generation unit 178 calculates a depth that satisfies the appropriate parallax. If the depth range is K1 to K2 and the depth value of each pixel is Gxy, the depth map is changed from Hxy to Gxy in FIG. Further, it is assumed that the appropriate parallax of the display device owned by the user is J1 to J2. In this case, in the depth map, the depth value G of each pixel is converted as follows, and a new depth value Fxy is obtained.
[0129]
Fxy = J1 + (Gxy−K1) × (J2−J1) / (K2−K1)
In the above example, if K1 = −4, K2 = 4, and J1 = −3 and J2 = 2, the depth map of FIG. 49 is converted to the depth map of FIG. The That is, “4” is converted to “2”, “2” is converted to “1”, and “−4” is converted to “−3”. Intermediate values between K1 and K2 are converted between J1 and J2. For example, the second square 252 has Gxy = 2 and Fxy = 0.75. When Fxy does not become an integer, rounding off or processing that reduces the near parallax may be performed.
[0130]
Note that the above conversion formula is an example of linear conversion, but a weighting function F (Gxy) for Gxy is applied, and various other nonlinear conversions are also conceivable. Also, it is possible to newly generate left and right images from the original planar image 204 by shifting the objects in opposite directions. In the case of a multi-view type, a similar process may be performed a plurality of times to generate a multi-viewpoint image.
[0131]
The above is the configuration and operation of the stereoscopic image processing apparatus 100 according to the embodiment. Although the stereoscopic image processing apparatus 100 has been described as an apparatus, this may be a combination of hardware and software, or may be configured only by software. In that case, it is convenient if an arbitrary part of the stereoscopic image processing apparatus 100 is made into a library and can be called from various programs. The programmer can skip programming that requires knowledge of 3D display. For the user, regardless of the software and contents, the operation related to stereoscopic display, that is, the GUI is common, and the set information can be shared by other software, so that the trouble of resetting can be saved.
[0132]
Note that it is useful not only for the processing related to stereoscopic display but also for sharing information among a plurality of programs. Various programs can determine the state of the image with reference to the information. An example of information to be shared is information acquired by the information acquisition unit 118 of the stereoscopic image processing apparatus 100 described above. This information may be held in a recording unit or correction map holding unit 140 (not shown).
[0133]
52 to 54 show an example in which the above-described stereoscopic image processing apparatus 100 is used as a library. FIG. 52 shows the usage of the stereoscopic display library 300. The stereoscopic display library 300 is referred to by calling functions from a plurality of programs A302, B304, C306, and the like. The parameter file 318 stores the appropriate parallax of the user in addition to the above information. The stereoscopic display library 300 is used by a plurality of apparatuses A 312, B 314, C 316, etc. via an API (Application Program Interface) 310.
[0134]
Examples of the program A302 and the like include a game, a so-called Web3D three-dimensional application, a three-dimensional desktop screen, a three-dimensional map, a two-dimensional image parallax image viewer, and a depth information-added viewer. Of course, some of the games use different coordinates, but the 3D display library 300 can handle this.
[0135]
On the other hand, as an example of the device A312 or the like, any stereoscopic display device using parallax, such as a binocular or multi-lens parallax barrier method, a shutter glasses method, or a polarized glasses method.
[0136]
FIG. 53 shows an example in which the stereoscopic display library 300 is incorporated in the three-dimensional data software 402. The three-dimensional data software 402 includes a program main body 404, a stereoscopic display library 300 for realizing appropriate parallax therefor, and a photographing instruction processing unit 406. The program body 404 communicates with the user via the user interface 410. The photographing instruction processing unit 406 virtually photographs a predetermined scene during the operation of the program body 404 in accordance with a user instruction. The captured image is recorded in the image recording device 412. Also output to the stereoscopic display device 408.
[0137]
For example, assume that the three-dimensional data software 402 is game software. In that case, the user can execute the game while experiencing an appropriate stereoscopic effect by the stereoscopic display library 300 during the game. When the user wants to keep a record during the game, for example, when the player wins a complete battle game, an instruction is given to the shooting instruction processing unit 406 via the user interface 410 and the scene is recorded. At this time, using the stereoscopic display library 300, a parallax image is generated so as to have an appropriate parallax when played back later on the stereoscopic display device 408, and this is recorded in an electronic album or the like of the image recording device 412. Note that recording is performed with a two-dimensional image called a parallax image, so that the three-dimensional data itself of the program body 404 does not flow out, and it is possible to consider copyright protection.
[0138]
FIG. 54 shows an example in which the three-dimensional data software 402 of FIG. 53 is incorporated into a network-based system 430.
The game machine 432 is connected to the server 436 and the user terminal 434 via a network (not shown). The game machine 432 is for an arcade game, and includes a communication unit 442, three-dimensional data software 402, and a stereoscopic display device 440 that locally displays the game. The three-dimensional data software 402 is the one shown in FIG. The parallax image displayed on the stereoscopic display device 440 from the three-dimensional data software 402 is optimally set in advance for the stereoscopic display device 440. The parallax adjustment by the three-dimensional data software 402 is used when an image is transmitted to the user via the communication unit 442 as described later. The display device used here only needs to have a function of adjusting a parallax and generating a stereoscopic image, and does not necessarily have to be a device capable of stereoscopic display.
[0139]
The user terminal 434 includes a communication unit 454, a viewer program 452 for viewing a stereoscopic image, and a stereoscopic display device 450 of any size and type for locally displaying the stereoscopic image. The viewer program 452 includes the stereoscopic image processing apparatus 100.
[0140]
The server 436 associates the communication unit 460, an image holding unit 462 that records images virtually taken by the user in connection with the game, the user's appropriate parallax information, the user's email address, and other personal information with the user. A user information holding unit 464 for recording. The server 436 functions as an official game site, for example, and records a scene that the user likes during the game, a moving image or a still image of a famous game. The stereoscopic display can be either a moving image or a still image.
[0141]
An example of image capturing in the above configuration is performed as follows. The user performs stereoscopic display on the stereoscopic display device 450 of the user terminal 434 in advance, acquires appropriate parallax based on the function of the stereoscopic image processing device 100, notifies the server 436 of this via the communication unit 454, and transmits user information. It is stored in the holding unit 464. This appropriate parallax is a general-purpose description not related to the hardware of the stereoscopic display device 450 owned by the user.
[0142]
The user plays a game with the game machine 432 at an arbitrary timing. In the meantime, the stereoscopic display device 440 performs stereoscopic display using the parallax initially set or the parallax manually adjusted by the user. When the user wishes to record an image during game play or replay, the 3D display library 300 built in the 3D data software 402 of the game machine 432 is stored in the server 436 via the two communication units 442 and 460. A parallax image relating to an image virtually acquired by the image holding unit 462 via the two communication units 442 and 460 is acquired again by acquiring the appropriate parallax of the user from the user information holding unit 464 and generating a parallax image accordingly. Is stored. If the user returns to the home and then downloads the parallax image to the user terminal 434, the stereoscopic display can be performed with a desired stereoscopic effect. Even in this case, the parallax can be manually adjusted by the stereoscopic image processing apparatus 100 included in the viewer program 452.
[0143]
As described above, according to this application example, the programming related to the stereoscopic effect that must be set for each hardware of the display device and for each user is concentrated in the stereoscopic image processing device 100 and the stereoscopic display library 300. Programmers don't have to worry about any complex requirements for stereoscopic display. This applies not only to game software but also to arbitrary software that uses stereoscopic display, and it eliminates restrictions on the development of content and applications that use stereoscopic display. Therefore, the spread of these can be dramatically promoted.
[0144]
Especially in the case of games and other applications that originally have 3D CG data, it has been difficult to code accurate 3D display in the past. In many cases, it was not used for stereoscopic display. According to the stereoscopic image processing apparatus 100 or the stereoscopic display library 300 according to the embodiment, such adverse effects can be removed, and the stereoscopic display application can be enhanced.
[0145]
In FIG. 54, the user's appropriate parallax is registered in the server 436, but the user may bring an IC card or the like on which the information is recorded to use the game machine 432. You may record the score about this game and the picture you like on this card.
[0146]
The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there. Here are some examples.
[0147]
The first stereoscopic image processing apparatus 100 can perform processing with high accuracy by inputting three-dimensional data. However, it is also possible to drop the three-dimensional data into an image with depth information and generate a parallax image using the third stereoscopic image processing apparatus 100. In some cases, it may be less computationally expensive. Similarly, when inputting a plurality of viewpoint images, it is also possible to create a depth map by using matching point matching with high accuracy. A parallax image may be generated using the image processing apparatus 100.
[0148]
In the first stereoscopic image processing apparatus 100, the temporary camera arrangement unit 130 is configured as the stereoscopic image processing apparatus 100, but this may be preprocessing of the stereoscopic image processing apparatus 100. This is because processing up to the temporary placement of the camera can be performed regardless of the appropriate parallax. Similarly, arbitrary processing units constituting the first, second, and third stereoscopic image processing apparatuses 100 can be taken out of the stereoscopic image processing apparatus 100, and the degree of freedom of the configuration of the stereoscopic image processing apparatus 100 is allowed. Will be understood by those skilled in the art.
[0149]
In the embodiment, the case where the parallax control is performed in the horizontal direction has been described, but the same processing can also be performed in the vertical direction.
[0150]
During the operation of the stereoscopic display library 300 and the stereoscopic image processing apparatus 100, a unit for enlarging the character data may be provided. For example, in the case of a parallax image with two horizontal viewpoints, the horizontal resolution of an image visible to the user is halved. As a result, the readability of the characters can be reduced, and therefore a process of extending the characters twice in the horizontal direction is effective. When there is parallax in the vertical direction, it is also useful to extend the character in the vertical direction as well.
[0151]
During the operation of the stereoscopic display library 300 and the stereoscopic image processing apparatus 100, an “in-operation display unit” that puts characters and marks such as “3D” in the displayed image may be provided. In this case, the user can know whether or not the image can adjust the parallax.
[0152]
A stereoscopic display / normal display switching unit may be provided. This unit includes a GUI. When the user clicks a predetermined button, the display is switched from the stereoscopic display to the normal two-dimensional display, and vice versa.
[0153]
The information acquisition unit 118 does not necessarily acquire information by user input, but may include information that can be automatically acquired by a function such as plug and play.
[0154]
In the embodiment, a method of deriving E and A is used. However, a method of deriving other parameters by fixing these may be used, and designation of variables is free.
[0155]
【The invention's effect】
The present invention has the following effects.
1. It is possible to generate or display a stereoscopic image that is easily adapted to human physiology.
2. Even if the display target image changes, a stereoscopic image appropriate for the user can be generated or displayed.
3. The stereoscopic effect of the 3D display can be adjusted with a simple operation.
4). It is possible to reduce the burden on the programmer when creating contents or applications capable of appropriate stereoscopic display.
5. The user's effort to optimize the stereoscopic display is reduced.
6). The same applies to devices that cannot easily be plug-and-played in principle, such as a parallax barrier attached later, that can easily achieve stereoscopic adjustment and head tracking information that are not subject to the plug-and-play function.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a positional relationship among a user, a screen, and a playback object 14 that are ideally viewed stereoscopically.
FIG. 2 is a diagram illustrating an example of an imaging system that realizes the state of FIG. 1;
FIG. 3 is a diagram illustrating another example of an imaging system that realizes the state of FIG. 1;
4 is a diagram illustrating another example of an imaging system that realizes the state of FIG. 1; FIG.
FIG. 5 is a diagram illustrating a model coordinate system used in the first stereoscopic image processing apparatus.
FIG. 6 is a diagram showing a world coordinate system used for the first stereoscopic image processing apparatus.
FIG. 7 is a diagram illustrating a camera coordinate system used for the first stereoscopic image processing apparatus.
FIG. 8 is a diagram showing a view volume used for the first stereoscopic image processing apparatus.
9 is a diagram showing a coordinate system after perspective transformation of the volume of FIG. 8; FIG.
FIG. 10 is a diagram illustrating a screen coordinate system used in the first stereoscopic image processing apparatus.
FIG. 11 is a configuration diagram of a first stereoscopic image processing apparatus.
FIG. 12 is a configuration diagram of a second stereoscopic image processing apparatus.
FIG. 13 is a configuration diagram of a third stereoscopic image processing apparatus.
FIGS. 14A and 14B are diagrams illustrating a left-eye image and a right-eye image displayed by the stereoscopic effect adjustment unit of the first stereoscopic image processing apparatus, respectively.
FIG. 15 is a diagram illustrating a plurality of objects having different parallaxes displayed by the stereoscopic effect adjusting unit of the first stereoscopic image processing apparatus.
FIG. 16 is a diagram illustrating an object whose parallax changes, which is displayed by the stereoscopic effect adjusting unit of the first stereoscopic image processing apparatus.
FIG. 17 is a diagram illustrating a relationship between a camera angle of view, an image size, and parallax when appropriate parallax is realized.
18 is a diagram illustrating a positional relationship of the imaging system that realizes the state of FIG. 17;
FIG. 19 is a diagram illustrating a positional relationship of an imaging system that realizes the state of FIG.
FIG. 20 is a diagram illustrating a camera arrangement when a multi-viewpoint image is generated with appropriate parallax.
FIG. 21 is a diagram illustrating a parallax correction map used by the distortion processing unit of the first stereoscopic image processing apparatus.
22 is a diagram illustrating a camera viewpoint when generating a parallax image according to the parallax correction map of FIG.
FIG. 23 is a diagram showing another camera viewpoint when generating a parallax image in accordance with the parallax correction map of FIG.
FIG. 24 is a diagram illustrating a parallax correction map used by the distortion processing unit of the first stereoscopic image processing apparatus.
FIG. 25 is a diagram showing a camera viewpoint when generating a parallax image according to the parallax correction map of FIG. 24;
FIG. 26 is a diagram illustrating a sense of distance correction map used by the distortion processing unit of the first stereoscopic image processing apparatus.
FIG. 27 is a diagram showing a camera viewpoint when generating a parallax image according to the sense of distance correction map of FIG.
FIG. 28 is a diagram showing another sense of distance correction map used by the distortion processing unit of the first stereoscopic image processing apparatus.
29 is a diagram illustrating a camera viewpoint when generating a parallax image according to the sense of distance correction map of FIG.
30 (a), FIG. 30 (b), FIG. 30 (c), FIG. 30 (d), FIG. 30 (e), and FIG. 30 (f) are all first stereoscopic image processing apparatuses. It is a top view of the parallax distribution obtained as a result of processing the distortion processing unit in the three-dimensional space.
FIG. 31 is a diagram illustrating a principle of processing by a distortion processing unit of the first stereoscopic image processing apparatus.
32 is a diagram specifically illustrating the process of FIG. 31. FIG.
FIG. 33 is a diagram specifically showing the processing of FIG. 31;
34 is a diagram specifically showing the processing of FIG. 31. FIG.
FIG. 35 is a diagram illustrating another example of processing by the distortion processing unit of the first stereoscopic image processing apparatus.
36 is a diagram specifically illustrating the processing of FIG. 35. FIG.
FIG. 37 is a diagram showing a depth map.
FIG. 38 is a diagram illustrating an example of processing by a distortion processing unit of the third stereoscopic image processing apparatus.
FIG. 39 is a diagram illustrating a depth map generated by processing by the distortion processing unit of the third stereoscopic image processing apparatus.
FIG. 40 is a diagram illustrating another example of processing by the distortion processing unit of the third stereoscopic image processing apparatus.
FIG. 41 is a diagram illustrating an example of processing by a two-dimensional image generation unit of the second stereoscopic image processing apparatus.
FIG. 42 is a diagram illustrating an example of a parallax image.
FIG. 43 is a diagram illustrating a parallax image in which the synthesis position is shifted by the two-dimensional image generation unit of the second stereoscopic image processing apparatus.
FIG. 44 is a diagram illustrating processing of an image edge adjustment unit of the second stereoscopic image processing apparatus.
FIG. 45 is a diagram illustrating processing of the second stereoscopic image processing apparatus.
FIG. 46 is a diagram illustrating another process of the second stereoscopic image processing apparatus.
FIG. 47 is a diagram illustrating another process of the second stereoscopic image processing apparatus.
FIG. 48 is a diagram illustrating a planar image to which a depth map is added.
FIG. 49 is a diagram showing a depth map.
FIG. 50 is a diagram illustrating a manner in which a parallax image is generated based on a depth map by a two-dimensional image generation unit of a second stereoscopic image processing apparatus.
FIG. 51 is a diagram illustrating a depth map corrected by the two-dimensional image generation unit of the second stereoscopic image processing apparatus.
FIG. 52 is a diagram illustrating a manner in which the stereoscopic image processing apparatus according to the embodiment is used as a library.
FIG. 53 is a configuration diagram in which a stereoscopic display library is incorporated into three-dimensional data software.
FIG. 54 is a diagram illustrating a manner in which a stereoscopic display library is used in a network-based system.
[Explanation of symbols]
10 users, 12 screens, 14 playback objects, 20 real objects, 22, 24, 26, 28 cameras, 30 front projection planes, 32 rear projection planes, 100 stereoscopic image processing devices, 112 stereoscopic effect adjustment units, 114, 152, 170 parallax control unit, 116 format conversion unit, 118 information acquisition unit, 122 instruction acquisition unit, 124 parallax identification unit, 132 camera arrangement determination unit, 136,174 distortion processing unit, 140,176 correction map holding unit, 142 two-dimensional image Generation unit, 150 parallax amount detection unit, 156 header inspection unit, 158 matching unit, 160 position shift unit, 164 parallax writing unit, 168 image edge adjustment unit, 178 two-dimensional image generation unit, 210 optical axis crossing plane, 300 three-dimensional Display library, 402 3D data software, 406 Shooting instruction processing unit 430 network utilization type of system, 432 game machine, 434 user terminal, 436 servers, 452 viewer program.

Claims (8)

A parallax control unit that corrects parallax between a plurality of viewpoint images for displaying a stereoscopic image;
A map holding unit that holds a correction map that the parallax control unit should refer to in the process,
In the correction map, the correction amount of the parallax is described so as to change corresponding to the position in the viewpoint image ,
The parallax control unit reduces the parallax in a peripheral part of the plurality of viewpoint images, or changes the parallax so that an object in the peripheral part is sensed farther from the user. . The parallax control unit, three-dimensional image processing apparatus according to claim 1, wherein the changing the parallax By selectively processing performed on any of the plurality of viewpoint images. The plurality of viewpoint images are generated starting from three-dimensional data,
The parallax control unit, when generating the plurality of viewpoint images, a stereoscopic image processing apparatus and controls the camera parameter according to claim 1 or 2, wherein the changing the parallax. The plurality of viewpoint images are generated starting from three-dimensional data,
The parallax control unit, the plurality of carrying generated viewpoint image, three-dimensional image processing apparatus according to claim 1, wherein the changing the parallax by distorting the three-dimensional space itself. Wherein the plurality of viewpoint images, which are generated from the planar image depth information is given, the parallax control unit according to claim 1, characterized in that changing the parallax by manipulating the depth information The three-dimensional image processing apparatus described. An image acquisition step of acquiring a plurality of viewpoint images for displaying a stereoscopic image;
A parallax correction step that changes the parallax between the acquired viewpoint images so that the correction amount changes corresponding to the position in the viewpoint images, and
In the parallax correction step, parallax correction is performed so that parallax is reduced in the peripheral portions of the plurality of viewpoint images, or parallax correction is performed so that peripheral objects are perceived further away from the user. A stereoscopic image processing method. 7. The stereoscopic image processing method according to claim 6 , wherein each of the steps is implemented as a function of a stereoscopic display library, and the function of the library can be called as a function from a plurality of programs. An image acquisition step of acquiring a plurality of viewpoint images for displaying a stereoscopic image;
A computer program for causing a computer to execute a parallax correction step for changing parallax between the acquired viewpoint images so that a correction amount thereof changes corresponding to a position in the viewpoint images ,
In the parallax correction step, parallax correction is performed such that parallax is reduced in the peripheral portions of the plurality of viewpoint images, or parallax correction is performed so that objects in the peripheral portions are sensed farther from the user. Computer program.

Images