JP3643511B2

JP3643511B2 - 3D image processing method, 3D modeling method, and recording medium recording 3D image processing program

Info

Publication number: JP3643511B2
Application number: JP2000050999A
Authority: JP
Inventors: 裕人松岡; 昭彦宮崎; 仁志北澤
Original assignee: Nippon Telegraph and Telephone Corp
Current assignee: Nippon Telegraph and Telephone Corp
Priority date: 2000-02-28
Filing date: 2000-02-28
Publication date: 2005-04-27
Anticipated expiration: 2020-02-28
Also published as: JP2001243497A

Description

【０００１】
【発明の属する技術分野】
本発明は，表示対象物体の３次元モデルデータを用い，コンピュータによって任意の視点方向から見た物体の画像を表示する３次元画像処理方法，３次元モデリング方法および３次元画像処理プログラムを記録した記録媒体に関し，特に表示対象の撮影画像をもとに処理を行うイメージベースモデリングならびにイメージベースレンダリングの技術に関する。
【０００２】
【従来の技術】
近年，インターネットの普及や画像処理能力の向上に伴い，商品等のプレゼンテーションに３次元モデルデータが用いられるようになってきた。３次元モデルデータを商品紹介に用いることにより，顧客は，実際に目の前に商品がなくても，パーソナルコンピュータ等の画面上で，商品を好きな角度／方向から見ることができる。しかし，商品の３次元モデルデータを生成するには，その形状データと表面の画像データが必要となる。
【０００３】
一般的な３次元モデルとして，ポリゴンモデルがある。図２２にそのポリゴンモデルの一例を示す。このモデルは，表示対象を複数のポリゴンで表し，各ポリゴンに色彩や光沢等の情報を与える。そのデータを透視投影することにより，所望の角度・距離から見たときの２次元画像を得ることができる。この方法では，正確なポリゴンモデルを生成することにより，任意の角度・距離からの観察画像を正確に得ることができる。
【０００４】
しかしながら，このポリゴンモデルを得るには，レンジファインダ等の専用の３次元モデリング装置が必要となる。また，表示対象に曲面がある場合，ポリゴン数が膨大となり，そのデータ量・処理量が共に膨大となるという問題がある。また，微妙な色彩が表現できないためリアリティが低く，実写画像に比べるとかなり見劣りするという問題がある。
【０００５】
リアリティを増すために，実写画像を複数の方向から撮影し，その画像をポリゴン表面に貼り付ける方法がある。しかしながら，この方法は，表示対象の表面がプラスチックや金属等の鏡面反射が大きいものである場合に，表面に映り込んだ像が実写画像ごとに異なるため，これらの画像をポリゴン表面に貼り付けると，逆にポリゴンの境界が見苦しくなるという問題がある
一方，実写画像を用いたモデルとして，複数の方向から実写画像を撮影し，表示したい角度に最も近い画像を表示する方法がある。この方法では，実写画像を用いているため，非常にリアリティの高い画像を得ることができ，例えば米国アップルコンピュータ社が開発したＱｕｉｃｋＴｉｍｅＶＲ等で用いられている。しかしながら，任意の角度からの画像を得るには，画像数が膨大となり，また，実写画像の撮影距離と所望の視点距離とが大きく異なる場合，表示対象の側面等の見え方が異なるという問題がある。
【０００６】
これを解決する方法として，２枚の実写画像から，その間の画像を補間して描画する方法が提案されており，ＩＧＧＲＡＰＨ９６で公表された“ＶｉｅｗＭｏｒｐｈｉｎｇ”はその代表的な例である。この方法は，２枚の画像間で共通である対応点をあらかじめ求めておき，対応点をもとに２枚の画像を変形してそれらを合成し，２枚の画像間の視点から見たときの画像を合成する。この方法では，正確な対応点が得られると，２枚の画像間の任意の視点からの画像を，かなり正確に描画することができ，必要となる実写画像の枚数を減らすことができる。
【０００７】
しかしながら，この方法では，２つの実写画像間の対応点が必須であるが，これを求める有効なアルゴリズムはいまだ実現されておらず，現在のところ手動で対応点の指定を行っているので，３次元モデルの自動生成には難がある。また，本方法では計算量がかなり大きいため，画像合成に数〜数十分程度かかるのが普通であり，リアルタイムで表示することができない。
【０００８】
【発明が解決しようとする課題】
前述したように，従来のモデリング手法であるポリゴンモデルでは，リアリティの高い画像を生成することができず，また，モデルデータの作成が難しいという問題がある。また，実写画像を用いたモデリング手法では，リアリティは増すが，データ量が膨大になったり，対応点を手動で指定したりしなければならなかったり，その処理時間が膨大になったりするという問題がある。
【０００９】
【課題を解決するための手段】
本発明は，複数の視点から撮影した実写画像と表示対象物体の概略形状を３次元モデルとする。そして，所望の視点からの表示画像を合成する場合に，まず，所望の視点方向と撮影方向とが最も近い実写画像を選ぶ。その実写画像を概略形状の表面に，実写画像の撮影位置を視点として投射する。そして，投射された概略形状を，所望の視点からの方向からスクリーン上に透視投影して，表示画像を得る。
【００１０】
本発明では，所望の視点方向と撮影方向が近い１枚の画像を用い，それを概略形状をもとに変形した画像を表示することを基本とする。このため，実写画像間で，鏡面反射等で写り込んだ画像に差が生じていても，それらに影響されることなく，リアリティの高い画像を得ることができる。また，実写画像間の対応点を用いていないため，対応点を求める必要もない。
【００１１】
さらに，実写画像を変形した画像で実写画像間の視点からの表示画像を補っているので，必要となる実写画像の枚数を低く抑えることができる。また，所望の視点方向と撮影方向が近い実写画像を変形させて表示させているので，実写画像の変形は比較的小さく，このため，概略形状の精度を高くしなくても，十分にリアリティの高い画像が得られるという特徴がある。
【００１２】
すなわち，概略形状を生成するために，高精度の３次元モデリング装置を必要とせず，概略形状の生成も簡易に行うことができる。また，概略形状のポリゴン数はかなり少なく抑えられるので，本方法における画像合成時間は短く，リアルタイムで描画することが可能である。
【００１３】
【発明の実施の形態】
以下，本発明の実施の形態について詳細に説明する。
【００１４】
本発明での，表示対象物体Ｏの３次元モデルデータＭＤは，物体の全周囲をカバーする異なる方向から撮影したｎ枚の撮影画像Ｐｋ（１≦ｋ≦ｎ）と，その撮影情報Ｉｋ（１≦ｋ≦ｎ）と，表示対象物体Ｏの概略を表すポリゴンデータ（概略形状Ｓ）からなる。
【００１５】
図１に３次元モデルデータＭＤの例を示す。この図において，（Ａ）は撮影画像の例を，（Ｂ）は撮影情報の例を，（Ｃ）は概略形状の例を示している。
【００１６】
撮影画像Ｐｋは，デジタルカメラ等で撮影する。撮影情報Ｉｋは，カメラと表示対象物体Ｏ（被写体）の距離ｄ，表示対象物体Ｏのある撮影方向を基準としてその基準方向との差を表す撮影方向を示す回転行列Ｒ，撮影画像の１ピクセルの長さを表す解像度ｒからなる。これらの値は厳密な値を必要としないので，定規等での計測で十分である。
【００１７】
概略形状Ｓは，表示対象物体Ｏの大まかな形状でかまわないので，撮影画像から表示対象物体Ｏの大まかな縁取りを行い，Shape from Silhouette 等のアルゴリズムを用いて求める。厳密に計算する必要はなく，生成するポリゴン数も少なくてよいので，一般的なポリゴン生成に比べ，簡易に生成することができる。なお，画像から形状を求める手法（Shape from Silhouette のアルゴリズムを含む）の参考文献としては，例えば「“From image to models (and beyond): a personal retrospective", SIGGRAPH Course Note, 1999. 」がある。
【００１８】
また，概略形状Ｓは，定規等による測定で大まかに生成する方法でも生成可能である。さらに，各撮影画像の背景を切り取っておくことにより，表示対象を表示するとき，表示対象よりも奥にある物体等と重なった場合でも，正確に表示することができる。
【００１９】
本発明での上記３次元モデルデータＭＤを任意の方向から見たときの第１の表示方法について述べる。図２に，第１の表示方法による処理の流れを示す。
【００２０】
まず，ステップＳ１１で，所望の視点Ｖの方向と３次元モデルデータＭＤの各撮影画像Ｐ１〜Ｐｎの撮影方向とを比較し，最も近い撮影画像Ｐｋ（１≦ｋ≦ｎ）を選ぶ。この比較は，視点Ｖの方向ベクトルと撮影方向のベクトルの内積を求め，その値が最も大きいもの（各方向ベクトルを単位ベクトルで表したとき，内積が１に最も近いもの）を選ぶことにより実現できる。
【００２１】
ステップＳ１２では，この撮影画像Ｐｋを，撮影位置を視点とした方向から見た概略形状Ｓの表面に投射する。この様子を，図３に示す。実際に計算機上では，撮影位置を視点，概略形状ＳをスクリーンＳＣ上に透視投影して，撮影画像ＰｋをスクリーンＳＣに重ねることにより，概略形状Ｓの各表面に投射される画像を得ることができる。
【００２２】
ステップＳ１３では，図４に示すように，この撮影画像Ｐｋを投射した概略形状Ｓを，所望の視点Ｖの方向からスクリーンＳＣ上に透視投影して，表示画像を得る。この操作を行うことにより，３次元モデルデータＭＤの所望の視点から見たときの投影画像を生成することができる。
【００２３】
なお，透視投影は，人間が眼で見る状態と同じ画像を作るため，視点に近い物体ほど大きく，遠い物体ほど小さくなるように投影する投影方法の一つであるが，周知の技術であるため，これについてのさらに詳しい説明は省略する。
【００２４】
図５に撮影画像および第１の表示方法による合成画像例を示す。図５（Ｂ）は撮影画像そのものであり，図５（Ａ）は視点を左１５度ずらして合成した画像，図５（Ｃ）は視点を右１５度ずらして合成した画像である。図５の（Ａ），（Ｃ）は，図１（Ｃ）に示す単純な概略形状Ｓで合成を行っているが，本アルゴリズムを用いることにより，任意の方向から見たときの表示画像を生成できることがわかる。
【００２５】
以上の第１の表示方法では，概略形状Ｓに対して，撮影画像Ｐｋの撮影位置を視点とした方向からの実写の投影と，所望の視点Ｖからの透視投影のため，２回の座標計算を行う必要がある。撮影情報Ｉｋに，概略形状Ｓの各項点が対応する撮影画像上のピクセル座標からなる投影点情報（以下，ピクセル座標情報という）ＩＶ，概略形状Ｓの多角形に対応する撮影画像上の多角形を表す多角形情報ＩＰを加えた３次元モデルデータＭＤ２を用いることにより，この座標計算を１回に減らすことができる。
【００２６】
図６に３次元モデルデータＭＤ２における撮影情報に追加されるピクセル座標情報ＩＶと多角形情報ＩＰの例を示す。他の撮影情報については，図１（Ｃ）に示す３次元モデルデータＭＤと同様である。ピクセル座標情報ＩＶは，２次元の座標データであり，撮影位置を視点，撮影画像ＰｋをスクリーンＳＣとして，概略形状Ｓの各頂点を透視投影することにより，その座標を求める。さらに，多角形情報ＩＰは，頂点のインデックス列のリストであり，概略形状Ｓの頂点の接続情報をもとに生成する。
【００２７】
次に，この３次元モデルデータＭＤ２を用いた第２の表示方法について説明する。図７に，第２の表示方法による処理の流れを示す。
【００２８】
第２の表示方法では，まず，所望の視点Ｖの方向と撮影方向とが最も近い撮影画像Ｐｋ（１≦ｋ≦ｎ）を選ぶ（ステップＳ２１）。次に，図８に示すように，視点Ｖからの概略形状Ｓの各頂点のスクリーン座標ＶＳを，透視投影により求める（ステップＳ２２）。
【００２９】
そして，多角形情報ＩＰから任意の多角形Ａを一つ選び（ステップＳ２３），多角形Ａの各頂点の座標をピクセル座標情報ＩＶから得て，撮影画像Ｐｋ内の多角形Ｂの領域を取得する（ステップＳ２４）。ステップＳ２４では，例えば図９（Ａ）に示す実写画像から，図９（Ｂ）に示す多角形Ｂの領域を取得することになる。
【００３０】
次に，多角形Ａの各頂点座標をスクリーン座標ＶＳから得て，図９（Ｃ）に示すようなスクリーンＳＣ上の多角形Ｃを得る（ステップＳ２５）。そして，多角形Ｂの領域をスクリーンＳＣ上の多角形Ｃの形状に合うように変形する（ステップＳ２６）。多角形Ｂの領域を変形した様子を図９（Ｄ）に示す。これをスクリーンＳＣに貼り付けて描画する（ステップＳ２７）。多角形情報ＩＰの全ての多角形に対して，多角形Ａと同様に処理してスクリーンＳＣ上に描画することにより（ステップＳ２８），図９（Ｅ）に示すように，第１の表示方法と同様の表示画像を得ることができる。
【００３１】
上記第１または第２の表示方法では，スクリーンＳＣに射影した多角形に対応する撮影画像上の多角形がない場合，その場所の画像が抜けるという問題がある。具体例で説明すると，図１０は，表示対象物体を正面から撮影した画像であるが，この画像をもとに視点を左に１５度ずらして第１の表示方法（第２の表示方法も同様）で合成すると，図１１に示すような画像になる。この画像では，左側面に対応する多角形が，図１０中の撮影画像中にないため，画像の欠けが生じている。
【００３２】
この問題を解決する第３の表示方法を次に示す。図１２に，第３の表示方法による処理の流れを示す。
【００３３】
第３の表示方法では，所望の視点方向と撮影方向とが近いｍ枚（２から４枚程度）の撮影画像を選び（ステップＳ３１），その中から視点方向と撮影方向が最も遠い撮影画像から順番に１枚ずつ選んで，上記第１の表示方法または第２の表示方法を用いることにより表示画像を生成する（ステップＳ３２）。生成した表示画像をスクリーンＳＣ上に順次上書きしていく（ステップＳ３３）。ステップＳ３２，Ｓ３３を，選択したｍ枚の撮影画像に対する処理を全て終了するまで繰り返す（ステップＳ３４）。
【００３４】
図１３は，第３の表示方法を用いた例であり，視点の方向を左１５度として画像を合成するにあたって，正面撮影画像を用いてスクリーンＳＣに描画する前に，左４５度方向から撮影した撮影画像について，視点を右に３０度ずらしたときの画像を生成し，それをスクリーンＳＣに描画し，その後に，図１１の画像を同様に第１または第２の表示方法を用いて上書きした例を示している。
【００３５】
これにより，画像の欠けがなくなっていることがわかる。第３の表示方法では，ｍ枚の画像を重ね書きした場合，その処理時間はｍ倍となる。ここで，表示欠けを生じさせる多角形Ｄについて考えてみる。３次元モデルデータＭＤあるいはＭＤ２内の，ある撮影画像Ｐｋを表示するときの視点の範囲は，撮影画像Ｐｋと撮影位置が隣接する撮影画像Ｐｍの撮影位置の中間であり，これを表示限界位置Ｌとする。表示限界位置Ｌを視点として概略形状Ｓの各多角形を透視投影したときにスクリーンＳＣ上に映し出される多角形のうち，撮影画像Ｐｋを撮影したときの撮影位置を視点として概略形状Ｓの各多角形を透視投影したときにスクリーンＳＣ上に現れない多角形が，表示欠けを生じさせる多角形Ｄである。
【００３６】
そこで表示欠けをなくすための描画時間の短縮のため，次のような付加画像情報ＩＴｋを用いる。付加画像情報ＩＴｋは，この多角形の各頂点が対応する撮影画像Ｐｍ上のピクセル座標情報と，その多角形の頂点のインデックスのリストで多角形を表す多角形情報，その多角形内の撮影画像からなる。上記ピクセル座標情報は，前記の多角形情報の生成の場合と同様に作成する。すなわち，付加画像情報ＩＴｋは，図６と同様な形式のピクセル座標情報，多角形情報のリストと，多角形内の撮影画像の画像データからなる。このような付加画像情報ＩＴｋを加えた３次元モデルデータをＭＤ３とし，これを描画に用いることにより，描画時間を短縮することができる。
【００３７】
図１４に，第４の表示方法による処理の流れを示す。この３次元モデルデータＭＤ３を用いた第４の表示方法では，まず，３次元モデルデータＭＤ３の撮影画像Ｐｋ（１≦ｋ≦ｎ）の中から，所望の視点方向と撮影方向とが最も近い撮影画像Ｐｊ（１≦ｊ≦ｎ）を１枚選び（ステップＳ４１），３次元モデルデータＭＤ３の付加画像情報ＩＴｋをもとに前述した第１または第２の表示方法によりスクリーンに描画し（ステップＳ４２），全ての必要な付加画像情報ＩＴｋの描画が終了したならば（ステップＳ４３），３次元モデルデータＭＤ３の撮影情報Ｉｋをもとに第１または第２の表示方法によりスクリーンに重ね書きを行い，表示画像を得る（ステップＳ４４）。
【００３８】
また，第３の表示方法または第４の表示方法では，違う撮影画像を変形したものを重ね合わせているので，撮影時の照明角度の違いなどで撮影画像の色彩に差がある場合や，概略形状が正確でないために変形した画像が一致しない場合がある。
【００３９】
例えば，図１５の画像のポリゴンＡＢＣＤＥに注目する。概略形状上では，このポリゴンは，ＡＦＣＤＥで構成されているが，線分ＣＧを含むポリゴンが手前にあるので，実際に見えている部分のポリゴンは，ＡＢＣＤＥとなる。視点方向を左にすると，点Ｂの位置が，線分ＣＧより左側になり，図１６のように表示画像に欠けが生じる。第３の表示方法または第４の表示方法のように，近傍の撮影画像で欠け部分を補ったのが図１７の例である。撮影画像の色彩が若干違い，形状にも差が生じているため，接合部が目立つことかわかる。
【００４０】
これを回避するための３次元モデルデータＭＤ４と第５の表示方法について説明する。３次元モデルデータＭＤ４は，頂点Ｂのように，手前のポリゴンの重なりにより生じた頂点を交点として，その交差点情報ＩＸを，３次元モデルデータの多角形情報ＩＰに加えたデータを，モデルデータとする。その交差点情報ＩＸは，これを表す２つの交線すなわち４つの頂点のインデックスからなる。図１５の例では，交点（頂点）Ｂは，頂点Ａ，Ｆ，Ｇ，Ｃのインデックスで与えられる。
【００４１】
第５の表示方法は，この３次元モデルデータＭＤ４を用いる。描画アルゴリズムは，基本的に第２の表示方法と同様である。違う点は，概略形状Ｓの各頂点のスクリーン座標を求めた後，それらから，交点のスクリーン座標を求めて描画する点にある。
【００４２】
図１８に，第５の表示方法による処理の流れを示す。まず，所望の視点方向と撮影方向とが最も近い撮影画像Ｐｋ（１≦ｋ≦ｎ）を選ぶ（ステップＳ５１）。視点Ｖからの概略形状Ｓの各頂点のスクリーン座標ＶＳを，透視投影により求める（ステップＳ５２）。次に，３次元モデルデータＭＤ４の交差点情報ＩＸの示す２辺の始点と終点に対応するスクリーン座標を代入して，その２辺の交点の座標を求めて頂点のスクリーン座標として加える（ステップＳ５３）。
【００４３】
そして，多角形情報ＩＰから任意の多角形Ａを一つ選び（ステップＳ５４），多角形Ａの各頂点の座標をピクセル座標情報ＩＶから得て，撮影画像Ｐｋ内の多角形Ｂの領域を取得する（ステップＳ５５）。次に，多角形Ａの各頂点座標をスクリーン座標ＶＳから得て，スクリーン上の多角形Ｃを求め（ステップＳ５６），多角形Ｂの領域をスクリーンＳＣ上の多角形Ｃの形状に合うように変形する（ステップＳ５７）。これをスクリーンＳＣに描画する（ステップＳ５８）。多角形情報ＩＰの全ての多角形に対して，多角形Ａと同様に処理してスクリーンＳＣ上に描画し表示画像を得る（ステップＳ５９）。
【００４４】
図１９は，第５の表示方法を用いた例を説明する図であり，頂点Ａ，Ｃ，Ｆ，Ｇのスクリーン座標を求め，それから，線分ＡＦおよび線分ＣＧの交点Ｂ’のスクリーン座標を求める。そして，図１５の多角形ＡＢＣＤＥの領域を図１９の多角形ＡＢ’ＣＤＥの形状に合うように変形して描画する方法を加えたものである。この方法により，画像の不自然さも回避できていることが分かる。
【００４５】
上記の３次元モデルデータＭＤ２では，ピクセル座標情報ＩＶとして，ｘ，ｙの２次元データを用い，第２の表示方法では，その座標で表された多角形を，スクリーン上の対応する多角形領域に変形して貼り付けていた。さらに，ピクセル座標情報ＩＶに，奥行き情報のｚ値を加えた情報を用いることにより，より自然な画像を得ることが可能になる。
【００４６】
図２０（Ａ）に示すスクリーンと平面の関係により，ピクセル座標情報ＩＶとしてｘ，ｙの２次元データを用いたときに生じる画像の歪みについて説明する。スクリーンＳＣ上の２点Ｓ₀とＳ₁間のｍ：（１−ｍ）の位置の点をＳ_mとし，視点ＶからＳ_mを通り，概略形状Ｓの平面との交点をＰ_mとすると，平面が奥行き方向に傾いている場合，Ｐ₀Ｐ_mとＰ_mＰ₁との比は，ｍ：（１−ｍ）にならない。このため，ｚ値を用いずに多角形を線形補間によって変形すると，図２０（Ａ）からも明らかなように画像に歪みが生じることになる。そこで，第６の表示方法では，多角形の変形時にｚ値を考慮して，いわゆるパースペクティブコレクション（perspective correction）による変形を行い，多角形への描画を実行する。
【００４７】
図７で説明した第２の表示方法におけるステップＳ２６での多角形の変形方法と，第６の表示方法による多角形の変形方法との違いについて説明する。図７のステップＳ２６では，撮影画像上の多角形ＢをスクリーンＳＣ上の多角形Ｃの形状に変形する。すなわち，多角形Ｃ内の任意の点が多角形Ｂのどこに対応するかがわかれば，変形を計算できる。
【００４８】
まず，第２の表示方法においてピクセル座標情報ＩＶとして，２次元データを用いる場合，図２１（Ａ）に示すような線形補間による多角形の変換を行うことになる。
【００４９】
多角形Ｃの辺上の２点をＳ₀（ｘ_s0，ｙ_s0），Ｓ₁（ｘ_s1，ｙ_s1）とし，それらが多角形Ｂ上で対応する２点をＰ₀（ｘ_p0，ｙ_p0），Ｐ₁（ｘ_p1，ｙ_p1）とする。Ｓ₀とＳ₁との間の点をＳ_m（ｘ_sm，ｙ_sm）とすると，点Ｓ_mに対応する多角形Ｂ上の点Ｐ_m（ｘ_pm，ｙ_pm）は，次のように計算される。
【００５０】
ｍ＝（ｘ_sm−ｘ_s0）／（ｘ_s1−ｘ_s0）
ｘ_pm＝（１−ｍ）×ｘ_p0＋ｍ×ｘ_p1
ｙ_pm＝（１−ｍ）×ｙ_p0＋ｍ×ｙ_p1
これに対し，第２の表示方法を改良した第６の表示方法では，ピクセル座標情報ＩＶとして，図２０（Ｂ）に示すような３次元データを用いて，図２１（Ｂ）に示すパースペクティブコレクションによって，撮影画像上の多角形ＢをスクリーンＳＣ上の多角形Ｃの形状に変形する。
【００５１】
多角形Ｃの辺上の２点をＳ₀（ｘ_s0，ｙ_s0，ｚ_s0），Ｓ₁（ｘ_s1，ｙ_s1，ｚ_s1）とし，それらが多角形Ｂ上で対応する２点をＰ₀（ｘ_p0，ｙ_p0，ｚ_p0），Ｐ₁（ｘ_p1，ｙ_p1，ｚ_p1）とする。また，Ｓ₀とＳ₁との間の点をＳ_m（ｘ_sm，ｙ_sm，ｚ_sm）とし，点Ｓ_mに対応する多角形Ｂ上の点をＰ_m（ｘ_pm，ｙ_pm，ｚ_pm）とする。点Ｐ_m（ｘ_pm，ｙ_pm，ｚ_pm）は，次のように計算される。
【００５２】
ｍ＝（ｘ_sm−ｘ_s0）／（ｘ_s1−ｘ_s0）
１／Ｔ＝（１−ｍ）／（ｚ_s0×ｚ_p1）＋ｍ／（ｚ_s1×ｚ_p0）
ｘ_pm／Ｔ＝（１−ｍ）×ｘ_p0／（ｚ_s0×ｚ_p1）＋ｍ×ｘ_p1／（ｚ_s1×ｚ_p0）
ｙ_pm／Ｔ＝（１−ｍ）×ｙ_p0／（ｚ_s0×ｚ_p1）＋ｍ×ｙ_p1／（ｚ_s1×ｚ_p0）
このようにピクセル座標情報ＩＶにｚ値を加えた３次元の座標情報を用いることによって，図２０（Ａ）に示すような概略形状Ｓの平面の傾きによって生じる画像の歪みを解消することができ，より自然な画像を得ることが可能になる。
【００５３】
上述した第３ないし第５の表示方法において利用する第２の表示方法に代えて，第６の表示方法を用いることができることは言うまでもない。
【００５４】
なお，以上説明した表示方法を，ＣＰＵおよびメモリを備えたコンピュータに上記の処理を行うソフトウェアプログラムを実行させることにより実現することができ，また，その一部をハードウェア回路によって実現することもできる。このようなソフトウェアやハードウェアによる画像処理のインプリメント方法自体は，以上の説明と従来のイメージベースレンダリングの技術とから，当業者には明らかであるので，これ以上の詳しい説明は省略する。また，本発明の方法をコンピュータによって実現するためのプログラムは，コンピュータが読み取り可能な可搬媒体メモリ，半導体メモリ，ハードディスクなどの適当な記録媒体に格納することができることは言うまでもない。
【００５５】
【発明の効果】
本発明では，視点と撮影方向が近い撮影画像を用い，概略形状をもとに変形させて描画させているので，従来のポリゴンモデルに比べてリアリティの高い画像を得ることができる。また，本発明で用いる概略形状は簡便なものでよいので，そのデータの作成は容易であるという利点もある。また，ＱｕｉｃｋＴｉｍｅＶＲのように撮影画像のみで描画する方法に比べ，必要となる撮影画像も少なく抑えることができるので，モデルデータ量を低く抑えることができる。また，概略形状のポリゴン数はかなり少なく抑えられるので，本発明を用いた画像合成時間は短く，リアルタイムで描画することも可能である。
【図面の簡単な説明】
【図１】３次元モデルデータＭＤの例を示す図である。
【図２】第１の表示方法による処理の流れを示す図である。
【図３】撮影画像の概略形状への投影例を示す図である。
【図４】投影された概略形状のスクリーンへの透視投影例を示す図である。
【図５】撮影画像および第１の表示方法による合成画像例を示す図である。
【図６】３次元モデルデータＭＤ２における撮影情報に追加されるピクセル座標情報ＩＶと多角形情報ＩＰの例を示す図である。
【図７】第２の表示方法による処理の流れを示す図である。
【図８】第２の表示方法の説明図である。
【図９】第２の表示方法の説明図である。
【図１０】表示対象を正面から撮影した画像を示す図である。
【図１１】視点を左１５度として正面撮影画像を第１の表示方法で合成した例を示す図である。
【図１２】第３の表示方法による処理の流れを示す図である。
【図１３】視点を左１５度として正面撮影画像を第３の表示方法で合成した例を示す図である。
【図１４】第４の表示方法による処理の流れを示す図である。
【図１５】表示対象を右１３５度，上３０度から撮影した例を示す図である。
【図１６】視点を左１５度として右１３５度撮影画像を第１の表示方法で合成した例を示す図である。
【図１７】視点を左１５度として右１３５度撮影画像を第３の表示方法で合成した例を示す図である。
【図１８】第５の表示方法による処理の流れを示す図である。
【図１９】視点を左１５度として右１３５度撮影画像を第５の表示方法で合成した例を示す図である。
【図２０】３次元ピクセル座標情報を用いる例を説明する図である。
【図２１】多角形の変形方法説明図である。
【図２２】ポリゴンモデルの例を示す図である。
【符号の説明】
Ｐｋ撮影画像
Ｉｋ撮影情報
Ｓ概略形状[0001]
BACKGROUND OF THE INVENTION
The present invention uses a 3D model data of a display target object, and records a 3D image processing method, a 3D modeling method, and a 3D image processing program for displaying an image of an object viewed from an arbitrary viewpoint direction by a computer. In particular, the present invention relates to image-based modeling and image-based rendering technology that performs processing based on captured images to be displayed.
[0002]
[Prior art]
In recent years, with the spread of the Internet and the improvement of image processing capability, 3D model data has been used for presentation of products and the like. By using the three-dimensional model data for product introduction, the customer can view the product from a desired angle / direction on the screen of a personal computer or the like without actually having the product in front of him. However, in order to generate the three-dimensional model data of the product, the shape data and the surface image data are required.
[0003]
There is a polygon model as a general three-dimensional model. FIG. 22 shows an example of the polygon model. In this model, a display target is represented by a plurality of polygons, and information such as color and gloss is given to each polygon. By perspective-projecting the data, it is possible to obtain a two-dimensional image when viewed from a desired angle and distance. In this method, it is possible to accurately obtain an observation image from an arbitrary angle and distance by generating an accurate polygon model.
[0004]
However, in order to obtain this polygon model, a dedicated three-dimensional modeling apparatus such as a range finder is required. In addition, when a display target has a curved surface, the number of polygons becomes enormous, and there is a problem that both the data amount and the processing amount become enormous. In addition, since the subtle colors cannot be expressed, the reality is low, and there is a problem that it is considerably inferior to the actual image.
[0005]
In order to increase the reality, there is a method in which a photographed image is taken from a plurality of directions and the image is pasted on the polygon surface. However, in this method, when the surface of the display object has a large specular reflection such as plastic or metal, the image reflected on the surface is different for each photographed image. On the contrary, there is a problem that the boundary of the polygon becomes unsightly
On the other hand, as a model using a photographed image, there is a method of photographing a photographed image from a plurality of directions and displaying an image closest to the desired display angle. In this method, since a photographed image is used, an image with extremely high reality can be obtained. For example, it is used in QuickTime VR developed by Apple Computer, Inc., USA. However, in order to obtain an image from an arbitrary angle, the number of images is enormous, and when the shooting distance of a live-action image and the desired viewpoint distance are greatly different, the appearance of the side surface of the display object is different. is there.
[0006]
As a method for solving this, there has been proposed a method of drawing an image between two real shot images by interpolating between them, and “View Morphing” published by IGGRAPH 96 is a typical example. In this method, corresponding points that are common between the two images are obtained in advance, the two images are deformed based on the corresponding points, and the images are synthesized and viewed from the viewpoint between the two images. Synthesize the image of the time. In this method, when an accurate corresponding point is obtained, an image from an arbitrary viewpoint between two images can be drawn fairly accurately, and the number of necessary photographed images can be reduced.
[0007]
However, in this method, a corresponding point between two live-action images is indispensable. However, an effective algorithm for obtaining this has not been realized yet, and at present, the corresponding point is manually specified. There are difficulties in automatically generating a dimensional model. In addition, since this method requires a large amount of calculation, it usually takes several to several tens of minutes for image synthesis, and cannot be displayed in real time.
[0008]
[Problems to be solved by the invention]
As described above, the polygon model, which is a conventional modeling method, has a problem that it is difficult to generate an image with high reality and it is difficult to create model data. In addition, although modeling techniques using live-action images increase reality, the amount of data becomes enormous, corresponding points must be specified manually, and the processing time becomes enormous. There is.
[0009]
[Means for Solving the Problems]
In the present invention, a real image taken from a plurality of viewpoints and a schematic shape of a display target object are set as a three-dimensional model. When a display image from a desired viewpoint is synthesized, first, a real image that has the closest desired viewpoint direction and shooting direction is selected. The photographed image is projected onto the surface of the approximate shape with the shooting position of the photographed image as the viewpoint. Then, the projected schematic shape is perspective-projected on the screen from the direction from the desired viewpoint to obtain a display image.
[0010]
The present invention basically uses a single image whose photographing direction is close to the desired viewpoint direction, and displays an image obtained by deforming the image based on the schematic shape. For this reason, even if there is a difference between images captured by specular reflection or the like between actual captured images, an image with high reality can be obtained without being influenced by the difference. In addition, since no corresponding points between live-action images are used, there is no need to obtain corresponding points.
[0011]
Furthermore, since the display image from the viewpoint between the captured images is supplemented with an image obtained by deforming the captured image, the number of necessary captured images can be kept low. In addition, since a live-action image that is close to the desired viewing direction is displayed in a deformed manner, the deformation of the live-action image is relatively small. Therefore, even if the accuracy of the approximate shape is not increased, sufficient reality can be achieved. There is a feature that a high image can be obtained.
[0012]
That is, in order to generate a rough shape, a high-precision three-dimensional modeling apparatus is not required, and the rough shape can be easily generated. In addition, since the number of polygons of the approximate shape can be suppressed to a very small value, the image synthesis time in this method is short, and drawing in real time is possible.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail.
[0014]
In the present invention, the three-dimensional model data MD of the display target object O includes n photographed images Pk (1 ≦ k ≦ n) photographed from different directions covering the entire periphery of the object, and photographing information Ik (1 ≦ k ≦ n) and polygon data (schematic shape S) representing the outline of the display target object O.
[0015]
FIG. 1 shows an example of the three-dimensional model data MD. In this figure, (A) shows an example of a photographed image, (B) shows an example of photographed information, and (C) shows an example of a schematic shape.
[0016]
The captured image Pk is captured with a digital camera or the like. The shooting information Ik includes a distance d between the camera and the display target object O (subject), a rotation matrix R indicating a shooting direction with respect to a shooting direction of the display target object O as a reference, and one pixel of the shot image. It consists of resolution r representing the length of. Since these values do not require exact values, measurement with a ruler is sufficient.
[0017]
Since the rough shape S may be a rough shape of the display target object O, the rough shape of the display target object O is obtained from the photographed image, and is obtained by using an algorithm such as Shape from Silhouette. Since it is not necessary to calculate strictly and the number of polygons to be generated may be small, it can be generated more easily than general polygon generation. For example, “From image to models (and beyond): a personal retrospective”, SIGGRAPH Course Note, 1999. is available as a reference for techniques for obtaining shapes from images (including the Shape from Silhouette algorithm).
[0018]
The rough shape S can also be generated by a method of generating roughly by measurement using a ruler or the like. Furthermore, by cutting out the background of each captured image, when displaying a display target, it is possible to display it accurately even if it overlaps with an object or the like located behind the display target.
[0019]
A first display method when the three-dimensional model data MD in the present invention is viewed from an arbitrary direction will be described. FIG. 2 shows a flow of processing by the first display method.
[0020]
First, in step S11, the direction of the desired viewpoint V is compared with the shooting directions of the captured images P1 to Pn of the three-dimensional model data MD, and the closest captured image Pk (1 ≦ k ≦ n) is selected. This comparison is realized by obtaining the inner product of the direction vector of the viewpoint V and the shooting direction vector and selecting the one with the largest value (when each direction vector is expressed as a unit vector, the inner product is closest to 1). it can.
[0021]
In step S12, the photographed image Pk is projected onto the surface of the general shape S viewed from the direction with the photographing position as the viewpoint. This is shown in FIG. Actually, on the computer, an image projected on each surface of the approximate shape S can be obtained by perspectively projecting the shooting position, perspectively projecting the approximate shape S on the screen SC, and superimposing the captured image Pk on the screen SC. it can.
[0022]
In step S13, as shown in FIG. 4, the schematic shape S on which the captured image Pk is projected is perspective-projected on the screen SC from the direction of the desired viewpoint V to obtain a display image. By performing this operation, it is possible to generate a projection image when the 3D model data MD is viewed from a desired viewpoint.
[0023]
In addition, perspective projection is one of the projection methods for projecting so that an object closer to the viewpoint is larger and a farther object is smaller in order to create the same image as a human sees with eyes, but is a well-known technique. , More detailed explanation about this is omitted.
[0024]
FIG. 5 shows a photographed image and a composite image example by the first display method. FIG. 5B is a captured image itself, FIG. 5A is an image synthesized by shifting the viewpoint 15 degrees to the left, and FIG. 5C is an image synthesized by shifting the viewpoint 15 degrees to the right. 5A and 5C are synthesized with a simple schematic shape S shown in FIG. 1C. By using this algorithm, a display image viewed from an arbitrary direction is displayed. It can be seen that it can be generated.
[0025]
In the first display method described above, the coordinate calculation is performed twice for the approximate shape S for the projection of the real image from the direction with the shooting position of the shot image Pk as the viewpoint and the perspective projection from the desired viewpoint V. Need to do. Projection information (hereinafter referred to as pixel coordinate information) IV, which is made up of pixel coordinates on the captured image corresponding to each item point of the approximate shape S, corresponds to the captured information Ik, and many on the captured image corresponding to the polygon of the approximate shape S. By using the three-dimensional model data MD2 to which polygon information IP representing a square is added, this coordinate calculation can be reduced to one time.
[0026]
FIG. 6 shows an example of pixel coordinate information IV and polygon information IP added to the photographing information in the three-dimensional model data MD2. Other imaging information is the same as that of the three-dimensional model data MD shown in FIG. The pixel coordinate information IV is two-dimensional coordinate data, and the coordinates are obtained by perspective-projecting each vertex of the schematic shape S with the shooting position as the viewpoint and the shot image Pk as the screen SC. Furthermore, the polygon information IP is a list of vertex index strings, and is generated based on the vertex connection information of the general shape S.
[0027]
Next, a second display method using the three-dimensional model data MD2 will be described. FIG. 7 shows the flow of processing by the second display method.
[0028]
In the second display method, first, a photographed image Pk (1 ≦ k ≦ n) in which the direction of the desired viewpoint V and the photographing direction are closest is selected (step S21). Next, as shown in FIG. 8, the screen coordinates VS of each vertex of the schematic shape S from the viewpoint V are obtained by perspective projection (step S22).
[0029]
Then, an arbitrary polygon A is selected from the polygon information IP (step S23), the coordinates of each vertex of the polygon A are obtained from the pixel coordinate information IV, and the area of the polygon B in the captured image Pk is acquired. (Step S24). In step S24, for example, the region of the polygon B shown in FIG. 9B is acquired from the photographed image shown in FIG.
[0030]
Next, each vertex coordinate of the polygon A is obtained from the screen coordinates VS, and a polygon C on the screen SC as shown in FIG. 9C is obtained (step S25). Then, the region of the polygon B is deformed so as to match the shape of the polygon C on the screen SC (step S26). FIG. 9D shows a state where the region of the polygon B is deformed. This is pasted and drawn on the screen SC (step S27). By processing all the polygons of the polygon information IP in the same manner as the polygon A and drawing them on the screen SC (step S28), the first display method as shown in FIG. 9E. The same display image can be obtained.
[0031]
The first or second display method has a problem that if there is no polygon on the photographed image corresponding to the polygon projected onto the screen SC, the image at that location is lost. Specifically, FIG. 10 is an image obtained by photographing the display target object from the front. Based on this image, the viewpoint is shifted 15 degrees to the left, and the first display method (the second display method is the same). ), An image as shown in FIG. 11 is obtained. In this image, since the polygon corresponding to the left side surface is not present in the captured image in FIG. 10, the image is missing.
[0032]
A third display method for solving this problem will be described below. FIG. 12 shows the flow of processing by the third display method.
[0033]
In the third display method, m (approximately 2 to 4) photographed images having a desired viewpoint direction and a photographing direction are selected (step S31), and the viewpoint image and the photographing direction are the farthest from the photographed images. One image is selected in order and a display image is generated by using the first display method or the second display method (step S32). The generated display image is sequentially overwritten on the screen SC (step S33). Steps S32 and S33 are repeated until all the processes for the selected m photographed images are completed (step S34).
[0034]
FIG. 13 shows an example using the third display method. When compositing an image with the viewpoint direction set to 15 degrees to the left, the image is taken from the direction of 45 degrees to the left before being drawn on the screen SC using the front image. For the captured image, an image is generated when the viewpoint is shifted 30 degrees to the right, and is drawn on the screen SC. Thereafter, the image of FIG. 11 is overwritten using the first or second display method in the same manner. An example is shown.
[0035]
As a result, it can be seen that the image is not missing. In the third display method, when m images are overwritten, the processing time is m times. Here, consider a polygon D that causes display defects. The range of the viewpoint when displaying a certain photographed image Pk in the three-dimensional model data MD or MD2 is an intermediate between the photographed positions of the photographed image Pk and the photographed image Pm adjacent to the photographed position. And Among the polygons displayed on the screen SC when the polygons of the schematic shape S are perspective-projected with the display limit position L as the viewpoint, each of the polygons of the schematic shape S with the imaging position when the captured image Pk is captured as the viewpoint. A polygon that does not appear on the screen SC when the square is perspective-projected is a polygon D that causes display defects.
[0036]
Therefore, the following additional image information ITk is used in order to shorten the drawing time for eliminating display defects. The additional image information ITk includes pixel coordinate information on the captured image Pm to which each vertex of the polygon corresponds, polygon information representing a polygon in a list of indices of the polygon vertices, and a captured image in the polygon. Consists of. The pixel coordinate information is created in the same manner as in the generation of the polygon information. That is, the additional image information ITk includes pixel coordinate information, a list of polygon information, and image data of a captured image in the polygon in the same format as in FIG. The three-dimensional model data to which such additional image information ITk is added is MD3 and is used for drawing, thereby shortening the drawing time.
[0037]
FIG. 14 shows the flow of processing by the fourth display method. In the fourth display method using the three-dimensional model data MD3, first, shooting is performed in which the desired viewpoint direction is closest to the shooting direction from the shot images Pk (1 ≦ k ≦ n) of the three-dimensional model data MD3. One image Pj (1.ltoreq.j.ltoreq.n) is selected (step S41), and drawn on the screen by the first or second display method described above based on the additional image information ITk of the three-dimensional model data MD3 (step S42). ), When drawing of all necessary additional image information ITk is completed (step S43), overwriting is performed on the screen by the first or second display method based on the shooting information Ik of the three-dimensional model data MD3. , A display image is obtained (step S44).
[0038]
In addition, in the third display method or the fourth display method, since different photographed images are superimposed, there is a difference in the color of the photographed image due to a difference in illumination angle at the time of photographing, etc. The deformed image may not match because the shape is not accurate.
[0039]
For example, pay attention to the polygon ABCDE in the image of FIG. On the schematic shape, this polygon is composed of AFCDE, but since the polygon including the line segment CG is in front, the polygon that is actually visible is ABCDE. When the viewpoint direction is set to the left, the position of the point B is on the left side of the line segment CG, and the display image is missing as shown in FIG. FIG. 17 shows an example in which a missing portion is compensated for by a nearby captured image as in the third display method or the fourth display method. Since the color of the photographed image is slightly different and the shape is also different, it can be seen that the joint is conspicuous.
[0040]
The 3D model data MD4 and the fifth display method for avoiding this will be described. The three-dimensional model data MD4 is obtained by adding the intersection information IX to the polygon information IP of the three-dimensional model data, with the vertex generated by the overlap of the preceding polygons like the vertex B as the intersection. To do. The intersection information IX includes two intersection lines representing this, that is, indexes of four vertices. In the example of FIG. 15, the intersection (vertex) B is given by the indices of the vertices A, F, G, and C.
[0041]
The fifth display method uses this three-dimensional model data MD4. The drawing algorithm is basically the same as in the second display method. The difference is that, after obtaining the screen coordinates of each vertex of the general shape S, the screen coordinates of the intersection are obtained and drawn.
[0042]
FIG. 18 shows a flow of processing by the fifth display method. First, a photographed image Pk (1 ≦ k ≦ n) having the closest desired viewpoint direction and photographing direction is selected (step S51). The screen coordinates VS of each vertex of the schematic shape S from the viewpoint V are obtained by perspective projection (step S52). Next, the screen coordinates corresponding to the start point and end point of the two sides indicated by the intersection information IX of the three-dimensional model data MD4 are substituted, the coordinates of the intersection of the two sides are obtained and added as the screen coordinates of the vertex (step S53). .
[0043]
Then, an arbitrary polygon A is selected from the polygon information IP (step S54), the coordinates of each vertex of the polygon A are obtained from the pixel coordinate information IV, and the area of the polygon B in the captured image Pk is acquired. (Step S55). Next, the vertex coordinates of the polygon A are obtained from the screen coordinates VS to obtain the polygon C on the screen (step S56), so that the area of the polygon B matches the shape of the polygon C on the screen SC. Deformation (step S57). This is drawn on the screen SC (step S58). All the polygons of the polygon information IP are processed in the same way as the polygon A and drawn on the screen SC to obtain a display image (step S59).
[0044]
FIG. 19 is a diagram for explaining an example using the fifth display method. The screen coordinates of the vertices A, C, F, and G are obtained, and then the screen coordinates of the intersection B ′ of the line segment AF and the line segment CG are obtained. Ask for. Then, a method is added in which the region of the polygon ABCDE in FIG. 15 is deformed and drawn so as to match the shape of the polygon ABC′CDE in FIG. 19. It can be seen that this method can also avoid the unnaturalness of the image.
[0045]
In the above three-dimensional model data MD2, x, y two-dimensional data is used as the pixel coordinate information IV, and in the second display method, a polygon represented by the coordinates is converted into a corresponding polygon region on the screen. It was deformed and pasted. Furthermore, a more natural image can be obtained by using information obtained by adding the z value of the depth information to the pixel coordinate information IV.
[0046]
Image distortion that occurs when two-dimensional data of x and y is used as the pixel coordinate information IV will be described based on the relationship between the screen and the plane shown in FIG. 2 points S on the screen SC₀And S₁The point of the position of m: (1-m) in between is S_mAnd from viewpoint V to S_mPassing through the plane of the general shape S_mIf the plane is inclined in the depth direction, P₀P_mAnd P_mP₁The ratio is not m: (1-m). For this reason, if the polygon is deformed by linear interpolation without using the z value, the image is distorted as is apparent from FIG. Therefore, in the sixth display method, in consideration of the z value when the polygon is deformed, deformation is performed by so-called perspective correction, and drawing on the polygon is executed.
[0047]
The difference between the polygonal deformation method at step S26 in the second display method described in FIG. 7 and the polygonal deformation method according to the sixth display method will be described. In step S26 of FIG. 7, the polygon B on the captured image is transformed into the shape of the polygon C on the screen SC. That is, if it is known where an arbitrary point in the polygon C corresponds to the polygon B, the deformation can be calculated.
[0048]
First, when two-dimensional data is used as the pixel coordinate information IV in the second display method, a polygon is converted by linear interpolation as shown in FIG.
[0049]
Two points on the side of polygon C are S₀(X_s0, Y_s0), S₁(X_s1, Y_s1) And the two corresponding points on polygon B are P₀(X_p0, Y_p0), P₁(X_p1, Y_p1). S₀And S₁The point between and S_m(X_sm, Y_sm), Point S_mPoint P on polygon B corresponding to_m(X_pm, Y_pm) Is calculated as follows:
[0050]
m = (x_sm-X_s0) / (X_s1-X_s0)
x_pm= (1-m) × x_p0+ M × x_p1
y_pm= (1-m) × y_p0+ M × y_p1
On the other hand, in the sixth display method improved from the second display method, the perspective collection shown in FIG. 21B is used by using three-dimensional data as shown in FIG. 20B as the pixel coordinate information IV. Thus, the polygon B on the captured image is transformed into the shape of the polygon C on the screen SC.
[0051]
Two points on the side of polygon C are S₀(X_s0, Y_s0, Z_s0), S₁(X_s1, Y_s1, Z_s1) And the two corresponding points on polygon B are P₀(X_p0, Y_p0, Z_p0), P₁(X_p1, Y_p1, Z_p1). S₀And S₁The point between and S_m(X_sm, Y_sm, Z_sm) And point S_mA point on polygon B corresponding to_m(X_pm, Y_pm, Z_pm). Point P_m(X_pm, Y_pm, Z_pm) Is calculated as follows:
[0052]
m = (x_sm-X_s0) / (X_s1-X_s0)
1 / T = (1-m) / (z_s0Xz_p1) + M / (z_s1Xz_p0)
x_pm/ T = (1-m) × x_p0/ (Z_s0Xz_p1) + M × x_p1/ (Z_s1Xz_p0)
y_pm/ T = (1-m) × y_p0/ (Z_s0Xz_p1) + M × y_p1/ (Z_s1Xz_p0)
As described above, by using the three-dimensional coordinate information obtained by adding the z value to the pixel coordinate information IV, it is possible to eliminate the image distortion caused by the inclination of the plane of the general shape S as shown in FIG. , It becomes possible to obtain a more natural image.
[0053]
It goes without saying that the sixth display method can be used in place of the second display method used in the third to fifth display methods described above.
[0054]
Note that the display method described above can be realized by causing a computer having a CPU and a memory to execute a software program for performing the above processing, and a part of the display method can also be realized by a hardware circuit. . The method of implementing image processing using software or hardware is apparent to those skilled in the art based on the above description and the conventional image-based rendering technique, and will not be described in detail. Needless to say, the program for realizing the method of the present invention by a computer can be stored in an appropriate recording medium such as a portable medium memory, a semiconductor memory, or a hard disk readable by the computer.
[0055]
【The invention's effect】
In the present invention, since a photographed image having a viewpoint and a photographing direction close to each other is used and the image is deformed and drawn based on the approximate shape, an image having higher reality than the conventional polygon model can be obtained. Further, since the schematic shape used in the present invention may be simple, there is an advantage that the data can be easily created. Further, compared to a method of drawing with only a photographed image such as QuickTimeVR, it is possible to reduce the number of necessary photographed images, so that the model data amount can be kept low. In addition, since the number of polygons of the approximate shape can be suppressed to a very small value, the image composition time using the present invention is short, and it is possible to draw in real time.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating an example of three-dimensional model data MD.
FIG. 2 is a diagram showing a flow of processing by a first display method.
FIG. 3 is a diagram illustrating an example of projection of a captured image onto a schematic shape.
FIG. 4 is a diagram showing an example of perspective projection onto a screen having a projected general shape.
FIG. 5 is a diagram illustrating an example of a composite image by a captured image and a first display method.
FIG. 6 is a diagram illustrating an example of pixel coordinate information IV and polygon information IP added to shooting information in the three-dimensional model data MD2.
FIG. 7 is a diagram showing a flow of processing by a second display method.
FIG. 8 is an explanatory diagram of a second display method.
FIG. 9 is an explanatory diagram of a second display method.
FIG. 10 is a diagram illustrating an image obtained by photographing a display target from the front.
FIG. 11 is a diagram illustrating an example in which a front-shot image is synthesized by the first display method with the viewpoint being 15 degrees to the left.
FIG. 12 is a diagram showing a flow of processing by a third display method.
FIG. 13 is a diagram illustrating an example in which a front-shot image is synthesized by a third display method with the viewpoint set to 15 degrees to the left.
FIG. 14 is a diagram showing a flow of processing by a fourth display method.
FIG. 15 is a diagram illustrating an example in which a display target is photographed from 135 degrees on the right and 30 degrees on the right.
FIG. 16 is a diagram illustrating an example in which a photographed image of 135 degrees to the right is synthesized by the first display method with the viewpoint being 15 degrees to the left.
FIG. 17 is a diagram illustrating an example in which a photographed image of 135 degrees to the right is synthesized by the third display method with the viewpoint being 15 degrees to the left.
FIG. 18 is a diagram showing a flow of processing by a fifth display method.
FIG. 19 is a diagram illustrating an example in which a photographed image of 135 degrees to the right is synthesized by the fifth display method with the viewpoint being 15 degrees to the left.
FIG. 20 is a diagram illustrating an example using three-dimensional pixel coordinate information.
FIG. 21 is an explanatory diagram of a polygonal deformation method.
FIG. 22 is a diagram illustrating an example of a polygon model.
[Explanation of symbols]
Pk image
Ik shooting information
S Outline shape

Claims

In a 3D image processing method for displaying an image of an object viewed from an arbitrary viewpoint direction by a computer using 3D model data of a display target object,
As the three-dimensional model data, n photographed images Pk (1 ≦ k ≦ n) obtained by photographing a display target object from different directions, photographing information Ik (1 ≦ k ≦ n) including at least photographing direction information, Using information on the outline shape S representing the outline of the display target object,
From the photographed images Pk (1 ≦ k ≦ n) of the three-dimensional model data, k photographed images that are close to the designated display viewpoint direction and the photographing direction are selected as a close-up photographed image set ,
From the close-up shot image set to the closest image one by one from the image with the direction of the display viewpoint and the shooting direction far away,
On the surface of general shape S of the three-dimensional model data, a process of projecting the photographed image photographing position of the photographing field image as the viewpoint,
A process of superposing and projecting the projected schematic shape on the screen with the display viewpoint as a viewpoint ;
A display image is generated by repeating for each image in the close-up photographed image set .

In a 3D image processing method for displaying an image of an object viewed from an arbitrary viewpoint direction by a computer using 3D model data of a display target object,
As the three-dimensional model data, n photographed images Pk (1 ≦ k ≦ n) obtained by photographing a display target object from different directions, photographing information Ik (1 ≦ k ≦ n) including at least photographing direction information, Information of the outline shape S representing the outline of the display target object. The position of the photographing apparatus when the photographed image Pk is photographed is used as the viewpoint in the photograph information Ik, and each vertex of the outline shape S is displayed on the photographed image Pk. Using information including projection point information IV composed of pixel coordinates on a captured image of each projection point that has been perspective-projected, and polygon information IP representing a connection of projection points representing a projected polygon of the approximate shape S,
From the photographed images Pk (1 ≦ k ≦ n) of the three-dimensional model data, k photographed images that are close to the designated display viewpoint direction and the photographing direction are selected as a close-up photographed image set ,
From the close-up shot image set to the closest image one by one from the image with the direction of the display viewpoint and the shooting direction far away,
The display viewpoint each vertex of general shape S of the three-dimensional model data perspectively projected on the screen as a viewpoint, a processing asking you to screen coordinates VS of each vertex of the projected outline shape S,
One arbitrary polygon A is taken out from the polygon information IP of the shooting information Ik, each vertex coordinate of the polygon A is acquired from the projection point information IV, and corresponds to the polygon A of the shooting image Pk pointed to by it. and processing to cut out the area of a polygon B,
Further, a polygon C on the screen corresponding to the polygon A is obtained from the screen coordinates VS of each vertex of the polygon A, and the shape of the polygon B is deformed so as to match the shape of the polygon C, and then on the screen. and processing that stuck to,
A process of repeating the same operation as the polygon A for all the polygons of the polygon information IP ,
A display image is generated by repeating for each image in the close-up photographed image set .

In a 3D image processing method for displaying an image of an object viewed from an arbitrary viewpoint direction by a computer using 3D model data of a display target object,
As the three-dimensional model data,
N captured images Pk (1 ≦ k ≦ n) obtained by capturing the display target object from different directions;
Shooting information Ik (1 ≦ k ≦ n) including at least information of the shooting direction;
Information of the outline shape S representing the outline of the display target object;
Further, between the photographed image Pk and the photographed image Pm whose photographing position is adjacent, an intermediate between the photographing position of the photographed image Pk and the photographing position of the photographed image Pm is set as the display limit position L of the photographed image Pk, and the display limit position L is viewed. When the polygons of the approximate shape S are perspectively projected, the polygons that are projected on the screen when the polygons of the approximate shape S are perspectively projected are perspectively projected with respect to the shooting position when the captured image Pk is captured. the polygon does not appear on the screen, projected on the photographed image Pm photographing position of the photographing image Pm as the viewpoint, and information obtained by adding the photographed image in the vertex coordinates and polygon of the polygon as the additional image information ITk Use
From the photographed images Pk (1 ≦ k ≦ n) of the three-dimensional model data, one photographed image Pj (1 ≦ j ≦ n) having the closest display viewpoint direction and photographing direction is selected.
Based on the additional image information ITk of the 3D model data, a captured image is projected on the surface of the approximate shape S of the 3D model data from the shooting position of the captured image as a viewpoint, and the projected approximate shape is Drawing on the screen by perspective projection on the screen with the display viewpoint as the viewpoint ,
Thereafter, based on the imaging information Ik of the 3D model data, a captured image is projected on the surface of the approximate shape S of the 3D model data with the imaging position of the captured image as a viewpoint, and the projected approximate shape A three-dimensional image processing method characterized in that a display image is generated by perspectively projecting on the screen with the display viewpoint as a viewpoint and drawing it superimposed on the screen.

In a 3D image processing method for displaying an image of an object viewed from an arbitrary viewpoint direction by a computer using 3D model data of a display target object,
As the three-dimensional model data,
N captured images Pk (1 ≦ k ≦ n) obtained by capturing the display target object from different directions;
Information of the outline shape S representing the outline of the display target object;
It consists of pixel coordinates on the captured image of each projection point obtained by perspectively projecting each vertex of the general shape S on the captured image Pk, with at least the information on the capturing direction and the position of the capturing device when capturing the captured image Pk as the viewpoint. Shooting information Ik (1 ≦ k ≦ n) including projection point information IV and polygon information IP representing connection of projection points representing a projected polygon of the approximate shape S;
Further, between the photographed image Pk and the photographed image Pm whose photographing position is adjacent, an intermediate between the photographing position of the photographed image Pk and the photographing position of the photographed image Pm is set as the display limit position L of the photographed image Pk, and the display limit position L is viewed. When the polygons of the approximate shape S are perspectively projected, the polygons that are projected on the screen when the polygons of the approximate shape S are perspectively projected are perspectively projected with respect to the shooting position when the captured image Pk is captured. A polygon that does not appear on the screen is projected onto the photographed image Pm with the photographing position of the photographed image Pm as a viewpoint, and information obtained by adding the vertex coordinates of the polygon and the photographed image in the polygon as additional image information ITk. Use
From the photographed images Pk (1 ≦ k ≦ n) of the three-dimensional model data, one photographed image Pj (1 ≦ j ≦ n) having the closest display viewpoint direction and photographing direction is selected.
Based on the additional image information ITk of the three-dimensional model data, each vertex of the schematic shape S of the three-dimensional model data is perspective-projected on the screen from the display viewpoint, and the screen of each vertex of the projected schematic shape S is projected. The coordinate VS is obtained, one arbitrary polygon A is extracted from the polygon information IP of the shooting information Ik, each vertex coordinate of the polygon A is acquired from the projection point information IV, and the number of the shot images Pk pointed to by it is obtained. A polygon B area corresponding to the polygon A is cut out, and a polygon C on the screen corresponding to the polygon A is obtained from the screen coordinates VS of each vertex of the polygon A. It is deformed to fit the shape of the square C and pasted on the screen, and all polygons of the polygon information IP are drawn on the screen by repeating the same operation as the polygon A.
Thereafter, based on the photographing information Ik of the three-dimensional model data, each vertex of the schematic shape S of the three-dimensional model data is perspective-projected on the screen from the display viewpoint, and each vertex of the projected schematic shape S is projected. The screen coordinate VS is obtained, one arbitrary polygon A is extracted from the polygon information IP of the shooting information Ik, each vertex coordinate of the polygon A is acquired from the projection point information IV, and the shot image Pk indicated by it is indicated. A region of polygon B corresponding to polygon A is cut out, and polygon C on the screen corresponding to polygon A is obtained from screen coordinates VS of each vertex of polygon A, and the shape of polygon B is obtained. paste on the screen is deformed to fit the shape of a polygon C, for all polygons of the polygon information IP, to repeat the operation similar to the polygon a, draw over the screen You 3-dimensional image processing method characterized by generating a display image by.

In a 3D image processing method for displaying an image of an object viewed from an arbitrary viewpoint direction by a computer using 3D model data of a display target object,
As the three-dimensional model data,
N captured images Pk (1 ≦ k ≦ n) obtained by capturing the display target object from different directions;
Shooting information Ik (1 ≦ k ≦ n) including at least information of the shooting direction;
Information of the outline shape S representing the outline of the display target object;
Further, the photographing information Ik includes pixel coordinates on the photographed image of each projection point obtained by perspectively projecting each vertex of the schematic shape S on the photographed image Pk with the viewpoint of the position of the photographing device when the photographed image Pk is photographed. Information obtained by adding projection point information IV and polygon information IP representing a connection of projection points representing a projected polygon of the approximate shape S;
Further, each polygon of the approximate shape S is projected onto the screen with the shooting position of the captured image Pk as a viewpoint, and the point where the sides of each polygon intersect is defined as an intersection X, and the starting points of the two sides including the intersection X And intersection information IXk which is information on the vertex of the end point and information obtained by adding the polygon information including the intersection point X projected on the screen,
From the photographed images Pk (1 ≦ k ≦ n) of the three-dimensional model data, one photographed image Pj (1 ≦ j ≦ n) having the closest display viewpoint direction and photographing direction is selected.
Perspectively projecting each vertex of the approximate shape S of the three-dimensional model data onto the screen from the display viewpoint, and obtaining a screen coordinate VS of each vertex of the projected approximate shape S;
Substituting the screen coordinates corresponding to the start point and end point of the two sides indicated by the intersection information IXk, the coordinates of the intersection of the two sides are obtained and added as the screen coordinates of the vertex,
One arbitrary polygon A is extracted from the polygon information IP of the shooting information Ik,
Obtaining the vertex coordinates of the polygon A from the projection point information IV;
Cut out a region of polygon B corresponding to polygon A of photographed image Pk to which it points,
Further, a polygon C on the screen corresponding to the polygon A is obtained from the screen coordinates VS of each vertex of the polygon A, and the shape of the polygon B is deformed so as to match the shape of the polygon C, and then on the screen. A display image is generated by repeating the same operation as that for the polygon A for all the polygons of the polygon information IP.

According to claim 2,請 Motomeko 4 or 3-dimensional image processing method according to claim 5,
As the projection point information IV, three-dimensional coordinate data including a coordinate value of depth is used, and when the shape of the polygon B is deformed to match the shape of the polygon C, the coordinate value of the depth is used, The three-dimensional image processing method characterized by performing correction | amendment according to the inclination of the plane of the said general shape S with respect to a screen.

In a three-dimensional modeling method for representing a three-dimensional object by a computer,
N captured images Pk (1 ≦ k ≦ n) obtained by capturing the display target object from different directions;
Shooting information Ik (1 ≦ k ≦ n) including at least information of the shooting direction;
While using the information with the outline shape S representing the outline of the display target object ,
Further, between the photographed image Pk and the photographed image Pm whose photographing position is adjacent, an intermediate between the photographing position of the photographed image Pk and the photographing position of the photographed image Pm is set as the display limit position L of the photographed image Pk, and the display limit position L is set as the display limit position L. When each polygon of the approximate shape S is perspective-projected, the polygon of the approximate shape S is perspectively projected out of the polygons projected on the screen when the captured image Pk is captured. A polygon that does not appear on the screen is projected onto the photographed image Pm with the photographing position of the photographed image Pm as the viewpoint, and information obtained by adding the vertex coordinates of the polygon and the photographed image in the polygon as additional image information ITk. make use of,
Generate 3D model data of display object on computer,
A three-dimensional modeling method comprising storing the three-dimensional model data in a computer-readable storage medium that performs three-dimensional image processing.

In a three-dimensional modeling method for representing a three-dimensional object by a computer,
N captured images Pk (1 ≦ k ≦ n) obtained by capturing the display target object from different directions;
Shooting information Ik (1 ≦ k ≦ n) including at least information of the shooting direction;
While using the information with the outline shape S representing the outline of the display target object,
Further, each polygon of the general shape S is projected onto the screen with the shooting position of the captured image Pk as a viewpoint, and the point where the sides of each polygon intersect is defined as an intersection X, and the starting points of the two sides including the intersection X Using information obtained by adding the intersection information IXk, which is information on the vertex of the end point, and polygon information including the intersection X projected on the screen to the shooting information Ik,
Generate 3D model data of display object on computer,
A three- dimensional modeling method , wherein the three-dimensional model data is stored in a computer-readable storage medium that performs three-dimensional image processing .

The three-dimensional modeling method according to claim 7 or 8,
A three-dimensional modeling method, wherein an image obtained by cutting a background image is used as the photographed image Pk.

In the three-dimensional modeling method according to claim 7, claim 8 or claim 9,
In the shooting information Ik,
Projection point information IV consisting of pixel coordinates on the photographic image of each projection point obtained by perspectively projecting each vertex of the schematic shape S on the photographic image Pk, with the position of the photographic device C when photographing the photographic image Pk as a viewpoint,
A three-dimensional modeling method comprising adding polygon information IP representing a connection of projection points representing a projected polygon of the approximate shape S.

The three-dimensional modeling method according to claim 10 ,
A three-dimensional modeling method characterized in that three-dimensional coordinate data including a coordinate value of depth is used as the projection point information IV including the pixel coordinates.

A three-dimensional image processing characterized in that a program for causing a computer to execute the three-dimensional image processing method according to any one of claims 1 to 6 is recorded on a computer-readable recording medium. A recording medium that records the program.