JP4010754B2 - Image processing apparatus, image processing method, and computer-readable recording medium - Google Patents

Description

ここで、上記の式（１）における９つの未知数ｂ_１からｂ_９を以下の行列Ｂとしてまとめることができる。
【００７６】
【数２】

なお、上記行列Ｂを射影変換行列と呼ぶこととする。そして、この射影変換行列Ｂを用いると、上記式（１）により示された射影変換は次の式（３）により表すことができる。
【００７７】
【数３】

なお、上記式（３）においてｓは、左辺の第三要素を１に正規化するための定数を示す。ここで、上記式（１）及び式（３）から明らかなように、行列Ｂはスケール倍の自由度を有しているため、実質的な自由度は８となる。以下においては、射影幾何学に基づき被写体像のあおり歪みを補正する射影変換行列を求める手順を詳しく説明する。
【００７８】
射影幾何学においては、画像面上の点（ｘ，ｙ）に定数ｚを付け加えた３次元ベクトル（ｘ，ｙ，ｚ）を計算に用いると便利なことが知られている。以下においては、この３次元ベクトルを拡張ベクトルと呼ぶことにする。そして特に、定数ｚの値が焦点距離ｆに一致する場合は、拡張ベクトルが画像面上の点（ｘ，ｙ）に対応する視線の向きを示す。そこで、以下においては画像面上の点（ｘ，ｙ）に焦点距離ｆを付加した３次元ベクトル（ｘ，ｙ，ｆ）を視線ベクトルと呼ぶこととする。
【００７９】
ここで、図１１（ｂ）に示された消失点Ｖ１及び消失点Ｖ２の視線ベクトルを、それぞれ以下の式（４）及び式（５）とおく。
【００８０】
【数４】

そして、射影幾何学においては、拡張ベクトルの第三要素ｚが焦点距離ｆである場合に限り、空間中で方向ベクトルがｍである直線は、画像面上において視線ベクトルがｍである消失点を持つという定理が成り立つ。すなわち、消失点Ｖ１を持つ画像パターン対に対応する平行パターン対の空間中の方向は視線ベクトルｖ_１で表され、消失点Ｖ２を持つ画像パターン対に対応する平行パターン対の空間中の方向は視線ベクトルｖ_２で表される。従って、被写体面の向きは視線ベクトルｖ_１及び視線ベクトルｖ_２の両方に直交し、次の式（６）により示されるベクトルｎにより表すことができる。
【００８１】
【数５】

なお、式（６）においてＮはベクトル（ｖ_１×ｖ_２）を単位ベクトルに正規化する作用素であり、任意のベクトルｖに対して以下の式（７）に示される演算を施すものである。
【００８２】
【数６】

なお、式（７）において｜ｖ｜はベクトルｖのノルムを示す。
【００８３】
次に、あおり歪みを補正する射影変換行列Ｂの算出手順を詳しく説明する。ここで、本実施の形態では撮像部３１における光軸の向きと被写体面の向きとの関係に基づいて、あおり歪みを補正するパラメータを計算する。具体的には、図１２に示されるように、上記式（６）により算出されたベクトルｎを単位法線ベクトルとする平面、すなわち被写体面に平行な平面を被投影面１０１として、該被投影面１０１に向かってあおり歪みを含んだ被写体像を持つ画像面１００を投影（射影変換）することにより歪み補正画像１０２を生成する。
【００８４】
以下において、あおり歪みを補正するパラメータの計算方法を説明する。まず、図１３に示されるように、撮像部３１の光軸をなすｚ軸を被投影面１０１の単位法線ベクトルｎに一致させる座標変換を表す回転行列Ｒを求める。この場合、次の式（８）に示された関係が成立する。
【００８５】
【数７】

そして、上記式（８）を満たす回転行列Ｒは多数存在するが、ここでは回転行列Ｒを次の式（９）により定義する。
【００８６】
【数８】

但し、行列Ｒ_ｘ及び行列Ｒ_ｙは次の式（１０）により示される。
【００８７】
【数９】

ここで、上記式（９）及び式（１０）により示された回転行列は、図１４に示されるように、以下の順序でｘｙｚ座標系を回転させることにより、ｘ’ｙ’ｚ’座標系に変換する演算に相当する。すなわち、最初にｘｙｚ座標系をｙ軸周りに角度βだけ回転させる。そして、このような回転により得られた座標系をｘ_１ｙ_１ｚ_１座標系とするとき、このｘ_１ｙ_１ｚ_１座標系をｘ_１軸周りに角度αだけ回転する。
【００８８】
ここで、ベクトルｎを次の式（１１）のように表す。
【００８９】
【数１０】

そして、上記の式（８）及び式（９）を用いると、回転角は次式のように導出される。
【００９０】
【数１１】

ここで、求められた回転角を式（９）及び式（１０）に代入することによって、行列Ｒを一意に定めることができる。
【００９１】
次に、上記のような手順により求められた回転行列Ｒを用いて、画像面上の任意の点（ｘ，ｙ，ｆ）を被投影面上の点（ｘ’，ｙ’，ｚ’）に座標変換する。そして、この座標変換は次式（１４）により示される射影変換により表すことができる。
【００９２】
【数１２】

ここで、ｋは撮像部３１の光学中心ｏから被投影面１０１までの距離を示す拡大係数であるため、ｋは作成される歪み補正画像の大きさを表す係数を意味する。また、ｓは射影変換後のベクトルのｚ’座標をｋにするための定数である。そして、以上の手順で式（３）の行列Ｂを以下の式（１５）により示される行列とすることにより、あおり歪みを補正するための射影変換行列Ｂを求めることができる。
【００９３】
【数１３】

そして、図９に示された座標変換部４０３は、射影変換部４０２により算出された射影変換行列Ｂを基に、被写体像を座標変換して歪み補正画像を作成する。ここで、より具体的には、射影変換後の点（ｘ’，ｙ’）に対応する変換前の点（ｘ，ｙ）を式（３）により計算し、計算された座標（ｘ，ｙ）の近傍における画素値を基に、歪み補正画像を構成する点（ｘ’，ｙ’）における画素値を補間演算により決定する。
【００９４】
なお、該補間演算は、双一次補間法やＢ−スプライン補間法などの方法を用いて行うことができる。
【００９５】
また、上記において消失点が無限遠点となる場合、すなわち画像パターン対を構成する２つの線状パターンが平行となる場合には、線状パターンの画像面上での向きが（ｍ_ｘ，ｍ_ｙ）であるとき、消失点の視線ベクトルを第三要素として０を加えた（ｍ_ｘ，ｍ_ｙ，０）とすればよい。
【００９６】
また、焦点距離ｆが未知である場合でも、適当な値ｆ’を用いて点（ｘ，ｙ）に対し拡張ベクトル（ｘ，ｙ，ｆ’）を生成し、上記手順を実行することによって、被写体像のあおり歪みを補正することができる。但し、この場合は実空間中における方向ベクトルと画像面上の消失点との間に成立する上記定理は成立しないため、実際には元の被写体面の縦横比が一致しない歪み補正画像が得られる。従って、焦点距離及び画像中心などの画像パラメータを用いて視線ベクトル（ｘ，ｙ，ｆ）を生成することにより、元の被写体面に相似な被写体像を持つ歪み補正画像を得ることができる。
【００９７】
以上より、本発明の実施の形態１に係る画像処理装置及び画像処理方法によれば、二組の画像パターン対を用いて被写体面の向きを求めることにより、あらゆる撮影条件下で撮影された被写体像であっても、簡易に該被写体像のあおり歪みを補正して適正な画像を得ることができる。また、上記画像パラメータを利用することにより、被写体面に相似であると共にあおり歪みが補正された画像を生成することもできる。
【００９８】
さらには、該あおり歪みの補正において上記のような射影変換を用いることにより、少ない処理コストで信頼性の高い該補正を実現することができる。
［実施の形態２］
図１５は、本発明の実施の形態２に係る画像処理装置の構成を示すブロック図である。図１５に示されるように、実施の形態２に係る画像処理装置は、図５に示された実施の形態１に係る画像処理装置と同様な構成を有するが、インタフェース３６には予め生成された合成画像が外部から入力されると共に、撮像部３１を備えていない点で相違するものである。
【００９９】
すなわち、本実施の形態２に係る画像処理装置は、合成画像に対してあおり歪みを補正するものである。ここで上記合成画像は、図１６（ａ）に示されるように、被写体面１０の一部を画像処理装置２００によって複数の視点Ｄ１〜Ｄｎから分割的に撮影し、該撮影により得られた複数の画像４ａ，４ｂ，４ｎを貼り合わせることによって生成されるものである。
【０１００】
より具体的には、図１６（ｂ）に示されるように、ある一つの画像例えば画像４ｎを基準として、分割された各画像４ａ，４ｂをこの画像４ｎに対して貼り合わせることにより合成画像４１が生成される。そして、本実施の形態２に係る画像処理装置２００においては、該合成画像４１のあおり歪みが補正され、図１６（ｃ）に示された歪み補正画像６０が得られる。
【０１０１】
以下において、本実施の形態２に係る画像処理装置の動作を、図１７のフローチャートを参照しつつ説明する。まず、ステップＳ１において、インタフェース３６を介して予め生成された合成画像を入力し、フレームメモリ３５に記憶する。そして、該合成画像の入力が終了すると、平行パターン検出部３９はステップＳ２において、被写体面における平行パターン対の投影像である画像パターン対を二組入力または検出する。
【０１０２】
次に、画像歪み補正部４０は、ステップＳ３において、入力された合成画像のあおり歪みを補正するためのあおり歪み補正パラメータを算出し、ステップＳ４において該パラメータを用いてあおり歪みを補正した合成画像を生成する。
【０１０３】
ここで、上記実施の形態１に係る画像処理方法と同様な方法によって、合成画像におけるあおり歪みを補正する場合、該合成画像の画像パラメータが既知である必要があるが、該合成画像の画像パラメータを知ることは可能である。例えば、被写体面の一部分をそれぞれ撮影した分割画像を、射影変換を用いて貼り合わせることにより合成画像を生成する場合には、各分割画像を基準となる画像に整合するよう変換して貼り合わせる。
【０１０４】
そして、このような方法により生成された合成画像は、上記基準となる画像を撮影した時の画像面を拡大する広画角の光学系を用いて、被写体面を撮影することにより得られる画像と等価なものとなる。すなわち、合成画像の焦点距離及び画像中心は、貼り合わせの基準となる画像の焦点距離及び画像中心とそれぞれ一致することとなる。
【０１０５】
従って、貼り合わせの基準となる画像の画像パラメータが既知ならば、合成画像の焦点距離及び画像中心を得ることができ、上記実施の形態１に係る画像処理方法を適用することによって、合成画像における被写体像のあおり歪みを補正することが可能である。
【０１０６】
以上より、本発明の実施の形態２に係る画像処理装置及び画像処理方法によれば、二組の画像パターン対を入力又は検出することによって、分割画像を貼り合わせるためあおり歪みが目立ち易い合成画像のあおり歪みを補正し、適正な合成画像を得ることができる。
【０１０７】
また、本実施の形態２に係る画像処理装置及び画像処理方法によれば、画像パラメータを利用することによって、被写体面に相似であると共に歪み補正が施された被写体像を生成することができる。そしてさらには、あおり歪み補正に射影変換を用いることによって、少ない処理コストで信頼性の高いあおり歪み補正を実現することができる。
［実施の形態３］
本発明の実施の形態３に係る画像処理装置は、図５に示された実施の形態１に係る画像処理装置と同様な構成を有するが、画像歪み補正部４０の機能が相違するものである。
【０１０８】
すなわち、本実施の形態３に係る画像処理装置は、被写体面において既知の角度で交わる２組以上の平行パターン対の投影像である画像パターン対を入力または検出する。ここで、入力した画像パターン対に対応する２組の平行パターン対が被写体面上でなす角度が既知である場合には、焦点距離ｆが未知でも被写体面に相似であると共に歪み補正がなされた被写体像を生成することができる。
【０１０９】
以下においては、被写体面における２組の平行パターン対のなす角度がψであると仮定して、本実施の形態を説明する。
【０１１０】
図１８は、本発明の実施の形態３に係る画像処理方法を示すフローチャートである。図１８に示されるように、まずステップＳ１においては、ユーザは撮像部により被写体面を撮像し、又は予め撮影された画像を外部から入力する。次に、ステップＳ２において、平行パターン検出部は被写体面においてなす角度がψである平行パターン対の投影像である画像パターン対を検出すると共に、インタフェースを介して上記角度ψを外部から入力する。なお、上記角度ψとしては、一例として９０度とすることができる。
【０１１１】
そして、画像歪み補正部は、ステップＳ３においてあおり歪み補正パラメータを算出し、ステップＳ４において算出された該パラメータを利用してあおり歪みを補正した画像を作成する。
【０１１２】
以下において、本実施の形態３に係る画像歪み補正部の動作について説明する。なお、本実施の形態３に係る画像歪み補正部は、図９に示された実施の形態１に係る画像歪み補正部４０と同様な構成を有する。
【０１１３】
まず、消失点検出部は、平行パターン検出部で得られた二組の画像パターン対における各消失点を検出する。なお、消失点の座標は、上記実施の形態１において説明された方法により求めることができる。
【０１１４】
また、射影変換部は、消失点検出部により求められた二組の消失点の座標及び上記角度ψに基づき、被写体像におけるあおり歪みを補正するための射影変換行列Ｂを求める。ここで、消失点検出部が検出した二つの消失点の拡張ベクトルを、それぞれ以下の式（１６）及び式（１７）とおく。
【０１１５】
【数１４】

なお、上記式（１６）及び式（１７）におけるｚは、定数を意味する。
【０１１６】
ここで、二つの消失点に対応する拡張ベクトルｖ_１’，ｖ_２’のなす角度はψであるとわかっているため、空間中における方向ベクトルと画面上における消失点との関係についての上記定理により、上記拡張ベクトルｖ_１’，ｖ_２’の第三要素ｚが焦点距離ｆである場合に限り、次式（１８）が成立する。
【０１１７】
【数１５】

すなわち、上記式（１８）を解くことにより、焦点距離ｆを計算することができる。
【０１１８】
以上のような方法により、既知の角度ψで交わる平行パターン対の投影像である画像パターン対を入力することによって、焦点距離ｆを計算することができる。そして、このように求められた焦点距離ｆと上記式（４）乃至式（１５）を用いて、上記実施の形態１と同様な方法により、あおり歪みを補正するための射影変換行列Ｂを求めることができる。
【０１１９】
また、座標変換部は、上記射影変換部により算出された射影変換行列Ｂを基に、画像を座標変換して歪み補正画像を作成する。具体的には、射影変換後の座標（ｘ’，ｙ’）に対応する変換前の座標（ｘ，ｙ）を式（３）に基づいて計算し、計算された座標（ｘ，ｙ）の近傍における画素値を基に座標（ｘ’，ｙ’）における画素値を補間演算により決定する。なお、この補間演算は、双一次補間法やＢ−スプライン補間法などの既存の方法を用いて行うことができる。
【０１２０】
また、ほとんどの被写体面においては、縦及び横方向の境界線や縦罫線及び横罫線など、少なくとも二組の平行パターン対が互いに直交しているため、上記平行パターン検出部においては、互いに直交している平行パターン対の投影像である画像パターン対を検出するようにしても良い。なお、この場合には、式（１８）において角度ψに９０度を代入して焦点距離ｆを求めればよいが、該焦点距離ｆは次式（１９）により求められる。
【０１２１】
【数１６】

すなわち、上記式（１９）によれば、式（１８）による計算よりも遥かに簡単に焦点距離ｆを求めることができる。そしてさらに、互いに直交している平行パターン対に対応する画像パターン対を入力するよう予め設定することとすれば、上記角度ψを別途入力する必要がなくなり、利便性を一層高めることができる。
【０１２２】
なお、上記において、いずれか一方の消失点が無限遠点となる場合には、焦点距離ｆを計算することはできない。このとき、上記実施の形態１と同様に、焦点距離ｆが未知の場合でも、適当な値ｆ’を用いて点（ｘ，ｙ）より拡張ベクトル（ｘ，ｙ，ｆ’）を生成して上記手順を実行することができるが、この場合にはあおり歪みを補正することはできるものの、実際の被写体面と縦横比が一致しない画像が生成される。
【０１２３】
以上より、本発明の実施の形態３に係る画像処理装置及び画像処理方法によれば、平行パターン対のなす角度が既知の画像パターン対を二組用いることにより、いずれか一方の消失点が無限遠点でない限り、撮影時の焦点距離が未知であっても、被写体面に相似であると共に歪み補正がなされた被写体像を生成することができる。
［実施の形態４］
本発明の実施の形態４に係る画像処理装置は、図５に示された上記実施の形態１に係る画像処理装置と同様な構成を有するが、画像歪み補正部４０の機能が相違するものである。以下においては、該相違点について詳しく説明する。
【０１２４】
本実施の形態４に係る画像処理装置は、あおり歪みを補正した画像における被写体像の傾きを補正することを特徴とする。例えば、上記実施の形態１においては、式（１４）に示された回転行列Ｒを用いて座標変換することにより、画像のあおり歪みを補正すると説明したが、この座標変換は被写体面の法線ベクトルを回転軸とする１軸周りの自由度を残している。すなわち、図１９（ａ）に示されるように、画像処理装置２が被写体面１に対して傾いている場合などには、撮影時の撮像部３１の姿勢により、画像面３内で被写体像が傾いた歪み補正画像が得られることがある。
【０１２５】
従って、本実施の形態に係る画像処理装置は、図１９（ｂ）に示されるように、画像処理装置２が被写体面１に対して傾いていない状態で撮影されたときに得られる歪み補正画像、すなわち画像面内３で被写体像が傾いていない補正画像を得ることを目的とするものである。
【０１２６】
以下において、本実施の形態４に係る画像処理装置の動作を、図２０に示されたフローチャートを参照しつつ説明する。まず、ステップＳ１において、信号処理部３２は撮像部３１により撮影された被写体面の画像を入力する。
【０１２７】
そして、ステップＳ２において、平行パターン検出部３９は、被写体面上における平行パターン対の画像への投影像である画像パターン対を入力又は検出する。なお、各画像パターン対は２つの線状パターンにより構成されている。
【０１２８】
次に、ステップＳ３において、ステップＳ２で入力された画像パターン対のうち一の画像パターン対に対して、あおり歪み補正後の向きを指定する。そして、以下に詳しく説明する画像歪み補正部４１は、ステップＳ４において、ステップＳ２において入力された画像パターン対と、ステップＳ３で指定された上記向きとに基づいて、撮影した画像のあおり歪みを補正するパラメータを算出する。またステップＳ５では、画像歪み補正部４１は、ステップＳ４で算出されたパラメータを用いてあおり歪みを補正した画像を作成する。
【０１２９】
以下において、本実施の形態４に係る画像歪み補正部４１について詳しく説明する。図２１は、本実施の形態４に係る画像歪み補正部４１の構成を示すブロック図である。図２１に示されるように、この画像歪み補正部４１は消失点検出部４０１と、射影変換部４０２と、座標変換部４０３と、向き指定部４０４とを備える。ここで、消失点検出部４０１は平行パターン検出部３９に接続され、射影変換部４０２は消失点検出部４０１に接続される。また、座標変換部４０３は射影変換部４０２に接続され、向き指定部４０４は射影変換部４０２に接続される。
【０１３０】
上記のような構成を有する画像歪み補正部４１においては、消失点検出部４０１は平行パターン検出部３９で得られた二組の画像パターン対においてそれぞれ消失点を検出する。そして、射影変換部４０２は、消失点検出部４０１から供給された二つの消失点の座標に基づき、被写体におけるあおり歪みを補正するための射影変換行列Ｂを求める。また、座標変換部４０３は、射影変換部４０２により算出された射影変換行列Ｂを基に、画像を座標変換して歪み補正画像を作成する。
【０１３１】
また、向き指定部４０４は、図２２に示されるように、表示部３７の表面に設置された透明タッチパネル４０４ａと、入力された画像パターン対のうち一つの画像パターン対に対してあおり歪み補正後の向きを指定するポインティングデバイス４０４ｂとにより構成される。
【０１３２】
ここで例えば、図２２（ａ）に示されるように、画像パターン対を入力した後、表示部３７において「向き指定」という文字がオーバレイ表示される。すると、ユーザはポインティングデバイス４０４ｂにより、表示部３７上で入力した画像パターン対のうち一つの画像パターン対に対して、あおり歪みを補正した後の画像における該画像パターンの向きを入力する。
【０１３３】
すなわち、例えば図２２（ｂ）に示されるように、第一の画像パターン対を構成する直線３９３ａ及び直線３９３ｂに対し、あおり歪み補正後に横方向（水平方向）を向くように指定すると、該画像パターン対はそれぞれ直線３９３ｃ及び直線３９３ｄのように横方向（水平方向）を向くように補正される。
【０１３４】
以下において、このような画像処理を実行するときの動作を説明する。まず、撮像部３１の光軸（ｚ軸）を被投影面の単位法線ベクトルｎに一致させる座標変換を示す回転行列Ｒは、上記式（８）で与えられる。次に、第一の画像パターン対の消失点に対応する単位視線ベクトルをｖ_１とおくと、ｖ_１を横方向すなわち単位ベクトル［１，０，０］^Ｔ に変換する座標変換は、回転行列Ｒを用いて以下の式（２０）により示される。
【０１３５】
【数１７】

ここで、回転行列Ｒは直交行列なので、同一直線上にない変換前後のベクトルが二組与えられれば一意に求められ、式（８）及び式（２０）により以下の式（２１）が求められる。
【０１３６】
【数１８】

但し、ｕはｎとｖ_１の両方に直交する単位ベクトルであり、次式（２２）で表される。
【０１３７】
【数１９】

これは、式（８）から式（１３）による計算において残された回転行列Ｒの１軸周りの自由度が、一つの画像パターン対における補正後の向きを規定することにより拘束されることを意味する。そして、以上の計算によって求められた回転行列Ｒを式（１５）に代入することにより、あおり歪みを補正する射影変換行列Ｂが求められる。
【０１３８】
また、座標変換部４０３は、射影変換部４０２により算出された射影変換行列Ｂを基に、画像を座標変換して歪み補正画像を生成する。具体的には、射影変換後の座標（ｘ’，ｙ’）に対応する変換前の座標（ｘ，ｙ）を式（３）に基づいて計算し、計算された座標（ｘ、ｙ）の近傍における画素値を基に座標（ｘ’，ｙ’）における画素値を補間演算により決定する。なお、該補間演算は、双一次補間法やＢ−スプライン補間法などの既存の方法を用いて行えばよい。
【０１３９】
以上より、本実施の形態４に係る画像処理装置及び画像処理方法によれば、上記実施の形態１と同様な効果を得ることができると共に、さらに、あおり歪みが補正された画像の画像面内における傾きを任意に変更することができるため、一層見た目の良い画像を得ることができる。
【０１４０】
また、例えば、得られた該画像に対して文字認識処理などを行う場合においては、該認識精度が向上するため、文書情報の取得から符号化までの後処理コストを大幅に減らすこともできる。
［実施の形態５］
本発明の実施の形態５に係る画像処理装置は、図５に示された上記実施の形態１に係る画像処理装置と同様な構成を有するが、画像歪み補正部４０の機能が相違するものである。以下においては、該相違点について詳しく説明する。
【０１４１】
また、本実施の形態５に係る画像処理装置は、上記実施の形態４と同様に、あおり歪みを補正した画像における被写体像の傾きを補正することを特徴とする。
【０１４２】
以下において、本実施の形態５に係る画像処理装置の動作を図２３のフローチャートを参照しつつ説明する。まず、ステップＳ１において、信号処理部３２は撮像部３１により撮影された被写体面の画像、あるいは予め撮影された画像を入力する。そして、ステップＳ２において、平行パターン検出部３９は、被写体面内における二つの平行パターン対の画像への投影像である二組の画像パターン対を入力又は検出する。なお、各画像パターン対は２つの線状パターンにより構成されている。
【０１４３】
次に、以下に詳しく説明する画像歪み補正部４２は、ステップＳ３において、撮影した画像のあおり歪みを補正するパラメータを算出し、ステップＳ４で該算出されたパラメータを用いてあおり歪みを補正した画像を作成する。そして、ステップＳ５において、歪み補正がなされた画像における被写体像の画面内における傾きを補正する。
【０１４４】
以下において、本実施の形態５に係る画像歪み補正部４２について詳しく説明する。図２４は、本実施の形態５に係る画像歪み補正部４２の構成を示すブロック図である。図２１に示されるように、この画像歪み補正部４２は消失点検出部４０１と、射影変換部４０２と、座標変換部４０３と、傾き補正部４０５とを備える。ここで、消失点検出部４０１は平行パターン検出部３９に接続され、射影変換部４０２は消失点検出部４０１に接続される。また、座標変換部４０３は射影変換部４０２に接続され、傾き補正部４０５は座標変換部４０３に接続される。
【０１４５】
上記のような構成を有する画像歪み補正部４２においては、消失点検出部４０１は平行パターン検出部３９で得られた二組の画像パターン対においてそれぞれ消失点を検出する。そして、射影変換部４０２は、消失点検出部４０１から供給された二つの消失点の座標に基づき、被写体におけるあおり歪みを補正するための射影変換行列Ｂを求める。また、座標変換部４０３は、射影変換部４０２により算出された射影変換行列Ｂを基に、画像を座標変換して歪み補正画像を作成する。
【０１４６】
また、傾き補正部４０５は、図２５に示されるように、表示部３７の表面に設置された透明タッチパネル４０５ａと、傾きを補正する角度を入力するポインティングデバイス４０５ｂにより構成され、歪み補正画像の画像面内における傾きを補正するための座標変換を求めるものである。例えば、本実施の形態に係る画像処理装置の筐体に配設されたスイッチ（図示していない）を押して、傾き補正モードを起動すると、図２５に示されるように、表示部３７の透明タッチパネル４０５ａ上において「傾き補正」という文字がオーバレイ表示される。すると、ユーザはポインティングデバイス４０５ｂにより表示部３７上で円弧を描き、被写体像の傾きを補正する回転の方向及び回転の大きさをゼスチャ入力する。
【０１４７】
そして、このようにして入力された回転角をθとすると、次式（２３）により歪み補正画像における点（ｘ’，ｙ’）から点（ｘ’’，ｙ’’）への合同変換を算出する。
【０１４８】
【数２０】

但し、点（ｘ_０’，ｙ_０’）は回転の中心となる座標であり、通常は画像中心が回転中心座標として選択される。
【０１４９】
そして、傾き補正部４０５の算出結果は座標変換部４０３に出力され、式（２３）の合同変換式を基に、歪み補正画像における被写体像の傾きを補正した画像を作成する。具体的には、射影変換後の座標（ｘ’’，ｙ’’）に対応する変換前の座標（ｘ’，ｙ’）を式（２３）に基づいて計算し、計算された座標（ｘ’、ｙ’）の近傍における画素値を基に座標（ｘ’’，ｙ’’）における画素値を補間演算により決定する。なお、該補間演算は、双一次補間法やＢ−スプライン補間法などの既存の方法を用いて行えばよい。
【０１５０】
以上より、本実施の形態５に係る画像処理装置及び画像処理方法によっても、上記実施の形態４と同様な効果を得ることができる。
［実施の形態６］
本発明の実施の形態６に係る画像処理装置は、図５に示された上記実施の形態１に係る画像処理装置と同様な構成を有するが、画像歪み補正部４０の機能が相違するものである。以下においては、該相違点について詳しく説明する。
【０１５１】
ここで、本実施の形態６に係る画像処理装置は、さらにあおり歪みを補正した画像の大きさを該補正前の画像の大きさと概ね等しくすることを特徴とするものである。
【０１５２】
以下において、本実施の形態６に係る画像処理装置の動作を図２６のフローチャートを参照しつつ説明する。まず、ステップＳ１において、信号処理部３２は撮像部３１により撮影された被写体面の画像、あるいは予め撮影された画像を入力する。そして、ステップＳ２において、平行パターン検出部３９は、被写体面内における二つの平行パターン対の画像への投影像である二組の画像パターン対を入力又は検出する。なお、各画像パターン対は２つの線状パターンにより構成されている。
【０１５３】
次に、以下に詳しく説明する画像歪み補正部４３は、ステップＳ３において、撮影した画像のあおり歪みを補正するパラメータを算出し、ステップＳ４で、あおり歪みを補正した画像の大きさを該補正前の大きさとほぼ同じにするための拡大係数ｋを計算する。そしてさらに、ステップＳ５において、画像歪み補正部４３は、該算出されたパラメータ及び拡大係数ｋを用いてあおり歪みを補正した画像を作成する。
【０１５４】
以下において、本実施の形態６に係る画像歪み補正部４３について詳しく説明する。図２７は、本実施の形態６に係る画像歪み補正部４３の構成を示すブロック図である。図２７に示されるように、この画像歪み補正部４３は消失点検出部４０１と、射影変換部４０２と、座標変換部４０３と、画像正規化部４０６とを備える。ここで、消失点検出部４０１は平行パターン検出部３９に接続され、射影変換部４０２は消失点検出部４０１に接続される。また、座標変換部４０３は射影変換部４０２に接続され、画像正規化部４０６は消失点検出部４０１と座標変換部４０３との間に射影変換部４０２と並列に接続される。
【０１５５】
上記のような構成を有する画像歪み補正部４３においては、消失点検出部４０１は平行パターン検出部３９で得られた二組の画像パターン対においてそれぞれ消失点を検出する。そして、射影変換部４０２は、消失点検出部４０１から供給された二つの消失点の座標に基づき、被写体におけるあおり歪みを補正するための射影変換行列Ｂを求める。また、座標変換部４０３は、射影変換部４０２により算出された射影変換行列Ｂを基に、画像を座標変換して歪み補正画像を作成する。
【０１５６】
また、画像正規化部４０５は、式（１４）において、あおり歪みを補正した画像を補正前とほぼ同じ大きさとするための拡大係数ｋの値を求める。ここで拡大係数ｋは、図２８に示されるように、撮像部３１の光学中心ｏと被写体面に平行な被投影面１０１との距離を表し、作成される歪み補正画像の大きさを規定する。
【０１５７】
そこで、図２８に示されるように、焦点距離ｆ、及び撮像部３１の光軸（ｚ軸）と被投影面１０１の法線ベクトルとのなす角φを用いて、拡大係数ｋを次式（２４）により算出される値とする。
【０１５８】
【数２１】

これにより、あおり歪みを補正した画像１０４の大きさは、図２８に示された斜線部で示されるように、補正前の画像１０３の大きさと概ね等しくなる。ここで、撮像部３１の光軸（ｚ軸）の単位ベクトルと、式（８）により示された被投影面１０１の法線ベクトルｎを用いて、以下の式（２５）で示される関係が成立する。
【０１５９】
【数２２】

これにより、上記式（２４）は次式（２６）のように変形される。
【０１６０】
【数２３】

従って、上記式（２６）により算出された拡大係数ｋを用いて式（１４）により示される座標変換を行うことにより、あおり歪み補正前後の画像の大きさをほぼ同じにすることができる。
【０１６１】
なお、上記拡大係数ｋの算出方法は、式（２６）を用いる方法に限定されるものでない。すなわち例えば、あおり歪み補正前の画像における図２９（ａ）に示された点Ｐ_１，Ｐ_２，Ｐ_３，Ｐ_４に対し、一旦ｋを１と置いて式（１４）に示された座標変換を施すことにより、図２９（ｂ）に示されたあおり歪み補正後の点Ｐ_１’，Ｐ_２’，Ｐ_３’，Ｐ_４’をそれぞれ計算する。次に、該点Ｐ_１’，Ｐ_２’，Ｐ_３’，Ｐ_４’に囲まれた外接矩形の面積Ｓ’を求める。ここで、あおり歪み補正前の画像における点Ｐ_１，Ｐ_２，Ｐ_３，Ｐ_４により囲まれた矩形の面積をＳとすると、上記拡大係数ｋは次式（２７）によっても算出することができる。
【０１６２】
【数２４】

以上より、本実施の形態６に係る画像処理装置及び画像処理方法によれば、上記実施の形態１と同様な効果を得ることができると共に、さらに、あおり歪みを補正した画像を所望の大きさにすることができる。
［実施の形態７］
本発明の実施の形態７に係る画像処理装置は、図５に示された上記実施の形態１に係る画像処理装置と同様な構成を有するが、画像歪み補正部４０の機能が相違するものである。以下においては、該相違点について詳しく説明する。
【０１６３】
ここで、本実施の形態７に係る画像処理装置についても、上記実施の形態６と同様に、あおり歪みを補正した画像の大きさを該補正前の画像の大きさと概ね等しくすることができることを特徴とするものである。
【０１６４】
そして、本実施の形態７に係る画像処理装置は、上記実施の形態６に係る画像処理装置と同様に動作するが、図２６に示されたステップＳ４においては、ユーザが画像歪み補正部に対して、あおり歪みを補正した画像の大きさを設定する。これにより、ステップＳ５において、本実施の形態７に係るあおり歪み補正部４４は、ステップＳ３において算出されたあおり歪み補正パラメータと上記のようにユーザによって設定された大きさとにより、あおり歪みを補正した画像を生成する。
【０１６５】
以下において、本実施の形態７に係る画像歪み補正部４４について詳しく説明する。図３０は、本実施の形態７に係る画像歪み補正部４４の構成を示すブロック図である。図３０に示されるように、この画像歪み補正部４４は消失点検出部４０１と、射影変換部４０２と、座標変換部４０３と、画像サイズ設定部４０７とを備える。ここで、消失点検出部４０１は平行パターン検出部３９に接続され、射影変換部４０２は消失点検出部４０１に接続される。また、座標変換部４０３は射影変換部４０２に接続され、画像サイズ設定部４０７は消失点検出部４０１と座標変換部４０３との間に射影変換部４０２と並列に接続される。
【０１６６】
上記において、画像サイズ設定部４０７は、図３１に示されるように、表示部３７の表面に設置された透明タッチパネル４０７ａと、あおり歪み補正前後の画像の拡大倍率を設定するポインティングデバイス４０７ｂにより構成される。
【０１６７】
ここで、例えば本実施の形態６に係る画像処理装置の筐体に配設されたスイッチを押すことにより、画像サイズ設定モードを起動すると、図３１に示されるように表示部３７において「画像サイズ設定」という文字がオーバレイ表示される。すると、ユーザはポインティングデバイス４０７ｂを用いて表示部３７上に表された倍率を選択することにより、歪み補正がなされた画像の倍率が入力される。
【０１６８】
このとき、該入力された倍率をｇとすると、上記実施の形態５に係る画像処理方法と同様に、式（１４）の拡大係数ｋを次式（２８）により算出する。
【０１６９】
【数２５】

従って、式（２８）により算出された拡大係数ｋの値を用いて式（１４）の座標変換を行うことにより、所望の大きさの歪み補正画像を得ることができる。なお、拡大係数ｋの計算方法は、式（２８）による計算に限定されるものではなく、式（２７）のｋの値に基づいて以下の式（２９）により算出することもできる。
【０１７０】
【数２６】

以上より、本発明の実施の形態７に係る画像処理装置及び画像処理方法によっても、上記実施の形態６と同様な効果を得ることができる。
【０１７１】
なお、上記全ての実施の形態に係る画像処理方法は、コンピュータプログラムにより記述できる。従って、該プログラムを例えばＣＤ−ＲＯＭ１７１やフロッピーディスク等のコンピュータ読み取り可能な記録媒体に記録して、図３２（ａ）に示されるようにコンピュータＰＣに装着することにより、該プログラムをコンピュータＰＣに実行させることにより容易に該画像処理を実現することができる。
【０１７２】
また、図３２（ｂ）に示されるように、上記全ての実施の形態に係る画像処理装置は、デジタルカメラ２ａとして一つの筐体に収容してもよく、この場合には上記プログラムが記録されたフロッピーディスク１７２等の小型記録媒体を該デジタルカメラ２ａに装着して、該プログラムを該デジタルカメラ２ａ自身に実行させることにより、該デジタルカメラ２ａに上記画像処理の機能を持たせることができる。
【０１７３】
【発明の効果】
上述の如く、本発明に係る画像処理によれば、様々な撮影条件において撮影されることにより生じた画像の歪みを簡易に補正することができるため、適正な画像を容易に得ることができる。
【０１７４】
また、少なくとも二つの消失点を用いて変換行列を求めることとすれば、様々な撮影条件において撮影されることにより生じた画像の歪みを簡易な画像処理により確実に補正することができるため、精度の高い適正な画像を得ることができる。
【０１７５】
また、光学系の画像パラメータ及び消失点を用いることによって変換行列を算出することとすれば、容易に補正後の画像を被写体の表面に描かれた原像と相似なものとすることができるため、より高精度な補正を実現することができる。
【０１７６】
また、合成画像を画像処理する場合には、合成画像の作成の際に基準とされた画像を撮影した光学系の画像パラメータ及び消失点を用いることにより変換行列を算出することとすれば、あおり歪みが生じ易い合成画像に対しても、簡易な画像処理により該歪みを補正することができる。
【０１７７】
また、少なくとも二組の線分が被写体上でなす角度及び消失点を用いて変換行列を算出することとすれば、被写体を撮影した光学系の焦点距離が未知の場合においても該歪みを高精度に補正することができるため、該焦点距離を得るための装置及び動作が不用となり、より簡易に適正な画像を得ることができる。
【０１７８】
また、一組の平行な線分における座標変換後の方向がユーザにより指定された場合には、該方向に応じて変換行列を算出することとすれば、歪みが補正された画像の向きを画像面内において容易に変更することができるため、一層見た目の良い画像を得ることができ、また、得られた画像に対する後処理コストを低減することもできる。
【図面の簡単な説明】
【図１】合成画像を得る際の従来からの問題を説明する図である。
【図２】本発明の実施の形態に係る画像処理装置の動作を説明する図である。
【図３】本発明の実施の形態に係る画像処理装置の基本構成を示すブロック図である。
【図４】本発明の実施の形態に係るデジタルカメラの構成を示すブロック図である。
【図５】本発明の実施の形態１に係る画像処理装置の構成を示すブロック図である。
【図６】図５に示された画像処理装置の動作を示すフローチャートである。
【図７】図５に示された平行パターン検出部の構成及び動作を説明する図である。
【図８】図５に示された平行パターン検出部の他の動作を説明する図である。
【図９】図５に示された画像歪み補正部の構成を示すブロック図である。
【図１０】図５に示された撮像部の光学系を説明する図である。
【図１１】図９に示された消失点検出部の動作を説明する図である。
【図１２】図５に示された画像歪み補正部によるパラメータ算出動作を説明する第一の図である。
【図１３】図５に示された画像歪み補正部によるパラメータ算出動作を説明する第二の図である。
【図１４】図５に示された画像歪み補正部によるパラメータ算出動作を説明する第三の図である。
【図１５】本発明の実施の形態２に係る画像処理装置の構成を示すブロック図である。
【図１６】本発明の実施の形態２に係る画像処理装置の動作を説明する図である。
【図１７】図１５に示された画像処理装置の動作を示すフローチャートである。
【図１８】本発明の実施の形態３に係る画像処理装置の動作を示すフローチャートである。
【図１９】本発明の実施の形態４に係る画像処理装置の動作を説明する図である。
【図２０】本発明の実施の形態４に係る画像処理装置の動作を示すフローチャートである。
【図２１】本発明の実施の形態４に係る画像歪み補正部の構成を示すブロック図である。
【図２２】図２１に示された向き指定部の構成及び動作を説明する図である。
【図２３】本発明の実施の形態５に係る画像処理装置の動作を示すフローチャートである。
【図２４】本発明の実施の形態５に係る画像歪み補正部の構成を示すブロック図である。
【図２５】図２４に示された傾き補正部の構成及び動作を説明する図である。
【図２６】本発明の実施の形態６に係る画像処理装置の動作を説明するフローチャートである。
【図２７】本発明の実施の形態６に係る画像歪み補正部の構成を示すブロック図である。
【図２８】図２７に示された画像歪み補正部の動作を説明する第一の図である。
【図２９】図２７に示された画像歪み補正部の動作を説明する第二の図である。
【図３０】本発明の実施の形態７に係る画像歪み補正部の構成を示すブロック図である。
【図３１】図３０に示された画像サイズ設定部の構成及び動作を説明する図である。
【図３２】本発明に係る他の実施の形態を説明する図である。
【符号の説明】
１，１０ 被写体面
２，２００ 画像処理装置
２ａ デジタルカメラ
３，１００ 画像面
４，４ａ〜４ｎ，６，４１，６０，１０３，１０４ 画像
５ 被写体像
７ 分割画像
１１ 中央演算処理装置（ＣＰＵ）
１２ 同期型ダイナミックランダムアクセスメモリ（ＳＤＲＡＭ）
１３ ハードディスクドライブ（ＨＤＤ）
１４ 入力インタフェース（入力Ｉ／Ｆ）
１５ 電源
１６ 表示インタフェース（表示Ｉ／Ｆ）
１７ 記録装置
１８ 外部インタフェース（外部Ｉ／Ｆ）
１９ ディスプレイ
２０ 読み出し専用メモリ（ＲＯＭ）
２１ レンズ
２２ 絞り機構
２３ シャッタ
２４ 光電変換素子
２５ 駆動部
２６ 前処理回路
２７ モータ駆動部
２８ 液晶ディスプレイ（ＬＣＤ）
３１ 撮像部
３２ 信号処理部
３３ メモリ制御部
３４ 主制御部
３５ フレームメモリ
３６ インタフェース（Ｉ／Ｆ）
３７ 表示部
３８ 外部記憶部
３９ 平行パターン検出部
４０，４１，４２，４３，４４ 画像歪み補正部
１０１ 被投影面
１０２ 歪み補正画像
１７１ ＣＤ−ＲＯＭ
１７２ フロッピーディスク
３９０ ポインタ
３９１，４０４ａ，４０５ａ，４０７ａ 透明タッチパネル
３９２，４０４ｂ，４０５ｂ，４０７ｂ ポインティングデバイス
３９３ａ〜３９３ｄ 直線
４０１ 消失点検出部
４０２ 射影変換部
４０３ 座標変換部
４０４ 向き指定部
４０５ 傾き補正部
４０６ 画像正規化部
４０７ 画像サイズ設定部
ＢＵＳ バス
ＰＯ 対象点
ＣＩ 画像中心
ＥＶ 視線ベクトル
ｖ１，ｖ２ 消失点
Ｄ１〜Ｄｎ 視点
Ｐ１，Ｐ２ 位置
ＰＣ コンピュータ[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus, an image processing method, and a computer recording medium, and more specifically, an image processing apparatus and an image processing method for obtaining an appropriate image regardless of an imaging state, and a program for realizing the method. The invention relates to a recording medium on which is recorded.
[0002]
[Prior art]
With the rapid advancement of computer networks, the way of doing business is diversifying, and it is necessary to quickly acquire important information in all aspects. Along with this, there is an increasing demand for easy and high-precision input of document information necessary for business by using a portable input device everywhere. In particular, along with the rapid spread and high resolution of digital still cameras, there have been some applications where the captured images are used as electronic information effective for work and entertainment by processing them. .
[0003]
As a typical first application example, when photographing a subject surface such as A4 paper or a large poster, if the photographing surface and the subject surface are not parallel, image distortion (this is referred to as “aori” There is a technique for improving the legibility of the document image information acquired by correcting this. With regard to such a technique, in Japanese Patent Laid-Open No. 3-94383, an input image is placed in a known assumed fixed shape, and the distortion of the fixed shape is examined, whereby the original object surface is examined. A technique of finding a conversion parameter for matching a shape and correcting tilt distortion is disclosed.
[0004]
However, according to this technique, it is necessary to place an object to be photographed in a fixed figure whose shape is already known, and it is also necessary to obtain data for the fixed figure in advance.
[0005]
In Japanese Patent Laid-Open No. 5-101221, an orthogonal grid is introduced into a subject surface to obtain spatial coordinates of each grid point, and an orthogonal transformation is performed on a coordinate system provided on an imaging surface to perform projection. A technique for correcting distortion is disclosed. However, in such a technique, since it is necessary to manually input the two-dimensional coordinates of each lattice point, it is difficult to photograph the subject surface by a simple operation.
[0006]
In Japanese Patent Laid-Open No. 6-197261, correction processing is performed after calculating a correction amount for tilt distortion from contour information obtained by detecting the contour of a subject image, and the corrected image is processed. A technique for making image quality uniform by performing image processing that thins out the maximum amount of interpolation is disclosed. However, in the above processing, for example, a trapezoidal subject image is stretched in the horizontal direction and converted into a rectangle without considering the aspect ratio of the subject, so that the obtained subject image is transformed into a figure that is not similar to the original subject. There is a problem of being.
[0007]
Japanese Patent Application Laid-Open No. 9-289600 includes angle setting means for inputting an inclination angle formed by an imaging surface of a camera and a subject surface, and distance measuring means for measuring a distance to the subject, to the angle setting means. A technique is disclosed in which a subject is photographed while an inclination angle is input, so that the subject is corrected to an image viewed from the front based on the inclination angle and the subject distance detected by the distance measuring means. However, in such a technique, it is very difficult to input the tilt angle accurately and manually. As a result, it is difficult to correct the distortion of the image with high accuracy.
[0008]
Japanese Patent Application Laid-Open No. 11-98485 discloses a technique for correcting the tilt distortion of an image so that a geometric shape of a photographing target is correctly reproduced based on information on the direction of a camera fixed to a rotation axis. Has been. However, in this technique, since it is necessary to fix the camera to the rotation axis, there is a problem that the apparatus scale increases, and the degree of freedom of photographing by the user decreases.
[0009]
Japanese Patent Application Laid-Open No. 11-238123 discloses a method for correcting the tilt distortion of an image by instructing the relationship between the image element and the image element on a display screen on which an image obtained by imaging is displayed. Etc. are disclosed. However, this method can correct distortion accurately except when the subject is photographed in a state where the plane formed by the normal vector of the subject surface and the base is parallel to the base of the imaging plate. However, there is a problem that the method can be applied only under limited photographing conditions.
[0010]
As a second application example, a portable image input device is used to divide and shoot large-area paper information such as newspapers and pictures drawn on panels and walls, and paste the obtained images. Some of them create one composite image by combining them. That is, as the number of pixels of an image sensor represented by a CCD (Charge Coupled Device) increases, the resolution of a digital camera has improved in recent years. There is still a lack of resolution to capture and digitize subjects. Therefore, an approach has been made in which a high-definition image is artificially created by pasting together images to compensate for the lack of resolution of the digital camera.
[0011]
Such an application is a technique for pasting together images obtained by dividing a part of a subject using geometric correction formulas such as affine transformation and projective transformation when the subject can be regarded as a flat shape. The image is converted to the appearance of the subject in the reference image and pasted. An outline of such technology is described in the document "Computer Vision -Technical Review and Future Prospect-" (Takashi Matsuyama et al., New Technology Communications).
[0012]
However, in such an application example, if there is a distortion in the subject image in the reference image, there is a problem in that the distortion is also included in the combined composite image. This problem will be described with reference to FIG. In the figure, f indicates the focal length of the optical system included in the photographing apparatus.
[0013]
As shown in FIG. 1, for example, consider a case where two images obtained by photographing a subject surface 1 from a position P1 and a position P2 are combined. Here, the divided image 7 obtained by photographing from the position P2 becomes a subject image 5 when it is mapped onto a surface obtained by enlarging the image surface 3 used as a reference in photographing from the position P1. Therefore, as shown in FIG. 1, the divided image 7 is combined with an image photographed from the position P <b> 1 on the image plane 3 as a subject image 5. The composite image obtained in this way is equivalent to an image obtained by photographing the subject surface 1 using an optical system with a wide angle of view that enlarges the reference image surface.
[0014]
Therefore, the synthesized image created as described above has a problem that the perspective effect on the subject is larger than that of an image obtained by normal shooting, and thus a larger tilt and distortion are likely to occur.
[0015]
[Problems to be solved by the invention]
The present invention has been made to solve the above-described problems, and records an image processing apparatus, an image processing method, and a program for realizing the method that can easily obtain an appropriate image regardless of shooting conditions. An object of the present invention is to provide a recording medium.
[0016]
[Means for Solving the Problems]
  The above purpose isShoot the subject surfaceOf the image obtained byAori distortionAn image processing apparatus for correctingAt least a parallel pattern pair including two patterns parallel to each other 2 An input unit that inputs an input image obtained by imaging a subject surface including two parallel pattern detection units that detect an image pattern pair that is a projection image of the two parallel pattern pairs in the input image;
An image distortion correction unit that corrects the distortion of the input image based on the two image pattern pairs, and the parallel pattern detection unit receives input of the two image pattern pairs in the input image or Detecting, and receiving an input of an angle formed by the two parallel pattern pairs on the subject surface with the image pattern pair as a projection image, and the image distortion correction unit, for each of the two image pattern pairs, Vanishing point detecting means for detecting coordinates of vanishing points that are intersections of patterns constituting the image pattern pair, and an optical in which the input image is photographed based on the coordinates of the two vanishing points and the input angle A tilt distortion of the input image based on a focal length calculation means for calculating a focal length in the system, the coordinates of the two vanishing points, and the focal distance A correction parameter calculating means for calculating a correction to the correction parameter, based on the correction parameter comprises a generating means for generating a corrected image obtained by correcting the tilt distortion of the input imageThis is achieved by providing an image processing apparatus characterized by the above. According to such means, it is possible to correct image distortion caused by shooting under various shooting conditions by an image processing apparatus having a simple configuration.
[0017]
  In addition, the present inventionImage processing deviceAccording to the above, it is possible to reliably correct the distortion of an image caused by shooting under various shooting conditions by the image processing apparatus having a simple configuration.
[0018]
Here, if the image correction means calculates the transformation matrix by using the image parameters and vanishing points of the optical system that captured the subject, the original image in which the corrected image is drawn on the surface of the subject Can be similar.
[0019]
In addition, the image correction means is used as a reference when creating a composite image when inputting a composite image generated by joining a plurality of images obtained by partially photographing a subject. The transformation matrix can be calculated by using the image parameters and vanishing points of the optical system that captured the captured image. According to such means, the distortion can be corrected by the image processing apparatus having a simple configuration with respect to the composite image in which tilt distortion is likely to occur. Note that the image parameter may include a focal length in the optical system.
[0020]
Further, if the image correction means calculates the transformation matrix using the angle and vanishing point formed by at least two sets of line segments on the subject, even when the focal length of the optical system photographing the subject is unknown The distortion can be corrected with high accuracy.
[0021]
Further, the image correction means may calculate a conversion matrix according to the direction when the direction after coordinate conversion in a set of line segments is designated by the user. According to such means, the orientation of the image whose distortion has been corrected can be freely changed in the image plane.
[0022]
  The object of the present invention is toShoot the subject surfaceOf the image obtained byAori distortionAn image processing method for correctingAt least a parallel pattern pair including two patterns parallel to each other 2 An input step of inputting an input image obtained by imaging a subject surface including two, a parallel pattern detection step of detecting an image pattern pair which is a projection image of the two parallel pattern pairs in the input image, and the two image patterns An image distortion correction step for correcting a tilt distortion of the input image based on a pair, wherein the parallel pattern detection step receives or detects the input of the two image pattern pairs in the input image, and An input of an angle formed by the two parallel pattern pairs having the image pattern pair as a projection image on the object plane is received, and the image distortion correction step configures the image pattern pair for each of the two image pattern pairs. A vanishing point detecting step for detecting the coordinates of the vanishing point that is the intersection of the patterns to be Based on the target and the input angle, a focal length calculating step for calculating a focal length in the optical system in which the subject surface is photographed, and based on the coordinates of the two vanishing points and the focal length, A correction parameter calculating step for calculating a correction parameter for correcting a tilt distortion of the input image; and a generation step for generating a corrected image in which the tilt distortion of the input image is corrected based on the correction parameter.This is achieved by providing an image processing method characterized by the above. According to such means, it is possible to correct image distortion caused by shooting under various shooting conditions by simple image processing.
[0023]
Another object of the present invention is an image processing method for correcting distortion of an image obtained by photographing a subject, wherein at least two sets of line segments extracted from the image are extrapolated, respectively. A first step of calculating a vanishing point; a second step of obtaining a transformation matrix using at least two vanishing points; and a third step of performing coordinate transformation of each point constituting the image by the transformation matrix. This is achieved by providing an image processing method characterized by the above. According to such means, it is possible to reliably correct distortion of an image caused by shooting under various shooting conditions by simple image processing.
[0024]
In the second step, if the transformation matrix is calculated by using the image parameters and vanishing points of the optical system that captured the subject, the corrected image can be easily drawn on the surface of the subject. It can be similar to an image.
[0025]
In addition, when image processing is performed on a composite image generated by stitching together a plurality of images obtained by partially photographing a subject, in the second step, a reference is provided when generating the composite image. If the transformation matrix is calculated by using the image parameters and vanishing points of the optical system that captured the image, the distortion is corrected by simple image processing for a composite image that is prone to tilt distortion. be able to. Note that the image parameter may include a focal length in the optical system.
[0026]
In the second step, if the transformation matrix is calculated using the angle and vanishing point formed by at least two sets of line segments on the subject, the focal length of the optical system that captured the subject is unknown. In addition, the distortion can be corrected with high accuracy.
[0027]
In addition, when the direction after coordinate transformation in a set of line segments is designated by the user, in the second step, if the transformation matrix is calculated according to the direction, the distortion corrected image The orientation can be easily changed in the image plane.
[0028]
Another object of the present invention is a computer-readable recording medium storing a program for correcting image distortion obtained by photographing a subject by image processing by a computer, and the program is stored in a computer. Each of the points constituting the image by calculating at least two intersection points obtained by extrapolating at least two pairs of opposing line segments extracted from the image, and obtaining a transformation matrix using at least two intersection points Is achieved by providing a computer-readable recording medium characterized in that the coordinate transformation is performed using a transformation matrix. According to such a means, by causing the computer to execute the above program, it is possible to correct image distortion caused by shooting under various shooting conditions by simple image processing.
[0029]
  The object of the present invention is toA computer-readable recording medium storing a program for causing a computer to execute the image processing method of the present invention is provided.Is achieved. According to such a means, by causing the computer to execute the program, it is possible to reliably correct image distortion caused by shooting under various shooting conditions by simple image processing.
[0030]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts.
[0031]
In general, on the subject surface, straight lines parallel to each other, such as a set of boundary lines facing each other, a set of ruled lines drawn in the same direction, or a set of character strings arranged in parallel to each other In many cases, a pattern is present. Hereinafter, such a set of parallel linear patterns on the subject surface is referred to as a “parallel pattern pair”.
[0032]
The parallel pattern pair often exists in both the vertical and horizontal directions. That is, there are often two pairs of parallel pattern pairs on one subject surface. Here, a certain algebraic relationship is established between the distortion method when the two parallel pattern pairs are projected on the image and the relative orientation of the subject surface and the imaging surface. Therefore, if two or more sets of images formed by projecting parallel pattern pairs onto an image (hereinafter referred to as “image pattern pairs”) can be input or detected, the relative orientations of the subject surface and the imaging surface can be calculated. The tilt distortion correction can be performed based on the obtained value.
[0033]
The present invention achieves the above object based on such a principle. That is, according to the image processing apparatus 2 according to the embodiment of the present invention, for example, as shown in FIG. 2A, the subject surface 1 on which the letters “ABC” are written is placed on the subject surface 1. Even when the image is taken from an inclined direction, not the image 4 having the tilt distortion but the image 6 in which the distortion shown in FIG. 2B is corrected can be obtained. In the following, for the sake of simplicity, it is assumed that the shape of a planar subject to be photographed is a rectangle.
[0034]
FIG. 3 is a block diagram showing a basic configuration of the image processing apparatus according to the embodiment of the present invention. As shown in FIG. 3, the image processing apparatus according to the present embodiment includes a central processing unit (CPU) 11, a synchronous dynamic random access memory (SDRAM) 12, a hard disk drive (HDD) 13, an input An interface (input I / F) 14, a power supply 15, a display interface (display I / F) 16, a recording device 17, an external interface (external I / F) 18, a display 19, and a bus BUS are provided. .
[0035]
Here, the CPU 11, SDRAM 12, HDD 13, input I / F 14, power supply 15, display I / F 16, recording device 17, and external I / F 18 are all connected to the bus BUS. The display 19 is connected to the display I / F 16.
[0036]
In the above, the input I / F 14 includes a pointing device such as a mouse, a keyboard, buttons, and the like, and the display 19 includes a CRT. The recording device 17 is configured by a CD-RW (Compact Disk ReWritable) drive or the like, and the external I / F 18 is connected to an external device such as a digital camera or a printer or an electric communication line such as the Internet by wire or wireless. Used for.
[0037]
The SDRAM 12 is used as a work area of the CPU 11 and stores fixed information such as a processing program for executing an image processing method described below and other control programs. The processing program is loaded into the SDRAM 12 via, for example, the recording device 17, or is temporarily stored in the HDD 13 and loaded into the SDRAM 12 when necessary, or connected to the external I / F 18. Is loaded into the SDRAM 12 via.
[0038]
On the other hand, an image signal to be subjected to image processing is supplied from a device such as a digital camera connected to the recording device 17 or the external I / F 18 or an electric communication line.
[0039]
FIG. 4 is a block diagram showing the configuration of the digital camera according to the embodiment of the present invention. As shown in FIG. 4, the digital camera according to the present embodiment includes a CPU 11, SDRAM 12, input interface (input I / F) 14, power supply 15, display interface (display I / F) 16, Recording device 17, external interface (external I / F) 18, display 19, read-only memory (ROM) 20, lens 21, aperture mechanism 22, shutter 23, photoelectric conversion element 24, and drive unit 25, a preprocessing circuit 26, a motor driving unit 27, and a liquid crystal display (LCD) 28.
[0040]
Here, the CPU 11 and SDRAM 12, input I / F 14, power supply 15, display I / F 16, recording device 17, external I / F 18, ROM 20, preprocessing circuit 26, and motor drive unit 27 are connected to the bus BUS and the lens 21. The diaphragm mechanism 22 and the shutter 23 are connected to a motor drive unit 27. The photoelectric conversion element 24 and the preprocessing circuit 26 are both connected to the drive unit 25, and the LCD 28 is connected to the display I / F 16.
[0041]
In the digital camera having such a configuration, the input I / F 14 includes a pointing device such as a pen, a button installed on the camera body, and the photoelectric conversion element 24 includes a CCD or the like.
[0042]
The drive unit 25 controls the photoelectric conversion element 24 and performs analog signal processing such as amplification and clamping on the image signal obtained by the photoelectric conversion element 24 by a preamplifier, an AGC (Auto Gain Control) circuit, or the like. The conversion is performed by an analog-to-digital (A / D) converter. The preprocessing circuit 26 performs preprocessing such as automatic white balance (AWB), edge enhancement, and γ correction on the digital signal generated by the driving unit 25, and also performs image compression / decompression processing. Execute.
[0043]
Further, the motor drive unit 27 drives the lens 21, the aperture mechanism 22, and the shutter 23 in order to control focus, zoom, iris, shutter speed, and the like when imaging a subject. The ROM 20 is used as a work area of the CPU 11 and stores a processing program for executing an image processing method described below. The SDRAM 12 is used as a work area of the CPU 11 and stores fixed information such as a processing program for executing an image processing method described below and other control programs. Here, the processing program is stored in advance in the ROM 20, for example, or loaded into the SDRAM 12 through the recording device 17, or loaded into the SDRAM 12 through an electric communication line connected to the external I / F 18. On the other hand, the control program is stored in advance in the ROM 20, for example.
[0044]
Hereinafter, the image processing apparatus and the image processing method according to the present embodiment will be described in more detail.
[Embodiment 1]
FIG. 5 is a block diagram showing a configuration of the image processing apparatus according to Embodiment 1 of the present invention. As shown in FIG. 5, the image processing apparatus according to Embodiment 1 includes an imaging unit 31, a signal processing unit 32, a memory control unit 33, a main control unit 34, a frame memory 35, and an interface (I / F) 36, a display unit 37, an external storage unit 38, a parallel pattern detection unit 39, and an image distortion correction unit 40.
[0045]
Here, the imaging unit 31 is connected to the signal processing unit 32, the signal processing unit 32 is connected to the memory control unit 33, the main control unit 34, and the interface 36, and the frame memory 35 is connected to the memory control unit 33. The parallel pattern detection unit 39 is connected to the main control unit 34, the memory control unit 33, and the frame memory 35, and the image distortion correction unit 40 is connected to the parallel pattern detection unit 39 and the frame memory 35.
[0046]
The display unit 37 and the external storage unit 38 are connected to the interface 36. In the following, the correspondence relationship between the image processing apparatus according to the first embodiment shown in FIG. 5 and the embodiments shown in FIGS. 3 and 4 will be described.
[0047]
5 includes the lens 21, the diaphragm mechanism 22, the shutter 23, the photoelectric conversion element 24, the driving unit 25, and the preprocessing circuit 26 illustrated in FIG. 4, and the signal processing unit 32 and the imaging unit 31 illustrated in FIG. The memory control unit 33 is configured by the CPU 11. The main control unit 34 is composed of the CPU 11 and the ROM 20, and the frame memory 35 is composed of the SDRAM 12. The interface 36 includes a display I / F 16 and an external I / F 18, and the display unit 37 includes the display 19 or the LCD 28.
[0048]
The external storage unit 38 is constituted by the HDD 13 or the recording device 17, and more specifically, a CD-RW, a magneto-optical disk, or the like can be used. However, an image signal is transmitted via an electric communication line by using a modem card or the like. May be transmitted directly to a remote recording medium.
[0049]
The parallel pattern detection unit 39 and the image distortion correction unit 40 are configured by the CPU 11 and the input I / F 14.
[0050]
In the image processing apparatus having the above-described configuration, the signal processing unit 32 performs predetermined processing on an image captured by the imaging unit 31 or input from the outside via the interface 36. The main control unit 34 centrally controls each unit constituting the image processing apparatus, and the frame memory 35 accumulates image signals according to instructions from the memory control unit 33. The display unit 37 displays image signals supplied via the interface 36, and the external storage unit 38 reads and writes various signals such as image signals via the interface 36.
[0051]
Hereinafter, the operation of each unit shown in FIG. 5 will be described in detail. First, the signal processing unit 32 performs various image processing such as compression, expansion, color separation, white balance adjustment, and γ correction on the supplied image signal. Further, the memory control unit 33 stores the image signal processed by the signal processing unit 32 in the frame memory 35, and conversely reads out the image signal stored in the frame memory 35. In this case, the image signal read from the frame memory 35 is compressed in the signal processing unit 32 and then stored in the external storage unit 38 via the interface 36.
[0052]
When reading out the image signal stored in the external storage unit 38, the image signal is first supplied to the signal processing unit 32 via the interface 36, and the image processing unit 32 performs image decompression. On the other hand, when displaying the image signal read from the external storage unit 38 and the frame memory 35, the read signal is first subjected to digital-analog conversion (D / A conversion) in the signal processing unit 32, and amplified. Signal processing is performed. Then, the processed signal is transmitted to the display unit 37 via the interface 36.
[0053]
The parallel pattern detection unit 39 inputs or detects at least two sets of image pattern pairs that are projection images of the parallel pattern pairs on the subject surface. This operation will be described in detail later. The image distortion correction unit 40 corrects the distortion of the subject image in the image stored in the frame memory 35. This operation will be described in detail later.
[0054]
Next, the operation of the image processing apparatus shown in FIG. 5 will be described with reference to the flowchart shown in FIG. First, in step S <b> 1, the signal processing unit 32 inputs an image of the subject surface captured by the imaging unit 31 or an image separately captured in advance and supplied via the interface 36.
[0055]
In step S2, the parallel pattern detection unit 39 inputs or detects an image pattern pair that is a projection image of the parallel pattern pair on the subject surface. Each image pattern pair includes two linear patterns.
[0056]
Hereinafter, the parallel pattern detection unit 39 will be described in detail. FIG. 7 is a diagram illustrating the configuration and operation of the parallel pattern detection unit illustrated in FIG. As shown in FIG. 7, the parallel pattern detection unit 39 includes a transparent touch panel 391 installed on the surface of the display unit 37 and a pointing device 392 that specifies an image pattern pair.
[0057]
When the characters “parallel pattern input” are overlaid on the display unit 37, the user designates a linear pattern existing on the image displayed on the display unit 37 to input two sets of image pattern pairs. To do. Specifically, when the display unit 37 instructs the user to input the first pair of image patterns, the user designates the starting point position of the first linear pattern using the pointing device 392. . Then, a cross-shaped pointer and the letters “1-1” are displayed on the display unit 37, and the user is notified that the start point of the first linear pattern has been input.
[0058]
Subsequently, when the end point of the first linear pattern is designated, a cross-shaped pointer and the letters “1-2” are displayed on the display unit 37, and the end point of the first linear pattern is input. The user is notified.
[0059]
As a result, as shown in FIG. 7, a straight line 393a connecting two pointers labeled “1-1” and “1-2” is stored in the parallel pattern detection unit 39 as the first linear pattern. Is done. Similarly, when the user designates the start point and end point of the second linear pattern with the pointing device 392, the straight line 393b connecting the two pointers displayed as "2-1" and "2-2" The second linear pattern is stored in the parallel pattern detection unit 39.
[0060]
As described above, the input to the parallel pattern detection unit 39 of a pair of image patterns composed of the first and second linear patterns is completed. The second and subsequent image pattern pairs are input to the parallel pattern detection unit 39 by the same operation.
[0061]
Moreover, the parallel pattern detection part 39 shown by FIG. 5 is good also as what memorize | stores the linear pattern which comprises this character string sequentially, when a user designates one point on a character string as follows. That is, such a parallel pattern detection unit 39 is designated when the user designates the vicinity of a linear pattern such as a character string shown on the display unit 37 by the pointing device 392 as shown in FIG. A linear pattern near the point is automatically detected and a straight line 393a indicating the first linear pattern is displayed. At this time, as shown in FIG. 8, a number “1” is displayed around the straight line 393a to notify the user that the input of the first linear pattern is completed.
[0062]
Here, a method for automatically detecting a linear pattern will be briefly described. When the linear pattern is a straight line, for example, a differential operator such as a Sobel filter is applied to the image. As a result, an edge image is generated in which the portion where the differential value is greater than the threshold value is white and the small portion is black, and the straight line is automatically generated by integrating the edge segments near the specified point in the edge image. Detected.
[0063]
If the linear pattern is a character string, for example, a rectangle or local integration process as shown in Japanese Patent No. 2895122 is used to cut out a row or column near the specified point and automatically detect the character string. can do.
[0064]
Further, as shown in FIG. 8, when the user designates a point near the second linear pattern with the pointing device 392, a straight line 393b indicating the second linear pattern and a number “2” are displayed. The user is notified that the second linear pattern has been input. Then, the input of the first pair of image patterns is completed by the operation as described above. By repeating the same operation, two or more pairs of image patterns are input to the parallel pattern detection unit 39.
[0065]
Next, in step S3 shown in FIG. 6, the image distortion correction unit 40 calculates a tilt distortion correction parameter for correcting the tilt distortion of the image supplied to the signal processing unit 32. In step S4, the parameter is corrected. Is used to generate an image in which tilt distortion is corrected (hereinafter also referred to as “distortion corrected image”).
[0066]
Hereinafter, the configuration and operation of the image distortion correction unit 40 illustrated in FIG. 5 will be described in detail. The image distortion correction unit 40 corrects the distortion of the image based on two sets of image pattern pairs input to or detected by the parallel pattern detection unit 39. Here, in order to correct the distortion of the image, there is a method of performing local enlargement or reduction on the image based on the angle formed by the linear patterns constituting each image pattern pair. However, an example in which the tilt distortion is corrected using projective transformation will be described here.
[0067]
FIG. 9 is a block diagram showing a configuration of the image distortion correction unit 40 shown in FIG. As shown in FIG. 9, the image distortion correction unit 40 includes a vanishing point detection unit 401, a projection conversion unit 402, and a coordinate conversion unit 403. Here, the vanishing point detection unit 401 is connected to the parallel pattern detection unit 39 and the frame memory 35, and the projection conversion unit 402 is connected to the vanishing point detection unit 401. The coordinate conversion unit 403 is connected to the projective conversion unit 402.
[0068]
In the following, it is assumed that the optical system of the imaging unit 31 shown in FIG. 5 is approximated by a central projection model (or perspective transformation model) having a focal length f as shown in FIG. The operation of each part shown in FIG. As shown in FIG. 10, the “center projection model” refers to a straight line connecting the optical center o and the target point PO when an image of the target point PO is formed on the image plane 3 having the image center CI. 3 is a model in which a point Q (x, y) that intersects 3 is an image of the target point PO. The point Q is expressed by a line-of-sight vector EV.
[0069]
The vanishing point detection unit 401 illustrated in FIG. 9 detects each intersection point in the two sets of image pattern pairs obtained by the parallel pattern detection unit 39. The vanishing point detection unit 401 is supplied with image parameters from the frame memory 35. Here, the “image parameter” refers to an imaging unit at the time of image capturing such as the coordinates of the point (image center CI) where the image plane 3 and the optical axis (z-axis) shown in FIG. 31 optical parameters are designated.
[0070]
On the other hand, when the optical system of the imaging unit 31 is a central projection model as described above, the theorem that parallel lines in real space intersect at one point in the image plane holds, and the intersection is called a “vanishing point” Yes.
[0071]
For example, a point P as shown in FIG.₁And point P₂, Point Q₂, Point Q₁By capturing a rectangular object surrounded by the image processing device 2 from an oblique direction, a corresponding point p as shown in FIG.₁And point p₂, Point q₂, Point q₁A case where an image surrounded by is obtained will be described.
[0072]
In this case, as shown in FIG. 11A, in the real space, the point P₁And point P₂And a line connecting₁And point Q₂A straight line connecting points and a point P₁And point Q₁And a line connecting the point P₂And point Q₂The straight lines connecting are parallel to each other.
[0073]
However, as shown in FIG.₁And point p₂And a point q₁And point q₂The line connecting the two points intersects at the vanishing point V1, and the point p₁And point q₁And a line connecting the points p₂And point q₂Intersects with the straight line connecting at the vanishing point V2. The vanishing point coordinates can be obtained by a general method such as obtaining a linear equation for each of the two linear patterns constituting the image pattern pair and solving the obtained two linear equations simultaneously. .
[0074]
Further, the projective transformation unit 402 shown in FIG. 9 performs the distortion in the subject image based on the coordinates of the two vanishing points obtained by the vanishing point detecting unit 401 and the image parameters supplied from the vanishing point detecting unit 401. Projective transformation for correction is obtained. In the projective transformation, the point (x, y) is mapped to the point (x ′, y ′) by the following equation (1).
[0075]
[Expression 1]

Here, nine unknowns b in the above equation (1)₁To b₉Can be summarized as the following matrix B.
[0076]
[Expression 2]

The matrix B is referred to as a projective transformation matrix. When this projective transformation matrix B is used, the projective transformation represented by the above formula (1) can be represented by the following formula (3).
[0077]
[Equation 3]

In the above formula (3), s represents a constant for normalizing the third element on the left side to 1. Here, as is clear from the above formulas (1) and (3), the matrix B has a degree of freedom of scale multiple, so the substantial degree of freedom is 8. In the following, a procedure for obtaining a projective transformation matrix for correcting tilt distortion of a subject image based on projective geometry will be described in detail.
[0078]
In projective geometry, it is known that it is convenient to use a three-dimensional vector (x, y, z) obtained by adding a constant z to a point (x, y) on the image plane. Hereinafter, this three-dimensional vector will be referred to as an extension vector. In particular, when the value of the constant z matches the focal length f, the extension vector indicates the direction of the line of sight corresponding to the point (x, y) on the image plane. Therefore, in the following, a three-dimensional vector (x, y, f) obtained by adding a focal length f to a point (x, y) on the image plane is referred to as a line-of-sight vector.
[0079]
Here, the line-of-sight vectors of the vanishing point V1 and the vanishing point V2 shown in FIG. 11B are set as the following equations (4) and (5), respectively.
[0080]
[Expression 4]

In projective geometry, only when the third element z of the extension vector is the focal length f, a straight line with a direction vector m in space is a vanishing point with a line-of-sight vector m on the image plane. The theorem holds. That is, the direction in the space of the parallel pattern pair corresponding to the image pattern pair having the vanishing point V1 is the line-of-sight vector v.₁The direction in the space of the parallel pattern pair corresponding to the image pattern pair having the vanishing point V2 is represented by the line-of-sight vector v.₂It is represented by Therefore, the direction of the subject surface is the line-of-sight vector v₁And line-of-sight vector v₂And can be represented by a vector n shown by the following equation (6).
[0081]
[Equation 5]

In Equation (6), N is a vector (v₁× v₂) Is normalized to a unit vector, and an operation represented by the following expression (7) is performed on an arbitrary vector v.
[0082]
[Formula 6]

In Equation (7), | v | indicates the norm of the vector v.
[0083]
Next, the calculation procedure of the projective transformation matrix B for correcting tilt distortion will be described in detail. Here, in the present embodiment, a parameter for correcting tilt distortion is calculated based on the relationship between the direction of the optical axis and the direction of the subject surface in the imaging unit 31. Specifically, as shown in FIG. 12, a plane having the vector n calculated by the above equation (6) as a unit normal vector, that is, a plane parallel to the subject plane, is used as the projection plane 101, and the projection target. A distortion-corrected image 102 is generated by projecting (projective transformation) an image plane 100 having a subject image including tilt distortion toward the plane 101.
[0084]
Hereinafter, a method for calculating parameters for correcting tilt distortion will be described. First, as shown in FIG. 13, a rotation matrix R representing coordinate transformation that matches the z-axis forming the optical axis of the imaging unit 31 with the unit normal vector n of the projection surface 101 is obtained. In this case, the relationship shown in the following formula (8) is established.
[0085]
[Expression 7]

There are many rotation matrices R that satisfy the above equation (8). Here, the rotation matrix R is defined by the following equation (9).
[0086]
[Equation 8]

However, the matrix R_xAnd matrix R_yIs represented by the following equation (10).
[0087]
[Equation 9]

Here, the rotation matrix represented by the above equations (9) and (10) can be obtained by rotating the xyz coordinate system in the following order as shown in FIG. It corresponds to the operation to convert to. That is, first, the xyz coordinate system is rotated around the y axis by an angle β. The coordinate system obtained by such rotation is expressed as x₁y₁z₁When the coordinate system is used, this x₁y₁z₁Coordinate system x₁Rotate around the axis by an angle α.
[0088]
Here, the vector n is expressed as the following equation (11).
[0089]
[Expression 10]

And if said Formula (8) and Formula (9) are used, a rotation angle will be derived | led-out like following Formula.
[0090]
## EQU11 ##

Here, the matrix R can be uniquely determined by substituting the obtained rotation angle into the equations (9) and (10).
[0091]
Next, an arbitrary point (x, y, f) on the image plane is converted to a point (x ′, y ′, z ′) on the projection plane using the rotation matrix R obtained by the procedure as described above. Convert coordinates to. This coordinate transformation can be expressed by a projective transformation represented by the following equation (14).
[0092]
[Expression 12]

Here, k is an enlargement coefficient indicating the distance from the optical center o of the image capturing unit 31 to the projection surface 101, and therefore k indicates a coefficient indicating the size of the distortion correction image to be created. S is a constant for setting the z ′ coordinate of the vector after projective transformation to k. Then, the projection transformation matrix B for correcting the tilt distortion can be obtained by setting the matrix B of the equation (3) to the matrix represented by the following equation (15) by the above procedure.
[0093]
[Formula 13]

Then, the coordinate conversion unit 403 illustrated in FIG. 9 generates a distortion-corrected image by performing coordinate conversion on the subject image based on the projection conversion matrix B calculated by the projection conversion unit 402. More specifically, a point (x, y) before conversion corresponding to the point (x ′, y ′) after projective transformation is calculated by the equation (3), and the calculated coordinates (x, y ), The pixel value at the point (x ′, y ′) constituting the distortion corrected image is determined by interpolation calculation.
[0094]
The interpolation calculation can be performed using a method such as a bilinear interpolation method or a B-spline interpolation method.
[0095]
In the above, when the vanishing point is an infinite point, that is, when two linear patterns constituting the image pattern pair are parallel, the orientation of the linear pattern on the image plane is (m_x, M_y), 0 is added with the gaze vector of the vanishing point as the third element (m_x, M_y, 0).
[0096]
Even when the focal length f is unknown, an extended vector (x, y, f ′) is generated for the point (x, y) using an appropriate value f ′, and the above procedure is executed. The distortion of the subject image can be corrected. However, in this case, since the above theorem that holds between the direction vector in the real space and the vanishing point on the image plane does not hold, a distortion-corrected image in which the aspect ratio of the original subject plane does not coincide is actually obtained. . Therefore, by generating the line-of-sight vector (x, y, f) using image parameters such as the focal length and the image center, a distortion-corrected image having a subject image similar to the original subject surface can be obtained.
[0097]
As described above, according to the image processing apparatus and the image processing method according to the first embodiment of the present invention, the subject imaged under any imaging condition can be obtained by obtaining the orientation of the subject surface using the two image pattern pairs. Even in the case of an image, an appropriate image can be obtained by simply correcting the distortion of the subject image. Further, by using the image parameter, it is possible to generate an image that is similar to the subject surface and in which the tilt distortion is corrected.
[0098]
Furthermore, by using the projective transformation as described above in correcting the tilt distortion, the correction with high reliability can be realized with a small processing cost.
[Embodiment 2]
FIG. 15 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 2 of the present invention. As shown in FIG. 15, the image processing apparatus according to the second embodiment has the same configuration as the image processing apparatus according to the first embodiment shown in FIG. The difference is that a composite image is input from the outside and the imaging unit 31 is not provided.
[0099]
That is, the image processing apparatus according to the second embodiment corrects the distortion of the composite image. Here, as shown in FIG. 16A, the composite image is obtained by photographing a part of the subject surface 10 in a divided manner from a plurality of viewpoints D1 to Dn by the image processing device 200, and obtaining a plurality of images. These images 4a, 4b and 4n are generated by pasting together.
[0100]
More specifically, as shown in FIG. 16B, a composite image 41 is obtained by pasting each of the divided images 4a and 4b on the image 4n on the basis of a certain image, for example, the image 4n. Is generated. In the image processing apparatus 200 according to the second embodiment, the distortion of the composite image 41 is corrected, and a distortion corrected image 60 shown in FIG. 16C is obtained.
[0101]
Hereinafter, the operation of the image processing apparatus according to the second embodiment will be described with reference to the flowchart of FIG. First, in step S <b> 1, a composite image generated in advance via the interface 36 is input and stored in the frame memory 35. When the input of the composite image is completed, the parallel pattern detection unit 39 inputs or detects two sets of image pattern pairs that are projection images of the parallel pattern pairs on the subject surface in step S2.
[0102]
Next, in step S3, the image distortion correction unit 40 calculates a tilt distortion correction parameter for correcting the tilt distortion of the input composite image, and in step S4, the composite image obtained by correcting the tilt distortion using the parameter. Is generated.
[0103]
Here, when correcting the distortion in the composite image by the same method as the image processing method according to the first embodiment, the image parameter of the composite image needs to be known. It is possible to know. For example, when a composite image is generated by combining divided images obtained by capturing a part of the subject surface using projective transformation, each divided image is converted and pasted so as to match the reference image.
[0104]
The composite image generated by such a method is an image obtained by photographing the subject surface using a wide-angle optical system that enlarges the image surface when the reference image is photographed. It is equivalent. That is, the focal length and the image center of the composite image coincide with the focal length and the image center of the image serving as a reference for pasting.
[0105]
Therefore, if the image parameters of the image used as a reference for pasting are known, the focal length and the image center of the composite image can be obtained. By applying the image processing method according to Embodiment 1 above, It is possible to correct tilt distortion of the subject image.
[0106]
As described above, according to the image processing apparatus and the image processing method according to Embodiment 2 of the present invention, a composite image in which the tilt distortion is conspicuous because the divided images are combined by inputting or detecting two image pattern pairs. The tilt distortion can be corrected and an appropriate composite image can be obtained.
[0107]
Further, according to the image processing apparatus and the image processing method according to the second embodiment, it is possible to generate a subject image similar to the subject surface and subjected to distortion correction by using the image parameter. Further, by using projective transformation for tilt distortion correction, it is possible to realize highly reliable tilt distortion correction at a low processing cost.
[Embodiment 3]
The image processing apparatus according to the third embodiment of the present invention has the same configuration as that of the image processing apparatus according to the first embodiment shown in FIG. 5, but the function of the image distortion correction unit 40 is different. .
[0108]
That is, the image processing apparatus according to the third embodiment inputs or detects an image pattern pair that is a projection image of two or more pairs of parallel pattern pairs that intersect at a known angle on the subject surface. Here, when the angle formed on the object plane by the two parallel pattern pairs corresponding to the input image pattern pair is known, it is similar to the object plane and the distortion is corrected even if the focal length f is unknown. A subject image can be generated.
[0109]
In the following, the present embodiment will be described on the assumption that the angle formed by the two pairs of parallel patterns on the subject surface is ψ.
[0110]
FIG. 18 is a flowchart showing an image processing method according to Embodiment 3 of the present invention. As shown in FIG. 18, first, in step S <b> 1, the user images the subject surface with the imaging unit, or inputs an image captured in advance from the outside. Next, in step S2, the parallel pattern detection unit detects an image pattern pair that is a projection image of the parallel pattern pair having an angle ψ formed on the object surface, and inputs the angle ψ from the outside through an interface. The angle ψ can be set to 90 degrees as an example.
[0111]
The image distortion correction unit calculates a tilt distortion correction parameter in step S3, and creates an image in which the tilt distortion is corrected using the parameter calculated in step S4.
[0112]
The operation of the image distortion correction unit according to the third embodiment will be described below. The image distortion correction unit according to the third embodiment has the same configuration as that of the image distortion correction unit 40 according to the first embodiment shown in FIG.
[0113]
First, the vanishing point detection unit detects each vanishing point in the two sets of image pattern pairs obtained by the parallel pattern detection unit. The vanishing point coordinates can be obtained by the method described in the first embodiment.
[0114]
The projective transformation unit obtains a projective transformation matrix B for correcting tilt distortion in the subject image based on the two vanishing point coordinates obtained by the vanishing point detection unit and the angle ψ. Here, the extension vectors of the two vanishing points detected by the vanishing point detection unit are set as the following equations (16) and (17), respectively.
[0115]
[Expression 14]

In addition, z in the said Formula (16) and Formula (17) means a constant.
[0116]
Here, the extended vector v corresponding to the two vanishing points₁', V₂Since the angle formed by ′ is known to be ψ, the extension vector v is determined by the above theorem on the relationship between the direction vector in space and the vanishing point on the screen.₁', V₂Only when the third element z of ′ is the focal length f, the following equation (18) is established.
[0117]
[Expression 15]

That is, the focal length f can be calculated by solving the above equation (18).
[0118]
The focal length f can be calculated by inputting an image pattern pair that is a projection image of a parallel pattern pair intersecting at a known angle ψ by the method as described above. Then, using the focal length f thus obtained and the above formulas (4) to (15), a projective transformation matrix B for correcting tilt distortion is obtained by the same method as in the first embodiment. be able to.
[0119]
Further, the coordinate conversion unit creates a distortion corrected image by performing coordinate conversion on the image based on the projection conversion matrix B calculated by the projection conversion unit. Specifically, the coordinates (x, y) before conversion corresponding to the coordinates (x ′, y ′) after projective conversion are calculated based on Expression (3), and the calculated coordinates (x, y) Based on the pixel value in the vicinity, the pixel value at the coordinates (x ′, y ′) is determined by interpolation. This interpolation calculation can be performed using an existing method such as a bilinear interpolation method or a B-spline interpolation method.
[0120]
In most of the subject planes, at least two parallel pattern pairs such as vertical and horizontal border lines, vertical ruled lines, and horizontal ruled lines are orthogonal to each other. An image pattern pair that is a projected image of the parallel pattern pair may be detected. In this case, the focal length f may be obtained by substituting 90 degrees for the angle ψ in the equation (18), but the focal length f is obtained by the following equation (19).
[0121]
[Expression 16]

That is, according to the above equation (19), the focal length f can be obtained much more easily than the calculation according to the equation (18). Furthermore, if the image pattern pair corresponding to the parallel pattern pair orthogonal to each other is set in advance, it is not necessary to input the angle ψ separately, and the convenience can be further enhanced.
[0122]
In the above description, when either one of the vanishing points is an infinite point, the focal length f cannot be calculated. At this time, as in the first embodiment, even when the focal length f is unknown, an extension vector (x, y, f ′) is generated from the point (x, y) using an appropriate value f ′. Although the above procedure can be executed, in this case, although the distortion can be corrected, an image whose aspect ratio does not coincide with the actual subject surface is generated.
[0123]
As described above, according to the image processing apparatus and the image processing method according to Embodiment 3 of the present invention, by using two sets of image pattern pairs whose angles formed by parallel pattern pairs are known, one of the vanishing points is infinite. Unless it is a far point, even if the focal length at the time of shooting is unknown, it is possible to generate a subject image that is similar to the subject surface and that has been subjected to distortion correction.
[Embodiment 4]
The image processing apparatus according to the fourth embodiment of the present invention has the same configuration as that of the image processing apparatus according to the first embodiment shown in FIG. 5 except that the function of the image distortion correction unit 40 is different. is there. In the following, this difference will be described in detail.
[0124]
The image processing apparatus according to the fourth embodiment is characterized in that the inclination of the subject image in the image in which the tilt distortion is corrected is corrected. For example, in the first embodiment, it has been described that the distortion of the image is corrected by performing coordinate transformation using the rotation matrix R shown in Expression (14). The degree of freedom around one axis with the vector as the rotation axis remains. That is, as shown in FIG. 19A, when the image processing apparatus 2 is tilted with respect to the subject plane 1, the subject image is displayed in the image plane 3 depending on the posture of the imaging unit 31 at the time of shooting. An inclined distortion corrected image may be obtained.
[0125]
Accordingly, the image processing apparatus according to the present embodiment, as shown in FIG. 19B, is a distortion-corrected image obtained when the image processing apparatus 2 is photographed without being inclined with respect to the subject surface 1. That is, the object is to obtain a corrected image in which the subject image is not inclined in the image plane 3.
[0126]
Hereinafter, the operation of the image processing apparatus according to the fourth embodiment will be described with reference to the flowchart shown in FIG. First, in step S <b> 1, the signal processing unit 32 inputs an image of the subject surface imaged by the imaging unit 31.
[0127]
In step S2, the parallel pattern detection unit 39 inputs or detects an image pattern pair that is a projection image of the parallel pattern pair on the subject surface. Each image pattern pair includes two linear patterns.
[0128]
Next, in step S3, the direction after tilt distortion correction is designated for one of the image pattern pairs input in step S2. In step S4, the image distortion correction unit 41 described in detail below corrects the distortion of the captured image based on the image pattern pair input in step S2 and the direction specified in step S3. The parameter to be calculated is calculated. In step S5, the image distortion correction unit 41 creates an image in which the tilt distortion is corrected using the parameters calculated in step S4.
[0129]
Hereinafter, the image distortion correction unit 41 according to the fourth embodiment will be described in detail. FIG. 21 is a block diagram illustrating a configuration of the image distortion correction unit 41 according to the fourth embodiment. As shown in FIG. 21, the image distortion correction unit 41 includes a vanishing point detection unit 401, a projective conversion unit 402, a coordinate conversion unit 403, and a direction designation unit 404. Here, the vanishing point detection unit 401 is connected to the parallel pattern detection unit 39, and the projective transformation unit 402 is connected to the vanishing point detection unit 401. The coordinate conversion unit 403 is connected to the projection conversion unit 402, and the direction designation unit 404 is connected to the projection conversion unit 402.
[0130]
In the image distortion correction unit 41 having the above configuration, the vanishing point detection unit 401 detects the vanishing point in each of the two sets of image pattern pairs obtained by the parallel pattern detection unit 39. Then, the projective transformation unit 402 obtains a projective transformation matrix B for correcting tilt distortion in the subject based on the coordinates of the two vanishing points supplied from the vanishing point detection unit 401. In addition, the coordinate conversion unit 403 generates a distortion correction image by performing coordinate conversion on the image based on the projection conversion matrix B calculated by the projection conversion unit 402.
[0131]
Further, as shown in FIG. 22, the orientation designating unit 404 corrects the distortion with respect to the transparent touch panel 404a installed on the surface of the display unit 37 and one of the input image pattern pairs. And a pointing device 404b for designating the direction of the screen.
[0132]
Here, for example, as shown in FIG. 22A, after the image pattern pair is input, the characters “direction designation” are overlaid on the display unit 37. Then, the user inputs the orientation of the image pattern in the image after correcting the distortion with respect to one of the image pattern pairs input on the display unit 37 by the pointing device 404b.
[0133]
That is, for example, as shown in FIG. 22B, when the straight line 393a and the straight line 393b constituting the first image pattern pair are specified to face in the horizontal direction (horizontal direction) after tilt correction, the image The pattern pairs are corrected so as to face in the horizontal direction (horizontal direction) as shown by a straight line 393c and a straight line 393d.
[0134]
Hereinafter, an operation when such image processing is executed will be described. First, the rotation matrix R indicating the coordinate transformation for matching the optical axis (z axis) of the imaging unit 31 with the unit normal vector n of the projection surface is given by the above equation (8). Next, the unit line-of-sight vector corresponding to the vanishing point of the first image pattern pair is represented by v.₁V₁In the horizontal direction, that is, the unit vector [1, 0, 0]^T  The coordinate transformation to be converted into is expressed by the following equation (20) using the rotation matrix R.
[0135]
[Expression 17]

Here, since the rotation matrix R is an orthogonal matrix, if two sets of vectors before and after the transformation that are not on the same straight line are given, the rotation matrix R is uniquely obtained, and the following equation (21) is obtained from equations (8) and (20). .
[0136]
[Expression 18]

Where u is n and v₁Is a unit vector orthogonal to both, and is expressed by the following equation (22).
[0137]
[Equation 19]

This is because the degree of freedom around one axis of the rotation matrix R left in the calculations according to the equations (8) to (13) is constrained by defining the corrected direction in one image pattern pair. means. Then, by substituting the rotation matrix R obtained by the above calculation into Expression (15), a projective transformation matrix B for correcting the tilt distortion is obtained.
[0138]
In addition, the coordinate conversion unit 403 generates a distortion-corrected image by performing coordinate conversion on the image based on the projection conversion matrix B calculated by the projection conversion unit 402. Specifically, the coordinates (x, y) before conversion corresponding to the coordinates (x ′, y ′) after projective conversion are calculated based on Expression (3), and the calculated coordinates (x, y) Based on the pixel value in the vicinity, the pixel value at the coordinates (x ′, y ′) is determined by interpolation. The interpolation calculation may be performed using an existing method such as a bilinear interpolation method or a B-spline interpolation method.
[0139]
As described above, according to the image processing apparatus and the image processing method according to the fourth embodiment, it is possible to obtain the same effects as those in the first embodiment, and furthermore, in the image plane of the image in which the tilt distortion is corrected. Since the inclination at can be arbitrarily changed, an image with better appearance can be obtained.
[0140]
For example, when character recognition processing or the like is performed on the obtained image, the recognition accuracy is improved, so that post-processing costs from acquisition of document information to encoding can be significantly reduced.
[Embodiment 5]
The image processing apparatus according to the fifth embodiment of the present invention has the same configuration as that of the image processing apparatus according to the first embodiment shown in FIG. 5, but the function of the image distortion correction unit 40 is different. is there. In the following, this difference will be described in detail.
[0141]
Further, the image processing apparatus according to the fifth embodiment is characterized in that, as in the fourth embodiment, the inclination of the subject image in the image in which the tilt distortion is corrected is corrected.
[0142]
Hereinafter, the operation of the image processing apparatus according to the fifth embodiment will be described with reference to the flowchart of FIG. First, in step S <b> 1, the signal processing unit 32 inputs an image of the subject surface captured by the imaging unit 31 or an image captured in advance. In step S2, the parallel pattern detection unit 39 inputs or detects two sets of image pattern pairs that are projection images of the two parallel pattern pairs on the subject plane. Each image pattern pair includes two linear patterns.
[0143]
Next, the image distortion correction unit 42, which will be described in detail below, calculates a parameter for correcting the distortion of the captured image in step S3, and the image in which the distortion is corrected using the calculated parameter in step S4. Create In step S5, the inclination of the subject image in the image subjected to distortion correction is corrected.
[0144]
Hereinafter, the image distortion correction unit 42 according to the fifth embodiment will be described in detail. FIG. 24 is a block diagram showing a configuration of the image distortion correction unit 42 according to the fifth embodiment. As shown in FIG. 21, the image distortion correction unit 42 includes a vanishing point detection unit 401, a projection conversion unit 402, a coordinate conversion unit 403, and an inclination correction unit 405. Here, the vanishing point detection unit 401 is connected to the parallel pattern detection unit 39, and the projective transformation unit 402 is connected to the vanishing point detection unit 401. The coordinate conversion unit 403 is connected to the projective conversion unit 402, and the inclination correction unit 405 is connected to the coordinate conversion unit 403.
[0145]
In the image distortion correction unit 42 having the above-described configuration, the vanishing point detection unit 401 detects the vanishing point in each of the two sets of image pattern pairs obtained by the parallel pattern detection unit 39. Then, the projective transformation unit 402 obtains a projective transformation matrix B for correcting tilt distortion in the subject based on the coordinates of the two vanishing points supplied from the vanishing point detection unit 401. In addition, the coordinate conversion unit 403 generates a distortion correction image by performing coordinate conversion on the image based on the projection conversion matrix B calculated by the projection conversion unit 402.
[0146]
In addition, as shown in FIG. 25, the inclination correction unit 405 includes a transparent touch panel 405a installed on the surface of the display unit 37 and a pointing device 405b that inputs an angle for correcting the inclination. The coordinate transformation for correcting the in-plane inclination is obtained. For example, when a tilt correction mode is activated by pressing a switch (not shown) provided in the housing of the image processing apparatus according to the present embodiment, the transparent touch panel of the display unit 37 is displayed as shown in FIG. The characters “tilt correction” are overlaid on 405a. Then, the user draws an arc on the display unit 37 by the pointing device 405b, and inputs a gesture for the direction of rotation and the magnitude of the rotation for correcting the inclination of the subject image.
[0147]
When the rotation angle input in this way is θ, joint conversion from the point (x ′, y ′) to the point (x ″, y ″) in the distortion corrected image is performed by the following equation (23). calculate.
[0148]
[Expression 20]

However, the point (x₀', Y₀′) Is a coordinate serving as the center of rotation, and usually the image center is selected as the rotation center coordinate.
[0149]
Then, the calculation result of the inclination correction unit 405 is output to the coordinate conversion unit 403, and an image in which the inclination of the subject image in the distortion corrected image is corrected is created based on the joint conversion equation of Expression (23). Specifically, the coordinates (x ′, y ′) before conversion corresponding to the coordinates (x ″, y ″) after projective conversion are calculated based on the equation (23), and the calculated coordinates (x Based on the pixel value in the vicinity of ', y'), the pixel value at the coordinates (x ″, y ″) is determined by interpolation. The interpolation calculation may be performed using an existing method such as a bilinear interpolation method or a B-spline interpolation method.
[0150]
As described above, the same effect as in the fourth embodiment can be obtained also by the image processing apparatus and the image processing method according to the fifth embodiment.
[Embodiment 6]
The image processing apparatus according to the sixth embodiment of the present invention has the same configuration as that of the image processing apparatus according to the first embodiment shown in FIG. 5, but the function of the image distortion correction unit 40 is different. is there. In the following, this difference will be described in detail.
[0151]
Here, the image processing apparatus according to the sixth embodiment is characterized in that the size of the image after correcting the tilt distortion is approximately equal to the size of the image before the correction.
[0152]
Hereinafter, the operation of the image processing apparatus according to the sixth embodiment will be described with reference to the flowchart of FIG. First, in step S <b> 1, the signal processing unit 32 inputs an image of the subject surface captured by the imaging unit 31 or an image captured in advance. In step S2, the parallel pattern detection unit 39 inputs or detects two sets of image pattern pairs that are projection images of the two parallel pattern pairs on the subject plane. Each image pattern pair includes two linear patterns.
[0153]
Next, the image distortion correction unit 43, which will be described in detail below, calculates a parameter for correcting the distortion of the captured image in step S3, and in step S4, calculates the size of the image in which the distortion is corrected before the correction. The enlargement factor k is calculated so as to be approximately the same as the size of. Further, in step S5, the image distortion correction unit 43 creates an image in which the tilt distortion is corrected using the calculated parameter and the enlargement coefficient k.
[0154]
Hereinafter, the image distortion correction unit 43 according to the sixth embodiment will be described in detail. FIG. 27 is a block diagram showing a configuration of the image distortion correction unit 43 according to the sixth embodiment. As illustrated in FIG. 27, the image distortion correction unit 43 includes a vanishing point detection unit 401, a projection conversion unit 402, a coordinate conversion unit 403, and an image normalization unit 406. Here, the vanishing point detection unit 401 is connected to the parallel pattern detection unit 39, and the projective transformation unit 402 is connected to the vanishing point detection unit 401. The coordinate conversion unit 403 is connected to the projection conversion unit 402, and the image normalization unit 406 is connected between the vanishing point detection unit 401 and the coordinate conversion unit 403 in parallel with the projection conversion unit 402.
[0155]
In the image distortion correction unit 43 having the above configuration, the vanishing point detection unit 401 detects the vanishing point in each of the two sets of image pattern pairs obtained by the parallel pattern detection unit 39. Then, the projective transformation unit 402 obtains a projective transformation matrix B for correcting tilt distortion in the subject based on the coordinates of the two vanishing points supplied from the vanishing point detection unit 401. In addition, the coordinate conversion unit 403 generates a distortion correction image by performing coordinate conversion on the image based on the projection conversion matrix B calculated by the projection conversion unit 402.
[0156]
In addition, the image normalization unit 405 obtains a value of an enlargement coefficient k in Equation (14) for making an image with corrected tilt distortion substantially the same size as before correction. Here, as shown in FIG. 28, the enlargement factor k represents the distance between the optical center o of the imaging unit 31 and the projection surface 101 parallel to the subject surface, and defines the size of the distortion-corrected image to be created. .
[0157]
Therefore, as shown in FIG. 28, the enlargement factor k is expressed by the following equation using the focal length f and the angle φ formed by the optical axis (z-axis) of the imaging unit 31 and the normal vector of the projection surface 101: 24).
[0158]
[Expression 21]

Thereby, the size of the image 104 in which the tilt distortion is corrected is substantially equal to the size of the image 103 before correction, as indicated by the hatched portion shown in FIG. Here, using the unit vector of the optical axis (z-axis) of the imaging unit 31 and the normal vector n of the projection surface 101 shown by the equation (8), the relationship represented by the following equation (25) is established. To establish.
[0159]
[Expression 22]

Thereby, the said Formula (24) is deform | transformed like following Formula (26).
[0160]
[Expression 23]

Therefore, by performing the coordinate transformation represented by the equation (14) using the enlargement coefficient k calculated by the equation (26), the sizes of the images before and after correcting the distortion correction can be made substantially the same.
[0161]
Note that the method for calculating the enlargement factor k is not limited to the method using Expression (26). That is, for example, the point P shown in FIG. 29A in the image before tilt distortion correction.₁, P₂, P₃, P₄On the other hand, once the k is set to 1 and the coordinate transformation shown in Expression (14) is performed, the point P after tilt distortion correction shown in FIG.₁', P₂', P₃', P₄'Is calculated respectively. Next, the point P₁', P₂', P₃', P₄The area S 'of the circumscribed rectangle surrounded by' is obtained. Here, the point P in the image before tilt correction is corrected.₁, P₂, P₃, P₄If the area of the rectangle surrounded by S is S, the expansion factor k can also be calculated by the following equation (27).
[0162]
[Expression 24]

As described above, according to the image processing device and the image processing method according to the sixth embodiment, it is possible to obtain the same effect as that of the first embodiment, and further, it is possible to obtain an image with a desired size corrected for tilt distortion. Can be.
[Embodiment 7]
The image processing apparatus according to the seventh embodiment of the present invention has the same configuration as that of the image processing apparatus according to the first embodiment shown in FIG. 5, but the function of the image distortion correction unit 40 is different. is there. In the following, this difference will be described in detail.
[0163]
Here, also in the image processing apparatus according to the seventh embodiment, as in the sixth embodiment, the size of the image whose tilt distortion is corrected can be made substantially equal to the size of the image before the correction. It is a feature.
[0164]
The image processing apparatus according to the seventh embodiment operates in the same manner as the image processing apparatus according to the sixth embodiment. However, in step S4 shown in FIG. To set the size of the image corrected for tilt distortion. Thus, in step S5, the tilt distortion correction unit 44 according to the seventh embodiment corrects the tilt distortion based on the tilt distortion correction parameter calculated in step S3 and the size set by the user as described above. Generate an image.
[0165]
Hereinafter, the image distortion correction unit 44 according to the seventh embodiment will be described in detail. FIG. 30 is a block diagram showing a configuration of the image distortion correction unit 44 according to the seventh embodiment. As illustrated in FIG. 30, the image distortion correction unit 44 includes a vanishing point detection unit 401, a projection conversion unit 402, a coordinate conversion unit 403, and an image size setting unit 407. Here, the vanishing point detection unit 401 is connected to the parallel pattern detection unit 39, and the projective transformation unit 402 is connected to the vanishing point detection unit 401. The coordinate conversion unit 403 is connected to the projection conversion unit 402, and the image size setting unit 407 is connected between the vanishing point detection unit 401 and the coordinate conversion unit 403 in parallel with the projection conversion unit 402.
[0166]
In the above, as shown in FIG. 31, the image size setting unit 407 includes a transparent touch panel 407a installed on the surface of the display unit 37 and a pointing device 407b for setting an enlargement magnification of the image before and after correcting the distortion. The
[0167]
Here, for example, when the image size setting mode is activated by pressing a switch provided in the housing of the image processing apparatus according to the sixth embodiment, the “image size” is displayed on the display unit 37 as shown in FIG. “Setting” is displayed as an overlay. Then, the user selects the magnification represented on the display unit 37 by using the pointing device 407b, thereby inputting the magnification of the image subjected to distortion correction.
[0168]
At this time, assuming that the input magnification is g, as in the image processing method according to the fifth embodiment, the enlargement coefficient k of Equation (14) is calculated by the following Equation (28).
[0169]
[Expression 25]

Therefore, a distortion-corrected image having a desired size can be obtained by performing the coordinate transformation of Equation (14) using the value of the enlargement coefficient k calculated by Equation (28). Note that the method of calculating the enlargement factor k is not limited to the calculation by Expression (28), and can be calculated by the following Expression (29) based on the value of k in Expression (27).
[0170]
[Equation 26]

As described above, the same effect as in the sixth embodiment can also be obtained by the image processing apparatus and the image processing method according to the seventh embodiment of the present invention.
[0171]
The image processing methods according to all the above embodiments can be described by a computer program. Accordingly, the program is recorded on a computer-readable recording medium such as a CD-ROM 171 or a floppy disk, for example, and mounted on the computer PC as shown in FIG. By doing so, the image processing can be easily realized.
[0172]
As shown in FIG. 32 (b), the image processing apparatus according to all the above embodiments may be housed in a single housing as the digital camera 2a. In this case, the program is recorded. A small recording medium such as a floppy disk 172 is attached to the digital camera 2a, and the digital camera 2a itself is caused to execute the program, whereby the digital camera 2a can be provided with the image processing function.
[0173]
【The invention's effect】
As described above, according to the image processing according to the present invention, it is possible to easily correct image distortion caused by shooting under various shooting conditions, so that an appropriate image can be easily obtained.
[0174]
In addition, if the transformation matrix is obtained using at least two vanishing points, distortion of an image caused by shooting under various shooting conditions can be reliably corrected by simple image processing. Highly appropriate images can be obtained.
[0175]
Further, if the transformation matrix is calculated by using the image parameters and vanishing points of the optical system, the corrected image can be easily similar to the original image drawn on the surface of the subject. More accurate correction can be realized.
[0176]
Also, when image processing is performed on a composite image, if the conversion matrix is calculated by using the image parameters and vanishing points of the optical system that captured the image used as the reference when creating the composite image, Even for a composite image in which distortion is likely to occur, the distortion can be corrected by simple image processing.
[0177]
Also, if the transformation matrix is calculated using the angle and vanishing point formed by at least two sets of line segments on the subject, the distortion can be accurately detected even when the focal length of the optical system that photographed the subject is unknown. Therefore, the apparatus and operation for obtaining the focal length are unnecessary, and an appropriate image can be obtained more easily.
[0178]
Further, when the direction after coordinate transformation in a set of parallel line segments is designated by the user, if the transformation matrix is calculated in accordance with the direction, the orientation of the image whose distortion has been corrected is Since it can be easily changed in the plane, it is possible to obtain a more visually appealing image and to reduce the post-processing cost for the obtained image.
[Brief description of the drawings]
FIG. 1 is a diagram for explaining a conventional problem in obtaining a synthesized image.
FIG. 2 is a diagram for explaining the operation of the image processing apparatus according to the embodiment of the present invention.
FIG. 3 is a block diagram showing a basic configuration of an image processing apparatus according to an embodiment of the present invention.
FIG. 4 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention.
FIG. 5 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 1 of the present invention.
6 is a flowchart showing an operation of the image processing apparatus shown in FIG.
7 is a diagram for explaining the configuration and operation of the parallel pattern detection unit shown in FIG. 5;
FIG. 8 is a diagram for explaining another operation of the parallel pattern detection unit shown in FIG. 5;
9 is a block diagram showing a configuration of an image distortion correction unit shown in FIG.
10 is a diagram for explaining an optical system of the imaging unit shown in FIG.
FIG. 11 is a diagram for explaining the operation of the vanishing point detection unit shown in FIG. 9;
12 is a first diagram for explaining a parameter calculation operation by the image distortion correction unit shown in FIG. 5;
13 is a second diagram illustrating a parameter calculation operation by the image distortion correction unit illustrated in FIG.
14 is a third diagram illustrating a parameter calculation operation by the image distortion correction unit illustrated in FIG.
FIG. 15 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 2 of the present invention.
FIG. 16 is a diagram for explaining the operation of the image processing apparatus according to the second embodiment of the present invention;
17 is a flowchart showing the operation of the image processing apparatus shown in FIG.
FIG. 18 is a flowchart showing an operation of the image processing apparatus according to the third embodiment of the present invention.
FIG. 19 is a diagram for explaining the operation of the image processing apparatus according to the fourth embodiment of the present invention.
FIG. 20 is a flowchart showing an operation of the image processing apparatus according to the fourth embodiment of the present invention.
FIG. 21 is a block diagram showing a configuration of an image distortion correction unit according to Embodiment 4 of the present invention.
22 is a diagram for explaining the configuration and operation of the orientation specifying unit shown in FIG. 21. FIG.
FIG. 23 is a flowchart showing an operation of the image processing apparatus according to the fifth embodiment of the present invention.
FIG. 24 is a block diagram showing a configuration of an image distortion correction unit according to Embodiment 5 of the present invention.
FIG. 25 is a diagram for explaining the configuration and operation of the inclination correction unit shown in FIG. 24;
FIG. 26 is a flowchart for explaining the operation of the image processing apparatus according to the sixth embodiment of the present invention;
FIG. 27 is a block diagram showing a configuration of an image distortion correction unit according to Embodiment 6 of the present invention.
FIG. 28 is a first diagram for explaining the operation of the image distortion correction unit shown in FIG. 27;
FIG. 29 is a second diagram for explaining the operation of the image distortion correction unit shown in FIG. 27;
FIG. 30 is a block diagram showing a configuration of an image distortion correction unit according to Embodiment 7 of the present invention.
31 is a diagram for explaining the configuration and operation of the image size setting unit shown in FIG. 30;
FIG. 32 is a diagram for explaining another embodiment according to the present invention.
[Explanation of symbols]
1,10 Subject surface
2,200 image processing device
2a Digital camera
3,100 image plane
4,4a-4n, 6,41,60,103,104 images
5 Subject image
7 split images
11 Central processing unit (CPU)
12 Synchronous dynamic random access memory (SDRAM)
13 Hard disk drive (HDD)
14 Input interface (input I / F)
15 Power supply
16 Display interface (Display I / F)
17 Recording device
18 External interface (external I / F)
19 Display
20 Read-only memory (ROM)
21 Lens
22 Aperture mechanism
23 Shutter
24 photoelectric conversion element
25 Drive unit
26 Pre-processing circuit
27 Motor drive unit
28 Liquid crystal display (LCD)
31 Imaging unit
32 Signal processor
33 Memory controller
34 Main control unit
35 frame memory
36 Interface (I / F)
37 Display
38 External storage
39 Parallel pattern detector
40, 41, 42, 43, 44 Image distortion correction unit
101 Projected surface
102 Distorted image
171 CD-ROM
172 floppy disk
390 pointer
391, 404a, 405a, 407a Transparent touch panel
392, 404b, 405b, 407b Pointing device
393a-393d straight line
401 Vanishing point detector
402 Projection conversion unit
403 Coordinate converter
404 Orientation specification part
405 Tilt correction unit
406 Image normalization unit
407 Image size setting section
BUS bus
PO target points
CI image center
EV line-of-sight vector
v1, v2 vanishing point
D1-Dn viewpoint
P1, P2 position
PC computer

Claims (13)

An image processing apparatus for correcting tilt distortion of an image obtained by photographing a subject surface ,
An input unit for inputting an input image in which a subject surface including at least two parallel pattern pairs including two parallel patterns is captured;
A parallel pattern detection unit that detects an image pattern pair that is a projection image of the two parallel pattern pairs in the input image;
An image distortion correction unit that corrects tilt distortion of the input image based on the two image pattern pairs,
The parallel pattern detection unit
Receiving or detecting input of the two image pattern pairs in the input image, and further receiving input of an angle formed by the two parallel pattern pairs having the image pattern pair as a projection image on the subject surface;
The image distortion correction unit
For each of the two image pattern pairs, vanishing point detecting means for detecting coordinates of vanishing points that are intersections of the patterns constituting the image pattern pair;
A focal length calculation means for calculating a focal length in the optical system in which the subject surface is imaged based on the coordinates of the two vanishing points and the input angle;
Correction parameter calculation means for calculating a correction parameter for correcting the tilt distortion of the input image based on the coordinates of the two vanishing points and the focal length;
Generating means for generating a corrected image in which tilt distortion of the input image is corrected based on the correction parameter;
An image processing apparatus comprising: The generating means includes
The image processing apparatus according to claim 2, wherein a correction image in which tilt distortion at the time of projection onto a predetermined projection surface is corrected is generated using a transformation matrix including the correction parameter . The image processing apparatus according to claim 2, wherein the projection surface is a plane parallel to the subject surface . The transformation matrix is a projective transformation matrix;
The means for generating the corrected image uses the projective transformation matrix to generate a corrected image in which each point constituting the image is coordinate-converted to a point on the projection surface.
The image processing apparatus according to claim 2, wherein: The calculating means includes
A parameter of an enlargement ratio of the corrected image on the projection surface with respect to the input image is calculated based on the coordinates of the two vanishing points and the focal length;
The image processing apparatus according to claim 4, wherein the transformation matrix includes a parameter of the enlargement ratio . The image distortion correction unit creates the composite image when the input image is a composite image generated by joining a plurality of images obtained by partially photographing the subject surface. 6. The image processing apparatus according to claim 1, wherein the transformation matrix is calculated by using a parameter of an optical system that captures a reference image and the vanishing point. . An image processing method for correcting distortion of an image obtained by photographing a subject surface ,
An input step of inputting an input image in which a subject surface including at least two parallel pattern pairs including two parallel patterns is captured;
A parallel pattern detection step of detecting an image pattern pair which is a projection image of the two parallel pattern pairs in the input image;
An image distortion correction step of correcting a tilt distortion of the input image based on the two image pattern pairs,
The parallel pattern detection step includes
Receiving or detecting input of the two image pattern pairs in the input image, and further receiving input of an angle formed by the two parallel pattern pairs having the image pattern pair as a projection image on the subject surface;
The image distortion correction step includes:
For each of the two image pattern pairs, a vanishing point detecting step of detecting coordinates of vanishing points that are intersections of patterns constituting the image pattern pair;
A focal length calculating step of calculating a focal length in the optical system in which the subject surface is imaged based on the coordinates of the two vanishing points and the input angle;
A correction parameter calculating step for calculating a correction parameter for correcting the tilt distortion of the input image based on the coordinates of the two vanishing points and the focal length;
A generation step of generating a corrected image in which tilt distortion of the input image is corrected based on the correction parameter;
An image processing method comprising : In the generating step,
The image processing method according to claim 7, wherein a correction image in which tilt distortion at the time of projection onto a predetermined projection surface is corrected is generated using a transformation matrix including the correction parameter . The image processing method according to claim 8, wherein the projection surface is a plane parallel to the subject surface . The transformation matrix is a projective transformation matrix;
The step of generating the corrected image uses the projective transformation matrix to generate a corrected image in which each point constituting the image is coordinate-converted to a point on the projection surface.
The image processing method according to claim 8 or 9, wherein: In the calculating step,
A parameter of an enlargement ratio of the corrected image on the projection surface with respect to the input image is calculated based on the coordinates of the two vanishing points and the focal length;
The image processing method according to claim 10, wherein the transformation matrix includes a parameter of the enlargement ratio . In the image distortion correction step, in the case where the input image is a composite image generated by joining a plurality of images obtained by partially photographing the subject surface, the composite image is generated. 12. The image processing method according to claim 7, wherein the transformation matrix is calculated by using a parameter of an optical system that captures a reference image and the vanishing point. . A computer-readable recording medium storing a program for causing a computer to execute the image processing method according to claim 7 .

Images