JP3957888B2 - Imaging apparatus and captured image composition method - Google Patents

Description

【００３５】
上記のように構成した撮像装置の動作を図５のフローチャートを参照して説明する。画像撮像部１は入力対象平面Ｇｃを第１視点から撮像し、撮像した第１画像を記憶装置に記憶する（ステップＳ１，Ｓ２）。その後、ユーザが画像撮像部１を撮像した画像の一部が重複するように第２視点に移動した後に第２視点から撮像し、撮像した第２画像を記憶装置に記憶する（ステップＳ３，Ｓ４）。このように、第１画像と第２画像の一部が重複するように撮像するので、画像撮像部１が撮像した各画像の関連を容易に検出することができる。
【００３６】
運動検出部２は第１画像を撮像した際から第２画像を撮像する際までの画像撮像部１の姿勢角の変化及び視点の位置の変化を検出する（ステップＳ５）。対応関係抽出部３は第１画像から複数の特徴点Ａｉを抽出し、第２画像から対応点Ｂｉを抽出する（ステップＳ６）。これにより、第１画像と第２画像との対応関係を検出することができる。３次元位置算出部４は運動検出部２が検出した姿勢角変化及び視点位置変化並びに対応関係抽出部３が抽出した特徴点Ａｉと対応点Ｂｉを基にして各特徴点Ａｉの３次元位置（Ｘｉ１，Ｙｉ１，Ｚｉ１）を算出する（ステップＳ７）。平面算出部５は対応関係抽出部３が抽出した各特徴点Ａｉが同一平面上にあるものとして、３次元位置算出部４が計測した各特徴点Ａｉの３次元位置（Ｘｉ１，Ｙｉ１，Ｚｉ１）を基に最小自乗法を用いて各特徴点Ａｉが存在する平面の平面方程式を算出する（ステップＳ８）。投影部６は運動検出部５が検出した姿勢角変化及び視点位置変化並びに平面算出部５が算出した平面方程式を基に、画像撮像部１が撮像した各画像を任意の同一画像面Ｉｍに投影して、複数の視点から撮像した各画像を合成する（ステップＳ９）。このようにして、画像撮像部１が並進移動することによる影響により合成画像に歪が生じることを防止できるとともに、画像撮像部１の回転が合成画像の歪の原因になることを防止できる。
【００３７】
調整部７は、画像撮像部１が撮像した各画像を画像面に投影した際にその重複部分の相互相関値が最大になるように投影位置及びスケールを微調整する（ステップＳ１０）。これにより、合成した各画像にずれが生じることを防止でき、さらに正確に画像の合成を行なうことができる。
【００３８】
次ぎに、この発明の第２の実施例の撮像装置について説明する。第２の実施例の撮像装置は、図６の構成図に示すように、画像撮像部１と姿勢検出部８と対応関係抽出部３と並進運動検出部９と３次元位置算出部４と平面算出部５と投影部６と調整部７を有する。
【００３９】
姿勢検出部８は、例えば図７に示すようにＸ軸方向磁気方位センサ８１とＺ軸方向磁気方位センサ８１と姿勢角検知部８３とを有する。Ｘ軸方向磁気方位センサ８１とＺ軸方向磁気方位センサ８２はそれぞれ画像面に垂直な方向をＺ軸とした場合のＸ軸方向とＺ軸方向の磁気の検出信号を出力する。ここで検出する磁気方位は地磁気によるものでも、人工的に発生させた磁界によるものでも良い。姿勢角検知部８３はＸ軸方向磁気方位センサ８１とＺ軸方向磁気方位センサ８２からの信号を基に画像撮像部１で撮像するときの各視点における画像撮像部１の角度である姿勢角を検知する。
【００４０】
並進運動検出部９は姿勢検出部８が検出した姿勢角並びに対応関係抽出部３が抽出した特徴点Ａｉ及び対応点Ｂｉを基に画像撮像部１の並進運動成分Ｔを算出する。並進運動成分Ｔを算出するには、例えば並進運動検出部９は姿勢検出部８が検出した第１画像撮像の際の姿勢角と第２画像撮像の際の姿勢角から姿勢角の変化を算出し、図８に示すような同じ座標系で特徴点Ａｉを通る視線ベクトルＶ１ｉと対応点Ｂｉを通る視線ベクトルＶ２ｉを算出する。視線ベクトルＶ１ｉ，Ｖ２ｉと並進運動ベクトルＴのスカラーの三重積は理想的には「０」である。そこで、並進運動ベクトルＴは各特徴点Ａｉにおけるスカラーの三重積の総和である下記（２）式に示す値を最小化する値として算出する。
【００４１】
【数２】

【００４２】
３次元位置算出部４は、姿勢検出部８が検出した姿勢角と対応関係抽出部３が抽出した特徴点Ａｉと対応点Ｂｉ及び並進運動検出部９が検出した並進運動ベクトルＴを基に三角測量の原理を用いて各特徴点Ａｉの３次元位置を算出する。
【００４３】
上記のように構成した撮像装置の動作を図９のフローチャートを参照して説明する。画像撮像部１は入力対象平面Ｇｃを第１視点で撮像し、撮像した第１画像を記憶装置に記憶する（ステップＳ１１）。姿勢検出部８は画像撮像部１が第１視点から撮像した際の画像撮像部１の姿勢角を検出する（ステップＳ１２）。その後、ユーザが画像撮像部１をそれぞれ撮像した画像の一部が重複するように第２視点に移動した後に第２視点で撮像し、撮像した第２画像を記憶装置に記憶する（ステップＳ１３）。姿勢検出部８は画像撮像部１が第１視点から撮像した際と同様に画像撮像部１が第１視点から撮像した際の画像撮像部１の姿勢角を検出し、さらに第１視点から撮像した際の姿勢角と第２視点から撮像した際の姿勢角の変化を検出する（ステップＳ１４）。このように、第１画像と第２画像の一部が重複するように撮像するとともにその際の姿勢角を検出するので、画像撮像部１が撮像した各画像の関連を容易に検出することができるとともに画像撮像部１の姿勢角の変化による影響をなくすことができる。
【００４４】
対応関係抽出部３は第１画像から複数の特徴点Ａｉを抽出し、第２画像から対応点Ｂｉを抽出する（ステップＳ１５）。これにより第１画像と第２画像との対応関係を検出することができる。並進運動検出部９は姿勢検出部８が検出した姿勢角並びに対応関係抽出部３が抽出した特徴点Ａi及び対応点Ｂiを基に画像撮像部１の並進運動ベクトルＴを算出する（ステップＳ１６）。３次元位置算出部４は運動検出部２が検出した姿勢角変化と視点位置変化と対応関係抽出部３が抽出した特徴点Ａｉと対応点Ｂｉ及び並進運動検出部９が検出した並進運動ベクトルＴを基に各特徴点Ａｉの３次元位置（Ｘｉ１，Ｙｉ１，Ｚｉ１）を算出する（ステップＳ１７）。平面算出部５は対応関係抽出部３が抽出した各特徴点Ａｉが同一平面上にあるものとして、３次元位置算出部４が計測した各特徴点Ａｉの３次元位置（Ｘｉ１，Ｙｉ１，Ｚｉ１）を基に最小自乗法を用いて各特徴点Ａｉが存在する平面の平面方程式を算出する（ステップＳ１８）。投影部６は運動検出部５が検出した姿勢角変化及び視点位置変化並びに平面算出部５が算出した平面方程式を基に、画像撮像部１が撮像した各画像を任意な同一画像面Ｉｍに投影して、複数の視点から撮像した各画像を合成する（ステップＳ１９）。これにより、画像撮像部１が平行移動することによる影響により合成画像に歪が生じることを防止できるとともに、画像撮像部１の回転が合成画像の歪の原因になることを防止できる。調整部７は、画像撮像部１が撮像した各画像を画像面に投影した際にその重複部分の相互相関値が最大になるように投影位置及びスケールを微調整する（ステップＳ２０）。これにより、合成した各画像にずれが生じることを防止でき、さらに正確に画像の合成を行なうことができる。
【００４５】
上記実施例では姿勢検出部８にＸ軸方向磁気方位センサ８１とＺ軸方向磁気方位センサ８２と姿勢角検知部８３を設けた場合について説明したが、さらにＹ軸方向磁気方位センサを設けても良い。
【００４６】
また、姿勢検出部８に、例えば図１０に示すようにＸ軸方向磁気方位センサ８１とＹ軸方向磁気方位センサ８４と画像撮像部１の重力方向に対する角度を検出する重力方向検出器８５と姿勢角検知部８３を設けても良い。ここで重力方向検出器８５としては、例えばＸ軸，Ｙ軸，Ｚ軸の各軸方向の加速度を検出する加速度センサを用いて画像撮像部１の姿勢角を検出するようにしても良いし、水準器を用いて姿勢角を検出するようにしても良い。このように重力方向に対する角度を重力方向検出器８５で検出し、重力方向周りの回転角度をＸ軸方向磁気方位センサ８１とＹ軸方向磁気方位センサ８４で検出することができるので、より正確に姿勢角を検出することができる。また重力方向検出器８は、２軸方向の磁気を検出するもので良いが、大きく傾く場合には３軸回り磁気を検出するようにすると良い。
【００４７】
さらに、姿勢検出部８に、例えば図１１に示すようにＸ軸回りジャイロ８６とＹ軸回りジャイロ８７と姿勢角検知部８３を設けても良い。Ｘ軸回りジャイロ８６とＹ軸回りジャイロ８７は、Ｘ軸回りとＹ軸回りの回転角速度を検出する角速度検出部としての機能を有し、それぞれＸ軸回りの回転角速度及びＹ軸回りの回転角速度を示す信号を出力する。姿勢角検知部８３はＸ軸回りジャイロ８６とＹ軸回りジャイロ８７が出力した信号をデジタル変換した後に積分して姿勢角を検出する。このようにして磁気が不安定な場所又は磁気を検出するのが困難な場合であっても正確に画像撮像部１の姿勢角を検出することができる。
【００４８】
次ぎにこの発明の第３の実施例の撮像装置について説明する。第３の実施例の撮像装置は、図１２の構成図に示すように、画像撮像部１と位置検出部１１と対応関係抽出部３と３次元位置算出部４と平面算出部５と投影部６を有する。位置検出部１１は、図２に示すように、画像撮像部１で入力対象平面Ｇｃを第１視点Ｏｎと第２視点Ｏ（ｎ＋１）でそれぞれ撮像したときの画像面の基準座標系における位置と姿勢角を検出する。３次元位置算出部４は、位置検出部１１で検出した画像面の基準座標系における位置と姿勢角及び対応関係抽出部３で抽出した第１画像Ｇｎの特徴点Ａｉと第２画像Ｇ（ｎ＋１）の対応点Ｂｉを基に各特徴点Ａｉの３次元座標値を算出する。
【００４９】
上記のように構成された撮像装置の動作を図１３のフローチャートを参照して説明する。画像撮像部１で対象物を第１視点Ｏｎにおいて撮像したときの画像Ｇｎを記憶し、位置検出部１１で基準座標系における画像面の位置と姿勢を算出する（ステップＳ２１）。その後、画像撮像部１を第２視点Ｏ（ｎ＋１）に移動して対象物を撮像し、撮像した画像Ｇ（ｎ＋１）を記憶し、位置検出部１１で基準座標系における画像面の位置と姿勢を算出して３次元位置算出部４に送る（ステップＳ２２）。対応関係抽出部３は第１視点Ｏｎにおいて撮像した画像Ｇｎ上の特徴点Ａｉを抽出し（ステップＳ２３）、第２視点Ｏ（ｎ＋１）で撮像した画像Ｇ（ｎ＋１）の特徴点Ａｉに対応する対応点Ｂｉを抽出し３次元位置算出部４に送る（ステップＳ２４）。３次元位置算出部４は位置検出部１１で検出した各視点Ｏｎ，Ｏ（ｎ＋１）における画像面の位置と姿勢角及び対応関係抽出部３で抽出した第１画像Ｇｎの特徴点Ａｉと第２画像Ｇ（ｎ＋１）の対応点Ｂｉを基に各特徴点Ａｉの３次元座標値を算出し平面算出部５に送る（ステップＳ２５）。平面算出部５は複数の対象平面上の各特徴点Ａｉの座標値より平面方程式を算出する（ステップＳ２６）。投影部６は平面算出部５で算出した平面方程式で表せる画像面に各視点Ｏｎ，Ｏ（ｎ＋１）で撮像した画像を再投影して合成する（ステップＳ２７）。複数の視点で画像を撮像した場合は、上記処理を繰り返す（ステップＳ２８，Ｓ２９）。そして必要な場合、別の画像面へ再投影する（ステップＳ３０）。
【００５０】
このようにして複数の視点で分割した撮像した画像を簡単に合成することができるとともに、歪がなく精細な画像を得ることができる。
【００５１】
上記第３の実施例の位置検出部１１を、図１４に示すように、姿勢算出部１２と並進計算部１３で構成しても良い。姿勢算出部１２は、例えば加速度センサと磁気方位センサとジャイロのいずれか又はこれらの組合せを有し、各視点における画像撮像部１の姿勢角あるいは姿勢角変化を算出する。並進計算部１３は姿勢算出部１２で算出した姿勢角から各視点で得られた画像面の基準座標系に対する姿勢角を算出し、対応関係抽出部３で抽出した第１画像Ｇｎの特徴点Ａｉと第２画像Ｇ（ｎ＋１）の対応点Ｂｉの対応関係より、図８に示す画像撮像部１の並進運動ベクトルＴを算出する。この並進計算部１３で算出した各視点で得られた画像面の基準座標系に対する姿勢角と画像撮像部１の並進運動ベクトルＴ及び対応関係抽出部３が抽出した特徴点Ａｉと対応点Ｂｉを基に３次元位置算出部４で三角測量の原理を用いて各特徴点Ａｉの３次元位置を算出する。
【００５２】
姿勢算出部１２として３軸加速度センサと２軸磁気方位センサを用いた場合、図１５に示すように、重力方向をＹ軸とした基準座標系Ｘ，Ｙ，Ｚ軸及び画像撮像部１の光軸をＷ軸と一致させた撮像部座標系Ｕ，Ｖ，Ｗ軸を定義し、基準座標系Ｘ，Ｙ，Ｚ軸に対する撮像部座標系Ｕ，Ｖ，Ｗ軸の傾きを姿勢角θ（Ｕ），θ（Ｖ），θ（Ｗ）とする。回転の順序は、基準座標系と撮像部座標系が一致する状態からＶ（Ｙ）軸回りにθ（Ｖ）だけ回転させてから、移動した撮像部座標系を基準にＵ軸回りにθ（Ｕ）だけ回転させ、移動した撮像部座標系を基準にＷ軸回りにθ（Ｗ）だけ回転させる。それぞれの回転行列をＲ（Ｖ），Ｒ（Ｕ），Ｒ（Ｕ）として、回転行列Ｒ＝Ｒ（Ｖ）・Ｒ（Ｕ）・Ｒ（Ｗ）を下記（３）式で表す。
【００５３】
【数３】

【００５４】
また、３軸加速度センサの出力Ａと２軸磁気方位センサの出力Ｍを下記（４）式で表す。
【００５５】
【数４】

【００５６】
ただし、２軸磁気方位センサを用いる場合Ｍ（Ｙ）は不定である。また、重力加速度ベクトルＧは下記（５）式で表せる。
【００５７】
【数５】

【００５８】
ここで地磁気の伏角φを既知とすると、地磁気べクトルＤは下記（６）式となる。
【００５９】
【数６】

【００６０】
ここで回転行列Ｒと３軸加速度センサの出力Ａと重力加速度ベクトルＧは下記（７）式の関係があるから（８）式が得られる。
【００６１】
【数７】

【００６２】
（８）式より姿勢角θ（Ｕ），θ（Ｗ）は下記（９）式で得られる。（９）式においてcosθ（Ｕ）が０の場合、θ（Ｗ）は任意に定めて良い。
【００６３】
【数８】

【００６４】
さらに、回転行列Ｒと２軸磁気方位センサの出力Ｍと地磁気べクトルＤは下記（１０）式の関係があるから（１１）式が得られる。
【００６５】
【数９】

【００６６】
（１１）式より下記（１２）式が算出され、（１２）式より姿勢角θ（Ｖ）は（１３）式で得られる。
【００６７】
【数１０】

【００６８】
このようにして姿勢算出部１２により３軸加速度センサと２軸磁気方位センサの出力から基準座標系Ｘ，Ｙ，Ｚ軸に対する画像撮像部座標系Ｕ，Ｖ，Ｗ軸の傾きを姿勢角θ（Ｕ），θ（Ｖ），θ（Ｗ）を算出することができる。
【００６９】
次ぎに、姿勢算出部１２として３軸加速度センサと３軸磁気方位センサの出力から基準座標系Ｘ，Ｙ，Ｚ軸に対する画像撮像部座標系Ｕ，Ｖ，Ｗ軸の傾きを姿勢角θ（Ｕ），θ（Ｖ），θ（Ｗ）を算出する場合を示す。この場合、姿勢角θ（Ｕ），θ（Ｗ）は上記（９）式で得られる。ここで地磁気の伏角は未知として良く、地磁気ベクトルＤを下記（１４）とする。
【００７０】
【数１１】

【００７１】
ここでベクトルＤ１を下記（１５）とおくと（１６）式が得られる。
【００７２】
【数１２】

【００７３】
（１６）式より（１７）式が得られる。
【００７４】
【数１３】

【００７５】
（１７）式より姿勢角θ（Ｖ）が下記（１８）式で得ることができる。
【００７６】
【数１４】

【００７７】
また、位置検出部１１を、図１６に示すように、運動検出部１４と位置姿勢計算部１５で構成しても良い。運動検出部１４は、例えば加速度センサとジャイロからなり、画像撮像部１が視点を変えて撮像するときに、画像撮像部１が移動する過程における姿勢角変化と並進運動成分を検出する。位置姿勢計算部１５は検出した画像撮像部１の姿勢角変化と並進運動成分から各視点で得られた画像面の基準座標系に対する位置と姿勢を算出する。この算出した各視点で得られた画像面の基準座標系に対する位置と姿勢及び対応関係抽出部３が抽出した特徴点Ａｉと対応点Ｂｉを基に３次元位置算出部４で三角測量の原理を用いて各特徴点Ａｉの３次元位置を算出する。
【００７８】
さらに、図１７に示すように、運動計算部１６で基準座標系に対する各画像面の位置と姿勢を算出するようにしても良い。この場合、運動計算部１６は対応関係抽出部３が抽出した特徴点Ａｉと対応点Ｂｉの対応関係より視点間の画像撮像部１の位置変化と姿勢角変化を求め、求めた位置変化と姿勢角変化より基準座標系における各画像面の位置と姿勢を算出する。この算出した各視点で得られた画像面の基準座標系に対する位置と姿勢及び対応関係抽出部３が抽出した特徴点Ａｉと対応点Ｂｉを基に３次元位置算出部４で三角測量の原理を用いて各特徴点Ａｉの３次元位置を算出する。
【００７９】
また、上記各実施例で投影部６において各撮像画像を任意の位置と姿勢の画像面に再投影後、その位置を微調整することにより合成画像の歪みを軽減することができる。すなわち、被写体は一部が重なるよう撮像されているので、再投影される画像面において、その領域は重複して投影される。そこで第１視点から投影されるこの重複部画像と第２視点から投影される重複部画像の（１）式に示す相関関数が最大となる微小量δｘ、δｙを求め、第２視点から投影される全画像に対して、δｘ、δｙだけ平行移動させる。ここで位置の微調整は相互相関値を用いるものに限られず、サブピクセルレベルの位置合わせを行っても良い。
【００８０】
次ぎにこの発明の他の実施例の撮像装置について説明する。この実施例の撮像装置は、図１８の構成図に示すように、画像撮像部１と３次元位置検出部１７と平面算出部５ａと投影部６ａ及び再調整部１８を有する。画像撮像部１は、図２（ａ）に示すように，平面状の被写体１０を視点を変えて撮像する。この視点を変えて撮像するときに、撮像する画像の一部が必ずしも重なり合っていなくても良い。３次元位置検出部１７は、例えば超音波やレーザーなどを用いた測距センサからなり、視点を変えて撮像したときに撮像対象物平面の複数の特徴点の距離を計測して各特徴点の３次元位置を検出する。平面算出部５ａは３次元位置検出部１７で検出した各特徴点の３次元位置から、最小自乗法などにより、被写体１０の面の空間上での平面方程式を算出する。ただし、この場合、算出した平面は各視点を基準とした平面となり、視点の数だけ存在し、図１９に示すように、視点Ｏｎと視点Ｏ（ｎ＋１）で撮像したとき、視点Ｏｎを基準にした平面Ｇｃｎと視点Ｏ（ｎ＋１）を基準とした平面Ｇｃ（ｎ＋１）を算出する。投影部６ａは視点Ｏｎで撮像した画像Ｇｎを平面Ｇｃｎへ再投影し、視点Ｏ（ｎ＋１）で撮像した画像Ｇ（ｎ＋１）を平面Ｇｃ（ｎ＋１）へ再投影する。再調整部１８は画像Ｇｎを投影した平面Ｇｃｎと画像Ｇ（ｎ＋１）を投影した平面Ｇｃ（ｎ＋１）の位置と姿勢を調整して合成画像を得る。この調整方法としては、画像Ｇｎと画像Ｇ（ｎ＋１）に重なる部分がある場合は、それらが適合するように調整し、重なる部分がほとんどない場合は、画像Ｇｎと画像Ｇ（ｎ＋１）が滑らかにつながるよう調整する。投影部６ａは調整して合成画像を、必要ならば任意の仮想的な視点の画像面Ｇｐへ投影して新たな画像を得る。
【００８１】
上記のように構成した撮像装置の動作を図２０のフローチャートを参照して説明する。第１視点において画像撮像部１で対象物を撮像したときの画像Ｇｎを記憶し、３次元位置検出部１７で撮像対象物平面の複数の特徴点の距離を計測して、各特徴点の３次元位置を求める。平面算出部５ａは各特徴点の３次元位置により画像撮像部１に対する対象平面Ｇｃｎの位置と姿勢を示す平面方程式を算出する（ステップＳ３１）。投影部６ａは算出した平面Ｇｃｎに撮像した画像Ｇｎを投影する（ステップＳ３２）。次ぎに第２視点において画像撮像部１で対象物を撮像したときの画像Ｇ（ｎ＋１）を記憶し、３次元位置検出部１７で撮像対象物平面の複数の特徴点の距離を計測し、各特徴点の３次元位置を求める。平面算出部５ａは各特徴点の３次元位置から画像撮像部１に対する対象平面Ｇｃ（ｎ＋１）の位置と姿勢を示す平面方程式を算出する（ステップＳ３３，Ｓ３４）。投影部６ａは算出した平面Ｇｃ（ｎ＋１）に撮像した画像Ｇ（ｎ＋１）を投影する（ステップＳ３５）。再調整部１８は画像Ｇｎを投影した平面Ｇｃｎと画像Ｇ（ｎ＋１）を投影した平面Ｇｃ（ｎ＋１）の位置と姿勢を調整して合成画像を得る（ステップＳ３６）。この処理を視点を変えて撮像するたびに繰り返し、所定の各視点で撮像と処理が終了したら（ステップＳ３７）、投影部６ａは合成した画像を別の画像面へ再投影する（ステップＳ３８）。
【００８２】
このように各視点で複数の特徴点の距離を直接検出して対象平面の位置と姿勢を定めるから、対象平面を算出する処理の高速化を図ることができる。
【００８３】
上記実施例は３次元位置検出部１７に超音波やレーザーなどを用いた測距センサを使用した場合について説明したが、画像撮像部１の画像面に対象物平面上の特徴点が合焦したときのレンズと画像面の位置関係から特徴点の距離を算出するようにしても良い。
【００８４】
また、図２１に示すように、３次元位置検出部１７とともに姿勢算出部１２を設け、３次元位置検出部１７で検出した各特徴点の３次元位置と姿勢算出部１２で得られた画像撮像部１の姿勢から基準座標系における平面方程式を算出することにより、対象平面の位置と姿勢を示す平面方程式を算出するとき、姿勢角の調整を行う必要がなく処理の高速化を図ることができる。
【００８５】
この場合の動作を図２２のフローチャートを参照して説明する。第１視点において、画像撮像部１で対象物を撮像したときの画像Ｇｎを記憶し、３次元位置検出部１７で撮像対象物平面の複数の特徴点の距離を計測して、各特徴点の３次元位置を求め、姿勢算出部１２で画像撮像部１の基準座標に対する姿勢を算出する。平面算出部５ａは各特徴点の３次元位置から、画像撮像部１に対する対象平面の位置と姿勢を示す平面方程式を算出する（ステップＳ４１）。この平面方程式を画像撮像部１の基準座標に対する姿勢をもとに基準座標へ変換する（ステップＳ４２）。投影部６ａは変換した平面方程式で示す平面に画像Ｇｎを投影する（ステップＳ４３）。次ぎに第２視点において画像撮像部１で対象物を撮像したときの画像Ｇ（ｎ＋１）を記憶し、３次元位置検出部１７で撮像対象物平面の複数の特徴点の距離を計測して、各特徴点の３次元位置を求め、姿勢算出部１２で画像撮像部１の基準座標に対する姿勢を算出する。平面算出部５ａは各特徴点の３次元位置から、画像撮像部１に対する対象平面の位置と姿勢を示す平面方程式を算出し（ステップＳ４４，Ｓ４５）、平面方程式を画像撮像部１の基準座標に対する姿勢をもとに基準座標へ変換する（ステップＳ４６）。投影部６ａは変換した平面方程式で示す平面に画像Ｇ（ｎ＋１）を投影する（ステップＳ４７）。再調整部１８は画像Ｇｎを投影した平面と画像Ｇ（ｎ＋１）を投影した平面の位置と姿勢を調整して合成画像を得る（ステップＳ４８）。この処理を視点を変えて撮像するたびに繰り返し、各視点で撮像と処理が終了したら（ステップＳ４９）、投影部６ａは合成した画像を別の画像面へ再投影する（ステップＳ５０）。
【００８６】
【発明の効果】
この発明は以上説明したように、入力対象平面をそれぞれ一部が重複するようにして複数の視点から撮像した際の姿勢角を算出し、算出した姿勢角と各視点で撮像した画像の対応関係から視点を変えたときの並進移動成分を検出し、姿勢角と特徴点と対応点及び並進運動成分を基に各特徴点に相当する入力対象平面上の３次元位置を算出して撮像した入力対象平面の各画像を任意の同一画像面に投影するので、姿勢角の変化による影響を取り除いて、複数の視点から撮像した画像をより正確に合成することができる。
【００８７】
また、複数の視点から撮像した画像を合成し、入力対象平面上の画像を復元するので、入力対象平面上の画像が大きい場合及び入力対象平面を有する対象物を移動できない場合であっても画像を撮像し復元をすることができる。
【００９１】
また、互いに直交する２軸周り又は３軸周りの角速度から撮像するときの姿勢角を検出することにより、磁気による方位検出が困難な場合であっても簡単に撮像するときの姿勢角を検出することができる。
【００９２】
また、各視点において撮像した入力対象平面の画像を投影した際に、その重複部分の相互相関値が最大になるように投影位置及びスケールを微調整するので、歪のない高精度の画像を再生することができる。
【００９３】
また、視点を変えて撮像するときに、基準座標系における画像面の位置と姿勢を算出し、各視点における画像面の位置と姿勢角及び各画像の対応関係から各特徴点の３次元座標値を算出して平面方程式を算出するから、平面状の被写体を分割撮像して合成するときに、歪みなしで合成することができ、簡単な構成で高精細な画像を得ることができる。
【００９４】
また、各視点で撮像するときの姿勢角を算出し、各視点で得られた画像面の基準座標系に対する姿勢角を算出し、算出した画像面の基準座標系に対する姿勢角と各視点で撮像した画像の対応関係から視点を変えたときの並進移動成分を算出するから、視点を変えたときの位置と姿勢を簡単に検出することができ、各特徴点の３次元座標値を算出して平面方程式を簡単に算出することができる。
【００９６】
また、視点を変えて撮像した画像の対応関係から視点間の姿勢変化と並進移動を算出することにより、撮像部の動きを検出するセンサが不要となり、装置全体の構成を簡略化できる。
【００９７】
さらに、各視点で撮像した画像を合成するときに隣接画像間で位置合わせを行うから、良質な合成画像を得ることができる。
【図面の簡単な説明】
【図１】この発明の実施例を示す構成図である。
【図２】入力対象平面に対する第１画像及び第２画像の関係を示す説明図である。
【図３】画像投影の原理を示す説明図である。
【図４】画像合成の原理を示す説明図である。
【図５】撮像装置の動作を示すフローチャートである。
【図６】第２の実施例の構成図である。
【図７】姿勢検出部の構成図である。
【図８】並進運動ベクトルを算出する原理を示す説明図である。
【図９】第２の実施例の動作を示すフローチャートである。
【図１０】重力方向検出器を有する姿勢検出部の構成図である。
【図１１】ジャイロを有する姿勢検出部の構成図である。
【図１２】第３の実施例の構成図である。
【図１３】第３の実施例の動作を示すフローチャートである。
【図１４】第４の実施例の構成図である。
【図１５】基準座標系と画像撮像部座標系を示す説明図であ。
【図１６】第５の実施例の構成図である。
【図１７】第６の実施例の構成図である。
【図１８】第７の実施例の構成図である。
【図１９】第７の実施例の画像合成を示す説明図である。
【図２０】第７の実施例の動作を示すフローチャートである。
【図２１】第８の実施例の構成図である。
【図２２】第８の実施例の動作を示すフローチャートである。
【符号の説明】
１ 画像撮像部
２ 運動検出部
３ 対応関係抽出部
４ ３次元位置算出部
５ 平面算出部
６ 投影部
７ 調整部
８ 姿勢検出部
９ 並進運動検出部
１１ 位置検出部
１２ 姿勢算出部
１３ 並進計算部
１４ 運動検出部
１５ 位置姿勢計算部
１６ 運動計算部
１７ ３次元位置検出部
１８ 再調整部[0001]
BACKGROUND OF THE INVENTION
The present invention captures an image on an input target plane a plurality of times so as to partially overlap each other, combines the captured images on one screen, and restores the captured image, and a captured image combining method, particularly an input to be captured The present invention relates to the synthesis of a high-definition flat image by reducing distortion of the divided image when the target plane is large.
[0002]
[Prior art]
There is a need for a device that can easily and accurately input and restore text or photographs on a plane. In the case of an image on a normal size paper such as A4 size or A3 size, high-accuracy image input is possible with a copying machine or a scanner. As described above, a copying machine or a scanner can read an image on a normal size paper with high accuracy, but it is large as information on a large paper, information written on a wall, information displayed on a panel, or the like. An image or an image drawn on a medium that cannot be moved cannot be input. Therefore, as an apparatus for inputting such a large image or an image drawn on a medium that cannot be moved using an electronic camera, for example, an electronic camera disclosed in Japanese Patent Laid-Open No. 6-141228 or Japanese Patent Laid-Open No. 9-90530. A panoramic image synthesizing apparatus or the like disclosed in the publication is used.
[0003]
The electronic camera disclosed in Japanese Patent Application Laid-Open No. 6-141228 forms a subject image on an image sensor via a photographing lens and a mirror, and the subject image so that a part of the current image overlaps a part of the previous image. Are intermittently captured a plurality of times and moved to a position where the current image is properly connected to the previous image. Further, the panoramic image synthesizing apparatus disclosed in Japanese Patent Application Laid-Open No. 9-90530 calculates parameters for synthesizing images, and synthesizes images based on the calculated parameters, thereby increasing the height of the divided images of the electronic camera. I'm trying to get an accurate image.
[0004]
[Problems to be solved by the invention]
However, the electronic camera disclosed in Japanese Patent Application Laid-Open No. 6-141228 requires a mirror and its driving mechanism, which increases the scale of the apparatus and is difficult to be mounted on a normal electronic camera.
[0005]
Further, when an image on a document or a panel is captured, the distance between the camera and the subject is short, so in the panorama image composition device disclosed in Japanese Patent Laid-Open No. 9-90530, the effect of perspective projection due to the translational movement of the camera. Distortion occurs in the composite image. Further, since a human holds the electronic camera, arbitrary rotation occurs in the image, which causes distortion of the composite image.
[0006]
The present invention has been made in order to eliminate such disadvantages, and an imaging apparatus and a captured image that can input a simple and high-definition planar image by reducing distortion when combining divided images. The object is to provide a synthesis method.
[0007]
[Means for Solving the Problems]
  An imaging apparatus according to the present invention isMeans for capturing an input target plane from a plurality of viewpoints so that a part thereof overlaps with the previously captured image, and an angle of the image capturing means with respect to the input target plane when each image is captured by the image capturing means; Means for detecting a certain posture angle;Means for extracting a plurality of feature points of the previously captured image from an overlapping portion between the previously captured image and the subsequently captured image, and extracting corresponding points corresponding to the feature points from the subsequently captured image;Means for detecting a translational motion component of the means for imaging based on the detected posture angle and the extracted feature points and corresponding points; the detected posture angle; and the extracted feature points and corresponding points; Based on the detected three-dimensional position of each feature point, the means for calculating the three-dimensional position of each feature point based on the detected translational movement component and the extracted feature points are on the same plane. The imaging is performed on the basis of means for calculating information on a plane in which each feature point exists on the same plane, the detected attitude angle, the translational motion component of each feature point, and the calculated plane information. Means for projecting each image onto an arbitrary same image plane and synthesizing the images.
[0012]
  The means for synthesizing the image includes the first captured image and the later captured image.OverlapMeans for finely adjusting the projection position and scale of the image so that the cross-correlation value of the image becomes the maximum, and further highly accurate image restoration without distortion is performed.
[0013]
  According to this inventionThe second imaging deviceA posture angle which is a position and an angle of an image plane of the image capturing unit when the image capturing unit captures each image so as to partially overlap the previously captured image, and the image capturing unit captures each image. And a plurality of feature points of the previously captured image from the overlapping portion of the previously captured image and the later captured image, and corresponding points corresponding to the feature points from the later captured image Of each point on the input target plane corresponding to each feature point based on the detected image plane position, posture angle change and viewpoint position change, and the extracted feature points and corresponding points. Means for calculating a three-dimensional position; means for calculating a plane equation indicating information of a plane passing through each point on the input target plane based on the calculated three-dimensional position of each point on the input target plane; Expressed by the plane equation The image plane, and means for synthesizing an image by re-projecting each image of which an image is capturedMeans for detecting an attitude angle which is a position and an angle of an image plane of the imaging means; means for calculating an attitude angle with respect to a reference coordinate system of an image plane obtained when the images are captured; and the calculation And a means for calculating a translational movement component of the image pickup means based on the correspondence angle obtained by the means for extracting the corresponding point and the attitude angle of the image plane with respect to the reference coordinate system.
[0015]
  Further, the means for detecting the attitude angle which is the position and angle of the image plane of the imaging means is a means for detecting a change in the attitude angle of the image plane detected when each image is captured and a translational movement component. And a means for calculating the position and orientation of the image plane obtained with respect to the reference coordinate system obtained when each of the images is captured based on the detected change in the orientation angle and the translational movement component.
[0016]
  Further, the means for detecting the posture angle which is the position and angle of the image plane of the image pickup means is the image pickup means based on the positional relationship between the plurality of feature points and the corresponding points obtained by the means for extracting the corresponding points. The position and orientation relative to the reference coordinate system of the image plane obtained when each image is imaged based on the orientation change and translational movement, and the calculated orientation change and translational movement of the imaging means. And means for calculating.
[0021]
  The captured image composition method of the present invention captures an input target plane so that a part of the captured image overlaps with the previously captured image,Detecting an attitude angle that is an imaging angle with respect to the input target surface when imaging each image;Extracting a plurality of feature points of the previously captured image from the overlapping portion of the previously captured image and the subsequently captured image, and extracting corresponding points corresponding to the feature points from the subsequently captured image;Based on the detected posture angle and the extracted feature points and corresponding points, a translational motion component when each image is captured is detected, and the detected posture angles, the extracted feature points and corresponding points are Based on the detected translational motion component, a three-dimensional position of each feature point is calculated, and the extracted feature points are assumed to be on the same plane, based on the calculated three-dimensional position of each feature point. Information on a plane in which each feature point exists on the same plane is calculated, and each captured image is arbitrarily selected based on the detected attitude angle, the translational motion component of each feature point, and the calculated plane information. The image is synthesized by projecting on the same image plane.
[0024]
  Between the previously captured image and the later captured image.OverlapIt is desirable to finely adjust the projection position and scale of the image so that the cross-correlation value becomes the maximum.
[0027]
DETAILED DESCRIPTION OF THE INVENTION
The imaging apparatus of the present invention includes an image imaging unit, a motion detection unit, a correspondence relationship extraction unit, a three-dimensional position calculation unit, a plane calculation unit, and a projection unit. The image capturing unit captures an input target plane from a plurality of different viewpoints, for example, two viewpoints so that a part of the image capturing unit overlaps with the previously captured image. The motion detection unit detects a change in posture angle, which is an angle of the image capturing unit at the viewpoint at the time of each imaging by the image capturing unit, and a change in the position of the viewpoint. The correspondence relationship extraction unit extracts a plurality of feature points from the previously captured image, and extracts corresponding points that correspond to the feature points extracted in the previous image in the image captured next to the image from which the feature points have been extracted. To do. The three-dimensional position calculation unit calculates the three-dimensional position of each feature point based on the posture angle change and the viewpoint position change detected by the motion detection unit, and the feature point and the corresponding point extracted by the correspondence relationship extraction unit. In order to calculate the three-dimensional position of the feature point, for example, the line of sight from the optical center of the first viewpoint to each feature point based on the posture change from the first viewpoint captured first to the second viewpoint captured next. A line-of-sight vector from the optical center of the vector and the second viewpoint to each corresponding point is calculated, and the three-dimensional of each feature point is calculated by the principle of triangulation based on the calculated line-of-sight vector and the position change of the first viewpoint and the second viewpoint. Calculate the position.
[0028]
The plane calculation unit assumes that the feature points extracted by the correspondence relationship extraction unit are on the same plane, and uses each of the feature points measured by the 3D position calculation unit, for example, using the least square method. A plane equation indicating the plane information where the feature point exists is calculated. Based on the posture angle change and viewpoint position change detected by the motion detection unit and the plane information calculated by the plane calculation unit, the projection unit projects each image captured by the image imaging unit onto an arbitrary same image plane, and The images taken from the viewpoint are synthesized and the image on the input target plane is restored.
[0029]
【Example】
FIG. 1 is a block diagram of an image pickup apparatus according to an embodiment of the present invention. As shown in the drawing, the imaging apparatus includes, for example, an image imaging unit 1, a motion detection unit 2, a correspondence relationship extraction unit 3, a three-dimensional position calculation unit 4, a plane calculation unit 5, a projection unit 6, and an adjustment unit 7. The image capturing unit 1 includes, for example, a CCD area sensor and the like, and as illustrated in FIG. 2A, the image of the input target plane Gc of the subject 10 is captured at the first viewpoint On and the second viewpoint O (n + 1), respectively. A part is imaged so as to overlap as shown in FIGS. 2 (b) and 2 (c). When one feature point A of the image Gn captured at the first viewpoint On shown in FIG. 2B is defined, the image G (n + 1) captured at the second view O (n + 1) shown in FIG. The corresponding point corresponding to the feature point A of Gn is the point B.
[0030]
The motion detection unit 2 detects a change in posture angle, which is an angle of the image imaging unit 1 with respect to the input target plane Gc at each imaging by the image imaging unit 1, and a change in viewpoint position. The correspondence extracting unit 3 extracts a plurality of feature points Ai (i is an integer, representing the order of each feature point) from the first image Gn captured at the first viewpoint On, and the second viewpoint O (n + 1). A corresponding point Bi of the second image G (n + 1) corresponding to the feature point Ai of the first image Gn is extracted from the second image G (n + 1) imaged in (1). The three-dimensional position calculation unit 4 determines the three-dimensional position (Xi1) of each feature point Ai based on the posture angle change detected by the motion detection unit 2, the viewpoint position change, and the feature point Ai and the corresponding point Bi extracted by the correspondence relationship extraction unit 3. , Yi1, Zi1). Thus, in order to calculate the three-dimensional position (Xi1, Yi1, Zi1) of the feature point Ai, the optical center of the first image plane imaged at the first viewpoint On is used as the origin, and the first viewpoint On to the second viewpoint O. Based on the posture change of the image capturing unit 1 when moving to (n + 1), the line-of-sight vector V1i from the optical center of the first viewpoint On to each feature point Ai and the corresponding point from the optical center of the second viewpoint O (n + 1) The line-of-sight vector V2i up to Bi is calculated, and the three-dimensional position (Xi1, Yi1, Zi1) of each feature point Ai is calculated by the principle of triangulation based on the calculated line-of-sight vectors V1i, V2i and the position change of each viewpoint. .
[0031]
The plane calculation unit 5 assumes that the feature points Ai extracted by the correspondence relationship extraction unit 3 are on the same plane, and determines the three-dimensional positions (Xi1, Yi1, Zi1) of the feature points Ai measured by the three-dimensional position calculation unit 4. Based on this, for example, a plane equation is calculated as information on the plane where each feature point Ai exists using the least square method. The projection unit 6 projects each image picked up by the image pickup unit 1 on an arbitrary same image plane based on the posture angle change and viewpoint position change detected by the motion detection unit 5 and the plane equation calculated by the plane calculation unit 5. Thus, the images taken from a plurality of viewpoints are combined. Here, an operation of projecting each image captured by the image capturing unit 1 onto an arbitrary same image plane will be described with reference to FIG.
[0032]
Let Pc be the point where a straight line connecting the point Pn on the image plane In obtained from a certain viewpoint and the optical center On of this viewpoint intersects the plane Gc calculated by the plane calculation unit 5. Also, Im is an image plane onto which a newly synthesized image is projected, and this optical center is Om. The pixel is matched with the point Pm where the straight line connecting the point Pc and the optical center Om intersects the image plane Im. By performing this operation on all necessary pixels, a new image is formed on the image plane Im as shown in FIG. This process is performed on each image captured by the image capturing unit 1, and each image captured by the image capturing unit 1 is projected onto an arbitrary same image plane to synthesize images captured from a plurality of viewpoints. Here, the image formed in this way may be stored in a storage device, displayed on a display device, or printed from a printing device.
[0033]
The adjustment unit 7 finely adjusts the projection position and scale so that the cross-correlation value of the overlapping portion becomes maximum when each image captured by the image capturing unit 1 is projected onto the image plane. Since the images on the input target plane Gc are picked up so as to partially overlap, the regions picked up overlappingly on the image plane Im on which the image is projected are projected redundantly. When the overlapping part image of the first image is In (x, y) and the overlapping part image of the second image is Im (x, y), the value S of the correlation function expressed by the following equation (1) is maximized. Thus, δx and δy are obtained, and all pixels of the second image are shifted by δx and δy.
[0034]
[Expression 1]

[0035]
The operation of the imaging apparatus configured as described above will be described with reference to the flowchart of FIG. The image capturing unit 1 captures the input target plane Gc from the first viewpoint, and stores the captured first image in the storage device (steps S1 and S2). Thereafter, after the user moves to the second viewpoint so that a part of the image captured by the image capturing unit 1 overlaps, the image is captured from the second viewpoint, and the captured second image is stored in the storage device (steps S3 and S4). ). In this way, since the first image and the second image are imaged so as to partially overlap, the relationship between the images captured by the image capturing unit 1 can be easily detected.
[0036]
The motion detection unit 2 detects a change in posture angle and a change in viewpoint position of the image capturing unit 1 from when the first image is captured until when the second image is captured (step S5). The correspondence extracting unit 3 extracts a plurality of feature points Ai from the first image, and extracts corresponding points Bi from the second image (step S6). Thereby, the correspondence between the first image and the second image can be detected. The three-dimensional position calculation unit 4 uses the three-dimensional position of each feature point Ai based on the posture angle change and viewpoint position change detected by the motion detection unit 2 and the feature point Ai and the corresponding point Bi extracted by the correspondence relationship extraction unit 3 ( Xi1, Yi1, Zi1) are calculated (step S7). The plane calculation unit 5 assumes that the feature points Ai extracted by the correspondence relationship extraction unit 3 are on the same plane, and the three-dimensional positions (Xi1, Yi1, Zi1) of the feature points Ai measured by the three-dimensional position calculation unit 4. The plane equation of the plane in which each feature point Ai exists is calculated using the least square method based on (step S8). The projection unit 6 projects each image picked up by the image pickup unit 1 onto an arbitrary same image plane Im based on the posture angle change and viewpoint position change detected by the motion detection unit 5 and the plane equation calculated by the plane calculation unit 5. Then, the respective images taken from a plurality of viewpoints are synthesized (step S9). In this way, it is possible to prevent the composite image from being distorted due to the effect of translational movement of the image capturing unit 1, and to prevent the rotation of the image capturing unit 1 from causing the distortion of the composite image.
[0037]
The adjustment unit 7 finely adjusts the projection position and the scale so that the cross-correlation value of the overlapping portion is maximized when each image captured by the image capturing unit 1 is projected onto the image plane (step S10). As a result, it is possible to prevent the synthesized images from being shifted, and more accurately synthesize the images.
[0038]
Next, an image pickup apparatus according to a second embodiment of the present invention will be described. As shown in the block diagram of FIG. 6, the image pickup apparatus of the second embodiment includes an image pickup unit 1, a posture detection unit 8, a correspondence relationship extraction unit 3, a translational motion detection unit 9, a three-dimensional position calculation unit 4, and a plane. The calculation unit 5, the projection unit 6, and the adjustment unit 7 are included.
[0039]
The attitude detection unit 8 includes an X-axis direction magnetic orientation sensor 81, a Z-axis direction magnetic direction sensor 81, and an attitude angle detection unit 83, for example, as shown in FIG. The X-axis direction magnetic direction sensor 81 and the Z-axis direction magnetic direction sensor 82 respectively output magnetic detection signals in the X-axis direction and the Z-axis direction when the direction perpendicular to the image plane is the Z-axis. The magnetic orientation detected here may be based on geomagnetism or an artificially generated magnetic field. The attitude angle detection unit 83 determines an attitude angle that is an angle of the image capturing unit 1 at each viewpoint when the image capturing unit 1 captures an image based on signals from the X-axis direction magnetic direction sensor 81 and the Z-axis direction magnetic direction sensor 82. Detect.
[0040]
The translational motion detection unit 9 calculates the translational motion component T of the image capturing unit 1 based on the posture angle detected by the posture detection unit 8 and the feature point Ai and the corresponding point Bi extracted by the correspondence relationship extraction unit 3. In order to calculate the translational motion component T, for example, the translational motion detection unit 9 calculates a change in posture angle from the posture angle at the time of first image capturing and the posture angle at the time of second image capture detected by the posture detection unit 8. Then, the line-of-sight vector V1i passing through the feature point Ai and the line-of-sight vector V2i passing through the corresponding point Bi are calculated in the same coordinate system as shown in FIG. The scalar triple product of the line-of-sight vectors V1i, V2i and the translational motion vector T is ideally “0”. Therefore, the translational motion vector T is calculated as a value that minimizes the value shown in the following equation (2), which is the sum of the scalar triple products at each feature point Ai.
[0041]
[Expression 2]

[0042]
The three-dimensional position calculation unit 4 is a triangle based on the posture angle detected by the posture detection unit 8, the feature point Ai extracted by the correspondence relationship extraction unit 3, the corresponding point Bi, and the translational motion vector T detected by the translational motion detection unit 9. The three-dimensional position of each feature point Ai is calculated using the principle of surveying.
[0043]
The operation of the imaging apparatus configured as described above will be described with reference to the flowchart of FIG. The image capturing unit 1 captures the input target plane Gc from the first viewpoint, and stores the captured first image in the storage device (step S11). The posture detection unit 8 detects the posture angle of the image capturing unit 1 when the image capturing unit 1 captures an image from the first viewpoint (step S12). After that, after the user moves to the second viewpoint so that a part of the images captured by the image capturing unit 1 overlaps, the user captures an image from the second viewpoint, and stores the captured second image in the storage device (step S13). . The posture detection unit 8 detects the posture angle of the image capturing unit 1 when the image capturing unit 1 captures images from the first viewpoint in the same manner as when the image capturing unit 1 captures images from the first viewpoint, and further captures images from the first viewpoint. A change in the posture angle when the image is taken and the posture angle when the image is taken from the second viewpoint is detected (step S14). As described above, since the first image and the second image are imaged so as to partially overlap and the posture angle at that time is detected, it is possible to easily detect the relationship between the images captured by the image imaging unit 1. In addition, it is possible to eliminate the influence of the change in the posture angle of the image capturing unit 1.
[0044]
The correspondence extracting unit 3 extracts a plurality of feature points Ai from the first image, and extracts corresponding points Bi from the second image (step S15). As a result, the correspondence between the first image and the second image can be detected. The translational motion detection unit 9 calculates the translational motion vector T of the image capturing unit 1 based on the posture angle detected by the posture detection unit 8 and the feature points Ai and the corresponding points Bi extracted by the correspondence relationship extraction unit 3 (step S16). . The three-dimensional position calculation unit 4 includes a posture angle change and a viewpoint position change detected by the motion detection unit 2, a feature point Ai and a corresponding point Bi extracted by the correspondence relationship extraction unit 3, and a translational motion vector T detected by the translational motion detection unit 9. Is used to calculate the three-dimensional position (Xi1, Yi1, Zi1) of each feature point Ai (step S17). The plane calculation unit 5 assumes that the feature points Ai extracted by the correspondence relationship extraction unit 3 are on the same plane, and the three-dimensional positions (Xi1, Yi1, Zi1) of the feature points Ai measured by the three-dimensional position calculation unit 4. The plane equation of the plane in which each feature point Ai exists is calculated using the least square method based on (step S18). The projection unit 6 projects each image captured by the image capturing unit 1 onto an arbitrary same image plane Im based on the posture angle change and viewpoint position change detected by the motion detection unit 5 and the plane equation calculated by the plane calculation unit 5. Then, the respective images taken from a plurality of viewpoints are synthesized (step S19). Thereby, it is possible to prevent the composite image from being distorted due to the influence of the parallel movement of the image capturing unit 1, and to prevent the rotation of the image capturing unit 1 from causing the distortion of the composite image. The adjustment unit 7 finely adjusts the projection position and the scale so that the cross-correlation value of the overlapping portion is maximized when each image captured by the image capturing unit 1 is projected onto the image plane (step S20). As a result, it is possible to prevent the synthesized images from being shifted, and more accurately synthesize the images.
[0045]
In the above embodiment, the case where the X axis direction magnetic direction sensor 81, the Z axis direction magnetic direction sensor 82, and the attitude angle detection unit 83 are provided in the attitude detection unit 8 has been described. good.
[0046]
Further, for example, as shown in FIG. 10, the posture detection unit 8 includes an X-axis direction magnetic orientation sensor 81, a Y-axis direction magnetic direction sensor 84, and a gravity direction detector 85 that detects an angle with respect to the gravity direction of the image capturing unit 1. An angle detector 83 may be provided. Here, as the gravity direction detector 85, for example, the attitude angle of the image capturing unit 1 may be detected using an acceleration sensor that detects acceleration in each of the X-axis, Y-axis, and Z-axis directions. The attitude angle may be detected using a level. As described above, the angle with respect to the gravity direction can be detected by the gravity direction detector 85, and the rotation angle around the gravity direction can be detected by the X-axis direction magnetic direction sensor 81 and the Y-axis direction magnetic direction sensor 84. The attitude angle can be detected. The gravitational direction detector 8 may detect the magnetism in the biaxial direction, but if it is greatly inclined, it is preferable to detect the magnetism around the three axes.
[0047]
Further, the attitude detection unit 8 may be provided with an X-axis gyro 86, a Y-axis gyro 87, and an attitude angle detection unit 83, for example, as shown in FIG. The X-axis gyro 86 and the Y-axis gyro 87 each have a function as an angular velocity detection unit that detects rotational angular velocities around the X and Y axes. The rotational angular velocity around the X axis and the rotational angular velocity around the Y axis, respectively. A signal indicating is output. The attitude angle detection unit 83 digitally converts the signals output from the X-axis gyro 86 and the Y-axis gyro 87 and then integrates them to detect the attitude angle. Thus, even when the magnetism is unstable or when it is difficult to detect the magnetism, the attitude angle of the image pickup unit 1 can be detected accurately.
[0048]
Next, an image pickup apparatus according to a third embodiment of the present invention will be described. As shown in the block diagram of FIG. 12, the image pickup apparatus of the third embodiment includes an image pickup unit 1, a position detection unit 11, a correspondence relationship extraction unit 3, a three-dimensional position calculation unit 4, a plane calculation unit 5, and a projection unit. 6. As illustrated in FIG. 2, the position detection unit 11 includes the position in the reference coordinate system of the image plane when the image capturing unit 1 images the input target plane Gc at the first viewpoint On and the second viewpoint O (n + 1). Detect posture angle. The three-dimensional position calculation unit 4 includes the feature point Ai of the first image Gn and the second image G (n + 1) extracted by the position and orientation angle of the image plane detected by the position detection unit 11 and the correspondence relationship extraction unit 3. The three-dimensional coordinate value of each feature point Ai is calculated on the basis of the corresponding point Bi.
[0049]
The operation of the imaging apparatus configured as described above will be described with reference to the flowchart of FIG. The image Gn when the object is imaged at the first viewpoint On is stored by the image capturing unit 1, and the position and orientation of the image plane in the reference coordinate system are calculated by the position detection unit 11 (step S21). Thereafter, the image capturing unit 1 is moved to the second viewpoint O (n + 1) to capture the object, the captured image G (n + 1) is stored, and the position detection unit 11 stores the position and orientation of the image plane in the reference coordinate system. Is calculated and sent to the three-dimensional position calculation unit 4 (step S22). The correspondence extraction unit 3 extracts a feature point Ai on the image Gn captured at the first viewpoint On (step S23), and corresponds to the feature point Ai of the image G (n + 1) captured at the second viewpoint O (n + 1). The corresponding points Bi are extracted and sent to the three-dimensional position calculation unit 4 (step S24). The three-dimensional position calculation unit 4 includes the position and orientation angle of the image plane at each viewpoint On, O (n + 1) detected by the position detection unit 11 and the feature points Ai and second of the first image Gn extracted by the correspondence relationship extraction unit 3. Based on the corresponding point Bi of the image G (n + 1), the three-dimensional coordinate value of each feature point Ai is calculated and sent to the plane calculation unit 5 (step S25). The plane calculation unit 5 calculates a plane equation from the coordinate values of the feature points Ai on the plurality of target planes (step S26). The projection unit 6 re-projects and synthesizes an image captured at each viewpoint On, O (n + 1) on the image plane that can be represented by the plane equation calculated by the plane calculation unit 5 (step S27). When images are taken from a plurality of viewpoints, the above process is repeated (steps S28 and S29). If necessary, it is reprojected onto another image plane (step S30).
[0050]
In this way, the captured images divided from a plurality of viewpoints can be easily combined, and a fine image without distortion can be obtained.
[0051]
As shown in FIG. 14, the position detection unit 11 of the third embodiment may be configured by an attitude calculation unit 12 and a translation calculation unit 13. The posture calculation unit 12 includes, for example, any one of an acceleration sensor, a magnetic direction sensor, and a gyro, or a combination thereof, and calculates the posture angle or posture angle change of the image capturing unit 1 at each viewpoint. The translation calculation unit 13 calculates a posture angle with respect to the reference coordinate system of the image plane obtained at each viewpoint from the posture angle calculated by the posture calculation unit 12, and the feature point Ai of the first image Gn extracted by the correspondence relationship extraction unit 3. And the corresponding point Bi of the second image G (n + 1), the translational motion vector T of the image capturing unit 1 shown in FIG. 8 is calculated. The orientation angle of the image plane with respect to the reference coordinate system obtained at each viewpoint calculated by the translation calculation unit 13, the translational motion vector T of the image capturing unit 1, the feature point Ai extracted by the correspondence extraction unit 3, and the corresponding point Bi. Based on the principle of triangulation, the three-dimensional position calculation unit 4 calculates the three-dimensional position of each feature point Ai.
[0052]
When a three-axis acceleration sensor and a two-axis magnetic azimuth sensor are used as the posture calculation unit 12, as shown in FIG. 15, the reference coordinate system X, Y, Z axes with the gravity direction as the Y axis and the light of the image capturing unit 1 An imaging unit coordinate system U, V, and W axes whose axes are aligned with the W axis are defined, and the inclinations of the imaging unit coordinate systems U, V, and W axes with respect to the reference coordinate system X, Y, and Z axes are expressed as posture angles θ (U ), Θ (V), θ (W). The rotation order is determined by rotating θ (V) around the V (Y) axis from a state where the reference coordinate system and the imaging unit coordinate system coincide with each other, and then rotating around the U axis with reference to the moved imaging unit coordinate system. Rotate only by U) and rotate by θ (W) around the W axis with reference to the moved imaging unit coordinate system. The respective rotation matrices are R (V), R (U), and R (U), and the rotation matrix R = R (V) · R (U) · R (W) is expressed by the following equation (3).
[0053]
[Equation 3]

[0054]
The output A of the triaxial acceleration sensor and the output M of the biaxial magnetic azimuth sensor are expressed by the following equation (4).
[0055]
[Expression 4]

[0056]
However, M (Y) is indefinite when a biaxial magnetic azimuth sensor is used. The gravitational acceleration vector G can be expressed by the following equation (5).
[0057]
[Equation 5]

[0058]
Here, assuming that the dip angle φ of geomagnetism is known, the geomagnetic vector D is expressed by the following equation (6).
[0059]
[Formula 6]

[0060]
Here, since the rotation matrix R, the output A of the triaxial acceleration sensor, and the gravitational acceleration vector G have the relationship of the following equation (7), equation (8) is obtained.
[0061]
[Expression 7]

[0062]
The posture angles θ (U) and θ (W) can be obtained from the following equation (9) from the equation (8). In the equation (9), when cos θ (U) is 0, θ (W) may be arbitrarily determined.
[0063]
[Equation 8]

[0064]
Furthermore, since the rotation matrix R, the output M of the biaxial magnetic azimuth sensor, and the geomagnetic vector D have the relationship of the following equation (10), equation (11) is obtained.
[0065]
[Equation 9]

[0066]
The following expression (12) is calculated from the expression (11), and the posture angle θ (V) is obtained from the expression (13) from the expression (12).
[0067]
[Expression 10]

[0068]
In this way, the attitude calculation unit 12 determines the inclinations of the image imaging unit coordinate systems U, V, and W axes with respect to the reference coordinate system X, Y, and Z axes from the outputs of the 3-axis acceleration sensor and the 2-axis magnetic orientation sensor by the attitude angle θ ( U), θ (V), θ (W) can be calculated.
[0069]
Next, the inclination of the image capturing unit coordinate system U, V, and W axes with respect to the reference coordinate system X, Y, and Z axes is calculated from the outputs of the triaxial acceleration sensor and the triaxial magnetic azimuth sensor as the posture calculating unit 12 by the posture angle θ (U ), Θ (V), θ (W) are calculated. In this case, the posture angles θ (U) and θ (W) are obtained by the above equation (9). Here, the dip angle of geomagnetism may be unknown, and the geomagnetic vector D is defined as (14) below.
[0070]
## EQU11 ##

[0071]
Here, when the vector D1 is set as the following (15), the equation (16) is obtained.
[0072]
[Expression 12]

[0073]
Equation (17) is obtained from Equation (16).
[0074]
[Formula 13]

[0075]
The posture angle θ (V) can be obtained from the following equation (18) from the equation (17).
[0076]
[Expression 14]

[0077]
Further, as shown in FIG. 16, the position detection unit 11 may be configured by a motion detection unit 14 and a position / orientation calculation unit 15. The motion detection unit 14 includes, for example, an acceleration sensor and a gyro, and detects a change in posture angle and a translational motion component in the process of moving the image capturing unit 1 when the image capturing unit 1 changes the viewpoint. The position / orientation calculation unit 15 calculates the position and orientation of the image plane obtained from each viewpoint from the detected orientation angle change and translational motion component of the image capturing unit 1 with respect to the reference coordinate system. Based on the feature points Ai and corresponding points Bi extracted by the position and orientation of the image plane obtained from each viewpoint and the corresponding relationship extracting unit 3 and the corresponding points Bi, the principle of triangulation is calculated by the three-dimensional position calculating unit 4. Using this, the three-dimensional position of each feature point Ai is calculated.
[0078]
Further, as shown in FIG. 17, the motion calculation unit 16 may calculate the position and orientation of each image plane with respect to the reference coordinate system. In this case, the motion calculation unit 16 obtains a change in position and posture angle of the image capturing unit 1 between viewpoints from the correspondence between the feature points Ai and the correspondence points Bi extracted by the correspondence extraction unit 3, and obtains the obtained position changes and postures. The position and orientation of each image plane in the reference coordinate system are calculated from the angle change. Based on the feature points Ai and corresponding points Bi extracted by the position and orientation of the image plane obtained from each viewpoint and the corresponding relationship extracting unit 3 and the corresponding points Bi, the principle of triangulation is calculated by the three-dimensional position calculating unit 4. Using this, the three-dimensional position of each feature point Ai is calculated.
[0079]
  In each of the above embodiments, the projection unit 6 can re-project each captured image onto an image plane having an arbitrary position and orientation, and then finely adjust the position to reduce distortion of the composite image. That is, since the subject is imaged so as to partially overlap, the region is projected in an overlapping manner on the image plane to be reprojected. Therefore, the overlapping image projected from the first viewpoint and the second viewpoint are projected.Equation (1) for overlapping imageThe minute amounts δx and δy that maximize the correlation function shown in FIG. 6 are obtained, and the entire images projected from the second viewpoint are translated by δx and δy. Here, the fine adjustment of the position is not limited to the one using the cross-correlation value, and the sub-pixel level alignment may be performed.
[0080]
Next, an image pickup apparatus according to another embodiment of the present invention will be described. As shown in the block diagram of FIG. 18, the imaging apparatus of this embodiment includes an image imaging unit 1, a three-dimensional position detection unit 17, a plane calculation unit 5a, a projection unit 6a, and a readjustment unit 18. As shown in FIG. 2A, the image capturing unit 1 captures an image of a planar subject 10 by changing the viewpoint. When taking an image while changing this viewpoint, a part of the image to be taken may not necessarily overlap. The three-dimensional position detection unit 17 includes a distance measuring sensor using, for example, an ultrasonic wave or a laser, and measures the distance between a plurality of feature points on the imaging target plane when taking an image while changing the viewpoint. A three-dimensional position is detected. The plane calculation unit 5a calculates a plane equation in the space of the surface of the subject 10 from the three-dimensional position of each feature point detected by the three-dimensional position detection unit 17 by the least square method or the like. However, in this case, the calculated plane is a plane based on each viewpoint, and there are as many viewpoints as there are, and as shown in FIG. The plane Gc (n + 1) based on the plane Gcn thus obtained and the viewpoint O (n + 1) is calculated. The projection unit 6a reprojects the image Gn captured at the viewpoint On to the plane Gcn, and reprojects the image G (n + 1) captured at the viewpoint O (n + 1) onto the plane Gc (n + 1). The readjustment unit 18 adjusts the position and orientation of the plane Gcn on which the image Gn is projected and the plane Gc (n + 1) on which the image G (n + 1) is projected to obtain a composite image. As an adjustment method, if there is an overlapping portion between the image Gn and the image G (n + 1), the adjustment is made so that they match. If there is almost no overlapping portion, the image Gn and the image G (n + 1) are smooth. Adjust to be connected. The projection unit 6a adjusts and projects the composite image onto an image plane Gp at an arbitrary virtual viewpoint if necessary, thereby obtaining a new image.
[0081]
The operation of the imaging apparatus configured as described above will be described with reference to the flowchart of FIG. An image Gn obtained when the image capturing unit 1 captures an image of the object at the first viewpoint is stored, and the distance between a plurality of feature points on the image capturing object plane is measured by the three-dimensional position detection unit 17, and 3 of each feature point is measured. Find the dimension position. The plane calculation unit 5a calculates a plane equation indicating the position and orientation of the target plane Gcn with respect to the image capturing unit 1 based on the three-dimensional position of each feature point (step S31). The projection unit 6a projects the captured image Gn on the calculated plane Gcn (step S32). Next, an image G (n + 1) when the object is imaged by the image capturing unit 1 at the second viewpoint is stored, and the distance between a plurality of feature points on the imaged object plane is measured by the three-dimensional position detecting unit 17. The three-dimensional position of the feature point is obtained. The plane calculation unit 5a calculates a plane equation indicating the position and orientation of the target plane Gc (n + 1) with respect to the image capturing unit 1 from the three-dimensional position of each feature point (steps S33 and S34). The projection unit 6a projects the captured image G (n + 1) on the calculated plane Gc (n + 1) (step S35). The readjustment unit 18 adjusts the position and orientation of the plane Gcn on which the image Gn is projected and the plane Gc (n + 1) on which the image G (n + 1) is projected to obtain a composite image (step S36). This process is repeated each time the viewpoint is changed for imaging, and when imaging and processing are completed at each predetermined viewpoint (step S37), the projection unit 6a re-projects the synthesized image onto another image plane (step S38).
[0082]
As described above, since the distance and the posture of the target plane are determined by directly detecting the distances of the plurality of feature points at each viewpoint, it is possible to speed up the process of calculating the target plane.
[0083]
In the above embodiment, the case where a distance measuring sensor using an ultrasonic wave or a laser is used for the three-dimensional position detection unit 17 has been described. However, the feature point on the object plane is focused on the image plane of the image pickup unit 1. The distance between the feature points may be calculated from the positional relationship between the lens and the image plane.
[0084]
In addition, as shown in FIG. 21, the posture calculation unit 12 is provided together with the three-dimensional position detection unit 17, and the three-dimensional position of each feature point detected by the three-dimensional position detection unit 17 and the image obtained by the posture calculation unit 12. By calculating a plane equation in the reference coordinate system from the posture of the unit 1, when calculating a plane equation indicating the position and posture of the target plane, it is not necessary to adjust the posture angle, and the processing speed can be increased. .
[0085]
The operation in this case will be described with reference to the flowchart of FIG. At the first viewpoint, the image Gn when the image capturing unit 1 captures the object is stored, and the three-dimensional position detection unit 17 measures the distances between a plurality of feature points on the image capturing object plane, The three-dimensional position is obtained, and the posture calculation unit 12 calculates the posture of the image capturing unit 1 with respect to the reference coordinates. The plane calculation unit 5a calculates a plane equation indicating the position and orientation of the target plane with respect to the image capturing unit 1 from the three-dimensional position of each feature point (step S41). This plane equation is converted into reference coordinates based on the attitude of the image capturing unit 1 with respect to the reference coordinates (step S42). The projection unit 6a projects the image Gn on the plane indicated by the converted plane equation (step S43). Next, the image G (n + 1) when the object is imaged by the image capturing unit 1 at the second viewpoint is stored, and the distances of the plurality of feature points on the imaging object plane are measured by the three-dimensional position detection unit 17, The three-dimensional position of each feature point is obtained, and the posture calculation unit 12 calculates the posture of the image capturing unit 1 with respect to the reference coordinates. The plane calculation unit 5a calculates a plane equation indicating the position and orientation of the target plane with respect to the image capturing unit 1 from the three-dimensional position of each feature point (steps S44 and S45), and the plane equation with respect to the reference coordinates of the image capturing unit 1 is calculated. Based on the posture, the coordinates are converted into reference coordinates (step S46). The projection unit 6a projects the image G (n + 1) on the plane indicated by the converted plane equation (step S47). The readjustment unit 18 adjusts the position and orientation of the plane on which the image Gn is projected and the plane on which the image G (n + 1) is projected to obtain a composite image (step S48). This process is repeated each time the viewpoint is changed for imaging, and when the imaging and processing are completed at each viewpoint (step S49), the projection unit 6a re-projects the synthesized image onto another image plane (step S50).
[0086]
【The invention's effect】
  As described above, the present invention is obtained when images are taken from a plurality of viewpoints so that the input target planes partially overlap each other.Calculate the attitude angle, detect the translational movement component when the viewpoint is changed from the correspondence between the calculated attitude angle and the image captured at each viewpoint, and each based on the attitude angle, the feature point, the corresponding point, and the translational motion component Since the three-dimensional position on the input target plane corresponding to the feature point is calculated and each image of the input target plane that is captured is projected on the same arbitrary image plane, the influence of the change in the attitude angle is removed, and from a plurality of viewpoints The captured images can be synthesized more accurately.
[0087]
In addition, since the images captured from a plurality of viewpoints are combined and the image on the input target plane is restored, the image is displayed even when the image on the input target plane is large and the object having the input target plane cannot be moved. Can be imaged and restored.
[0091]
In addition, by detecting the posture angle when imaging from the angular velocities around two or three axes orthogonal to each other, the posture angle when imaging is easily detected even when it is difficult to detect the direction by magnetism. be able to.
[0092]
Also, when projecting an input target plane image captured at each viewpoint, the projection position and scale are fine-tuned so that the cross-correlation value of the overlapped portion is maximized, so that high-precision images without distortion are reproduced. can do.
[0093]
In addition, when imaging is performed with different viewpoints, the position and orientation of the image plane in the reference coordinate system are calculated, and the three-dimensional coordinate value of each feature point is calculated from the position and orientation angle of the image plane at each viewpoint and the correspondence between the images. Therefore, when a plane subject is divided and imaged and synthesized, it can be synthesized without distortion, and a high-definition image can be obtained with a simple configuration.
[0094]
In addition, the attitude angle when imaging from each viewpoint is calculated, the attitude angle of the image plane obtained at each viewpoint with respect to the reference coordinate system is calculated, and the attitude angle with respect to the reference coordinate system of the calculated image plane and imaging is performed at each viewpoint Because the translational movement component when the viewpoint is changed is calculated from the correspondence between the images, the position and orientation when the viewpoint is changed can be easily detected, and the three-dimensional coordinate value of each feature point is calculated. The plane equation can be easily calculated.
[0096]
Also, by calculating the posture change and translational movement between the viewpoints from the correspondence relationship of the images picked up by changing the viewpoint, a sensor for detecting the movement of the imaging unit is unnecessary, and the configuration of the entire apparatus can be simplified.
[0097]
Furthermore, since the alignment is performed between the adjacent images when the images captured from the respective viewpoints are combined, a high-quality combined image can be obtained.
[Brief description of the drawings]
FIG. 1 is a configuration diagram showing an embodiment of the present invention.
FIG. 2 is an explanatory diagram illustrating a relationship between a first image and a second image with respect to an input target plane.
FIG. 3 is an explanatory diagram showing the principle of image projection.
FIG. 4 is an explanatory diagram showing the principle of image composition.
FIG. 5 is a flowchart illustrating an operation of the imaging apparatus.
FIG. 6 is a configuration diagram of a second embodiment.
FIG. 7 is a configuration diagram of a posture detection unit.
FIG. 8 is an explanatory diagram showing the principle of calculating a translational motion vector.
FIG. 9 is a flowchart showing the operation of the second embodiment.
FIG. 10 is a configuration diagram of an attitude detection unit having a gravity direction detector.
FIG. 11 is a configuration diagram of an attitude detection unit having a gyro.
FIG. 12 is a configuration diagram of a third embodiment.
FIG. 13 is a flowchart showing the operation of the third embodiment.
FIG. 14 is a configuration diagram of a fourth embodiment.
FIG. 15 is an explanatory diagram showing a reference coordinate system and an image capturing unit coordinate system.
FIG. 16 is a block diagram of a fifth embodiment.
FIG. 17 is a block diagram of a sixth embodiment.
FIG. 18 is a block diagram of a seventh embodiment.
FIG. 19 is an explanatory diagram showing image composition in the seventh embodiment.
FIG. 20 is a flowchart showing the operation of the seventh embodiment.
FIG. 21 is a block diagram of an eighth embodiment.
FIG. 22 is a flowchart showing the operation of the eighth embodiment.
[Explanation of symbols]
1 Image capturing unit
2 motion detector
3 correspondence extraction part
4 3D position calculator
5 Plane calculation part
6 Projection unit
7 Adjustment section
8 Attitude detection unit
9 Translational motion detector
11 Position detector
12 Attitude calculation unit
13 Translation calculator
14 Motion detector
15 Position and orientation calculator
16 motion calculator
17 3D position detector
18 Readjustment section

Claims (7)

Means for capturing an input target plane from a plurality of viewpoints so that a part of the image captured previously overlaps;
Means for detecting an attitude angle that is an angle of the image capturing unit with respect to the input target surface when each image is captured by the image capturing unit;
Means for extracting a plurality of feature points of the previously captured image from an overlapping portion between the previously captured image and the subsequently captured image, and extracting corresponding points corresponding to the feature points from the subsequently captured image;
Means for detecting a translational motion component of the means for imaging based on the detected attitude angle and the extracted feature points and corresponding points;
Means for calculating a three-dimensional position of each feature point from the detected posture angle, the extracted feature point and corresponding point, and the detected translational motion component;
Means for calculating information on a plane in which each feature point exists on the same plane based on the three-dimensional position of each calculated feature point, assuming that each extracted feature point is on the same plane;
Means for projecting the captured images on any same image plane based on the detected posture angle, the translational motion component of each feature point, and the calculated plane information, and combining the images;
An imaging apparatus comprising: The means for synthesizing the image includes means for finely adjusting a projection position and a scale of the image so that a cross-correlation value of an overlapping portion between the image captured earlier and an image captured later is maximized. The imaging device described. Means for imaging the input target plane so that a part of the previously captured image overlaps;
Means for detecting a posture angle which is a position and an angle of an image plane of the image capturing means when each image is captured by the image capturing means;
Means for extracting a plurality of feature points of a previously captured image from an overlapping portion between the previously captured image and a later captured image, and extracting corresponding points corresponding to the feature points from the subsequently captured image;
A three-dimensional position of each point on the input target plane corresponding to each feature point is calculated based on the detected image plane position, posture angle change, viewpoint position change, and the extracted feature points and corresponding points. Means,
Means for calculating a plane equation indicating information of a plane passing through each point on the input target plane based on the calculated three-dimensional position of each point on the input target plane;
Means for re-projecting each captured image and synthesizing the image on the image plane represented by the calculated plane equation;
The means for detecting the attitude angle, which is the position and angle of the image plane of the imaging means, calculates the attitude angle with respect to the reference coordinate system of the image plane obtained when each of the images is captured, and the calculated An image pickup apparatus comprising: an orientation angle of an image plane with respect to a reference coordinate system; and means for calculating a translational movement component of the image pickup means based on a correspondence obtained by the means for extracting the corresponding points.   Means for detecting an attitude angle which is a position and an angle of an image plane of the imaging means; means for detecting a change in the attitude angle of the image plane detected when each image is captured and a translational movement component; 4. The imaging apparatus according to claim 3, further comprising means for calculating a position and an attitude of an image plane obtained with respect to a reference coordinate system obtained when the images are taken on the basis of the detected change in the attitude angle and the translational movement component.   The means for detecting the posture angle, which is the position and angle of the image plane of the image pickup means, changes the posture of the image pickup means from the positional relationship between the plurality of feature points and the corresponding points obtained by the means for extracting the corresponding points. And calculating the position and orientation of the image plane obtained when the respective images are captured based on the calculated orientation change and translational movement of the imaging means. The imaging apparatus according to claim 4, further comprising: means. Capture the input target plane so that part of it overlaps with the previously captured image,
Detecting an attitude angle that is an imaging angle with respect to the input target surface when imaging each image;
Extracting a plurality of feature points of the previously captured image from the overlapping portion of the previously captured image and the subsequently captured image, and extracting corresponding points corresponding to the feature points from the subsequently captured image;
Detecting a translational motion component when each of the images is captured based on the detected posture angle and the extracted feature points and corresponding points;
A three-dimensional position of each feature point is calculated from the detected posture angle, the extracted feature points and corresponding points, and the detected translational motion component,
Assuming that each extracted feature point is on the same plane, the information on the plane where each feature point exists on the same plane is calculated based on the calculated three-dimensional position of each feature point;
Based on the detected posture angle, the translational motion component of each feature point, and the calculated plane information, the captured images are projected onto an arbitrary same image plane to synthesize the images. Captured image composition method. The captured image composition method according to claim 6, wherein the projection position and the scale of the image are finely adjusted so that the cross-correlation value of the overlapping portion between the image captured earlier and the image captured later is maximized.

Images