JP3683258B2 - Vehicle periphery monitoring device, image generation method, and image generation program - Google Patents

Description

【００３７】
また、対応点の画像座標値（ｕ１，ｖ１），（ｕ２，ｖ２）と、そのカメラ１の視点座標系の座標値（ｘ１，ｙ１，ｚ１）、カメラ間の位置関係Ｍ１２およびカメラパラメタｆ１，ｆ２との間には、次の関係式が成り立つ。カメラパラメタｆ１，ｆ２は、カメラの焦点距離に画素サイズと画像座標値の変換係数を掛けたものである。
【数２】

【００３８】
予め計測されたパラメタＭ１２，ｆ１，ｆ２と、算出した対応点の画像座標値（ｕ１，ｖ１）、（ｕ２，ｖ２）を（数２）に代入することによって、対応点の３次元座標（ｘ１，ｙ１，ｚ１）を算出することができる。ここで、（数２）には未知数３に対して独立した一次式が４つあることと、画像から算出した対応点の画像座標値（ｕ１，ｖ１）、（ｕ２，ｖ２）は誤差を含むと考えられることから、最小二乗法を用いて３次元座標値（ｘ１，ｙ１，ｚ１）を算出するものとする。これにより、誤差の影響を低減することができる。
【００３９】
このようにして求めた、ある点に対応する、第１のカメラ画像における画像座標値（ｕ１，ｖ１）、第２のカメラ画像における画像座標値（ｕ２，ｖ２）、および３次元位置座標値（ｘ１，ｙ１，ｚ１）をまとめた（ｕ１，ｖ１，ｕ２，ｖ２，ｘ１，ｙ１，ｚ１）を、当該点の位置情報と呼ぶ。
【００４０】
さらに、位置算出部１４は、最小二乗法で求めた３次元座標値（ｘ１，ｙ１，ｚ１）から、第１および第２のカメラ画像における画像座標値（ｕ１’，ｖ１’）、（ｕ２’，ｖ２’）を逆算する（Ｓ１３）。すなわち、３次元座標値（ｘ１，ｙ１，ｚ１）およびパラメタＭ１２，ｆ１，ｆ２を（数２）に代入して、第１および第２のカメラ画像における画像座標値を新たに求める。
【００４１】
ここで、逆算により求めた画像座標値（ｕ１’，ｖ１’），（ｕ２’，ｖ２’）は、計測誤差がない場合は、元の原画像座標値（ｕ１，ｖ１），（ｕ２，ｖ２）と一致するはずである。しかしながら、一般には、計測誤差があるため両者は一致しない。そこで、原画像座標値（ｕ１，ｖ１），（ｕ２，ｖ２）と逆算により求めた画像座標値（ｕ１’，ｖ１’），（ｕ２’，ｖ２’）との距離誤差平均ｅを用いて、次式によって、信頼度ｒを求める（Ｓ１４）。
【００４２】
【数３】

【００４３】
この信頼度ｒは、対応点の座標値に計測誤差が含まれていない場合は１になり、計測誤差が大きくなるにつれて０に近づく。すなわち、算出した対応点の位置情報（ｕ１，ｖ１，ｕ２，ｖ２，ｘ１，ｙ１，ｚ１）が誤差を含む度合を表しており、位置情報が信頼できるか否かの指標となる。
【００４４】
このような手順によって、位置算出部１４は、フレームメモリ１３ａ，１３ｂに格納された第１および第２のカメラ画像から、所定数の対応点の位置情報ＰＩ（（ｕ１ｉ，ｖ１ｉ，ｕ２ｉ，ｖ２ｉ，ｘ１ｉ，ｙ１ｉ，ｚ１ｉ）：ｉ＝１〜ｎ）とその信頼度ＲＬ（ｒｉ：ｉ＝１〜ｎ）を算出して出力する。
【００４５】
可変画像合成部１５はフレームメモリ１３ａ，１３ｂから第１および第２のカメラ画像のデータを読み出し、位置算出部１４によって算出された対応点の位置情報ＰＩを用いて、第１の合成画像ＳＩ１を生成する。ここでは、図４に示すように配置された合成カメラを仮定し、予め与えられた合成カメラの位置Ｍ１０とパラメタｆ０とを用いて、テクスチャマッピングの手法によって画像合成を行うものとする。画像データと画像中の点の３次元座標値とを用いて、任意の視点からみた画像を合成する、いわゆるテクスチャマッピングは、コンピュータグラフィックスの分野の広く知られた技術であり、ここでは詳細な説明を省略する。なお、可変画像合成部１７での画像合成はテクスチャマッピングに限定されるものではなく、例えば、合成画像において、位置情報が示す位置に、障害物を表す所定の色や図形を重畳表示する方法など、位置情報を反映させた画像合成であればよい。
【００４６】
一方、固定画像合成部１６はフレームメモリ１３ａ，１３ｂから第１および第２のカメラ画像のデータを読み出し、位置算出部１４によって算出された対応点の位置情報ＰＩを用いないで、予め定められた方法によって、第２の合成画像ＳＩ２を生成する。ここでは、図５のように、路面平面を基準にして画像合成を行うものとする。すなわち、カメラ１、カメラ２は全て路面平面を撮影していると仮定すると、カメラ１，カメラ２および合成カメラについてそれぞれ、路面に対する位置およびカメラパラメタを与えることによって、合成カメラの画像座標に対応するカメラ１またはカメラ２の画像座標を計算することができる。固定画像合成部１６は合成カメラの全ての画像座標（ｕｏｉ，ｖｏｉ）について、対応するカメラを示す値ｃｉ、およびその画像座標（ｕｉ，ｖｉ）を予め計算して参照テーブルとして記録している。そしてこの参照テーブルに従って、合成画像の各座標（ｕｏｉ，ｖｏｉ）に対応する画素データ（ｃｉ，ｕｉ，ｖｉ）をフレームメモリ１３ａ，１３ｂから順次読み出し、合成画像を生成する。このような路面平面を基準にした画像合成については、例えば国際公開第００／０７３７３号パンフレットに開示されており、ここでは詳細な説明は省略する。
【００４７】
図６は図２の状況において、可変画像合成部１５が生成した第１の合成画像ＳＩ１の例であり、（ａ）は位置算出部１４によって対応点の位置情報ＰＩが正しく算出された場合、（ｂ）は位置情報ＰＩに誤差が多く含まれた場合である。１Ａは車両１を表す画像である。図６（ａ）のように、第１の合成画像ＳＩ１は、元のカメラ画像中の物体の３次元位置が維持されたまま視点が変換されているため、図２（ｂ）（ｃ）のようなカメラ画像と比べて、障害物ＯＢまでの距離が分かりやすいという特徴がある。一方、図６（ｂ）のように、位置情報ＰＩが誤差を多く含む場合は、障害物ＯＢの位置や形状が正確でなくなる、という特徴もある。
【００４８】
これに対して図７は図２の状況において、固定画像合成部１６が生成した第２の合成画像ＳＩ２の例である。図７のように、第２の合成画像ＳＩ２は、障害物ＯＢが路面と接する領域と車両との位置関係は正確に示されるものの、路面平面に対して高さのある領域は歪んで表示され、不自然になる、という特徴がある。
【００４９】
そこで合成方式選択部１７は、位置算出部１４によって算出された信頼度ＲＬに応じて、第１および第２の合成画像ＳＩ１，ＳＩ２のうちいずれか一方を、提示対象の合成画像として選択し、出力する。すなわち、信頼度ＲＬが所定の基準よりも高いと判断した場合、言い換えると、位置情報ＰＩが誤差をあまり含まない場合は、可変画像合成部１５が生成した第１の合成画像ＳＩ１を選択し、一方、信頼度ＲＬが所定の基準よりも低いと判断した場合、言い換えると、位置情報ＰＩが誤差を多く含む場合は、固定画像合成部１６が生成した第２の合成画像ＳＩ２を選択する。
【００５０】
信頼度ＲＬと所定の基準との比較は、具体的には例えば次のように行う。まず、位置算出部１４によって算出されたｎ個の対応点の信頼度ｒｉと、所定の閾値ｒｔｈとをそれぞれ比較し、閾値ｒｔｈ以上である対応点の個数ｍを算出する。そして、閾値ｒｔｈ以上の対応点の割合ｍ／ｎと所定の閾値ｍｔｈとを比較し、割合ｍ／ｎが閾値ｍｔｈよりも大きいときは、信頼度ＲＬが所定の基準よりも高いと判断して、可変画像合成部１５が生成した第１の合成画像ＳＩ１を選択する。一方、割合ｍ／ｎが閾値ｍｔｈよりも小さいときは、信頼度ＲＬが所定の基準よりも低いと判断して、固定画像合成部１５が生成した第２の合成画像ＳＩ２を選択する。もちろん、信頼度と所定の基準との比較は、このような方法に限定されるものではなく、例えば、信頼度ｒｉが閾値ｒｔｈ以上である対応点の個数を、閾値と大小比較してもよい。
【００５１】
カメラ１１ａ，１１ｂはそれぞれ，一定周期で画像を撮影する。そして、位置算出部１４、可変画像合成部１５、固定画像合成部１６および合成方式選択部１７は、カメラ画像が得られる毎に、上述した動作を繰り返し実行する。これにより、時間的に変化する車両の周辺状況に応じた合成画像がディスプレイ２０に表示される。
【００５２】
運転者はディスプレイ２０に表示された合成画像を見て、これを参考にして運転を行う。例えば図２（ａ）の状況において、可変画像合成部１５は、位置算出部１４で位置情報ＰＩが正しく算出されたときは図６（ａ）の合成画像を、位置情報ＰＩが正しく算出されなかったときは図６（ｂ）の合成画像を生成し、固定画像合成部１６は、図７の合成画像を生成する。ここで、図６（ａ）の画像は図７と比べて、障害物ＯＢの歪みが無く自然である。ところが図６（ｂ）の画像は、運転者に障害物ＯＢの位置について誤解を与える可能性が高いため、図７の画像の方が好ましい、といえる。
【００５３】
そこで本実施形態では、合成方式選択部１７が、信頼度ＲＬを用いて位置情報ＰＩが正しく算出されたか否かを判断し、位置情報ＰＩが正しく算出されたときは可変画像合成部１５が生成した図６（ａ）の合成画像を選択し、位置情報ＰＩが誤差を多く含む場合は固定画像合成部１６が生成した図７の合成画像を選択して出力する。この結果、運転者にとってより安全で、かつ自然な画像の提示が実現される。
【００５４】
以上のように本実施形態によると、画像中の物体の位置情報とその信頼度を算出する位置算出部と、位置情報を用いてカメラ画像を変形加工して第１の合成画像を生成する可変画像合成部と、位置情報を用いないで、予め定められた手順でカメラ画像を変形加工して第２の合成画像を生成する固定画像合成部とを設けて、位置情報の信頼度を参照して、位置情報が正しく算出されたと判断したときは第１の合成画像を選択し、位置情報が誤差を多く含む可能性があると判断したときは第２の合成画像を選択する。これにより、合成画像において、障害物が誤った位置に表示されたり、障害物が表示されないということがなく、かつ状況に応じて歪みの少ない合成画像を表示することができるので、運転者にとってより安全で、より自然な画像の提示が可能になる。
【００５５】
なお、本実施形態では、可変画像合成部１５および固定画像合成部１６が生成する第１および第２の合成画像は、その構図について特に限定はないものとしたが、可変画像合成部１５および固定画像合成部１６は、第１および第２の合成画像において、路面の位置が互いに一致するように、画像合成を行うのが好ましい。これにより、画像合成方式が頻繁に切り替わったときでも、合成画像が見にくくなることがなく、運転者が車両周辺の状況を把握しやすくなる。
【００５６】
例えば、撮影状況によっては、処理を繰り返す毎に位置情報の信頼度ＲＬが大きく変動し、第１の合成画像と第２の合成画像とが頻繁に切り替えられる場合がある。このような場合、第１および第２の合成画像の構図が大きく異なっていると、運転者が車両周辺の状況を把握しづらくなる。そこで、可変画像合成部１５と固定画像合成部１６とで、画像合成における合成カメラの位置と内部パラメタを予め一致させておくことによって、図６および図７のように、路面がほぼ同じ位置に示された合成画像を生成することができる。この結果、合成画像が頻繁に切り替わっても、見にくくなることはない。
【００５７】
また本実施形態では、位置情報の信頼度に応じて画像合成方式を選択するものとしたが、選択のためのしきい値にヒステリシスを持たせることによって、信頼度が早い周期で微小変動した場合などに、画像合成方式の頻繁な切替を抑制することができる。同様に、所定時間が経過するまでは画像合成方式の切替が生じないよう制約を加えることによっても、画像合成方式の頻繁な切替を抑制することができる。これにより、合成画像が見やすくなり、運転者が車両周辺の状況を把握しやすくなる。
【００５８】
また本実施形態では、位置情報の信頼度に応じて画像合成方式を選択するものとしたが、さらに位置情報自体も参照して、画像合成方式を選択するようにしてもよい。例えば、合成方式選択部１７は、位置情報に含まれる各対応点の３次元位置および画像座標値を参照して、可変画像合成部１５および固定画像合成部１６における画像合成に用いられる領域に属する点を抽出する。そして、抽出した点に係る信頼度を、合成画像の選択に用いる。具体的には例えば、抽出した点の個数が所定値以上のときは、抽出した点の信頼度を用いて、上述した手順によって画像合成方式を選択する。一方、抽出した点の個数が所定値未満のときは、固定画像合成部１６によって生成された第２の合成画像ＳＩ２を選択する。これにより、画像合成に用いられない領域に属する対応点の信頼度が、画像合成方式の選択に考慮されなくなるので、より適切な合成画像の提示が可能になる。
【００５９】
また本実施形態では、カメラは、図２のように車両１後方を撮影するように、かつ、水平位置が異なるように設置されているものとしたが、撮影範囲やカメラ設置位置を限定するものではなく、カメラの撮影範囲が重複するように配置されていれば、その重複領域に対して本実施形態と同様な処理を行うことができる。
【００６０】
また本実施形態では、対応点の位置情報の信頼度は、画像座標を基にして（数３）により求めるものとしたが、信頼度の算出はこれに限定されるものではなく、対応点の位置情報に含まれる誤差の大きさや誤りの可能性を示す尺度であればどのような基準値を用いてもよい。
【００６１】
また本実施形態では、可変画像合成部１５は、位置算出部１４によって算出されたｎ個の対応点の位置情報を用いて画像合成を行うものとしたが、ｎ個の対応点のうち信頼度が所定の閾値以上である点の位置情報のみを選択して用いてもよい。これにより、信頼度の低い位置情報が排除されるので、合成画像における合成誤りを低減できる。
【００６２】
また本実施形態では、可変画像合成部１５と固定画像合成部１６はそれぞれ１個ずつとしたが、画像合成部の個数はこれに限定されるものではない。例えば、可変画像合成部および固定画像合成部のうち少なくとも一方を複数個設けて、合成方式選択部１７が位置情報の信頼度に応じて、各画像合成部がそれぞれ生成した合成画像のうち１つを選択するようにしてもよい。
【００６３】
また本実施形態では、位置算出部１４は、２個のカメラで同時に撮影された画像を用いて位置を算出する、いわゆる２眼ステレオ方式を利用するものとしたが、この代わりに、時間的に連続した２個のカメラ画像を用いて位置を算出する、いわゆる動きステレオ方式を利用してもよく、本実施形態と同様の効果を得ることができる。
【００６４】
（第２の実施形態）
本発明の第２の実施形態に係る車両周辺監視装置の構成は第１の実施形態とほぼ同様であり、図１に示すとおりである。第１の実施形態と異なるのは、合成方式選択部１７Ａが、選択した合成画像に、当該合成画像の特徴に応じた所定の表示を重畳する点である。また、合成画像の選択に際し、位置情報の信頼度ＲＬと併せて、位置情報ＰＩ自体も組み合わせて用いる。
【００６５】
具体的には次のように動作する。合成方式選択部１７Ａは、位置算出部１４によって特定されたｎ個の対応点から、合成画像に現れる可能性がある障害物に属する点を抽出する。すなわち、位置情報ＰＩに含まれる３次元位置座標値および画像座標値を基準にして、路面平面から所定距離だけ離れ、かつ画像合成に用いられる領域に属する点を抽出する。
【００６６】
そして、抽出した点の個数が所定値以上である場合は、抽出した点の信頼度を用いて第１の実施形態で示した処理を行い、信頼度が所定の基準よりも高いときは可変画像合成部１５が生成した第１の合成画像ＳＩ１を選択する。このとき、障害物ＯＢの位置は正確に検出されたと判断されるので、図８（ａ）に示すように、例えば「障害物位置検出中」というメッセージを重畳する。一方、信頼度が所定の基準よりも低いときは固定画像合成部１６が生成した第２の合成画像ＳＩ２を選択し、このときは、障害物ＯＢの位置が正確には検出されなかったと判断されるので、図８（ｂ）に示すように、例えば「位置算出不可：画像に歪みがあります！」というメッセージを重畳する。
【００６７】
一方、抽出した点の個数が所定値未満である場合は、固定画像合成部１６が生成した第２の合成画像ＳＩ２を選択し、このときは車両近傍に障害物が検出されなかったと判断されるので、図８（ｃ）に示すように、例えば「障害物を検出していません」というメッセージを重畳する。
【００６８】
ここで、可変画像合成部１５が生成する第１の合成画像ＳＩ１では、図６（ａ）のように障害物の形状が正しく表示されるのに対して、固定画像合成部１６が生成する第２の合成画像ＳＩ２では、図７のように障害物の形状が歪む場合がある。このため、利用者が図７のような合成画像を見た場合、障害物は直方体状で合成画像が歪んでいるのか、障害物がつぶれた直方体状であって合成画像は歪んでいないのかが、利用者には判断できないため、障害物の形状を誤解してしまう可能性がある。例えば運転者が、図７に示された障害物は低いので車で乗り越えることができる、と誤解した場合、安全性の面で問題がある。
【００６９】
これに対して本実施形態では、図８のように、選択した合成画像に当該合成画像の特徴に応じた表示が重畳されるので、利用者は、例えば合成画像が歪みを含むことを容易に知ることができ、したがって、合成画像の性質に起因した車両周辺状況に対する誤解を避けることができる。この結果、より安全な運転が実現される。
【００７０】
（第３の実施形態）
図９は本発明の第３の実施形態に係る車両周辺監視装置の構成を示すブロック図である。図１と共通の構成要素については図１と同一の符号を付しており、ここではその詳細な説明を省略する。図１と異なるのは、車両周辺の障害物を検出してその距離情報ＤＩを出力する障害物センサ３１が車両に設けられており、合成方式選択部１７Ｂは、障害物センサ３１から得られた距離情報ＤＩを加味して、合成画像の選択を行う点である。障害物センサ３１は近くにある障害物までの距離を一定周期で計測し、所定距離以内に障害物があるとき、距離情報ＤＩを出力する。ここでは障害物センサ３１は、超音波を発信してそれが反射して戻って来るまでの時間から物体までの距離を計測する、いわゆる超音波センサであるものとする。
【００７１】
図１０は本実施形態におけるカメラおよび障害物センサの設置態様である。図１０のように、障害物センサ３１はカメラ１とカメラ２の間に設置されている。なお、障害物センサ３１とカメラ１，カメラ２との位置関係は予め計測されているものとする。
【００７２】
本実施形態における合成方式選択部１７Ｂの動作について説明する。合成方式選択部１７Ｂは、障害物センサ３１で障害物が検出されなかった場合、第１の実施形態と同様に、位置情報の信頼度ＲＬを所定の基準と比較し、その比較結果に応じて合成画像の選択を行う。一方、障害物センサ３１で障害物が検出された場合、まず、位置算出部１４によって特定されたｎ個の対応点について、障害物センサ３１からの距離ｄｉ（ｉ＝１，…，ｎ）をそれぞれ算出する。ここでの算出は、各対応点の位置情報と、障害物センサ３１とカメラ１，カメラ２との位置関係を用いて行うことができる。次に合成方式選択部１７Ｂは、算出した距離ｄｉ（ｉ＝１，…，ｎ）と、障害物センサ３１によって検出された距離情報ＤＩとを比較する。そして、その差が所定の誤差閾値dth以下の対応点が存在するか否か、すなわち、距離情報ＤＩとほぼ等しい距離ｄｉを有する対応点が存在するか否かを判断する。この処理は、距離情報ＤＩと位置情報ＰＩとを参照して、障害物ＯＢに関する位置情報ＰＩの精度を評価している、といえる。
【００７３】
そして合成方式選択部１７Ｂは、距離情報ＤＩとほぼ等しい距離ｄｉを有する対応点が存在する場合は、第１の実施形態と同様に、位置情報の信頼度ＲＬを所定の基準と比較し、その比較結果に応じて、合成画像の選択を行う。一方、距離情報ＤＩとほぼ等しい距離ｄｉを有する対応点が存在しない場合は、言い換えると、障害物ＯＢに関する位置情報ＰＩの精度が十分でないと判断されるときは、信頼度ＲＬに応じた合成画像の選択を行わないで、固定画像合成部１６が生成した第２の合成画像ＳＩ２を選択する。
【００７４】
ここで、障害物センサ３１は、近距離にある物体については比較的確実にその距離を検出できる。言い換えると、障害物センサ３１が距離情報ＤＩを出力したとき、距離情報ＤＩが示す距離の位置に障害物が実際に存在する可能性は高い。これに対して、位置算出部１４で実行される画像上での対応点探索や位置検出では、特に障害物の輝度や色合いが均一である場合や、障害物が背景と似ている場合などに、位置情報に誤差が多く含まれたり、障害物上の対応点自体が算出されなかったりする可能性がある。
【００７５】
位置情報に誤差が含まれるという問題は、第１の実施形態で説明したように、位置情報の信頼度を基準にして画像合成方式を選択することによって、ある程度回避することができる。ところが、障害物上の対応点自体が算出されなかった場合、このことは信頼度を用いた評価からは認識できない。
【００７６】
これに対して本実施形態では、合成方式選択部１７Ｂが、距離情報ＤＩとほぼ等しい距離ｄｉを有する対応点が存在するか否かを判断する。存在しないときは、位置情報が多くの誤差を含んでいるか、障害物上の対応点が算出されていない可能性が高い。言い換えると、障害物センサ３１によって検出された障害物ＯＢに関して、位置算出部１４によって得られた位置情報ＰＩの精度が十分でない、といえる。このとき、固定画像合成部１６で生成された第２の合成画像ＳＩ２を選択することによって、障害物の位置に誤りの無い合成画像を表示できる。
【００７７】
逆に、距離情報ＤＩとほぼ等しい距離を有する対応点が存在する場合は、第１の実施の形態と同様に、信頼度を用いて画像合成方式を選択する。これにより、障害物が誤った位置に表示されたり、表示されなかったりするということがなく、かつ状況に応じて歪みの少ない合成画像を表示することができる。したがって、運転者にとってより安全で、より自然な画像の提示が可能になる。
【００７８】
（第４の実施形態）
本発明の第４の実施形態に係るる車両周辺監視装置の構成は第３の実施形態とほぼ同様であり、図９に示すとおりである。第３の実施形態と異なるのは、合成方式選択部１７Ｃが、可変画像合成部１５および固定画像合成部１６によって生成された第１および第２の合成画像ＳＩ１，ＳＩ２のうちいずれか一方を選択する代わりに、第１および第２の合成画像ＳＩ１，ＳＩ２から、信頼度ＲＬに応じて、画像を部分的に選択し、提示対象の合成画像を生成する点である。
【００７９】
図１１は本実施形態におけるカメラの設置態様とカメラ画像の例である。ここでは、車両１の後方にある２個の障害物ＯＢａ，ＯＢｂがカメラに映されているものとする。
【００８０】
位置算出部１４は、図１１（ｂ）（ｃ）のカメラ画像から複数の点の位置情報とその信頼度を算出する。一般に、複数の物体がある場合、位置算出部１４によって算出される位置情報は、誤差を含む可能性や誤差の大きさが画像の領域によって異なっている。ここでは、障害物ＯＢａの領域に属する点の位置情報は誤差が小さく、障害物ＯＢｂの領域に属する点の位置情報は誤差が大きいと仮定する。このとき、障害物ＯＢａの領域に属する点の位置情報は信頼度が高くなり、障害物ＯＢｂの領域に属する点の位置情報は信頼度が低くなる。
【００８１】
図１２（ａ）は可変画像合成部１５によって生成された第１の合成画像ＳＩ１の例であり、図１２（ｂ）は固定画像合成部１６によって生成された第２の合成画像ＳＩ２の例である。ここで図１２（ａ）を見ると、障害物ＯＢａの領域の位置情報は信頼度が高いため、障害物ＯＢａは正しい位置に表示されているのに対して、障害物ＯＢｂの領域の位置情報は信頼度が低く誤差を多く含むため、障害物ＯＢｂは誤った位置に合成されている。
【００８２】
合成方式選択部１７Ｃは、図１３に示すように分割された複数の領域（図１３では１２個の領域）において、それぞれ、合成画像の部分的な選択を行う。すなわち、位置算出部１４によって算出された各点の位置情報と、可変画像合成部１５および固定画像合成部１６の合成パラメタとを用いて、各点が対応する合成画像上での位置を算出する。そして図１３に示す部分領域毎に、信頼度が所定の閾値以上のときは可変画像合成部１５が生成した第１の合成画像ＳＩ１を選択し、信頼度が閾値以下のときは固定画像合成部１６が生成した第２の合成画像ＳＩ２を選択する。
【００８３】
図１４は合成方式選択部１７Ｃの出力画像の例である。図１４において、障害物ＯＢａの領域（図１３における第９の部分領域）については、位置情報の信頼度が高いので、第１の合成画像ＳＩ１が選択されている。それ以外の領域では、第２の合成画像ＳＩ２が選択されている。ここで図１４と図１２（ａ）（ｂ）とを比較すると、図１２（ａ）では、障害物ＯＢａは正しい位置に歪みが少なく表示されているものの、障害物ＯＢｂの位置がずれており、また図１２（ｂ）では、障害物ＯＢａ，ＯＢｂはともに、路面と接する領域の位置は正しく合成されているものの、路面平面に対して高さのある領域は歪んでいる。これに対して図１４では、障害物ＯＢａ，ＯＢｂともに位置は正しく、障害物ＯＢａは歪みも少ない画像として、良好に表示されており、したがってより適切な合成画像が表示されているといえる。
【００８４】
以上のように本実施形態によると、第１および第２の合成画像ＳＩ１，ＳＩ２を、位置情報の信頼度に応じて部分的に選択して、合成画像を生成しているので、障害物の位置がより正しく、歪みがより少ない合成画像を表示することができる。
【００８５】
なお、本実施形態では、図１３に示すような所定の部分領域毎に、合成画像の選択を行うものとしたが、部分領域の個数や分割位置、大きさなどはこれに限定されるものではない。
【００８６】
また、合成画像を選択する部分領域は、予め定めておく代わりに、動的に定めてもかまわない。例えば、各点の位置情報と画像合成のパラメタとから、各点が対応する合成画像上での位置を算出した後、所定サイズの単位矩形領域においてそれぞれ、点の密度と、信頼度の平均値とを算出する。そして、これらの値が所定のしきい値よりも大きいときは第１の合成画像ＳＩ１を選択し、しきい値以下のときは第２の合成画像ＳＩ２を選択するようにしてもよい。点の密度と信頼度が高い領域は、障害物の位置が正しく計測された領域なので、このような部分領域を動的に選択することによって、歪みがより少ない合成画像を生成することができる。
【００８７】
なおもちろん、画像の選択に際し、上述の実施形態で示したように、位置情報ＰＩや距離情報ＤＩを加味してもかまわない。
【００８８】
（第５の実施形態）
図１５は本発明の第５の実施形態に係る車両周辺監視装置の構成を示すブロック図である。図９の構成と異なるのは、車両が、当該車両の移動方向を検出する移動方向検出手段３２を有しており、合成方式選択部１７Ｄが、移動方向検出手段３２から得た移動方向を加味して、合成画像の選択を行う点である。移動方向検出手段３２は例えば、車速センサと、舵角センサと、車速センサおよび舵角センサの出力から車両の移動方向を算出する手段とによって構成されるが、このような構成はすでに広く知られており、ここでは詳細な説明は省略する。
【００８９】
いま図１１の状況において、位置算出部１４で算出された位置情報ＰＩの信頼度ＲＬが、障害物ＯＢａの領域では高く、障害物ＯＢｂの領域では障害物ＯＢａの領域に比べて低いものと仮定する。このとき、可変画像合成部１５によって、図１６のような第１の合成画像ＳＩ１が生成される。図１６では、障害物ＯＢａは、位置情報の誤差が小さいため正しい位置に表示されているのに対して、障害物ＯＢｂは、位置情報が誤差を多く含むため少しずれた位置に合成されている。
【００９０】
合成方式選択部１７Ｄは、移動方向検出手段３２によって検出された車両の移動方向を用いて、位置算出部１４によって特定された複数の点のうち、車両の移動方向の画像に対応する点を特定する。そして、特定した点に係る信頼度に応じて、合成画像の選択を行う。
【００９１】
例えば合成方式選択部１７Ｄは、車両が左後方へ移動した場合、すなわち障害物ＯＢａの方向に進んだ場合は、障害物ＯＢａの領域に属する点の信頼度を基準にして合成画像の選択を行い、一方、車両が右後方へ移動した場合、すなわち障害物ＯＢｂの方向に進んだ場合は、障害物ＯＢｂの領域に属する点の信頼度を基準にして合成画像の選択を行う。
【００９２】
図１７（ａ）は車両が左後方へ移動した場合に合成方式選択部１７Ｄが出力した合成画像の例である。この場合、移動方向にあたる障害物ＯＢａの領域の点の信頼度は高いため、可変画像合成部１５が生成した第１の合成画像ＳＩ１が選択されている。一方、図１７（ｂ）は車両が右後方へ移動した場合に合成方式選択部１７Ｄが出力した合成画像の例である。この場合、移動方向にあたる障害物ＯＢｂの領域の点の信頼度が低いため、固定画像合成部１６が生成した第２の合成画像ＳＩ２が選択されている。
【００９３】
ここで、車両を運転する場合、車両の移動方向の画像は非常に重要であり、移動方向にある障害物はより正しい位置に表示される方が望ましいのに対して、移動方向以外にある障害物は多少位置がずれて表示されてもさほど問題にならないと考えられる。また、わずかな位置ずれよりも歪みによる違和感の方が、影響が大きい場合があり得るので、このような場合には、本実施形態によってより適切な合成画像が出力される。
【００９４】
以上のように本実施形態によると、合成画像の選択に、車両の移動方向が加味され、移動方向の画像に対応する点の信頼度が用いられるので、車両の移動方向において障害物が誤った位置に表示されず、また、移動方向以外については歪みの少ない合成画像を表示することができる。
【００９５】
なお、本実施形態において、合成画像の選択基準のしきい値にヒステリシスを持たせて、さらに、そのヒステリシスの幅を車両の移動速度に応じて変化させるようにしてもよい。具体的には、車両の移動速度が大きいほど、しきい値のヒステリシスの幅を大きくする。一般に、車両の移動速度が大きいほど画像の時間的な変動が大きくなり、これに伴って信頼度の時間的な変動も大きくなる。このため、移動速度に応じてヒステリシスの幅を大きくすることによって、合成画像の頻繁な切り替わりが抑制され、したがって、運転者が車両周辺の状況を把握しづらくなるという問題を回避することができる。
【００９６】
なお、本発明に係る車両周辺監視装置としては、カメラやディスプレイ、Ａ／Ｄ変換器、Ｄ／Ａ変換器、フレームメモリ、障害物センサおよび移動方向検出手段は、必須の構成要素では必ずしもない。
【００９７】
また、上述の各実施形態では、本発明に係る個別の処理手段、すなわち位置算出部、可変画像合成部、固定画像合成部および合成方式選択部は、個別のハードウェアによって実現されるものとしたが、ソフトウェアによって実現することも可能である。この場合、本発明に係る車両周辺監視装置は、例えば図１８のようなＣＰＵ４１、ＲＯＭ４２、ＲＡＭ４３を備えたコンピュータによって構成され、位置算出部、可変画像合成部、固定画像合成部、合成方法選択部などの処理手段は、ＲＯＭ４２またはＲＡＭ４３に格納されたプログラムをＣＰＵ４１によって実行することによって、実現される。
【００９８】
図１９は図１８の構成における処理の流れの一例を示すフローチャートであり、本発明に係る画像生成方法、および画像生成プログラムに対応している。まず、フレームメモリ１３ａ，１３ｂからカメラ画像を入力し（Ｓ２１）、入力したカメラ画像から、上述の位置算出部１４の動作と同様に、複数の点について、位置情報と、その位置情報の信頼度とを求める（Ｓ２２）。そして、第１の実施形態で述べたように、信頼度と所定の基準とを比較して（Ｓ２３）、信頼度が所定の基準よりも高いときは（Ｓ２４でＹｅｓ）、可変画像合成部１５の動作と同様にして、位置情報を用いて、カメラ画像から合成画像を生成する（Ｓ２５）。一方、信頼度が所定の基準よりも低いときは（Ｓ２４でＮｏ）、固定画像合成部１６の動作と同様にして、位置情報を用いないで所定の方法によって、カメラ画像から合成画像を生成する（Ｓ２６）。そして、生成した合成画像を、フレームメモリ１８へ出力する（Ｓ２７）。
【００９９】
なお、図１９の方法では、信頼度に応じて、いずれかの合成画像を選択的に生成するものとしたが、もちろん上述した各実施形態と同様に、２種類の合成画像を生成してから、信頼度に応じて選択するようにしてもかまわない。また逆に、例えば図１のようなハード構成において、合成方式選択部１７が、信頼度に応じて、可変画像合成部１５および固定画像合成部１６のいずれか一方を、選択的に動作させるようにしてもかまわない。
【０１００】
なお、本発明の用途は、車両の周辺監視に限定されるものではなく、例えば、ロボットにも適用することも可能である。例えば、遠隔操作が可能な移動ロボットにカメラと本発明に係る監視装置とを設置し、ロボット周囲の状況を示す合成画像は、ロボットのユーザが監視できるように、遠隔地に置かれたディスプレイに表示させる。本発明の各実施形態と同様の効果を得ることができる。
【０１０１】
【発明の効果】
以上のように本発明によると、車両の周辺状況を示す画像として、位置情報が正しく算出された場合は歪みの少ない合成画像が表示され、位置情報が大きな誤差を含む場合には、障害物が誤った位置に表示されたり消えたりしない合成画像が表示される。したがって、より自然で、かつ、より安全な画像を提示することができる。
【図面の簡単な説明】
【図１】本発明の第１および第２の実施形態に係る車両周辺監視装置の構成を示すブロック図である。
【図２】（ａ）は第１の実施形態におけるカメラの設置態様の例、（ｂ）（ｃ）は（ａ）におけるカメラ画像の例である。
【図３】位置算出部の動作を示すフローチャートである。
【図４】カメラ座標系の関係を示す概念図である。
【図５】路面平面を基準にした画像合成を示す概念図である。
【図６】本発明の第１の実施形態において、可変画像合成部が生成した第１の合成画像の例である。
【図７】本発明の第１の実施の形態において、固定画像合成部が生成した第２の合成画像の例である。
【図８】本発明の第２の実施形態において、提示される合成画像の例である。
【図９】本発明の第３および第４の実施形態に係る車両周辺監視装置の構成を示すブロック図である。
【図１０】第３の実施形態におけるカメラおよび障害物センサの設置態様の例である。
【図１１】（ａ）は第４の実施形態におけるカメラの設置態様の例、（ｂ）（ｃ）は（ａ）におけるカメラ画像の例である。
【図１２】（ａ）は第４の実施形態において、可変画像合成部が生成した第１の合成画像の例、（ｂ）は固定画像合成部が生成した第２の合成画像の例である。
【図１３】合成画像の部分的な選択を行うための領域分割の例である。
【図１４】第４の実施形態において、合成方式選択部が生成した合成画像の例である。
【図１５】本発明の第５の実施形態に係る車両周辺監視装置の構成を示すブロック図である。
【図１６】第５の実施形態において、可変画像合成部が生成した第１の合成画像の例である。
【図１７】第５の実施形態において、合成方式選択部が出力した合成画像の例である。
【図１８】本発明に係る車両周辺監視装置をコンピュータを用いて構成した場合の構成を示すブロック図である。
【図１９】図１８の構成における処理の流れを示すフローチャートである。
【図２０】従来技術を示す図である。
【図２１】従来技術と、その問題点を示す図である。
【符号の説明】
１ 車両
１１ａ，１１ｂ カメラ
１４ 位置算出部
１５ 可変画像合成部
１６ 固定画像合成部
１７，１７Ａ，１７Ｂ，１７Ｃ，１７Ｄ 合成方式選択部
２０ ディスプレイ
３１ 障害物センサ
３２ 移動方向検出手段
ＰＩ 位置情報
ＲＬ 信頼度
ＳＩ１ 第１の合成画像
ＳＩ２ 第２の合成画像
ＤＩ 距離情報[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a technique related to a vehicle periphery monitoring apparatus that deforms and processes a camera image that reflects the periphery of a vehicle and presents a composite image that indicates the surrounding state of the vehicle.
[0002]
[Prior art]
In recent years, vehicle periphery monitoring devices that monitor the periphery of a vehicle with a camera have been put into practical use so that the driver can drive safely.
[0003]
As an example of a conventional vehicle periphery monitoring device, a camera is installed so as to reflect a driver's blind spot such as the rear of the vehicle, and an image photographed by the camera is deformed by a predetermined fixing method. Some are shown on the display. For example, in the technique disclosed in Patent Document 1, a plurality of cameras are installed around a vehicle as shown in FIG. 20A, and an image around the vehicle photographed by the camera is deformed, and FIG. An image viewed from above centered on the vehicle is generated and presented on a display device in the vehicle.
[0004]
As another example, the position of the obstacle in the image is calculated from the image taken by the camera installed in the vehicle, and the image is deformed according to the position of the obstacle and displayed on the display. is there. For example, in the technique disclosed in Patent Document 2, when a vehicle is moving, an obstacle position in the image is calculated using two images with slightly different shooting times, and the image is deformed using the position. Thus, the composite image from the side viewpoint as shown in FIG.
[0005]
Using such a device, you can check the blind spot that cannot be seen directly from the driver's seat on the display, or it will be easier to grasp the distance to the obstacle by the image seen from the top or side. , Safer driving becomes possible.
[0006]
[Patent Document 1]
Japanese Patent Laid-Open No. 03-99952
[Patent Document 2]
Japanese Patent Laid-Open No. 06-333200
[0007]
[Problems to be solved by the invention]
However, the above-described prior art has the following problems.
[0008]
First, when the camera image is deformed by a predetermined fixed method, the shape of the obstacle in the image is not taken into consideration, so the shape of the obstacle is distorted on the composite image, and as a result, the composite image becomes unnatural. There was a problem of becoming. For example, in Patent Document 1, since the deformation processing of the camera image is performed on the assumption that the object in the image exists on the same plane, for example, the road plane, the road plane is correctly displayed as shown in FIG. However, an object that is not on the road surface, such as an obstacle, becomes a distorted and unnatural image.
[0009]
In addition, when the image is deformed according to the position of the obstacle calculated from the image, when the position of the obstacle is correctly calculated, the image is not distorted and a natural composite image is generated, but the calculation is performed. If there are many errors in the position, the obstacle may be displayed at an incorrect position, or the obstacle may not be displayed in the composite image. For example, in Patent Document 2, the position of the obstacle is calculated by extracting the movement of the luminance edge from two images with slightly different shooting times. However, in the actual driving environment, there are many errors in the calculated position. In many cases, the position of an obstacle cannot be detected at all. When the position includes a lot of errors, as shown in FIG. 21B, the obstacle is displayed farther than it actually is or is not displayed on the composite image. This could interfere with safe driving.
[0010]
In view of the above problems, the present invention provides a vehicle periphery monitoring device capable of presenting an image showing a situation around the vehicle more naturally and without causing a driver's misunderstanding. Let it be an issue.
[0011]
[Means for Solving the Problems]
In order to solve the above-described problem, the present invention calculates position information and reliability indicating how accurately the position information is calculated for a plurality of points in a camera image, and this reliability is calculated. Accordingly, an image composition method for deforming a camera image using position information, and an image composition method for deforming a camera image by a predetermined method without using position information, in whole or in part, Select as appropriate. Thus, for example, when the reliability is high and the position information is calculated correctly, a composite image with less distortion is displayed using the position information, while the reliability is low and the position information includes a large error. If there is, the position information is not used, but a composite image is displayed in which the obstacle is not displayed at the wrong position or disappears. Therefore, it is possible to present an image that shows a situation around the vehicle that is more natural and does not cause misunderstanding of the driver.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
According to the first aspect of the present invention, as a vehicle periphery monitoring device that generates and presents a composite image indicating a surrounding situation of the vehicle from an image of a camera that reflects the periphery of the vehicle, a plurality of points in the camera image are A position calculating unit that calculates position information and reliability of the position information; a variable image combining unit that generates a first composite image from the camera image using the position information; and the position information from the camera image. A fixed image composition unit that generates a second composite image by a predetermined method without using it, and selects one of the first and second composite images as a composite image to be presented according to the reliability And a synthesis method selection unit.
[0013]
According to a second aspect of the present invention, there is provided the vehicle periphery monitoring device according to the first aspect, wherein the composition method selection unit selects a composite image in consideration of the position information.
[0014]
According to the third aspect of the present invention, the composition method selection unit refers to the position information, extracts points belonging to the region used for image composition among the plurality of points, and relates to the extracted points. A vehicle periphery monitoring device according to a second aspect in which the degree is used for selecting a composite image is provided.
[0015]
According to the fourth aspect of the present invention, the variable image composition unit and the fixed image composition unit perform image composition so that the positions of the road surfaces coincide with each other in the first and second composite images. A vehicle periphery monitoring device is provided.
[0016]
According to the fifth aspect of the present invention, instead of selecting any one of the first and second synthesized images, the synthesis method selection unit selects the reliability from the first and second synthesized images. According to the present invention, there is provided a vehicle periphery monitoring apparatus according to a first aspect that partially selects an image and generates a composite image to be presented from the selected image.
[0017]
According to a sixth aspect of the present invention, there is provided the vehicle periphery monitoring device according to the first aspect, wherein the composition method selection unit superimposes a predetermined display corresponding to the characteristics of the composite image on the selected composite image.
[0018]
According to a seventh aspect of the present invention, the vehicle is provided with an obstacle sensor, and the synthesis method selection unit takes into account distance information indicating a distance value to the obstacle obtained from the obstacle sensor. A vehicle periphery monitoring apparatus according to a first aspect for selecting a composite image is provided.
[0019]
According to the eighth aspect of the present invention, when the combining method selection unit evaluates the accuracy of the position information related to the obstacle with reference to the distance information and the position information, and determines that the accuracy is not sufficient. There is provided a vehicle periphery monitoring apparatus according to a seventh aspect, wherein the second composite image is selected without selecting a composite image according to the reliability.
[0020]
According to a ninth aspect of the present invention, the vehicle has a movement direction detection unit that detects a movement direction of the vehicle, and the combining method selection unit takes into account the movement direction obtained from the movement direction detection unit. Thus, a vehicle periphery monitoring device according to a first aspect for selecting a composite image is provided.
[0021]
According to the tenth aspect of the present invention, the composition method selection unit identifies a point belonging to the image region in the moving direction among the plurality of points, and uses the reliability related to the identified point for selecting a composite image. A vehicle periphery monitoring device according to a ninth aspect to be used is provided.
[0022]
According to the eleventh aspect of the present invention, as an image generation method for generating a composite image indicating a surrounding situation of the vehicle from an image of a camera that reflects the periphery of the vehicle, position information and a plurality of points in the camera image are obtained. A first step for determining the reliability of the position information, a second step for comparing the reliability with a predetermined reference, and the reliability in the second step is higher than the predetermined reference. When the determination is made, using the position information obtained in the first step from the camera image, a third step of generating a composite image, and the reliability in the second step is based on the predetermined reference When it is determined that the camera image is lower than the camera image, the camera image is provided with a fourth step of generating a composite image by a predetermined method without using the position information.
[0023]
According to the twelfth aspect of the present invention, in the first step, the corresponding points are searched for the first and second camera images, and the identified corresponding points are the original in the first and second camera images. A three-dimensional position coordinate value is obtained from the image coordinate value, the original image coordinate value and the three-dimensional position coordinate value are determined as position information of the corresponding point, and the first and second cameras are determined from the three-dimensional position coordinate value. An image generation method according to an eleventh aspect is provided, in which image coordinate values in an image are calculated backward, and the reliability of the position information is obtained using the calculated image coordinate values and the original image coordinate values.
[0024]
According to the thirteenth aspect of the present invention, as a program for causing a computer to execute a method of generating a composite image indicating a surrounding situation of the vehicle from an image of a camera that reflects the periphery of the vehicle, For a plurality of points, a first step for obtaining position information and reliability of the position information, a second step for comparing the reliability obtained in the first step with a predetermined reference, A third step of generating a composite image from the camera image using the position information obtained in the step 1 when the reliability is determined to be higher than the predetermined reference in the step 2; When it is determined in the second step that the reliability is lower than the predetermined reference, a synthesized image is obtained from the camera image by a predetermined method without using the position information. It provides that to execute a fourth step of generating.
[0025]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0026]
(First embodiment)
FIG. 1 is a block diagram showing a configuration of a vehicle periphery monitoring device according to the first embodiment of the present invention. The vehicle periphery monitoring apparatus in FIG. 1 generates a composite image indicating the surrounding situation of the vehicle from an image of a camera that reflects the periphery of the vehicle, and presents this on a display.
[0027]
In FIG. 1, 11a and 11b are cameras that reflect the periphery of the vehicle, 12a and 12b are A / D converters that digitize analog video signals obtained by the cameras 11a and 11b, and 13a and 13b are digitized camera images. Is a frame memory that temporarily holds. Reference numeral 14 denotes a camera image as input, a position calculation unit that calculates position information PI and its reliability RL for a plurality of points in the camera image, and 15 uses the position information PI to transform the camera image, The variable image composition unit 16 that generates the first composite image SI1 from another viewpoint, 16 does not use the position information PI, and deforms the camera image by a predetermined method to change the second composite image SI2 from another viewpoint. A fixed image composition unit 17 to be generated is a composition method selection unit that selects one of the first and second composite images SI1 and SI2 as a composite image to be presented in accordance with the reliability RL. Reference numeral 18 denotes a frame memory that temporarily stores a composite image to be presented, 19 a D / A converter that converts digital image data stored in the frame memory 18 into an analog video signal, and 20 an analog video signal. Display.
[0028]
FIG. 2 shows an example of camera installation and camera images in this embodiment. As shown in FIG. 2 (a), in this embodiment, two cameras (camera 1, camera 2: corresponding to the cameras 11a and 11b in FIG. 1) are photographed at the rear of the vehicle 1 in the rear of the vehicle. is set up. OB is an obstacle and WL is a white line. FIGS. 2B and 2C are examples of camera images taken by the camera 1 and the camera 2, respectively. As can be seen from FIGS. 2B and 2C, the shooting ranges of the cameras 1 and 2 overlap each other. However, in the present invention, the camera may be installed in a place other than the vehicle, such as a parking lot or a facility around the road. Moreover, the display 20 shall be installed in the vehicle 1 in the position where it is easy to see from the driver. Of course, you may install in places other than a vehicle.
[0029]
The position calculation unit 14 calculates position information and reliability of a plurality of points in the camera image using so-called binocular stereo vision three-dimensional position detection. FIG. 3 is a flowchart showing the operation of the position calculation unit 14.
[0030]
First, the position calculation unit 14 reads camera image data from the frame memories 13a and 13b, specifies corresponding points from the two camera images (first and second camera images), and calculates their coordinates (S11). .
[0031]
Specifically, this is performed as follows. First, the image data of the first camera image is sequentially read from the frame memory 13a, and the luminance change amount in a small area centered on each pixel is calculated.
[0032]
Next, a predetermined number of pixels (u1i, v1i) (i = 1... N) are selected from all the pixels in descending order of the calculated luminance change amount. By selecting the luminance change amount in descending order, the pixels on the boundary portion of the object are easily selected. For example, it is assumed that the pixel (u1, v1) corresponding to the edge of the obstacle OB shown in FIG.
[0033]
Next, using the small region centered on each selected pixel (u1i, v1i) (i = 1... N) as a comparison reference, the small region having the smallest luminance difference is searched for in the second camera image. Then, the image coordinate value (u2i, v2i) (i = 1... N) of the center of the searched small area is calculated. The point (u2, v2) shown in FIG. 2C is an example of the selected point, and corresponds to the edge of the obstacle OB like the point (u1, v1) in FIG.
[0034]
For such a method for selecting corresponding points from an image and a corresponding point search method for detecting corresponding points between two images, for example, “Good Features to Track”, Jianbo Shi, Carlo Tomasi, IEEE Conference on Computer Vision and Pattern Recognition, 1994, "An Iterative Image Registration Technique with an application to Stereo Vision", Bruce D. Lucas, Takeo Kanade, Proceedings of Imaging Understandig Workshop, pp121-130, 1981 Therefore, detailed description is omitted here.
[0035]
Next, the position calculation unit 14 calculates three-dimensional position coordinates from the image coordinates (u1i, v1i) (u2i, v2i) (i = 1... N) for each corresponding point specified in step S11 (S12). Here, the positional relationship (coordinate transformation matrix) M12 of the cameras 1 and 2 and the camera parameters f1 and f2 measured in advance are used.
[0036]
Assume that the coordinate systems of cameras 1 and 2 are arranged as shown in FIG. The positional relationship M12 between cameras between a coordinate value (x1, y1, z1) of the viewpoint coordinate system of the camera 1 and a coordinate value (x2, y2, z2) of the viewpoint coordinates of the camera 2 for a certain point. The following relational expression is established using
[Expression 1]

[0037]
Also, the image coordinate values (u1, v1), (u2, v2) of the corresponding points, the coordinate values (x1, y1, z1) of the viewpoint coordinate system of the camera 1, the positional relationship M12 between the cameras, and the camera parameters f1, The following relational expression holds between f2. Camera parameters f1 and f2 are obtained by multiplying the focal length of the camera by the pixel size and the conversion coefficient of the image coordinate value.
[Expression 2]

[0038]
By substituting the previously measured parameters M12, f1, f2 and the calculated image coordinate values (u1, v1), (u2, v2) of the corresponding points into (Equation 2), the three-dimensional coordinates (x1) of the corresponding points , Y1, z1) can be calculated. Here, (Equation 2) has four linear expressions independent of the unknown 3 and the image coordinate values (u1, v1) and (u2, v2) of corresponding points calculated from the image include errors. Therefore, the three-dimensional coordinate values (x1, y1, z1) are calculated using the least square method. Thereby, the influence of an error can be reduced.
[0039]
The image coordinate value (u1, v1) in the first camera image, the image coordinate value (u2, v2) in the second camera image, and the three-dimensional position coordinate value ( (u1, v1, u2, v2, x1, y1, z1) that summarizes (x1, y1, z1) is called position information of the point.
[0040]
Further, the position calculation unit 14 calculates the image coordinate values (u1 ′, v1 ′), (u2 ′) in the first and second camera images from the three-dimensional coordinate values (x1, y1, z1) obtained by the least square method. , V2 ′) is calculated backward (S13). That is, by substituting the three-dimensional coordinate values (x1, y1, z1) and the parameters M12, f1, f2 into (Equation 2), new image coordinate values in the first and second camera images are obtained.
[0041]
Here, the image coordinate values (u1 ′, v1 ′) and (u2 ′, v2 ′) obtained by back calculation are the original original image coordinate values (u1, v1), (u2, v2) when there is no measurement error. ) Should match. However, in general, there is a measurement error and the two do not match. Therefore, using the distance error average e between the original image coordinate values (u1, v1), (u2, v2) and the image coordinate values (u1 ′, v1 ′), (u2 ′, v2 ′) obtained by back calculation, The reliability r is obtained by the following equation (S14).
[0042]
[Equation 3]

[0043]
The reliability r is 1 when the coordinate value of the corresponding point does not include a measurement error, and approaches 0 as the measurement error increases. That is, the calculated position information (u1, v1, u2, v2, x1, y1, z1) of the corresponding points represents the degree of error, and serves as an index of whether the position information is reliable.
[0044]
By such a procedure, the position calculation unit 14 obtains the position information PI ((u1i, v1i, u2i, v2i, x1i, y1i, z1i): i = 1 to n) and its reliability RL (ri: i = 1 to n) are calculated and output.
[0045]
The variable image composition unit 15 reads the data of the first and second camera images from the frame memories 13a and 13b, and uses the position information PI of the corresponding points calculated by the position calculation unit 14 to generate the first composite image SI1. Generate. Here, it is assumed that a composite camera arranged as shown in FIG. 4 is used, and image synthesis is performed by a texture mapping method using a pre-given position M10 of the composite camera and a parameter f0. So-called texture mapping, which synthesizes an image viewed from an arbitrary viewpoint using image data and a three-dimensional coordinate value of a point in the image, is a well-known technique in the field of computer graphics. Description is omitted. Note that image composition in the variable image composition unit 17 is not limited to texture mapping. For example, a method of superimposing a predetermined color or figure representing an obstacle on a position indicated by position information in a composite image, or the like. Any image composition reflecting position information may be used.
[0046]
On the other hand, the fixed image synthesizing unit 16 reads the data of the first and second camera images from the frame memories 13a and 13b, and does not use the position information PI of the corresponding points calculated by the position calculating unit 14, and is determined in advance. A second composite image SI2 is generated by the method. Here, as shown in FIG. 5, it is assumed that image composition is performed with reference to a road surface plane. That is, assuming that the camera 1 and the camera 2 are all shooting a road plane, the camera 1 and the camera 2 correspond to the image coordinates of the composite camera by giving the position and camera parameters with respect to the road surface, respectively. The image coordinates of camera 1 or camera 2 can be calculated. The fixed image composition unit 16 pre-calculates and records a value ci indicating the corresponding camera and the image coordinates (ui, vi) as a reference table for all image coordinates (uoi, voi) of the synthesized camera. Then, according to the reference table, pixel data (ci, ui, vi) corresponding to each coordinate (ui, voi) of the composite image is sequentially read out from the frame memories 13a, 13b to generate a composite image. Such image composition based on the road surface plane is disclosed in, for example, the pamphlet of International Publication No. 00/07373, and detailed description thereof is omitted here.
[0047]
FIG. 6 is an example of the first composite image SI1 generated by the variable image composition unit 15 in the situation of FIG. 2, and (a) is a case where the position information PI of the corresponding point is correctly calculated by the position calculation unit 14. (B) is a case where the position information PI contains many errors. 1A is an image representing the vehicle 1. As shown in FIG. 6 (a), the first synthesized image SI1 is converted in viewpoint while the three-dimensional position of the object in the original camera image is maintained. Compared with such a camera image, the distance to the obstacle OB is easy to understand. On the other hand, as shown in FIG. 6B, when the position information PI includes many errors, there is a feature that the position and shape of the obstacle OB are not accurate.
[0048]
On the other hand, FIG. 7 shows an example of the second composite image SI2 generated by the fixed image composition unit 16 in the situation of FIG. As shown in FIG. 7, the second composite image SI2 shows the positional relationship between the vehicle and the region where the obstacle OB is in contact with the road surface accurately, but the region having a height relative to the road surface is displayed distorted. There is a feature of becoming unnatural.
[0049]
Therefore, the composition method selection unit 17 selects one of the first and second composite images SI1 and SI2 as a composite image to be presented according to the reliability RL calculated by the position calculation unit 14, Output. That is, when it is determined that the reliability RL is higher than the predetermined reference, in other words, when the position information PI does not include much error, the first synthesized image SI1 generated by the variable image synthesizer 15 is selected, On the other hand, when it is determined that the reliability RL is lower than the predetermined reference, in other words, when the position information PI includes many errors, the second composite image SI2 generated by the fixed image composition unit 16 is selected.
[0050]
Specifically, the comparison between the reliability RL and a predetermined standard is performed as follows, for example. First, the reliability ri of the n corresponding points calculated by the position calculation unit 14 is compared with a predetermined threshold value rth, and the number m of corresponding points equal to or greater than the threshold value rth is calculated. Then, the ratio m / n of the corresponding points equal to or higher than the threshold value rth is compared with a predetermined threshold value mth, and when the ratio m / n is larger than the threshold value mth, it is determined that the reliability RL is higher than the predetermined reference. The first composite image SI1 generated by the variable image composition unit 15 is selected. On the other hand, when the ratio m / n is smaller than the threshold value mth, it is determined that the reliability RL is lower than a predetermined reference, and the second combined image SI2 generated by the fixed image combining unit 15 is selected. Of course, the comparison between the reliability and the predetermined standard is not limited to such a method, and for example, the number of corresponding points whose reliability ri is equal to or greater than the threshold rth may be compared with the threshold. .
[0051]
Each of the cameras 11a and 11b captures an image at a constant period. Then, the position calculation unit 14, the variable image synthesis unit 15, the fixed image synthesis unit 16, and the synthesis method selection unit 17 repeatedly perform the above-described operation every time a camera image is obtained. As a result, a composite image corresponding to the surrounding situation of the vehicle that changes with time is displayed on the display 20.
[0052]
The driver looks at the composite image displayed on the display 20 and performs the driving with reference to this. For example, in the situation of FIG. 2A, when the position information PI is correctly calculated by the position calculation unit 14, the variable image composition unit 15 cannot correctly calculate the position information PI of the composite image of FIG. When this occurs, the composite image of FIG. 6B is generated, and the fixed image composition unit 16 generates the composite image of FIG. Here, the image of FIG. 6A is natural with no distortion of the obstacle OB compared to FIG. However, since the image of FIG. 6B is likely to mislead the driver about the position of the obstacle OB, it can be said that the image of FIG. 7 is preferable.
[0053]
Therefore, in this embodiment, the composition method selection unit 17 determines whether or not the position information PI is correctly calculated using the reliability RL, and the variable image composition unit 15 generates the position information PI when the position information PI is correctly calculated. 6A is selected, and when the position information PI includes many errors, the composite image of FIG. 7 generated by the fixed image combining unit 16 is selected and output. As a result, a safer and more natural image presentation for the driver is realized.
[0054]
As described above, according to the present embodiment, the position calculation unit that calculates the position information of the object in the image and its reliability, and the variable that deforms the camera image using the position information to generate the first composite image. An image synthesizing unit and a fixed image synthesizing unit that generates a second synthesized image by deforming the camera image in a predetermined procedure without using position information, and referencing the reliability of the position information. When it is determined that the position information is correctly calculated, the first composite image is selected, and when it is determined that the position information may contain many errors, the second composite image is selected. As a result, in the composite image, an obstacle is not displayed at an incorrect position or an obstacle is not displayed, and a composite image with less distortion can be displayed depending on the situation, so it is more for the driver. It is possible to present safe and more natural images.
[0055]
In the present embodiment, the composition of the first and second composite images generated by the variable image composition unit 15 and the fixed image composition unit 16 is not particularly limited, but the variable image composition unit 15 and the fixed image composition unit 15 are fixed. The image composition unit 16 preferably performs image composition so that the positions of the road surfaces coincide with each other in the first and second composite images. Thereby, even when the image composition method is frequently switched, it is not difficult to see the composite image, and the driver can easily grasp the situation around the vehicle.
[0056]
For example, depending on the shooting situation, the reliability RL of the position information varies greatly every time the process is repeated, and the first composite image and the second composite image may be frequently switched. In such a case, if the composition of the first and second composite images is greatly different, it becomes difficult for the driver to grasp the situation around the vehicle. Therefore, the variable image composition unit 15 and the fixed image composition unit 16 make the position of the composition camera and the internal parameters in the image composition coincide with each other in advance, so that the road surface is almost at the same position as shown in FIGS. The indicated composite image can be generated. As a result, even if the composite image is frequently switched, it is not difficult to see.
[0057]
In this embodiment, the image composition method is selected according to the reliability of the position information. However, when the threshold for selection has a hysteresis, the reliability is slightly changed in a fast cycle. For example, frequent switching of the image composition method can be suppressed. Similarly, frequent switching of the image composition method can also be suppressed by adding a restriction so that the image composition method is not switched until a predetermined time elapses. This makes it easy to see the composite image and makes it easier for the driver to grasp the situation around the vehicle.
[0058]
In this embodiment, the image composition method is selected according to the reliability of the position information. However, the image composition method may be selected with reference to the position information itself. For example, the composition method selection unit 17 refers to the three-dimensional position and the image coordinate value of each corresponding point included in the position information, and belongs to the region used for image composition in the variable image composition unit 15 and the fixed image composition unit 16. Extract points. Then, the reliability related to the extracted point is used for selecting a composite image. Specifically, for example, when the number of extracted points is equal to or greater than a predetermined value, an image composition method is selected by the above-described procedure using the reliability of the extracted points. On the other hand, when the number of extracted points is less than the predetermined value, the second composite image SI2 generated by the fixed image composition unit 16 is selected. As a result, the reliability of corresponding points belonging to a region that is not used for image synthesis is not considered in the selection of the image synthesis method, so that a more appropriate synthesized image can be presented.
[0059]
Further, in this embodiment, the camera is installed so as to photograph the rear of the vehicle 1 as shown in FIG. 2 and at a different horizontal position, but the photographing range and the camera installation position are limited. Instead, if the shooting ranges of the cameras are arranged so as to overlap, the same processing as in this embodiment can be performed on the overlapping area.
[0060]
In the present embodiment, the reliability of the position information of the corresponding point is obtained by (Equation 3) based on the image coordinates, but the calculation of the reliability is not limited to this, and the reliability of the corresponding point is not limited to this. Any reference value may be used as long as it is a scale indicating the magnitude of error contained in the position information and the possibility of error.
[0061]
In the present embodiment, the variable image composition unit 15 performs image composition using the position information of the n corresponding points calculated by the position calculation unit 14, but the reliability of the n corresponding points is determined. Only the position information of a point where is equal to or greater than a predetermined threshold value may be selected and used. As a result, position information with low reliability is eliminated, so that synthesis errors in the synthesized image can be reduced.
[0062]
In the present embodiment, one variable image composition unit 15 and one fixed image composition unit 16 are provided, but the number of image composition units is not limited to this. For example, at least one of a variable image composition unit and a fixed image composition unit is provided, and one of the composite images generated by each image composition unit according to the reliability of the position information by the composition method selection unit 17 is generated. May be selected.
[0063]
In the present embodiment, the position calculation unit 14 uses a so-called binocular stereo method in which the position is calculated using images simultaneously captured by two cameras. A so-called motion stereo system that calculates a position using two continuous camera images may be used, and the same effect as in the present embodiment can be obtained.
[0064]
(Second Embodiment)
The configuration of the vehicle periphery monitoring apparatus according to the second embodiment of the present invention is substantially the same as that of the first embodiment, as shown in FIG. The difference from the first embodiment is that the composition method selection unit 17A superimposes a predetermined display corresponding to the characteristics of the composite image on the selected composite image. Further, when selecting a composite image, the position information PI itself is used in combination with the reliability RL of the position information.
[0065]
Specifically, it operates as follows. The composition method selection unit 17A extracts points belonging to obstacles that may appear in the composite image from the n corresponding points specified by the position calculation unit 14. That is, with reference to the three-dimensional position coordinate value and the image coordinate value included in the position information PI, a point that is separated from the road surface by a predetermined distance and belongs to an area used for image composition is extracted.
[0066]
If the number of extracted points is equal to or greater than a predetermined value, the process described in the first embodiment is performed using the reliability of the extracted points. If the reliability is higher than a predetermined reference, the variable image The first combined image SI1 generated by the combining unit 15 is selected. At this time, since it is determined that the position of the obstacle OB has been accurately detected, for example, a message “Detecting obstacle position” is superimposed as shown in FIG. On the other hand, when the reliability is lower than the predetermined reference, the second composite image SI2 generated by the fixed image composition unit 16 is selected. At this time, it is determined that the position of the obstacle OB has not been accurately detected. Therefore, as shown in FIG. 8B, for example, a message “Position calculation impossible: image is distorted!” Is superimposed.
[0067]
On the other hand, when the number of extracted points is less than the predetermined value, the second composite image SI2 generated by the fixed image composition unit 16 is selected, and at this time, it is determined that no obstacle is detected in the vicinity of the vehicle. Therefore, as shown in FIG. 8C, for example, a message “No obstacle detected” is superimposed.
[0068]
Here, in the first composite image SI1 generated by the variable image combining unit 15, the shape of the obstacle is correctly displayed as shown in FIG. 6A, whereas the first image generated by the fixed image combining unit 16 is displayed. In the second composite image SI2, the shape of the obstacle may be distorted as shown in FIG. For this reason, when the user looks at the composite image as shown in FIG. 7, whether the obstacle is a rectangular parallelepiped and the composite image is distorted or whether the obstacle is a crushed rectangular parallelepiped and the composite image is not distorted. Because the user cannot judge, there is a possibility of misunderstanding the shape of the obstacle. For example, if the driver misunderstands that the obstacle shown in FIG. 7 is low and can be overcome by car, there is a problem in terms of safety.
[0069]
On the other hand, in the present embodiment, as shown in FIG. 8, since the display corresponding to the characteristics of the composite image is superimposed on the selected composite image, the user can easily include, for example, the composite image including distortion. Therefore, it is possible to avoid misunderstanding of the situation around the vehicle due to the properties of the composite image. As a result, safer driving is realized.
[0070]
(Third embodiment)
FIG. 9 is a block diagram showing a configuration of a vehicle periphery monitoring apparatus according to the third embodiment of the present invention. Constituent elements common to those in FIG. 1 are denoted by the same reference numerals as those in FIG. 1, and detailed description thereof is omitted here. The difference from FIG. 1 is that an obstacle sensor 31 that detects an obstacle around the vehicle and outputs distance information DI thereof is provided in the vehicle, and the synthesis method selection unit 17B is obtained from the obstacle sensor 31. The point is that the composite image is selected in consideration of the distance information DI. The obstacle sensor 31 measures the distance to nearby obstacles at a constant period, and outputs distance information DI when there is an obstacle within a predetermined distance. Here, it is assumed that the obstacle sensor 31 is a so-called ultrasonic sensor that measures the distance from the time until the ultrasonic wave is transmitted and reflected and returned to the object.
[0071]
FIG. 10 shows an installation mode of the camera and the obstacle sensor in the present embodiment. As shown in FIG. 10, the obstacle sensor 31 is installed between the camera 1 and the camera 2. It is assumed that the positional relationship between the obstacle sensor 31 and the cameras 1 and 2 is measured in advance.
[0072]
The operation of the composition method selection unit 17B in this embodiment will be described. When no obstacle is detected by the obstacle sensor 31, the synthesis method selection unit 17B compares the reliability RL of the position information with a predetermined reference, according to the comparison result, as in the first embodiment. Select a composite image. On the other hand, when an obstacle is detected by the obstacle sensor 31, first, the distance di (i = 1,..., N) from the obstacle sensor 31 is determined for the n corresponding points specified by the position calculation unit 14. Calculate each. The calculation here can be performed using the positional information of each corresponding point and the positional relationship between the obstacle sensor 31 and the cameras 1 and 2. Next, the composition method selection unit 17B compares the calculated distance di (i = 1,..., N) with the distance information DI detected by the obstacle sensor 31. Then, it is determined whether or not there is a corresponding point whose difference is equal to or smaller than a predetermined error threshold dth, that is, whether or not there is a corresponding point having a distance di that is substantially equal to the distance information DI. It can be said that this process evaluates the accuracy of the position information PI related to the obstacle OB with reference to the distance information DI and the position information PI.
[0073]
Then, when there is a corresponding point having a distance di that is substantially equal to the distance information DI, the composition method selection unit 17B compares the reliability RL of the position information with a predetermined reference, as in the first embodiment, A composite image is selected according to the comparison result. On the other hand, when there is no corresponding point having a distance di substantially equal to the distance information DI, in other words, when it is determined that the accuracy of the position information PI regarding the obstacle OB is not sufficient, the composite image according to the reliability RL. Is selected, the second composite image SI2 generated by the fixed image composition unit 16 is selected.
[0074]
Here, the obstacle sensor 31 can detect the distance of an object at a short distance relatively reliably. In other words, when the obstacle sensor 31 outputs the distance information DI, there is a high possibility that an obstacle actually exists at the position of the distance indicated by the distance information DI. On the other hand, in the corresponding point search and position detection on the image executed by the position calculation unit 14, particularly when the brightness and color of the obstacle are uniform or when the obstacle is similar to the background. There is a possibility that the position information includes many errors or the corresponding points themselves on the obstacles are not calculated.
[0075]
As described in the first embodiment, the problem that the position information includes an error can be avoided to some extent by selecting an image composition method based on the reliability of the position information. However, if the corresponding point on the obstacle itself is not calculated, this cannot be recognized from the evaluation using the reliability.
[0076]
On the other hand, in this embodiment, the composition method selection unit 17B determines whether there is a corresponding point having a distance di that is substantially equal to the distance information DI. When it does not exist, there is a high possibility that the position information includes many errors or corresponding points on the obstacle are not calculated. In other words, it can be said that the position information PI obtained by the position calculator 14 is not sufficiently accurate with respect to the obstacle OB detected by the obstacle sensor 31. At this time, by selecting the second synthesized image SI2 generated by the fixed image synthesizing unit 16, a synthesized image having no error can be displayed at the position of the obstacle.
[0077]
On the other hand, if there is a corresponding point having a distance substantially equal to the distance information DI, the image composition method is selected using the reliability as in the first embodiment. Thereby, an obstacle is not displayed at an incorrect position or not displayed, and a composite image with little distortion can be displayed depending on the situation. Therefore, it is possible to present images that are safer and more natural for the driver.
[0078]
(Fourth embodiment)
The configuration of the vehicle periphery monitoring apparatus according to the fourth embodiment of the present invention is substantially the same as that of the third embodiment, as shown in FIG. The difference from the third embodiment is that the composition method selection unit 17C selects one of the first and second composite images SI1 and SI2 generated by the variable image composition unit 15 and the fixed image composition unit 16. Instead, the image is partially selected from the first and second composite images SI1 and SI2 according to the reliability RL, and a composite image to be presented is generated.
[0079]
FIG. 11 shows an example of camera installation and camera images in the present embodiment. Here, it is assumed that two obstacles OBa and OBb in the rear of the vehicle 1 are projected on the camera.
[0080]
The position calculation unit 14 calculates position information and reliability of a plurality of points from the camera images of FIGS. In general, when there are a plurality of objects, the position information calculated by the position calculation unit 14 may include an error and the size of the error differs depending on the area of the image. Here, it is assumed that the position information of the points belonging to the area of the obstacle OBa has a small error, and the position information of the points belonging to the area of the obstacle OBb has a large error. At this time, the position information of the points belonging to the area of the obstacle OBa has high reliability, and the position information of the points belonging to the area of the obstacle OBb has low reliability.
[0081]
FIG. 12A shows an example of the first composite image SI1 generated by the variable image composition unit 15, and FIG. 12B shows an example of the second composite image SI2 generated by the fixed image composition unit 16. is there. Referring to FIG. 12 (a), since the position information of the area of the obstacle OBa has high reliability, the position information of the area of the obstacle OBb is displayed while the obstacle OBa is displayed at the correct position. Since the reliability is low and includes many errors, the obstacle OBb is synthesized at an incorrect position.
[0082]
The composition method selection unit 17C performs partial selection of the composite image in each of a plurality of regions (12 regions in FIG. 13) divided as illustrated in FIG. That is, using the position information of each point calculated by the position calculation unit 14 and the synthesis parameters of the variable image synthesis unit 15 and the fixed image synthesis unit 16, the position on the synthesized image corresponding to each point is calculated. . For each partial area shown in FIG. 13, when the reliability is equal to or higher than a predetermined threshold, the first composite image SI1 generated by the variable image combining unit 15 is selected. When the reliability is equal to or lower than the threshold, the fixed image combining unit is selected. 16 selects the second composite image SI2 generated.
[0083]
FIG. 14 is an example of an output image of the composition method selection unit 17C. In FIG. 14, since the reliability of the position information is high for the area of the obstacle OBa (the ninth partial area in FIG. 13), the first composite image SI1 is selected. In the other area, the second composite image SI2 is selected. Here, comparing FIG. 14 with FIGS. 12A and 12B, in FIG. 12A, the obstacle OBa is displayed at the correct position with little distortion, but the position of the obstacle OBb is shifted. In FIG. 12B, both the obstacles OBa and OBb are correctly combined in the position of the area in contact with the road surface, but the area having a height with respect to the road surface plane is distorted. On the other hand, in FIG. 14, the positions of both the obstacles OBa and OBb are correct, and the obstacle OBa is displayed well as an image with little distortion, and therefore it can be said that a more appropriate composite image is displayed.
[0084]
As described above, according to the present embodiment, the first and second composite images SI1 and SI2 are partially selected according to the reliability of the position information to generate the composite image. It is possible to display a composite image with a more accurate position and less distortion.
[0085]
In the present embodiment, the composite image is selected for each predetermined partial area as shown in FIG. 13, but the number, division position, size, etc. of the partial areas are not limited to this. Absent.
[0086]
Further, the partial region for selecting the composite image may be dynamically determined instead of being determined in advance. For example, after calculating the position on the composite image corresponding to each point from the position information of each point and the image composition parameters, the density of the points and the average value of the reliability in the unit rectangular area of a predetermined size, respectively And calculate. The first composite image SI1 may be selected when these values are greater than a predetermined threshold value, and the second composite image SI2 may be selected when these values are less than or equal to the threshold value. An area with high dot density and high reliability is an area where the position of an obstacle is correctly measured. Therefore, by dynamically selecting such a partial area, a composite image with less distortion can be generated.
[0087]
Of course, when selecting an image, as shown in the above-described embodiment, the position information PI and the distance information DI may be taken into consideration.
[0088]
(Fifth embodiment)
FIG. 15 is a block diagram showing a configuration of a vehicle periphery monitoring apparatus according to the fifth embodiment of the present invention. 9 is different from the configuration of FIG. 9 in that the vehicle has a movement direction detection unit 32 that detects the movement direction of the vehicle, and the combining method selection unit 17D takes the movement direction obtained from the movement direction detection unit 32 into account. Thus, a composite image is selected. The movement direction detection means 32 is constituted by, for example, a vehicle speed sensor, a steering angle sensor, and a means for calculating the movement direction of the vehicle from the output of the vehicle speed sensor and the steering angle sensor. Such a configuration is already widely known. Detailed description is omitted here.
[0089]
In the situation of FIG. 11, it is assumed that the reliability RL of the position information PI calculated by the position calculation unit 14 is high in the area of the obstacle OBa and low in the area of the obstacle OBb compared to the area of the obstacle OBa. To do. At this time, the variable image composition unit 15 generates a first composite image SI1 as shown in FIG. In FIG. 16, the obstacle OBa is displayed at the correct position because the position information error is small, whereas the obstacle OBb is synthesized at a slightly shifted position because the position information contains a lot of errors. .
[0090]
The combining method selection unit 17D specifies a point corresponding to the image of the moving direction of the vehicle among the plurality of points specified by the position calculating unit 14 using the moving direction of the vehicle detected by the moving direction detection unit 32. To do. Then, a composite image is selected according to the reliability related to the identified point.
[0091]
For example, when the vehicle moves left rearward, that is, in the direction of the obstacle OBa, the composition method selection unit 17D selects a composite image based on the reliability of points belonging to the area of the obstacle OBa. On the other hand, when the vehicle moves to the right rear, that is, in the direction of the obstacle OBb, the composite image is selected based on the reliability of the points belonging to the area of the obstacle OBb.
[0092]
FIG. 17A is an example of a composite image output by the composite method selection unit 17D when the vehicle moves to the left rear. In this case, since the reliability of the point of the area of the obstacle OBa corresponding to the moving direction is high, the first composite image SI1 generated by the variable image composition unit 15 is selected. On the other hand, FIG. 17B is an example of a synthesized image output by the synthesis method selection unit 17D when the vehicle moves to the right rear. In this case, since the reliability of the point of the area of the obstacle OBb corresponding to the moving direction is low, the second composite image SI2 generated by the fixed image composition unit 16 is selected.
[0093]
Here, when driving a vehicle, the image of the moving direction of the vehicle is very important, and it is desirable that obstacles in the moving direction are displayed in the correct position, whereas obstacles other than the moving direction are displayed. Even if the object is displayed with a slight shift, it is not considered to be a problem. In addition, since a sense of discomfort due to distortion may have a greater influence than a slight misalignment, in this case, a more appropriate composite image is output according to the present embodiment.
[0094]
As described above, according to this embodiment, since the moving direction of the vehicle is added to the selection of the composite image and the reliability of the point corresponding to the image in the moving direction is used, the obstacle is erroneous in the moving direction of the vehicle. It is possible to display a composite image that is not displayed at a position and has little distortion in directions other than the moving direction.
[0095]
In the present embodiment, hysteresis may be given to the threshold value for selecting the composite image, and the width of the hysteresis may be changed in accordance with the moving speed of the vehicle. Specifically, the threshold hysteresis width is increased as the moving speed of the vehicle increases. Generally, as the moving speed of the vehicle increases, the temporal variation of the image increases, and accordingly, the temporal variation of the reliability also increases. For this reason, by increasing the width of the hysteresis in accordance with the moving speed, frequent switching of the composite image is suppressed, and thus it is possible to avoid the problem that it becomes difficult for the driver to grasp the situation around the vehicle.
[0096]
In addition, as a vehicle periphery monitoring apparatus according to the present invention, a camera, a display, an A / D converter, a D / A converter, a frame memory, an obstacle sensor, and a moving direction detection unit are not necessarily essential components.
[0097]
Further, in each of the above-described embodiments, the individual processing means according to the present invention, that is, the position calculation unit, the variable image synthesis unit, the fixed image synthesis unit, and the synthesis method selection unit are realized by individual hardware. However, it can also be realized by software. In this case, the vehicle periphery monitoring device according to the present invention is configured by a computer including a CPU 41, a ROM 42, and a RAM 43 as shown in FIG. 18, for example, and includes a position calculation unit, a variable image synthesis unit, a fixed image synthesis unit, and a synthesis method selection unit. The processing means such as are realized by the CPU 41 executing a program stored in the ROM 42 or the RAM 43.
[0098]
FIG. 19 is a flowchart showing an example of the processing flow in the configuration of FIG. 18, and corresponds to the image generation method and the image generation program according to the present invention. First, camera images are input from the frame memories 13a and 13b (S21), and the position information and reliability of the position information are obtained from the input camera images for a plurality of points in the same manner as the operation of the position calculation unit 14 described above. (S22). Then, as described in the first embodiment, the reliability is compared with a predetermined standard (S23), and when the reliability is higher than the predetermined standard (Yes in S24), the variable image combining unit 15 In the same manner as in the above operation, a composite image is generated from the camera image using the position information (S25). On the other hand, when the reliability is lower than the predetermined reference (No in S24), the composite image is generated from the camera image by the predetermined method without using the position information in the same manner as the operation of the fixed image combining unit 16. (S26). Then, the generated composite image is output to the frame memory 18 (S27).
[0099]
In the method of FIG. 19, one of the composite images is selectively generated according to the reliability. Of course, like the above-described embodiments, two types of composite images are generated. The selection may be made according to the reliability. On the other hand, in the hardware configuration as shown in FIG. 1, for example, the composition method selection unit 17 selectively operates either the variable image composition unit 15 or the fixed image composition unit 16 according to the reliability. It doesn't matter.
[0100]
The application of the present invention is not limited to vehicle periphery monitoring, and can be applied to, for example, a robot. For example, a camera and a monitoring device according to the present invention are installed in a mobile robot capable of remote operation, and a composite image showing the situation around the robot is displayed on a display placed in a remote place so that the robot user can monitor it. Display. The same effects as those of the embodiments of the present invention can be obtained.
[0101]
【The invention's effect】
As described above, according to the present invention, as the image indicating the surrounding situation of the vehicle, a composite image with less distortion is displayed when the position information is correctly calculated, and when the position information includes a large error, an obstacle is displayed. A composite image that does not appear or disappear in the wrong position is displayed. Therefore, a more natural and safer image can be presented.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a vehicle periphery monitoring device according to first and second embodiments of the present invention.
FIGS. 2A and 2B are examples of camera installation modes in the first embodiment, and FIGS. 2B and 2C are examples of camera images in FIG.
FIG. 3 is a flowchart showing an operation of a position calculation unit.
FIG. 4 is a conceptual diagram showing the relationship of camera coordinate systems.
FIG. 5 is a conceptual diagram showing image composition based on a road surface plane.
FIG. 6 is an example of a first composite image generated by a variable image composition unit in the first embodiment of the present invention.
FIG. 7 is an example of a second composite image generated by the fixed image composition unit in the first embodiment of the present invention.
FIG. 8 is an example of a composite image presented in the second embodiment of the present invention.
FIG. 9 is a block diagram showing a configuration of a vehicle periphery monitoring device according to third and fourth embodiments of the present invention.
FIG. 10 is an example of an installation mode of a camera and an obstacle sensor in the third embodiment.
FIGS. 11A and 11B are examples of camera installation modes in the fourth embodiment, and FIGS. 11B and 11C are examples of camera images in FIG.
12A is an example of a first composite image generated by a variable image composition unit in the fourth embodiment, and FIG. 12B is an example of a second composite image generated by a fixed image composition unit. .
FIG. 13 is an example of region division for performing partial selection of a composite image.
FIG. 14 is an example of a synthesized image generated by a synthesis method selection unit in the fourth embodiment.
FIG. 15 is a block diagram showing a configuration of a vehicle periphery monitoring device according to a fifth embodiment of the present invention.
FIG. 16 is an example of a first composite image generated by a variable image composition unit in the fifth embodiment.
FIG. 17 is an example of a synthesized image output by a synthesis method selection unit in the fifth embodiment.
FIG. 18 is a block diagram showing a configuration when a vehicle periphery monitoring apparatus according to the present invention is configured using a computer.
FIG. 19 is a flowchart showing a process flow in the configuration of FIG. 18;
FIG. 20 is a diagram showing a conventional technique.
FIG. 21 is a diagram showing a conventional technique and its problems.
[Explanation of symbols]
1 vehicle
11a, 11b camera
14 Position calculator
15 Variable image composition unit
16 Fixed image composition part
17, 17A, 17B, 17C, 17D Compositing method selection unit
20 display
31 Obstacle sensor
32 Moving direction detection means
PI location information
RL reliability
SI1 First composite image
SI2 Second composite image
DI distance information

Claims (13)

A vehicle periphery monitoring device that generates and presents a composite image indicating a surrounding situation of the vehicle from an image of a camera that reflects the periphery of the vehicle,
A position calculation unit that calculates position information and reliability of the position information for a plurality of points in the camera image;
A variable image composition unit that generates a first composite image from the camera image using the position information;
A fixed image composition unit that generates a second composite image by a predetermined method without using the position information from the camera image;
A vehicle periphery monitoring apparatus, comprising: a synthesis method selection unit that selects one of the first and second synthesized images as a synthesized image to be presented according to the reliability. In claim 1,
The synthesis method selection unit
A vehicle periphery monitoring device, wherein a composite image is selected in consideration of the position information. In claim 2,
The synthesis method selection unit
With reference to the position information, out of the plurality of points, a point belonging to a region used for image composition is extracted,
A vehicle periphery monitoring device, wherein reliability relating to an extracted point is used for selecting a composite image. In claim 1,
The variable image combining unit and the fixed image combining unit are
A vehicle periphery monitoring device that performs image composition so that road positions coincide with each other in the first and second composite images. In claim 1,
The synthesis method selection unit
Instead of selecting either one of the first and second composite images, an image is partially selected from the first and second composite images according to the reliability, and from the selected image A vehicle periphery monitoring device that generates a composite image to be presented. In claim 1,
The synthesis method selection unit
A vehicle periphery monitoring device, wherein a predetermined display corresponding to the characteristics of the composite image is superimposed on the selected composite image. In claim 1,
The vehicle is provided with an obstacle sensor,
The synthesis method selection unit
A vehicle periphery monitoring device, wherein a composite image is selected in consideration of distance information obtained from the obstacle sensor and indicating a distance value to the obstacle. In claim 7,
The synthesis method selection unit
With reference to the distance information and position information, the accuracy of the position information related to the obstacle is evaluated, and when it is determined that the accuracy is not sufficient, the composite image is not selected according to the reliability, and the first A vehicle periphery monitoring device, wherein two composite images are selected. In claim 1,
The vehicle has a moving direction detecting means for detecting a moving direction of the vehicle,
The synthesis method selection unit
A vehicle periphery monitoring apparatus, wherein a composite image is selected in consideration of a moving direction obtained from the moving direction detecting means. In claim 9,
The synthesis method selection unit
Among the plurality of points, specify a point belonging to the image area in the moving direction,
The vehicle periphery monitoring device, wherein the reliability related to the identified point is used for selecting a composite image. An image generation method for generating a composite image indicating a surrounding situation of the vehicle from an image of a camera that reflects the periphery of the vehicle,
A first step of obtaining position information and reliability of the position information for a plurality of points in the camera image;
A second step of comparing the reliability with a predetermined criterion;
A third step of generating a composite image from the camera image using the position information obtained in the first step when the reliability is determined to be higher than the predetermined reference in the second step; When,
A fourth step of generating a composite image by a predetermined method without using the position information from the camera image when the reliability is determined to be lower than the predetermined reference in the second step; An image generation method comprising: In claim 11,
In the first step,
Search for corresponding points for the first and second camera images;
For the identified corresponding point, a three-dimensional position coordinate value is obtained from the original image coordinate values in the first and second camera images, and the original image coordinate value and the three-dimensional position coordinate value are determined as position information of the corresponding point. ,
Back-calculating image coordinate values in the first and second camera images from the three-dimensional position coordinate values;
A method of generating an image, comprising: calculating a reliability of the position information by using a back-calculated image coordinate value and the original image coordinate value. A program for causing a computer to execute a method for generating a composite image indicating a surrounding situation of the vehicle from an image of a camera that reflects the periphery of the vehicle,
A first step of obtaining position information and reliability of the position information for a plurality of points in the camera image;
A second step of comparing the reliability obtained in the first step with a predetermined standard;
A third step of generating a composite image from the camera image using the position information obtained in the first step when the reliability is determined to be higher than the predetermined reference in the second step; ,
A fourth step of generating a composite image by a predetermined method without using the position information from the camera image when the reliability is determined to be lower than the predetermined reference in the second step; An image generation program that is executed.

Images