JP3968867B2 - Display processing device - Google Patents

Description

である。
【００７１】
横方向の拡大処理を終えると、次に、ステップＳ１６〜Ｓ１１０の縦方向の拡大処理に入る。ここでは、図６（ｂ）に示すように１〜Ｃのうちのｎ番目の画素列の中の高輝度領域の始端Ｑnstart・終端Ｑnendを調査し（ステップＳ１７，Ｓ１８）、同図（ｃ）に示すように画素列ｎの高輝度領域Ｑnstart〜Ｑnendを縦方向に拡大率Ｋだけ拡大して拡大領域Ｑ'nstart〜Ｑ'nendを得る（ステップＳ１９）。そしてこの処理を画素列１〜Ｃそれぞれに対して繰返す（ステップＳ１６〜Ｓ１１０）。
【００７２】
ここで、

である。
【００７３】
これによって、図２のフローチャートにおけるステップＳ５の拡大処理を行い、可視光による第１の風景画像における図４（ｂ）に示した高輝度領域２０を拡大して同図（ｃ）に示した重畳領域２１に決定することになる。
【００７４】
次に図４（ｄ）に示すように、この高輝度拡大領域に対応する領域を赤外線による第２の風景画像上から重畳領域２２として切出して、図４（ａ）に示した２値化前の可視光による第１の風景画像の同じ領域上に重畳合成し、重畳画像フレームメモリ６に保存し（ステップＳ６）、これを表示モニタ７に表示させる（ステップＳ７）。図５は表示モニタ７の表示された重畳合成画像を示している。なお、この重畳合成処理では、赤外線画像の切出した重畳領域は不透明にして、背面になる第１の風景画像が透けて見えないようにしている。
【００７５】
この実施の形態では、可視光カメラ１も赤外線カメラ３も動画撮影カメラであり、第１、第２の風景画像は動画像であるので、以上のステップＳ２〜Ｓ７の処理を画像フレームごとに繰返すことによって表示モニタ７に重畳合成画像を動画像として表示させる。
【００７６】
こうして、第１の実施の形態の表示処理装置では、所定の閾値Ｂによって２値化した可視光による第１の風景画像から対向車のヘッドライトのような高輝度の光源の見かけの大きさで決まる不快グレア領域を高輝度領域２０と決定し、さらに所定の拡大率Ｋを掛けることによってこの高輝度領域２０の周囲に位置して光源からの光がドライバの眼球内で散乱するために眩しさを感じる減能グレア領域まで重畳合成領域を拡大し、この重畳合成領域２１に対応する領域２２を第２の風景画像から切出して２値化前の可視光による第１の風景画像の該当する部位に合成することにより、対向車のヘッドライトのような高輝度の光源によって眩惑され、ドライバの肉眼では見にくいあるいは見えないような対象物を表示手段に表示されている合成画像上で容易に発見できるようになる。これによって、特に対向車のヘッドライトに入ってしまって歩行者がドライバの視野から消滅する現象が発生するような場合にもその歩行者の存在を表示モニタ７で一瞥して認識できるようになる。
【００７７】
また表示モニタ７の輝度設定に応じて所定の輝度閾値Ｂを可変設定するようにしたので、ドライバの眩しさを感じる個人差に合わせて重畳領域を調整することができ、ドライバが自ら感じる眩惑範囲と表示モニタ７上での表示範囲との相違による違和感を低減することができる。
【００７８】
なお、上記の実施の形態における高輝度領域の決定のための輝度閾値の設定は特に限定されるわけではなく、実験を繰返すことによってふさわしい値に設定することができる。また輝度閾値を可変とせず、特に本発明の表示処理装置の使用が必要となる状況、夜間のような低照度環境で使用するのにふさわしい値に固定することもできる。そしてその場合には図１において示したモニタ輝度設定スイッチ８は不要となり、構成が単純になる。また輝度閾値を固定とする場合には、その輝度閾値を上記の実施の形態の場合よりも若干低めに設定すれば、通常の不快グレア領域よりもやや広めの領域を高輝度領域２０として決定することができるようになり、拡大率を設定して高輝度領域を拡大処理しなくても十分実用に供することができ、構成がより単純化できる。
【００７９】
次に、本発明の第２の実施の形態の表示処理装置を図７及び図８に基づいて説明する。第２の実施の形態の表示処理装置の特徴は、ＣＰＵ５が可視光による第１の風景画像に対して高輝度領域を決定するための所定の閾値をヘッドライト操作スイッチ９によるヘッドライトの点／消灯操作状態に応じて可変設定するようにした点にある。なお、図７において、図１に示した第１の実施の形態と共通する部分には同一の符号を付して示し、詳しい説明は省略する。
【００８０】
この第２の実施の形態の表示処理装置では、図８のフローチャートに示すように、ＣＰＵ５がまず、ヘッドライト操作スイッチ９の操作状態からヘッドライトの点灯状況を読取る（ステップＳ１′）。そして、図２に示した第１の実施の形態と同様に、赤外線画像フレームメモリ４から赤外線による第２の風景画像を取込み、可視光画像フレームメモリ２から可視光による第１の風景画像を取込む（ステップＳ２，Ｓ３）。
【００８１】
続いて、可視光による第１の風景画像を輝度に対する所定の閾値をもって２値化して２値化画像を得る（ステップＳ４）。ここで用いる閾値は、ヘッドライト操作スイッチ９から得たヘッドライト点灯状況に基づいて調整する。例えば、自車のヘッドライトが消灯されている状態では昼間走行と見なし、対向車のヘッドライトのような強力な光源の輪郭部分が検出される程度の輝度とし、ヘッドライトが点灯されている状態では夜間走行と見なし、対向車のヘッドライトのような強力な光源の周囲以外の間接反射光がカットされる程度の輝度とする。
【００８２】
次に第１の実施の形態と同様にして、所定の閾値をもって２値化した第１の風景画像の高輝度領域を拡大処理して重畳領域を決定し（ステップＳ５）、この高輝度拡大領域に対応する領域を赤外線による第２の風景画像上から重畳領域として切出し、２値化前の可視光による第１の風景画像の同じ領域上に重畳合成し（ステップＳ６）、これを表示モニタ７に表示させる（ステップＳ７）。
【００８３】
この実施の形態でも、可視光カメラ１も赤外線カメラ３も動画撮影カメラであり、第１、第２の風景画像は動画像であるので、以上のステップＳ２〜Ｓ７の処理を画像フレームごとに繰返すことによって表示モニタ７に重畳合成画像を動画像として表示させる。
【００８４】
こうして、第２の実施の形態の表示処理装置でも第１の実施の形態と同様、図４及び図６に示した画像処理ができ、所定の閾値Ｂによって２値化した可視光による第１の風景画像から対向車のヘッドライトのような高輝度の光源の見かけの大きさで決まる不快グレア領域を高輝度領域２０と決定し、さらに所定の拡大率Ｋを掛けることによってこの高輝度領域２０の周囲に位置して光源からの光がドライバの眼球内で散乱するために眩しさを感じる減能グレア領域まで重畳合成領域２１を拡大し、この重畳合成領域２１に対応する領域２２を第２の風景画像から切出して２値化前の第１の風景画像の該当する部位に合成することにより、対向車のヘッドライトのような高輝度の光源によって眩惑され、ドライバの肉眼では見にくいあるいは見えないような対象物を表示モニタ７に表示されている合成画像上で容易に発見できるようになる。これによって、特に対向車のヘッドライトに入ってしまって歩行者がドライバの視野から消滅する現象が発生するような場合にもその歩行者の存在を表示モニタ７で一瞥して認識できるようになる。
【００８５】
またヘッドライトの点／消灯状態に応じて所定の輝度閾値Ｂを可変設定するようにしたので、運転時の周囲の明るさに合わせて重畳領域を調整することができ、ドライバが自ら感じる眩惑範囲と表示モニタ７上での表示範囲との相違による違和感を低減することができる。
【００８６】
なお、上記の実施の形態における高輝度領域の決定のための輝度閾値の設定は特に限定されるわけではなく、実験を繰返すことによってふさわしい値に設定することができる。また拡大率についても装置機能を単純化するためには省略することができる。そしてその場合に、輝度閾値を若干低めに設定すれば、通常の不快グレア領域よりもやや広めの領域を高輝度領域２０として決定することができるようになり、十分実用できる。
【００８７】
次に、本発明の第３の実施の形態の表示処理装置を図９〜図１４に基づいて説明する。第３の実施の形態の表示処理装置はＣＰＵ５０が実行する画像処理に特徴を有している。まず、図９に基づいて機能構成について説明する。図１に示した第１の実施の形態と同様に、この第３の実施の形態の表示処理装置は、可視光による第１の風景画像を得るための可視光カメラ１、この可視光カメラ１の撮影した第１の風景画像を記憶しておく可視光画像フレームメモリ２、可視光以外の光線又は電磁波による第２の風景画像を撮影するための第２の撮影手段としての赤外線カメラ３、この赤外線カメラ３の撮影した第２の風景画像を記憶しておく赤外線画像フレームメモリ４、後述する所定の画像処理を行うＣＰＵ５０、このＣＰＵ５０の合成処理画像を記憶する重畳画像フレームメモリ６、この重畳画像フレームメモリ６に記憶されている合成画像を表示する表示モニタ７を備えている。
【００８８】
第１及び第２の実施の形態ではＣＰＵ５が、画像の切出しと重畳合成処理の対象となるドライバの視認困難な視界部位として眩惑領域を用い、第１の風景画像上の眩惑領域に対して可視光以外による第２風景画像から対応する領域の画像を切出して第１の風景画像の眩惑領域に重畳する処理を行ったが、この第３の実施の形態のＣＰＵ５０は、画像の切出しと重畳合成処理の対象となるドライバの視認困難な視界部位として低視界（可視光ではドライバが眩惑を起こす高輝度環境、あるいはドライバが周囲の情景と歩行者とを識別できないような低輝度、低コントラストの環境のいずれをも問わない）において歩行者が検出された領域を用い、可視光以外による第２の風景画像上で歩行者の存在している領域を判別してその歩行者の存在している所定の領域の画像を切出し、可視光による第１の風景画像の対応する部位に重畳合成する処理を行うことを特徴とする。
【００８９】
以下、このＣＰＵ５０の実行する画像処理について、図１０のフローチャート及び図１１〜図１４の説明図を用いて説明する。この表示処理装置の電源投入でスタートすれば、可視光カメラ１、赤外線カメラ３、表示モニタ７は起動する。そして可視光カメラ１は設定された視野、画角の可視光による、図１３（ａ）に示すような第１の風景画像を撮影して可視光画像フレームメモリ２に１フレームずつ順次記憶していく。同時に、赤外線カメラ３も設定された視野、画角の赤外線による、図１３（ｂ）に示すような第２の風景画像を撮影して赤外線画像フレームメモリ４に１フレームずつ順次記憶していく。
【００９０】
そしてＣＰＵ５０は赤外線画像フレームメモリ４から図１３（ｂ）に示す赤外線による第２の風景画像を取込み（ステップＳ２１）、可視光画像フレームメモリ２から図１３（ａ）に示す可視光による第１の風景画像を取込む（ステップＳ２２）。
【００９１】
続いて、赤外線による第２の風景画像から歩行者の有無を判別する（ステップＳ２３）。この歩行者の検出には赤外線を放射しやすい頭部の画像の存在を目安とする。つまり、赤外線による第２の風景画像内から重心位置、縦横比、充足率、実面積などの条件を満たす部位の存在を調べる。具体的には特開平０９−１２８５４８号公報に開示されている方法を採用する。これによって、歩行者の存在位置を画面上の２次元座標上で画面全体の幅と高さに対する割合で求める。なお、ここで、実際の表示処理では赤外線画像や可視光画像を表示モニタ７に表示し、その表示画面上で処理する訳ではなく、実際にはＣＰＵ５０が赤外線画像フレームメモリ４の画像フレームデータに対して実行する。
【００９２】
こうして歩行者の存在を検出すれば、続いて図１３（ｃ）に示すように、その歩行者の像を含む所定の領域を切出し領域として決定する（ステップＳ２４）。この赤外線による第２の風景画像に対する切出し領域の決定は、次のようにして行う。
【００９３】
図１１を参照して、画像の水平画角中心から歩行者の頭部までの実距離ｆｅと、画面上での歩行者の身長ｅｂは、カメラの設置高Ｈcamと俯角θ、歩行者の身長Ｈped、カメラからの水平距離Ｌとの間で、それぞれ次の数１〜数３の式で示す関係が存在する。
【００９４】
【数１】

【数２】

【数３】

ここで、歩行者の身長Ｈpedとして成人の平均身長を代表値として当てはめると、上記のｆｅとｅｂとの関係式を得ることができる。一方、図１２を参照して、赤外線による第２の風景画像３１上での歩行者の頭部の検出位置により、画面上にてｆｅに該当する値を得ることができる。これにより、Ｌに応じたｅｂの値を求めることができる。
【００９５】
こうして求めた歩行者の頭部に該当する位置（＝ｅ）と、領域の高さ方向の大きさ（＝ｅｂ）をもとにして画面上で歩行者を包含する図形を当てはめ、切出し領域３２とするのである。
【００９６】
この実施の形態で歩行者を包含する図形としては、（ａ）上部の辺の位置を垂直座標ｅとし、（ｂ）水平方向の中心線を水平座標とし、（ｃ）垂直方向の大きさをｅｂとし、（ｄ）水平方向の大きさをｅｂ／２とする長方形を用いる。
【００９７】
次に、赤外線による第２の風景画像から決定した切出し領域３２の切出しと、切出した領域３２の可視光による第１の風景画像への重畳合成を行う（ステップＳ２５）。この領域切出しの処理では、赤外線画像を画素ごとに走査し、その画素の座標がステップＳ２４で決定した切出し領域３２に含まれているか否かを調べ、含まれている場合には赤外線画像の該当する画素を可視光による第１の風景画像の該当する画素上にコピーする処理を全画面に対して繰返す。
【００９８】
ＣＰＵ５０は上記の重畳合成処理が済んだ重畳合成画像をいったん重畳画像フレームメモリ６に書出し、これを表示モニタ７が表示する（ステップＳ２６）。最終的に表示モニタ７に表示される重畳合成画像は、図１４に示すものとなる。なお、この重畳合成処理では、赤外線画像から切出した切出し領域３２の画像は不透明にして、背面になる第１の風景画像が透けて見えないようにしている。これによって歩行者の画像が鮮明になる。
【００９９】
この第３の実施の形態では、可視光カメラ１も赤外線カメラ３も動画撮影カメラであり、第１、第２の風景画像は動画像であるので、以上のステップＳ２１〜Ｓ２６の処理を画像フレームごとに繰返すことによって表示モニタ７に重畳合成画像を動画像として表示させる。
【０１００】
こうして第３の実施の形態の表示処理装置によれば、対向車のヘッドライトのような高輝度の光源によって眩惑され、あるいは低輝度、低コントラストの環境のためにドライバの肉眼では見にくいあるいは見えない位置に存在する歩行者を表示モニタ７に表示されている合成画像上で容易に発見できるようになる。
【０１０１】
しかも、歩行者を検出した場合にその歩行者を含む限定された領域の画像だけを赤外線のような可視光以外による第２の風景画像から切出して、可視光による第１の風景画像の該当する部位に重畳合成して表示するので、表示画像の大部分はドライバが肉眼で見ている風景と一致し、歩行者の存在する部分だけが強調されるような表示になり、実視野と表示モニタの表示画面上との位置関係の照合がしやすい。
【０１０２】
また歩行者の画像上での大きさに応じて、つまり、歩行者が自車から遠方に存在するか近くに存在するかに応じて切出し領域も広狭変化し、表示モニタ７の表示画面上での歩行者の画像が遠ければ小さく、近ければ大きく表示されることになり、ドライバの肉眼で視認できる実風景に照らして、どの程度接近していると離れているかを容易に把握できる。
【０１０３】
また画像上で歩行者の身体を包含するのと同程度の領域を切出し領域と決定するのに、赤外線画像上にて縦方向の歩行者の存在位置、カメラの設置高、撮影方向の俯角、歩行者の身長から求めるようにしたので、撮影手段以外の距離計測手段を必要とせず、装置構成の簡素化が図れる。
【０１０４】
なお、上記の第３の実施の形態において、切出し領域として歩行者を包含する長方形領域を用いたが、これに限定されず、歩行者の頭部を頂点とし歩行者のほぼ全身を含むだ円形領域とすることができる。
【０１０５】
また比較的近距離に存在する歩行者を目標物とする場合、頭部に該当する大きさを調べ、頭部と体全体との比率、つまり、図１２におけるα：βの値を固定値として、領域３２の高さ方向の大きさｅｂを簡易的に決定して切出し領域を決定することもできる。
【０１０６】
次に、本発明の第４の実施の形態の表示処理装置を図１５〜図１７に基づいて説明する。第４の実施の形態の特徴は、ＣＰＵ５０が歩行者の存在を検出してその歩行者を含む所定の領域を切出し領域として決定すれば、第２の風景画像からの切出し画像を白黒反転処理してから可視光による第１の風景画像に重畳合成処理する点にある。
【０１０７】
この第４の実施の形態の表示処理装置の機能構成は、第３の実施の形態と同様に図９に示したものである。そしてＣＰＵ５０が実行する処理が、図１５のフローチャートに示すようになる。
【０１０８】
すなわち、図１０に示した第３の実施の形態のフローチャートと同様に、赤外線画像フレームメモリ４から赤外線による第２の風景画像を取込み（ステップＳ２１）、可視光画像フレームメモリ２から可視光による第１の風景画像を取込む（ステップＳ２２）。続いて、赤外線による第２の風景画像から歩行者の有無を判別し（ステップＳ２３）、その歩行者の像を含む所定の領域を切出し領域として決定する（ステップＳ２４）。
【０１０９】
次に、赤外線による第２の風景画像から決定した切出し領域の切出し、さらにその白黒反転処理を行い、白黒反転後の画像を可視光による第１の風景画像の該当する部位に重畳合成する（ステップＳ２５′）。このステップＳ２５′での画像の切出しと白黒反転と重畳処理では、赤外線による第２の風景画像を２次元的に１画素ごとに走査し、その画素の座標がステップＳ２４で決定した切出し領域に含まれているか否かを調べる。そして含まれている場合には、赤外線画像の該当する画素について、図１６に示すように明度の反転処理を行い、これを可視光による第１の風景画像の該当する座標の画素上にコピーする処理を全画面に対して繰返す。
【０１１０】
ＣＰＵ５０は上記の重畳合成処理が済んだ重畳合成画像をいったん重畳画像フレームメモリ６に書出し、これを表示モニタ７に表示させる（ステップＳ２６）。最終的に表示モニタ７に表示される重畳合成画像は、図１７に示すものとなる。
【０１１１】
この第４の実施の形態でも、可視光カメラ１も赤外線カメラ３も動画撮影カメラであり、第１、第２の風景画像は動画像であるので、以上のステップＳ２１〜Ｓ２６の処理を画像フレームごとに繰返すことによって表示モニタ７に重畳合成画像を動画像として表示させる。
【０１１２】
こうして第４の実施の形態の表示処理装置によれば、第３の実施の形態と同様の効果を奏し、さらに、赤外線による第２の風景画像の中の歩行者の画像を白黒反転させて可視光による第１の風景画像の該当する部位に重畳合成するので、自ら発光することがない歩行者が赤外線画像上で周囲よりも明るく映る違和感を低減することができる。
【０１１３】
なお、上記の第４の実施の形態においても、切出し領域として歩行者を包含する長方形領域を用いたが、これに限定されず、歩行者の頭部を頂点とし歩行者のほぼ全身を含むだ円形領域とすることができる。また比較的近距離に存在する歩行者を目標物とする場合、頭部に該当する大きさを調べ、頭部と体全体との比率、つまり、図１２におけるα：βの値を固定値として、領域３２の高さ方向の大きさを簡易的に決定して切出し領域を決定することもできる。
【図面の簡単な説明】
【図１】本発明の第１の実施の形態の機能構成を示すブロック図。
【図２】上記の実施の形態による重畳画像の表示処理のフローチャート。
【図３】上記の実施の形態による高輝度領域の拡大処理のフローチャート。
【図４】上記の実施の形態による画像処理の説明図。
【図５】上記の実施の形態による表示画像の説明図。
【図６】上記の実施の形態による高輝度領域の拡大処理の説明図。
【図７】本発明の第２の実施の形態の機能構成を示すブロック図。
【図８】上記の実施の形態による重畳画像の表示処理のフローチャート。
【図９】本発明の第３の実施の形態の機能構成を示すブロック図。
【図１０】上記の実施の形態による重畳画像の表示処理のフローチャート。
【図１１】上記の実施の形態による歩行者の切出し領域を決定するための計算処理の説明図。
【図１２】上記の実施の形態による歩行者の切出し領域を決定するための計算処理の説明図。
【図１３】上記の実施の形態による画像処理の説明図。
【図１４】上記の実施の形態による表示画像の説明図。
【図１５】本発明の第４の実施の形態の重畳画像の表示処理のフローチャート。
【図１６】上記の実施の形態による切出し領域の白黒反転処理の説明図。
【図１７】上記の実施の形態による表示画像の説明図。
【符号の説明】
１ 可視光カメラ
２ 可視光画像フレームメモリ
３ 赤外線カメラ
４ 赤外線画像フレームメモリ
５ ＣＰＵ
６ 重畳画像フレームメモリ
７ 表示モニタ
８ モニタ輝度設定スイッチ
９ ヘッドライト操作スイッチ
２０ 高輝度領域
２１ 重畳領域
２２ 重畳領域
３１ 第２の風景画像
３２ 切出し領域[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a display processing device having a function of assisting a driver's vision in a situation that hinders visual recognition of a scenery outside a vehicle.
[0002]
[Prior art]
Visual information from outside the vehicle is extremely important for the driver for driving. However, it is often necessary to drive in an environment that is dazzled by a low-light environment at night or the presence of a powerful light source such as a light from an oncoming vehicle. In such an environment with poor visibility with the naked eye, even a careful driver tends to be late in finding objects such as pedestrians approaching the vehicle.
[0003]
As a technique for covering such physiological problems on the driver side, for example, Japanese Patent Application Laid-Open No. 5-137030 discloses a scan for detecting an infrared ray and forming an image of a low-light environment landscape where illumination does not reach. An apparatus (conventional example 1) for projecting on a display unit using the unit is disclosed. Japanese Patent Application Laid-Open No. 6-270751 discloses a device (conventional example 2) that issues an alarm when an object such as a pedestrian is present in a dazzling area by a headlight of an oncoming vehicle. Further, JP-A-8-263784 discloses an apparatus (conventional example 3) for synthesizing a temperature distribution in front of a vehicle detected by a temperature detecting means such as an infrared camera with an image obtained by an imaging means such as a video camera. .
[0004]
[Problems to be solved by the invention]
However, the conventional display processing apparatus for assisting the visual perception of the driver has the following problems.
[0005]
In the devices of the above-described conventional examples 1 and 3, not only the vicinity of an object such as a pedestrian but also the entire infrared image on the display means is displayed alone or in combination. On the other hand, in order to clearly discriminate an object such as a pedestrian, detection means and display means with high resolution to some extent are required. However, in the devices of Conventional Example 1 and Conventional Example 3, since the infrared image displayed on the display means shows buildings, planting, sky and clouds, such a high resolution image is displayed on the driver. In the case, there were the following problems.
[0006]
◇ It is difficult for a driver to grasp complicated visual information at a glance while driving.
[0007]
◇ The difference between the low-visibility scenery with the naked eye and the clear image displayed is too large, and it is difficult to collate the position of the object in the display screen within the real field of view.
[0008]
◇ When displaying a moving image on the display means, it is easy for the driver to be unconscious about the entire vehicle because it unconsciously looks at the screen.
[0009]
Further, in the apparatus of Conventional Example 2, since the setting of the dazzling area on the apparatus side does not take into account ambient illuminance and individual differences of drivers, an alarm is issued even though the driver is not actually dazzling, which is bothersome. There was a problem that made me feel.
[0010]
Furthermore, as an application of the device disclosed in the conventional example 1, a device that captures a front landscape with an infrared camera and displays it on a display means when dazzling can be considered. However, if you are dazzled by the presence of a powerful light source such as the headlight of an oncoming vehicle, you will not be able to see a near-distance object that is present in the irradiation range of the headlight of your vehicle and is normally visible to the driver. Therefore, it is impossible for the driver to instantly understand why the object displayed on the display means cannot be seen with the naked eye, and when the dazzling is resolved, the object is approaching as close as possible to the vehicle. There is a problem that can happen.
[0011]
The present invention has been made in order to solve such a conventional problem, and reliably assists the driver's vision in a situation that hinders visual recognition of the scenery outside the vehicle. It is an object of the present invention to provide a display processing device capable of grasping visual information at a glance while driving.
[0012]
Another object of the present invention is to provide a display processing device capable of adjusting the visual assisting action according to the physiological personality of the driver.
[0016]
[Means for Solving the Problems]
Claim 1 The display processing apparatus according to the present invention provides a driver with a first photographing unit that obtains a first landscape image with visible light, a second photographing unit that obtains a second landscape image with light rays or electromagnetic waves other than the visible light, and a driver. Region determining means for determining a region on the first landscape image corresponding to a field of view that is difficult to visually recognize as a combined region, and extracting a region corresponding to the combined region from the second landscape image, and the first landscape. An image composition unit that obtains a composite image by synthesizing the corresponding part of the image and a display unit that displays the composite image are provided.
[0017]
Claim 1 In the display processing apparatus of the invention, the first photographing means obtains a first landscape image by visible light, the second photographing means obtains a second landscape image by light rays or electromagnetic waves other than visible light, and region determining means. However, the region on the first landscape image corresponding to the field of view that is difficult for the driver to visually recognize is determined as the synthesis region. In response to this, the image synthesizing unit cuts out a part corresponding to the synthesis region from the second landscape image, and synthesizes it with the corresponding part of the first landscape image to obtain a synthesized image, which is displayed on the display unit.
[0018]
As a result, the display means can display a normal visible light scene for the part visible to the driver, and can synthesize and display a scenery other than the visible light or electromagnetic waves on the visual field that is difficult to see with the naked eye. This allows the driver to grasp the entire front landscape by looking at the display image on the display means.
[0019]
Claim 2 The invention of claim 1 In the display processing apparatus, the area determining means determines a visual field part that is dazzled by a high-luminance light source as a visual field part that is difficult for the driver to visually recognize. An object that is dazzled by the luminance light source and is difficult or invisible to the driver's naked eyes can be easily found on the composite image displayed on the display means.
[0020]
Claim 3 The invention of claim 1 In the display processing apparatus, the region determining means determines a field-of-view portion where an object is captured in a low-light environment as a field-of-view region that is difficult for the driver to visually recognize, so that it is not projected on an illuminating lamp. Therefore, an object that is difficult to see or cannot be seen with the naked eye of the driver can be easily found on the composite image displayed on the display means.
[0021]
Claim 4 The display processing apparatus of the invention includes a first photographing means for obtaining a first landscape image by visible light, a second photographing means for obtaining a second landscape image by light rays or electromagnetic waves other than the visible light, and the first Region determining means for determining a region that exceeds a predetermined luminance threshold on one landscape image as a composite region, and a region corresponding to the composite region is cut out from the second landscape image, and the first landscape image corresponds to the region The image synthesizing unit is provided with an image synthesizing unit that synthesizes the region and obtains a synthesized image and a display unit that displays the synthesized image.
[0022]
Claim 4 In the display processing apparatus of the invention, the first photographing means obtains a first landscape image by visible light, the second photographing means obtains a second landscape image by light rays or electromagnetic waves other than visible light, and region determining means. Is determined as a synthesis region in the first landscape image that exceeds a predetermined luminance threshold. In response to this, the image synthesizing unit cuts out a part corresponding to the synthesis region from the second landscape image, and synthesizes it with the corresponding part of the first landscape image to obtain a synthesized image, which is displayed on the display unit.
[0023]
Thus, if the brightness threshold is set so that the field of view that is dazzled by the high-brightness light source becomes a composite area, it will be dazzled by the high-brightness light source such as the headlight of an oncoming vehicle, and it will be difficult or difficult to see with the naked eye of the driver. An object that does not exist can be easily found on the composite image displayed on the display means.
[0024]
Claim 5 The invention of claim 4 In the display processing apparatus, the region determining means determines a region expanded vertically and horizontally by a predetermined ratio as a composite region with respect to a portion exceeding the predetermined luminance threshold on the first landscape image. The discomfort glare area determined by the apparent size of a high-intensity light source such as the headlight of an oncoming vehicle and its surroundings are located around it, and the light from the light source scatters within the driver's eyeballs, making it feel dazzling The reduced glare region can be cut out from the second landscape image as a composite region and combined with the corresponding part of the first landscape image, which is dazzled by a high-intensity light source such as a headlight of an oncoming vehicle, An object that is difficult or invisible to the naked eye can be easily found on the composite image displayed on the display means.
[0025]
Claim 6 The invention of claim 4 Or 5 In the display processing apparatus, the screen brightness setting means for variably setting the screen brightness of the display means, and the predetermined brightness threshold used by the area determination means corresponds to the screen brightness set by the screen brightness setting means. Brightness threshold setting means for variably setting.
[0026]
Claim 7 The invention of claim 6 In the display processing apparatus, the luminance threshold value setting unit sets the luminance threshold value low when the screen luminance setting unit sets the screen luminance to a low value.
[0027]
Claim 6 as well as 7 In the display processing apparatus according to the invention, it is possible to synthesize images corresponding to individual differences in which the driver feels dazzling.
[0028]
Claim 8 The invention of claim 5 In the display processing apparatus, the screen brightness setting means for variably setting the screen brightness of the display means, and the vertical and horizontal area enlargement ratios used by the area determination means correspond to the screen brightness set by the screen brightness setting means. And an enlargement ratio setting means for variably setting.
[0029]
Claim 9 The invention of claim 8 In the display processing apparatus, the enlargement ratio setting means sets the area enlargement ratio large when the screen brightness setting means sets the screen brightness low.
[0030]
Claim 8 as well as 9 In the display processing device according to the invention, it is possible to perform image composition corresponding to individual differences in which the driver feels dazzling, including the unpleasant glare region and the reduced glare region.
[0031]
Claim 10 The invention of claim 4 Or 5 The display processing apparatus further includes brightness threshold setting means for variably setting the predetermined brightness threshold used by the area determination means in accordance with the lighting / extinguishing state of the illumination lamp of the vehicle.
[0032]
Claim 1 1 The invention of claim 1 0 In the display processing apparatus, the brightness threshold setting means sets the brightness threshold lower when the headlight of the vehicle is turned on than when the headlight is turned off.
[0033]
Claim 1 0 And 1 1 In the display processing apparatus according to the invention, it is possible to synthesize an image corresponding to the brightness of the landscape by the illumination of the illumination lamp.
[0034]
Claim 1 2 The invention of claim 5 The display processing apparatus further includes an enlargement ratio setting means for variably setting the vertical and horizontal area enlargement ratios used by the area determination means in accordance with the lighting / extinguishing state of the illumination lamp of the vehicle.
[0035]
Claim 1 3 The invention of claim 1 2 In the display processing apparatus, the enlargement ratio setting means sets the area enlargement ratio larger when the headlight of the vehicle is turned on than when the headlight is turned off.
[0036]
Claim 1 2 And 1 3 In the display processing apparatus according to the invention, it is possible to synthesize an image corresponding to the brightness of the landscape by irradiation of the illumination lamp, including the unpleasant glare area and the reduced glare area.
[0037]
Claim 1 4 The display processing apparatus of the invention includes a first photographing means for obtaining a first landscape image by visible light, a second photographing means for obtaining a second landscape image by light rays or electromagnetic waves other than the visible light, and the first Target object discrimination means for discriminating the position of the pedestrian on the second landscape image, and a part including the position of the pedestrian on the second landscape image is cut out, and the corresponding part of the first landscape image And an image composition means for obtaining a composite image and a display means for displaying the composite image.
[0038]
Claim 1 4 In the display processing apparatus of the invention, the first photographing means obtains a first landscape image by visible light, the second photographing means obtains a second landscape image by light rays or electromagnetic waves other than visible light, and target discrimination The means determines the position of the pedestrian on the second landscape image. In response to this, the image synthesizing means cuts out a part including the position of the pedestrian on the second landscape image and synthesizes it with the corresponding part of the first landscape image to obtain a synthesized image and display it on the display means. .
[0039]
As a result, a composite image in which the display means displays a pedestrian that is dazzled by a high-intensity light source such as a headlight of an oncoming vehicle or that is present at a position that is difficult or invisible to the naked eye of the driver due to a low-light environment. It will be easy to find above.
[0040]
Claim 1 5 The invention of claim 1 4 In this display processing apparatus, the image synthesizing unit cuts out a region as much as the region including the pedestrian's body as a part including the pedestrian's presence position.
[0041]
Claim 1 6 The invention of claim 1 5 In the display processing apparatus, the image composition means cuts out the size of the region as the distance to the pedestrian becomes farther as a part including the position where the pedestrian is present.
[0042]
Claim 1 7 The invention of claim 1 5 In the display processing apparatus, the image synthesizing means includes a position of the pedestrian in the vertical direction on the screen, an installation height of the second photographing means, a depression angle in the photographing direction, as a part including the position of the pedestrian. The size of the area is determined based on the height of the pedestrian and is cut out.
[0043]
Claim 1 5 ~ 1 7 In the display processing device of the invention, the cutout area is large when the pedestrian is close, the cutout area is small when the pedestrian is far away, and the target by visible light in the part far away from the pedestrian is not inadvertently painted. The comparison of the positional relationship between the real field of view and the display screen of the display means is easy for the driver to understand.
[0044]
Claim 1 8 The invention of claim 1 4 ~ 1 7 In the display processing apparatus, the image synthesizing unit cuts out a portion including the position of the pedestrian on the second landscape image, inverts it in black and white, and synthesizes it to the corresponding portion of the first landscape image. A composite image is obtained, and the contrast between the image of the pedestrian and the surrounding image on the composite image of the display means is reduced, so that the driver can easily see the image.
[0045]
Claim 19 The invention of claim 1 to 1 8 In the display processing apparatus, the second photographing means is an infrared photographing means, and a second landscape image can be obtained by reliably photographing an image of an area that cannot be captured by visible light photographing by infrared photographing. The display means can display a normal visible light image on the part visible to the driver, and can display an infrared image on a portion of the field of view that is difficult to see with the naked eye or on the target object. It is possible for the driver to glance at the display image of the display means to grasp the entire front landscape or to find the object.
[0047]
【The invention's effect】
Claim 1 According to the invention, a portion corresponding to a visual field portion that is difficult for the driver to visually recognize is extracted from the second landscape image by light rays or electromagnetic waves other than visible light, and is combined with the corresponding portion of the first landscape image to form a composite image. Therefore, the part visible to the driver with respect to the display means displays a landscape with normal visible light, and the view portion difficult to see with the naked eye is synthesized and displayed with a light ray or electromagnetic wave other than visible light. This allows the driver to grasp the entire front landscape by glanceping at the display image on the display means.
[0048]
Claim 2 According to the invention, the visual field part that is dazzled by the high-intensity light source is determined as the visual field part that is difficult for the driver to visually recognize, and the corresponding part is cut out from the second landscape image to the corresponding part of the first landscape image. Since it is synthesized and displayed as a composite image, on the composite image displayed on the display means an object that is dazzled by a high-intensity light source such as a headlight of an oncoming vehicle and that is difficult or invisible to the naked eye of the driver Can be easily discovered.
[0049]
Claim 3 According to the invention, the visual field part in which the object is reflected in the low illumination environment is determined as the visual field part that is difficult for the driver to visually recognize, and the corresponding part is cut out from the second landscape image to correspond to the first landscape image. Since it is combined with the part and displayed as a composite image, the object that is difficult to see or cannot be seen with the naked eye of the driver is displayed on the display means because it exists in a low illuminance area that does not appear in the illumination lamp. It can be easily found on the composite image.
[0050]
Claim 4 According to the invention, a portion that exceeds a predetermined luminance threshold on the first landscape image by visible light is defined as a synthesis region, and a portion corresponding to this synthesis region is cut out from the second landscape image by other than visible light to obtain the first. Since the composite image is displayed on the corresponding part of the landscape image and the composite image is displayed, if the brightness threshold is set so that the visual field part that is dazzled by the high-intensity light source becomes the composite area, An object that is dazzled by the high-intensity light source and is difficult or invisible to the driver's naked eyes can be easily found on the composite image displayed on the display means.
[0051]
Claim 5 According to the invention, the discomfort glare region determined by the apparent size of a high-intensity light source such as a headlight of an oncoming vehicle and the surrounding light are dazzled because light from the light source is scattered in the driver's eyeball. The degrading glare area that feels good can be cut out from the second landscape image other than visible light as a composite area and combined with the corresponding part of the first landscape image by visible light, like a headlight of an oncoming vehicle An object that is dazzled by the high-intensity light source and is difficult or invisible to the driver's naked eyes can be easily found on the composite image displayed on the display means.
[0052]
Claim 6 as well as 7 According to the invention, since the brightness threshold value for cutting out the composite area from the second landscape image other than visible light is variably set according to the screen brightness setting of the display means, it corresponds to the individual difference in which the driver feels dazzling Image composition is possible.
[0053]
Claim 8 as well as 9 According to the invention, the discomfort glare region determined by the apparent size of a high-intensity light source such as a headlight of an oncoming vehicle and the surrounding light are dazzled because light from the light source is scattered in the driver's eyeball. Decomposition glare area that feels good is used as the synthesis area, and the brightness threshold for cutting out this synthesis area is variably set according to the screen brightness setting of the display means, so images that correspond to individual differences that make the driver feel dazzling Synthesis is possible.
[0054]
Claim 1 0 And 1 1 According to the invention, the brightness threshold for cutting out the synthesis region from the second landscape image other than visible light is variably set according to the lighting / lighting-off state of the illumination light. Image composition corresponding to the size is possible.
[0055]
Claim 1 2 And 1 3 According to the invention, the discomfort glare region determined by the apparent size of a high-intensity light source such as a headlight of an oncoming vehicle and the surrounding light are dazzled because light from the light source is scattered in the driver's eyeball. Degradation glare area that feels good is used as the synthesis area, and the brightness threshold for cutting out this synthesis area is variably set according to the lighting / extinguishing status of the lighting lamp. Image synthesis is possible.
[0056]
Claim 1 4 According to this invention, the presence position of the pedestrian is determined on the second landscape image by a light ray or electromagnetic wave other than visible light, and a part including the location of the pedestrian is cut out and visible on the second landscape image. Since the composite image is displayed on the corresponding portion of the first landscape image by light and the composite image is displayed, it is dazzled by a high-intensity light source such as a headlight of an oncoming vehicle, or is difficult to see with the naked eye of the driver due to a low-light environment. Alternatively, a pedestrian present at an invisible position can be easily found on the composite image displayed on the display means.
[0057]
Claim 1 5 ~ 1 7 According to the invention, since the image synthesizing means cut out an area of the same extent as including the pedestrian's body as a part including the position of the pedestrian, the cut-out area is large if the pedestrian is close, If the pedestrian is far away, the cut-out area will be smaller and the driver will be able to check the positional relationship between the real field of view and the display screen of the display means without inadvertently painting the target with visible light away from the pedestrian. Easier to understand.
[0058]
Claim 1 8 According to the invention, the part including the position of the pedestrian on the second landscape image is cut out and reversed in black and white, and is combined with the corresponding part of the first landscape image to display the composite image. The contrast between the image of the pedestrian and the surrounding image on the composite image becomes small so that it can be easily seen by the driver.
[0059]
Claim 19 According to the invention, since the infrared photographing means is used as the second photographing means for obtaining the second landscape image other than the visible light, it is possible to surely capture a pedestrian who is hard to see or cannot see in the real field of view. Can be displayed.
[0060]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 shows a functional configuration of the display processing apparatus according to the first embodiment of the present invention. The display processing apparatus of this embodiment has a normal visible light camera 1 such as a CCD camera as a first photographing means for obtaining a first landscape image by normal visible light, and photographing by the visible light camera 1. A visible light image frame memory 2 for storing the first landscape image, an infrared camera 3 as a second photographing means for photographing a second landscape image by light rays or electromagnetic waves other than visible light, and the infrared camera. 3 is provided with an infrared image frame memory 4 for storing the second landscape image taken by the third person. In addition to the infrared camera, for example, a radio wave radar, an ultrasonic radar, or the like can be employed as the second photographing unit that obtains the second landscape image other than visible light. The visual field of the visible light camera 1 is preferably set so as to include the visual field of the infrared camera 3. Further, by unifying the angle of view and the number of pixels, subsequent image composition processing is simplified, and high-speed processing is possible. Note that these can also be applied to all other embodiments described below.
[0061]
The display processing apparatus of this embodiment further includes a first landscape image by visible light stored in the visible light image frame memory 2 and a second by infrared light other than visible light stored in the infrared image frame memory 4. A CPU 5 that performs a predetermined image composition process described later on the landscape image, a superimposed image frame memory 6 that stores a composite image processed by the CPU 5, a liquid crystal that displays a composite image stored in the superimposed image frame memory 6, A display monitor 7 as an image display means such as a CRT, and a monitor brightness setting switch 8 for setting screen brightness for the display monitor 7 are provided. The monitor brightness setting switch 8 is for the driver to variably set the brightness suitable for himself / herself by viewing the screen brightness of the display monitor 7, and the brightness setting signal is input to the CPU 5 together with the display monitor 7. .
[0062]
Next, the operation of the display processing apparatus having the above configuration will be described. As shown in the flowchart of FIG. 2, when the display processing apparatus is started by turning on the power, the visible light camera 1, the infrared camera 3, and the display monitor 7 are activated. The visible light camera 1 captures the first landscape image by the visible light having the set field of view and angle of view, and sequentially stores it in the visible light image frame memory 2 frame by frame. At the same time, the infrared camera 3 also captures a second landscape image by infrared rays having a set field of view and angle of view and sequentially stores them in the infrared image frame memory 4 frame by frame.
[0063]
Then, the CPU 5 reads the monitor brightness setting value adjusted by the driver while viewing the screen brightness of the display monitor 7 from the monitor brightness setting switch 8 (step S1). Then, the second landscape image by infrared rays (image of FIG. 4D) is taken in from the infrared image frame memory 4 (step S2), and the first landscape image by visible light from the visible light image frame memory 2 (FIG. 4A). )) Is captured (step S3).
[0064]
Subsequently, the first landscape image by visible light is binarized with a predetermined threshold value for luminance to obtain a binarized image shown in FIG. 4B (step S4). The threshold used here is adjusted based on the monitor brightness setting value obtained from the monitor brightness setting switch 8. For example, the brightness used in the daytime is such that the contour of a powerful light source such as a headlight of an oncoming car is detected, and the brightness used at night is a strong light source such as a headlight of an oncoming car. The brightness is such that indirect reflected light other than the surrounding area is cut off.
[0065]
Next, as shown in FIG. 4 (c), the high luminance area 20 of the first landscape image binarized with a predetermined threshold B for luminance is enlarged to determine the overlapping area 21 (step S5). Here, the enlargement ratio is different depending on the threshold value, and the high luminance region by the luminance threshold value Bnight in the night time zone is doubled in both the vertical and horizontal directions with respect to the high luminance region by the luminance threshold value Bday in the daytime time zone. It is set to expand to the size of. That is, assuming that the luminance threshold is B and the enlargement ratio is K, the enlargement ratio K is obtained based on the following linear expression.
[0066]
K = Kday · (3- (B / Bnight))
However, Knight / Kday = 2; Bnight / Bday = 1/2. Further, as the luminance threshold B and the enlargement ratio K, Bday and Kday are used in the daytime, and Bnight and Knight are used in the nighttime.
[0067]
Furthermore, the screen brightness setting of the display monitor 7 is increased because the surroundings are bright during the day, and is decreased because the surroundings are dark at night (this is a setting that is automatically switched in conjunction with lighting / extinguishing of the illumination lamps). Therefore, when the screen setting brightness is high, it is determined that it is daytime, and when the screen setting brightness is low, it is determined that it is nighttime, and the above B and K are switched and set.
[0068]
The details of the enlargement process of the high luminance area are as shown in the flowchart of FIG. Upon entering this process, the processes in steps S12 to S14 are repeated up to the maximum pixel line R for each pixel line m of the binarized image with the predetermined threshold B (steps S11 to S15). Further, after this processing, the processing of steps S17 to S19 is repeated up to the maximum pixel row C for each pixel row n of the binarized image with the predetermined threshold B (steps S16 to S110).
[0069]
In step S12, as shown in FIG. 6A, the binarized data of the mth pixel row among 1 to R is read, and the start end Pmstart of the pixel exceeding the threshold B (that is, the high luminance pixel). The end Pmend is investigated (Steps S12 and S13). Subsequently, as shown in FIG. 6B, the high luminance pixel areas Pmstart to Pmend of the pixel row m are expanded in the horizontal direction by the enlargement factor K to obtain enlarged areas P′mstart to P′mend (step S14). This process is repeated for each of the pixel rows 1 to R (steps S11 to S15).
[0070]
here,

It is.
[0071]
When the horizontal enlargement process is completed, the process proceeds to the vertical enlargement process in steps S16 to S110. Here, as shown in FIG. 6B, the start end Qnstart and end end Qnend of the high luminance region in the n-th pixel column of 1 to C are examined (steps S17 and S18), and FIG. As shown in FIG. 4, the high luminance areas Qnstart to Qnend of the pixel row n are enlarged by the enlargement factor K in the vertical direction to obtain enlarged areas Q′nstart to Q′nend (step S19). This process is repeated for each of the pixel columns 1 to C (steps S16 to S110).
[0072]
here,

It is.
[0073]
Thereby, the enlargement process of step S5 in the flowchart of FIG. 2 is performed, and the high luminance region 20 shown in FIG. 4B in the first landscape image by visible light is enlarged to show the superposition shown in FIG. The area 21 is determined.
[0074]
Next, as shown in FIG. 4D, an area corresponding to the high-intensity enlarged area is cut out as a superimposed area 22 from the second landscape image by infrared rays, and before binarization shown in FIG. Are superimposed on the same region of the first landscape image by the visible light, stored in the superimposed image frame memory 6 (step S6), and displayed on the display monitor 7 (step S7). FIG. 5 shows the superimposed composite image displayed on the display monitor 7. In this superimposing and synthesizing process, the superposed region from which the infrared image is cut out is made opaque so that the first landscape image on the back cannot be seen through.
[0075]
In this embodiment, both the visible light camera 1 and the infrared camera 3 are moving image shooting cameras, and the first and second landscape images are moving images. Therefore, the processes in steps S2 to S7 are repeated for each image frame. Thus, the superimposed composite image is displayed on the display monitor 7 as a moving image.
[0076]
Thus, in the display processing apparatus of the first embodiment, the apparent size of a high-intensity light source such as a headlight of an oncoming vehicle is obtained from the first landscape image by visible light binarized by a predetermined threshold B. The determined unpleasant glare area is determined as the high brightness area 20 and further multiplied by a predetermined magnification K, so that the light from the light source located around the high brightness area 20 is scattered in the driver's eyeball. The superimposition synthesis area is expanded to the degrading glare area where the image is felt, and a region 22 corresponding to the superposition synthesis area 21 is cut out from the second landscape image, and the corresponding part of the first landscape image by the visible light before binarization By combining the image, the display means displays an object that is dazzled by a high-intensity light source such as a headlight of an oncoming vehicle and that is difficult or invisible to the naked eye of the driver. It becomes possible to easily found above. This makes it possible to recognize the presence of the pedestrian at a glance on the display monitor 7 even when the pedestrian enters the headlight of the oncoming vehicle and the pedestrian disappears from the driver's field of view. .
[0077]
In addition, since the predetermined luminance threshold B is variably set according to the luminance setting of the display monitor 7, the overlapping area can be adjusted according to individual differences in which the driver feels dazzling, and the dazzling range that the driver feels himself / herself And the discomfort due to the difference between the display range on the display monitor 7 can be reduced.
[0078]
Note that the setting of the brightness threshold for determining the high brightness area in the above embodiment is not particularly limited, and can be set to a suitable value by repeating the experiment. In addition, the luminance threshold value is not variable, and can be fixed to a value suitable for use in a low-light environment such as a situation where the display processing device of the present invention is required, particularly at night. In this case, the monitor brightness setting switch 8 shown in FIG. 1 is not necessary, and the configuration is simplified. When the luminance threshold value is fixed, if the luminance threshold value is set slightly lower than that in the above embodiment, an area slightly wider than the normal discomfort glare area is determined as the high luminance area 20. Thus, it is possible to provide a practical use without setting the enlargement ratio and enlarging the high-luminance region, and the configuration can be further simplified.
[0079]
Next, a display processing apparatus according to a second embodiment of the present invention will be described with reference to FIGS. The display processing apparatus according to the second embodiment is characterized in that a predetermined threshold value for the CPU 5 to determine a high-luminance area for the first landscape image by visible light is a headlight dot / This is in the point that it is variably set according to the light-off operation state. In FIG. 7, parts common to the first embodiment shown in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted.
[0080]
In the display processing apparatus according to the second embodiment, as shown in the flowchart of FIG. 8, the CPU 5 first reads the lighting state of the headlight from the operation state of the headlight operation switch 9 (step S1 ′). Then, as in the first embodiment shown in FIG. 2, the second landscape image by infrared is taken from the infrared image frame memory 4 and the first landscape image by visible light is taken from the visible light image frame memory 2. (Steps S2 and S3).
[0081]
Subsequently, the first landscape image by visible light is binarized with a predetermined threshold for luminance to obtain a binarized image (step S4). The threshold used here is adjusted based on the headlight lighting status obtained from the headlight operation switch 9. For example, when the headlight of the own vehicle is turned off, it is regarded as daytime driving, the brightness is such that the outline of a powerful light source such as the headlight of an oncoming vehicle is detected, and the headlight is turned on In the case of driving at night, the brightness is set such that indirect reflected light other than around a powerful light source such as a headlight of an oncoming vehicle is cut off.
[0082]
Next, in the same manner as in the first embodiment, the high luminance area of the first landscape image binarized with a predetermined threshold is enlarged to determine a superimposition area (step S5). The region corresponding to is cut out as a superimposition region from the second landscape image by infrared rays and superimposed on the same region of the first landscape image by the visible light before binarization (step S6), and this is displayed on the display monitor 7. (Step S7).
[0083]
Also in this embodiment, since the visible light camera 1 and the infrared camera 3 are both motion picture cameras, and the first and second landscape images are moving images, the processes in steps S2 to S7 are repeated for each image frame. Thus, the superimposed composite image is displayed on the display monitor 7 as a moving image.
[0084]
In this way, the display processing apparatus according to the second embodiment can perform the image processing shown in FIGS. 4 and 6 as in the first embodiment, and the first processing using the visible light binarized by the predetermined threshold B can be performed. An unpleasant glare area determined by the apparent size of a high-intensity light source such as a headlight of an oncoming vehicle is determined as a high-intensity area 20 from the landscape image, and further multiplied by a predetermined enlargement factor K. The superimposition synthesis region 21 is expanded to a degenerated glare region in which the light from the light source is scattered around the driver's eyeball and feels dazzling, and the region 22 corresponding to the superposition synthesis region 21 is defined as the second region 22. By cutting out from a landscape image and combining it with the corresponding part of the first landscape image before binarization, it is dazzled by a high-intensity light source such as a headlight of an oncoming vehicle, and is difficult or difficult to see with the naked eye of the driver. It becomes possible to easily found on the composite image being displayed without such objects on the display monitor 7. This makes it possible to recognize the presence of the pedestrian at a glance on the display monitor 7 even when the pedestrian enters the headlight of the oncoming vehicle and the pedestrian disappears from the driver's field of view. .
[0085]
In addition, since the predetermined luminance threshold B is variably set according to the headlight on / off state, the overlapping area can be adjusted according to the ambient brightness during driving, and the dazzling range felt by the driver himself / herself And the discomfort due to the difference between the display range on the display monitor 7 can be reduced.
[0086]
Note that the setting of the brightness threshold for determining the high brightness area in the above embodiment is not particularly limited, and can be set to a suitable value by repeating the experiment. Also, the enlargement ratio can be omitted to simplify the device function. In that case, if the luminance threshold is set slightly lower, a region slightly wider than the normal discomfort glare region can be determined as the high luminance region 20, which is sufficiently practical.
[0087]
Next, a display processing apparatus according to a third embodiment of the present invention will be described with reference to FIGS. The display processing apparatus according to the third embodiment is characterized by image processing executed by the CPU 50. First, the functional configuration will be described with reference to FIG. Similar to the first embodiment shown in FIG. 1, the display processing apparatus of the third embodiment includes a visible light camera 1 for obtaining a first landscape image by visible light, and the visible light camera 1. A visible light image frame memory 2 for storing the photographed first landscape image, an infrared camera 3 as a second photographing means for photographing a second landscape image by light rays or electromagnetic waves other than visible light, An infrared image frame memory 4 for storing a second landscape image taken by the infrared camera 3, a CPU 50 for performing predetermined image processing to be described later, a superimposed image frame memory 6 for storing a composite processed image of the CPU 50, and this superimposed image A display monitor 7 for displaying the composite image stored in the frame memory 6 is provided.
[0088]
In the first and second embodiments, the CPU 5 uses a dazzling area as a field of view that is difficult to visually recognize by the driver that is the target of image cropping and superimposition and synthesis processing, and is visible to the dazzling area on the first landscape image. The processing of extracting the image of the corresponding region from the second landscape image other than light and superimposing it on the dazzling region of the first landscape image is performed. The CPU 50 of this third embodiment extracts and superimposes the image. Low visibility as a field of view that is difficult for the driver to view (a high-luminance environment in which the driver is dazzled with visible light, or a low-luminance, low-contrast environment in which the driver cannot distinguish surrounding scenes from pedestrians) The area where the pedestrian is detected is determined on the second landscape image other than visible light by using the area where the pedestrian is detected. That cut out images of a predetermined area, and performs a process of superimposing the synthesis into the corresponding sites of the first scenic image by visible light.
[0089]
Hereinafter, the image processing executed by the CPU 50 will be described with reference to the flowchart of FIG. 10 and the explanatory diagrams of FIGS. If the display processing apparatus is started by turning on the power, the visible light camera 1, the infrared camera 3, and the display monitor 7 are activated. The visible light camera 1 shoots a first landscape image as shown in FIG. 13A with visible light having a set field of view and angle of view, and sequentially stores it in the visible light image frame memory 2 frame by frame. Go. At the same time, the infrared camera 3 also captures a second landscape image as shown in FIG. 13B using infrared rays having a set field of view and angle of view and sequentially stores them in the infrared image frame memory 4 frame by frame.
[0090]
Then, the CPU 50 fetches the second landscape image by infrared shown in FIG. 13B from the infrared image frame memory 4 (step S21), and the first scenery by visible light shown in FIG. 13A from the visible light image frame memory 2. A landscape image is captured (step S22).
[0091]
Subsequently, the presence or absence of a pedestrian is determined from the second landscape image using infrared rays (step S23). The detection of the pedestrian is based on the presence of an image of the head that easily emits infrared rays. That is, the existence of a part satisfying conditions such as the position of the center of gravity, the aspect ratio, the sufficiency rate, and the actual area is examined from the second landscape image by infrared rays. Specifically, the method disclosed in JP 09-128548 A is adopted. As a result, the position of the pedestrian is obtained as a ratio to the width and height of the entire screen on the two-dimensional coordinates on the screen. Here, in the actual display processing, an infrared image or a visible light image is displayed on the display monitor 7 and is not processed on the display screen. In practice, the CPU 50 converts the image frame data of the infrared image frame memory 4 into image frame data. Run against.
[0092]
If the presence of a pedestrian is detected in this way, then, as shown in FIG. 13C, a predetermined area including the pedestrian image is determined as a cut-out area (step S24). The cutout area for the second landscape image by infrared rays is determined as follows.
[0093]
Referring to FIG. 11, the actual distance fe from the center of the horizontal angle of view of the image to the head of the pedestrian and the height eb of the pedestrian on the screen are the camera installation height Hcam, the depression angle θ, and the height of the pedestrian. There is a relationship represented by the following equations 1 to 3 between Hped and the horizontal distance L from the camera.
[0094]
[Expression 1]

[Expression 2]

[Equation 3]

Here, when the average height of an adult is applied as a representative value as the pedestrian's height Hped, the above-described relational expression between fe and eb can be obtained. On the other hand, referring to FIG. 12, a value corresponding to fe can be obtained on the screen based on the detection position of the head of the pedestrian on the second landscape image 31 by infrared rays. Thereby, the value of eb according to L can be obtained.
[0095]
Based on the position corresponding to the head of the pedestrian thus obtained (= e) and the size of the area in the height direction (= eb), a figure including the pedestrian is applied on the screen, and the cutout area 32 is applied. It is.
[0096]
In this embodiment, a figure including a pedestrian includes (a) the position of the upper side as the vertical coordinate e, (b) the horizontal center line as the horizontal coordinate, and (c) the size in the vertical direction. eb, and (d) a rectangle whose horizontal size is eb / 2 is used.
[0097]
Next, the cutout region 32 determined from the second landscape image by infrared rays is cut out and superimposed on the first landscape image by the visible light of the cutout region 32 (step S25). In this area extraction process, the infrared image is scanned for each pixel, and it is checked whether or not the coordinates of the pixel are included in the extraction area 32 determined in step S24. The process of copying the pixels to be copied onto the corresponding pixels of the first landscape image by visible light is repeated for the entire screen.
[0098]
The CPU 50 once writes the superimposed composite image that has undergone the above-described superimposition composite processing to the superimposed image frame memory 6 and displays it on the display monitor 7 (step S26). The superimposed composite image finally displayed on the display monitor 7 is as shown in FIG. In this superimposing and combining process, the image of the cutout region 32 cut out from the infrared image is made opaque so that the first landscape image on the back cannot be seen through. As a result, the image of the pedestrian becomes clear.
[0099]
In the third embodiment, the visible light camera 1 and the infrared camera 3 are both moving image capturing cameras, and the first and second landscape images are moving images. Therefore, the processes in steps S21 to S26 described above are performed on an image frame. The superimposed composite image is displayed as a moving image on the display monitor 7 by repeating each time.
[0100]
Thus, according to the display processing apparatus of the third embodiment, it is dazzled by a high-intensity light source such as a headlight of an oncoming vehicle, or is difficult or invisible to the naked eye of the driver due to a low-intensity and low-contrast environment. A pedestrian present at the position can be easily found on the composite image displayed on the display monitor 7.
[0101]
In addition, when a pedestrian is detected, only an image of a limited area including the pedestrian is cut out from the second landscape image other than visible light such as infrared rays, and the first landscape image corresponding to the visible light corresponds. Since it is superimposed on the part and displayed, most of the displayed image matches the scenery that the driver sees with the naked eye, and only the part where the pedestrian is present is emphasized. It is easy to collate the positional relationship with the display screen.
[0102]
Depending on the size of the pedestrian on the image, that is, depending on whether the pedestrian is located far away or close to the vehicle, the cutout area also changes widely, and on the display screen of the display monitor 7 If the pedestrian's image is far away, it will be displayed as small, and if it is close, it will be displayed large, and it will be easy to grasp how far away it is in light of the actual scenery that can be visually recognized by the driver's naked eye.
[0103]
In addition, in order to determine a region equivalent to the pedestrian's body on the image as the cutout region, the position of the pedestrian in the vertical direction on the infrared image, the installation height of the camera, the depression angle in the shooting direction, Since it is obtained from the height of the pedestrian, no distance measuring means other than the photographing means is required, and the apparatus configuration can be simplified.
[0104]
In the third embodiment, a rectangular region including a pedestrian is used as the cutout region. However, the present invention is not limited to this, and the elliptical region includes the pedestrian's head as a vertex and substantially the entire pedestrian. Can be an area.
[0105]
When a target is a pedestrian present at a relatively short distance, the size corresponding to the head is examined, and the ratio between the head and the whole body, that is, the value of α: β in FIG. The cutout region can be determined by simply determining the size eb of the region 32 in the height direction.
[0106]
Next, a display processing apparatus according to a fourth embodiment of the present invention will be described with reference to FIGS. A feature of the fourth embodiment is that when the CPU 50 detects the presence of a pedestrian and determines a predetermined area including the pedestrian as a cut-out area, the cut-out image from the second landscape image is subjected to black-and-white reversal processing. Then, it is superimposed on the first landscape image by visible light.
[0107]
The functional configuration of the display processing apparatus according to the fourth embodiment is the same as that shown in FIG. 9 as in the third embodiment. The processing executed by the CPU 50 is as shown in the flowchart of FIG.
[0108]
That is, similar to the flowchart of the third embodiment shown in FIG. 10, the second landscape image by infrared is taken from the infrared image frame memory 4 (step S21), and the second landscape image by visible light is taken from the visible light image frame memory 2. One landscape image is captured (step S22). Subsequently, the presence or absence of a pedestrian is determined from the second landscape image using infrared rays (step S23), and a predetermined area including the pedestrian image is determined as a cut-out area (step S24).
[0109]
Next, the cut-out area determined from the second landscape image by infrared rays is cut out, and further the black-and-white reversal process is performed, and the image after black-and-white reversal is superimposed and synthesized on the corresponding part of the first landscape image by visible light (step) S25 '). In the image cropping, black-and-white reversal, and superimposition processing in step S25 ′, the infrared second landscape image is scanned two-dimensionally pixel by pixel, and the coordinates of the pixels are included in the cropping region determined in step S24. Check if it is. If it is included, the inversion processing of the brightness is performed on the corresponding pixel of the infrared image as shown in FIG. 16, and this is copied onto the pixel of the corresponding coordinate of the first landscape image by visible light. Repeat the process for all screens.
[0110]
The CPU 50 once writes the superimposed composite image that has undergone the above-described superimposition composite processing in the superimposed image frame memory 6 and displays it on the display monitor 7 (step S26). The superimposed composite image finally displayed on the display monitor 7 is as shown in FIG.
[0111]
Also in the fourth embodiment, the visible light camera 1 and the infrared camera 3 are both moving image capturing cameras, and the first and second landscape images are moving images. Therefore, the processing in the above steps S21 to S26 is performed on an image frame. The superimposed composite image is displayed as a moving image on the display monitor 7 by repeating each time.
[0112]
Thus, according to the display processing apparatus of the fourth embodiment, the same effects as those of the third embodiment can be obtained. Further, the pedestrian image in the second landscape image by infrared rays is reversed in black and white to be visible. Since it is superimposed and synthesized on the corresponding part of the first landscape image by light, it is possible to reduce the uncomfortable feeling that a pedestrian who does not emit light by himself / herself appears brighter than the surroundings on the infrared image.
[0113]
In the fourth embodiment, the rectangular area including the pedestrian is used as the cutout area. However, the present invention is not limited to this, and includes the pedestrian's head as a vertex and almost the entire pedestrian. It can be a circular area. When a target is a pedestrian present at a relatively short distance, the size corresponding to the head is examined, and the ratio between the head and the whole body, that is, the value of α: β in FIG. The cutout area can be determined by simply determining the size of the area 32 in the height direction.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a functional configuration of a first embodiment of the present invention.
FIG. 2 is a flowchart of superimposed image display processing according to the embodiment.
FIG. 3 is a flowchart of high-intensity region enlargement processing according to the embodiment.
FIG. 4 is an explanatory diagram of image processing according to the embodiment.
FIG. 5 is an explanatory diagram of a display image according to the above embodiment.
FIG. 6 is an explanatory diagram of enlargement processing of a high luminance area according to the above embodiment.
FIG. 7 is a block diagram showing a functional configuration of a second embodiment of the present invention.
FIG. 8 is a flowchart of superimposed image display processing according to the embodiment.
FIG. 9 is a block diagram showing a functional configuration of a third embodiment of the present invention.
FIG. 10 is a flowchart of superimposed image display processing according to the embodiment.
FIG. 11 is an explanatory diagram of a calculation process for determining a pedestrian cutout area according to the embodiment.
FIG. 12 is an explanatory diagram of a calculation process for determining a pedestrian cutout region according to the embodiment.
FIG. 13 is an explanatory diagram of image processing according to the embodiment.
FIG. 14 is an explanatory diagram of a display image according to the above embodiment.
FIG. 15 is a flowchart of superimposed image display processing according to the fourth embodiment of the present invention;
FIG. 16 is an explanatory diagram of black and white reversal processing of a cutout area according to the above embodiment.
FIG. 17 is an explanatory diagram of a display image according to the above embodiment.
[Explanation of symbols]
1 Visible light camera
2 Visible light image frame memory
3 Infrared camera
4 Infrared image frame memory
5 CPU
6 Superimposed image frame memory
7 Display monitor
8 Monitor brightness setting switch
9 Headlight operation switch
20 High brightness area
21 Superimposition area
22 Overlapping area
31 Second landscape image
32 Cutout area

Claims (19)

First photographing means for obtaining a first landscape image by visible light;
Second photographing means for obtaining a second landscape image by light rays or electromagnetic waves other than the visible light;
A region determining means for determining a portion on the first landscape image corresponding to a field of view difficult for the driver to visually recognize as a composite region;
Image synthesizing means for cutting out a portion corresponding to the synthesis region from the second landscape image and synthesizing the corresponding portion of the first landscape image to obtain a synthesized image;
A display processing device comprising display means for displaying the composite image. The display processing apparatus according to claim 1, wherein the area determination unit determines a visual field part that is dazzled by a high-luminance light source as a visual field part that is difficult for the driver to visually recognize. The display processing apparatus according to claim 1, wherein the area determination unit determines a field of view where an object is captured in a low illumination environment as a field of view that is difficult for the driver to visually recognize. First photographing means for obtaining a first landscape image by visible light;
Second photographing means for obtaining a second landscape image by light rays or electromagnetic waves other than the visible light;
Region determining means for determining a region on the first landscape image that exceeds a predetermined luminance threshold as a composite region;
Image synthesizing means for cutting out a portion corresponding to the synthesis region from the second landscape image and synthesizing the corresponding portion of the first landscape image to obtain a synthesized image;
A display processing device comprising display means for displaying the composite image. The area determination unit determines an area enlarged vertically and horizontally by a predetermined ratio as a composite area with respect to a portion exceeding the predetermined luminance threshold on the first landscape image. Display processing device. Screen brightness setting means for variably setting the screen brightness of the display means;
The display processing apparatus according to claim 4, further comprising: a luminance threshold setting unit that variably sets the predetermined luminance threshold used by the region determining unit in accordance with the screen luminance set by the screen luminance setting unit. . The display processing apparatus according to claim 6, wherein the luminance threshold setting unit sets the luminance threshold low when the screen luminance setting unit sets the screen luminance low. Screen brightness setting means for variably setting the screen brightness of the display means;
The display processing apparatus according to claim 5, further comprising: an enlargement ratio setting unit that variably sets the vertical and horizontal area enlargement ratio used by the area determination unit in accordance with the screen brightness set by the screen brightness setting unit. 9. The display processing apparatus according to claim 8, wherein the enlargement ratio setting means sets the area enlargement ratio large when the screen brightness setting means sets the screen brightness low. The display processing apparatus according to claim 4, further comprising a luminance threshold setting unit configured to variably set the predetermined luminance threshold used by the region determining unit according to a lighting / extinguishing state of an illumination lamp of the vehicle. The display processing apparatus according to claim 10, wherein the brightness threshold setting unit sets the brightness threshold lower when the headlight of the vehicle is turned on than when the headlight is turned off. The display processing apparatus according to claim 5, further comprising an enlargement ratio setting unit configured to variably set the vertical and horizontal area enlargement ratios used by the area determination unit according to the lighting / extinguishing state of the illumination lamp of the vehicle. The display processing device according to claim 12, wherein the enlargement ratio setting means sets the area enlargement ratio when the headlight of the vehicle is turned on larger than when the headlight is turned off. First photographing means for obtaining a first landscape image by visible light;
Second photographing means for obtaining a second landscape image by light rays or electromagnetic waves other than the visible light;
Target discrimination means for discriminating the position of the pedestrian on the second landscape image;
Image synthesizing means for cutting out a part including the position of the pedestrian on the second landscape image and synthesizing it with the corresponding part of the first landscape image;
A display processing device comprising display means for displaying the composite image. The display processing apparatus according to claim 14, wherein the image synthesizing unit cuts out a region having the same extent as that including the body of the pedestrian as a part including the position where the pedestrian is present. The display processing according to claim 15, wherein the image synthesizing unit cuts out the size of the region smaller as the distance to the pedestrian becomes farther as a part including the position where the pedestrian is present. apparatus. The image synthesizing means includes the position of the pedestrian in the vertical direction on the screen, the height of the second photographing means, the depression angle in the photographing direction, and the height of the pedestrian as a part including the position of the pedestrian. The display processing apparatus according to claim 15, wherein the size of the area is determined and cut out. The image synthesizing means cuts out a part including the position of the pedestrian on the second landscape image, inverts it in black and white, and synthesizes it with a corresponding part of the first landscape image to obtain a composite image. The display processing device according to claim 14, wherein the display processing device is a display processing device. The display processing apparatus according to claim 1, wherein the second imaging unit is an infrared imaging unit.

Images