JP2004320525A - Vehicle perimeter visual recognition device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a vehicle perimeter visual recognition device capable of displaying a vehicle perimeter image at a state that it is made legible by correcting its camera optical distortion, making a driver, etc. easily recognize positional relation and a sense of distance between the present vehicle and each point around it and transition of the position of the present vehicle in accordance with travel, etc. based on the image and enhancing visibility of the vehicle perimeter based on a displayed image. <P>SOLUTION: An image processor 3 performs lens distortion correction, etc. to an original image of the vehicle perimeter to be obtained by imaging of an imaging apparatus 1, predicts a prediction trace on which the vehicle or a reference point provided around the vehicle passes in accordance with travel of the vehicle based on a detection result of a steering sensor 5 and displays the predicted prediction trace on an image to which correction processing is performed in piles. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

【００２８】
ここで、上記式（１）のＣｘ，Ｃｙは原画像Ｉ１上に設定したｘｙ座標系の原点に対するレンズ歪みの中心座標（Ｃｘ，Ｃｙ）の座標値に対応しており、ｋはレンズ歪み係数に対応しており、αは撮像装置１における撮像素子の縦横比に対応し、縦：横＝２：１であればα＝２となる。なお、図４及び図５の設定例では、レンズ歪みの中心座標（Ｃｘ，Ｃｙ）とｘｙ座標系の原点とが一致している。
【００２９】
そして、上記式（１）を補正後の座標値ｘ２，ｙ２について解くと下記の関係式が得られる。
【００３０】
【数２】

【００３１】
これより、例えば、原画像Ｉ１上の各点の画像データ（画素データ等）の位置（座標）を、その補正前の座標値ｘ１，ｙ１を上記式（２）に代入して得られる座標点（ｘ２，ｙ２）に補正することにより、レンズ歪みが補正される。なお、原画像Ｉ１の各点の座標点（ｘ１，ｙ１）と対応する補正後の座標点（ｘ２，ｙ２）との対応関係を示す変換テーブルを予めＲＯＭに内蔵しておき、その変換テーブルを用いて歪み補正を行うのが望ましい。これによって、座標値についての計算を逐次行う必要がなくなるという利点がある。
【００３２】
図６は、図３の原画像Ｉ１に対して歪み補正処理を施して得られる画像を示す図である。図６に示されるように、歪み補正後の画像Ｉ２はカメラ光学的歪みが補正されていることが分かる。
【００３３】
次に、第２の処理モードによる処理について説明する。この第２の処理モードでは、原画像Ｉ１に対して上述の歪み補正処理を施した上に、その処理後の画像Ｉ２に対して第１の視点変換処理が行われる。この第１の視点変換処理とは、画像Ｉ２を、対応する撮像領域Ａをあたかもその真上から見たような画像に変換するものである。
【００３４】
第１の視点変換処理の具体的内容について、図７を参照して説明する。図７中におけるｘｙ座標系は、レンズ歪み補正処理後の画像Ｉ２に対応したカメラ座標系に対応し、ＸＹＺ座標系は世界座標系に対応している。ｘｙ座標系とＸＹＺ座標系との関係は、撮像装置１の撮像光軸ＧがＸＹＺ座標系の原点Ｏを通り、かつ、撮像光軸ＧがＹＺ面と平行をなし、かつ、ｘｙ座標系の原点ｏとＸＹＺ座標系の原点Ｏとの間の距離が撮像装置１の焦点距離Ｆと等しくなるように設定されている。
【００３５】
また、図７中の仮想平面Ｐは、地面に対応しており、ＸＹＺ座標系においてＸＹ面と平行をなし、Ｚ＝−Ｈを通る。このＨは撮像装置１の地面からの取付高さに対応している。また、撮像光軸ＧとＹ軸とのなす角θは、撮像装置１の斜め下向きに設定された撮像光軸Ｇが水平面となす角（カメラ取付角）に対応している。
【００３６】
そして、第１の視点変換処理は、カメラ座標系（ｘｙ座標系）の画像Ｉ２を、世界座標系（ＸＹＺ座標系）に設定した仮想平面Ｐに原点Ｏに関して対称になるように投影（写像）することにより行われる。この写像において、カメラ座標系の点Ｐ２（ｘ２，ｙ２）と世界座標系の点Ｐ３（ｘ３，ｙ３，−Ｈ）との間には、幾何学的な関係より、下記の関係式が成り立つ。
【００３７】
【数３】

【００３８】
上記式（３）を用いれば、座標点（ｘ３，ｙ３）に対応するカメラ座標系の座標点（ｘ２，ｙ２）が求まる。これにより、例えば、構成すべき画像上の各座標点（ｘ３，ｙ３）に対応する画像Ｉ２上の座標点（ｘ２，ｙ２）の画像データ（画素データ等）を割り当てて画像を変換することにより、画像Ｉ２を、撮像領域Ａをあたかもその真上から見たような画像に変換することができる。なお、原画像Ｉ１に対してレンズ歪み補正処理及び第１の視点変換処理を施して得られる画像上における各座標点（ｘ３，ｙ３）と原画像Ｉ１上の各座標点（ｘ１，ｙ１）との対応関係を示す変換テーブルを予めＲＯＭに内蔵しておき、その変換テーブルを用いて歪み補正及び第１の視点変換処理を行うのが望ましい。これによって、座標値についての計算を逐次行う必要がなくなるという利点がある。
【００３９】
次に、第３の処理モードによる処理について説明する。図８に示すように、撮像装置１の撮像光軸方向Ｊが車両の直進方向Ｋと平行でなく、直進方向Ｋと所定角度ψを成している。このため、原画像Ｉ１あるいはレンズ歪み補正処理等を施した画像を表示装置７に表示した際に、図６に示すように、表示画像上における車両の直進方向Ｌが画像の縦方向Ｍと平行になっていないため、運転者等がその表示画像中の景色と自車との位置関係を直感的に視認しにくい場合がある。
【００４０】
そこで、本実施形態では、第３の処理モードの処理では、原画像Ｉ１に対して上述の歪み補正処理及び第１の視点変換処理を施した上に、その処理後の画像に対して所定の回転変換処理を行うようになっている。
【００４１】
具体的には、回転変換処理は、図９に示すように、その画像上における車両の直進方向Ｌが画像の縦方向Ｍと平行な関係になるように、画像上の各点Ｐ３（ｘ３，ｙ３）の画像データ（画素データ等）を画像と平行な面内で角度ψだけ回転座標変換した点Ｐ４（ｘ４，ｙ４）に回転させることにより行われる。
【００４２】
すなわち、点Ｐ３（ｘ３，ｙ３）から点Ｐ４（ｘ４，ｙ４）への座標変換（アフィン変換）は、下記の関係式で与えられる。
【００４３】
【数４】

【００４４】
上記式（４）による変換の逆変換は、
【００４５】
【数５】

【００４６】
の関係式で与えられる。この式（５）を用いれば、構成すべき画像上における各点（ｘ４，ｙ４）に対応する回転処理前の画像の各点（ｘ３，ｙ３）が求まる。これにより、例えば、構成すべき画像上の各座標点（ｘ４，ｙ４）に対応する回転処理前のもとの画像の座標点（ｘ３，ｙ３）の画像データ（画素データ等）を割り当てて画像を変換することにより、画像上における車両の直進方向Ｌと画像の縦方向Ｍとが平行な関係にある画像を得ることができる。なお、原画像Ｉ１に対してレンズ歪み補正処理、第１の視点変換処理及び回転変換処理を施して得られる画像上における各座標点（ｘ４，ｙ４）と原画像Ｉ１上の各座標点（ｘ１，ｙ１）との対応関係を示す変換テーブルを予めＲＯＭに内蔵しておき、その変換テーブルを用いて歪み補正、第１の視点変換処理及び回転変換処理を行うのが望ましい。これによって、座標値についての計算を逐次行う必要がなくなるという利点がある。
【００４７】
次に、第４の処理モードによる処理について説明する。この第４の処理モードでは、原画像Ｉ１に対して上述の歪み補正処理、第１の視点変換処理及び回転変換処理を施した上に、その処理後の画像に対して第２の視点変換処理が行われる。この第２の視点変換処理とは、画像を、対応する撮像領域Ａをあたかも助手席の窓から顔を出して見下ろすようにして視認したような画像に変換するものである。
【００４８】
第２の視点変換処理の具体的内容について、図１０を参照して説明する。図１０中におけるＸＹＺ座標系は世界座標系に対応している。また、ｘｙ座標系は、視認者が助手席の窓から顔を出して撮像領域Ａを視認する際の視点位置を仮想的に設定し、その視点位置から撮像領域Ａの撮像を行って得られる画像の座標系（仮想視点座標系）に対応している。ｘｙ座標系とＸＹＺ座標系との関係は、上記視点位置に仮想的に設置された撮像装置１の撮像光軸ＮがＸＹＺ座標系の原点Ｏを通り、かつ、撮像光軸ＮがＹＺ面と平行をなし、かつ、ｘｙ座標系の原点ｏとＸＹＺ座標系の原点Ｏとの間の距離が撮像装置１の焦点距離Ｆと等しくなるように設定されている。
【００４９】
また、図１０中の仮想平面Ｐ’は、地面に対応しており、ＸＹＺ座標系においてＸＹ面と平行をなし、Ｚ＝−Ｈ’を通る。このＨ’は前記視点位置の地面からの取付高さに対応している。また、撮像光軸ＮとＹ軸とのなす角φは、前記視点位置から撮像領域Ａを見下ろす方向の水平面となす角に対応している。
【００５０】
そして、第２の視点変換処理は、前述の歪み補正処理、第１の視点変換処理及び回転変換処理が施された仮想平面Ｐ’上の画像（もとの画像）を、仮想視点座標系（ｘｙ座標系）に原点Ｏに関して対称となるように投影（写像）することにより行われる。この写像において、仮想平面Ｐ’上のもとの画像上の点Ｐ４（ｘ４，ｙ４，−Ｈ’）と仮想視点座標系の点Ｐ５（ｘ５，ｙ５）との間には、幾何学的な関係より、下記の関係式が成り立つ。
【００５１】
【数６】

【００５２】
上記式（６）を用いれば、座標点（ｘ５，ｙ５）に対応する座標点（ｘ４，ｙ４）が求まる。これにより、例えば、構成すべき画像上の各座標点（ｘ５，ｙ５）に対応するもとの画像上の座標点（ｘ４，ｙ４）の画像データ（画素データ等）を割り当てて画像を変換することにより、助手席の窓から顔を出して撮像領域Ａを見下ろしたときに視認されるような画像を得ることができる。なお、原画像Ｉ１に対してレンズ歪み補正処理、第１の視点変換処理、回転変換処理及び第２の視点変換処理を施して得られる画像上における各座標点（ｘ５，ｙ５）と原画像Ｉ１上の各座標点（ｘ１，ｙ１）との対応関係を示す変換テーブルを予めＲＯＭに内蔵しておき、その変換テーブルを用いて歪み補正、第１の視点変換処理、回転変換処理及び第２の視点変換処理を行うのが望ましい。これによって、座標値についての計算を逐次行う必要がなくなるという利点がある。
【００５３】
図１１は、図３の原画像Ｉ１に対して第４の処理モードによる歪み補正処理、第１の視点変換処理、回転変換処理及び第２の視点変換処理を施して得られる画像を示す図である。図１１に示されるように、処理後の画像Ｉ３は、カメラ光学的歪みが補正され、画像Ｉ３上における車両の直進方向Ｌと画像の縦方向Ｍとが平行な関係にあり、しかも、あたかも助手席の窓から顔を出して撮像領域Ａを見下ろしたときに視認されるような画像に変換されていることが分かる。ここで、図１１の画像Ｉ３は、図１２に示す略矩形の領域Ａ１からハッチングを付して示す領域Ａ２，Ａ３を除いた領域を撮像領域Ａとして撮像して得られた画像である。また、画像Ｉ３上におけるハッチングを付して示す部分Ｂ５，Ｂ６は、変換処理に伴って画像の欠けが生じた部分を示しており、所定の色（例えば、黒色）によるベタ画像が表示される。
【００５４】
次に、上述の第１ないし第４の処理モードの切り替え動作等について説明する。撮像装置１の撮像によって得られた原画像Ｉ１に対して第１ないし第４のいずれの処理モードによる処理を施して表示装置７に表示させるかは、運転者等が入力装置９を介して操作入力を行うことにより切り替えられるようになっている。すなわち、例えば、第１の処理モードが選択された場合には、原画像Ｉ１に対してレンズ歪み補正処理が施され、その処理後の画像Ｉ２が表示装置７に表示され、第４の処理モードが選択された場合には、原画像Ｉ１に対してレンズ歪み補正処理、第１の視点変換処理、回転変換処理及び第２の視点変換処理が施され、その処理後の画像Ｉ３が表示装置７に表示される。
【００５５】
また、原画像Ｉ１を第１ないし第４の処理モードによる処理を施して表示するか、それともその処理を施さずに表示するかについても、運転者等が入力装置９を介して操作入力を行うことにより切り替えられるようになっている。すなわち、原画像Ｉ１を処理を施して表示するモードが選択されている場合には、第１ないし第４の処理モードのうちのその時点で選択されている処理モードによる処理が原画像Ｉ１に施され、その処理後の画像が表示装置７に表示され、原画像Ｉ１をそのまま表示装置７に表示させるモードが選択されている場合には、原画像Ｉ１ががそのまま表示装置７に表示される。
【００５６】
次に、画像処理装置３の予測軌跡表示機能について説明する。画像処理装置３は、車両又は車両周辺に設けられた基準点が車両の走行に伴って通過してゆく予測軌跡Ｒ１，Ｒ２（図１３参照）をステアリングセンサ５の検出結果に基づいて予測し、その予測軌跡Ｒ１，Ｒ２を、表示装置７に表示する撮像領域Ａの撮像画像上の対応する位置に重ねた状態で表示する。本実施形態では、前記基準点としては、車両の左端部（ここでは、車両前端側における左側のコーナ部１１の端部）に対応する地面上の地点に対応して設けられた第１の基準点と、その左端部（例えば、コーナ部１１の端部）から左方向に所定距離（例えば、５０ｃｍ）だけ離れた地面上の地点に対応して設けられた第２の基準点とが設定されている。そして、予測軌跡Ｒ１は第１の基準点に対応しており、予測軌跡Ｒ２は第２の基準点に対応している。
【００５７】
予測軌跡Ｒ１，Ｒ２の導出方法としては種々の方法が考えられる。例えば、一例として次の方法が考えられる。すなわち、予め第１及び第２の基準点に対応する撮像画像Ｉ１上の各座標点（基準座標点）をメモリに登録しておき、ステアリングセンサ５の検出信号が示すその時点のステアリングの舵角に基づいて、撮像画像Ｉ１上においてその各基準座標点が車両の進行に伴って移動する軌跡を計算する。そして、算出された予測軌跡Ｒ１，Ｒ２の描画像を作成し、撮像画像Ｉ１上の対応する位置に重ね合わせて、重畳画像を作成し、その撮像画像と予測軌跡Ｒ１，Ｒ２の描画像とに対して一緒に上述の第１ないし第４の処理モードによる処理を施して表示を行う。
【００５８】
ここで、図１３及び図１４は原画像Ｉ１に対して第４の処理モードによる処理を施して得られた画像Ｉ３上に予測軌跡Ｒ１，Ｒ２が重ねて表示された例を示し、図１３はステアリングの舵角が０のときの予測軌跡Ｒ１，Ｒ２の状態を示し、図１４はステアリングが右に切られているときの予測軌跡Ｒ１，Ｒ２の状態を示している。また、図１５は原画像Ｉ１に対して第１の処理モードによる処理を施して得られた画像Ｉ２上にステアリングの舵角が０のときの予測軌跡Ｒ１，Ｒ２が重ねて表示された例を示し、図１６は原画像Ｉ１上にステアリングの舵角が０のときの予測軌跡Ｒ１，Ｒ２が重ねて表示された例を示している。
【００５９】
また、画像処理装置３の機能には、予測軌跡Ｒ１，Ｒ２の表示のオン、オフ切り替えを行う機能が含まれている。すなわち、運転者等が入力装置９に対して所定の操作入力を行うと、それに応答して、画像処理装置３が、予測軌跡Ｒ１，Ｒ２を撮像画像上に重ねて表示するか否かのオン、オフ切り替えを行うようになっている。
【００６０】
なお、図１３等に示す例では、２つの予測軌跡Ｒ１，Ｒ２の両方を表示しているが、いずれか一方のみを表示するようにしてもよく、あるいは、入力装置９からの操作入力によりいずれの予測軌跡Ｒ１，Ｒ２を表示させるかを選択できるようにしてもよい。
【００６１】
また、本実施形態では第１及び第２の基準点に対応する２つの予測軌跡Ｒ１，Ｒ２を表示するようにしたが、３つ以上の基準点を設け、それらに対応する３つ以上の予測軌跡を表示するようにしてもよい。また、第２の基準点についても、コーナ部１１の端部から５０ｃｍ以上（例えば、１ｍ）離れた地面上の地点に設定してもよい。
【００６２】
このようにして得られた表示画像を参照することによって、図１７に示すように他の車両等の障害物の近傍をすり抜ける際や、図１８に示すように道路の側縁部等の仮想線で示すスペースＳに幅寄せして駐車する際や、図１９に示すように駐車場の狭い駐車スペースに駐車する際などにおいて、車両周辺の障害物等の状況を的確に認識することができる。
【００６３】
以上のように、本実施形態よれば、上述の第１ないし第４のいずれかの処理モードによる処理を原画像Ｉ１に施して得られる画像上に、車両又はその周辺に設けられた基準点の車両の走行に伴って通過してゆく予測軌跡Ｒ１，Ｒ２を重ねて表示することができるため、車両周辺の画像をそのカメラ光学的歪みを補正して見やすくした状態で表示することができるとともに、表示画像に基づいて自車とその周辺の各地点との位置関係及び距離感や、走行に伴う自車位置の推移等を運転者等に容易に認識させることができ、表示画像に基づく車両周辺の視認性の向上が図れる。
【００６４】
さらに、本実施形態では、車両の左端部に対応する地面上の地点に対応して設けられた第１の基準点、及び左端部から所定距離だけ離れた地面上の地点に対応して設けられた第２の基準点に対応する予測軌跡Ｒ１，Ｒ２を画像に重ねて表示するため、表示画像に基づいて自車とその周辺の各地点との位置関係及び距離感や、走行に伴う自車位置の推移等を運転者等により容易に認識させることができる。
【００６５】
また、上述の第２又は第３の処理モードによる処理では、原画像Ｉ１を、その対応する撮像領域Ａをあたかも真上から見たような画像に変換して表示するため、画像を直感的に分かりやすい状態で表示することができる。
【００６６】
さらに、上述の第３又は第４の処理モードによる処理では、原画像Ｉ１を、その画像Ｉ１上における車両の直進方向が画像の縦方向と平行な関係になるように変換して表示するため、画像を直感的に分かりやすい状態で表示することができる。
【００６７】
また、上述の第４の処理モードによる処理では、撮像領域Ａの原画像Ｉ１を、その撮像領域Ａをあたかも助手席の窓から顔を出して見下ろすようにして視認したような画像Ｉ３に変換して表示するため、画像Ｉ３を直感的に分かりやすい状態で表示することができる。
【００６８】
さらに、原画像Ｉ１に上述の第１ないし４の処理モードによる処理を施すか否か、及び、処理を施す場合に第１ないし第４のいずれの処理モードによる処理を施すかを運転者等が自由に選択することができるため、運転者の好みや車両の走行状況の変化等に柔軟に対応することができる。
【００６９】
また、撮像画像に重ねて表示する予測軌跡Ｒ１，Ｒ２の表示の有無を運転者等が自由に切り替えることができるため、運転者の好みや車両の走行状況の変化等に柔軟に対応することができる。
【００７０】
＜第２実施形態＞
図２０は、本発明の第２実施形態に係る車両周辺視認装置のブロック図である。本実施形態に係る車両周辺視認装置が前述の第１実施形態に係る車両周辺視認装置と実質的に異なる点は、図２０に示すようにクリアランスセンサ２１を追加した点、及びそのクリアランスセンサ２１の追加に伴って動作内容が一部変更された点のみであり、互いに対応する部分には同一の参照符号を付して説明を省略する。
【００７１】
本実施形態において追加されたクリアランスセンサ２１は、本発明に係る障害物センサに相当するものであり、超音波センサ等により構成され、撮像装置１に対応して車両のコーナ部１１等に設けられ、撮像装置１の自車と接触の可能性のある障害物の有無、及び撮像領域Ａ内に存在する障害物と自車との間の距離を検出する。より詳細には、クリアランスセンサ２１は、撮像領域Ａ内における自車から第１の基準距離の範囲内に自車と接触する可能性のある障害物があるか否かを検出し、その検出結果を画像処理装置３に与える。
【００７２】
画像処理装置３は、表示装置７が撮像装置１の撮像画像の表示を行っていない状態において（例えば、カーナビゲーションのための画像が表示されている場合など）、クリアランスセンサ２１が前記障害物の存在を検出した場合には、それに応じて撮像装置１の撮像画像を表示装置７に表示させるようになっている。このときの撮像画像の表示モード（第１ないし第４の処理モードの処理を行うか否か、処理を行う場合にいずれの処理モードの処理を行うか、及び、予測軌跡Ｒ１，Ｒ２の重ね合わせ表示の有無）は、その時点で設定されているモードが採用されて、撮像画像の表示が行われる。なお、撮像装置１の撮像画像の表示装置７による表示が行われているときに、クリアランスセンサ２１による前記障害物の検出が行われた際には、その撮像画像の表示が継続される。
【００７３】
これによって、表示装置７によってナビゲーション画像等が表示されるなどして、撮像装置１の撮像画像の表示が行われていないときに、撮像領域Ａ内における自車から第１の基準距離の範囲内に自車と接触の可能性がある障害物が検出された場合には、それに伴って、表示装置７の表示状態が切り替えられて、撮像装置１の撮像画像が自動的に表示されるようになっている。
【００７４】
したがって、本実施形態においても、前述の第１実施形態とほぼ同様な効果が得られるとともに、表示装置７が撮像装置１の撮像画像の表示を行っていない状態においても、障害物の接近に伴って表示装置１の表示状態が自動的に切り替えられて、撮像装置１の撮像画像が表示されるため、撮像領域Ａの画像の表示を適切なタイミングで自動的に行わせることができる。
【００７５】
＜第３実施形態＞
本発明の第３実施形態に係る車両周辺視認装置のブロック構成は、前述の図２０に示す第２実施形態に係る構成と同様であるため、図示を省略する。本実施形態に係る車両周辺視認装置が前述の第１実施形態に係る車両周辺視認装置と実質的に異なる点は、図２０に示すようにクリアランスセンサ２１を追加した点、及びそのクリアランスセンサ２１の追加に伴って動作内容が一部変更された点のみであり、互いに対応する部分には同一の参照符号を付して説明を省略する。
【００７６】
本実施形態において追加されたクリアランスセンサ２１は、本発明に係る距離センサに相当するものであり、超音波センサ等により構成され、撮像装置１に対応して車両のコーナ部１１等に設けられ、撮像領域Ａ内に存在するにおける自車と接触の可能性のある障害物と自車との距離を検出し、その検出結果を画像処理装置３に与える。
【００７７】
画像処理装置３は、クリアランスセンサ２１が検出した前記障害物との距離が第２の基準距離を上回っている場合には、上述の第１ないし第４の処理モードのうちのその時点で運転者等により選択されている処理モードによる処理を撮像装置１が撮像した原画像Ｉ１に対して施して、その処理後の画像を表示装置７に表示させる。
【００７８】
一方、クリアランスセンサ２１が検出した前記障害物との距離が前記第２の基準距離を下回っている場合には、画像処理装置３による以下の動作が行われる。この場合には、第２ないし第４の処理モードのよる処理が禁止され、第１の処理モードによる処理が選択可能な状態とされる。そして、前記障害物との距離が前記第２の基準距離を下回った時点で、運転者等により第２ないし第４の処理モードが選択されている場合には、処理モードが第２ないし第４の処理モードから第１の処理モードに強制的に切り替えられ、原画像Ｉ１に対して第１の処理モードによる処理（レンズ歪み補正処理のみ）が施されて、その処理後の画像Ｉ２が表示装置７に表示される。このように処理モードの強制的な切り替えが行われている場合には、前記障害物が前記第２の基準距離の範囲外に遠ざかるのに伴って、処理モードが第１の処理モードから運転者等によって選択されている第２ないし第４の処理モードに自動的に切り替えられる。なお、処理モードとして第１の処理モードが選択されている場合には、前記障害物との距離が前記第２の基準距離を下回っても、そのまま第１の処理モードによる処理が継続される。
【００７９】
ここで、運転者等によって第１ないし第４の処理モードの処理を行わずに原画像１を表示するモードが選択されている場合には、前記障害物との距離が前記第２の基準距離を下回ったか否かにかかわらず、そのモードがそのまま保持される。また、予測軌跡Ｒ１，Ｒ２の表示の有無についての切り替えは、前記障害物との距離が前記第２の基準距離を下回ったか否かにかかわらず、運転者等の操作に基づいて行われる。
【００８０】
このように原画像Ｉ１に対する処理内容を前記障害物との距離に応じて制限するのは、次のような理由による。すなわち、撮像領域Ａ内に障害物等が存在しない場合には、実質的に平らな地面を撮像していることと等価であるため、原画像Ｉ１に対して上述の第１及び第２の視点変換処理を施しても、視点変換処理による画像の歪みの影響は小さなものとなる。しかし、撮像領域Ａ内に障害物等が存在した場合には、障害物は高さを有しているため、視点変換処理により原画像Ｉ１中の障害物等の画像に歪みが生じてしまい、視認性が低下する。そして、その視点変換処理により生じる歪みは、障害物と自車（撮像装置１）との距離が小さくなるほど大きくなる。
【００８１】
そこで、本実施形態では、障害物と自車との距離が一定の水準以下になった場合には、処理内容に視点変換処理を含む第２ないし第４の処理モードの処理が行われないように制限している。
【００８２】
以上のように、本実施形態においても、前述の第１実施形態とほぼ同様な効果が得られるとともに、一定以上の高さを有する障害物が自車の至近距離に存在しているのにもかかわらず、原画像Ｉ１に対して視点変換処理を含む第２なしい第４の処理モードによる処理が行われて画像の歪みが生じ、表示画像の視認性が低下するのを防止することができる。
【００８３】
【発明の効果】
請求項１に記載の発明によれば、カメラ光学的歪みを補正した車両周辺の画像上に、車両又は車両周辺に設けられた基準点が車両の走行に伴って通過してゆく予測軌跡を重ねて表示することができるため、車両周辺の画像をそのカメラ光学的歪みを補正して見やすくした状態で表示することができるとともに、表示画像に基づいて自車とその周辺の各地点との位置関係及び距離感や、走行に伴う自車位置の推移等を運転者等に容易に認識させることができ、表示画像に基づく車両周辺の視認性の向上が図れる。
【００８４】
請求項２に記載の発明によれば、車両周辺の画像を、その対応する撮像領域をあたかも真上から見たような画像に変換して表示するため、画像を直感的に分かりやすい状態で表示することができる。
【００８５】
請求項３に記載の発明によれば、車両周辺の画像を、その画像上における車両の直進方向が画像の縦方向と平行な関係になるように変換して表示するため、画像を直感的に分かりやすい状態で表示することができる。
【００８６】
請求項４に記載の発明によれば、車両周辺の撮像領域の画像を、その撮像領域をあたかも車窓から顔を出して視認したような画像に変換して表示するため、画像を直感的に分かりやすい状態で表示することができる。
【００８７】
請求項５に記載の発明によれば、車両の左端部又は右端部に対応する地面上の地点に対応して設けられた第１の基準点、及び左端部又は右端部から所定距離だけ離れた地面上の地点に対応して設けられた第２の基準点のうちの少なくともいずれか一方に対応する前記予測軌跡を画像に重ねて表示するため、表示画像に基づいて自車とその周辺の各地点との位置関係及び距離感や、走行に伴う自車位置の推移等を運転者等により容易に認識させることができる。
【００８８】
請求項６に記載の発明によれば、車両周辺の画像に対して行う処理の内容を運転者等が自由に切り替えることができるため、運転者の好みや車両の走行状況の変化等に柔軟に対応することができる。
【００８９】
請求項７に記載の発明によれば、画像に重ねて表示する前記予測軌跡の表示の有無を運転者等が自由に切り替えることができるため、運転者の好みや車両の走行状況の変化等に柔軟に対応することができる。
【００９０】
請求項８に記載の発明によれば、表示部が撮像カメラの画像の表示を行っていない状態において、自車から第１の基準距離の範囲内に障害物があることが障害物センサによって検出された場合には、処理部が撮像カメラの画像を表示部に表示させるようになっているため、車両周辺の画像の表示を適切なタイミングで自動的に行わせることができる。
【００９１】
請求項９に記載の発明によれば、自車と障害物との距離が第２の基準距離を下回っているか否かに応じて、車両周辺の画像に対して視点変換処理を施すか否かを切り替える構成であるため、一定以上の高さを有する障害物が自車の至近距離に存在しているのにもかかわらず、画像に対して視点変換処理が行われて画像の歪みが生じ、表示画像の視認性が低下するのを防止することができる。
【図面の簡単な説明】
【図１】本発明の第１実施形態に係る車両周辺視認装置のブロック図である。
【図２】撮像装置の設置位置及びその撮像領域を示す平面図である。
【図３】撮像装置によって撮像された原画像を示す図である。
【図４】撮像装置の撮像によって生じるカメラ光学的歪みの様子を示す図である。
【図５】歪み補正処理によってカメラ光学的歪みが補正された様子を示す図である。
【図６】図３の原画像に対して歪み補正処理を施して得られる画像を示す図である。
【図７】第１の視点変換処理の説明図である。
【図８】撮像装置の撮像光軸と車両の直進方向との関係を示す図である。
【図９】回転変換処理の説明図である。
【図１０】第２の視点変換処理の説明図である。
【図１１】図３の画像に対して第１の視点変換処理、回転変換処理及び第２の視点変換処理を施して得られる画像を示す図である。
【図１２】撮像装置の撮像領域と図１１の画像との関係を説明するための図である。
【図１３】図１１の画像に予測軌跡を重ねて表示した状態を示す図である。
【図１４】図１１の画像に予測軌跡を重ねて表示した状態を示す図である。
【図１５】図６の画像に予測軌跡を重ねて表示した状態を示す図である。
【図１６】図３の画像に予測軌跡を重ねて表示した状態を示す図である。
【図１７】停車中の他の車両の側方をすり抜ける際の様子を示す図である。
【図１８】道路の側縁部等に幅寄せして駐車する際の様子を示す図である。
【図１９】駐車場での駐車時の様子を示す図である。
【図２０】本発明の第２及び第３実施形態に係る車両周辺視認装置のブロック図である。
【符号の説明】
１ 撮像装置
３ 画像処理装置
５ ステアリングセンサ
７ 表示装置
９ 入力装置
２１ クリアランスソナー
Ａ 撮像領域
Ｉ１ 原画像
Ｉ２，Ｉ３ 画像
Ｌ 直進方向
Ｍ 縦方向
Ｒ１，Ｒ２ 予測軌跡[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a vehicle periphery recognition device that captures an image around a vehicle, subjects the image to distortion correction or the like, and displays the image on a display device in a vehicle compartment.
[0002]
[Prior art]
As this kind of conventional technology, for example, there is one described in Patent Document 1. According to this conventional technique, a correction process for correcting image distortion (camera optical distortion) caused by the influence of the imaging characteristics of an imaging camera is performed on an image around a vehicle, and a predetermined viewpoint conversion is performed. A composite image showing a situation around the vehicle is created and displayed based on the composite image.
[0003]
As an invention related to the present invention, there is one described in, for example, Patent Document 2.
[0004]
[Patent Document 1]
JP-A-2001-339716 (Claims 1 to 3, FIG. 5 and the like)
[Patent Document 2]
JP 2002-218451 A
[0005]
[Problems to be solved by the invention]
However, in the above-described conventional technology, since only a synthesized image indicating a situation around the vehicle is synthesized and displayed, the positional relationship between the host vehicle and each point in the surrounding area based on the synthesized image, a sense of distance, and traveling. It is difficult to recognize the change in the position of the own vehicle accompanying the vehicle.
[0006]
Further, in the above-described related art, since a synthesized image synthesized based on the image is displayed, the situation around the vehicle is not sufficiently reflected in the synthesized image, and the visibility around the vehicle based on the display image may be reduced. is there.
[0007]
Therefore, the present invention can display an image around the vehicle in a state in which it is easy to see by correcting the optical distortion of the camera, and based on the image, the positional relationship between the vehicle and each point around the vehicle and the sense of distance. Also, it is an object of the present invention to provide a vehicle periphery visual recognition device that allows a driver or the like to easily recognize a change in the position of the own vehicle due to traveling and to improve visibility around the vehicle based on a display image.
[0008]
[Means for Solving the Problems]
The technical means for achieving the above-mentioned object is to capture an image of an area around the left or right corner of the front end of the vehicle in a vehicle periphery visualization device that images the periphery of the vehicle and displays the image in the vehicle interior. Imaging camera, a display unit provided in the vehicle interior, a steering sensor for detecting the steering angle of the steering of the vehicle, and an image obtained by the imaging camera, due to the influence of the imaging characteristics of the imaging camera. A distortion correction process for correcting the generated distortion is performed and an image thereof is displayed on the display unit, and a predicted trajectory in which a predetermined reference point provided around the vehicle or the vehicle passes as the vehicle travels passes through the steering trajectory. A processing unit that makes a prediction based on the detection result of the sensor and causes the display unit to display the predicted trajectory in a state of being superimposed on a corresponding position on the image. Eteiru.
[0009]
Further, preferably, the processing unit further includes a first viewpoint conversion process for converting the image on which the distortion correction process has been performed into an image as if the imaging region were viewed from directly above the image. And the image subjected to the first viewpoint conversion process is displayed on the display unit with the predicted trajectory superimposed thereon.
[0010]
Further preferably, the processing unit further includes, for the image on which the distortion correction processing and the first viewpoint conversion processing have been performed, a relationship in which a straight traveling direction of the vehicle on the image is parallel to a vertical direction of the image. The rotation conversion process is performed by a predetermined rotation angle in a plane parallel to the image, and the image subjected to the rotation conversion process is displayed on the display unit with the predicted trajectory superimposed thereon. Is good.
[0011]
In addition, preferably, the processing unit further visually recognizes the imaging area as if it were a face coming out of a car window with respect to the image on which the distortion correction processing, the first viewpoint conversion processing, and the rotation conversion processing were performed. It is preferable to perform a second viewpoint conversion process for converting the image into the image as described above, and display the image subjected to the second viewpoint conversion process on the display unit with the predicted trajectory superimposed thereon. .
[0012]
Further preferably, the processing unit is a first reference point provided corresponding to a point on the ground corresponding to the left end or the right end of the vehicle, and a predetermined distance from the left end or the right end The predicted trajectory corresponding to at least one of the second reference points provided corresponding to a point on the ground may be displayed on the display unit so as to be superimposed on the image.
[0013]
Preferably, the image processing apparatus further includes a first input receiving unit that receives a first operation input for switching a mode of processing performed on the image, wherein the processing unit further includes a first input receiving unit that obtains an image obtained by the imaging camera. The mode of processing to be performed on the image is switched to any of the following first to fourth processing modes in accordance with the first operation input received by the first input receiving unit, and the processing mode corresponds to any one of the following processing modes. To the image, display on the display device, in the first processing mode, the processing unit performs the distortion correction processing on the image obtained by the imaging of the imaging camera, In a second processing mode, the processing unit performs the distortion correction processing on the image obtained by the imaging of the imaging camera, and as if the imaging area were formed on the image. Further performing a first viewpoint conversion process for converting into an image viewed from directly above, and in the third processing mode, the processing unit performs processing on the image obtained by the imaging of the imaging camera. The distortion correction processing and the first viewpoint conversion processing are performed, and the image is processed in a plane parallel to the image such that the straight traveling direction of the vehicle and the vertical direction of the image are parallel to each other on the image. In the fourth processing mode, the processing section performs the distortion correction processing and the first viewpoint conversion on the image obtained by the imaging camera. It is preferable to further perform a process and the rotation conversion process, and further perform a second viewpoint conversion process on the image so as to convert the imaging region into an image as if the user had viewed his / her head out of a vehicle window and viewed.
[0014]
Further preferably, the apparatus further includes a second input receiving unit that receives a second operation input for switching on / off of whether or not the predicted trajectory to be displayed on the image is displayed, and the processing unit further includes: It is preferable that the presence / absence of the display of the predicted trajectory to be displayed on the image is switched on / off according to the second operation input received by the second input receiving unit.
[0015]
Preferably, the apparatus further comprises an obstacle sensor for detecting whether there is an obstacle that may contact the vehicle within a range of the first reference distance from the vehicle in the imaging area. The unit may further include: in a state where the display unit is not displaying the image obtained by the imaging of the imaging camera, when the obstacle sensor detects the presence of the obstacle, Preferably, an image is displayed on the display unit.
[0016]
Furthermore, preferably, the apparatus further includes a distance sensor for detecting a distance between the own vehicle and an obstacle that may contact the own vehicle in the imaging region, and the processing unit further includes: If the distance is greater than the second reference distance, the image obtained by the imaging camera is subjected to the distortion correction processing, and the image is subjected to the imaging region as if the imaging camera were not. Further performing a viewpoint conversion process for converting into an image as viewed from a viewpoint other than the viewpoint, and superimposing the predicted trajectory thereon and displaying the predicted trajectory on the display unit, while the distance detected by the distance sensor is equal to the second distance. When the distance is smaller than the reference distance of 2, the image obtained by the imaging camera is subjected to the distortion correction processing, and the viewpoint conversion processing is not performed. Serial it is preferable to be displayed on the display unit overlapping the predicted trajectory.
[0017]
BEST MODE FOR CARRYING OUT THE INVENTION
<First embodiment>
FIG. 1 is a block diagram of the vehicle periphery recognition device according to the first embodiment of the present invention. As shown in FIG. 1, the vehicle periphery visual recognition device includes an imaging device (imaging camera) 1, an image processing device (processing unit) 3, a steering sensor 5, a display device (display unit) 7, and an input device ( A first and a second input receiving unit) 9.
[0018]
As shown in FIG. 2, the imaging device 1 is installed at the left corner 11 or the like at the front end of the vehicle, and captures an imaging area A around the vehicle including the periphery of the corner 11. In the present embodiment, the imaging device 1 is installed at the left corner 11 or the like of the vehicle and images the periphery of the corner 11. However, the imaging device 1 is mounted on the right corner 13 at the front end side of the vehicle. And the like, and the periphery of the corner 13 may be imaged. In the present embodiment, the imaging direction of the imaging apparatus 1 is set so that the imaging optical axis is directed obliquely left forward with respect to the straight traveling direction of the vehicle, and the imaging optical axis is directed obliquely downward with respect to the horizontal direction. It is set to face.
[0019]
Any configuration can be adopted as the configuration of the imaging device 1. For example, an imaging element and an imaging lens system are installed in an upward direction, a mirror is provided above the imaging lens system, and light from an imaging area A introduced through a small window provided in a case is reflected by the mirror to form an image. A configuration (mirror type) in which an image is captured by being incident on an image sensor via a lens system may be employed, or the image sensor and the image lens system may be installed toward the image area A without using a mirror. Alternatively, a configuration (direct-view type) in which light from the imaging region A introduced through the small window of the case is made incident on the imaging device via the imaging lens system to perform imaging may be adopted. In addition, the imaging device 1 may be installed in a vehicle in any form and location, for example, the imaging device 1 may be embedded in a front bumper or a headlight. In addition, it is necessary to adopt an optimal method for mounting the imaging device 1 on a vehicle after considering designability, ease of installation, durability, and the like.
[0020]
The steering sensor 5 detects a steering angle of the steering of the vehicle. The display device 7 is installed around the driver's seat in the vehicle interior, and displays an image of the imaging device 1 and the like. The input device 9 receives an operation input from a driver or the like. The input device 9 includes an operation input (first operation input) for switching the content of a process performed on the image of the imaging device 1 by the image processing device 3 and a display that is superimposed on the image and described later. An operation input (second operation input) for switching the display of the predicted trajectories R1 and R2 (see FIG. 13) on and off is input.
[0021]
The image processing device 3 performs a process corresponding to the first operation input input via the input device 9 on the image of the imaging device 1, and displays the processed imaging device on the display device 7. (Image processing function) and a function of predicting the predicted trajectories R1 and R2 and displaying the predicted trajectories R1 and R2 so as to be superimposed on an image around the vehicle (predicted trajectory display function). The function and operation of the image processing device 3 will be described in detail below.
[0022]
First, the image processing function of the image processing device 3 will be described. This image processing function includes first to fourth processing modes, which will be described later, that can be switched. In the image processing function, which of the first to fourth processing modes is applied to the original image I1 (see FIG. 3) obtained by the imaging of the imaging device 1 and displayed on the display device 7 Is switched according to an operation input through the input device 9, and whether the original image I1 is displayed by performing processing in the first to fourth processing modes or is displayed without performing the processing. 9 includes a function of switching in response to an operation input via the control unit 9.
[0023]
The processing in the first processing mode will be described in order. FIG. 3 is a diagram illustrating an original image obtained by imaging by the imaging device 1. The original image I1 shown in FIG. 3 is obtained by imaging the imaging region A shown in FIG. That is, portions indicated by reference numerals B1, B2, and B3 on the original image I1 in FIG. 3 correspond to the road C1, the greenery C2, and the sidewalk C3 in FIG. 2, respectively. The portion indicated by reference numeral B4 on the original image I1 reflects a part of the vehicle body.
[0024]
Such an original image I1 is distorted due to camera optical distortion caused by imaging using the imaging device 1. Arrows D1 to D4 in FIG. 3 indicate trajectories in which scenes corresponding to points E1 to E4 on the original image I1 move as the vehicle goes straight ahead. It can also be seen that the original image I1 is distorted.
[0025]
Therefore, in the present embodiment, as the processing in the first processing mode, the distortion of the original image I1 is corrected by performing a generally known lens distortion correction processing on the original image I1. Specifically, as shown in FIG. 4, an xy coordinate system is set on the original image I1, and the positions (coordinates) of image data (pixel data and the like) corresponding to each point of the xy coordinate system are set in a predetermined relationship. The distortion is corrected by performing the correction according to the equation.
[0026]
The coordinates (x1, y1) of the point P1 on the original image I1 before correction shown in FIG. 4 and the coordinates (x2, y2) of the point P2 on the image I2 after correction shown in FIG. When approximated by a third-order polynomial, the following relational expression generally holds.
[0027]
(Equation 1)

[0028]
Here, Cx and Cy in the above equation (1) correspond to the coordinate values of the center coordinates (Cx and Cy) of the lens distortion with respect to the origin of the xy coordinate system set on the original image I1, and k is the lens distortion coefficient. Α corresponds to the aspect ratio of the image sensor in the imaging apparatus 1, and if vertical: horizontal = 2: 1, α = 2. 4 and 5, the center coordinates (Cx, Cy) of the lens distortion coincide with the origin of the xy coordinate system.
[0029]
When the above equation (1) is solved for the corrected coordinate values x2 and y2, the following relational expression is obtained.
[0030]
(Equation 2)

[0031]
Thus, for example, the position (coordinate) of image data (pixel data or the like) of each point on the original image I1 is obtained by substituting the coordinate values x1 and y1 before the correction into the above equation (2). By correcting to (x2, y2), lens distortion is corrected. A conversion table indicating the correspondence between the coordinate point (x1, y1) of each point of the original image I1 and the corresponding coordinate point (x2, y2) after correction is previously stored in the ROM, and the conversion table is stored in the ROM. It is desirable to perform distortion correction by using this. As a result, there is an advantage that it is not necessary to sequentially calculate the coordinate values.
[0032]
FIG. 6 is a diagram showing an image obtained by performing a distortion correction process on the original image I1 of FIG. As shown in FIG. 6, it can be seen that the image I2 after the distortion correction has the camera optical distortion corrected.
[0033]
Next, processing in the second processing mode will be described. In the second processing mode, the above-described distortion correction processing is performed on the original image I1, and then the first viewpoint conversion processing is performed on the processed image I2. The first viewpoint conversion process is a process of converting the image I2 into an image as if the corresponding imaging region A was viewed from directly above.
[0034]
The specific contents of the first viewpoint conversion process will be described with reference to FIG. The xy coordinate system in FIG. 7 corresponds to the camera coordinate system corresponding to the image I2 after the lens distortion correction processing, and the XYZ coordinate system corresponds to the world coordinate system. The relationship between the xy coordinate system and the XYZ coordinate system is such that the imaging optical axis G of the imaging device 1 passes through the origin O of the XYZ coordinate system, the imaging optical axis G is parallel to the YZ plane, and the xy coordinate system The distance between the origin o and the origin O of the XYZ coordinate system is set to be equal to the focal length F of the imaging device 1.
[0035]
The virtual plane P in FIG. 7 corresponds to the ground, is parallel to the XY plane in the XYZ coordinate system, and passes through Z = −H. This H corresponds to the mounting height of the imaging device 1 from the ground. The angle θ between the imaging optical axis G and the Y axis corresponds to the angle (camera mounting angle) between the imaging optical axis G set obliquely downward of the imaging device 1 and the horizontal plane.
[0036]
In the first viewpoint conversion process, the image I2 in the camera coordinate system (xy coordinate system) is projected (mapped) on a virtual plane P set in the world coordinate system (XYZ coordinate system) so as to be symmetric with respect to the origin O. It is done by doing. In this mapping, the following relational expression holds between a point P2 (x2, y2) in the camera coordinate system and a point P3 (x3, y3, -H) in the world coordinate system due to a geometrical relationship.
[0037]
[Equation 3]

[0038]
Using the above equation (3), a coordinate point (x2, y2) in the camera coordinate system corresponding to the coordinate point (x3, y3) is obtained. Thereby, for example, the image is converted by assigning the image data (pixel data or the like) of the coordinate point (x2, y2) on the image I2 corresponding to each coordinate point (x3, y3) on the image to be composed. , The image I2 can be converted into an image as if the imaging area A were viewed from directly above. Note that each coordinate point (x3, y3) on the image obtained by performing the lens distortion correction process and the first viewpoint conversion process on the original image I1 and each coordinate point (x1, y1) on the original image I1 It is preferable that a conversion table indicating the correspondence between the two is stored in advance in the ROM, and the distortion correction and the first viewpoint conversion process be performed using the conversion table. As a result, there is an advantage that it is not necessary to sequentially calculate the coordinate values.
[0039]
Next, processing in the third processing mode will be described. As shown in FIG. 8, the imaging optical axis direction J of the imaging device 1 is not parallel to the straight traveling direction K of the vehicle, but forms a predetermined angle と with the straight traveling direction K. Therefore, when the original image I1 or the image subjected to the lens distortion correction processing or the like is displayed on the display device 7, as shown in FIG. 6, the straight traveling direction L of the vehicle on the display image is parallel to the vertical direction M of the image. Therefore, it may be difficult for the driver or the like to intuitively visually recognize the positional relationship between the scene in the displayed image and the own vehicle.
[0040]
Therefore, in the present embodiment, in the processing in the third processing mode, after performing the above-described distortion correction processing and the first viewpoint conversion processing on the original image I1, a predetermined image is processed on the processed image. A rotation conversion process is performed.
[0041]
Specifically, as shown in FIG. 9, the rotation conversion processing is performed such that each point P3 (x3, x3) on the image is such that the straight traveling direction L of the vehicle on the image is parallel to the vertical direction M of the image. This is performed by rotating the image data (pixel data and the like) of (y3) to a point P4 (x4, y4) obtained by rotating coordinate conversion by an angle で in a plane parallel to the image.
[0042]
That is, the coordinate transformation (affine transformation) from the point P3 (x3, y3) to the point P4 (x4, y4) is given by the following relational expression.
[0043]
(Equation 4)

[0044]
The inverse conversion of the conversion by the above equation (4) is
[0045]
(Equation 5)

[0046]
Is given by Using this equation (5), each point (x3, y3) of the image before the rotation process corresponding to each point (x4, y4) on the image to be composed is obtained. Thereby, for example, image data (pixel data or the like) of the coordinate point (x3, y3) of the original image before the rotation process corresponding to each coordinate point (x4, y4) on the image to be composed is assigned. Can be obtained by obtaining an image in which the straight traveling direction L of the vehicle and the vertical direction M of the image are parallel to each other on the image. Each coordinate point (x4, y4) on the image obtained by performing the lens distortion correction process, the first viewpoint conversion process, and the rotation conversion process on the original image I1 and each coordinate point (x1) on the original image I1. , Y1), it is preferable that a conversion table indicating the correspondence relationship with y1) is previously stored in the ROM, and the distortion correction, the first viewpoint conversion process, and the rotation conversion process are performed using the conversion table. As a result, there is an advantage that it is not necessary to sequentially calculate the coordinate values.
[0047]
Next, processing in the fourth processing mode will be described. In the fourth processing mode, the above-described distortion correction processing, the first viewpoint conversion processing, and the rotation conversion processing are performed on the original image I1, and the second viewpoint conversion processing is performed on the processed image. Is performed. This second viewpoint conversion processing is to convert an image into an image as if the corresponding imaging area A was visually recognized as if the face was looked down from the window of the passenger seat.
[0048]
The specific contents of the second viewpoint conversion processing will be described with reference to FIG. The XYZ coordinate system in FIG. 10 corresponds to the world coordinate system. Further, the xy coordinate system is obtained by virtually setting a viewpoint position when the viewer puts his / her face out of the window on the passenger seat and visually recognizes the imaging region A, and performs imaging of the imaging region A from the viewpoint position. It corresponds to the image coordinate system (virtual viewpoint coordinate system). The relationship between the xy coordinate system and the XYZ coordinate system is such that the imaging optical axis N of the imaging device 1 virtually set at the viewpoint position passes through the origin O of the XYZ coordinate system, and the imaging optical axis N is The focal length F is set to be parallel and the distance between the origin o of the xy coordinate system and the origin O of the XYZ coordinate system is equal to the focal length F of the imaging device 1.
[0049]
The virtual plane P ′ in FIG. 10 corresponds to the ground, is parallel to the XY plane in the XYZ coordinate system, and passes through Z = −H ′. This H ′ corresponds to the mounting height of the viewpoint from the ground. An angle φ between the imaging optical axis N and the Y axis corresponds to an angle formed with a horizontal plane in a direction in which the imaging area A is viewed from the viewpoint position.
[0050]
The second viewpoint conversion process converts the image (original image) on the virtual plane P ′ that has been subjected to the above-described distortion correction process, first viewpoint conversion process, and rotation conversion process into a virtual viewpoint coordinate system ( This is performed by projecting (mapping) on the xy coordinate system) symmetrically with respect to the origin O. In this mapping, a point P4 (x4, y4, −H ′) on the original image on the virtual plane P ′ and a point P5 (x5, y5) on the virtual viewpoint coordinate system are geometrical. From the relation, the following relational expression holds.
[0051]
(Equation 6)

[0052]
Using the above equation (6), a coordinate point (x4, y4) corresponding to the coordinate point (x5, y5) is obtained. Thereby, for example, the image is converted by allocating the image data (pixel data and the like) of the coordinate point (x4, y4) on the original image corresponding to each coordinate point (x5, y5) on the image to be composed. This makes it possible to obtain an image that can be visually recognized when the user faces his / her face out of the window on the passenger seat and looks down at the imaging area A. Note that each coordinate point (x5, y5) on the image obtained by performing lens distortion correction processing, first viewpoint conversion processing, rotation conversion processing, and second viewpoint conversion processing on the original image I1 and the original image I1 A conversion table indicating a correspondence relationship with each of the above coordinate points (x1, y1) is previously stored in the ROM, and distortion correction, first viewpoint conversion processing, rotation conversion processing, and second conversion processing are performed using the conversion table. It is desirable to perform a viewpoint conversion process. As a result, there is an advantage that it is not necessary to sequentially calculate the coordinate values.
[0053]
FIG. 11 is a diagram showing an image obtained by performing distortion correction processing, first viewpoint conversion processing, rotation conversion processing, and second viewpoint conversion processing on the original image I1 of FIG. 3 in the fourth processing mode. is there. As shown in FIG. 11, in the processed image I3, the camera optical distortion is corrected, and the straight traveling direction L of the vehicle and the vertical direction M of the image on the image I3 are in a parallel relationship. It can be seen that the image has been converted into an image that can be visually recognized when the user looks out the imaging area A with his / her face out of the window of the seat. Here, the image I3 in FIG. 11 is an image obtained by capturing, as the imaging region A, an area obtained by removing the areas A2 and A3 indicated by hatching from the substantially rectangular area A1 illustrated in FIG. Further, hatched portions B5 and B6 on the image I3 indicate portions where the image is missing due to the conversion process, and a solid image of a predetermined color (for example, black) is displayed. .
[0054]
Next, the switching operation of the above-described first to fourth processing modes and the like will be described. The driver or the like operates via the input device 9 whether the original image I1 obtained by the imaging of the imaging device 1 is to be processed in any of the first to fourth processing modes and displayed on the display device 7. Switching can be performed by inputting. That is, for example, when the first processing mode is selected, lens distortion correction processing is performed on the original image I1, the processed image I2 is displayed on the display device 7, and the fourth processing mode is selected. Is selected, the original image I1 is subjected to lens distortion correction processing, first viewpoint conversion processing, rotation conversion processing, and second viewpoint conversion processing, and the processed image I3 is displayed on the display device 7. Will be displayed.
[0055]
In addition, the driver or the like performs an operation input via the input device 9 on whether to display the original image I1 by performing the processing in the first to fourth processing modes or to display the original image I1 without performing the processing. It can be switched by changing. That is, when the mode for performing processing and displaying the original image I1 is selected, the processing in the processing mode selected at that time among the first to fourth processing modes is performed on the original image I1. Then, the processed image is displayed on the display device 7, and when the mode for displaying the original image I1 on the display device 7 is selected, the original image I1 is displayed on the display device 7 as it is.
[0056]
Next, the predicted trajectory display function of the image processing device 3 will be described. The image processing device 3 predicts predicted trajectories R1 and R2 (see FIG. 13) through which a reference point provided around the vehicle or the vehicle passes as the vehicle travels, based on the detection result of the steering sensor 5, The predicted trajectories R1 and R2 are displayed in a state where the predicted trajectories R1 and R2 are superimposed on corresponding positions on the captured image of the imaging region A displayed on the display device 7. In the present embodiment, as the reference point, a first reference point provided corresponding to a point on the ground corresponding to the left end of the vehicle (here, the end of the left corner portion 11 on the front end side of the vehicle). A point and a second reference point provided corresponding to a point on the ground separated by a predetermined distance (for example, 50 cm) in the left direction from the left end (for example, the end of the corner portion 11) are set. ing. The predicted trajectory R1 corresponds to a first reference point, and the predicted trajectory R2 corresponds to a second reference point.
[0057]
Various methods can be considered as a method for deriving the predicted trajectories R1 and R2. For example, the following method can be considered as an example. That is, each coordinate point (reference coordinate point) on the captured image I1 corresponding to the first and second reference points is registered in the memory in advance, and the steering angle of the steering at that time indicated by the detection signal of the steering sensor 5 is stored. , The trajectory of each of the reference coordinate points on the captured image I1 moving with the progress of the vehicle is calculated. Then, a drawn image of the calculated predicted trajectories R1 and R2 is created, and superimposed on the corresponding position on the captured image I1 to create a superimposed image. On the other hand, processing is performed together with the above-described first to fourth processing modes to perform display.
[0058]
Here, FIGS. 13 and 14 show examples in which predicted trajectories R1 and R2 are superimposed and displayed on an image I3 obtained by performing processing in the fourth processing mode on the original image I1, and FIG. FIG. 14 shows the states of the predicted trajectories R1 and R2 when the steering angle of the steering is 0, and FIG. 14 shows the states of the predicted trajectories R1 and R2 when the steering is turned right. FIG. 15 shows an example in which predicted trajectories R1 and R2 when the steering angle is 0 are superimposed and displayed on an image I2 obtained by performing a process in the first processing mode on the original image I1. FIG. 16 shows an example in which predicted trajectories R1 and R2 when the steering angle is 0 are superimposed on the original image I1.
[0059]
The functions of the image processing device 3 include a function of switching the display of the predicted trajectories R1 and R2 on and off. That is, when the driver or the like performs a predetermined operation input to the input device 9, the image processing device 3 responds to the predetermined operation input to determine whether or not to display the predicted trajectories R1 and R2 on the captured image. , Is switched off.
[0060]
In the example shown in FIG. 13 and the like, both of the two predicted trajectories R1 and R2 are displayed. However, only one of the two predicted trajectories R1 and R2 may be displayed. May be selected to display the predicted trajectories R1 and R2.
[0061]
In the present embodiment, two prediction trajectories R1 and R2 corresponding to the first and second reference points are displayed. However, three or more reference points are provided, and three or more prediction trajectories corresponding thereto are provided. The trajectory may be displayed. Also, the second reference point may be set at a point on the ground that is at least 50 cm (for example, 1 m) away from the end of the corner 11.
[0062]
By referring to the display image obtained in this way, when passing through the vicinity of an obstacle such as another vehicle as shown in FIG. 17 or a virtual line such as a side edge of a road as shown in FIG. For example, when the vehicle is parked in a narrow space S as shown in FIG. 19 or when the vehicle is parked in a narrow parking space as shown in FIG. 19, it is possible to accurately recognize the situation of obstacles and the like around the vehicle.
[0063]
As described above, according to the present embodiment, on the image obtained by performing the processing in any of the above-described first to fourth processing modes on the original image I1, the reference point provided around the vehicle or its periphery is obtained. Since the predicted trajectories R1 and R2 that pass along with the traveling of the vehicle can be displayed in an overlapping manner, the image around the vehicle can be displayed in a state where the optical distortion of the camera is corrected and the camera is easily viewed. Based on the display image, the driver can easily recognize the positional relationship and the sense of distance between the own vehicle and each point in the vicinity thereof, the change in the position of the own vehicle due to traveling, etc., and the vehicle periphery based on the display image Can be improved.
[0064]
Furthermore, in the present embodiment, a first reference point provided corresponding to a point on the ground corresponding to the left end of the vehicle, and a first reference point provided corresponding to a point on the ground separated by a predetermined distance from the left end. In order to display the predicted trajectories R1 and R2 corresponding to the second reference point superimposed on the image, based on the displayed image, the positional relationship and the sense of distance between the vehicle and each of its surrounding points, The driver can easily recognize the change of the position and the like.
[0065]
In the processing in the above-described second or third processing mode, the original image I1 is converted into an image as if the corresponding imaging area A is viewed from directly above and displayed, so that the image is intuitively displayed. It can be displayed in an easy-to-understand state.
[0066]
Further, in the processing in the third or fourth processing mode described above, the original image I1 is converted and displayed so that the straight traveling direction of the vehicle on the image I1 is parallel to the vertical direction of the image. Images can be displayed in an intuitive and easy-to-understand manner.
[0067]
Further, in the processing in the above-described fourth processing mode, the original image I1 of the imaging region A is converted into an image I3 in which the imaging region A is visually recognized as if the face is viewed from the front passenger's seat window. Therefore, the image I3 can be displayed in an intuitive and easy-to-understand state.
[0068]
Furthermore, the driver or the like determines whether or not to perform the processing in the first to fourth processing modes on the original image I1 and whether to perform the processing in the first to fourth processing modes when performing the processing. Since the selection can be made freely, it is possible to flexibly cope with a driver's preference, a change in the running condition of the vehicle, and the like.
[0069]
In addition, since the driver or the like can freely switch whether or not to display the predicted trajectories R1 and R2 to be displayed so as to be superimposed on the captured image, it is possible to flexibly respond to a driver's preference, a change in the running condition of the vehicle, and the like. it can.
[0070]
<Second embodiment>
FIG. 20 is a block diagram of the vehicle periphery recognition device according to the second embodiment of the present invention. The vehicle peripheral visual recognition device according to the present embodiment is substantially different from the vehicle peripheral visual recognition device according to the above-described first embodiment in that a clearance sensor 21 is added as shown in FIG. Only the operation contents are partially changed with the addition, and the corresponding parts are denoted by the same reference numerals and description thereof is omitted.
[0071]
The clearance sensor 21 added in the present embodiment corresponds to the obstacle sensor according to the present invention, and is configured by an ultrasonic sensor or the like, and is provided in the corner portion 11 or the like of the vehicle corresponding to the imaging device 1. Then, the presence or absence of an obstacle that may come into contact with the own vehicle of the imaging device 1 and the distance between the own vehicle and the obstacle existing in the imaging area A are detected. More specifically, the clearance sensor 21 detects whether there is an obstacle that may come into contact with the own vehicle within a range of the first reference distance from the own vehicle in the imaging area A, and the detection result Is given to the image processing apparatus 3.
[0072]
When the display device 7 is not displaying the captured image of the imaging device 1 (for example, when an image for car navigation is displayed), the image processing device 3 determines that the clearance sensor 21 When the presence is detected, the captured image of the imaging device 1 is displayed on the display device 7 accordingly. At this time, the display mode of the captured image (whether or not to perform the processing in the first to fourth processing modes, which processing mode to perform when performing the processing, and the superposition of the predicted trajectories R1 and R2) Regarding the presence / absence of display, the mode set at that time is adopted, and the captured image is displayed. When the obstacle is detected by the clearance sensor 21 while the captured image of the imaging device 1 is being displayed by the display device 7, the display of the captured image is continued.
[0073]
As a result, when the captured image of the imaging device 1 is not displayed, such as when a navigation image is displayed on the display device 7 or the like, when the captured image of the imaging device A is within the range of the first reference distance from the own vehicle. When an obstacle that may come into contact with the own vehicle is detected, the display state of the display device 7 is switched accordingly, and the captured image of the imaging device 1 is automatically displayed. Has become.
[0074]
Therefore, also in the present embodiment, substantially the same effects as those of the above-described first embodiment can be obtained, and even in a state where the display device 7 is not displaying the image captured by the imaging device 1, it is possible to obtain the same effect as the approach of an obstacle. Accordingly, the display state of the display device 1 is automatically switched to display the captured image of the imaging device 1, so that the image of the imaging region A can be automatically displayed at an appropriate timing.
[0075]
<Third embodiment>
The block configuration of the vehicle periphery recognition device according to the third embodiment of the present invention is the same as the configuration according to the above-described second embodiment shown in FIG. The vehicle peripheral visual recognition device according to the present embodiment is substantially different from the vehicle peripheral visual recognition device according to the above-described first embodiment in that a clearance sensor 21 is added as shown in FIG. Only the operation contents are partially changed with the addition, and the corresponding parts are denoted by the same reference numerals and description thereof is omitted.
[0076]
The clearance sensor 21 added in the present embodiment corresponds to the distance sensor according to the present invention, is configured by an ultrasonic sensor or the like, and is provided in the corner portion 11 or the like of the vehicle corresponding to the imaging device 1, The distance between the own vehicle and an obstacle that may be in contact with the own vehicle in the imaging area A is detected, and the detection result is provided to the image processing device 3.
[0077]
When the distance to the obstacle detected by the clearance sensor 21 exceeds the second reference distance, the image processing device 3 determines whether the driver is at that time in the first to fourth processing modes. The processing according to the processing mode selected as described above is performed on the original image I1 captured by the imaging device 1, and the processed image is displayed on the display device 7.
[0078]
On the other hand, when the distance to the obstacle detected by the clearance sensor 21 is smaller than the second reference distance, the following operation by the image processing device 3 is performed. In this case, the processing in the second to fourth processing modes is prohibited, and the processing in the first processing mode can be selected. When the driver or the like selects the second to fourth processing modes when the distance to the obstacle falls below the second reference distance, the processing mode is changed to the second to fourth processing modes. Is forcedly switched from the processing mode to the first processing mode, the original image I1 is subjected to the processing in the first processing mode (only the lens distortion correction processing), and the processed image I2 is displayed on the display device. 7 is displayed. In the case where the processing mode is forcibly switched, the processing mode is changed from the first processing mode to the driver as the obstacle moves away from the range of the second reference distance. The mode is automatically switched to the second to fourth processing modes selected by the above. When the first processing mode is selected as the processing mode, the processing in the first processing mode is continued even if the distance to the obstacle is smaller than the second reference distance.
[0079]
Here, when the mode for displaying the original image 1 without performing the processing of the first to fourth processing modes is selected by the driver or the like, the distance to the obstacle is equal to the second reference distance. Irrespective of whether or not it falls below, the mode is kept as it is. The switching of the display of the predicted trajectories R1 and R2 is performed based on the operation of the driver or the like regardless of whether the distance to the obstacle is smaller than the second reference distance.
[0080]
The reason for restricting the processing content of the original image I1 in accordance with the distance from the obstacle is as follows. That is, when there is no obstacle or the like in the imaging region A, it is equivalent to imaging a substantially flat ground, and therefore, the first and second viewpoints described above with respect to the original image I1. Even if the conversion processing is performed, the influence of the image distortion due to the viewpoint conversion processing is small. However, when an obstacle or the like exists in the imaging region A, the obstacle has a height, and the viewpoint conversion processing causes distortion of an image of the obstacle or the like in the original image I1. Visibility decreases. The distortion generated by the viewpoint conversion process increases as the distance between the obstacle and the vehicle (the imaging device 1) decreases.
[0081]
Therefore, in the present embodiment, when the distance between the obstacle and the host vehicle becomes equal to or less than a certain level, the processing in the second to fourth processing modes including the viewpoint conversion processing in the processing content is not performed. Limited to.
[0082]
As described above, also in this embodiment, substantially the same effects as those of the first embodiment can be obtained, and even if an obstacle having a certain height or more exists at a close distance of the own vehicle. Regardless, it is possible to prevent the original image I1 from being subjected to the processing in the second to fourth processing modes including the viewpoint conversion processing, causing image distortion, and reducing the visibility of the displayed image. .
[0083]
【The invention's effect】
According to the first aspect of the present invention, a predicted trajectory in which a vehicle or a reference point provided around the vehicle passes as the vehicle travels is superimposed on the image around the vehicle corrected for the optical distortion of the camera. The image around the vehicle can be displayed in an easy-to-view state by correcting the camera optical distortion, and the positional relationship between the vehicle and each point around the vehicle can be displayed based on the displayed image. In addition, the driver and the like can easily recognize the sense of distance, the change in the position of the vehicle accompanying the travel, and the like, and the visibility around the vehicle based on the display image can be improved.
[0084]
According to the second aspect of the present invention, the image around the vehicle is converted into an image as if the corresponding imaging area is viewed from directly above, and the image is displayed in an intuitive and easy-to-understand manner. can do.
[0085]
According to the third aspect of the present invention, since the image around the vehicle is converted and displayed so that the straight traveling direction of the vehicle on the image is parallel to the vertical direction of the image, the image is intuitively displayed. It can be displayed in an easy-to-understand state.
[0086]
According to the fourth aspect of the present invention, the image of the imaging region around the vehicle is displayed by converting the imaging region into an image as if the user were visually recognizing the person with his / her face out of the vehicle window. It can be displayed in an easy state.
[0087]
According to the invention described in claim 5, the first reference point provided corresponding to the point on the ground corresponding to the left end or the right end of the vehicle, and a predetermined distance from the left end or the right end In order to display the predicted trajectory corresponding to at least one of the second reference points provided corresponding to the point on the ground so as to be superimposed on an image, the own vehicle and various places around it are displayed based on the display image. The driver and the like can easily recognize the positional relationship with the point and the sense of distance, the change in the position of the own vehicle due to traveling, and the like.
[0088]
According to the invention described in claim 6, since the content of the processing performed on the image around the vehicle can be freely switched by the driver or the like, it is possible to flexibly respond to the driver's preference, a change in the running condition of the vehicle, and the like. Can respond.
[0089]
According to the invention as set forth in claim 7, the driver or the like can freely switch the presence or absence of the display of the predicted trajectory to be superimposed and displayed on an image. It can respond flexibly.
[0090]
According to the invention described in claim 8, when the display unit is not displaying the image of the imaging camera, the obstacle sensor detects that there is an obstacle within the range of the first reference distance from the own vehicle. In this case, the processing unit causes the image of the imaging camera to be displayed on the display unit, so that the image around the vehicle can be automatically displayed at an appropriate timing.
[0091]
According to the ninth aspect of the present invention, whether or not to perform the viewpoint conversion process on the image around the vehicle according to whether or not the distance between the own vehicle and the obstacle is smaller than the second reference distance Since the configuration is such that an obstacle having a height equal to or higher than a certain level exists at a close distance of the own vehicle, the image is subjected to the viewpoint conversion process and the image is distorted. It is possible to prevent the visibility of the displayed image from being reduced.
[Brief description of the drawings]
FIG. 1 is a block diagram of a vehicle periphery recognition device according to a first embodiment of the present invention.
FIG. 2 is a plan view illustrating an installation position of an imaging device and an imaging region thereof.
FIG. 3 is a diagram illustrating an original image captured by an imaging device.
FIG. 4 is a diagram illustrating a state of camera optical distortion caused by imaging by an imaging device.
FIG. 5 is a diagram illustrating a state in which camera optical distortion is corrected by the distortion correction processing.
6 is a diagram showing an image obtained by performing a distortion correction process on the original image of FIG. 3;
FIG. 7 is an explanatory diagram of a first viewpoint conversion process.
FIG. 8 is a diagram illustrating a relationship between an imaging optical axis of an imaging device and a straight traveling direction of a vehicle.
FIG. 9 is an explanatory diagram of a rotation conversion process.
FIG. 10 is an explanatory diagram of a second viewpoint conversion process.
11 is a diagram illustrating an image obtained by performing a first viewpoint conversion process, a rotation conversion process, and a second viewpoint conversion process on the image of FIG. 3;
FIG. 12 is a diagram for explaining a relationship between an imaging region of the imaging device and the image of FIG. 11;
FIG. 13 is a diagram showing a state in which a predicted trajectory is superimposed on the image of FIG. 11 and displayed.
14 is a diagram showing a state in which a predicted trajectory is superimposed on the image of FIG. 11 and displayed.
FIG. 15 is a diagram showing a state in which a predicted trajectory is superimposed on the image of FIG. 6 and displayed.
FIG. 16 is a diagram showing a state in which a predicted trajectory is superimposed on the image of FIG. 3 and displayed.
FIG. 17 is a diagram showing a state in which the vehicle passes a side of another stopped vehicle.
FIG. 18 is a diagram illustrating a state in which the vehicle is parked with a width approaching a side edge of a road or the like.
FIG. 19 is a diagram showing a state at the time of parking in a parking lot.
FIG. 20 is a block diagram of a vehicle periphery recognition device according to second and third embodiments of the present invention.
[Explanation of symbols]
1 Imaging device
3 Image processing device
5 Steering sensor
7 Display device
9 Input device
21 Clearance Sonar
A Imaging area
I1 Original image
I2, I3 image
L Straight direction
M Vertical direction
R1, R2 predicted trajectory

Claims (9)

In a vehicle periphery recognition device that images the periphery of the vehicle and displays the image in the vehicle interior,
An imaging camera that captures an imaging area including the periphery of the left or right corner on the vehicle front end side,
A display unit provided in the vehicle interior;
A steering sensor for detecting a steering angle of a vehicle steering;
The image obtained by the imaging camera is subjected to a distortion correction process for correcting distortion caused by the influence of the imaging characteristics of the imaging camera, and the image is displayed on the display unit, and is provided around the vehicle or the vehicle. A predicted trajectory through which the predetermined reference point passes as the vehicle travels is predicted based on the detection result of the steering sensor, and the predicted trajectory is displayed in a state of being superimposed on a corresponding position on the image. A processing unit to be displayed on the unit;
A vehicle periphery visual recognition device comprising: The vehicle periphery recognition device according to claim 1,
The processing unit further includes:
The image subjected to the distortion correction processing is subjected to a first viewpoint conversion processing for converting the imaging region into an image as if viewed from directly above, and the first viewpoint conversion processing is performed. A vehicle periphery visual recognition device, wherein the image is displayed on the display unit with the predicted trajectory superimposed thereon. The vehicle periphery visual recognition device according to claim 2,
The processing unit further includes:
With respect to the image on which the distortion correction processing and the first viewpoint conversion processing have been performed, an in-plane parallel to the image is provided such that a straight traveling direction of the vehicle on the image is parallel to a vertical direction of the image. A vehicle periphery visual recognition device that performs a rotation conversion process by a predetermined rotation angle and displays the image subjected to the rotation conversion process on the display unit with the predicted trajectory superimposed thereon. The vehicle periphery recognition device according to claim 3,
The processing unit further includes:
A second image processing unit that converts the image that has been subjected to the distortion correction processing, the first viewpoint conversion processing, and the rotation conversion processing into an image as if the imaged area was viewed from a car window as if the face were viewed. A vehicle periphery visual recognition device that performs a viewpoint conversion process and displays the image on which the second viewpoint conversion process has been performed on the display unit with the predicted trajectory superimposed thereon. The vehicle periphery visual recognition device according to any one of claims 1 to 4,
The processing unit includes:
A first reference point provided corresponding to a point on the ground corresponding to the left end or the right end of the vehicle, and a first reference point provided corresponding to a point on the ground separated by a predetermined distance from the left end or the right end A vehicle periphery recognition device that causes the display unit to display the predicted trajectory corresponding to at least one of the second reference points on the image. The vehicle periphery recognition device according to claim 1,
A first input receiving unit that receives a first operation input for switching a mode of processing performed on the image,
The processing unit further includes:
The mode of processing performed on the image obtained by the imaging of the imaging camera is set to one of the following first to fourth processing modes according to the first operation input received by the first input receiving unit. Switching, performing a process corresponding to any one of the processing modes on the image, and displaying the image on the display device,
In the first processing mode, the processing unit performs the distortion correction processing on the image obtained by the imaging of the imaging camera,
In the second processing mode, the processing unit performs the distortion correction processing on the image obtained by the imaging of the imaging camera, and views the imaging area of the image as if viewed from directly above the image. Further performing a first viewpoint conversion process for converting into such an image,
In the third processing mode, the processing unit performs the distortion correction processing and the first viewpoint conversion processing on the image obtained by the imaging of the imaging camera, and performs an image processing on the image. Further performing a rotation conversion process by a predetermined rotation angle in a plane parallel to the image, such that the straight traveling direction of the vehicle and the vertical direction of the image in have a parallel relationship,
In the fourth processing mode, the processing unit performs the distortion correction processing, the first viewpoint conversion processing, and the rotation conversion processing on the image obtained by the imaging of the imaging camera, and performs processing on the image. On the other hand, a vehicle periphery visual recognition device that further performs a second viewpoint conversion process for converting the image capturing area into an image as if the user had viewed his / her head out of a vehicle window and viewed. The vehicle periphery recognition device according to claim 1,
A second input receiving unit that receives a second operation input for switching on / off of whether or not the predicted trajectory is displayed to be superimposed on the image,
The processing unit further includes:
A vehicle periphery visual recognition device that switches on / off of whether or not to display the predicted trajectory to be superimposed on the image according to the second operation input received by the second input reception unit. The vehicle periphery recognition device according to claim 1,
An obstacle sensor for detecting whether or not there is an obstacle that may come into contact with the own vehicle within a range of the first reference distance from the own vehicle in the imaging region;
The processing unit further includes:
When the obstacle sensor detects the presence of the obstacle while the display unit is not displaying the image obtained by the imaging camera, the image is displayed on the display unit accordingly. The vehicle periphery recognition device to be displayed. The vehicle periphery recognition device according to claim 1,
Further comprising a distance sensor that detects the distance between the vehicle and an obstacle that may be in contact with the vehicle in the imaging region,
The processing unit further includes:
If the distance detected by the distance sensor is greater than a second reference distance, the distortion correction processing is performed on the image obtained by the imaging of the imaging camera, and the image Further performing a viewpoint conversion process for converting the region into an image as if viewed from a viewpoint other than the viewpoint of the imaging camera, and superimposing the predicted trajectory thereon and displaying the predicted trajectory on the display unit, while the distance sensor is When the detected distance is less than the second reference distance, the distortion correction processing is performed on the image obtained by the imaging of the imaging camera, and the viewpoint conversion processing is not performed. A vehicle periphery visual recognition device that superimposes the predicted trajectory on the display and displays the predicted trajectory on the display unit.

Images