JP3669205B2 - Obstacle recognition device - Google Patents

Description

【００５５】
なお、上記（数２）式において、ｘは基準画像（画像Ａ）の輝度、ｘはｘの平均値、ｙは他方の画像（画像Ｂ）の輝度、ｙはｙの平均値である。
【００５６】
二つのカメラを撮像面の同一ラインが同一直線上に並ぶように設置した場合、互いの画像上でのマッチング位置は同一ライン上にのる。このことから、類似画像の探索は図９に示したように同じライン上において視差方向に１画素ずつずらしながら行なうことで求められる。そしてそのウインドウの視差は、この操作において最も差分値が低いと判断された画像が存在する位置と基準画像のウインドウを定義した位置との差として求められる。また、右画像を基準画像とした場合、右画像上に定義したウインドウとのマッチング位置は、図９（Ｄ）に示すように左画像上では右画像でウインドウを定義した位置よりも右側になる。つまり、画像の探索範囲は、右画像でウインドウを定義した位置から右方向に走査すればよく、視差は走査の始点からマッチング位置までのずれとなる。図９の例では、図９（Ａ）の（ｃ）のグラフに示したようになる。
【００５７】
上記のように、図９（Ａ）は四つのウインドウそれぞれにおいて各画素毎に求めた差分値（基準画像と差分を求めた画像との差）と視差（基準画像と差分を求めた画像との位置の差）との関係をグラフ化したものであり、差分マッチングの場合、差分値が最小の位置が類似度の最も高い位置となる。しかし、例えば図９（Ａ）の（ａ）のように、ウインドウもウインドウの探索範囲も共に無地の画像であると、どの位置においても差分が同じ値となるため視差を決めることができない。実画像では、このようなウインドウでも小さなノイズがあるために、それぞれの位置で異なる値の差分が求められ、最小値の位置も求められてしまうが、このような場合、この位置は物体の視差を検出したものでないため、この視差を物体検知に利用すると誤検知の原因となる。また、図９（Ａ）の（ｂ）のように視差と平行なエッジしかないウインドウでは、物体の一部と考えられるエッジが撮像されているが、（ａ）と同様に、差分値は走査範囲内で全て同じ値なので、求められた視差は正確なものでない可能性が高い。すなわち上記（ａ）、（ｂ）のような場合に求めた視差は誤計測の原因になる可能性が高いため、視差は求めるべきではないと言える。
【００５８】
以下、上記のような誤検知の要因を取り除く方法を説明をする。
図９（Ｃ）は、各ウインドウにおいて視差方向に切った横１ライン上の輝度分散を複数箇所で求めた状態を示す図である。視差と平行に切られた横１ラインのウインドウの輝度分散は、（ａ）のような無地のウインドウにおいても、（ｂ）のような視差と平行な方向にだけエッジを持つウインドウにおいても、共に小さな値となる。逆に、（ｃ）のように視差の算出が可能なウインドウでは分散は大きくなる。
【００５９】
また、ウインドウ内の全体の輝度分散を計算すると、（ａ）の無地のウインドウでは分散が小さくなるが、（ｂ）のウインドウの分散は大きくなってしまう。しかし、図９（Ｃ）に示したように横１ラインの分散を算出すれば、（ａ）（ｂ）は共に分散が小さくなるので、視差算出が不適なウインドウを正確に検出することができる。このように複数箇所の１ライン上の分散の合計または平均値に基づいて視差を求めることが適当であるか否かを、マッチングを行なう前に判断すれば、誤検知のウインドウを取り除くことができ、同時に計算量を削減することができる。
【００６０】
また、視差の算出は、図９（Ａ）に示した差分値のグラフから最小値の位置を探すことで行なうため、最小値に近い値が複数箇所で見られるものは、無地のウインドウや視差方向と平行なエッジしか持たない画像と同様に、求めた視差の確実性は低いものとなる。例えば、図９（Ａ）の（ｄ）のようにノイズのある画像では、前述の図９（Ｃ）の方法で求めた分散は大きくなるが、差分値のグラフは、最小値に非常に近い値が複数箇所で求められるため、その中から視差を断定することは困難である。仮に、求められた差分値最小位置を視差としても、この位置は誤対応位置である可能性が高い。したがってこのような場合には視差を求めない方がよい。そのため次のような方法を用いる。
【００６１】
図９（Ａ）に示すように、走査範囲内で求めた差分値の平均値とその中の最小差分値を求めると、全体の差分値のうち、最小値に近い値が多く現れる場合にはマッチング範囲内で求めた差分の平均値と差分の最小値との差が小さくなる、つまり、この二つの値の差が或るしきい値より小さい場合には、そのウインドウで求めた視差は誤対応の高いものであると判断することができる。したがって、上記の判断において誤対応の可能性の高いと判断した視差は、この後の処理である表への投票に用いないようにすれば、物体の検知をより確実にすることができる。なお、上記図９（Ａ）のグラフは縦軸に差分をとっているが、差分の逆数の類似度をとった場合には、類似度の最大値と類似度の平均値との差が所定のしきい値より小さい場合に誤対応と判断する。
【００６２】
上記のように、マッチング領域内画像全体が黒の画像、画像全体が白の画像、或いはエッジや画像内に撮像されるテクスチャ（模様や形状など）に特徴のない領域で求められた視差は、誤計測である可能性が非常に高い。車両のボディ面では確実性の高い距離が算出されない原因はこのような原理に基づく。
【００６３】
また、ボディ面のような画像中のある一部分の領域だけでなく、霧や街灯のない夜間などの環境によっては、ステレオ画像処理による距離画像自体が正確に求められない場合もある。そのため、このような計測値を表へ投票することは何も存在しない位置への投票が行われるなどの誤計測につながる。したがって、各ウインドウにおいて視差方向に切った横１ライン上の輝度分散が各ラインにおいて小さい場合、およびマッチング範囲内で求めた差分の平均値と差分の最小値との差（類似度の最大値と類似度の平均値との差）が小さい場合には、誤対応の可能性が高いので、その位置で求めた計測値は表への投票に反映させない構成とするのが望ましい。具体的には、上記のような範囲の値は全く用いないか（重み＝０）、或いは重み付けの値を大幅に小さな値とする。
【００６４】
上記の画像上の輝度分散の考慮は誤計測の除去につながるため、ステレオ画像処理単体の場合でも誤計測削減のための有効な方法となる。しかし、ステレオ画像処理のみを用いる場合は、上記のようにその計測値を用いない場合は計測不可能な状況となるが、ステレオ画像処理とレーザレーダとを組み合わせた装置では、上記のような環境下では、レーザレーダなどの他の計測値の結果だけを反映させることで、確実な計測を行なうことができる。したがって、計測が不可能になることもなく、誤計測だけを削減することができる。
【００６５】
また、上記の説明は、ステレオ画像処理の計測値を用いないでレーザレーダの計測値のみを用いる場合について説明したが、その逆もあり得る。すなわち、レーザレーダなどの光の反射時間を利用した測距装置では、計測対象物が光を吸収するような物体の場合には、反射光がなくなるため計測が困難になる。通常、道路上などの外界には、光を１００％吸収するものはほとんどないため、多少の反射光は計測されるが、反射強度の低い計測値は、信頼性が低く、距離を誤計測する可能性が高い。そこで、スキャニング角度毎に計測距離と同時に強度を計測し、その光の強度が所定値よりも低い場合は、その位置での計測値を表への投票に反映させないようにする。具体的には、上記のような範囲の値は全く用いないか（重み＝０）、或いは重み付けの値を大幅に小さな値とする。この場合にはステレオ画像処理の計測値のみがほぼ使用されることになる。これによりレーザレーダによる誤計測を防ぐことができる。
【００６６】
次に、これまでの説明は、ステレオ画像処理とレーザレーダとを用いた装置について説明したが、それらの何れか一方の代わり、もしくはそれらに追加してミリ波レーダを用いる場合について説明する。
ミリ波レーダは、濃霧、雨中や暗闇など、ステレオ画像処理やレーザレーダでは測定不可能な場合でも周囲の物体までの測距が可能という特徴がある。しかし、一般に、ミリ波レーダは、レーザレーダのような細かい走査は行わないため、スキャニングレーザレーダや走査線のスキャンによりレーザレーダよりも細かい分解能の得られるステレオ画像処理を用いた計測と比べて、横方向の検知角度分解能が粗いという問題がある。そこで、他の装置では計測不可能な悪天候下では、ミリ波レーダでの計測結果に基づいた投票の重みを大きくする。一方、ステレオ画像処理やレーザレーダによる高精度な測定が可能な環境下では、ミリ波レーダの計測値に基づいた投票値の重みは小さくする構成とする。組み合わせとしては、ステレオ画像処理とミリ波レーダ、レーザレーダとミリ波レーダ、ステレオ画像処理とレーザレーダとミリ波レーダの３種が考えられる。上記の構成により、悪天候に強く、かつ、好天候下では、分解能の粗い測定に影響されることなく高精度な測定が可能となる。
【００６７】
次に、これまで説明した種々の方法を用いて前方の物体の位置計測および物体の認識を行なう実施例を説明する。
図１０は、２台のカメラ１、２と、レーザレンジファインダ７を自車両に搭載した構成を示す側面図である。ここでは、２台のカメラ１、２は路面に対して縦に平行に並べ、２台のカメラの撮像面のｙ軸が同一ライン上にのるように配置している。また、レーザレンジファインダ７は、カメラの光軸とレーザレンジファインダの走査中心軸を平行とし、カメラの撮像面〔図４（Ａ）の撮像面〕とレーザレンジファインダの計測面〔図７（Ａ）の計測面〕が平行になるように搭載した場合を示す。なお、図１０においては、カメラを１個のみ示しているが、実際には２台（この例では縦方向に）存在する。例えば、２台のカメラの中間にレーザレンジファインダを設置するように構成すれば、両者の画像の位置のずれを少なく出来る。
【００６８】
図１１は、この実施例の処理の流れを示す図である。
図１１において、まず、ステップＳ１０１では、２台のカメラ１、２からの画像Ａと画像Ｂを入力する。同時に、ステップＳ１０２では、レーザレンジファインダ７から距離画像とその位置の反射強度である輝度画像を入力する。
【００６９】
次に、ステップＳ１０３では、ステレオ画像を用いて距離画像を作成する（前記図３の説明参照）。この距離画像作成では、領域毎の視差を求める前に、まず画像上に定義した領域毎の輝度分散を求め、その分散がしきい値以上の部分だけ視差を算出するようにする（前記図９の説明参照）。このような前処理を行なうことで、信頼性の高いステレオ画像処理での計測値だけが後の投票に使用されるようにする。また、分散がしきい値以上の領域ではステレオ画像間のマッチング位置（視差）を求め、各領域毎の距離を求める。これには、画像間の正規化相関法や差分法などの一般的な方法を用いればよい。
【００７０】
次に、ステップＳ１０４では、ステレオ画像処理で求めた距離画像とレーザレンジファインダで計測した距離画像とのそれぞれを用いて、距離×水平方向の表への投票を行なう（図５、図８の説明参照）。
ここでは、まず、ステレオ画像処理で求めた距離画像を用いた投票を行なう。これは、図５で説明したように、距離画像の各領域に着目し、表の横の位置がその領域のｘ軸方向の角度、縦の位置はその領域で算出された距離ｚとなる位置に投票値を加算するという処理を繰り返す。例えば、水平方向が３の位置に定義された領域において距離がＬ２と求められた領域に検出値があった場合は、表中の水平方向が３、距離ｚがＬ２の位置に投票する。ここで表に加算する投票値は、後述する環境に応じた重みを付けてもよいし、＋１としてもよい。このような操作をステレオ画像処理の距離画像作成において距離が算出されている領域すべてにおいて行なう。
【００７１】
また、レーザレンジファインダで計測した距離画像を用いて同じ操作を行なう。ただし、通常、ステレオ画像処理で求めたｘ軸方向の一つの領域の角度（図４、図５のα）とレーザレンジファインダの１回の計測毎に動く走査角度（図６、図７のθ）は異なるので、表への投票の際には、実空間上において同じ方向を計測した値が表中の同じ位置に投票されるように変換する必要がある。それには次のような計算を施せばよい。すなわち、前記図４に示したように、カメラからの距離がｚ、カメラの光軸からｘ軸方向への距離がｘｐの位置にある点Ｐが撮像される画像上の領域のｘ方向の位置（これをｎ番目の領域とする）を求める。画像上で一つの領域に撮像される範囲角をα、光軸が撮像される領域を０番目としたときに、点Ｐが撮像される位置を光軸からｎ番目の領域とすると、ｎは、下記（数３）式で求められる。
ｎ＝ｘｐ／（α×ｚ） …（数３）
ただし、αは微小角なので、ｔａｎα≒αとする。
【００７２】
また、レーザレンジファインダで計測した輝度画像上において点Ｐまでの距離および反射強度を計測した画像上の位置とステレオカメラで撮像した画像上に切った領域の位置との対応を求める。前記図７に示したように、レーザレンジファインダの水平方向の１回のサンプリング毎に動く走査角度をθ、レーザレンジファインダの中心軸の計測値を０番目とする。レーザレンジファインダの距離画像において点Ｐを計測した位置を中心軸からｍ番目とすると、ｎとｍとの間には下記（数４）式の関係が成り立つ。ただし、θは微小角なので、ｔａｎθ≒θとする。
【００７３】
ｎ＝ｍ・（θ／α） …（数４）
上記（数４）式は、レーザレンジファインダにおいて、中心軸から水平方向にｎ番目の位置で計測した点は、ステレオ画像処理ではｍ番目の領域で計測されることを表す。レーザレンジファインダで計測した距離画像を用いた投票においては、この（数４）式による水平方向位置の変換を施した上で表への投票を加算する。加算する値は、その位置の反射強度に比例した値を加算する。なお、この際、ステレオ画像処理とレーザレンジファインダの両方が計測可能な環境下では、レーザレンジファインダとステレオ画像処理との投票値の割合が１対１となる値でよい。すなわち、この場合においては、両者の重みは共に１である。
【００７４】
ここで、投票値する値の定義方法の例を説明する。例えば、ステレオ画像の距離画像の一つの領域に撮像される縦方向のなす角（図４のβ）が１度、レーザレンジファインダの一つ分のデータを得るときの縦方向の走査角度（図７のφ）がその２倍にあたる２度のときに、両者の投票の重みを同じにすることを考える。このとき、縦方向の画角１０度の範囲に撮像される物体が前方に存在すると、ステレオ画像処理では表中の同じ位置に最大１０回、レーザレンジファインダでは最大５回の投票が行われる。つまり、ステレオ画像処理の結果の投票値を一つの距離につき「１」、レーザレンジファインダの計測結果の投票値を一つの距離につき「２」とすれば、最大の投票値が両者で同じ値となる。このように、同じ画角内で計測される物体を対象としたときに投票される最大の値が同じになるような投票値にすれば、両者の投票値の割合が１対１となる。
【００７５】
次に、ステップＳ１０５では、ステレオ画像処理とレーザレンジファインダで計測した距離画像全体の投票結果から、物体の有無とその物体の位置を求める。物体の有無は、表中からしきい値以上の投票値となる位置の有無により判断できる。また、その物体の位置は、（数３）式の逆算により求められる。例えば、表中において距離ｚｐ、水平方向ｎの位置にしきい値以上の値が投票されたとする。この位置に検出された物体の距離は表からｚｐと求められる。また、その物体の、カメラの光軸に対する水平方向の位置ｘｐ（図４参照）は、（数３）式の逆算により、ｘｐ＝ｎ×α×ｚｐ（α：表に定義した一つの領域に撮像される水平方向の画角）として求めることができる。表中にしきい値以上の点が複数個所存在すれば前方に複数の物体があることがわかり、それぞれの位置は前述と同様の方法で求められる。また、しきい値以上の値が同じ距離上にかたまりとして存在する場合は、それは一つの物体として判断することができ、そのかたまりの両端が物体端となる。
【００７６】
次に、ステップＳ１０４の投票における重み付けについて説明する。なお、図１１では、重み付けのステップは記載していないが、ステップＳ１０３とＳ１０２の次に、それぞれの計測値に重みを付けるステップを設け、その結果に基づいてステップＳ１０４の投票を行なえばよい。
【００７７】
前記の説明では、ステレオ画像処理とレーザレンジファインダの計測値を通常は１対１の重みになるよう定義しているが、装置の特徴に基づいて天候に応じた重みを付ける。例えば、雨の日、逆光の晴れ、順光の晴れ、曇りの日などの異なる天候下において、既知の位置に存在する物体をレーザレンジファインダとステレオ画像処理の両方で計測し、その計測値の真値に対する誤差の大きさや測定値の分散の大きさなどから、真値に近く分散の小さい計測が可能な装置の方の投票値の重みを大きくし、その逆であれば重みを小さくする。これにより、逆光などステレオ画像処理で誤計測が多い場面では、正確な計測が行われているレーザレンジファインダの計測結果の重みを大きくすることによって正確な結果が得られる。逆にレーザレンジファインダの計測が不安定で誤計測のある環境では、ステレオ画像処理の計測の重みを大きくすることにより、総合して天候の状況に影響を受けにくいロバストな計測が可能となる。また、投票値の重み付けを求める基準を、天候変化だけでなく、他の環境変化を基準とすることで、同様の方法で他の場面でも影響を受けにくくすることが出来る。
【００７８】
さらに、ステレオ画像処理とレーザレンジファインダの装置の特徴に基づいた投票の重みの付け方を説明する。ステレオ画像処理は、画像内が無地（エッジが存在しない）の場合には、原理的に距離を求めることができないため、距離を求める領域画像内の輝度分散の小さい場合は、誤計測の可能性が高い（前記図９の説明参照）。このことから、輝度分散がしきい値以下の位置で求められた値については重みを０にするか、または、重みを０.５〜０.１にするなど、計測値の信頼性に応じて重みを小さくする。これによって誤計測の可能性の高い値は表中に大きく反映されないようにする。また、レーザレンジファインダは、レーザレーダの届かない遠方の場合や光を吸収する物体に照射した場合は計測不可能もしくは誤計測値となる場合が多い。このことから、スキャニング方向毎にその方向への反射強度も計測し、その強度の低い位置での計測値では、重みを０にするか、もしくはステレオ画像処理と同様に、重みを０.５〜０.１にするなど信頼性に応じた重みを付ける。これにより、誤計測の値が表の投票に大きな影響を及ぼさないようにする。
【００７９】
以下、重み付けの具体的な方法について説明する。なお、通常時の重みを１とする。
（１）レーザレンジファインダやスキャンニングレーザレーダは雨、霧、雪に弱いことから次のように重みを付ける。
（１ａ）車両のワイパの動作段階に応じてレーザレーダの重みを小さくする。例えば、ワイパ不作動では重み１、低速動作では０.８、中速動作では０.６、最速動作では０.４とする。
（１ｂ）ＶＩＣＳ（Vehicle Information Communication System：道路交通情報通信システム）から得た当該地点および時間での降雨（雪）量によってレーザレーダの重みを小さくする。例えば降雨量０では重み１とし、降雨（雪）量があるときは、降雨量に逆比例する値を重みとする。例えば５０ｍｍのときは１／５、１００ｍｍの場合は１／１０、２００ｍｍのときは１／２０等の値にする。
（１ｃ）フォグランプのＯＮ、ＯＦＦに応じてレーザレーダの重みを変える。例えばフォグランプＯＦＦでは重みを１とし、フォグランプＯＮのときは重みを０.１または０とする。
（１ｄ）天候に応じて、既知の位置でレーザレーダの測距の正解率を調べ、その正解率を重みとする。例えば、異なる降雨量に応じて正解率を調べ、降雨量の情報をＶＩＣＳから求め、その降雨量のときに得られた正解率を重みとする。例えば降雨量１００ｍｍのときの正解率が２０％、５０ｍｍのときの正解率が５０％であれば降雨量１００ｍｍのときの重みを０.２、５０ｍｍのときの重みを０.５とする。
【００８０】
（２）ステレオ画像処理は、大雨、大雪、霧、逆光に弱い（ただし雨に対してはレーザレーダよりは良い）ことから次のように重みを付ける。
（２ａ）ＶＩＣＳから得た当該地点および時間での降雨、雪、霧に応じてステレオ画像処理の重みを小さくする。例えば降雨（雪）がないときは重み１とし、雨のときは、前記レーザレーダと同様の方法で重みを小さくする。
（２ｂ）ステレオ画像処理とレーザレーダの両方を載せた組み合わせの場合では、降雨時の重みは小さくするが、その重みはレーザレーダよりは大きく（例えばレーザレーダの重み×１.５にするなど）とする。このようにすれば、後述するミリ波レーダを加えた装置の場合に、ミリ波レーダの計測結果を有効に活用することが出来る。
（２ｃ）夕方に西に向かう場合や明け方に東へ向かうなどの逆光になる情報を時計とナビゲーションやジャイロなどの方位計測装置などから得て、それに応じて、逆光と予想されるときは、ステレオ画像処理の重みを小さくする。
（２ｄ）天候に応じて、既知の位置でステレオ画像処理での計測正解率を調べ、その正解率を重みとする。
（２ｅ）前記（１）で述べたレーザレーダと同様に、異なる霧の状態で、降雨量同様に透過率毎の正解率を重みとするものや、霧、雨、雪の異なる天候でのそれぞれの正解率を調べ、ＶＩＧＳより得た天候に応じて、そのときの正解率を重みとすることも出来る。
【００８１】
（３）ステレオ画像処理やレーザレーダの検知の状態から次のように重みを付ける。
（３ａ）ステレオ画像処理における画像内の分散の値がしきい値以下の部分で計測された距離値は重みを０にする。
（３ｂ）レーザレーダの反射強度がしきい値以下の部分で計測された距離値は重みを０にする。
（３ｃ）画像全体の分散が低い（しきい値以下）のときは、ステレオ画像処理の計測値の重みを０.２などの小さな値にする。
（３ｄ）スキャニングレーザレーダの同じ方向における検出値が時間的に連続して不安定な個所が多い（例えばしきい値の半分程度）場合は、雨などによるノイズの場合が多いので、レーザレーダの重みを小さくする。例えば、ノイズの数／全体のデータ数を重みとする。具体例を示すと、１回の走査で８０個のデータを得るスキャニングレーザレーダでノイズと判断されるデータが４０個の場合には重みを１／２＝０.５とする。
上記の各重み付けは、状況の応じてそれぞれを組み合わせて適用することが出来る。なお、重み付けは上記の数値を測定値に乗算することによって行なう。例えば、重み０.５というのは測定値に０.５を乗算した値を投票することを意味する。
【００８２】
次に、物体の認識を行なう例について説明する。なお、図１１では、物体認識のステップは記載していないが、ステップＳ１０５の次に、物体認識のステップを設ければよい。
前記図５〜図７に示した投票結果からもわかるように、車両を検知した個所での投票値の分布と木を検知した個所での投票値の分布は、エッジの有無や反射強度の強さによってそれぞれ特徴がある。このことから、例えば、既知の位置にある異なる対象物を、レーザレンジファインダとステレオ画像処理で計測したときにおける表への投票結果の分布形状を学習し、車両、人物、木など様々な対象物を、投票結果の形状のパターンで表現し、その投票結果のパターンと認識対象画像で計測された表への投票結果とのマッチングを行なうように構成することにより、物体の認識・判別処理が行える。パターンマッチングは、投票結果の形状同士の正規化相関や判別分析など、一般的な認識装置でよい。
【００８３】
さらに、この物体の認識をもとに、より計測を確実にするための投票の重み付けの方法を説明する。通常、木や人物は細長い形状をしていることから、路上に存在し、投票結果の分布の幅が狭い物体は、人物や木である確率が高い。人物や木は反射強度が低いことが多い一方で、エッジを多く含むものが多い。このことから、投票の際に、人物や木と認識された位置ではステレオ画像処理の結果に重みを大きくして投票するように構成するとよい。
【００８４】
次に、ステレオ画像処理やレーザレーダを用いた測距装置で問題となりがちな悪天候下での性能向上方法について説明する。
図１および図１１では、ステレオ画像処理とレーザレンジファインダまたはスキャンニングレーザレーダを用いた場合を示しているが、それらの何れかの代りもしくはそれらに追加して、雨、霧や暗闇でも測定可能なミリ波レーダを搭載することで対応が可能となる。構成としては、図１の構成にミリ波レーダを追加した構成、図１のステレオカメラ１、２の代わりにミリ波レーダを設けた構成、または図１のレーザレンジファインダ７の代わりにミリ波レーダを設けた構成が考えられる。また、図１１のフローチャートにおいては、ステップＳ１０１またはＳ１０２の代わりにミリ波レーダからの情報入力のステップとするか、もしくはステップＳ１０１、Ｓ１０２と並列にミリ波レーダからの情報入力のステップを設ければよい。
【００８５】
レーザレーダやステレオ画像処理は、反射強度や画像内の輝度分散の大きさによって、それらの装置での測距の信頼性を判断することが出来る。したがって、それらの装置の信頼性が、走査範囲全体において低い場合には、それらの装置では測定が不安定な悪環境下、悪天候下であると判断し、そのような場合は、ミリ波レーダでの測定結果に基づいた投票値に重みを付けるように構成する。これにより、悪環境下では測定不可能な装置による誤計測の悪影響を受けることなく、悪環境下でも計測が可能となる。ただし、ミリ波レーダには、悪環境下でも測定が可能な一方で、横方向の測定分解能が粗いという問題がある。したがって、ステレオ画像処理とレーザレンジファインダなどの信頼性が高い環境下では、横方向の分解能を高精度に維持するために、ミリ波レーダでの計測値に基づいた投票値の重みは小さくする。これにより、好天候下における高精度な測定を維持したまま、悪天候下での計測にも対応可能となる。
【００８６】
ミリ波レーダを追加した装置における重み付けの具体例としては、次のような構成が考えられる。すなわち、ミリ波レーダは悪天候には強いが精度が悪いことから、ＶＩＣＳから得た天候に応じて、降雨（雪）量がしきい値以上の場合（例えば他の測距装置の重みが０.５以下になるような場合）では、ミリ波レーダの計測値の重みを１とする。また、降雨（雪）量が０の場合には、ミリ波レーダの計測値の重みを０.５程度以下の小さな値とする。これにより、好天候では、ミリ波レーダによる分解能の粗い値が結果に反映されることを回避することが出来る。
【図面の簡単な説明】
【図１】本発明の一実施例のおける全体構成を示すブロック図。
【図２】ステレオ画像を用いて三角測量の原理でカメラから検出対象までの距離を求める原理を説明する図。
【図３】両画像の対応する位置毎の視差を求めた結果を示す図。
【図４】ステレオ画像上に距離画像作成のために切った一つ一つの小領域の画像上の位置と、その領域内に撮像される実空間上の物体との位置関係を示した図であり、（Ａ）は前方から見た斜視図、（Ｂ）は上方から見た平面図、（Ｃ）は側面図。
【図５】表への投票状況を示す図であり、（Ａ）は斜視図、（Ｂ）は撮像した画像と先行車に相当する距離ｚ＝Ｌ２における投票値を示す図、（Ｃ）は距離×路面に平行な角度（ｘ軸方向の角度）からなる投票用の表。
【図６】スキャニングレーザレーダによる計測状態を示した図であり、（Ａ）は全体の斜視図、（Ｂ）は反射強度と１次元方向毎の距離を示す図、（Ｃ）は投票状態を示す斜視図、（Ｄ）は距離×路面に平行な角度（ｘ軸方向の角度）からなる投票用の表。
【図７】レーザレンジファインダによる計測状態を示した図であり、（Ａ）は全体の斜視図、（Ｂ）は輝度画像、（Ｃ）は距離画像、（Ｄ）は投票状態を示す斜視図。
【図８】ステレオ画像処理での投票結果に加え、スキャニングレーザレーダでの投票結果を加算した様子を示した図。
【図９】物体の誤検知を防止する機能を説明するための図であり、（Ａ）は処理対象画像上に設定した各ウインドウの画像を示す図、（Ｂ）は画像上に定義したウインドウの位置と、それらの定義したウインドウの走査範囲を示す図、（Ｃ）はそれぞれのウインドウにおいて、視差方向に切った１ライン上の輝度分散を複数個所で求める状態をあらわした図、（Ｄ）は二つのカメラで撮像した対象物とその対象物が画像上に撮像されたときの位置関係を表した図。
【図１０】２台のカメラ１、２と、レーザレンジファインダ７を自車両に搭載した構成を示す側面図。
【図１１】一実施例における処理の流れを示す図。
【符号の説明】
１、２…電子式のカメラ ３、４…画像メモリ
５…演算部 ６…先行車
７…スキャンニングレーザレーダまたはレーザレンジファインダ
８…画像メモリ[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an obstacle recognizing device using two different types of distance measuring means, for example, a distance measuring device using stereo image processing and a distance measuring device such as a laser range finder, and is present in the traveling direction of a vehicle or the like. The technology relates to the recognition of obstacles.
[0002]
[Prior art]
As a conventional vehicle obstacle detection device, a device that detects an obstacle by performing image processing on an image in front of the vehicle captured by an electronic camera, or a light such as a laser range finder or a scanning laser radar is used. An apparatus for detecting obstacles ahead using radio waves, such as a millimeter wave radar, has been proposed.
[0003]
Also, as an obstacle detection device combining two different types of distance measuring devices, for example, a distance measuring device using image processing and another distance measuring device, it is described in, for example, JP-A-8-329393. There is something. This device is a combination of a device that performs monocular image processing using an image captured by a single camera and a laser radar. The monocular image processing determines an area where a preceding vehicle is present, and is reflected from the direction of the existing area. The distance measured by the laser radar is used as the inter-vehicle distance to the preceding vehicle.
[0004]
[Problems to be solved by the invention]
In the various distance measuring devices described above, image processing by an electronic camera has the advantage of being able to detect even objects with little reflection such as people and standing trees and having high resolution, but in fog, at night, etc. In such a situation where the contrast of the image is low, measurement is difficult. In addition, devices that use light reflection, such as laser radar, can be used at night and have high resolution, but are difficult to use in dense fog or rainy weather, and are less reflective objects such as people and standing trees. Not very suitable for detection. In addition, a device using radio waves such as a millimeter wave radar can be used even in dense fog, rain, or at night, but is unsuitable for identifying a detected object because the resolution in the distance and the lateral direction is rough. As described above, the obstacle detection device using the conventional distance measuring device alone has advantages and disadvantages, and it has been difficult to obtain a sufficient effect in all situations.
[0005]
Further, as described in Japanese Patent Laid-Open No. 8-329393, an obstacle detection device combining two types of distance measuring devices has also been proposed. In this device, an area where a preceding vehicle exists in image processing is proposed. Since the distance measured by the laser radar reflected from the direction of the existing area is used as the inter-vehicle distance, inter-vehicle distance measurement is impossible in fog or at night when image processing cannot be applied. Also, distance measurement is difficult in an environment where laser radar is inappropriate, such as in the rain or in dark clothes. In other words, the device described in the above publication cannot detect obstacles and measure distances unless both image processing and laser radar are sufficiently functioning. The problem with the obstacle detection device used was not solved.
[0006]
The present invention was made to solve the problems of the prior art as described above, and compensates for the disadvantages according to the type of distance measuring device, and improves the reliability of obstacle recognition in a wider range of situations. An object is to provide an object recognition device.
[0007]
[Means for Solving the Problems]
In order to achieve the above object, the present invention is configured as described in the claims. That is, in the first aspect of the invention, the measurement process is performed on a one-dimensional direction that is mounted on a vehicle and that is perpendicular to the traveling direction and parallel to the road surface, or a two-dimensional plane perpendicular to the traveling direction. At least two types of distance measuring means having different measurement principles for detecting an object existing ahead, a distance from the host vehicle in the z-axis direction, a position in which the x-axis direction is parallel to the road surface and the running direction as a reference, the y-axis Each table corresponding to the table in which the object is detected based on a table in which the direction indicates a vote value and a result measured by each of the at least two types of distance measuring means. of Voting means for voting on a position; and a determination means for searching for a place where the vote value in the table is equal to or greater than a predetermined threshold from the result of voting by the voting means and judging that an object exists at the place. The presence of the object is determined based on the result of adding the measurement values obtained by detecting the object in each distance measuring means. It is configured as follows.
[0008]
As described above, in the present invention, at least two types of distance measuring means having different measurement principles are used, and the respective distances and lateral positions are voted on the basis of the respective measurement results, and the added results are obtained. Is used to determine the presence and position (distance and direction) of the object, so that the presence and position of the object can be detected if at least one of the distance measuring means can measure the object. I can do it. Therefore, accurate measurement and discrimination can be performed even in a situation where one of the two types of distance measuring means is difficult to measure. Further, even in a situation where both distance measuring means are somewhat difficult to measure, by combining both results, it is possible to make a more accurate determination than when one type of distance measuring means is used.
[0009]
According to a second aspect of the present invention, as one of at least two types of distance measuring means, an image is picked up by two electronic cameras installed such that the optical axes are parallel to each other and face the traveling direction. The image processing means that detects the distance and the brightness for each area set in the image by performing stereo image processing on the image, and as another one-dimensional one perpendicular to the traveling direction and parallel to the road surface Using a distance measuring means using a laser radar that scans a two-dimensional plane perpendicular to the direction or the traveling direction with a laser beam and detects the distance and the reflection intensity for each region of the scanning direction or scanning surface It is. The distance measuring means using the laser radar includes, for example, a scanning laser radar that scans a one-dimensional direction and a laser range finder that scans a two-dimensional plane.
[0010]
As described above, as the two types of distance measuring means, an apparatus having completely different distance measuring principles, ie, an image processing means and a distance measuring means using a laser radar, are used. Can be measured. Therefore, by making a determination based on the result obtained by adding the measurement results of both, an accurate determination can be made under a wider situation.
[0011]
According to a third aspect of the present invention, reliability corresponding to the surrounding situation is obtained for each distance measuring means according to the characteristics of each distance measuring means, and each distance measuring means is used as a vote value in the voting means. Each means is configured to perform weighting according to the reliability. For example, in a state where the presence and location of obstacles in front are known, and under different circumstances of the surrounding environment such as various weather changes, the measurement reliability of each ranging device installed is learned, and the weather at the time of measurement, etc. According to the surrounding environment, the weight is increased for the measurement result of the distance measuring means with high reliability of measurement, and the weight is decreased for the measurement result of the distance measuring means with low reliability, and the vote is made. This makes it less susceptible to changes in the surrounding environment such as the weather.
[0012]
Further, in the invention described in claim 4, each voting result based on each measurement result of at least two types of distance measuring means having different measurement principles is compared and contrasted with a voting result for a previously known object. Thus, the type of the detected object is recognized. For example, when stereo image processing and laser radar are used as ranging means, the angle widths detected by both of these devices are both narrow and the value is high in stereo image processing, but the reflection intensity is high in laser radar. If it is low, it is judged as a person or a tree. In laser radar, the vote value is wide and high. In stereo image processing, the vote is judged to be a vehicle when both ends have high and low vote values. By making a determination based on the shape of the value distribution and the reflection intensity, the shape and type of the obstacle can be recognized. As a result, an object that cannot be detected by one type of distance measuring means can be accurately recognized under a wide range of conditions by using two types of distance measuring means.
[0013]
Further, in the invention according to claim 5, in the result of voting to the table based on the distance obtained by the image processing means, for a region where the width of the voting distribution at the position where the object is measured is narrow, measurement of stereo image processing is performed. The weight of the voting value based on the result is configured to be larger than other areas or the normal value. That is, when an object with a narrow width is detected, the object is likely to be a vertically long object other than a vehicle such as a person, and a vertically long object on the road surface has a texture (such as a surface pattern). Since there is no reflection surface and the probability of being an object suitable for stereo image processing is high, the result obtained by stereo image processing is voted with a large weight.
[0014]
In the invention described in claim 6, since the reliability is low when the contrast (luminance dispersion) of the image obtained by stereo image processing is low, the weight is reduced in the region where the contrast of the image is low, or The results of ranging means other than stereo image processing are weighted and voted on the table. This prevents erroneous measurement caused by stereo image processing, so obstacles can be detected by utilizing the results of other ranging means even in environments that cannot be measured by stereo image processing such as fog and nighttime.・ Measurement is possible.
[0015]
In the invention according to claim 7, in the configuration using the distance measuring means using the measurement of the reflection time of the laser beam such as the scanning laser radar or the laser range finder, when the reflection intensity of the laser beam is low Since the reliability of the measured value at that position is low, when voting the measurement result at that position to the table, the value with the reduced weight is voted. As a result, erroneous measurement caused by distance measurement by the laser radar can be prevented. Furthermore, when the other distance measuring means is a measuring apparatus using a camera such as a stereo image processing apparatus, measurement is possible even when the front object is an object having no reflecting surface. By reducing the weight on the measurement result by the radar, forward measurement can be performed without adversely affecting the measurement of stereo image processing by erroneous measurement of the laser radar.
[0016]
Further, in the invention described in claim 8, in the configuration using the millimeter wave radar as one of at least two types of distance measuring means having different measurement principles, the millimeter wave radar is a camera such as darkness without fog or reflecting objects. Because measurement is possible in bad weather and bad environments that are difficult to measure with laser radar, and in bad environments that are difficult to measure with other measurement devices, the voting value based on the measurement results with millimeter wave radar is weighted. wear. Millimeter-wave radars are resistant to adverse weather conditions, but are often rougher in measurement accuracy in the lateral direction parallel to the distance and road surface than other distance measurement methods such as laser radar and stereo image processing. Then, the weight of the vote value based on the measured value by the millimeter wave radar is reduced. By such a voting method, it is possible to measure with high accuracy without being affected by the measurement with low resolution under strong weather and in good weather. As a case where millimeter wave radar is used as one of the ranging means, for example, a combination of image processing means and millimeter wave radar, a combination of ranging means using laser radar and millimeter wave radar, and an image A combination of ranging means using processing means and laser radar and millimeter wave radar can be considered.
[0017]
【The invention's effect】
In the present invention, at least two types of distance measuring means having different measurement principles are used, and based on the respective measurement results, the corresponding distance and the lateral position are each voted, and the added result is used to determine the object. The existence and position of the object can be detected as long as at least one of the distance measurement means can measure the presence and position of the object. Therefore, accurate measurement and discrimination can be performed even in a situation where one of the two types of distance measuring means is difficult to measure. Further, even in a situation where both distance measuring means are somewhat difficult to measure, by combining both results, it is possible to make a more accurate determination than when one type of distance measuring means is used. Therefore, the certainty of obstacle recognition can be improved under a wider situation than before.
[0018]
Further, in the second aspect, as the two types of distance measuring means, an apparatus having completely different distance measuring principles, ie, an image processing means and a distance measuring means using a laser radar, are used. With each, accurate measurement can be performed. Therefore, by making a determination based on the result obtained by adding the measurement results of both, an accurate determination can be made under a wider situation.
[0019]
Further, according to claim 3, by voting a value obtained by weighting the measurement result obtained by the highly reliable distance measuring means under the measurement condition, the reliable measurement result is greatly increased in voting on the table. Therefore, if any one of the devices mounted on the vehicle can be measured, an object in front can be accurately detected and measured. In addition, since the ratio of voting values of erroneous measurement values decreases, even if there is a device that mismeasured among the ranging means, the measurement results of other devices can be used to make accurate measurements, and the whole As a result, robust measurement that is not easily affected by the surrounding environment such as the weather is possible.
[0020]
Further, according to claim 4, since different features appear in the shape of the vote value distribution according to the characteristics of the distance measuring means, the shape of the vote pattern is learned, and the patterns classified by type using the vote value distribution shape By performing matching, the shape and type of the object ahead can be recognized. In addition, even if an object that cannot be measured by one distance measuring means is an object that can be measured by the other distance measuring means, one pattern cannot be measured on the other hand and can be measured on the other. As a result, it is possible to recognize an object that cannot be measured with one device, and the information that the measurement is impossible is also used as effective information for object recognition, enabling wider and more reliable object recognition. It becomes.
[0021]
Further, according to the fifth aspect, an area where the width of the voting distribution at the position where the object is measured is reflected as a laser radar by increasing the weight of the voting value based on the measurement result of the stereo image processing. Even when combined with a distance measuring unit that is unstable in measuring objects that do not have a surface, the measurement of objects that do not have a reflective surface is reliable. Reliable distance measurement is possible.
[0022]
Further, in the sixth aspect, by reducing the weight in the region where the luminance dispersion is low in the stereo image processing, erroneous measurement due to the stereo image processing can be prevented, and robust measurement is possible.
[0023]
Further, in the seventh aspect of the present invention, in the measurement apparatus using the reflection time of the laser light, the reliability of the measurement result in the low reflection intensity portion is low, so that the measurement value in the low reflection intensity portion is weighted. By adopting a small configuration, it is possible to prevent erroneous measurement due to the distance measuring means using the reflection of the laser beam, and robust measurement is possible.
[0024]
Further, according to claim 8, when one of the distance measuring means is a millimeter wave radar, the voting value based on the measurement result of the millimeter wave radar is obtained under bad weather that is difficult to measure with other distance measuring means. In a favorable environment where the weight is increased and other distance measuring means can be measured, the weight of the voting value based on the measurement result of the millimeter wave radar is reduced, so that measurement is possible even under bad weather conditions. Under favorable weather conditions, it is possible to maintain high-accuracy measurement based on the measurement of other ranging means with fine horizontal resolution without being adversely affected by the roughness of the horizontal resolution of the millimeter wave radar.
[0025]
DETAILED DESCRIPTION OF THE INVENTION
The present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention, and exemplifies a case where a stereo image processing using two electronic cameras and a distance measuring device using a laser radar are used as a distance measuring device.
In FIG. 1, reference numerals 1 and 2 denote electronic cameras, which are installed in the front part of the vehicle facing forward, the optical axes of both cameras are parallel to each other, and the vertical axis of the imaging surface is the same. It is installed so that it is aligned on the top. In addition, you may install so that the horizontal axis of an imaging surface may align on the same line. Reference numerals 3 and 4 denote image memories for storing image signals input from the cameras 1 and 2, respectively. Reference numeral 5 denotes an arithmetic unit, which is composed of a microcomputer including, for example, a CPU, RAM, ROM, and the like. Reference numeral 6 denotes an object to be detected such as an obstacle existing in front of the host vehicle. FIG. 1 illustrates a preceding vehicle. Reference numeral 7 denotes a distance measuring device using a laser radar, for example, a scanning laser radar or a laser range finder, which is installed facing the front of the host vehicle. A scanning laser radar is a device that measures the distance to an object ahead and the reflection intensity (luminance) of the irradiation object in a one-dimensional manner by scanning the laser radar irradiation in the horizontal direction. This is a device that two-dimensionally measures the distance to the object ahead and the reflection intensity (luminance) of the irradiation target by scanning the radar irradiation vertically and horizontally. An image memory 8 stores an image signal input from a scanning laser radar or a laser range finder, and stores a distance image and a luminance image (reflection intensity). The cameras 1 and 2 and the scanning laser radar or the laser range finder 7 may be installed facing the rear of the vehicle so as to detect obstacles behind the vehicle. In addition to the apparatus using the laser beam as described above, a millimeter wave radar or the like that radiates radio waves can be used as 7.
[0026]
In the following, various calculation means and methods used in this embodiment will be described first, and then the overall calculation flow will be described based on a flowchart.
FIG. 2 is a diagram for explaining the principle of obtaining the distance from the camera to the detection target by the principle of triangulation using a stereo image. In FIG. 2, an image captured by the camera A (corresponding to the camera 1) is indicated by an image A, an image captured by the camera B (corresponding to the camera 2) is indicated by an image B, and the position of the detection target is indicated by a point p (x , Y, z).
[0027]
As can be seen from FIG. 2, the focal length f and the interocular distance (distance between both cameras) D are known, and two stereo images taken by two cameras A and B whose optical axes are parallel to each other are If the matching positions ya and yb between the images can be obtained, the distance z from the camera to the object p can be obtained from the following equation (Equation 1).
[0028]
z = f · D / (ya−yb) = f · D / S (Equation 1)
However, ya−yb = S is a parallax, and when an object is imaged by two cameras A and B installed at a predetermined interval with parallel optical axes as shown in FIG. This is the difference between the positions of images taken by the camera, that is, the difference between the position ya in the image A and the position yb in the image B. In this example, the unit of the interocular distance D and the distance z is m, and the unit of the focal length f, the parallax S, and the positions ya and yb is a pixel. For example, the cameras A and B use CCDs, and when the number of pixels is 640 × 480, the size of one pixel is about 10 μm.
[0029]
The above equation (1) is a case where the optical axes of both cameras are parallel to each other and the vertical axes of the imaging surfaces are aligned on the same line, but the horizontal axes of the imaging surfaces are on the same line. When installed so as to be aligned, the following (Equation 1 ′) is obtained.
z = f · D / (xa−xb) = f · D / S (Equation 1 ′)
Where xa−xb = S is parallax
In the following description, the case where the vertical axes of the imaging surfaces are all aligned on the same line will be described as an example.
[0030]
In order to detect the parallax S, the point (xb,) on the other image (eg, image B) corresponding to the point (xa, ya) where the point p is imaged on one image (eg, image A) is detected. It is necessary to detect yb). As the method, the image B can be obtained by searching a range most similar to an image (window) in a certain range including the point (xa, ya) on the image A. This similarity calculation includes a difference method between images and a normalized correlation method. And distance images (images for which parallax up to the object imaged inside each window is obtained) are positions where all defined windows have windows with high similarity to the other by the difference method or normalized correlation method Can be created by seeking.
[0031]
FIG. 3 is a diagram illustrating the result of obtaining the parallax for each corresponding position of both images. Specifically, in an image obtained by imaging the front of the road, one image (for example, image B) is cut for each window, and all of them are cut. In this window, the position of the image having the highest similarity to the window is obtained from the other image (for example, image A), so that the corresponding position in both images is detected, and the parallax for each window is determined from each corresponding position. It shows the obtained result. 3, (A) is a lower image (corresponding to image A), (B) is an upper image (corresponding to image B), (C) is a table of parallax, and (D) is a window portion where the parallax is “15”. (E) shows an image of a road surface on which no preceding vehicle exists. Also, (1) to (20) in FIGS. 3B and 3C show the positions of the windows in the horizontal direction (hereinafter referred to as the horizontal direction). However, in the figure, (1) to (20) are represented by circled numbers. One window has a width (length in the x direction) of xw and a height (length in the y direction) of yw.
As described above, if the parallax for each window is known, the distance to the object imaged in the corresponding window can be obtained by using the equation (1).
[0032]
Hereinafter, as shown in FIG. 3C, an image obtained by obtaining the parallax to the object imaged in each window is called a “distance image”. The parallax calculated for each window corresponds to a line segment that is imaged inside the window and has an edge (the point where the image changes from light to dark or from dark to light is continuous, indicating the edge of the image, etc. This corresponds to the distance to an object having a characteristic part such as (S)), and when one object is imaged across a plurality of windows, the same parallax is obtained in adjacent windows. For example, in the case of a distance image in an image captured in front of a road, the preceding vehicle and the road surface immediately below where the preceding vehicle exists are the same distance, and as shown by the thick line window in FIG. The window on the same y coordinate is calculated with the same parallax as the preceding vehicle. For example, “15” in the second row from the bottom in FIG. 3C is continuous in the horizontal direction, which corresponds to the above portion. In FIG. 3C, the part where the parallax “15” is gathered at the center corresponds to the preceding vehicle, and the part where the parallax “19” is gathered in the columns (3) and (4) is “ This corresponds to the “left tree”, and the portion (6) in which the parallax “5” continues in the row corresponds to the “center tree”. Further, as shown in FIG. 3E, when a road surface where no preceding vehicle is present is imaged, the y coordinate on the image and the distance z on the road imaged at that position are from the bottom to the top of the screen. It gradually becomes far away. That is, a near place is imaged in the lower part of the screen, and a distant area is imaged as it goes on the screen.
[0033]
As described above, when an object having a height in the front exists in the distance image, the same parallax is detected in the window at the x coordinate position where the object is imaged. On the other hand, at a position that does not have a height such as a road surface, such as a white line portion on the side of the vehicle, there is one window where the same parallax is detected on the same x coordinate. That is, when an object is present ahead, the object can be detected by counting the same number of parallaxes in a window in the same x-coordinate direction. According to this method, a plurality of objects and a single object can be detected by the same method, and the object can be detected regardless of the detection target and the background color. In addition, the road surface display such as a white line or a stop line has an advantage that a window showing the same parallax does not appear in the same direction, and thus it is not erroneously detected as a road surface display and an obstacle having a height. In addition, since only the distance image is used, a plurality of objects can be detected by the same process regardless of the color, shape, and background color of the detection target.
[0034]
FIG. 4 shows the position on the image of each small area cut for creating a distance image on the stereo image captured by the cameras 1 and 2, and the object in real space imaged in that area. (A) is a perspective view seen from the front, (B) is a plan view seen from above, and (C) shows a side view. In (A) and (B), the cameras 1 and 2 are shown as one.
[0035]
In FIG. 4, regions aligned on the same x coordinate defined on the image are images in the same direction. For example, an object (point P) imaged in the third region from the center on the image exists at a position where the angle from the optical axis is about 3 × α (where α is the horizontal direction angle for one region). It is an object. The distance from the camera to the object is obtained from the distance image obtained from the parallax. Therefore, as shown in FIG. 5, the angle (for example, the angle from the optical axis) in the left-right direction (x-axis direction) is parallel to the road surface, and the z-axis direction corresponds to the distance. Presence of a position with a high value in the table by voting for the table position corresponding to the defined position of the area on the distance image and the distance of the area by providing a table of “distance × angle parallel to the road surface” , The distance from the position where the value is high to the obstacle, the position in the horizontal direction, and the width of the object can be obtained. 5A is a perspective view, FIG. 5B is a diagram showing a captured image and a vote value at a distance z = L2 corresponding to the preceding vehicle, and FIG. 5C is a distance × an angle parallel to the road surface (x-axis direction). A voting table consisting of That is, FIG. 5 is a graph showing values in the y-axis direction, with the distance being taken in the z-axis direction and the x-axis being the x-axis direction of the image. This graph is created by voting to the corresponding position in FIG. 5 based on the value of the distance image as shown in FIG. Voting means, for example, an operation of adding y = 1 to x = 2 and z = 3 in the graph when the pixel is on x = 2 and the distance is 3 m.
[0036]
Such an operation is performed on the value of the entire distance image. When an object is present at the front z position, voting is repeated a plurality of times at the position of the distance z of the x coordinate at which the object is imaged, so that the value of that position in the table increases. For example, if there are five values of the distance 3m on the same x = 2 in the distance image value, for example, y = 5. Therefore, if a tall object is present at the same position x, the vote value (histogram height) at that location becomes a large value. On the other hand, on the x coordinate where no object exists, as shown in FIG. 3E, the same distance is not measured on the same x coordinate, so there is no position where the value is high in the table. Therefore, based on the voting result, the presence / absence of an object can be determined from the presence of a position having a large value in the table. Further, the position where the object exists is determined by the position in the x-axis direction of the range xl to xr (xl is the left end of the existence range and xr is the right end) where the vote value is a predetermined threshold value or more in the x-axis direction range. The distance to the object can be obtained from the value of distance z in the table.
[0037]
Specifically, for example, as shown in FIG. 3, a large number of values of the parallax 15 are collected at the position where the preceding vehicle exists, and therefore, as shown in FIGS. 5B and 5C, the value corresponds to the parallax 15. The voting values are collected in the range of the distance L2 at the left end xl and the right end xr. Similarly, the “left tree” in FIG. 5B is located at the second area from the left end and the distance is L1, and the “central tree” is the vote value at the fourth area from the left end and the distance is L3. Becomes larger.
[0038]
Next, FIG. 6 shows a measurement state when the distance in each angular direction is measured by a scanning laser radar that scans one-dimensionally in a direction parallel to the road surface in the same environment as FIG. (A) is a perspective view of the whole, (B) is a diagram showing the reflection intensity and the distance for each one-dimensional direction, (C) is a perspective view showing a voting state, and (D) is a distance × parallel to the road surface. A voting table composed of various angles (angles in the x-axis direction) is shown.
[0039]
In scanning laser radar, the distance to an object existing in each angular direction and the reflection intensity when the object is irradiated with a laser are measured. Since the vehicle has a reflective surface, if there is a vehicle ahead, the reflected light with high intensity returns, and the reflected intensity also leads to the reliability of the measurement result. In other words, in the same manner as the distance image obtained by stereo image processing, the reflection intensity is calculated from the one-dimensional data measured for each angle in the direction parallel to the road surface, for example, at the position of the distance table measured for each direction. By adding the proportional values, it is possible to measure the presence / absence of an obstacle ahead, the distance to the object, and the lateral range angle of the existing range. That is, FIGS. 5A and 6C, FIG. 5C and FIG. 6D are almost the same, and the position and distance of the obstacle in the traveling direction can be detected by the same processing. In the image processing, since the value of the edge portion is large, the histogram at the end portion of the preceding vehicle is high in FIG. 5A. However, since the scanning laser radar measures the distance by reflection from the object, FIG. In C), the height of the histogram is almost the same in the range where the preceding vehicle exists.
[0040]
Next, FIG. 7 is a diagram showing a state in which the same environment as FIG. 5 is measured by a laser range finder having a measurement surface whose measurement surface is parallel to the imaging surface of the camera, and FIG. (B) is a luminance image (reflection intensity), (C) is a distance image, and (D) is a perspective view showing a voting state.
[0041]
The laser range finder is a device that measures the distance in a two-dimensional plane by shaking the scanning of the laser radar shown in FIG. 6 not only in the horizontal direction but also in the vertical direction. In this apparatus, as shown in FIG. 7, a distance image for each angle in the vertical and horizontal directions similar to the distance image obtained by stereo image processing is obtained. From this, by using the distance image measured by the laser range finder and the reflection intensity, voting to the table in the same manner as in FIG. 5, the presence / absence, position, and distance of an obstacle existing ahead are measured. I can do it.
[0042]
FIG. 8 is based on the result of voting performed based on the distance image obtained by the stereo image processing illustrated in FIG. 5A and based on the measurement result of the scanning laser radar illustrated in FIG. It is the figure which showed a mode that the vote result was added. In FIG. 8, the black bar portion indicates the voting result of the stereo image processing shown in FIG. 5A, and the white bar portion indicates the voting result portion in the scanning laser radar shown in FIG. 6C.
[0043]
In FIG. 8, the result of stereo image processing and the result of scanning laser radar are scaled and added so as to be added with a weight of a ratio of about 1: 1. In this way, by adding the voting results of both, for example, if the object is an object that can be measured by scanning laser radar even in an environment where a distance image cannot be obtained by stereo image processing, such as in fog or at night, Since the value of the obstacle location in the table becomes high, the object can be detected. Conversely, even if there is an object that is difficult to measure with scanning laser radar, such as a person with clothes with low reflection intensity, the value in the table will be high if it can be measured with stereo image processing, so detection and measurement are possible. It becomes. Of course, in an environment where both can be measured, the position of the object is further increased, and detection and measurement with higher certainty is possible.
[0044]
In FIG. 8, the case where the stereo image processing and the scanning laser radar are combined is illustrated, but the voting result by the stereo image processing, the distance image obtained by the laser range finder, and the voting result based on the reflection intensity are added. May be.
[0045]
Next, a configuration in which weighting is performed when adding voting results will be described.
Measurement by stereo image processing has high certainty in an environment where the luminance contrast of the entire image is high, but low certainty in an environment with low contrast such as fog or night. In addition, scanning laser radars and laser range finders have advantages and disadvantages that differ depending on the distance measuring device, such as, for example, the reliability of measurement is low for rainy weather or objects with low reflection intensity, depending on the characteristics of the radar. Therefore, if the features of the stereo camera mounted on the vehicle and the environment of the distance measuring device such as the laser radar are learned, and the addition in FIG. 8 is performed, weighting corresponding to the environment is performed according to the respective features. The certainty of obstacle detection can be improved under a wide range of situations. For example, in an environment where the reliability of distance measurement by stereo image processing is high, weighting is performed so as to increase the vote value based on stereo image processing in the addition in FIG. And weight to make it smaller. Further, when the reliability of the measurement value of the laser radar is high, the weighting is performed so as to increase the value based on the measurement of the laser radar. As a result, a value with higher certainty is reflected more greatly in the distance measurement, and conversely, an influence of a value having a possibility of erroneous measurement is reduced, so that the measurement becomes more reliable.
[0046]
Next, a configuration for identifying the detected object from the measurement result will be described.
[0047]
5, 6, and 7 show a state in which the measured values when two trees and one vehicle are ahead are voted on the table. In these results, when the vehicle is measured, since the vehicle body has a relatively strong reflection intensity, the laser light applied to the vehicle usually has a strong reflection intensity. Therefore, the scanning laser radar or the laser range finder can detect the distance to the vehicle within the entire angle (within the range of xl to xr on the distance image) for imaging the vehicle. That is, at the time of voting on the table, as shown in FIGS. 6 and 7, histograms having the same height are formed in the entire range of the vehicle. On the other hand, in the case of stereo image processing, a certain distance is required near both ends of the vehicle including an edge in the region (near x = xl and x = xr), but a certain distance is required inside the vehicle compared to both ends. The number of measured areas is small. That is, in the voting on the table based on the result of the stereo image processing, as shown in FIG. 5, the voting value for both sides of the vehicle is high, and the inside is low compared to both sides. In general, a non-artificial object such as a person or a tree has many edges inside, and therefore, in stereo image processing, the distance can be reliably measured in all areas where the image is captured. Therefore, the vote value for the table at the position where the tree or person is imaged becomes high. However, trees and people can be measured with a laser radar, but since they do not have a metal reflecting surface, they are often weaker than the reflected light from the car body surface. As described above with reference to FIG. 6 and the like, in voting in an apparatus using a laser radar, a value proportional to the reflection intensity is added to the position in the distance table, so the voting value for an object with low reflected light is low. Become. Therefore, the vote value of the position where the tree or the person is detected using the measurement result by the scanning laser radar or the laser range finder is lower than that of the vehicle and is often lower than the result of the stereo image processing. .
[0048]
Summarizing the above results, the measurement results of the scanning laser radar and laser range finder have high reflection intensity (that is, high voting value) and the same value in a wide range of voting values, and stereo image processing In the measurement result, it can be determined that an object whose voting value is high at both ends and whose central portion is low is highly likely to be a vehicle.
[0049]
Moreover, in the measurement results of the scanning laser radar and the laser range finder, the reflection intensity is weak (that is, the vote value is low), and in the measurement result of the stereo image processing, an object having a high vote value in the entire region is a person or a standing tree. It can be judged that the possibility is high. In addition, since people and trees are generally narrower than vehicles and buildings, etc., it is possible to make a determination by adding width information to the vote value.
As described above, the distribution of voting values in the table for each measurement value is characteristic depending on the object. Therefore, by statistically learning this feature, it is possible to recognize an object using the distribution feature.
[0050]
Next, a configuration in which weighting is combined with object identification will be described.
For example, people and trees are often narrower than artificial objects such as vehicles, fences, and buildings. From this, the width of the detected object is obtained from the angle in the x-axis direction of the table, and when the width is narrow, the measured value of the stereo image processing excellent in detecting a person or a tree is weighted and voted. Thereby, detection by voting becomes more reliable. In other words, even when there is an erroneous measurement value due to noise or the like, the voting value can be made higher than the noise by weighting the position where the object is actually detected, so that adverse effects due to noise can be suppressed.
[0051]
Next, a configuration for preventing erroneous detection of an object in stereo image processing will be described.
FIG. 9 is a diagram for explaining a function for preventing erroneous detection of an object. FIG. 9A is a diagram showing images of windows set on a processing target image. FIG. When a star-shaped figure as shown is imaged, (a) in FIG. 9 (A) is a plain window (a portion without an imaged figure), and (b) has an edge only in a direction parallel to the parallax. (C) is a window having edges, and (d) is a window containing noise. The graphs shown under each figure in (a) to (d) are obtained by calculating the parallax by difference matching, and the difference value obtained for each pixel (parallax) is the difference value on the vertical axis and the parallax is on the horizontal axis. The graph shows a state in which a matching position (a position where the difference value is minimum = a position where the similarity is maximum) is obtained by searching for the minimum value from the graph. FIG. 9B is a diagram showing the positions of the windows (a) to (d) defined on the image and the scanning range of the defined windows. FIG. 9C is a diagram showing a state where the luminance dispersion on one line cut in the parallax direction is obtained at a plurality of locations in each of the windows (a) to (d). Further, FIG. 9D is a diagram showing a positional relationship when an object imaged by two cameras and the object imaged on the image.
[0052]
FIG. 9 shows the case where the optical axes of the two cameras are parallel to each other and the horizontal axes of the imaging surfaces are aligned on the same line for convenience of illustration (configuration in which the cameras are arranged in the horizontal direction). ) Shows the parallax detection. Therefore, as described with reference to FIG. 2, the parallax S is S = xa−xb, and the distance z is expressed by the above equation (1 ′). As shown in FIG. 2, when two cameras are arranged in the vertical direction, the parallax is obtained in the vertical direction, so that the parallax is S = ya−yb, but the basic functions are the same. .
[0053]
Moreover, said difference is the sum total of the brightness | luminance difference for every pixel of a reference | standard image and the other image, and similarity becomes large, so that a difference is small. Usually, the similarity and the difference are expressed by the following (Equation 2).
[0054]
[Expression 2]

[0055]
In the above formula (2), x is the luminance of the reference image (image A), x Is the average value of x, y is the brightness of the other image (image B), y Is the average value of y.
[0056]
When two cameras are installed so that the same line on the imaging surface is aligned on the same straight line, the matching positions on the images are on the same line. Thus, the search for similar images can be obtained by shifting one pixel at a time in the parallax direction on the same line as shown in FIG. The parallax of the window is obtained as the difference between the position where the image determined to have the lowest difference value in this operation and the position where the window of the reference image is defined. Further, when the right image is used as the reference image, the matching position with the window defined on the right image is on the right side of the left image with respect to the position defined on the right image as shown in FIG. 9D. . That is, the image search range may be scanned rightward from the position where the window is defined in the right image, and the parallax is shifted from the scanning start point to the matching position. In the example of FIG. 9, it is as shown in the graph of (c) of FIG.
[0057]
As described above, FIG. 9A shows the difference between the difference value (difference between the reference image and the image for which the difference is obtained) and the parallax (reference image and the image for which the difference is obtained) obtained for each pixel in each of the four windows. In the case of difference matching, the position with the smallest difference value is the position with the highest degree of similarity. However, as shown in FIG. 9A, for example, if the window and the search range of the window are both plain images, the difference becomes the same value at any position, so that the parallax cannot be determined. In a real image, there is a small noise even in such a window, so a difference between different values is obtained at each position, and the position of the minimum value is also obtained. In such a case, this position is the parallax of the object. Therefore, if this parallax is used for object detection, it may cause erroneous detection. Further, in a window having only an edge parallel to the parallax as shown in FIG. 9A (b), an edge considered to be a part of the object is imaged, but as in (a), the difference value is scanned. Since all values are the same within the range, the calculated parallax is likely not accurate. That is, it can be said that the parallax obtained in the cases (a) and (b) described above should not be obtained because the possibility of erroneous measurement is high.
[0058]
Hereinafter, a method for removing the cause of the erroneous detection as described above will be described.
FIG. 9C is a diagram illustrating a state in which luminance dispersion on one horizontal line cut in the parallax direction in each window is obtained at a plurality of locations. The luminance dispersion of a horizontal line window cut parallel to the parallax is the same for both a plain window as shown in (a) and a window with edges only in the direction parallel to the parallax as shown in (b). Small value. On the other hand, as shown in (c), the variance is large in the window where the parallax can be calculated.
[0059]
Further, when the luminance distribution of the entire window is calculated, the variance is small in the plain window of (a), but the variance of the window of (b) is large. However, if the variance of one horizontal line is calculated as shown in FIG. 9C, since the variances of both (a) and (b) are reduced, it is possible to accurately detect a window in which parallax calculation is inappropriate. . Thus, by determining whether it is appropriate to obtain the parallax based on the total or average of the variances on one line at a plurality of locations before matching is performed, the erroneous detection window can be removed. At the same time, the amount of calculation can be reduced.
[0060]
In addition, since the parallax is calculated by searching for the position of the minimum value from the difference value graph shown in FIG. 9A, a value close to the minimum value can be seen at a plurality of locations. As in the case of an image having only edges parallel to the direction, the certainty of the obtained parallax is low. For example, in a noisy image such as (d) in FIG. 9 (A), the variance obtained by the method of FIG. 9 (C) described above is large, but the difference value graph is very close to the minimum value. Since the values are obtained at a plurality of locations, it is difficult to determine the parallax from the values. Even if the minimum position of the obtained difference value is used as the parallax, this position is highly likely to be an erroneous correspondence position. Therefore, in such a case, it is better not to obtain parallax. Therefore, the following method is used.
[0061]
As shown in FIG. 9A, when the average value of the difference values obtained within the scanning range and the minimum difference value among them are obtained, if many values close to the minimum value among the entire difference values appear. If the difference between the average value of the differences obtained within the matching range and the minimum value of the differences is small, that is, if the difference between the two values is smaller than a certain threshold value, the parallax obtained in that window is incorrect. It can be determined that the response is high. Therefore, if the parallax determined to have a high possibility of miscorrespondence in the above determination is not used for voting on the table which is the subsequent processing, the detection of the object can be made more reliable. In the graph of FIG. 9A, the vertical axis indicates the difference. However, when the similarity of the reciprocal of the difference is taken, the difference between the maximum value of the similarity and the average value of the similarity is predetermined. If the threshold value is smaller than the threshold value, it is determined that the correspondence is incorrect.
[0062]
As described above, the parallax obtained in the entire matching region image is a black image, the entire image is a white image, or an edge or a texture (pattern, shape, etc.) captured in the image, The possibility of erroneous measurement is very high. The reason why a highly reliable distance is not calculated on the vehicle body surface is based on such a principle.
[0063]
In addition to a certain area in the image such as the body surface, the distance image itself by the stereo image processing may not be accurately obtained depending on an environment such as nighttime without fog or a streetlight. Therefore, voting such measurement values to the table leads to erroneous measurement such as voting to a position where nothing exists. Therefore, when the luminance dispersion on one horizontal line cut in the parallax direction in each window is small in each line, and the difference between the average value of the differences obtained within the matching range and the minimum value of the differences (the maximum similarity) If the difference between the similarity and the average value is small, there is a high possibility of erroneous correspondence. Therefore, it is desirable that the measured value obtained at that position is not reflected in the vote on the table. Specifically, a value in the above range is not used at all (weight = 0), or the weighting value is set to a significantly small value.
[0064]
Considering the luminance dispersion on the image leads to elimination of erroneous measurement, and therefore, it is an effective method for reducing erroneous measurement even in the case of a single stereo image processing. However, when only stereo image processing is used, measurement is impossible unless the measurement value is used as described above. However, in an apparatus that combines stereo image processing and laser radar, the environment described above is used. Below, reliable measurement can be performed by reflecting only the results of other measurement values such as laser radar. Therefore, only erroneous measurement can be reduced without making measurement impossible.
[0065]
In the above description, the case where only the measurement value of the laser radar is used without using the measurement value of the stereo image processing is described, but the reverse is also possible. That is, in a distance measuring device using light reflection time, such as a laser radar, if the object to be measured is an object that absorbs light, there is no reflected light, making measurement difficult. Normally, there is almost no light that absorbs 100% of light on the road or other environment, so some reflected light is measured, but the measured value with low reflection intensity is low in reliability and mismeasures the distance. Probability is high. Therefore, the intensity is measured simultaneously with the measurement distance for each scanning angle, and when the intensity of the light is lower than a predetermined value, the measurement value at that position is not reflected in the vote on the table. Specifically, a value in the above range is not used at all (weight = 0), or the weighting value is set to a significantly small value. In this case, only the measurement value of the stereo image processing is almost used. Thereby, erroneous measurement by the laser radar can be prevented.
[0066]
Next, in the description so far, the apparatus using the stereo image processing and the laser radar has been described. However, a case in which the millimeter wave radar is used instead of or in addition to any one of them will be described.
The millimeter-wave radar has a feature that it can measure distances to surrounding objects even when it cannot be measured by stereo image processing or laser radar, such as foggy, rainy or dark. However, in general, millimeter wave radar does not perform fine scanning like laser radar, so compared to measurement using stereo image processing that can obtain finer resolution than laser radar by scanning laser radar and scanning line scanning, There is a problem that the lateral detection angle resolution is coarse. Therefore, in bad weather that cannot be measured by other devices, the voting weight based on the measurement result by the millimeter wave radar is increased. On the other hand, in an environment where high-accuracy measurement by stereo image processing or laser radar is possible, the weight of the vote value based on the measurement value of the millimeter wave radar is set to be small. There are three possible combinations: stereo image processing and millimeter wave radar, laser radar and millimeter wave radar, stereo image processing, laser radar and millimeter wave radar. With the above-described configuration, it is possible to measure with high accuracy without being affected by the measurement with coarse resolution under strong weather and in good weather.
[0067]
Next, an embodiment will be described in which the position measurement of an object ahead and the object recognition are performed using the various methods described so far.
FIG. 10 is a side view showing a configuration in which two cameras 1 and 2 and a laser range finder 7 are mounted on the host vehicle. Here, the two cameras 1 and 2 are arranged vertically parallel to the road surface so that the y-axis of the imaging surfaces of the two cameras is on the same line. The laser range finder 7 has the optical axis of the camera parallel to the scanning center axis of the laser range finder, and the imaging surface of the camera [imaging surface of FIG. 4A] and the measurement surface of the laser range finder [FIG. The measurement surface of () is mounted in parallel. In FIG. 10, only one camera is shown, but actually there are two (in this example, in the vertical direction). For example, if a laser range finder is installed in the middle of two cameras, it is possible to reduce the positional deviation between the two images.
[0068]
FIG. 11 is a diagram showing the flow of processing in this embodiment.
In FIG. 11, first, in step S101, the images A and B from the two cameras 1 and 2 are input. At the same time, in step S102, a distance image and a luminance image that is the reflection intensity at that position are input from the laser range finder 7.
[0069]
Next, in step S103, a distance image is created using a stereo image (see the description of FIG. 3). In this distance image creation, before obtaining the parallax for each region, first, the luminance variance for each region defined on the image is obtained, and the parallax is calculated only for the portion where the variance is equal to or greater than the threshold (see FIG. 9). See description). By performing such pre-processing, only the measurement value in the highly reliable stereo image processing is used for the subsequent voting. Further, in a region where the variance is greater than or equal to a threshold value, a matching position (parallax) between stereo images is obtained, and a distance for each region is obtained. For this, a general method such as a normalized correlation method between images or a difference method may be used.
[0070]
Next, in step S104, the distance × horizontal table is voted using each of the distance image obtained by the stereo image processing and the distance image measured by the laser range finder (description of FIGS. 5 and 8). reference).
Here, first, voting is performed using a distance image obtained by stereo image processing. As described with reference to FIG. 5, focusing on each area of the distance image, the horizontal position of the table is an angle in the x-axis direction of the area, and the vertical position is a position where the distance z is calculated in the area. The process of adding the vote value to is repeated. For example, if there is a detected value in an area where the distance is determined as L2 in an area defined as a position where the horizontal direction is 3, the vote is given to a position where the horizontal direction in the table is 3 and the distance z is L2. Here, the vote value added to the table may be weighted according to the environment described later, or may be +1. Such an operation is performed in all areas where the distance is calculated in the distance image creation of the stereo image processing.
[0071]
The same operation is performed using the distance image measured by the laser range finder. However, in general, the angle of one region in the x-axis direction obtained by stereo image processing (α in FIGS. 4 and 5) and the scanning angle that moves for each measurement of the laser range finder (θ in FIGS. 6 and 7). ) Are different, it is necessary to convert so that values measured in the same direction in real space are voted at the same position in the table when voting on the table. To do so, the following calculation should be performed. That is, as shown in FIG. 4, the position in the x direction of the region on the image where the point P having the distance z from the camera and the distance xp from the optical axis of the camera to the x axis is imaged. (This is the nth region). Assuming that the range angle captured in one region on the image is α and the region where the optical axis is captured is 0th, and the position where the point P is captured is the nth region from the optical axis, n is The following equation (3) is used.
n = xp / (α × z) (Equation 3)
However, since α is a minute angle, tan α≈α.
[0072]
Further, the correspondence between the position on the image obtained by measuring the distance to the point P and the reflection intensity on the luminance image measured by the laser range finder and the position of the region cut on the image taken by the stereo camera is obtained. As shown in FIG. 7, it is assumed that the scanning angle that moves every sampling in the horizontal direction of the laser range finder is θ, and the measured value of the central axis of the laser range finder is zero. Assuming that the position where the point P is measured in the distance image of the laser range finder is m-th from the central axis, the relationship of the following formula (4) is established between n and m. However, since θ is a small angle, tan θ≈θ is set.
[0073]
n = m · (θ / α) (Equation 4)
In the laser range finder, the above equation (Equation 4) indicates that a point measured at the nth position in the horizontal direction from the central axis is measured in the mth region in stereo image processing. In the voting using the distance image measured by the laser range finder, the voting to the table is added after the conversion of the horizontal position according to the equation (4). As a value to be added, a value proportional to the reflection intensity at the position is added. At this time, in an environment where both the stereo image processing and the laser range finder can be measured, the value of the vote value between the laser range finder and the stereo image processing may be a value of 1: 1. That is, in this case, both weights are 1.
[0074]
Here, an example of a method for defining a vote value will be described. For example, an angle formed by a vertical direction (β in FIG. 4) captured in one region of a stereo image distance image is 1 degree, and a vertical scanning angle when obtaining data for one laser range finder (see FIG. Considering that the voting weights of both are the same when (φ of 7) is twice that of twice. At this time, if an object to be imaged exists in the range of the vertical angle of view of 10 degrees, voting is performed up to 10 times at the same position in the table in stereo image processing and up to 5 times in the laser range finder. That is, if the voting value of the stereo image processing result is “1” per distance and the voting value of the laser range finder measurement result is “2” per distance, the maximum voting value is the same for both. Become. In this way, if the voting values are such that the maximum values voted when objects measured within the same angle of view are the same, the ratio between the two voting values becomes 1: 1.
[0075]
Next, in step S105, the presence / absence of the object and the position of the object are obtained from the voting result of the entire distance image measured by the stereo image processing and the laser range finder. The presence / absence of an object can be determined from the presence / absence of a position having a vote value equal to or greater than a threshold value from the table. Further, the position of the object can be obtained by the reverse calculation of Equation (3). For example, it is assumed that a value equal to or greater than the threshold value is voted at a position in the table with a distance zp and a horizontal direction n. The distance of the object detected at this position is obtained as zp from the table. Further, the horizontal position xp (see FIG. 4) of the object with respect to the optical axis of the camera is calculated as xp = n × α × zp (α: one region defined in the table) by the inverse calculation of Equation (3). The angle of view in the horizontal direction to be imaged). If there are a plurality of points above the threshold in the table, it can be seen that there are a plurality of objects ahead, and the respective positions can be obtained by the same method as described above. In addition, when a value equal to or greater than the threshold exists as a lump on the same distance, it can be determined as one object, and both ends of the lump are object ends.
[0076]
Next, the weighting in voting in step S104 will be described. In FIG. 11, although the weighting step is not described, a step of weighting each measured value is provided after steps S103 and S102, and voting in step S104 may be performed based on the result.
[0077]
In the above description, the stereo image processing and the measurement value of the laser range finder are normally defined to have a one-to-one weight. However, a weight corresponding to the weather is given based on the characteristics of the apparatus. For example, in different weather conditions such as rainy days, sunny daylight, sunny daylight, and cloudy days, an object existing at a known position is measured by both laser range finder and stereo image processing. From the magnitude of error with respect to the true value, the magnitude of variance of the measured value, etc., the weight of the voting value is increased for the apparatus capable of measuring the variance close to the true value, and vice versa, the weight is reduced. Accordingly, in a scene where there are many erroneous measurements in stereo image processing such as backlight, an accurate result can be obtained by increasing the weight of the measurement result of the laser range finder in which the accurate measurement is performed. Conversely, in an environment where measurement of the laser range finder is unstable and erroneous measurement is performed, robust measurement that is not easily affected by the weather conditions can be achieved by increasing the weight of measurement in stereo image processing. In addition, by using not only the weather change but also other environmental changes as a reference for determining the weight of the vote value, it is possible to make it less susceptible to other scenes by the same method.
[0078]
Further, how to assign voting weights based on the features of the stereo image processing and laser range finder apparatus will be described. In stereo image processing, if the image is plain (no edge), the distance cannot be calculated in principle. Therefore, if the luminance dispersion in the area image for which the distance is calculated is small, there is a possibility of erroneous measurement. Is high (see the description of FIG. 9). For this reason, the value obtained at the position where the luminance variance is equal to or less than the threshold value is set to a weight of 0 or a weight of 0.5 to 0.1, depending on the reliability of the measured value. Reduce the weight. As a result, a value with a high possibility of erroneous measurement is not reflected in the table. In addition, the laser range finder often has a measurement impossible value or an erroneous measurement value when it is far away from the laser radar or when an object that absorbs light is irradiated. For this reason, the reflection intensity in that direction is also measured for each scanning direction, and the measured value at a position where the intensity is low is set to a weight of 0, or the weight is set to 0.5 to 0.5 as in the stereo image processing. Weight is set according to reliability, such as 0.1. This prevents erroneous measurement values from significantly affecting the table vote.
[0079]
Hereinafter, a specific method of weighting will be described. The normal weight is 1.
(1) Since the laser range finder and scanning laser radar are vulnerable to rain, fog, and snow, they are weighted as follows.
(1a) The weight of the laser radar is reduced according to the operation stage of the wiper of the vehicle. For example, the weight is 1 when the wiper is not operated, 0.8 is set for the low speed operation, 0.6 is set for the medium speed operation, and 0.4 is set for the maximum speed operation.
(1b) The weight of the laser radar is reduced by the amount of rainfall (snow) at the relevant point and time obtained from VICS (Vehicle Information Communication System). For example, when the rainfall amount is 0, the weight is 1, and when there is a rainfall (snow) amount, the weight is inversely proportional to the rainfall amount. For example, the value is 1/5 for 50 mm, 1/10 for 100 mm, and 1/20 for 200 mm.
(1c) The weight of the laser radar is changed according to whether the fog lamp is on or off. For example, the weight is set to 1 when the fog lamp is OFF, and the weight is set to 0.1 or 0 when the fog lamp is ON.
(1d) Check the accuracy rate of laser radar ranging at a known position according to the weather, and use the accuracy rate as a weight. For example, the correct answer rate is checked according to different rainfalls, the rainfall information is obtained from VICS, and the correct answer rate obtained at the time of the rainfall is used as a weight. For example, if the accuracy rate is 20% when the rainfall is 100 mm and the accuracy rate is 50% when the rainfall is 50 mm, the weight when the rainfall is 100 mm is 0.2, and the weight when the rainfall is 50 mm is 0.5.
[0080]
(2) Since stereo image processing is weak against heavy rain, heavy snow, fog, and backlight (however, it is better than laser radar for rain), the following weighting is applied.
(2a) The weight of stereo image processing is reduced according to the rain, snow, and fog at the point and time obtained from VICS. For example, the weight is set to 1 when there is no rain (snow), and the weight is reduced by the same method as the laser radar when it is raining.
(2b) In the case of a combination in which both stereo image processing and laser radar are mounted, the weight at the time of raining is reduced, but the weight is larger than that of laser radar (for example, the weight of laser radar × 1.5). And In this way, the measurement result of the millimeter wave radar can be used effectively in the case of an apparatus to which a millimeter wave radar described later is added.
(2c) When information that becomes backlit, such as when heading west in the evening or heading east at dawn, is obtained from a clock and a direction measuring device such as a navigation or gyroscope, Reduce the weight of image processing.
(2d) According to the weather, the measurement accuracy rate in stereo image processing is checked at a known position, and the accuracy rate is set as a weight.
(2e) Similar to the laser radar described in (1) above, in a different fog state, each of which is weighted with a correct answer rate for each transmittance as in the case of rainfall, or in different weather such as fog, rain, and snow The correct answer rate at that time can be used as a weight according to the weather obtained from VIGS.
[0081]
(3) Weights are assigned as follows based on the state of stereo image processing or laser radar detection.
(3a) The distance value measured at the portion where the variance value in the image in the stereo image processing is equal to or less than the threshold value has a weight of zero.
(3b) The distance value measured in the portion where the reflection intensity of the laser radar is equal to or less than the threshold value is set to zero.
(3c) When the variance of the entire image is low (below the threshold value), the weight of the measurement value of the stereo image processing is set to a small value such as 0.2.
(3d) When there are many places where the detected values in the same direction of the scanning laser radar are unstable continuously in time (for example, about half of the threshold value), there are many cases of noise due to rain or the like. Reduce the weight. For example, the number of noises / total number of data is used as a weight. As a specific example, when the number of data determined as noise by the scanning laser radar that obtains 80 data in one scan is 40, the weight is set to 1/2 = 0.5.
Each of the above weights can be applied in combination depending on the situation. The weighting is performed by multiplying the measured value by the above numerical value. For example, a weight of 0.5 means that a value obtained by multiplying a measured value by 0.5 is voted.
[0082]
Next, an example in which an object is recognized will be described. In FIG. 11, the object recognition step is not shown, but an object recognition step may be provided after step S105.
As can be seen from the voting results shown in FIGS. 5 to 7, the distribution of voting values at locations where a vehicle is detected and the distribution of voting values at locations where a tree is detected are the presence or absence of edges and the strength of reflection intensity. Each has its own characteristics. From this, for example, learning the distribution shape of the voting results to the table when different objects at known positions are measured by laser range finder and stereo image processing, various objects such as vehicles, people, trees, etc. Can be recognized and discriminated by configuring the voting result pattern to match the voting result pattern with the voting result of the table measured by the recognition target image. . The pattern matching may be a general recognition device such as a normalized correlation between shapes of voting results or discriminant analysis.
[0083]
Further, a voting weighting method for ensuring the measurement will be described based on the recognition of the object. Usually, since trees and people have an elongated shape, an object that exists on the road and has a narrow distribution of voting results has a high probability of being a person or a tree. While people and trees often have low reflection intensity, they often contain many edges. For this reason, at the time of voting, it is preferable to make a vote by increasing the weight of the result of stereo image processing at a position recognized as a person or a tree.
[0084]
Next, a description will be given of a method for improving performance under bad weather, which tends to be a problem in a distance measuring device using stereo image processing or laser radar.
1 and 11 show the case of using stereo image processing and a laser range finder or scanning laser radar, but it is possible to measure in rain, fog or darkness instead of or in addition to any of them. It is possible to respond by installing a simple millimeter wave radar. As a configuration, a configuration in which a millimeter wave radar is added to the configuration in FIG. 1, a configuration in which a millimeter wave radar is provided in place of the stereo cameras 1 and 2 in FIG. 1, or a millimeter wave radar in place of the laser range finder 7 in FIG. A configuration in which is provided is conceivable. In the flowchart of FIG. 11, instead of step S101 or S102, the step of inputting information from the millimeter wave radar is used, or the step of inputting information from the millimeter wave radar is provided in parallel with steps S101 and S102. Good.
[0085]
Laser radar and stereo image processing can determine the reliability of distance measurement by these devices based on the reflection intensity and the magnitude of luminance dispersion in the image. Therefore, if the reliability of these devices is low over the entire scanning range, they are judged to be in bad environments and in bad weather where measurement is unstable. The voting value based on the measurement result is weighted. As a result, measurement can be performed even in an adverse environment without being adversely affected by erroneous measurement by a device that cannot be measured in an adverse environment. However, the millimeter wave radar has a problem that the measurement resolution in the lateral direction is rough while it can be measured even in a bad environment. Therefore, in a highly reliable environment such as stereo image processing and a laser range finder, the weight of the vote value based on the measurement value by the millimeter wave radar is reduced in order to maintain the horizontal resolution with high accuracy. As a result, it is possible to support measurement under bad weather while maintaining high-precision measurement under good weather.
[0086]
As a specific example of weighting in a device to which a millimeter wave radar is added, the following configuration is conceivable. In other words, the millimeter wave radar is strong in bad weather but has low accuracy. Therefore, according to the weather obtained from VICS, the amount of rainfall (snow) is equal to or greater than a threshold (for example, the weight of another distance measuring device is 0. In the case of 5 or less), the measurement value weight of the millimeter wave radar is set to 1. When the amount of rainfall (snow) is 0, the weight of the measurement value of the millimeter wave radar is set to a small value of about 0.5 or less. Thereby, in favorable weather, it can be avoided that a coarse value of resolution by the millimeter wave radar is reflected in the result.
[Brief description of the drawings]
FIG. 1 is a block diagram showing an overall configuration in an embodiment of the present invention.
FIG. 2 is a diagram for explaining a principle for obtaining a distance from a camera to a detection target by using a stereo image by a principle of triangulation.
FIG. 3 is a diagram illustrating a result of obtaining a parallax for each corresponding position of both images.
FIG. 4 is a diagram showing a positional relationship between an image of each small area cut for creating a distance image on a stereo image and an object in real space imaged in the area; Yes, (A) is a perspective view seen from the front, (B) is a plan view seen from above, (C) is a side view.
5A is a perspective view, FIG. 5B is a perspective view, FIG. 5B is a diagram illustrating a voted value at a distance z = L2 corresponding to a captured image and a preceding vehicle, and FIG. A voting table consisting of distance x angle parallel to the road surface (angle in the x-axis direction).
6A and 6B are diagrams showing a measurement state by a scanning laser radar, where FIG. 6A is a perspective view of the whole, FIG. 6B is a diagram showing reflection intensity and distance in one-dimensional direction, and FIG. 6C is a voting state. The perspective view shown, (D) is a table for voting consisting of an angle parallel to the road surface x (angle in the x-axis direction).
7A and 7B are diagrams showing a measurement state by a laser range finder, where FIG. 7A is an overall perspective view, FIG. 7B is a luminance image, FIG. 7C is a distance image, and FIG. 7D is a perspective view showing a voting state. .
FIG. 8 is a diagram illustrating a state in which a voting result obtained by a scanning laser radar is added to a voting result obtained by stereo image processing.
FIGS. 9A and 9B are diagrams for explaining a function for preventing erroneous detection of an object, in which FIG. 9A is a diagram showing an image of each window set on a processing target image, and FIG. 9B is a window defined on the image; (C) is a diagram showing the positions of and the scanning ranges of those defined windows, (C) is a diagram showing the state in which the luminance dispersion on one line cut in the parallax direction is obtained at a plurality of locations in each window, (D) FIG. 3 is a diagram illustrating a positional relationship when an object captured by two cameras and the object are captured on an image.
FIG. 10 is a side view showing a configuration in which two cameras 1 and 2 and a laser range finder 7 are mounted on the own vehicle.
FIG. 11 is a diagram showing a flow of processing in one embodiment.
[Explanation of symbols]
1, 2 ... Electronic camera 3, 4 ... Image memory
5 ... Calculation unit 6 ... Advance vehicle
7. Scanning laser radar or laser range finder
8 ... Image memory

Claims (8)

It is mounted on a vehicle and detects an object existing ahead by performing measurement processing on a one-dimensional direction perpendicular to the traveling direction and parallel to the road surface or a two-dimensional plane perpendicular to the traveling direction. At least two different ranging means;
a table in which the z-axis direction is the distance from the host vehicle, the x-axis direction is parallel to the road surface and the position is based on the traveling direction, and the y-axis direction is the vote value;
Voting means for said at least two distance measuring means based on the result of measurement, respectively, to vote to the appropriate position of each of the table object is detected,
From the voting result by the voting means, search for a place where the voting value in the table is equal to or greater than a predetermined threshold, and judging means for judging that an object exists at the place;
An obstacle recognition apparatus comprising: determining the presence of an object based on a result obtained by adding measurement values obtained by detecting objects in the respective distance measuring means . One of the distance measuring means performs stereo image processing on an image captured by two electronic cameras installed so that the optical axes are parallel to each other and face the traveling direction, thereby setting each image set to an image. Image processing means for detecting distance and brightness for each region;
The other distance measuring means performs scanning with a laser beam on a one-dimensional direction perpendicular to the traveling direction and parallel to the road surface or a two-dimensional plane perpendicular to the traveling direction. Ranging means using a laser radar that detects the distance and reflection intensity for each region,
The voting means sequentially votes a predetermined value to the position of the table corresponding to the distance and position of the area where the distance is detected by the image processing means, and the distance is detected by the distance measuring means using the laser radar. In order to vote the value in proportion to the reflection intensity of the region to the position of the table corresponding to the distance and position of the region,
The determination means searches for a place where the vote value in the table is equal to or greater than a predetermined threshold for the vote result obtained by adding the result of the image processing means and the result of the distance measurement means using the laser radar. The obstacle recognizing apparatus according to claim 1, wherein it is determined that an object exists at the location. According to the characteristics of each distance measuring means, the reliability according to the surrounding situation is obtained for each distance measuring means, and the voting value in the voting means is weighted according to the reliability for each distance measuring means. The obstacle recognition apparatus according to claim 1 or 2, wherein the obstacle recognition apparatus performs the following. The respective voting results based on the respective measurement results of at least two types of distance measuring means having different measurement principles are compared with the voting results for previously known objects, thereby detecting the shape and type of the detected object. The obstacle recognition apparatus according to any one of claims 1 to 3, wherein at least one of the two is recognized. In the result of voting to the table based on the distance obtained by the image processing means, for a region where the width of the voting distribution at the position where the object is measured is narrow, the weight of the voting value based on the measurement result of the stereo image processing is different. The front obstacle recognition apparatus according to any one of claims 2 to 4, wherein the area is larger than the normal area or the normal value. Regarding the luminance obtained by the image processing means, in an area where the luminance dispersion is low, the voting value weight based on the measurement result of the image processing means is made smaller than other areas, or distance measurement other than the image processing means 6. The obstacle recognition apparatus according to claim 2, wherein a weight of a voting value based on a measurement result of the means is larger than a weight of the image processing means. Regarding the reflection intensity obtained by the distance measuring means using the laser radar, the weight of the vote value of the area is made smaller than the weight in the other area or the other distance measuring means for the area where the reflection intensity is low. The obstacle recognition apparatus according to any one of claims 2 to 6. The millimeter wave radar is used as one of at least two types of distance measuring means having different measurement principles, and the weight of the vote value based on the measurement result of the millimeter wave radar is made larger than the weight of other distance measuring means in bad weather. The obstacle recognition according to any one of claims 1 to 7, wherein a weight of a voting value based on a measurement result of the millimeter wave radar is set to be smaller than a weight of another distance measuring means in good weather. apparatus.

Images