JP4106163B2 - Obstacle detection apparatus and method - Google Patents

Description

という関係式が成り立つ。
【００４４】
ｈ＝（ｈ_１１，ｈ_１２，……，ｔ_３）^Ｔ，ｈ'＝（ｈ'_１１，ｈ'_１２，……，ｔ'_３）^Ｔは各カメラの自車座標系に対する３次元位置と姿勢、各カメラに装着したレンズの焦点距離、画像中心に関するパラメータである。
【００４５】
式(1)，(2)からＸとＹを消去すると、
【数２】

となる。
【００４６】
ただし、Ｈ＝（Ｈ_１１，Ｈ_１２，……Ｈ_３３）^Ｔはｈ，ｈ'で表される定数である。
【００４７】
この式により、左画像上の点（ｕ，ｖ）が道路面上の点であると仮定した時の右画像上の対応点を求めることができる。
【００４８】
言い換えると、左画像上の点（ｕ，ｖ）が道路平面上にあると仮定すると、その右画像上の対応点（ｕ_ｒ，ｖ_ｒ）は上式によって決まる。この式を道路平面拘束式と呼ぶ。
【００４９】
なお、この式のパラメータＨ＝（Ｈ_１１，Ｈ_１２，……Ｈ_３３）^Ｔは、静止時に予め求めておく。
【００５０】
以下にその方法について説明する。
【００５１】
まず、左画像から道路面上の特徴点（道路平面上に描かれた直線の交点や、ペイントの角点等）をＮ（≧４）個抽出する。
【００５２】
次に、抽出した各特徴点の右画像上の対応点を求める。なお、この特徴点抽出と対応づけ処理は、マウスで画像上の点をポインティングすることにより行ってもよい。これらのＮ組の対応関係は、各々、式(3)を満たすので、２Ｎ本の連立方程式を得る。
【００５３】
この連立方程式をＨについて解くことによりパラメータＨを求めることができる。
【００５４】
この道路平面拘束式を用いて障害物を検出する方法について以下に示す。
【００５５】
左画像の任意の点Ａ（ｕ，ｖ）の輝度をＩ_Ｌ（ｕ，ｖ）とし、点Ａが道路平面上に存在すると仮定した場合の右画像上の対応点Ａ'（ｕ_ｒ，ｖ_ｒ）を式(3)から求め、その輝度をＩ_Ｒ（ｕ_ｒ，ｖ_ｒ）とする。
【００５６】
点（ｕ，ｖ）が実際に道路平面上に存在すれば、点ＡとＡ'は正しい対応点の組（点ＡとＡ'が道路面上の同じ点の投影点）となるから、基本的には点Ａと点Ａ'の輝度が同じになる。したがって、
【数３】

とおくとＤｉｆｆ＝０となる。
【００５７】
逆に、Ｄｉｆｆ＝０でなければ点Ａ（ｕ，ｖ）とＡ'（ｕ_ｒ，ｖ_ｒ）は正しい対応点の組ではないことになり、点（ｕ，ｖ）は道路平面上に存在しない点、つまり障害物上の点となる。
【００５８】
これら一連の処理を基準画像上の全ての点に対して行うと、障害物領域を検出することができる。
【００５９】
この際、障害物か否かの判定基準として、ある程度の誤差を考慮し、Ｄｉｆｆ＞Ｔｈｒとなる点は障害物領域に属すると判定してもよい。
【００６０】
ここで、Ｔｈｒは予め設定した閾値である。
【００６１】
例えば、図６上に示すステレオ画像から、同図下に示すような障害物領域が検出される。
【００６２】
（衝突時間計測部５）
衝突時間計測部５は、ステレオ画像のうちのどちらかの画像（これを基準画像と呼び、本実施例では、前記したように左画像を基準画像とする）上の障害物の軌跡（障害物の位置の時間変化）から、自車がその障害物に衝突するまでの時間（衝突時間）を計測する。
【００６３】
まず、衝突時間について説明する。
【００６４】
式(1)の分母をＤとおく。すなわち、
Ｄ＝ｈ_３１Ｘ＋ｈ_３２Ｙ＋ｔ_３ (5)
である。
【００６５】
Ｄは、図７に示すように、障害物の道路平面との接点Ｔ（Ｘ，Ｙ）と基準カメラの視点Ｃの光軸方向の距離、すなわち奥行きである。
【００６６】
時刻ｔ−ｄｔ，ｔにおける障害物の道路平面との接点Ｔ'，Ｔの奥行きを各々Ｄ'，Ｄとする。ここで、障害物が自車座標系に対して時刻ｔ−ｄｔ，ｔ間の運動を継続した場合に奥行きが０になるまでの時間を衝突時間ｔ_ｃと定義する。
【００６７】
図８に奥行きの時間変化を示す。
【００６８】
現在の時刻ｔから奥行きが０になるまでの時間が衝突時間ｔ_ｃなので、ｔ_ｃは同図のように表される。斜線で示した２つの三角形の相似関係から、
【数４】

ただし、γ＝Ｄ'／Ｄである。
【００６９】
つまり、異なる２時刻における奥行きの比γから衝突時間ｔ_ｃを求めることができる。
【００７０】
以下に検出部４によって求めた各時刻の障害物領域の位置から、その衝突時間を計算する方法について説明する。
【００７１】
現在の時刻ｔとそれより前の時刻ｔ−ｄｔにおいて障害物領域が基準画像上で図９に示すように検出されたとし、時刻ｔにおいて検出された障害物領域の最下端の線（障害物領域と道路領域の境界線）上の任意の１点をｕ＝（ｕ，ｖ）とする。点ｕ＝（ｕ，ｖ）の時刻ｔ−ｄｔにおける位置ｕ'＝（ｕ'，ｖ'）は、点ｕ _０，ｕを結ぶ直線と、時刻ｔ−ｄｔにおける障害物領域と道路領域の境界線の交点として求めることができる。
【００７２】
また、時刻ｔ−ｄｔ，ｔにおける障害物の道路平面上の位置を各々、Ｘ'＝（Ｘ，Ｙ＋ｄＹ），Ｘ＝（Ｘ，Ｙ）とする。
【００７３】
本実施例では、自車と障害物はＹ軸に沿って運動する（進行方向が一致する）と仮定しているので、各時刻の障害物の位置のＸ座標は同一となる。Ｘ'までの奥行きは、
【数５】

ここで、Ｙ軸に平行な直線ｌ，ｌ'上の無限遠方の点の投影点がｕ _０＝（ｕ_０，ｖ_０）であるから、式(1)でＹ→∞として、
【数６】

このγを式(7)に代入することによって、障害物の衝突時間を計算することができる。
【００７４】
つまり、図９に示すように道路平面上を運動する障害物の衝突時間は、その障害物の道路平面との接点の時刻ｔ−ｄｔ，ｔにおける画像上の位置ｕ'，ｕと、直線ｌ，ｌ'の消失点ｕ _０のみから計算でき、自車座標系に対するカメラの位置や姿勢に依存しない。
【００７５】
以上のようにして、車載のステレオカメラからカメラパラメータを用いずに、かつ、奥行き探索を行うことなく道路平面上の障害物を検出し、さらに、その障害物が自車に衝突するまでの時間を計算することができる。
【００７６】
（実施例の効果）
以上により、本実施例であると、ステレオカメラで取得したステレオ画像を処理して、道路平面からの高さの有無により障害物を検出するため、明るさの変動や影の影響を受けず、画像中から先行車や歩行者等の障害物を検出することができる。また、従来のステレオ視の問題であった、多大な時間と労力を要するカメラキャリブレーションの作業と、計算コストの高い奥行き探索（対応探索）処理が不要なため、実用的効果は多大である。
【００７７】
（変更例１）
本実施例は画像入力部１で、２台のＴＶカメラを左右に並べて２枚の画像を入力しているが、これらのカメラは上下に配置してもよい。
【００７８】
また、３台以上のカメラを配置してもよい。
【００７９】
（変更例２）
特徴抽出部３は、道路平面上の２本の線を抽出する場合について説明したが、３本以上の線を抽出してもよいし、方向ベクトルが同じであれば、道路平面上にない線、例えば、ガードレール上の線を抽出してもよい。
【００８０】
（変更例３）
検出部４はさらに図１０に示す構成をとることもできる。
【００８１】
ここでは、画像変換部４−１，差異計算部４−２から構成している。
【００８２】
画像変換部４−１は、右画像を以下の手順に従って画像変換する。一般に、画像は画像上の点（ｕ，ｖ）を変数とし、その各点に対して輝度値が定義された関数ｆ（ｕ，ｖ）として表現できる。以下では画像をこのように表現する。
【００８３】
図１１に示すようなステレオ画像が入力されたとし、右画像をｇ（ｕ，ｖ）、その変換後の画像をｇ'（ｕ，ｖ）を求める。以下のように、ｇ'（ｕ，ｖ）を求める。
【００８４】
【数７】

ただし、（ｕ_ｒ，ｖ_ｒ）は、式(3)より求める。ｇ'（ｕ，ｖ）は、画像ｇ（ｕ，ｖ）上の全ての点が道路平面上に存在すると仮定した場合に、左カメラで得られる画像である。
【００８５】
例えば、図１２の右画像からは、同図に示すような変換画像を得る。
【００８６】
図１３に示すように、道路平面上にある点の投影点は、左画像と変換画像で同一となるのに対し、道路平面上にない点、すなわち、障害物（この場合は先行車両）上の点は、道路からの高さに応じて異なる位置に投影される。
【００８７】
したがって、この左画像と変換画像の差分を取ることにより、道路平面上の障害物を検出する。つまり、左画像をｆ（ｕ，ｖ）とすると、
【数８】

として、Ｄｉｆｆ'＝０でなければ、あるいは誤差を考慮して、Ｄｉｆｆ'＞Ｔｈｒ（Ｔｈｒは予め設定した閾値）となる点（ｕ，ｖ）は障害物領域に属すると判定する。
【００８８】
（変更例４）
検出部４は画素間差分をとることによって２枚の画像の差異を検出したが、各点に対して（２ｗ＋１）×（２ｗ＋１）のウィンドウを設定し、ウィンドウ内の輝度値の平均や分散、正規化相互相関等から差異を検出してもよい。
【００８９】
２枚の画像Ｆ（ｕ，ｖ），Ｇ（ｕ，ｖ）の点（ｕ，ｖ）の正規化相互相関Ｃは以下の式から算出できる。
【００９０】
【数９】

ここで、Ｎ＝（２ｗ＋１）×（２ｗ＋１），ａ_１，ａ_２は２枚の画像のウィンドウ内の輝度の平均、σ^２ _１，σ^２ _２は２枚の画像ウィンドウ内の輝度の分散であり、−１≦Ｃ≦１である。
【００９１】
この場合、Ｃ＜Ｔｈｒとなる点（ｕ，ｖ）が障害物領域に属すると判定する。ここで、Ｔｈｒ（≦１）は予め設定した閾値である。
【００９２】
（変更例５）
本実施例では道路両端の２本の白線を直線として抽出したが、道路がカーブしている場合には白線は曲線となる。
【００９３】
この場合には、白線を曲線として抽出すれば、同様に障害物を検出することができる。
【００９４】
（変更例６）
道路面として平面を仮定して説明したが、曲面の場合であっても、平面の場合と同様に障害物を検出することができる。
【００９５】
（変更例７）
自車と障害物に対し、車線に平行な運動を仮定したが、平行でない場合（例えば、車線変更する場合）にも適用可能である。
【００９６】
（変更例８）
本実施例は、車載カメラからの障害物検出に関して記述したが、例えば、移動ロボットの自律走行にも適用することが可能であり、本手法は車載カメラからの障害物検出に限定されるものではない。
【００９７】
（変更例９）
ステレオカメラを車の前方に設置し、前方の障害物を検出する場合について記述したが、車の側方や後方に設置して、各々の方向の障害物を検出することも可能である。
【００９８】
（変更例１０）
衝突時間計測部５は、計算した障害物に衝突するまでの時間が小さい場合には自車ドライバーに警報を発したり、衝突時間の大小に応じて音量や音質の異なる警報を発することも可能である。
【００９９】
（変更例１１）
車載ステレオカメラのカメラパラメータは走行中に変化しないものとしたが、変化する場合には道路平面拘束を更新すれば同様に障害物を検出し、その衝突時間を計算することができる。
【０１００】
その他、本発明の要旨を逸脱しない範囲で変形を実施できる。
【０１０１】
【発明の効果】
本発明であると複数台のカメラで取得した複数の画像を処理して、３次元空間中のある面からの高さの有無により障害物を検出するため、明るさの変動や影の影響を受けず、画像中から障害物を検出することができる。
【０１０２】
また、従来のステレオ視の問題であった、多大な時間と労力を要するカメラキャリブレーションの作業と、計算コストの高い奥行き探索（対応探索）処理が不要なため、実用的効果は多大である。
【図面の簡単な説明】
【図１】ステレオ視の対応探索を説明するための図である。
【図２】本実施例の自車の説明図である。
【図３】本実施例の障害物検出装置のブロック図である。
【図４】自車座標系を説明するための図である。
【図５】特徴抽出部３における直線抽出処理を説明するための図である。
【図６】検出部４における障害物検出処理を説明するための図である。
【図７】奥行きを説明するための図である。
【図８】衝突時間を説明するための図（その１）である。
【図９】衝突時間を説明するための図（その２）である。
【図１０】検出部４の変更例のブロック図である。
【図１１】ステレオ画像である。
【図１２】右画像とその変換画像を説明するための図である。
【図１３】左画像と右画像の変換画像を説明するための図である。
【符号の説明】
１ 画像入力部
２ 画像蓄積部
３ 特徴抽出部
４ 検出部
５ 衝突時間計算部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an obstacle detection device for detecting obstacles existing on a road, such as a preceding vehicle, a pedestrian, a parked vehicle, etc., with a vehicle-mounted camera, for example, in order to support safe driving of an automobile, its method and recording medium About.
[0002]
[Prior art]
A technique for detecting obstacles on a road such as a pedestrian, another walking vehicle, and a parked vehicle using a sensor mounted on a car is a very important technique for preventing a traffic accident.
[0003]
Obstacle detection technology using this in-vehicle sensor is broadly classified into those using lasers and ultrasonic waves and those using TV cameras.
[0004]
Those using a laser are expensive, and those using an ultrasonic wave have a low resolution of the ultrasonic wave, so that there is a problem in obstacle detection accuracy.
[0005]
On the other hand, the TV camera is relatively inexpensive and is suitable for obstacle detection in terms of resolution, measurement accuracy, and measurement range. The TV camera can also detect the traveling lane.
[0006]
When using this TV camera, there are a method using one camera and a method using a plurality of cameras (stereo cameras).
[0007]
In the method using one camera, a road area and an obstacle area are separated from one image captured by the camera using information such as luminance, color, texture (pattern), and the like.
[0008]
For example, a medium luminance area with low accuracy in the image, that is, a gray area is extracted as a road area, an area with less texture is extracted as a road area, and the other areas are defined as obstacle areas.
[0009]
However, since there are many obstacles having luminance, color, or texture similar to roads, it is difficult to separate an obstacle area and a road area under this general method.
[0010]
On the other hand, the method using a plurality of cameras detects an obstacle using the three-dimensional information as a clue.
[0011]
A technique for obtaining three-dimensional information of a target scene using a plurality of cameras is generally called stereo vision. Stereo vision refers to, for example, arranging two cameras on the left and right sides, associating the same point in the three-dimensional space between the left and right images, and obtaining the three-dimensional position of the point in the manner of triangulation. If the height of each camera from the road plane, the inclination with respect to the road plane, and the like are obtained in advance, the height from the road plane at an arbitrary point in the image can be obtained by stereo viewing. Therefore, the obstacle area and the road area can be separated based on the presence or absence of the height from the road plane.
[0012]
In the method using one camera, as described above, it is difficult to detect an area having luminance, color, and texture similar to a road as an obstacle. Since the obstacle is detected using the height of the object as a clue, it is possible to detect the obstacle in a more general scene.
[0013]
However, normal stereo vision is a technique for obtaining the distance from a stereo camera at an arbitrary point on the image. For this purpose, the distance and orientation of a plurality of cameras, the focal length of the camera lens, the image center, etc. It is necessary to obtain a parameter (camera parameter). The operation for obtaining these parameters is called calibration (camera calibration).
[0014]
Calibration prepares a large number of sample points with known three-dimensional positions, calculates the projection position on the image, and calculates the camera parameters by combining the relational expressions established between the three-dimensional position and the projection position of each sample point. It is work to do.
[0015]
However, this calibration requires a lot of time and labor, and is one of the causes that hinder the practical use of obstacle detection by stereo vision.
[0016]
Further, in normal stereo vision, it is necessary to associate the same point in the three-dimensional space between the left and right images.
[0017]
This association process is performed as shown in FIG. A corresponding point on the right image of a certain point on the left screen is on a line called an epipolar line shown in FIG.
[0018]
For example, when two cameras are arranged in parallel, the epipolar line coincides with the scanning line. A point having the most similar luminance pattern on the epipolar line is taken as a corresponding point.
[0019]
However, since this association is basically performed by search calculation, there is a problem that the calculation cost is extremely high.
[0020]
[Problems to be solved by the invention]
As described above, obstacle detection devices can be broadly classified into those using lasers and ultrasonic waves and those using TV cameras. Obstacle detection devices using lasers and ultrasonic waves are expensive and have high measurement accuracy. There was a problem of being low.
[0021]
Moreover, the obstacle detection apparatus using a TV camera has a problem that the use environment is limited, calibration that requires a lot of time and labor, and correspondence search of left and right images that require high calculation costs are necessary. there were.
[0022]
Therefore, the present invention has been made in view of the above circumstances, and uses a stereo camera capable of detecting obstacles with high accuracy and relatively low price in a general environment, and has been a problem of conventional stereo vision calibration. Obstacles that detect obstacles on the road plane at high speed without performing correspondence search processing, and measure the time from the movement of the obstacles on the image until the obstacles collide with the vehicle A detection apparatus, a method thereof, and a recording medium are provided.
[0023]
[Means for Solving the Problems]
The present invention includes: an image input unit that inputs and stores at least two images having different imaging points; (1) one image among a plurality of images stored by the image input unit is set as a reference image; set a reference image and another image, (3) the upper surfaces of the three-dimensional space, stationary relative to the imaging point of the reference image or the moving object, endless movement of the object Feature extraction means for extracting the position (u0, v0) of the projection point U on the reference image corresponding to the far point;
The corresponding point P ′ on the reference image when an arbitrary point P on the reference image is assumed to be on the surface is calculated, and does not exist on the surface from the luminance difference between these points P and P ′. A detection unit that detects a faulty area including a point, a position (u, v) on the reference image at a time t with respect to a point on a boundary line between the faulty area and the surface detected by the detection unit, and a time Based on the position (u ′, v ′) on the reference image at (t−dt) and the position (u0, v0) of the projection point U extracted by the feature extraction means, A collision time calculation means for calculating a collision time tc until the image capture point coincides with the image capturing point. The collision time calculation means includes γ = (v−v0) / (v′−v0) and tc = dt / The obstacle detection device is characterized in that the collision time tc is obtained from (γ−1) .
[0030]
In the present invention, a plurality of images acquired by a plurality of cameras are processed to detect an obstacle based on the presence or absence of a height from a certain surface in a three-dimensional space. The obstacle can be detected from the image without receiving it.
[0031]
In addition, since there is no need for a camera calibration operation that requires a lot of time and labor and a depth search (correspondence search) process with a high calculation cost, which is a problem of conventional stereo vision, the practical effect is great.
[0032]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
[0033]
In the present embodiment, as shown in FIG. 2, two left and right cameras are mounted on a vehicle (hereinafter referred to as “own vehicle”), and a stereo image acquired by the stereo camera is processed, so that a pedestrian or a preceding vehicle is processed. It is assumed that an obstacle such as a car, a parked vehicle, or the like existing on the road plane is detected and the time until the obstacle collides with the own vehicle is calculated.
[0034]
FIG. 3 shows a schematic configuration of the obstacle detection apparatus in the present embodiment, which is composed of an image input unit 1, an image storage unit 2, a feature extraction unit 3, a detection unit 4, and a collision time calculation unit 5. Yes.
[0035]
In the obstacle detection device, stereo images are acquired by two cameras whose position / posture, focal length of the lens, image center, etc. are unknown, and left and right images of points on the road plane that have been obtained in advance at rest Using a formula that expresses the relationship between the projected positions on the screen (hereinafter referred to as the “road plane constraint formula”), it is determined whether each point on the image has a height from the road plane, Separate the object area from the road area. Furthermore, the time until the obstacle collides with the host vehicle is calculated from the locus on the obstacle image.
[0036]
FIG. 4 shows the host vehicle coordinate system.
[0037]
In the own vehicle coordinate system, the traveling (front-rear) direction of the own vehicle is the Y-axis direction, the left and right and up-and-down directions are the X and Z axes, respectively, and the road plane is the XY plane. In this embodiment, it is assumed that both the vehicle and the obstacle travel along two white lines (straight lines l and l ′) at both ends of the road.
[0038]
(Image input unit 1)
The image input unit 1 acquires two images using two left and right TV cameras. Although the position and orientation of these two cameras with respect to the vehicle coordinate system, the focal length of the lens of each camera, and the image center may be unknown, in this embodiment, each camera is fixed to the car and is running. Shall not change.
[0039]
(Image storage unit 2)
The image storage unit 2 stores the two images input by the image input unit 1 in the image memory.
[0040]
(Feature extraction unit 3)
The feature extraction unit 3 uses one of the two images stored by the image storage unit 2 as a reference image (hereinafter, the left image is used as a reference image). As shown in FIG. The straight lines l ₁ and l ₂ are detected.
[0041]
Let u ₀ = (u ₀ , v ₀ ) be the intersection (projection point on an image at an infinite point on two straight lines, hereinafter referred to as “vanishing point”). This straight line detection is performed using, for example, edge extraction processing and Hough transformation.
[0042]
(Detector 4)
The detection unit 4 detects an obstacle using a constraint (hereinafter referred to as “road plane constraint”) established between points projected on the left and right images of points on the road plane. The road plane constraint will be described below.
[0043]
Any point on the road plane (XY plane) (X, Y) of the projected points to the left and right images for each _(u, v), when the _{(u r, v} r),
[Expression 1]

The following relational expression holds.
[0044]
h = (h ₁₁ , h ₁₂ ,..., t ₃ ) ^T , h ′ = (h ′ ₁₁ , h ′ ₁₂ ,..., t ′ ₃ ) ^T is the three-dimensional position of each camera relative to its own vehicle coordinate system. These are parameters relating to the posture, the focal length of the lens mounted on each camera, and the image center.
[0045]
If X and Y are deleted from the equations (1) and (2),
[Expression 2]

It becomes.
[0046]
Here, H = (H ₁₁ , H ₁₂ ,... H ₃₃ ) ^T is a constant represented by h 1 and h ′.
[0047]
From this equation, the corresponding point on the right image when the point (u, v) on the left image is assumed to be a point on the road surface can be obtained.
[0048]
In other words, assuming that the point (u, v) on the left image is on the road plane, the corresponding point (u _r , v _r ) on the right image is determined by the above equation. This equation is called a road plane constraint equation.
[0049]
It should be noted that the parameter H = (H ₁₁ , H ₁₂ ,... H ₃₃ ) ^T in this equation is obtained in advance when the vehicle is stationary.
[0050]
The method will be described below.
[0051]
First, N (≧ 4) feature points on the road surface (intersections of straight lines drawn on the road plane, corner points of paint, etc.) are extracted from the left image.
[0052]
Next, a corresponding point on the right image of each extracted feature point is obtained. Note that this feature point extraction and association processing may be performed by pointing a point on the image with a mouse. Each of these N sets of correspondences satisfies Equation (3), and thus 2N simultaneous equations are obtained.
[0053]
The parameter H can be obtained by solving this simultaneous equation for H.
[0054]
A method for detecting an obstacle using this road plane constraint formula will be described below.
[0055]
The luminance of an arbitrary point A (u, v) in the left image is I _L (u, v), and the corresponding point A ′ (u _r , v on the right image when it is assumed that the point A exists on the road plane. the _r) obtained from the equation (3), to the luminance _I R _(u r, and _{v r).}
[0056]
If the point (u, v) actually exists on the road plane, the points A and A ′ are a correct pair of corresponding points (the points A and A ′ are projection points of the same point on the road surface). Specifically, the brightness of the point A and the point A ′ is the same. Therefore,
[Equation 3]

In other words, Diff = 0.
[0057]
Conversely, Diff = 0 unless the point A (u, v) and _{A '(u r, v r} ) becomes not a correct corresponding point pairs, the point (u, v) is present on the road plane It is a point that does not work, that is, a point on the obstacle.
[0058]
When these series of processes are performed on all points on the reference image, the obstacle region can be detected.
[0059]
At this time, as a criterion for determining whether or not the object is an obstacle, a point where Diff> Thr may be determined to belong to the obstacle region in consideration of a certain amount of error.
[0060]
Here, Thr is a preset threshold value.
[0061]
For example, an obstacle region as shown in the lower part of the figure is detected from the stereo image shown in the upper part of FIG.
[0062]
(Collision time measurement unit 5)
The collision time measuring unit 5 is a trajectory (obstacle) of an obstacle on one of the stereo images (this is referred to as a reference image, and in the present embodiment, the left image is used as the reference image as described above). The time until the host vehicle collides with the obstacle (collision time) is measured.
[0063]
First, the collision time will be described.
[0064]
Let D be the denominator of equation (1). That is,
D = h ₃₁ X + h ₃₂ Y + t ₃ (5)
It is.
[0065]
As shown in FIG. 7, D is the distance in the optical axis direction between the contact T (X, Y) with the road plane of the obstacle and the viewpoint C of the reference camera, that is, the depth.
[0066]
Depths of the contact points T ′ and T with the road plane of the obstacle at time t-dt and t are D ′ and D, respectively. Here, the obstacle is defined as the time t-dt, collision time the time until the depth is 0 when continuing the movement between t t _c relative to the vehicle coordinate system.
[0067]
FIG. 8 shows the time change of the depth.
[0068]
Since the time from the current time t until the depth becomes 0 is the collision time t _c , t _c is expressed as shown in FIG. From the similarity of two triangles indicated by diagonal lines,
[Expression 4]

However, γ = D ′ / D.
[0069]
That is, the collision time t _c can be obtained from the depth ratio γ at two different times.
[0070]
A method for calculating the collision time from the position of the obstacle area at each time obtained by the detection unit 4 will be described below.
[0071]
Assume that an obstacle area is detected on the reference image as shown in FIG. 9 at the current time t and a time t-dt before that, and the line at the lowest end of the obstacle area (obstacles) detected at the time t. Arbitrary one point on the boundary line between the area and the road area) is set to u = (u, v). The position u ′ = (u ′, v ′) at the time point t = dt of the point u = (u, v) is a straight line connecting the points u ₀ , u and the boundary between the obstacle region and the road region at the time t-dt. It can be determined as the intersection of lines.
[0072]
Further, the positions of the obstacles on the road plane at the time t-dt, t are set as X ′ = (X, Y + dY) and X = (X, Y), respectively.
[0073]
In this embodiment, it is assumed that the own vehicle and the obstacle move along the Y axis (the traveling directions coincide), so the X coordinate of the position of the obstacle at each time is the same. The depth to X 'is
[Equation 5]

Here, since the projection point of the point at infinity on the straight lines l and l ′ parallel to the Y axis is u ₀ = (u ₀ , v ₀ ), Y → ∞ in the equation (1)
[Formula 6]

By substituting γ into equation (7), the collision time of the obstacle can be calculated.
[0074]
That is, as shown in FIG. 9, the collision time of the obstacle moving on the road plane is determined by the position u ′, u on the image at the time t−dt, t of the contact point of the obstacle with the road plane and the straight line l. , L ′ can be calculated only from the vanishing point u ₀ and does not depend on the position and orientation of the camera with respect to the vehicle coordinate system.
[0075]
As described above, an obstacle on the road plane is detected from the in-vehicle stereo camera without using camera parameters and without performing a depth search, and further, the time until the obstacle collides with the own vehicle. Can be calculated.
[0076]
(Effect of Example)
As described above, in the present embodiment, the stereo image acquired by the stereo camera is processed, and the obstacle is detected based on the presence or absence of the height from the road plane. Obstacles such as preceding cars and pedestrians can be detected from the image. In addition, since there is no need for a camera calibration operation that requires a lot of time and labor and a depth search (correspondence search) process with a high calculation cost, which is a problem of conventional stereo vision, the practical effect is great.
[0077]
(Modification 1)
In the present embodiment, the image input unit 1 inputs two images by arranging two TV cameras on the left and right, but these cameras may be arranged vertically.
[0078]
Three or more cameras may be arranged.
[0079]
(Modification 2)
The feature extraction unit 3 has been described with respect to extracting two lines on the road plane. However, three or more lines may be extracted. If the direction vector is the same, a line that is not on the road plane may be extracted. For example, a line on the guard rail may be extracted.
[0080]
(Modification 3)
The detection unit 4 can further have the configuration shown in FIG.
[0081]
Here, the image conversion unit 4-1 and the difference calculation unit 4-2 are used.
[0082]
The image conversion unit 4-1 converts the right image according to the following procedure. In general, an image can be expressed as a function f (u, v) in which a point (u, v) on the image is a variable and a luminance value is defined for each point. In the following, the image is expressed in this way.
[0083]
If a stereo image as shown in FIG. 11 is input, g (u, v) is obtained for the right image, and g ′ (u, v) is obtained for the converted image. G ′ (u, v) is obtained as follows.
[0084]
[Expression 7]

However, (u _r , v _r ) is obtained from equation (3). g ′ (u, v) is an image obtained by the left camera when it is assumed that all points on the image g (u, v) exist on the road plane.
[0085]
For example, a converted image as shown in FIG. 12 is obtained from the right image in FIG.
[0086]
As shown in FIG. 13, the projected point of a point on the road plane is the same in the left image and the converted image, whereas it is not on the road plane, that is, on an obstacle (in this case, a preceding vehicle). These points are projected at different positions depending on the height from the road.
[0087]
Therefore, an obstacle on the road plane is detected by taking the difference between the left image and the converted image. That is, if the left image is f (u, v),
[Equation 8]

As such, it is determined that the point (u, v) where Diff ′> Thr (Thr is a preset threshold value) does not belong to the obstacle region, if Diff ′ = 0 is not taken into consideration.
[0088]
(Modification 4)
The detection unit 4 detects the difference between the two images by taking the difference between the pixels, but sets a (2w + 1) × (2w + 1) window for each point, and averages and variances of luminance values in the window, A difference may be detected from normalized cross-correlation or the like.
[0089]
The normalized cross-correlation C of the point (u, v) of the two images F (u, v) and G (u, v) can be calculated from the following equation.
[0090]
[Equation 9]

Here, N = (2w + 1) × (2w + 1), a ₁ , a ₂ are the average luminance in the windows of the two images, and σ ² ₁ , σ ² ₂ are the variances of the luminance in the two image windows. Yes, -1 ≦ C ≦ 1.
[0091]
In this case, it is determined that the point (u, v) where C <Thr belongs to the obstacle region. Here, Thr (≦ 1) is a preset threshold value.
[0092]
(Modification 5)
In this embodiment, two white lines at both ends of the road are extracted as straight lines. However, when the road is curved, the white lines are curved.
[0093]
In this case, if a white line is extracted as a curve, an obstacle can be detected in the same manner.
[0094]
(Modification 6)
Although the description has been made on the assumption that a road surface is a plane, an obstacle can be detected even in the case of a curved surface, as in the case of a plane.
[0095]
(Modification 7)
Although the movement parallel to the lane is assumed with respect to the own vehicle and the obstacle, the present invention is also applicable when the movement is not parallel (for example, when changing the lane).
[0096]
(Modification 8)
Although the present embodiment has been described with respect to obstacle detection from an in-vehicle camera, for example, it can be applied to autonomous traveling of a mobile robot, and this method is not limited to the detection of an obstacle from an in-vehicle camera. Absent.
[0097]
(Modification 9)
Although the case where the stereo camera is installed in front of the vehicle and the obstacle in the front is detected has been described, it is also possible to install the stereo camera in the side or rear of the vehicle and detect the obstacle in each direction.
[0098]
(Modification 10)
The collision time measurement unit 5 can issue a warning to the driver of the vehicle when the time until the collision with the calculated obstacle is small, or can issue a warning with different volume and sound quality depending on the magnitude of the collision time. is there.
[0099]
(Modification 11)
The camera parameters of the in-vehicle stereo camera are assumed not to change during traveling. However, if the vehicle parameters change, the obstacle can be detected in the same manner by updating the road plane constraint, and the collision time can be calculated.
[0100]
In addition, modifications can be made without departing from the scope of the present invention.
[0101]
【The invention's effect】
In the present invention, a plurality of images acquired by a plurality of cameras are processed to detect an obstacle based on the presence or absence of a height from a certain surface in a three-dimensional space. The obstacle can be detected from the image without receiving it.
[0102]
In addition, since there is no need for a camera calibration operation that requires a lot of time and labor and a depth search (correspondence search) process with a high calculation cost, which is a problem of conventional stereo vision, the practical effect is great.
[Brief description of the drawings]
FIG. 1 is a diagram for explaining correspondence search for stereo vision;
FIG. 2 is an explanatory diagram of the own vehicle according to the present embodiment.
FIG. 3 is a block diagram of an obstacle detection apparatus according to the present embodiment.
FIG. 4 is a diagram for explaining a host vehicle coordinate system;
FIG. 5 is a diagram for explaining straight line extraction processing in a feature extraction unit 3;
FIG. 6 is a diagram for explaining an obstacle detection process in a detection unit 4;
FIG. 7 is a diagram for explaining depth.
FIG. 8 is a diagram (part 1) for explaining a collision time;
FIG. 9 is a diagram (part 2) for explaining the collision time;
FIG. 10 is a block diagram of a modified example of the detection unit 4;
FIG. 11 is a stereo image.
FIG. 12 is a diagram for explaining a right image and its converted image.
FIG. 13 is a diagram for explaining a converted image of a left image and a right image.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Image input part 2 Image storage part 3 Feature extraction part 4 Detection part 5 Collision time calculation part

Claims (6)

Image input means for inputting and storing at least two images having different imaging points;
(1) One image among a plurality of images accumulated by the image input means is set as a reference image , (2) another image is set as a reference image, and (3) a certain surface in a three-dimensional space , stationary relative to the imaging point of the reference image, or extracting the moving object, the position of the projected point U on the reference image corresponding to the point at infinity of the motion of the object of (u0, v0) Feature extraction means for
The corresponding point P ′ on the reference image when an arbitrary point P on the reference image is assumed to be on the surface is calculated, and does not exist on the surface from the luminance difference between these points P and P ′. Detecting means for detecting a faulty region including a point;
Regarding the point on the boundary line between the obstacle area and the surface detected by the detecting means, the position (u, v) on the reference image at time t and the position on the reference image at time (t-dt) ( u ′, v ′) and the position (u0, v0) of the projection point U extracted by the feature extraction means, the collision time tc until the obstacle area coincides with the imaging point of the reference image is calculated . A collision time calculation means for calculating,
The obstacle detection device, wherein the collision time calculation means obtains a collision time tc from γ = (v−v0) / (v′−v0) and tc = dt / (γ−1) . The feature extraction means is
Extracting a plurality of lines that match the direction of movement of the object from the reference image;
Obstacle detection device according to claim 1, wherein the intersections of a plurality of lines and their extracts, characterized in that said projection point U. The mutual position in the three-dimensional space corresponding to the imaging points of the plurality of images, the attitude of the camera that captures the image, the focal length of each camera that captures each image , and the image center are unknown. The obstacle detection device according to claim 1, wherein: The detection means includes
Calculating a corresponding point P ′ on the reference image when an arbitrary point P on the reference image is assumed to be on the surface;
Obstacle detection device according to claim 1, wherein the detecting a point that is not on the surface when the similarity of the brightness of the surroundings is less than the threshold value of the point P and the point P '. An image input step of inputting and storing at least two images having different imaging points;
(1) One image among the plurality of images accumulated in the image input step is set as a reference image , (2) another image is set as a reference image, and (3) a certain surface in the three-dimensional space , stationary relative to the imaging point of the reference image, or extracting the moving object, the position of the projected point U on the reference image corresponding to the point at infinity of the motion of the object of (u0, v0) A feature extraction step,
The corresponding point P ′ on the reference image when an arbitrary point P on the reference image is assumed to be on the surface is calculated, and does not exist on the surface from the luminance difference between these points P and P ′. A detecting step for detecting a faulty region including a point;
Regarding the point on the boundary line between the obstacle area and the surface detected by the detection step, the position (u, v) on the reference image at time t and the position on the reference image at time (t-dt) ( u ′, v ′) and the position (u0, v0) of the projection point U extracted by the feature extraction step, the collision time tc until the obstacle area coincides with the imaging point of the reference image is calculated . A collision time calculation step for calculating,
The obstacle detection method characterized in that the collision time calculation step obtains a collision time tc from γ = (v−v0) / (v′−v0) and tc = dt / (γ− 1) . An image input function for inputting and storing at least two images having different imaging points;
(1) One image among a plurality of images accumulated by the image input function is set as a reference image , (2) another image is set as a reference image, and (3) a certain surface in a three-dimensional space , stationary relative to the imaging point of the reference image, or extracting the moving object, the position of the projected point U on the reference image corresponding to the point at infinity of the motion of the object of (u0, v0) A feature extraction function,
The corresponding point P ′ on the reference image when an arbitrary point P on the reference image is assumed to be on the surface is calculated, and does not exist on the surface from the luminance difference between these points P and P ′. A detection function for detecting a faulty area including a point;
Regarding the point on the boundary line between the obstacle area and the surface detected by the detection function, the position (u, v) on the reference image at time t and the position (t-dt) on the reference image ( u ′, v ′) and the position (u0, v0) of the projection point U extracted by the feature extraction function, the collision time tc until the obstacle area coincides with the imaging point of the reference image is calculated . Realize the collision time calculation function to calculate,
The obstacle detection method recording medium in which the collision time calculation function records a program for obtaining the collision time tc from γ = (v−v0) / (v′−v0) and tc = dt / (γ−1) .

Images