JP4344860B2 - Road plan area and obstacle detection method using stereo image - Google Patents

Description

  In order to realize driver assistance for safe driving of a car and automatic driving of an autonomous mobile vehicle, the present invention uses a vehicle-mounted stereo camera on a road plane area and a road such as a preceding vehicle, an oncoming vehicle, a parked vehicle, and a pedestrian. The present invention relates to a road plane region and an obstacle detection method using a stereo image for detecting all existing obstacles.

  Conventionally, laser radar, ultrasonic waves, and millimeter wave radar are used for visually recognizing a driving environment, especially for detecting a region where the vehicle can travel, and for detecting obstacles existing in the driving environment in guiding an autonomous mobile vehicle. It can be roughly divided into a method and a method using an image.

  In the detection method using a laser radar or a millimeter wave radar, there is a problem that the apparatus is generally expensive and sufficient spatial resolution cannot be obtained. In the detection method using an ultrasonic sensor, there is a problem that it is difficult to measure far away and the spatial resolution is low.

  In addition, the method using an image can be divided into a method using a single eye and a method using a compound eye. Conventionally, most of the methods using an image use a single eye, that is, an image obtained from one viewpoint, and are mainly used in a developed environment such as an expressway, and white lines (for example, separation lines) And the center line) are detected to detect the travel area. However, in general roads and parking lots where the presence of white lines etc. is not guaranteed and the color and pattern of the road surface are various, by the monocular image method, that is, only from the density pattern projected on the monocular image, There is a problem that it is difficult to stably distinguish the travelable area and the obstacle.

  On the other hand, in the image method using a compound eye, that is, the detection method using a stereo image, since the three-dimensional structure of the environment can be used in principle, there is a possibility that the driving environment can be recognized more stably. In particular, since the travelable area can be regarded as almost a plane in the space, the image is subjected to two-dimensional projective transformation, and it is detected whether it is a planar area or an obstacle from the overlapping state of the images. For example, as shown in Non-Patent Literature 1, Non-Patent Literature 2, Non-Patent Literature 3, and Non-Patent Literature 4, there has already been proposed.

However, in the conventional detection methods shown in Non-Patent Document 1, Non-Patent Document 2, Non-Patent Document 3, and Non-Patent Document 4, calibration is performed in advance on the assumption that the positional relationship between the camera and the road plane is always constant. By performing the same fixed two-dimensional projective transformation, the plane area of the road and the obstacle are detected. However, since the traveling surface (road surface) may be inclined or the vehicle body may be inclined with movement, the assumption that the positional relationship between the in-vehicle camera and the road plane is always constant may not actually hold. Often. Therefore, with the conventional detection method using the same fixed two-dimensional projective transformation, it is extremely difficult to cope with the traveling surface (road surface) tilting or the vehicle body tilting with movement. There was a problem.
Luon cue. -Tong (Luong, Q.-T), Weber J (Weber, J), (Koller, D), Malik Jay (Malik, J), "Integrated Stereo Based Approach Too"・ Automatic Vehicle Guidance (An Integrated stereo-based approach to automatic vehicle guidance), 1995, Proc. ICCV, p.52-57 Onoguchi, K, Takeda, N, Watanabe, M, "Planar Projection Stereopsis Method for Road Extraction (Planar Projection Stereopsis Method for Road Extraction), 1995, Procedure. I.R.O.S. (Proc.IROS), p. 249-256 Co-authored by Storjohann, K, Zielke, T, Mallot, H, Vonseelen, W, “Visual Obstacle Detection for Automata Visual Obstacle Detection for Automatically Guided Vehicles, 1990, Proc. ICRA, p. 716-766 Xie, M, “Matching free stereo vision for detecting obstacles on a ground, machine vision and application” plane, Machine Vision and Application), 1996, Volume 9. Number 1 (Vol. 9, No. 1), p. 9-13 Heung-Yeung Shum, Richard Szeliski, “Panoramic Image Mosaics” Technical Report “Technical report”, 1997, MSR -T-R-97-23, Microsoft Research (MSR-TR-97-23, Microsoft Research) Co-authored by ODFaugeras and F.Lustman, “Motion and Structure from Motion in Peacewise Planner Environment” (“Motion and Structure from Motion In A Piecewise Planar Environment "), 1998, Proc. International Journal of Pattern Recognition and Artificial Intelligence, Volume 1.2, Number 3 (Proc. International Journal of Pattern Recognition and Artificial Intelligence, Vol. .2, No. 3), p458-508

  As described above, the road plane area and the obstacle detection method can be broadly classified into those using laser radar, ultrasonic waves and millimeter wave radar and those using images. Detection methods using laser radar, ultrasonic waves, and millimeter wave radar have problems that the apparatus is expensive, the measurement accuracy is low, and the spatial resolution is low. In addition, detection methods using images are limited to use environments such as expressways, and cannot cope with vibrations during driving and slopes of road surfaces. However, there was a problem that the measurement accuracy deteriorated remarkably.

  The present invention has been made under the circumstances described above, and an object of the present invention is to deal with camera vibrations caused by road surface inclination or vehicle running using only image information obtained from an in-vehicle stereo camera. A stereo image that can dynamically calculate the area that can travel in the real space of the vehicle, and calculate and present the distance and relative speed to the obstacle in each direction as seen from the vehicle. The object is to provide a road plane region and an obstacle detection method used.

  The present invention relates to a road plane region and an obstacle detection method using a stereo image, and the object of the present invention is a stereo moving image composed of a standard image and a reference image, which is taken by an imaging means mounted on an automobile. A road plane area and obstacle detection method using a stereo image that can detect a road plane area that can be traveled and all obstacles existing on the road surface using only A first step of dynamically estimating a transformation matrix, a second step of extracting a plane area that can be traveled using the projection transformation matrix obtained in the first step, and a first step A third step of calculating the inclination of the road surface by decomposing the projective transformation matrix, and a virtual view of the road surface viewed from above based on the inclination of the road surface calculated in the third step A fourth step of generating a shadow plane image, and a fifth step of calculating the travelable plane area and the position and direction of the vehicle on the virtual projection plane image generated in the fourth step; Is achieved by having

  In addition, the present invention provides a sixth step of calculating a direction-specific distance from the vehicle to the obstacle from the travelable plane area on the virtual projection plane image, and on the virtual projection plane image. And a seventh step of calculating a relative speed for each direction from the vehicle to the obstacle from the travelable plane area, or by superimposing the travelable plane area on the virtual projection plane image. And further including an eighth step of displaying the direction-specific distance and the direction-specific relative velocity information. Alternatively, in the first step, the reference image and the reference image are subjected to LOG filter processing. After performing the histogram flattening process, a projection transformation matrix corresponding to the road surface between the stereo images is dynamically estimated by an area-based method, and the second step is performed. Then, the reference image is projectively transformed by the projective transformation matrix, a difference image between the base image and the projective transformed reference image is obtained, a smoothing filter is applied to the difference image, and then a threshold value is used. A binarized image is obtained by using the estimation result of the previous time plane area in the binary image and taking the textureless area into consideration, thereby extracting a plane area that can be traveled. In the step, the road plane attitude parameter indicating the inclination of the road plane is a distance from the reference camera optical center to the road plane and a normal vector of the road plane. In the fourth step, the road plane attitude parameter Is more effectively achieved by projectively transforming the reference image to generate a virtual projection plane image parallel to the road plane.

  First, according to the road plane area and obstacle detection method using stereo images according to the present invention, a CCD camera is used as an image sensor that is an imaging means for stereo images. Excellent effects such as achievement of

  Next, according to the road plane area and the obstacle detection method using the stereo image according to the present invention, it is possible to dynamically obtain the stable driving area and the distance and relative speed according to the direction from the vehicle to the obstacle. Because it is possible, based on the required travelable area and the distance and relative speed by direction from the vehicle to the obstacle, it enables driver assistance such as warning of collision risk, collision avoidance etc. and automatic driving of the vehicle etc. Excellent effect.

  Furthermore, in the road plane area and obstacle detection method using the stereo image according to the present invention, it is very easy to install and calibrate the camera, so that the present invention is very suitable for practical use. There is no doubt.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the present embodiment, left and right stereo cameras are mounted on a vehicle, and vibrations and roads caused by the travel of a vehicle equipped with the stereo camera (hereinafter, the vehicle equipped with the stereo camera is referred to as a vehicle or a vehicle). It is assumed that all obstacles existing on the road surface such as a pedestrian, a parked vehicle, an oncoming vehicle, and a preceding vehicle are detected under a condition in which the inclination of the surface is changed.

  In the present embodiment, the right stereo camera is the reference camera, and the left stereo camera is the reference camera. Then, a stereo image taken by the reference camera is set as a reference image, and a stereo image taken by the reference camera is set as a reference image. In the present invention, the plane area and the obstacle that can be traveled are detected using only the stereo image (that is, the standard image and the reference image), and therefore the camera that captures the stereo image (that is, the standard camera). In some cases, the left stereo camera may be the reference camera, and the right stereo camera may be the reference camera.

  Incidentally, the reference camera is mounted on the left side of the vehicle and the reference camera is mounted on the right side of the vehicle as a stereo camera that captures a stereo image in steps S100 and S110 described below. As a stereo camera for capturing a stereo image in other steps, the reference camera is mounted on the right side of the automobile, and the reference camera is mounted on the left side of the automobile.

  First, as a point of interest of the present invention, by using two in-vehicle stereo cameras, an area corresponding to a plane of space and an orientation of the plane are obtained from stereo images captured by the stereo cameras. A travelable area in a coordinate system corresponding to the space is dynamically obtained, and the distance to the obstacle and the relative speed for each direction of the vehicle are calculated and presented.

  That is, the road plane area and the obstacle detection method using the stereo image according to the present invention are a flat area where the vehicle can travel (hereinafter referred to as a travelable area) using image information obtained from two in-vehicle stereo cameras. The distance and relative speed to the obstacle in each direction of the vehicle are calculated and presented by mapping the obtained travelable area to the coordinate system corresponding to the real space. This is a method for enabling driver assistance such as warning of collision risk of vehicles, avoidance of collision, automatic driving of vehicles, and the like.

  In the present invention, it is desirable to use a CCD camera as a stereo camera mounted on an automobile, but the present invention is not limited to this. In the following embodiments, it is assumed that a CCD camera is used as a stereo camera.

  FIG. 1 is a flowchart showing the overall flow of the present invention. As shown in FIG. 1, in the road plane area and obstacle detection method using the stereo image according to the present invention, first, a projective transformation matrix for the road surface is dynamically estimated (step S100). Next, using the projective transformation matrix obtained in step S100, a road surface plane region (hereinafter, the road surface plane region is referred to as a road plane region and sometimes referred to as a travelable plane region) is extracted. (Step S110). Further, the slope of the road surface is calculated by decomposing the projective transformation matrix obtained in step S100 (step S120). Based on the inclination of the road surface calculated in step S120, an image of the road surface viewed from above (in the present invention, this image is referred to as a virtual projection plane (VPP) image) is generated (step S130). . Then, on the virtual projection plane image (VPP image) generated in step S130, the plane area where the vehicle can travel and the position and direction of the host vehicle are presented (step S140). Next, the distance in each direction to the farthest part from the vehicle in the road plane area on the virtual projection plane image, that is, the obstacle is calculated (step S150). Similarly, the relative speed in each direction to the farthest part from the vehicle in the road plane area on the virtual projection plane image, that is, the obstacle is calculated (step S160). Finally, a plane area that can be traveled is displayed superimposed on the virtual projection plane image (VPP image), and the direction-specific distance and direction-specific relative velocity information calculated in steps S150 and S160 is displayed (step S170). .

  The invention will now be described in more detail step by step. First, the principle of plane projection and two-dimensional projective transformation used in the present invention will be described.

As shown in FIG. 2, when a point M on a plane in the space is projected onto two stereo images, m _b is a homogeneous coordinate on the standard image I _b in the stereo image, and the reference image I When the homogeneous coordinates on _r and m _r, respectively, as shown in formula 1 below, be related by two-dimensional projective transformation it is well known.

ここで、

は３×３の射影変換行列であり、

は定数倍の不定性を許して等しいものとする。また、図２において、Ｏは基準カメラの光学中心で、Ｏ^´は参照カメラの光学中心で、

は基準カメラ座標系から参照カメラ座標系への回転行列で、

は参照カメラ座標系における基準カメラと参照カメラの並進ベクトルで、ｄは基準カメラと平面の距離で、

は平面の法線ベクトルである。

here,

Is a 3 × 3 projective transformation matrix,

Are equal, allowing a constant multiple of indefiniteness. In FIG. 2, O is the optical center of the reference camera, O ^′ is the optical center of the reference camera,

Is the rotation matrix from the base camera coordinate system to the reference camera coordinate system,

Is the translation vector of the reference camera and reference camera in the reference camera coordinate system, d is the distance between the reference camera and the plane,

Is the normal vector of the plane.

  Therefore, if it can be assumed that the road and other areas that can travel can be considered to be almost flat in the space, by applying an appropriate projective transformation to one of the stereo images, the area of the plane will match. Images can be obtained.

That is, in the present invention in which obstacle detection is performed using plane extraction, when an obstacle is present on the plane, the area is not recognized as a plane, and thus it is possible to detect that the area is not a travelable area. Moreover, obstacles are included in areas other than the plane area, and as a result, the obstacle can be detected by measuring the spread of the plane.

<Step S100> Dynamically Estimate Projection Transformation Matrix for Road Surface First, the difference in brightness between cameras, etc., for stereo images captured and input by two stereo cameras mounted on a car In order to eliminate this, a LOG (Laplacian Of Gaussian) filter is applied, and a histogram flattening process is further performed. FIG. 3 shows an example of the input stereo original image. FIG. 3A shows a standard image, and FIG. 3B shows a reference image. FIG. 4 shows a result image obtained by subjecting the stereo original image shown in FIG. 3 to LOG filter processing and histogram flattening processing. FIG. 4A shows an image after processing of the standard image of FIG. 3A, and FIG. 4B shows an image after processing of the reference image of FIG. 3B.

を最小化する

を、繰り返し最適化により求める。 Next, a projective transformation matrix corresponding to a plane (road surface in the present embodiment) is obtained by the region-based method disclosed in Non-Patent Document 5. That is, the following evaluation function within a certain region R ^I

Minimize

Is obtained by iterative optimization.

但し、Ｉ_b(ｍ)、Ｉ_r(ｍ)は、それぞれ画像位置ｍでの基準画像、参照画像の濃度値を表す。

However, I _b (m) and I _r (m) represent density values of the standard image and the reference image at the image position m, respectively.

と、領域Ｒ^I（本発明では、以後「計算領域」と呼ぶことにする）を必要とする。そこで本発明では、時系列情報を利用し、以下に述べる手順により、その射影変換行列と計算領域を求める。 For the above estimation, a projective transformation matrix as an initial value suitably close to the true value

And region R ^I (in the present invention, hereinafter referred to as “calculation region”). Therefore, in the present invention, the time series information is used, and its projective transformation matrix and calculation area are obtained by the procedure described below.

と、基準画像の時間的に連続する２画像間の射影変換行列

、及び基準画像に対する平面領域Ｒを求めていく。そして、ある時刻ｔの推定の際には、前時刻ｔ−１までの上記推定結果を利用する。以下、図５を参照しながら、その手順を更に詳細に説明する。
（１）まず、連続する基準画像Ｉ_b(ｔ−１)、Ｉ_b(ｔ)間の射影変換行列

を求める。その時、射影変換行列の初期値としては、前時刻に推定された射影変換行列

を、また計算領域としては、Ｉ_b(ｔ−１)に対して前時刻で求められている平面領域Ｒ(ｔ−１)を用いることができる。
（２）次に、（１）で求められた射影変換行列

を用いて、前時刻の平面領域Ｒ(ｔ−１)を変換することにより、現時刻における平面領域の予測値Ｒ^I(ｔ)を求める。
（３）更に、ステレオ画像Ｉ_b(ｔ)、Ｉ_r(ｔ)間の射影変換行列

は、前時刻において推定された

を初期値とし、（２）で求められたＲ^I(ｔ)を計算領域として、求められる。 First, the projection transformation matrix between stereo images at each time

And a projective transformation matrix between two consecutive images of the reference image

And a plane region R with respect to the reference image. Then, when estimating a certain time t, the estimation result up to the previous time t-1 is used. Hereinafter, the procedure will be described in more detail with reference to FIG.
(1) First, a projective transformation matrix between successive reference images I _b (t−1) and I _b (t)

Ask for. At that time, the initial value of the projection transformation matrix is the projection transformation matrix estimated at the previous time.

Further, as the calculation region, the plane region R (t−1) obtained at the previous time with respect to I _b (t−1) can be used.
(2) Next, the projective transformation matrix obtained in (1)

Is used to convert the plane area R (t−1) at the previous time to obtain the predicted value R ^I (t) of the plane area at the current time.
(3) Further, a projective transformation matrix between the stereo images I _b (t) and I _r (t)

Was estimated at the previous time

Is the initial value, and R ^I (t) obtained in (2) is used as the calculation region.

  In the present invention, the projective transformation matrix and the calculation area as the initial value sufficiently close to the true value can be used in the continuous estimation using the time series image by the procedures (1), (2), and (3). Therefore, stable estimation is possible.

は、その都度ステレオ画像から求めることが可能なため、カメラの内部パラメータや２台のカメラ配置、カメラと道路面との位置関係等は知る必要がなく、また、道路面自体が傾いたり、走行中にカーブや道路面の凹凸で車体が傾斜したり、カメラが振動したりすることによっての変化があっても対応することができる。

＜ステップＳ１１０＞求まった射影変換行列を用いて、道路面の平面領域（走行可能な平面領域）を抽出する
ステップＳ１００で求めた射影変換行列を利用して、基準画像中から平面に対応した領域を抽出する。図６に図３のステレオ原画像を用いた一連の処理結果を示す。図６(Ａ)は、ＬＯＧフィルタ処理とヒストグラム平坦化処理を施した基準画像（図４(Ａ)と同じ）である。図６(Ｂ)は、ステップＳ１００で述べた方法により推定した射影変換行列

を用いて、図４(Ｂ)の参照画像を射影変換した結果である。図６(Ａ)の画像と図６(Ｂ)の画像の差分（差の絶対値）を求めたものが図６(Ｃ)である。図６(Ｃ)の画像を見ると、平面（道路面）に対しては、図６(Ａ)、図６(Ｂ)両画像の位置が一致しているため、全体的に黒く、それ以外の部分ではずれているため、白っぽく見える。次に、図６(Ｃ)の差分画像に対し、平均化フィルタをかけた上で２値化したものが図６(Ｄ)である。図６(Ｄ)は、図６(Ａ)と図６(Ｂ)の画像に対し、ＳＡＤ(Sum of Absolute Difference)を用いたマッチングを行っていることに相当する。 In the present invention, the projection transformation matrix and the calculation region estimation procedure described above, the projection transformation matrix

Can be obtained from the stereo image each time, so it is not necessary to know the internal parameters of the camera, the arrangement of the two cameras, the positional relationship between the camera and the road surface, etc. Even if there is a change due to the car body tilting or the camera vibrating due to curves or irregularities on the road surface, it can be dealt with.

<Step S110> Using the obtained projective transformation matrix, a road surface plane area (a plane area where the vehicle can run) is extracted. Using the projective transformation matrix obtained in step S100, an area corresponding to a plane from the reference image. To extract. FIG. 6 shows a series of processing results using the stereo original image of FIG. FIG. 6A is a reference image (same as FIG. 4A) subjected to LOG filter processing and histogram flattening processing. FIG. 6B shows a projective transformation matrix estimated by the method described in step S100.

This is a result of projective transformation of the reference image of FIG. FIG. 6C shows the difference (absolute value) between the image in FIG. 6A and the image in FIG. 6B. Looking at the image in FIG. 6C, the positions of both the images in FIG. 6A and FIG. 6B coincide with the plane (road surface), so the whole image is black. Because it is out of place, it looks whitish. Next, FIG. 6D shows a binary image obtained by applying an averaging filter to the difference image shown in FIG. 6C. FIG. 6D corresponds to performing matching using the SAD (Sum of Absolute Difference) on the images of FIGS. 6A and 6B.

より、現時刻において予想される平面領域が得られる。時系列画像間では平面領域がそれほど急激には変化しないとすれば、図７に示すように、現時刻における平面領域と非平面領域の境界は、前時刻から予想される境界に対してある一定の幅に収まると考えられる。そこで、それ以外の部分に関しては、前時刻の結果より平面領域ないし非平面領域とみなすことができるため、それを図６(Ｄ)のマッチング結果に反映する。さらに、走行可能な平面領域はある程度の大きさを持っていると考えられるため、収縮・膨張処理により、面積が小さな領域や細長い領域を除く処理を施した結果が図６(Ｅ)である。図６(Ｆ)は、抽出された結果が原画像とどう対応しているのかが分かるように、原画像に対し、抽出された平面領域以外の明るさを落して表示したものである。図６(Ｆ)より、抽出された領域が実際の道路面に良く一致していることが分かる。 Here, when the plane area extraction result of the previous time can be used, the result is used as follows. As described in step S100, the projective transformation matrix between the plane region R (t−1) at the previous time and the time series of the reference image.

As a result, a plane area expected at the current time is obtained. If the plane area does not change so rapidly between time-series images, as shown in FIG. 7, the boundary between the plane area and the non-planar area at the current time is constant relative to the boundary expected from the previous time. It is considered to be within the range of. Therefore, other portions can be regarded as a planar region or a non-planar region from the result of the previous time, and this is reflected in the matching result of FIG. Further, since it is considered that the plane area that can be traveled has a certain size, FIG. 6E shows the result of performing the process of excluding the area having a small area or the elongated area by the contraction / expansion process. FIG. 6F shows the original image displayed with reduced brightness other than the extracted plane area so that it can be seen how the extracted result corresponds to the original image. From FIG. 6 (F), it can be seen that the extracted area is in good agreement with the actual road surface.

  On the other hand, the method described above is based on the premise that the position on the image other than the plane is shifted between the base image and the reference image after projective transformation, and as a result, a difference in image density occurs. However, for example, a problem arises when a wall of a building without texture occupies a large area on the image. This is because in such a region, even if the position is shifted, a difference in density does not occur. FIG. 8A and FIG. 8B show examples of such images, and the matching result (corresponding to FIG. 6D) in this case is as shown in FIG. 8D. . As shown in FIG. 8D, the matching result clearly includes the wall of the front building.

  Therefore, in order to deal with such a problem, the following textureless area processing is performed. The specific processing procedure will be described below with reference to FIG.

  First, a region where the output result of the LOG filter is close to 0 is extracted as a textureless region and labeled for each region (A and B in FIG. 9A). That is, these areas are difficult to determine a planar area by matching. Actually, when such an image is determined by matching, an area indicated by a hatched portion in FIG. 9C is obtained, and as a result, the portion C is erroneously determined.

  Next, the following determination is performed on the textureless areas A and B (see FIG. 9D). If the entire textureless region is included in the matching result of FIG. 9C, the textureless region is a planar region (that is, in the case of region B). Note that it is considered that some boundary such as an edge or a difference in shading exists on the image between the actual planar region and the non-planar region. In the present invention, the boundary portion is not a textureless region and belongs to the matching region. Therefore, even if the textureless region exists at the edge of the planar region, the textureless region appears inside the matching region by the boundary portion in the processing result. If the entire textureless area is not included in the matching result of FIG. 9C, the entire textureless area is set as a non-planar area (that is, in the case of area A).

  Finally, an accurate planar region is obtained by excluding the textureless region determined as a non-planar region from the matching result of FIG. 9C (see FIG. 9E).

  The textureless region processing described above can be interpreted as follows. First, in the textureless region, it is not possible to determine whether or not a positional deviation has occurred even if local matching is performed. Therefore, the textureless areas are grouped, and each area is regarded as a “lumb” to determine the deviation. In that case, in the textureless area included in the planar area, there is naturally no positional shift, so the whole will match, but in the case of the textureless area in the non-planar area, the position will shift and there will be an area. The part necessarily overlaps with another surrounding area, and the part is determined to be a non-planar part by matching. Therefore, the entire textureless area is determined as a non-planar area.

  The results obtained by performing the above textureless region processing are shown in FIG. FIG. 10B shows the result of extracting the textureless region from the reference image shown in FIG. FIG. 10C shows the result of extracting the planar area by applying the textureless area processing described above. FIG. 10D shows a superimposed display with the original image.

  FIG. 11 shows another result obtained by performing the textureless area processing. In this example, there are textureless areas in the sky (non-planar area) and the paint (planar area) on the road surface, but textures corresponding to the sky (non-planar area) are processed by textureless area processing. It can be clearly seen that only the less regions are excluded from the matching results, and good results are obtained as shown in FIGS. 11 (E) and 11 (F).

To summarize the above, the overall flow of plane extraction in the present invention is as shown in FIG. 12, and the processing of each part will be described below.
<1> LOG Filter As a disadvantage of the region-based method (see Non-Patent Document 5) used in the present invention, if the brightness of two images is different, processing cannot be performed well. Therefore, in order to remove the brightness difference between the images, a LOG filter is applied to the input stereo original image. The LOG filter is expressed by the following equation (3). In this embodiment, σ = 1 and window size = 7.

また、ＬＯＧフィルタをかけた画像は、コントラストがとても低いので、本発明では、ヒストグラム平坦化によってコントラストを上げるようにしている。
＜２＞射影変換行列の推定
ステップＳ１００に説明された方法により、ステレオ画像間の平面の射影変換行列を動的に推定する。
＜３＞射影変換画像
＜２＞により得られた平面を表す射影変換行列によって、参照画像を射影変換する。射影変換行列が表す平面は、あたかも基準カメラから撮影したかのように一致し、射影変換行列が表す平面上にないものは、歪んで投影される。本発明ではこの性質を利用して平面領域を求める。
＜４＞差分画像
平面上に存在する点は基準画像へ正確に投影されるので、輝度値の差分は小さい。逆に、平面上にない点の輝度値の差分は大きくなる。閾値処理を施して差分の小さい領域を平面領域とする。このとき、ピクセルごとの差分ではノイズによる影響が大きいので、ＳＡＤによるマッチングを施す。具体的には、差分画像に平滑化フィルタをかけることで実現している。その後、閾値を使って２値化する。
＜５＞前時刻平面領域の利用
ステレオ動画像において、前時刻と現時刻に撮影された画像にはとても高い相関がある。同様に、処理結果においても相関が高いことを利用して、前時刻の処理結果を利用することで安定した処理を行うようしている。ここでは現時刻に抽出される平面領域と非平面領域の境界が前時刻から予想される境界に対してある一定幅に収まると考えられる。そこで、それ以外の部分に関しては、前時刻の結果より平面領域ないし非平面領域と見なすことができるため、それを２値画像(マッチング結果)に反映する。
＜６＞テクスチャレス領域
＜４＞の差分画像だけを利用すると、画像上であまりテクスチャの無い領域、つまりテクスチャレス領域では、濃度値の変化が無いため、たとえ異なる場所が重なりあっていたとしても道路領域として認識してしまう恐れがある。また、テクスチャレス領域をそのまま道路領域からすべて取り除いてしまうと、本来道路領域であるのに誤って非道路領域としてしまう恐れがある。道路領域内にもテクスチャレス領域が存在するためである。そこでＬＯＧフィルタをかけると、空間周波数が０に近い領域をテクスチャレス領域として抽出し、領域ごとにラベリングする（図１３(ａ)のＡ、Ｂ）。
＜７＞テクスチャレス領域の考慮
＜６＞でラベリングしたテクスチャレス領域ごとに基準画像と参照画像のマッチング結果（図１３(ｃ)）に完全に収まっている領域は道路領域とし、ずれが生じている領域（図１３(ｃ)のＣ）は非道路領域とする。
＜８＞Opening処理
求めたい道路領域は、自動車の走行可能な道路領域である。そこで、抽出された結果から、自動車の走行できない領域を取り除いている。まず、自動車の走行不可能な小さな領域を道路領域抽出結果から取り除く。得られた道路領域抽出結果からその面積を求め、閾値以下の領域を除去している。また、道路領域全体に収縮膨張処理を施す。

＜ステップＳ１２０＞射影変換行列を分解することにより道路面の傾きを算出する
ここでは平面領域抽出において得られた射影変換行列から、道路面の姿勢を表すパラメータである、基準カメラから道路平面までの距離ｄと道路平面の法線ベクトル

を求める方法について説明する。

In addition, since an image to which a LOG filter is applied has a very low contrast, the present invention increases the contrast by flattening the histogram.
<2> Estimation of Projection Transformation Matrix The plane projection transformation matrix between stereo images is dynamically estimated by the method described in step S100.
<3> Projection Conversion Image The reference image is projectively converted by the projection conversion matrix representing the plane obtained by <2>. The plane represented by the projective transformation matrix matches as if it was taken from the reference camera, and those not on the plane represented by the projective transformation matrix are distorted and projected. In the present invention, this property is used to obtain a planar area.
<4> Difference image Since the points existing on the plane are accurately projected onto the reference image, the difference in luminance value is small. On the contrary, the difference of the luminance value of the point which is not on a plane becomes large. An area having a small difference is set as a plane area by performing threshold processing. At this time, since the difference for each pixel is greatly affected by noise, matching by SAD is performed. Specifically, this is realized by applying a smoothing filter to the difference image. Then, binarization is performed using a threshold value.
<5> Use of Previous Time Plane Area In a stereo moving image, there is a very high correlation between images taken at the previous time and the current time. Similarly, by using the fact that the correlation is high in the processing result, stable processing is performed by using the processing result at the previous time. Here, it is considered that the boundary between the planar region and the non-planar region extracted at the current time falls within a certain width with respect to the boundary expected from the previous time. Therefore, the other portions can be regarded as a planar region or a non-planar region from the result at the previous time, and are reflected in the binary image (matching result).
<6> Textureless area If only the difference image of <4> is used, there is no change in the density value in the area without much texture on the image, that is, in the textureless area, even if different places overlap. There is a risk of being recognized as a road area. Further, if the textureless area is completely removed from the road area as it is, there is a possibility that it may be a non-road area accidentally although it is originally a road area. This is because there are textureless areas in the road area. Therefore, when a LOG filter is applied, an area having a spatial frequency close to 0 is extracted as a textureless area and is labeled for each area (A and B in FIG. 13A).
<7> Consideration of textureless area For each textureless area labeled in <6>, the area completely included in the matching result of the reference image and the reference image (FIG. 13C) is a road area, and a shift occurs. The area (C in FIG. 13C) is a non-road area.
<8> Opening process The road area to be obtained is a road area where a car can travel. Therefore, the area where the automobile cannot travel is removed from the extracted result. First, a small area where the automobile cannot travel is removed from the road area extraction result. The area is obtained from the obtained road area extraction result, and the area below the threshold is removed. Further, the entire road area is subjected to contraction and expansion processing.

<Step S120> The slope of the road surface is calculated by decomposing the projective transformation matrix. Here, from the projective transformation matrix obtained in the planar region extraction, a parameter representing the posture of the road surface, from the reference camera to the road plane. Normal vector of distance d and road plane

A method for obtaining the value will be described.

は図２のようなカメラ配置の場合、下記数４で表される。 First, the projective transformation matrix of the road plane

In the case of the camera arrangement as shown in FIG.

但し、

は基準カメラ座標系から参照カメラ座標系への回転行列で、

は参照カメラ座標系における基準カメラと参照カメラの並進ベクトルで、ｄは基準カメラと道路平面の距離で、

は道路平面の法線ベクトルで、Ａ₁、Ａ₂はそれぞれ基準カメラと参照カメラの内部パラメータである。

However,

Is the rotation matrix from the base camera coordinate system to the reference camera coordinate system,

Is the translation vector of the reference camera and reference camera in the reference camera coordinate system, d is the distance between the reference camera and the road plane,

Is a normal vector of the road plane, and A ₁ and A ₂ are internal parameters of the reference camera and the reference camera, respectively.

Here, the addition of the constant term k ≠ 0 indicates that the projective transformation matrix obtained from the image has a degree of freedom that is a constant multiple. Assuming that the camera internal parameters A ₁ and A ₂ are known, the following equation 5 holds.

ここで、非特許文献６に記載されたFaugerasらの手法によって射影変換行列

を特異値分解する。

Here, the projective transformation matrix is obtained by the method of Faugeras et al. Described in Non-Patent Document 6.

Singular value decomposition.

上記数６において、

でそれぞれ対応する行、列を入れ換えれば

のように並べ替えることができる。以後、降順に並んでいるとする。上記数５と上記数６を比較して、

In the above equation 6,

If you replace the corresponding rows and columns with

Can be sorted as follows. Hereinafter, it is assumed that they are arranged in descending order. Comparing Equation 5 and Equation 6 above,

ここで、

とすると、下記数９を得る。

here,

Then, the following formula 9 is obtained.

もとの

は、以下のような関係式から求めることができる。

Original

Can be obtained from the following relational expression.

上記数１０から、

を得るには

を求めればよいことが分かる。

From the above formula 10,

To get

It can be seen that

を導入し、

とし、上記数９から得られる３つの方程式と

は単位ベクトルであること、

は正規直交行列より

も単位ベクトルとなること、また

は回転行列なのでノルム変化がないことより、下記数１１が導出される。 Where the base of the reference camera coordinate system

Introduced

And the three equations obtained from Equation 9 above and

Is a unit vector,

Is from an orthonormal matrix

Is also a unit vector, and

Since is a rotation matrix, the following equation 11 is derived from the fact that there is no norm change.

これを

と置き、

の連立一次方程式として解くと、下記数１２を得る。

this

And put

Is solved as a simultaneous linear equation, the following equation 12 is obtained.

よって、下のように

の特異値が（I）重解を持たない場合、（II）重解を持つ場合、（III）３重解を持つ場合に場合分けして考える。
（I）

（II）

（III）

ここで、ｋ＝±σ₁もしくはｋ＝±σ₃は成り立たない。これは背理法で示される。よって、

にかかっている定数倍の項ｋは、

を特異値分解したときのσ₂として求まることが分かる。次に、ｋ＞０のときには、２台のカメラを平面の同じ側に配置にして平面の同じ面をカメラが撮影している場合に相当し、本発明で扱うカメラ配置である。また(III)の３重解の場合は、２つのカメラの光学中心が一致、つまり回転だけの場合なので、除外する。これらの関係より、一般的に、

は下記数１３のように書ける。

So, as below

The singular values of (1) have no multiple solutions, (II) have multiple solutions, and (III) have triple solutions.
(I)

(II)

(III)

Here, k = ± σ ₁ or k = ± σ ₃ does not hold. This is shown in a contradiction. Therefore,

The constant k term k depends on

It can be seen that it is obtained as σ ₂ when singular value decomposition is performed. Next, when k> 0, this corresponds to the case where two cameras are arranged on the same side of the plane and the same plane is taken by the camera, and is the camera arrangement handled in the present invention. In the case of the triple solution (III), the optical centers of the two cameras are coincident, that is, only rotation is excluded. From these relationships, in general,

Can be written as Equation 13 below.

（I）

のとき

より

を得る。つまり、

は

周りの回転行列である。以下それぞれ条件を入れて計算すると、下記数１４、数１５及び数１６を得る。

(I)

When

Than

Get. In other words,

Is

The surrounding rotation matrix. When calculation is performed with the following conditions, the following equations 14, 15, and 16 are obtained.

（II）

のとき及び（III）σ₁＝σ₂(σ₂＝σ₃)のとき
上記数１３より、

は単位行列になる。このとき、

(II)

And (III) when σ ₁ = σ ₂ (σ ₂ = σ ₃ )

Becomes the identity matrix. At this time,

と書ける。

Can be written.

の符号の違いにより、

は４つの解の不定性が発生している。そこで、「２台のカメラから平面が見えている」という条件を与える。つまり、光学中心を出た視線がレンズを通り、平面へ到達するということである。得られた

の候補のうち、

との内積が正になる

を選べば良い。これで４つから２つに絞れる。さらに、解を一意にするために、本実施例では２台のカメラ同士の位置関係が撮影中変化しないという条件を加えることで解決している。また、ｄを求めるためには、得られた

の絶対値と

から次の式より求めることができる。

つまり、２台のカメラのベースライン

を与えることでｄが決まる。

＜ステップＳ１３０＞道路平面を上方から見た画像（仮想投影面画像）を生成する
ステップＳ１２０で求められた道路平面の姿勢パラメータを使って、図１４に示されるように、基準画像に基づいて、道路平面と平行な仮想投影面(ＶＰＰ：Virtual Projection Plane)画像を生成する。図１４の中の記号を以下のように定義する。

は道路平面と基準カメラの座標系における法線ベクトルで、ｆは基準カメラの焦点距離で、ｄは基準カメラ光学中心と道路平面との距離で、また、ｅ_OZは基準カメラの光軸である。 here,

Due to the difference in the sign of

There are four uncertainties in the solution. Therefore, a condition that “a plane can be seen from two cameras” is given. That is, the line of sight that exits the optical center passes through the lens and reaches the plane. Obtained

Of the candidates for

The dot product with is positive

You can choose. This reduces the number from 4 to 2. Furthermore, in order to make the solution unique, in this embodiment, the problem is solved by adding a condition that the positional relationship between the two cameras does not change during shooting. Also, to obtain d, obtained

Absolute value of

From the following equation.

In other words, the baseline of the two cameras

Gives d.

<Step S130> Generating an image (virtual projection plane image) when the road plane is viewed from above Using the road plane attitude parameters obtained in step S120, as shown in FIG. A virtual projection plane (VPP) image parallel to the road plane is generated. Symbols in FIG. 14 are defined as follows.

Is the normal vector in the coordinate system of the road plane and the reference camera, f is the focal length of the reference camera, d is the distance between the optical center of the reference camera and the road plane, and e _OZ is the optical axis of the reference camera. .

とする。 Here, the normal vector obtained by decomposing the projective transformation matrix in step S120 is

And

との外積を回転軸として、ｅ_OZと

のなす角をθ回転させる変換行列を

とすると、 First, the optical axis e _OZ of the reference camera

E _OZ with the outer product of

A transformation matrix that rotates the angle between

Then,

と表せる。基準カメラをこの

により回転させたＶＰＰ画像へ変換する射影変換行列は、基準カメラの内部パラメータ行列を

、仮想カメラ（つまり、ＶＰＰカメラ）の内部パラメータ行列を

とすると、

It can be expressed. This is the reference camera

The projective transformation matrix to be converted into the VPP image rotated by the is the internal parameter matrix of the reference camera.

, The internal parameter matrix of the virtual camera (ie VPP camera)

Then,

と表せる。

It can be expressed.

をＶＰＰカメラ座標系から見たベクトル

を考え、それをＶＰＰ画像座標系に正射影したベクトル

を考える。 Next, the direction of the vertical axis of the VPP image and the direction of the optical axis of the reference camera are matched. Unit vector in the optical axis direction of the reference camera

Vector viewed from the VPP camera coordinate system

A vector that is orthogonally projected onto the VPP image coordinate system

think of.

First,

である。

It is.

のｘ成分をｕ_x、ｙ成分をｕ_yと表すとき、

をｘ−ｙ平面に正射影した

なるベクトル

を考える。 next,

When the x component of u is represented as u _x and the y component as u _y ,

Is orthogonally projected onto the xy plane.

Vector

think of.

をＶＰＰ画像座標に射影すると、同次ＶＰＰ画像座標上での無限遠点

に変換される。 this

Is projected onto the VPP image coordinates, the point at infinity on the homogeneous VPP image coordinates

Is converted to

と

を定義し、これをカメラ方向と呼ぶ。求める回転行列

はカメラ方向

を、「ＶＰＰ画像座標の−ｖ方向」＝「ＶＰＰカメラ座標系の−ｙ方向」に一致させる回転変換であるので、

When

This is called the camera direction. Desired rotation matrix

Is the camera direction

Is a rotational transformation that matches “−V direction of VPP image coordinates” = “− y direction of VPP camera coordinate system”.

を満たす

を求める。上記数２２の

と組み合わせることにより、基準画像からＶＰＰ画像への射影変換行列

は、

Meet

Ask for. Number 22 above

Projection transformation matrix from reference image to VPP image

Is

となる。

It becomes.

により、基準画像を射影変換してＶＰＰ画像を生成する。３次元空間でｍと表すことのできる点を基準画像へ投影して得られる基準画像座標の点

と、同じく点ｍをＶＰＰ画像へ投影したときのＶＰＰ画像座標の点

の間には、 The number 26

Thus, the reference image is projectively transformed to generate a VPP image. A point of reference image coordinates obtained by projecting a point that can be expressed as m in a three-dimensional space onto the reference image

Similarly, the point of the VPP image coordinates when the point m is projected onto the VPP image

In between

が成り立つので、ＶＰＰ画像上の点ｍ_iの輝度は、

の逆行列

を用いて対応する原画像上の点ｍ_0iの輝度値を代入することで得られる。

Since established, the luminance of the point m _i on the VPP image,

Inverse matrix of

_Is used to substitute the luminance value of the point m _0i on the corresponding original image.

ただし、

は行列

のｉ行ｊ列成分を表すとする。

However,

Is a matrix

Let i represent the i-row and j-column components.

Using the above projective transformation matrix and calculating the VPP image with the reference image of FIG. 16 as an input, FIG. 19 is generated. 15 is a reference image, and FIG. 18 is a result of superimposing the planar area shown in FIG. 17 on the original image.

<Step S140> Calculate the plane area that can be traveled on the VPP image and the position and direction of the vehicle. First, by the method described in step S130, only the plane area (white area) is converted into a VPP image as shown in FIG. The obtained image is shown in FIG. 20 (that is, a white area in this image is a plane area).

  Next, features of the VPP image will be described. A point on the estimated plane is converted into a VPP image having the correct coordinates by the conversion of Equation 27, but a point floating from the plane (for example, a car or a wall) is correct when converted to a VPP image. It is not converted to a position. That is, by converting the image into a VPP image, an image in which the estimated road plane area is viewed from above in parallel is generated, and the coordinates obtained from the image are a coordinate system corresponding to the real space.

と仮想カメラの内部パラメタ

と道路平面の法線ベクトル

、上記数２５の

を使い、同次ＶＰＰ画像座標上での光学中心の同次座標を

とすると、

より、ＶＰＰ画像の画像座標は

は

として得る。図２１の中ではＯが基本カメラの光学中心を道路平面に投影した点である。以後、この点を基準カメラ位置原点とする。よって、カメラと車輌の位置関係からＶＰＰ画像上での車輌の位置がわかる。また、図２１に示すように、ＶＰＰ画像で垂直な軸Ｙは光軸に一致する。つまり、光軸と車輌との取り付け角が既知であれば、ＶＰＰ画像中で自車輌の方向を算出できる。

＜ステップＳ１５０＞ＶＰＰ画像上での道路平面領域の自車輌からの最遠部すなわち障害物までの各方向毎の距離を算出する
領域の拡がりを求めるには、図２０のような平面領域をＶＰＰ画像に変換した画像を用意し、その画像上での基準カメラ位置原点から光軸（Ｙ軸）に対して、θ傾いた直線を伸ばし、平面領域の端を計算する（図２２参照）。画像上での基準カメラ位置原点Ｏから平面領域の端の座標を求めることにより、光軸からθ傾いた場所における画像上での平面の拡がり長さを求めることができる。画像上での長さは仮想カメラの内部パラメータと、基準カメラ光学中心と道路平面との距離ｄを使うことで実際の距離に変換できる。つまり、この処理により平面領域の方向別距離が算出されたわけである。 Next, in order to obtain the optical center dropped on the VPP image,

And virtual camera internal parameters

And road plane normal vector

, The above number 25

To calculate the homogeneous coordinates of the optical center on the homogeneous VPP image coordinates.

Then,

Therefore, the image coordinates of the VPP image are

Is

Get as. In FIG. 21, O is the point where the optical center of the basic camera is projected onto the road plane. Hereinafter, this point is set as the reference camera position origin. Therefore, the position of the vehicle on the VPP image can be known from the positional relationship between the camera and the vehicle. Further, as shown in FIG. 21, the vertical axis Y in the VPP image coincides with the optical axis. That is, if the mounting angle between the optical axis and the vehicle is known, the direction of the vehicle can be calculated in the VPP image.

<Step S150> Calculate the distance in each direction to the farthest part from the vehicle in the road plane area on the VPP image, that is, the obstacle. In order to obtain the area expansion, the plane area as shown in FIG. An image converted to an image is prepared, a straight line inclined by θ is extended from the reference camera position origin on the image with respect to the optical axis (Y axis), and the end of the planar area is calculated (see FIG. 22). By obtaining the coordinates of the end of the plane area from the reference camera position origin O on the image, the spread length of the plane on the image at a position inclined by θ from the optical axis can be obtained. The length on the image can be converted to an actual distance by using the internal parameter of the virtual camera and the distance d between the optical center of the reference camera and the road plane. That is, the distance for each direction of the planar area is calculated by this process.

In the present embodiment, due to restrictions such as the resolution of the photographed image, the upper limit is calculated that allows the area from the reference camera position origin to 32 m in the optical axis direction to be calculated as the area expansion. In addition, the measurement range was 25 degrees from the optical axis and in increments of 0.5 degrees.

<Step S160> Calculate the relative speed in each direction to the farthest part from the vehicle in the road plane area on the VPP image, that is, the obstacle. Next, from the vehicle in the road plane area on the VPP image The relative speed for each direction to the farthest part, that is, the obstacle is calculated. For each time, using the method described in step S150 for each direction, the distance for each direction of the road plane area is calculated.

Here, for example, in order to obtain the relative velocity of the plane inclined by θ from the optical axis (Y axis), the distance for each direction inclined by θ from the optical axis (Y axis) is used for the past five frames, and the image is 30 fps. Since shooting is 1/30 s in one frame, the least square method is used to obtain the slope from the time-series data of the direction-specific distance for each 1/30 s, and by converting it to speed, The relative speed was calculated (see FIG. 23). Therefore, by performing the same processing for each direction, the relative speed of the planar area is calculated for almost the entire planar area.

<Step S170> A plane area that can be traveled is displayed superimposed on the VPP image, and the calculated distance by direction and relative speed information by direction are displayed by the method described in Step S150 and Step S160. The distance and relative speed of the planar area could be calculated.

  In this invention, it devised so that the distance according to direction and the relative speed according to direction of the obtained road plane could be displayed easily. The result is the image shown in FIG. First, the image on the right side of the image shown in FIG. 24 is obtained by superimposing a reference image converted into a VPP image and a plane region converted into a VPP image. In the image on the right side of the image shown in FIG. 24, the light blue region is a combined planar region, and each point at the end of the planar region is an end point of the region (each point is a distance by plane direction). , The point indicated by arrow A), and the method of coloring the end points has the following rules.

  First, when the expansion rate of the plane is negative (the plane is contracted, that is, in the direction toward the camera), there is a risk, and the display is performed with warm color points. On the other hand, when the plane expansion speed is positive (the plane expands, that is, away from the camera), the danger is low, so it is displayed with a cold color point, and there is no plane expansion Is indicated by a green dot. The gradation below the left side of the image shown in FIG. 24 indicates the transition of the color of the point depending on the spread speed of the plane, and each number corresponds to the speed (km / h). In addition, a concentric arc is drawn every 5 m so as to be a fixed distance from the reference camera position origin (indicated by the arrow B in the figure), and the width of the vehicle on which the camera is mounted is indicated by a red line (in the figure, the arrow C).

を使って変換することで、基準画像での領域の端点に対応する座標が計算されるため、その座標を利用し、表示している（図中では矢印Ｄで示している）。ここで使用される色の決定の仕方は、図２４に示された画像の右側の画像の端点と同じルールを利用する。また、光軸から一定角度毎（本実施例では５度ごと）の点の上にその点の基準カメラ位置原点からの距離をメートル単位で表示している（図中では矢印Ｅで示している）。 Next, the image on the left side of the image shown in FIG. 24 is obtained by synthesizing the plane area with the reference image, and the plane area is indicated in light blue. Each end point of the plane area is expressed by the coordinates of the image calculated based on the VPP image as shown in Equation 27.

Since the coordinates corresponding to the end points of the region in the reference image are calculated by using the conversion, the coordinates are used and displayed (indicated by an arrow D in the figure). The method of determining the color used here uses the same rule as the end point of the image on the right side of the image shown in FIG. Further, the distance from the reference camera position origin of each point on a point at a certain angle from the optical axis (every 5 degrees in this embodiment) is displayed in meters (indicated by an arrow E in the figure). ).

  FIG. 25 and FIG. 26 show examples of road plane regions and obstacle detection results using the present invention by the above-described method of displaying the distance by direction and the relative speed information by direction. 25 and 26 show the result of applying the present invention to a stereo moving image taken by a camera mounted on an automobile. Incidentally, the result is a road plane area and an obstacle detection result in an urban area, and is a stereo. A result for every three frames (1/10 second) is extracted from the processing result for the moving image and displayed.

  As described above, in the road plane region and obstacle detection method using the stereo image according to the present invention having steps S100 to S170, the stereo image obtained from the two stereo cameras mounted on the automobile is obtained. It is possible to stably estimate the projection transformation matrix and the road plane area continuously by using the time series information that is obtained, the road surface itself tilts, and the vehicle body tilts due to curves and road surface irregularities during driving Even if the camera mounted on the car may vibrate, the projective transformation matrix is not constant, and the projective transformation matrix is always obtained from the stereo video, so solving these problems I was able to.

  Further, in the present invention, a virtual projection plane image (VPP image) is generated based on the slope of the road surface calculated by decomposing the projective transformation matrix, and further, from the vehicle in the plane area on the VPP image. The distance by direction and relative speed by direction to the obstacle are calculated, and finally the plane area that can be traveled is superimposed on the VPP image, and the calculated distance by direction and relative speed information by direction are displayed. Therefore, the display of the area where the vehicle can travel and the obstacle is a visual display, which is a highly practical method.

  When applying the present invention, it is preferable to use a CCD camera as a stereo camera mounted on an automobile. However, the present invention is not limited to this, and any other imaging means can be used as long as it can capture a stereo moving image. It is also possible to use photographing means. Further, the present invention can be applied to a traveling object other than an automobile.

It is a flowchart which shows the whole flow of this invention. It is a schematic diagram for demonstrating planar projection and two-dimensional projective transformation. It is a figure which shows an example of a stereo original image. It is a figure which shows the result image which performed the LOG filter process and the histogram flattening process to the stereo original image shown by FIG. It is a schematic diagram for demonstrating the projection transformation matrix and the estimation procedure of a calculation area | region using time series information. It is a figure which shows a planar area | region extraction result. It is a schematic diagram for demonstrating the area | region estimated from the previous time. It is a figure for demonstrating the planar area | region extraction of the stereo image in which a big textureless area | region exists. It is a schematic diagram for demonstrating a textureless area | region process. It is a figure which shows the planar area extraction result obtained by performing a textureless area process with respect to the stereo image in which a big textureless area exists. It is a figure which shows the planar area extraction result obtained by performing a textureless area process with respect to the stereo image in which a textureless area exists also in a planar area. It is the flowchart of the whole plane extraction in this invention. It is a schematic diagram for demonstrating the consideration method of a textureless area | region. It is a schematic diagram for demonstrating the virtual projection plane image (VPP image) of this invention. It is a figure which shows an example of a reference image. It is a figure which shows an example of a reference | standard image. It is a figure which shows a planar area | region extraction result. FIG. 18 is a diagram illustrating a result of superimposing the planar area of FIG. 17 on the original image of FIG. 16. It is a figure which shows the VPP image of the reference | standard image of FIG. It is a figure which shows the VPP image of the plane area | region of FIG. It is a figure for demonstrating the characteristic of a VPP image. It is a schematic diagram for demonstrating the calculation of the distance according to direction from the own vehicle of the plane area | region on a VPP image to the obstruction. It is a figure for demonstrating the relative velocity calculation of the plane area according to direction. It is an image which shows the road plane area | region and obstacle detection result using this invention. It is a figure which shows an example of the road plane area | region and obstacle detection result using this invention. It is a figure which shows the other example of the road plane area | region and obstacle detection result using this invention.

Claims (4)

It is possible to detect a road plane area that can be driven and all obstacles on the road surface by using only a stereo moving image composed of a base image and a reference image, which is taken by an imaging means mounted on an automobile. A road plane area and an obstacle detection method using stereo images,
A first step of dynamically estimating a projective transformation matrix for a road surface;
A second step of extracting a plane region that can be traveled using the projective transformation matrix obtained in the first step;
A third step of calculating a slope of the road surface by decomposing the projective transformation matrix obtained in the first step;
A fourth step of generating a virtual projection plane image in which the road surface is viewed from above based on the inclination of the road surface calculated in the third step;
On the virtual projection plane image generated in the fourth step, a fifth step of calculating the travelable plane area and the position and direction of the own vehicle;
A road plane area and an obstacle detection method using a stereo image characterized by comprising: A sixth step of calculating a distance by direction from the vehicle to the obstacle from the plane area where the vehicle can travel on the virtual projection plane image;
A seventh step of calculating a relative speed for each direction from the vehicle to the obstacle from the plane area where the vehicle can travel on the virtual projection plane image;
The road plane area | region and obstacle detection method using the stereo image of Claim 1 which further has these. An eighth step of displaying the travelable plane region superimposed on the virtual projection plane image, and displaying the distance by direction and the relative speed information by direction;
The road plane area | region and obstacle detection method using the stereo image of Claim 2 which further have these. In the first step, a LOG filter process and a histogram flattening process are performed on the base image and the reference image, and then a projective transformation matrix corresponding to a road surface between the stereo images is dynamically generated by an area-based method. To estimate
In the second step, the reference image is projectively transformed by the projective transformation matrix to obtain a difference image between the reference image and the projective transformed reference image, and a smoothing filter is applied to the difference image. A binary image is obtained by binarization using a threshold value, and a plane area that can be traveled is extracted by using the estimation result of the previous time plane area in the binary image and considering the textureless area. ,
In the third step, the road plane attitude parameter indicating the inclination of the road surface is a distance from the reference camera optical center to the road plane and a normal vector of the road plane,
The road plane using a stereo image according to claim 3, wherein in the fourth step, the reference image is projectively transformed using the road plane attitude parameter to generate a virtual projection plane image parallel to the road plane. Area and obstacle detection method.

Images