JP2003083742A - Distance correction apparatus and method of monitoring system - Google Patents
Distance correction apparatus and method of monitoring systemInfo
- Publication number
- JP2003083742A JP2003083742A JP2001277998A JP2001277998A JP2003083742A JP 2003083742 A JP2003083742 A JP 2003083742A JP 2001277998 A JP2001277998 A JP 2001277998A JP 2001277998 A JP2001277998 A JP 2001277998A JP 2003083742 A JP2003083742 A JP 2003083742A
- Authority
- JP
- Japan
- Prior art keywords
- vanishing point
- parallax
- distance
- calculating
- calculated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Traffic Control Systems (AREA)
- Image Analysis (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Measurement Of Optical Distance (AREA)
- Image Processing (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
Description
【０００１】[0001]
【発明の属する技術分野】本発明は、ステレオカメラの
位置ずれに起因した誤差を含んだ距離情報を補正する、
監視システム用の距離補正装置および距離補正方法に関
する。BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention corrects distance information including an error caused by a positional shift of a stereo camera,
The present invention relates to a distance correction device and a distance correction method for a surveillance system.
【０００２】[0002]
【従来の技術】近年、ＣＣＤ等の固体撮像素子を内蔵し
た一対の車載カメラ（ステレオカメラ）を用いたステレ
オ式車外監視装置が注目されている。三次元計測技術の
一つであるステレオ法では、一方の画像におけるある画
素ブロックと相関を有する領域を他方の画像において特
定する。そして、この画素ブロックの視差、すなわち、
双方の画像（ステレオ画像）における画素ブロックの相
対的なずれ量に基づき、三角測量の原理を用いて対象物
までの距離を算出する。したがって、ステレオマッチン
グの精度を高めるためには、換言すると、信頼性の高い
距離情報を得るためには、視差以外の位置的なずれがス
テレオ画像に存在しないことが望ましい。しかしなが
ら、実際は、ステレオカメラの機械的な取付精度に起因
した位置ずれ（例えば、水平方向や垂直方向への並進ず
れ、或いは回転ずれ等）が存在する。この位置ずれのう
ち、特に、水平方向の並進ずれ（以下「水平ずれ」とい
う）は、ステレオ画像における視差の誤差となって現れ
るため、それに基づき算出された距離は実測値と異なっ
てしまう。2. Description of the Related Art In recent years, a stereo type vehicle exterior monitoring apparatus using a pair of vehicle-mounted cameras (stereo cameras) having a solid-state image pickup device such as a CCD has attracted attention. In the stereo method, which is one of the three-dimensional measurement techniques, a region having a correlation with a pixel block in one image is specified in the other image. Then, the parallax of this pixel block, that is,
The distance to the object is calculated using the principle of triangulation based on the relative shift amount of the pixel blocks in both images (stereo images). Therefore, in order to improve the accuracy of stereo matching, in other words, in order to obtain highly reliable distance information, it is desirable that there is no positional deviation other than parallax in the stereo image. However, in reality, there is a positional deviation (for example, translational deviation in the horizontal direction or vertical direction, rotational deviation, etc.) due to the mechanical mounting accuracy of the stereo camera. Of these positional deviations, translational deviations in the horizontal direction (hereinafter referred to as “horizontal deviations”) appear as parallax errors in stereo images, and thus the distance calculated based on them is different from the actually measured value.
【０００３】例えば、特開２００１−１６０１３７号公
報には、ステレオカメラを用いた監視システムにおい
て、ステレオカメラの水平ずれに起因した誤差を含む算
出距離を、消失点視差ＤＰを用いて補正する手法が開示
されている。この距離補正手法では、一方の撮像画像
（基準画像）平面上に写し出された左右の車線の交点か
ら消失点JV2Dを算出し、この消失点JV2Dに基づいて道路
面の傾き角度ａを検出する。この傾き角度ａは、消失点
JV2Dに基づいて、写真解析の手法を用いて算出される。
また、ステレオ処理によって算出された距離画像（算出
距離群の二次元的な配列）を用いて、三次元空間におけ
る道路面の傾き角度ａ’を検出する。そして、これらの
傾き角度ａ，ａ’の差異を求め、両者が一致するように
消失点視差ＤＰを補正する。For example, Japanese Unexamined Patent Publication No. 2001-160137 discloses a method of correcting a calculated distance including an error caused by a horizontal displacement of a stereo camera using a vanishing point parallax DP in a monitoring system using a stereo camera. It is disclosed. In this distance correction method, the vanishing point JV2D is calculated from the intersection of the left and right lanes projected on one of the captured image (reference image) planes, and the inclination angle a of the road surface is detected based on the vanishing point JV2D. This inclination angle a is the vanishing point
Calculated using the method of photo analysis based on JV2D.
Further, the inclination angle a ′ of the road surface in the three-dimensional space is detected by using the distance image calculated by the stereo processing (two-dimensional array of calculated distance groups). Then, the difference between these inclination angles a and a ′ is obtained, and the vanishing point parallax DP is corrected so that the two match.
【０００４】また、特開平６−３４１８３７号公報に
は、上述した水平ずれの影響の低減を図る車間距離計測
装置が開示されている。この計測装置では、自車両前方
を撮影することにより得られる一対の撮像画像のそれぞ
れについて、左右の車線の交点より算出される消失点
と、先行車の像の中心軸（対称軸）とを求める。つぎ
に、一方の撮像画像上における消失点と中心線との視差
を算出するとともに、他方の撮像画像上における消失点
と中心線との視差とを算出する。そして、両視差を合計
することによって、先行車までの距離を算出する。Further, Japanese Laid-Open Patent Publication No. 6-341837 discloses an inter-vehicle distance measuring device for reducing the influence of the above-mentioned horizontal deviation. With this measuring device, the vanishing point calculated from the intersection of the left and right lanes and the central axis (symmetry axis) of the image of the preceding vehicle are obtained for each of a pair of captured images obtained by photographing the front of the vehicle. . Next, the parallax between the vanishing point and the center line on one captured image is calculated, and the parallax between the vanishing point and the center line on the other captured image is calculated. Then, the distance to the preceding vehicle is calculated by adding up both parallaxes.
【０００５】[0005]
【発明が解決しようとする課題】本発明の目的は、ステ
レオカメラの位置ずれ、特に水平ずれに起因した誤差を
含む視差を補正する、新規な距離補正手法を提供するこ
とである。SUMMARY OF THE INVENTION An object of the present invention is to provide a novel distance correction method for correcting a parallax including an error caused by a positional shift of a stereo camera, especially a horizontal shift.
【０００６】また、本発明の別の目的は、補正された視
差を用いて対象物までの距離を算出することにより、距
離計測の精度の向上を図ることである。Another object of the present invention is to improve the accuracy of distance measurement by calculating the distance to the object using the corrected parallax.
【０００７】[0007]
【課題を解決するための手段】かかる課題を解決するた
めに、第１の発明は、一対の撮像画像を得るステレオ撮
像手段と、ステレオ撮像手段により得られた一対の撮像
画像に基づいて、ステレオマッチングにより視差を算出
する視差算出手段と、視差算出手段により算出された視
差と消失点視差とに基づいて、対象物までの距離を算出
する距離算出手段と、一方の撮像画像平面において、距
離方向に延在する互いに空間的に平行な複数の近似直線
を算出し、この近似直線の交点から第１の消失点を算出
するとともに、他方の撮像画像平面において、距離方向
に延在する互いに空間的に平行な複数の近似直線を算出
し、この近似直線の交点から第２の消失点を算出する消
失点算出手段と、消失点算出手段により算出された第１
の消失点と第２の消失点とのずれ量に基づいて、消失点
視差を補正する補正手段とを有する監視システムの距離
補正装置を提供する。In order to solve such a problem, a first aspect of the present invention is based on a stereo image pickup means for obtaining a pair of picked-up images, and a stereo image based on the pair of picked-up images obtained by the stereo image pickup means. A parallax calculation unit that calculates parallax by matching, a distance calculation unit that calculates a distance to an object based on the parallax calculated by the parallax calculation unit and the vanishing point parallax, and the distance direction in one of the captured image planes. A plurality of spatially parallel approximation lines that are parallel to each other are calculated, and a first vanishing point is calculated from the intersection of the approximation straight lines. And a first vanishing point calculated by the vanishing point calculating means for calculating a second vanishing point from the intersection of the approximate straight lines.
There is provided a distance correction device for a monitoring system having a correction means for correcting the vanishing point parallax based on the amount of deviation between the vanishing point and the second vanishing point.
【０００８】ここで、上記第１の発明において、撮像画
像中に写し出された景色から、距離方向に延在する互い
に平行な複数の基準対象物を検出するとともに、撮像画
像平面における基準対象物の位置を特定する検出手段を
さらに設けてもよい。この場合、消失点算出手段は、検
出手段によって複数の基準対象物が検出されたならば、
検出された基準対象物のそれぞれについて撮像画像平面
における近似直線を算出することが好ましい。Here, in the first aspect of the invention, a plurality of reference objects parallel to each other extending in the distance direction are detected from the scenery projected in the picked-up image, and the reference object on the picked-up image plane is detected. You may further provide the detection means which pinpoints a position. In this case, the vanishing point calculation means, if a plurality of reference objects are detected by the detection means,
It is preferable to calculate an approximate straight line in the captured image plane for each of the detected reference objects.
【０００９】上記基準対象物は、撮像画像に写し出され
た道路上の左右の車線であってもよく、撮像画像に写し
出された壁と床との境界を示す左右の境界線であっても
よい。また、撮像画像に写し出された線路の左右のレー
ルを基準対象物として用いてもよい。The reference object may be the left and right lanes on the road shown in the picked-up image, or the left and right boundary lines showing the boundary between the wall and the floor shown in the picked-up image. . Further, the left and right rails of the railroad track shown in the captured image may be used as the reference object.
【００１０】第２の発明は、同一の景色を同一の時刻に
撮像した一対の撮像画像に基づいて、ステレオマッチン
グにより視差を算出するステップと、視差と消失点視差
とに基づいて、対象物までの距離を算出するステップ
と、一方の撮像画像平面において、距離方向に延在する
互いに空間的に平行な複数の近似直線を算出し、これら
の近似直線の交点から第１の消失点を算出するステップ
と、他方の撮像画像平面において、距離方向に延在する
互いに空間的に平行な複数の近似直線を算出し、これら
の近似直線の交点から第２の消失点を算出するステップ
と、第１の消失点と第２の消失点とのずれ量に基づい
て、消失点視差を補正するステップとを有する監視シス
テムの距離補正方法を提供する。According to a second aspect of the present invention, a step of calculating parallax by stereo matching based on a pair of picked-up images obtained by picking up the same scene at the same time, and even an object based on the parallax and the vanishing point parallax. And calculating a plurality of spatially parallel approximate straight lines extending in the distance direction on one of the captured image planes, and calculating a first vanishing point from the intersection of these approximate straight lines. A step of calculating a plurality of spatially parallel approximation lines extending in the distance direction on the other captured image plane, and calculating a second vanishing point from the intersection of these approximation lines; And a second vanishing point based on the amount of deviation between the vanishing point and the second vanishing point.
【００１１】[0011]
【発明の実施の形態】図１は、本実施形態に係るステレ
オ式車外監視装置の構成を示すブロック図である。ルー
ムミラーの近傍に取付けられたステレオカメラは、通常
の走行状態において、自車輛前方の道路や先行車等を含
む景色（同一の景色）を同一の時刻で撮像する。このス
テレオカメラは、ＣＣＤやＣＭＯＳセンサ、或いは、赤
外線カメラ等のイメージセンサを内蔵した一対のカメラ
１，２で構成されており、各カメラ１，２は、車幅方向
において所定のカメラ基線長で取付けられている。基準
画像信号を出力するメインカメラ１は、車輌の進行方向
に向かって右側に取付けられている。一方、比較画像信
号を出力するサブカメラ２は、進行方向に向かって左側
に取付けられている。カメラ対１，２は、互いに同期が
取れており、同一タイミングで前方の景色を撮像し、２
系統のアナログ画像信号を出力する。これらのアナログ
画像信号は、後段の回路の入力レンジに合致するよう
に、アナログインターフェース３において調整される。
また、アナログインターフェース３中のゲインコントロ
ールアンプ（ＧＣＡ）３ａにおいて画像の明るさバラン
スが調整される。1 is a block diagram showing the configuration of a stereo type vehicle exterior monitoring apparatus according to this embodiment. The stereo camera mounted near the rearview mirror captures a scene (same scene) including a road ahead of the vehicle and a preceding vehicle at the same time in a normal traveling state. This stereo camera is composed of a pair of cameras 1 and 2 having a built-in image sensor such as a CCD or CMOS sensor or an infrared camera, and each camera 1 and 2 has a predetermined camera baseline length in the vehicle width direction. Installed. The main camera 1 that outputs the reference image signal is mounted on the right side in the traveling direction of the vehicle. On the other hand, the sub camera 2 that outputs the comparative image signal is attached on the left side in the traveling direction. The camera pairs 1 and 2 are synchronized with each other, and capture a scene in the front at the same timing.
The analog image signal of the system is output. These analog image signals are adjusted in the analog interface 3 so as to match the input range of the circuit in the subsequent stage.
In addition, the brightness balance of the image is adjusted in the gain control amplifier (GCA) 3a in the analog interface 3.
【００１２】アナログインターフェース３において調整
されたアナログ画像信号は、Ａ／Ｄコンバータ４によ
り、所定の輝度階調（例えば、２５６階調のグレースケ
ール）のデジタル画像データに変換される。デジタル化
された各データは、補正回路５においてアフィン変換が
施される。各カメラ１，２の位置ずれ、そしてそれに起
因したステレオ画像のずれは、画像にアフィン変換を施
すことにより等価的に補正される。ここで、「アフィン
変換」とは、画像を回転、移動または拡大・縮小する幾
何学的な座標変換の総称をいう。補正回路５は、４つの
アフィンパラメータＫ，θ，SHFTI，SHFTJを用いて、原
画像に対して下式で示した線形変換を施す。The analog image signal adjusted by the analog interface 3 is converted by the A / D converter 4 into digital image data having a predetermined brightness gradation (for example, gray scale of 256 gradations). Each digitized data is subjected to affine transformation in the correction circuit 5. The positional deviation of each of the cameras 1 and 2 and the resulting stereo image deviation are equivalently corrected by subjecting the images to affine transformation. Here, “affine transformation” is a general term for geometrical coordinate transformation for rotating, moving, or enlarging / reducing an image. The correction circuit 5 uses the four affine parameters K, θ, SHFTI, and SHFTJ to perform the linear transformation shown in the following expression on the original image.
【数１】 [Equation 1]
【００１３】この数式において、（ｉ，ｊ）は原画像の
座標であり、（ｉ’，ｊ’）は変換後の座標である。ま
た、アフィンパラメータSHFTI，SHFTJはそれぞれ、ｉ方
向（画像の水平方向）への移動、ｊ方向（画像の垂直方
向）への移動を表している。また、アフィンパラメータ
θ，Ｋはそれぞれθの回転、Ｋ倍の拡大（|Ｋ|＜1の場
合は縮小）を示している。ステレオ画像にアフィン変換
を施すことによって、ステレオマッチングの精度を確保
する上で重要な「ステレオ画像における水平線の一致」
が保証される。なお、補正回路５の詳細なハードウェア
構成については、特開平１０−３０７３５２号公報に記
述されているので、必要ならば参照されたい。In this equation, (i, j) is the coordinate of the original image, and (i ', j') is the coordinate after conversion. Further, the affine parameters SHFTI and SHFTJ respectively represent movement in the i direction (horizontal direction of the image) and j direction (vertical direction of the image). Further, the affine parameters θ and K respectively indicate rotation of θ and expansion by K times (reduction when | K | <1). "Matching of horizontal lines in stereo images" is important for ensuring the accuracy of stereo matching by applying affine transformation to stereo images.
Is guaranteed. The detailed hardware configuration of the correction circuit 5 is described in Japanese Patent Laid-Open No. 10-307352, so refer to it if necessary.
【００１４】このような画像処理を経て、メインカメラ
１の出力信号から、例えば、水平方向が５１２画素、垂
直方向が２００画素の画像領域における各画素の輝度値
が基準画像データとして得られる。また、サブカメラ２
の出力信号から、基準画像と垂直方向長が同じで、基準
画像よりも大きな水平方向長を有する比較画像データが
得られる（一例として、水平方向が６４０画素、垂直方
向が２００画素）。なお、二次元平面である画像のｉ−
ｊ座標系は、画像の左下隅を原点として、水平方向をｉ
座標軸、垂直方向をｊ座標軸とする（単位は画素）。基
準画像データおよび比較画像データは、画像データメモ
リ７に格納される。Through such image processing, from the output signal of the main camera 1, for example, the brightness value of each pixel in the image area of 512 pixels in the horizontal direction and 200 pixels in the vertical direction is obtained as the reference image data. In addition, the sub camera 2
From this output signal, comparative image data having the same horizontal length as the reference image and a horizontal length larger than that of the reference image can be obtained (for example, 640 pixels in the horizontal direction and 200 pixels in the vertical direction). Note that i- of an image that is a two-dimensional plane
The j coordinate system uses the lower left corner of the image as the origin and sets the horizontal direction to i.
The coordinate axis and the vertical direction are j coordinate axes (unit is pixel). The reference image data and the comparison image data are stored in the image data memory 7.
【００１５】ステレオ演算回路６は、基準画像データと
比較画像データとに基づいて視差ｄを算出する。視差ｄ
は、基準画像において、例えば４×４画素の画素ブロッ
ク毎に一つ算出されるため、一フレーム分の基準画像全
体では最大で１２８×５０個算出され得る。基準画像中
の一画素ブロック（以下「対象画素ブロック」という）
の視差ｄiを算出する場合、まず、その対象画素ブロッ
クの輝度特性と相関を有する領域を比較画像において特
定する。周知のとおり、ステレオ画像に写し出された対
象物までの距離は、ステレオ画像における視差、すなわ
ち、基準画像と比較画像との間における画素ブロックの
水平方向のずれ量として現れる。したがって、比較画像
中の画素ブロック（以下「比較画素ブロック」という）
を探索する場合、対象画素ブロックのｊ座標と同じ水平
線（エピポーラライン）上を探索すればよい。ステレオ
演算回路６は、このエピポーラライン上を一画素ずつシ
フトしながら、比較画素ブロック毎に対象画素ブロック
との相関を評価する（ステレオマッチング）。The stereo operation circuit 6 calculates the parallax d based on the reference image data and the comparison image data. Parallax d
Is calculated for each pixel block of 4 × 4 pixels in the reference image, for example, 128 × 50 can be calculated at the maximum in the entire reference image for one frame. One pixel block in the reference image (hereinafter referred to as "target pixel block")
When calculating the parallax di of, the area having a correlation with the luminance characteristic of the target pixel block is first specified in the comparison image. As is well known, the distance to the object imaged in the stereo image appears as the parallax in the stereo image, that is, the horizontal shift amount of the pixel block between the reference image and the comparison image. Therefore, the pixel block in the comparison image (hereinafter referred to as "comparison pixel block")
When searching for, the search should be performed on the same horizontal line (epipolar line) as the j coordinate of the target pixel block. The stereo calculation circuit 6 evaluates the correlation with the target pixel block for each comparison pixel block while shifting this epipolar line pixel by pixel (stereo matching).
【００１６】二つの画素ブロックの相関関係は、例え
ば、周知の相関評価手法であるシティブロック距離を用
いて評価することができる。ステレオ演算回路６は、エ
ピポーラライン上に存在する領域（対象画素ブロックと
同一面積）毎にシティブロック距離を求め、基本的に
は、シティブロック距離の値が最小となる領域を対象画
素ブロックの相関先として特定する。そして、対象画素
ブロックと相関先に係る画素ブロックとの水平方向のず
れ量が視差ｄiとなる。なお、シティブロック距離の算
出に関するハードウェア構成および相関先の詳細な決定
手法については、特開平５−１１４０９９号公報に開示
されているので必要ならば参照されたい。ステレオ演算
回路６によって算出された視差ｄは距離データメモリ８
に格納される。The correlation between two pixel blocks can be evaluated using, for example, the city block distance which is a well-known correlation evaluation method. The stereo operation circuit 6 obtains the city block distance for each region (the same area as the target pixel block) existing on the epipolar line, and basically, the region where the value of the city block distance is the smallest is correlated with the target pixel block. Identify as the destination. The amount of horizontal shift between the target pixel block and the pixel block related to the correlation destination is the parallax di. The hardware configuration for calculating the city block distance and the detailed determination method of the correlation destination are disclosed in Japanese Patent Laid-Open No. 5-114099, so refer to them if necessary. The parallax d calculated by the stereo operation circuit 6 is the distance data memory 8
Stored in.
【００１７】マイクロコンピュータ９（機能的に捉えた
場合、その機能的ブロックである認識部１０）は、画像
データメモリ７から基準画像データを読み出し、基準画
像中に写し出された対象物（例えば先行車等）を周知の
画像認識技術を用いて認識する。また、認識部１０は、
距離データメモリ８から読み出した視差ｄを基本パラメ
ータとして、下式に基づいて対象物までの距離Ｚを算出
する。The microcomputer 9 (recognizing section 10 which is a functional block when functionally grasping) reads the reference image data from the image data memory 7 and targets the object (for example, the preceding vehicle) projected in the reference image. Etc.) is recognized using a known image recognition technique. Further, the recognition unit 10
Using the parallax d read from the distance data memory 8 as a basic parameter, the distance Z to the object is calculated based on the following formula.
【数２】 [Equation 2]
【００１８】同式において、ＫＺＨは所定の定数（カメ
ラ基線長／水平視野角）であり、ＤＰは消失点視差であ
る。本実施形態において、消失点視差ＤＰは視差補正値
（可変）であり、その値は後述する補正演算部１３にお
いて算出される。In the equation, KZH is a predetermined constant (camera base line length / horizontal viewing angle), and DP is a vanishing point parallax. In the present embodiment, the vanishing point parallax DP is a parallax correction value (variable), and the value is calculated by the correction calculation unit 13 described later.
【００１９】また、認識部１０は「道路形状の認識」を
行う。ここで、「道路形状の認識」とは、三次元的な道
路形状を左右の車線（白線や追い越し禁止ライン等）に
関する関数で表現し、この関数の各パラメータを、実際
の道路形状（直線、カーブ曲率または起伏）に合致する
ような値に設定することである。なお、以下の説明で
は、車線の典型である白線を例に説明するが、追越し車
線等を含めた各種車線に対しても適用可能である。以
下、本実施形態における白線モデルの算出手法を図２を
参照しつつ説明する。The recognition unit 10 also performs "recognition of road shape". Here, "recognition of road shape" means that a three-dimensional road shape is expressed by a function relating to left and right lanes (white line, overtaking prohibited line, etc.), and each parameter of this function is expressed as an actual road shape (straight line, Curve curvature or undulation). In the following description, a white lane, which is a typical lane, will be described as an example, but the present invention can be applied to various lanes including an overtaking lane. The method of calculating the white line model in this embodiment will be described below with reference to FIG.
【００２０】まず、基準画像において白線エッジＰedg
e、すなわち、水平方向の輝度エッジ（隣接画素間の輝
度の変化量が大きい箇所）の内、白線に起因して生じた
ものが特定される。白線エッジＰedgeは、走行路の左側
と右側とについて別個に探索され、複数の左白線エッジ
Ｐedge1と複数の右白線エッジＰedge2とがそれぞれ特定
される。具体的には、下記の３つの条件を具備する輝度
エッジが白線エッジＰedgeとして認識される。First, the white line edge Pedg in the reference image
e, that is, a horizontal luminance edge (a portion where the amount of change in luminance between adjacent pixels is large) caused by a white line is specified. The white line edge Pedge is searched separately for the left side and the right side of the traveling road, and a plurality of left white line edges Pedge1 and a plurality of right white line edges Pedge2 are respectively specified. Specifically, a luminance edge having the following three conditions is recognized as a white line edge Pedge.
【００２１】（白線エッジの３条件）
１．輝度変化量が所定値以上である輝度エッジで、か
つ、輝度エッジの内側（画像中央側）の画素よりも外側
（画像端側）の画素の方が輝度が大きいこと
すなわち走行路の左右の白線に起因した白線エッジＰed
geは、図２に示したように、白線の内側の境界（白線と
舗装路との境界）における輝度エッジである。(3 conditions of white line edge) 1. At the luminance edge where the amount of luminance change is more than a predetermined value,
Outside the pixels inside the luminance edge (on the center side of the image)
The pixels on the (image edge side) have higher brightness, that is, the white line edges Ped due to the white lines on the left and right of the road.
As shown in FIG. 2, ge is the luminance edge at the inner boundary of the white line (the boundary between the white line and the paved road).
【００２２】２．条件１を満たす白線エッジＰedgeの候
補に関して、それと同一水平線上の外側にさらに輝度エ
ッジが存在し、かつ、この輝度エッジの外側の画素より
も内側の画素の方が輝度が大きいこと
白線は所定の幅を有しているため、白線エッジＰedgeの
外側にも境界が存在する。この条件は、このような白線
の特徴に鑑みて設けられたものである。2. White line edge Pedge that satisfies condition 1
In addition, the luminance
Edge is present, and from pixels outside this luminance edge
Also, the luminance is higher in the inner pixel. Since the white line has a predetermined width, there is a boundary outside the white line edge Pedge. This condition is provided in view of such characteristics of the white line.
【００２３】３．条件１を満たす白線エッジＰedgeを含
む画素ブロックに関して、視差ｄが算出されていること
白線エッジＰedgeが存在する箇所に視差ｄが算出されて
いなければ、その白線エッジＰedgeは道路形状を認識す
るのに有効な情報とはならない。3. Include the white line edge Pedge that satisfies the condition 1.
The parallax d is calculated for the defective pixel block. If the parallax d is not calculated at the position where the white line edge Pedge exists, the white line edge Pedge is not effective information for recognizing the road shape.
【００２４】認識部１０は、特定された白線エッジＰed
ge毎に、その座標（ｉ，ｊ）およびその視差ｄを下記の
数式３および数式４に示した周知の座標変換式に代入す
ることにより、その実空間上の座標（Ｘ，Ｙ，Ｚ）を算
出する。The recognizing unit 10 recognizes the specified white line edge Ped.
By substituting the coordinates (i, j) and the parallax d for each ge into the well-known coordinate conversion formulas shown in the following formulas 3 and 4, the coordinates (X, Y, Z) in the real space are calculated. calculate.
【数３】 [Equation 3]
【数４】 [Equation 4]
【００２５】ここで、定数ＣＡＨはステレオカメラ１，
２の取付高さ、定数ｒはステレオカメラ１，２の取り付
け間隔、定数ＰＷＶ，ＰＷＨはそれぞれ１画素当たりの
垂直視野角、水平視野角である。また、定数ＩＶ，ＪＶ
はそれぞれ予め設定された消失点Ｖのｉ座標値、ｊ座標
値である。また、自車輌の位置を基準に設定された実空
間の座標系は、ステレオカメラ１，２の中央真下の道路
面を原点として、車幅方向をＸ軸、車高方向をＹ軸、車
長方向（距離方向）をＺ軸とする。撮像画像に写し出さ
れた対象物（先行車、立体物、または道路等）に関し
て、画像平面上の座標（ｉ，ｊ）および視差ｄが特定さ
れると、数式２〜数式４に示した変換式に基づいて、実
空間上の座標（Ｘ，Ｙ，Ｚ）を一義的に特定することが
できる。Here, the constant CAH is the stereo camera 1,
2 is a mounting height, a constant r is a mounting interval between the stereo cameras 1 and 2, and constants PWV and PWH are a vertical viewing angle and a horizontal viewing angle per pixel, respectively. Also, the constants IV, JV
Are the i-coordinate value and j-coordinate value of the vanishing point V set in advance, respectively. In addition, the coordinate system of the real space set based on the position of the vehicle is based on the road surface directly under the center of the stereo cameras 1 and 2 as the origin, the vehicle width direction is the X axis, the vehicle height direction is the Y axis, and the vehicle length. The direction (distance direction) is the Z axis. When the coordinates (i, j) on the image plane and the parallax d are specified with respect to the target object (preceding vehicle, three-dimensional object, road, etc.) shown in the captured image, the conversion formulas shown in Formulas 2 to 4 are obtained. Based on, the coordinates (X, Y, Z) in the real space can be uniquely specified.
【００２６】このようにして特定された各白線エッジＰ
edgeの実空間上の座標（Ｘ，Ｙ，Ｚ）に基づいて白線モ
デルが特定される。白線モデルは、認識範囲（例えばカ
メラ位置から車輌前方８４ｍ先まで）内の左右の白線エ
ッジＰedge1，Ｐedge2のそれぞれ関して所定区間ごとに
近似直線を求め、これを折れ線状に連結して表現したも
のである。一例として示す図３の白線モデルは、認識範
囲を７つの区間に分けて、各区間における左右の白線エ
ッジＰedge1，Ｐedge2毎に最小二乗法を用いて下式の直
線で近似している。Each white line edge P specified in this way
The white line model is specified based on the coordinates (X, Y, Z) of the edge in the real space. The white line model is an expression obtained by obtaining approximate straight lines for each predetermined section for each of the left and right white line edges Pedge1 and Pedge2 within the recognition range (for example, from the camera position to 84 m ahead of the vehicle) and connecting them in a polygonal line shape. Is. In the white line model of FIG. 3 shown as an example, the recognition range is divided into seven sections, and the left and right white line edges Pedge1 and Pedge2 in each section are approximated by the straight line of the following equation using the least square method.
【数５】（左白線モデルＬ） Ｘ ＝ ａL・Ｚ ＋ ｂL Ｙ ＝ ｃL・Ｚ ＋ ｄL （右白線モデルＲ） Ｘ ＝ ａR・Ｚ ＋ ｂR Ｙ ＝ ｃR・Ｚ ＋ ｄR Equation 5] (left white line model L) X = a L · Z + b L Y = c L · Z + d L ( right white line model R) X = a R · Z + b R Y = c R · Z + d R
【００２７】これらの白線モデルＬ，Ｒは、道路のカー
ブ曲率を表現したカーブ関数（Ｘ＝ｆ(Ｚ)）と、道路の
勾配や起伏を表現した勾配関数（Ｙ＝ｆ(Ｚ)）とで構成
されている。したがって、実空間における道路の三次元
的な変化状態は、左右の白線モデルＬ，Ｒによって把握
することができる。認識部１０により算出された各白線
エッジＰedgeおよび左右の白線モデルＬ，Ｒは補正演算
部１３に伝達される。These white line models L and R are a curve function (X = f (Z)) expressing the curve curvature of the road and a gradient function (Y = f (Z)) expressing the road gradient or undulation. It is composed of. Therefore, the three-dimensional change state of the road in the real space can be grasped by the left and right white line models L and R. Each white line edge Pedge and the left and right white line models L and R calculated by the recognition unit 10 are transmitted to the correction calculation unit 13.
【００２８】認識部１０は、先行車や道路形状に関する
認識結果に基づいて、警報が必要と判定した場合、モニ
タやスピーカー等の警報装置１１を作動させてドライバ
ーに注意を促す。また、必要に応じて制御装置１２を制
御することにより、ＡＴ（自動変速機）のシフトダウン
やエンジン出力の抑制、或いはブレーキの作動といった
車輌制御が実行される。When it is determined that an alarm is necessary, the recognition unit 10 activates an alarm device 11 such as a monitor or a speaker based on the recognition result of the preceding vehicle and the shape of the road to call the driver's attention. Further, by controlling the control device 12 as necessary, vehicle control such as downshifting of an AT (automatic transmission), suppression of engine output, or operation of a brake is executed.
【００２９】つぎに、本実施形態にかかる視差補正の詳
細を、図４および図５に示すフローチャートを参照しな
がら説明する。補正演算部１３は、このフローチャート
に示した一連の手順に従って消失点視差ＤＰの値を更新
し、その値を認識部１０にフィードバックする。なお、
このフローチャートは、所定期間のサイクル毎に繰り返
し実行される。The details of parallax correction according to this embodiment will be described below with reference to the flow charts shown in FIGS. The correction calculation unit 13 updates the value of the vanishing point parallax DP according to the series of procedures shown in this flowchart, and feeds back the value to the recognition unit 10. In addition,
This flowchart is repeatedly executed every cycle of a predetermined period.
【００３０】まず、ステップ１において、補正演算部１
３は、認識部１０において算出された一対の撮像画像
（基準画像および比較画像）のそれぞれに関して、白線
エッジＰedgeと白線モデルＬ，Ｒとを読み込む。つぎ
に、ステップ２において、基準画像において左右の白線
が存在するか否かが判断される。これは、認識部１０に
おいて左右の白線モデルＬ，Ｒが算出されているかを調
べることにより判断することができる。また、左白線エ
ッジＰedge1と右白線エッジＰedge2とが算出されている
か否かを調べることにより判断してもよい。First, in step 1, the correction calculation unit 1
3 reads the white line edge Pedge and the white line models L and R for each of the pair of captured images (reference image and comparison image) calculated by the recognition unit 10. Next, in step 2, it is judged whether or not there are left and right white lines in the reference image. This can be determined by checking whether the left and right white line models L and R are calculated in the recognition unit 10. Alternatively, the determination may be made by checking whether or not the left white line edge Pedge1 and the right white line edge Pedge2 have been calculated.
【００３１】ステップ２において否定判定された場合、
すなわち左右の双方に白線が存在しない場合は、互いに
平行な線分を抽出できないので消失点を算出することが
できない。したがって、制御の安定性を図るため、消失
点視差ＤＰの現在値を変更することなくリターンへ進
み、今回のサイクルにおける本フローチャートの実行を
終了する。一方、ステップ２において肯定判定された場
合は、ステップ３に進む。If a negative decision is made in step 2,
That is, when there are no white lines on both the left and right sides, line segments parallel to each other cannot be extracted, and thus the vanishing point cannot be calculated. Therefore, in order to stabilize the control, the process proceeds to the return without changing the current value of the vanishing point parallax DP, and the execution of this flowchart in this cycle is finished. On the other hand, if an affirmative decision is made in step 2, the operation proceeds to step 3.
【００３２】ステップ３において、左右の白線の信頼性
が評価される。具体的には以下の２つの点が評価され
る。そしてステップ４において白線として信頼できると
判断された場合のみステップ５に進む。一方、白線とし
て信頼できないと判断された場合は、消失点視差ＤＰの
値を変更することなくリターンへ進む。In step 3, the reliability of the left and right white lines is evaluated. Specifically, the following two points are evaluated. Then, the process proceeds to step 5 only when it is determined in step 4 that the white line is reliable. On the other hand, if it is determined that the white line is not reliable, the process proceeds to return without changing the value of the vanishing point parallax DP.
【００３３】（白線の信頼性評価）
１．前回のサイクルにおける白線位置と今回のサイクル
における白線位置とのずれが所定値よりも大きい場合は
白線としての信頼性が低いと判断する。具体的には、従
前のサイクルで検出された白線エッジＰedgeの位置と今
回のサイクルで検出された白線エッジＰedgeの位置的が
大きくずれている場合には白線としての信頼性が低いと
判断する。(Evaluation of reliability of white line) 1. When the deviation between the white line position in the previous cycle and the white line position in the current cycle is larger than a predetermined value, it is determined that the reliability of the white line is low. Specifically, if the position of the white line edge Pedge detected in the previous cycle and the position of the white line edge Pedge detected in the current cycle are significantly deviated, it is determined that the reliability of the white line is low.
【００３４】２．白線がどのくらい先まで見えているか
を検証する。白線は少なくともある程度の長さを有して
いる。したがって、フレーム間の白線の推移も考慮し
て、白線エッジＰedgeが奥行き方向において一定の長さ
以上延在していない場合には白線としての信頼性が低い
と判断する。2. Verify how far the white line can be seen. The white line has at least some length. Therefore, in consideration of the transition of the white line between the frames, it is determined that the reliability of the white line is low when the white line edge Pedge does not extend for a certain length or more in the depth direction.
【００３５】ステップ５において、白線モデルＲ，Ｌに
基づいて、白線の直線性が評価される。正確な消失点を
算出するためには、その算出ベースとなる左右の白線が
直線的に延在している必要があり、カーブした白線から
は正確な消失点を算出することができない。そこで、ス
テップ６において白線が直線であると判断された場合の
みステップ７へ進み、それ以外の場合は、消失点視差Ｄ
Ｐの値を変更することなくリターンへ進む。In step 5, the linearity of the white line is evaluated based on the white line models R and L. In order to calculate an accurate vanishing point, the left and right white lines that are the basis for the calculation need to extend linearly, and an accurate vanishing point cannot be calculated from a curved white line. Therefore, the process proceeds to step 7 only when the white line is determined to be a straight line in step 6, and the vanishing point parallax D otherwise.
Proceed to return without changing the value of P.
【００３６】白線の直線性は、例えば、認識部１０にお
いて算出された白線モデル（カーブ関数Ｘ＝ｆ(Ｚ)）に
基づいて評価することができる。図３を参照しながら説
明すると、まず、Ｚ−Ｘ平面における所定の距離レンジ
（例えば０〜Ｚ2）におけるカーブ関数の傾きＡ１（左
右の白線Ｌ，Ｒの傾きａL，ａRの平均）を算出する。こ
の傾きＡ１は、第１区間における傾きａ1と第２区間に
おける傾きａ2との平均値を用いる。つぎに、その先の
所定の距離レンジ（例えばＺ2〜Ｚ4）におけるカーブ関
数の傾きＡ２を算出する。この傾きＡ２は、第３区間に
おける傾きａ3と第４区間における傾きａ4との平均値を
用いる。そして、傾きＡ１と傾きＡ２との差（絶対値）
を求め、この差が所定のしきい値以下であれば白線が直
線であると判断する。The linearity of the white line can be evaluated, for example, based on the white line model (curve function X = f (Z)) calculated by the recognition unit 10. To illustrate with reference to FIG. 3, first, Z-X slope of the curve function at a predetermined distance range (e.g. 0～Z2) in the plane A1 (left and right white lines L, the slope of R a L, the average of a R) a calculate. As the slope A1, an average value of the slope a1 in the first section and the slope a2 in the second section is used. Next, the slope A2 of the curve function in the predetermined distance range (for example, Z2 to Z4) is calculated. As the slope A2, an average value of the slope a3 in the third section and the slope a4 in the fourth section is used. The difference between the slope A1 and the slope A2 (absolute value)
If this difference is less than or equal to a predetermined threshold value, it is determined that the white line is a straight line.
【００３７】図５に示すステップ７以降の手順は、消失
点視差ＤＰの更新処理に関する。まず、ステップ７にお
いて、基準画像における所定の距離レンジ（例えば０〜
Ｚ2）内に存在する複数の左白線エッジＰedge1の近似直
線Ｌ１mが、最小自乗法により算出される（図６参
照）。同様に、その距離レンジ内に存在する複数の右白
線エッジＰedge2の近似直線Ｌ２mも最小自乗法により算
出される。そして、ステップ８において、図６に示した
ように近似直線Ｌ１m，Ｌ２mの交点を特定することで、
基準画像における消失点Ｖm（ＩＶm，ＪＶm）のｉ座標
値ＩＶmが算出される。The procedure after step 7 shown in FIG. 5 relates to the process of updating the vanishing point parallax DP. First, in step 7, a predetermined distance range (for example, 0 to
The approximate straight line L1m of the plurality of left white line edges Pedge1 existing in Z2) is calculated by the least square method (see FIG. 6). Similarly, the approximate straight line L2m of the plurality of right white line edges Pedge2 existing within the distance range is also calculated by the least square method. Then, in step 8, by specifying the intersection of the approximate straight lines L1m and L2m as shown in FIG.
The i coordinate value IVm of the vanishing point Vm (IVm, JVm) in the reference image is calculated.
【００３８】続くステップ９において、基準画像の場合
と同様の手法で、比較画像における所定の距離レンジ
（例えば０〜Ｚ2）内に存在する複数の左白線エッジＰe
dge1の近似直線Ｌ１sが最小自乗法により算出される。
それとともに、所定の距離レンジ内に存在する複数の右
白線エッジＰedge2の近似直線Ｌ２sも最小自乗法により
算出される。そして、ステップ１０において、近似直線
Ｌ１s，Ｌ２sの交点を特定することで、比較画像におけ
る消失点Ｖs（ＩＶs，ＪＶs）のｉ座標値ＩＶsが算出さ
れる。In the following step 9, a plurality of left white line edges Pe existing within a predetermined distance range (for example, 0 to Z2) in the comparison image are obtained by the same method as in the reference image.
The approximate straight line L1s of dge1 is calculated by the least square method.
At the same time, the approximate straight line L2s of the plurality of right white line edges Pedge2 existing within the predetermined distance range is also calculated by the least square method. Then, in step 10, the i coordinate value IVs of the vanishing point Vs (IVs, JVs) in the comparative image is calculated by specifying the intersection of the approximate straight lines L1s and L2s.
【００３９】そして、ステップ１１において、これらの
消失点ＩＶm，ＩＶsに基づいて、視差補正値、すなわち
消失点視差ＤＰの更新処理が行われる。基本的には、基
準画像側に関して算出された消失点Ｖmのｉ座標値ＩＶm
と、比較画像側に関して算出された消失点Ｖsのｉ座標
値ＩＶsとのずれ量が、消失点視差ＤＰになる。そし
て、算出された消失点視差ＤＰが認識部１０に対して出
力され、今回のサイクルにおける本フローチャートの処
理が終了する。なお、制御の安定性を考慮して、このス
テップ１１で算出された消失点視差ＤＰを１〜ｎ回の処
理サイクルに亘って保存し、その平均値を距離補正に使
用するパラメータ（消失点視差）として適用してもよ
い。Then, in step 11, the parallax correction value, that is, the vanishing point parallax DP is updated based on these vanishing points IVm and IVs. Basically, the i coordinate value IVm of the vanishing point Vm calculated for the reference image side
Then, the amount of deviation from the i coordinate value IVs of the vanishing point Vs calculated for the comparative image side becomes the vanishing point parallax DP. Then, the calculated vanishing point parallax DP is output to the recognition unit 10, and the processing of this flowchart in the current cycle ends. In consideration of the control stability, the vanishing point parallax DP calculated in step 11 is stored for 1 to n processing cycles, and the average value thereof is used as a parameter (vanishing point parallax). ) May be applied.
【００４０】本実施形態によれば、消失点視差ＤＰに関
するフィードバック調整を監視制御と並行して行うこと
で、ステレオカメラの水平ずれが生じた場合であっても
常に精度の高い距離を算出することができる。したがっ
て、経年変化や衝撃等によってステレオカメラの取付け
位置が初期設定状態から変化した場合であっても、信頼
性の高い距離情報を安定して得ることができる。そし
て、このような算出距離に基づいて監視制御を行うこと
により、車外監視の信頼性の向上を図ることができる。
特に、本実施形態によれば、一対の撮像画像を用いて消
失点視差ＤＰを直接的に検出するため、消失点視差が大
きくずれた場合でも、それを安定して検出することがで
きる。According to the present embodiment, the feedback adjustment relating to the vanishing point parallax DP is performed in parallel with the monitoring control, so that a highly accurate distance can always be calculated even when the stereo camera is displaced horizontally. You can Therefore, even when the mounting position of the stereo camera changes from the initial setting state due to aging, impact, or the like, highly reliable distance information can be stably obtained. Then, by performing monitoring control based on such a calculated distance, it is possible to improve the reliability of vehicle exterior monitoring.
In particular, according to the present embodiment, since the vanishing point parallax DP is directly detected using the pair of captured images, even if the vanishing point parallax is significantly deviated, it can be stably detected.
【００４１】なお、上述した説明において、消失点視差
の更新は、比例制御や統計制御等によって行ってもよ
い。例えば、消失点視差ＤＰの１０００サンプル分のヒ
ストグラムを取り、その最頻値を用いてもよい。In the above description, the vanishing point parallax may be updated by proportional control, statistical control, or the like. For example, a histogram of 1000 samples of the vanishing point parallax DP may be taken and the mode value thereof may be used.
【００４２】（撮像画像を用いた各種監視システムへの
適用）上述した実施形態では、撮像画像に写し出された
道路上の左右の車線（白線）を利用して、消失点視差Ｄ
Ｐを算出する手法について説明した。これは、自動車の
前方監視の場合、通常、距離方向（Ｚ方向）に向かって
延在する車線が道路の左右に存在し、これらは空間的に
互いに平行であることが多いという一般的な傾向に鑑み
たものである。本明細書では、車線のように、距離方向
に向かって互いに平行に延在し、消失点算出のベースと
なる直線的な対象物を「基準対象物」という。そして、
本発明は、「基準対象物」が写し出される撮像画像を用
いた各種監視システムに広く適用することができる。(Application to Various Monitoring Systems Using Captured Images) In the above-described embodiment, the vanishing point parallax D is obtained by using the left and right lanes (white lines) on the road shown in the captured images.
The method of calculating P has been described. This is a general tendency in the case of monitoring the front of a vehicle, where there are usually lanes extending in the distance direction (Z direction) on the left and right sides of the road, and these are often spatially parallel to each other. In view of. In the present specification, a linear object that extends parallel to each other in the distance direction and serves as a base for vanishing point calculation, such as a lane, is referred to as a “reference object”. And
INDUSTRIAL APPLICABILITY The present invention can be widely applied to various monitoring systems using a captured image in which a “reference object” is projected.
【００４３】一例として、撮像画像に基づき周囲の状況
を認識する屋内ロボットに適用する場合、壁と床との２
つの境界線を「基準対象物」として用いることができ
る。図７は、屋内ロボットにおける撮像画像の一例であ
る。なお、同図に示す直線Ｌ１（またはＬ２）は、基準
画像側の直線Ｌ１m（またはＬ２m）と比較画像側の直線
Ｌ１s（またはＬ２s）とを総称する意味で用いている。
通常、左壁と床との境界線および右壁と床との境界線
は、距離方向（奥行方向）に向かって互いに平行に延在
していることが多い。したがって、左右の境界線を利用
して、消失点補正や距離補正を行うことができる。以
下、境界線を利用した消失点の調整手順の概略を説明す
る。As an example, in the case of applying to an indoor robot that recognizes the surrounding situation based on a captured image, it is possible to use a wall and a floor.
One boundary line can be used as a “reference object”. FIG. 7 is an example of a captured image of the indoor robot. The straight line L1 (or L2) shown in the figure is used as a generic term for the straight line L1m (or L2m) on the reference image side and the straight line L1s (or L2s) on the comparative image side.
Usually, the boundary line between the left wall and the floor and the boundary line between the right wall and the floor often extend parallel to each other in the distance direction (depth direction). Therefore, vanishing point correction and distance correction can be performed using the left and right boundary lines. The outline of the vanishing point adjustment procedure using the boundary line will be described below.
【００４４】まず、基準画像に基づき複数の直線Ｌ１
m，Ｌ２mを検出する。上述した（白線エッジの条件）と
同様に、壁と床との境界部分の輝度エッジや視差に関す
る条件を予め設定しておく。そして、撮像画像におい
て、この条件に合致する部分を境界線として認識し、そ
の直線性を適宜評価した上で、それぞれの近似直線Ｌ１
m，Ｌ２mを算出する。また、別の手法としては、周知の
ハフ変換等を用いて、撮像画像において直線を形成する
点（境界部分のエッジ画素）を抽出することにより、
「基準対象物」となる直線Ｌ１m，Ｌ２mを算出してもよ
い。First, based on the reference image, a plurality of straight lines L1
Detect m and L2m. Similar to the above (condition of white line edge), the condition regarding the luminance edge and the parallax of the boundary portion between the wall and the floor is set in advance. Then, in the picked-up image, a portion that meets this condition is recognized as a boundary line, its linearity is appropriately evaluated, and then each approximate straight line L1
Calculate m and L2m. In addition, as another method, by using a well-known Hough transform or the like, by extracting points (edge pixels at the boundary portion) forming a straight line in the captured image,
You may calculate the straight lines L1m and L2m used as a "reference | standard object."
【００４５】つぎに、距離画像に基づき直線Ｌ１m，Ｌ
２mが空間的に概ね平行であることを判定する。上述し
たように、距離画像に基づいて、直線Ｌ１m，Ｌ２mを構
成する各領域の実空間上の位置を特定することができ
る。したがって、２本の直線Ｌ１m，Ｌ２mが検出された
場合、周知の手法を用いて、これらの直線Ｌ１m，Ｌ２m
の空間的な平行性を判定する。Next, based on the range image, straight lines L1m, L
It is determined that 2m is spatially almost parallel. As described above, the position in the real space of each region forming the straight lines L1m and L2m can be specified based on the distance image. Therefore, when two straight lines L1m and L2m are detected, these straight lines L1m and L2m are detected using a known method.
To determine the spatial parallelism of.
【００４６】直線Ｌ１m，Ｌ２mが空間的に平行である場
合、基準画像におけるこれらの交点より、消失点Ｖmを
算出する。同様の手法で、比較画像における直線Ｌ１
s，Ｌ２sを検出し、比較画像におけるこれらの交点より
消失点Ｖsを算出する。そして、これらの消失点Ｖm，Ｖ
sのｉ座標値の差異から消失点視差ＤＰを算出する。When the straight lines L1m and L2m are spatially parallel to each other, the vanishing point Vm is calculated from the intersection of these in the reference image. In the same manner, the straight line L1 in the comparison image
s and L2s are detected, and the vanishing point Vs is calculated from these intersections in the comparative image. And these vanishing points Vm, V
The vanishing point parallax DP is calculated from the difference in the i coordinate value of s.
【００４７】また、他の例として、鉄道車両の前方状況
を監視するシステムに適用する場合、左右のレールを
「基準対象物」として用いることができる。図８は、鉄
道車両前方の撮像画像の一例である。左右のレールは、
距離方向に向かって互いに平行に延在している。したが
って、左右のレールを「基準対象物」として用いること
で２本の平行な直線Ｌ１，Ｌ２を特定できるため、上述
した手法で消失点視差ＤＰを調整することが可能とな
る。Further, as another example, when applied to a system for monitoring the front situation of a railway vehicle, the left and right rails can be used as "reference objects". FIG. 8 is an example of a captured image in front of the railway vehicle. The left and right rails
They extend parallel to each other in the distance direction. Therefore, since the two parallel straight lines L1 and L2 can be specified by using the left and right rails as the “reference object”, the vanishing point parallax DP can be adjusted by the above-described method.
【００４８】[0048]
【発明の効果】このように本発明では、距離情報等の三
次元情報を算出する際に用いられる消失点視差を、一対
の撮像画像平面上における平行な近似直線の交点として
算出された消失点のずれに基づいて補正している。した
がって、ステレオカメラの位置ずれが生じた場合であっ
ても、それに起因した誤差を相殺するような消失点視差
値が自動的に算出されるため、精度の高い三次元情報
（距離情報）を安定して得ることができる。As described above, in the present invention, the vanishing point parallax used when calculating three-dimensional information such as distance information is calculated as the vanishing point calculated as the intersection of parallel approximate straight lines on a pair of captured image planes. It is corrected based on the deviation. Therefore, even if the position of the stereo camera is displaced, the vanishing point parallax value that cancels the error caused by the displacement is automatically calculated, so that highly accurate three-dimensional information (distance information) is stabilized. You can get it.
【図１】ステレオ式車外監視装置の構成を示すブロック
図FIG. 1 is a block diagram showing a configuration of a stereo-type exterior monitoring device.
【図２】基準画像の白線エッジの説明図FIG. 2 is an explanatory diagram of a white line edge of a reference image.
【図３】白線モデルを示す図FIG. 3 is a diagram showing a white line model.
【図４】視差補正手順の一部を示すフローチャートFIG. 4 is a flowchart showing a part of a parallax correction procedure.
【図５】図４に続く手順を示したフローチャートFIG. 5 is a flowchart showing a procedure following FIG.
【図６】撮像画像平面上における消失点の算出説明図FIG. 6 is an explanatory diagram of calculation of a vanishing point on a captured image plane.
【図７】屋内ロボットの撮像画像の一例を示す図FIG. 7 is a diagram showing an example of a captured image of an indoor robot.
【図８】鉄道車両前方の撮像画像の一例を示す図FIG. 8 is a diagram showing an example of a captured image in front of a railway vehicle.
１ メインカメラ ２ サブカメラ ３ アナログインターフェース ３ａ ゲインコントロールアンプ ４ Ａ／Ｄコンバータ ５ 補正回路 ６ ステレオ演算回路 ７ 画像データメモリ ８ 距離データメモリ ９ マイクロコンピュータ １０ 認識部 １１ 警報装置 １２ 制御装置 １３ 補正演算部 1 Main camera 2 sub camera 3 analog interface 3a Gain control amplifier 4 A / D converter 5 Correction circuit 6 stereo operation circuit 7 Image data memory 8 distance data memory 9 Microcomputer 10 recognition unit 11 Alarm device 12 Control device 13 Correction calculator
───────────────────────────────────────────────────── フロントページの続き (51)Int.Cl.7 識別記号 ＦＩ テーマコート゛(参考） Ｂ６０Ｒ 21/00 ６２７ Ｂ６０Ｒ 21/00 ６２７ ５Ｌ０９６ Ｇ０１Ｂ 11/02 Ｇ０１Ｂ 11/02 Ｈ Ｇ０６Ｔ 1/00 ３１５ Ｇ０６Ｔ 1/00 ３１５ ３３０ ３３０Ａ 7/00 ３００ 7/00 ３００Ｅ 7/60 １５０ 7/60 １５０Ｄ Ｇ０８Ｇ 1/16 Ｇ０８Ｇ 1/16 Ｃ Ｈ０４Ｎ 7/18 Ｈ０４Ｎ 7/18 Ｊ Ｆターム(参考） 2F065 AA22 CC40 FF04 FF05 JJ03 JJ26 QQ03 QQ08 QQ13 QQ18 QQ21 QQ24 QQ29 QQ31 QQ38 SS09 2F112 AC03 AC06 BA06 CA04 CA05 FA07 FA21 FA25 FA35 FA45 5B057 AA19 BA13 CA13 CA16 CB12 CB16 DA07 DA08 DB03 DC02 DC05 DC16 DC32 5C054 AA01 AA05 CA04 CC02 CH01 EA01 EA05 FA02 FD01 HA30 5H180 CC04 CC24 CC30 LL02 LL06 5L096 BA02 CA05 FA06 FA09 FA13 FA46 FA66 FA69 HA08 JA18─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. 7 Identification code FI theme code (reference) B60R 21/00 627 B60R 21/00 627 5L096 G01B 11/02 G01B 11/02 H G06T 1/00 315 G06T 1 / 00 315 330 330A 7/00 300 7/00 300E 7/60 150 7/60 150D G08G 1/16 G08G 1/16 C H04N 7/18 H04N 7/18 JF term (reference) 2F065 AA22 CC40 FF04 FF05 JJ03 JJ26 QQ03 QQ08 QQ13 QQ18 QQ21 QQ24 QQ29 QQ31 QQ38 SS09 2F112 AC03 AC06 BA06 CA04 CA05 FA07 FA21 FA25 FA35 FA45 5B057 AA19 BA13 CA13 CA16 CB12 CB16 DA07 DA08 DB03 DC02 DC05 DC16 DC32 5C054A01 CH05 CA04 A04 A05 CHA4 CC30 LL02 LL06 5L096 BA02 CA05 FA06 FA09 FA13 FA46 FA66 FA69 HA08 JA18
Claims (6)
基づいて、ステレオマッチングにより視差を算出する視
差算出手段と、 前記視差算出手段により算出された視差と消失点視差と
に基づいて、対象物までの距離を算出する距離算出手段
と、 一方の撮像画像平面において、距離方向に延在する互い
に空間的に平行な複数の近似直線を算出し、当該近似直
線の交点から第１の消失点を算出するとともに、他方の
撮像画像平面において、距離方向に延在する互いに平行
な複数の近似直線を算出し、当該近似直線の交点から第
２の消失点を算出する消失点算出手段と、 前記消失点算出手段により算出された前記第１の消失点
と前記第２の消失点とのずれ量に基づいて、前記消失点
視差を補正する補正手段とを有することを特徴とする監
視システムの距離補正装置。1. A distance correction device of a monitoring system, wherein a stereo image pickup means for obtaining a pair of picked-up images, and a parallax calculation means for calculating a parallax by stereo matching based on the pair of picked-up images obtained by the stereo image pickup means. A distance calculation means for calculating the distance to the object based on the parallax calculated by the parallax calculation means and the vanishing point parallax, and one of the captured image planes spatially extending in the distance direction. A plurality of parallel approximate straight lines are calculated, a first vanishing point is calculated from the intersection of the approximate straight lines, and a plurality of parallel approximate straight lines extending in the distance direction are calculated in the other captured image plane, Vanishing point calculating means for calculating a second vanishing point from the intersection of the approximate straight line, the first vanishing point and the second vanishing point calculated by the vanishing point calculating means. Based on the shift amount, distance correction device of the monitoring system; and a correcting means for correcting the vanishing point parallax.
方向に延在する互いに空間的に平行な複数の基準対象物
を検出するとともに、撮像画像平面における基準対象物
の位置を特定する検出手段をさらに有し、 前記消失点算出手段は、前記検出手段によって複数の基
準対象物が検出された場合、当該基準対象物のそれぞれ
について撮像画像平面における近似直線を算出すること
を特徴とする請求項１に記載された監視システムの距離
補正装置。2. A detection for detecting a plurality of reference objects extending in the distance direction and being spatially parallel to each other from a scene image displayed in the picked-up image, and for specifying a position of the reference object on a picked-up image plane. Wherein the vanishing point calculation means calculates an approximate straight line in a captured image plane for each of the reference objects when the detection means detects a plurality of reference objects. A distance correction device for a surveillance system according to Item 1.
た道路上の左右の車線であることを特徴とする請求項２
に記載された監視システムの距離補正装置。3. The reference object is a right and left lane on a road shown in a captured image.
The distance correction device for the monitoring system described in 1.
た壁と床との境界を示す左右の境界線であることを特徴
とする請求項２に記載された監視システムの距離補正装
置。4. The distance correction device for a surveillance system according to claim 2, wherein the reference object is a left and right boundary line indicating a boundary between a wall and a floor shown in a captured image.
た線路の左右のレールであることを特徴とする請求項２
に記載された監視システムの距離補正装置。5. The reference object is the left and right rails of a railroad track shown in a captured image.
The distance correction device for the monitoring system described in 1.
づいて、ステレオマッチングにより視差を算出するステ
ップと、 前記視差と消失点視差とに基づいて、対象物までの距離
を算出するステップと、 一方の撮像画像平面において、距離方向に延在する互い
に空間的に平行な複数の近似直線を算出し、当該近似直
線の交点から第１の消失点を算出するステップと、 他方の撮像画像平面において、距離方向に延在する互い
に空間的に平行な複数の近似直線を算出し、当該近似直
線の交点から第２の消失点を算出するステップと、 前記第１の消失点と前記第２の消失点とのずれ量に基づ
いて、前記消失点視差を補正するステップとを有するこ
とを特徴とする監視システムの距離補正方法。6. A distance correction method for a surveillance system, which calculates a parallax by stereo matching based on a pair of captured images of the same scene at the same time, and based on the parallax and the vanishing point parallax. And calculating a distance to the object, and calculating a plurality of spatially parallel approximate straight lines extending in the distance direction on one of the captured image planes, and calculating the first disappearance from the intersection of the approximate straight lines. Calculating a point, and calculating a plurality of spatially parallel approximate straight lines extending in the distance direction on the other captured image plane, and calculating a second vanishing point from the intersection of the approximate straight lines. A step of correcting the vanishing point parallax on the basis of the amount of deviation between the first vanishing point and the second vanishing point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2001277998A JP4803927B2 (en) | 2001-09-13 | 2001-09-13 | Distance correction apparatus and distance correction method for monitoring system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2001277998A JP4803927B2 (en) | 2001-09-13 | 2001-09-13 | Distance correction apparatus and distance correction method for monitoring system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| JP2003083742A true JP2003083742A (en) | 2003-03-19 |
| JP4803927B2 JP4803927B2 (en) | 2011-10-26 |
Family
ID=19102438
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| JP2001277998A Expired - Fee Related JP4803927B2 (en) | 2001-09-13 | 2001-09-13 | Distance correction apparatus and distance correction method for monitoring system |
Country Status (1)
| Country | Link |
|---|---|
| JP (1) | JP4803927B2 (en) |
Cited By (45)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006287789A (en) * | 2005-04-04 | 2006-10-19 | Toyota Motor Corp | Attitude corrector for vehicle-mounted camera, and correcting method thereof |
| JP2008509619A (en) * | 2004-08-11 | 2008-03-27 | ルノー・エス・アー・エス | Method for automatically calibrating stereo vision equipment |
| US7385680B2 (en) | 2004-06-03 | 2008-06-10 | Matsushita Electric Industrial Co., Ltd. | Camera module |
| JP2009008539A (en) * | 2007-06-28 | 2009-01-15 | Fuji Heavy Ind Ltd | Stereo image processing apparatus |
| JP2009080113A (en) * | 2007-09-25 | 2009-04-16 | Seiko Epson Corp | Distance estimation method, distance estimation device, imaging device, and computer readable medium |
| WO2011063785A1 (en) * | 2009-11-25 | 2011-06-03 | Conti Temic Microelectronic Gmbh Patente & Lizenzen | Method for estimating the roll angle in a travelling vehicle |
| US20110285850A1 (en) * | 2007-08-17 | 2011-11-24 | Yuesheng Lu | Vehicular imaging system |
| WO2013030932A1 (en) * | 2011-08-29 | 2013-03-07 | パイオニア株式会社 | Navigation device, image display device, server, adjustment device, and forward image display method |
| US8446470B2 (en) | 2007-10-04 | 2013-05-21 | Magna Electronics, Inc. | Combined RGB and IR imaging sensor |
| US8451107B2 (en) | 2007-09-11 | 2013-05-28 | Magna Electronics, Inc. | Imaging system for vehicle |
| US8593521B2 (en) | 2004-04-15 | 2013-11-26 | Magna Electronics Inc. | Imaging system for vehicle |
| US8599001B2 (en) | 1993-02-26 | 2013-12-03 | Magna Electronics Inc. | Vehicular vision system |
| US8629768B2 (en) | 1999-08-12 | 2014-01-14 | Donnelly Corporation | Vehicle vision system |
| US8636393B2 (en) | 2006-08-11 | 2014-01-28 | Magna Electronics Inc. | Driver assistance system for vehicle |
| US8637801B2 (en) | 1996-03-25 | 2014-01-28 | Magna Electronics Inc. | Driver assistance system for a vehicle |
| US8643724B2 (en) | 1996-05-22 | 2014-02-04 | Magna Electronics Inc. | Multi-camera vision system for a vehicle |
| US8665079B2 (en) | 2002-05-03 | 2014-03-04 | Magna Electronics Inc. | Vision system for vehicle |
| US8977008B2 (en) | 2004-09-30 | 2015-03-10 | Donnelly Corporation | Driver assistance system for vehicle |
| US9014904B2 (en) | 2004-12-23 | 2015-04-21 | Magna Electronics Inc. | Driver assistance system for vehicle |
| US9041806B2 (en) | 2009-09-01 | 2015-05-26 | Magna Electronics Inc. | Imaging and display system for vehicle |
| US9085261B2 (en) | 2011-01-26 | 2015-07-21 | Magna Electronics Inc. | Rear vision system with trailer angle detection |
| US9146898B2 (en) | 2011-10-27 | 2015-09-29 | Magna Electronics Inc. | Driver assist system with algorithm switching |
| CN104949619A (en) * | 2014-03-28 | 2015-09-30 | 富士重工业株式会社 | Stereo image processing device for vehicle |
| JP2015184121A (en) * | 2014-03-24 | 2015-10-22 | 国立大学法人東京工業大学 | distance calculation device |
| US9191574B2 (en) | 2001-07-31 | 2015-11-17 | Magna Electronics Inc. | Vehicular vision system |
| KR101574020B1 (en) * | 2014-06-24 | 2015-12-03 | 국민대학교산학협력단 | Method and Apparatus for stereo rectification of left and right view images captured from stereo cameras |
| US9245448B2 (en) | 2001-07-31 | 2016-01-26 | Magna Electronics Inc. | Driver assistance system for a vehicle |
| US9264672B2 (en) | 2010-12-22 | 2016-02-16 | Magna Mirrors Of America, Inc. | Vision display system for vehicle |
| DE102015220884A1 (en) | 2014-10-27 | 2016-04-28 | Denso Corporation | Image processing device |
| US9446713B2 (en) | 2012-09-26 | 2016-09-20 | Magna Electronics Inc. | Trailer angle detection system |
| US9495876B2 (en) | 2009-07-27 | 2016-11-15 | Magna Electronics Inc. | Vehicular camera with on-board microcontroller |
| US9508014B2 (en) | 2013-05-06 | 2016-11-29 | Magna Electronics Inc. | Vehicular multi-camera vision system |
| WO2016203989A1 (en) * | 2015-06-17 | 2016-12-22 | ソニー株式会社 | Image processing device and image processing method |
| US9558409B2 (en) | 2012-09-26 | 2017-01-31 | Magna Electronics Inc. | Vehicle vision system with trailer angle detection |
| WO2017056484A1 (en) * | 2015-09-28 | 2017-04-06 | 京セラ株式会社 | Image processing device, stereo camera device, vehicle, and image processing method |
| US9743002B2 (en) | 2012-11-19 | 2017-08-22 | Magna Electronics Inc. | Vehicle vision system with enhanced display functions |
| JP2018005891A (en) * | 2016-06-28 | 2018-01-11 | キヤノン株式会社 | Image processing device, imaging device, image processing method, and program |
| US9900522B2 (en) | 2010-12-01 | 2018-02-20 | Magna Electronics Inc. | System and method of establishing a multi-camera image using pixel remapping |
| WO2018173908A1 (en) | 2017-03-24 | 2018-09-27 | 日立オートモティブシステムズ株式会社 | Stereo image processing device |
| JP2019082468A (en) * | 2017-10-27 | 2019-05-30 | キヤノン株式会社 | Imaging device and control method for imaging device |
| WO2019163493A1 (en) * | 2018-02-20 | 2019-08-29 | 日立オートモティブシステムズ株式会社 | Image capturing device |
| US10457209B2 (en) | 2012-02-22 | 2019-10-29 | Magna Electronics Inc. | Vehicle vision system with multi-paned view |
| US10793067B2 (en) | 2011-07-26 | 2020-10-06 | Magna Electronics Inc. | Imaging system for vehicle |
| WO2021215199A1 (en) * | 2020-04-22 | 2021-10-28 | ソニーセミコンダクタソリューションズ株式会社 | Information processing device, image capturing system, information processing method, and computer program |
| US11277558B2 (en) | 2016-02-01 | 2022-03-15 | Magna Electronics Inc. | Vehicle vision system with master-slave camera configuration |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06341837A (en) * | 1993-06-01 | 1994-12-13 | Matsushita Electric Ind Co Ltd | Distance-between-cars measuring apparatus, camera-position correction device and collision warning device |
| JPH09126759A (en) * | 1995-11-06 | 1997-05-16 | Mitsubishi Motors Corp | Range finding method by on-vehicle camera and range finder using the camera |
| JPH09133525A (en) * | 1995-11-10 | 1997-05-20 | Nippon Soken Inc | Distance measuring device |
| JP2001160137A (en) * | 1999-09-22 | 2001-06-12 | Fuji Heavy Ind Ltd | Device for correcting distance of monitoring system and device for correcting vanishing point of monitoring system |
-
2001
- 2001-09-13 JP JP2001277998A patent/JP4803927B2/en not_active Expired - Fee Related
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH06341837A (en) * | 1993-06-01 | 1994-12-13 | Matsushita Electric Ind Co Ltd | Distance-between-cars measuring apparatus, camera-position correction device and collision warning device |
| JPH09126759A (en) * | 1995-11-06 | 1997-05-16 | Mitsubishi Motors Corp | Range finding method by on-vehicle camera and range finder using the camera |
| JPH09133525A (en) * | 1995-11-10 | 1997-05-20 | Nippon Soken Inc | Distance measuring device |
| JP2001160137A (en) * | 1999-09-22 | 2001-06-12 | Fuji Heavy Ind Ltd | Device for correcting distance of monitoring system and device for correcting vanishing point of monitoring system |
Cited By (170)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8599001B2 (en) | 1993-02-26 | 2013-12-03 | Magna Electronics Inc. | Vehicular vision system |
| US8917169B2 (en) | 1993-02-26 | 2014-12-23 | Magna Electronics Inc. | Vehicular vision system |
| US8993951B2 (en) | 1996-03-25 | 2015-03-31 | Magna Electronics Inc. | Driver assistance system for a vehicle |
| US8637801B2 (en) | 1996-03-25 | 2014-01-28 | Magna Electronics Inc. | Driver assistance system for a vehicle |
| US8842176B2 (en) | 1996-05-22 | 2014-09-23 | Donnelly Corporation | Automatic vehicle exterior light control |
| US9131120B2 (en) | 1996-05-22 | 2015-09-08 | Magna Electronics Inc. | Multi-camera vision system for a vehicle |
| US8643724B2 (en) | 1996-05-22 | 2014-02-04 | Magna Electronics Inc. | Multi-camera vision system for a vehicle |
| US9436880B2 (en) | 1999-08-12 | 2016-09-06 | Magna Electronics Inc. | Vehicle vision system |
| US8629768B2 (en) | 1999-08-12 | 2014-01-14 | Donnelly Corporation | Vehicle vision system |
| US9834142B2 (en) | 2001-07-31 | 2017-12-05 | Magna Electronics Inc. | Driving assist system for vehicle |
| US9191574B2 (en) | 2001-07-31 | 2015-11-17 | Magna Electronics Inc. | Vehicular vision system |
| US10406980B2 (en) | 2001-07-31 | 2019-09-10 | Magna Electronics Inc. | Vehicular lane change system |
| US9463744B2 (en) | 2001-07-31 | 2016-10-11 | Magna Electronics Inc. | Driver assistance system for a vehicle |
| US9656608B2 (en) | 2001-07-31 | 2017-05-23 | Magna Electronics Inc. | Driver assist system for vehicle |
| US10611306B2 (en) | 2001-07-31 | 2020-04-07 | Magna Electronics Inc. | Video processor module for vehicle |
| US10046702B2 (en) | 2001-07-31 | 2018-08-14 | Magna Electronics Inc. | Control system for vehicle |
| US9245448B2 (en) | 2001-07-31 | 2016-01-26 | Magna Electronics Inc. | Driver assistance system for a vehicle |
| US9376060B2 (en) | 2001-07-31 | 2016-06-28 | Magna Electronics Inc. | Driver assist system for vehicle |
| US10099610B2 (en) | 2001-07-31 | 2018-10-16 | Magna Electronics Inc. | Driver assistance system for a vehicle |
| US8665079B2 (en) | 2002-05-03 | 2014-03-04 | Magna Electronics Inc. | Vision system for vehicle |
| US11203340B2 (en) | 2002-05-03 | 2021-12-21 | Magna Electronics Inc. | Vehicular vision system using side-viewing camera |
| US9171217B2 (en) | 2002-05-03 | 2015-10-27 | Magna Electronics Inc. | Vision system for vehicle |
| US9555803B2 (en) | 2002-05-03 | 2017-01-31 | Magna Electronics Inc. | Driver assistance system for vehicle |
| US10118618B2 (en) | 2002-05-03 | 2018-11-06 | Magna Electronics Inc. | Vehicular control system using cameras and radar sensor |
| US10683008B2 (en) | 2002-05-03 | 2020-06-16 | Magna Electronics Inc. | Vehicular driving assist system using forward-viewing camera |
| US9834216B2 (en) | 2002-05-03 | 2017-12-05 | Magna Electronics Inc. | Vehicular control system using cameras and radar sensor |
| US9643605B2 (en) | 2002-05-03 | 2017-05-09 | Magna Electronics Inc. | Vision system for vehicle |
| US10351135B2 (en) | 2002-05-03 | 2019-07-16 | Magna Electronics Inc. | Vehicular control system using cameras and radar sensor |
| US10110860B1 (en) | 2004-04-15 | 2018-10-23 | Magna Electronics Inc. | Vehicular control system |
| US8593521B2 (en) | 2004-04-15 | 2013-11-26 | Magna Electronics Inc. | Imaging system for vehicle |
| US9428192B2 (en) | 2004-04-15 | 2016-08-30 | Magna Electronics Inc. | Vision system for vehicle |
| US10015452B1 (en) | 2004-04-15 | 2018-07-03 | Magna Electronics Inc. | Vehicular control system |
| US9948904B2 (en) | 2004-04-15 | 2018-04-17 | Magna Electronics Inc. | Vision system for vehicle |
| US10187615B1 (en) | 2004-04-15 | 2019-01-22 | Magna Electronics Inc. | Vehicular control system |
| US8818042B2 (en) | 2004-04-15 | 2014-08-26 | Magna Electronics Inc. | Driver assistance system for vehicle |
| US10306190B1 (en) | 2004-04-15 | 2019-05-28 | Magna Electronics Inc. | Vehicular control system |
| US9736435B2 (en) | 2004-04-15 | 2017-08-15 | Magna Electronics Inc. | Vision system for vehicle |
| US9008369B2 (en) | 2004-04-15 | 2015-04-14 | Magna Electronics Inc. | Vision system for vehicle |
| US10462426B2 (en) | 2004-04-15 | 2019-10-29 | Magna Electronics Inc. | Vehicular control system |
| US11847836B2 (en) | 2004-04-15 | 2023-12-19 | Magna Electronics Inc. | Vehicular control system with road curvature determination |
| US10735695B2 (en) | 2004-04-15 | 2020-08-04 | Magna Electronics Inc. | Vehicular control system with traffic lane detection |
| US9191634B2 (en) | 2004-04-15 | 2015-11-17 | Magna Electronics Inc. | Vision system for vehicle |
| US9609289B2 (en) | 2004-04-15 | 2017-03-28 | Magna Electronics Inc. | Vision system for vehicle |
| US11503253B2 (en) | 2004-04-15 | 2022-11-15 | Magna Electronics Inc. | Vehicular control system with traffic lane detection |
| US7385680B2 (en) | 2004-06-03 | 2008-06-10 | Matsushita Electric Industrial Co., Ltd. | Camera module |
| JP2008509619A (en) * | 2004-08-11 | 2008-03-27 | ルノー・エス・アー・エス | Method for automatically calibrating stereo vision equipment |
| US10623704B2 (en) | 2004-09-30 | 2020-04-14 | Donnelly Corporation | Driver assistance system for vehicle |
| US8977008B2 (en) | 2004-09-30 | 2015-03-10 | Donnelly Corporation | Driver assistance system for vehicle |
| US9193303B2 (en) | 2004-12-23 | 2015-11-24 | Magna Electronics Inc. | Driver assistance system for vehicle |
| US9014904B2 (en) | 2004-12-23 | 2015-04-21 | Magna Electronics Inc. | Driver assistance system for vehicle |
| US11308720B2 (en) | 2004-12-23 | 2022-04-19 | Magna Electronics Inc. | Vehicular imaging system |
| US9940528B2 (en) | 2004-12-23 | 2018-04-10 | Magna Electronics Inc. | Driver assistance system for vehicle |
| US10509972B2 (en) | 2004-12-23 | 2019-12-17 | Magna Electronics Inc. | Vehicular vision system |
| JP4622637B2 (en) * | 2005-04-04 | 2011-02-02 | トヨタ自動車株式会社 | In-vehicle camera attitude correction device and in-vehicle camera attitude correction method |
| JP2006287789A (en) * | 2005-04-04 | 2006-10-19 | Toyota Motor Corp | Attitude corrector for vehicle-mounted camera, and correcting method thereof |
| US11623559B2 (en) | 2006-08-11 | 2023-04-11 | Magna Electronics Inc. | Vehicular forward viewing image capture system |
| US8636393B2 (en) | 2006-08-11 | 2014-01-28 | Magna Electronics Inc. | Driver assistance system for vehicle |
| US10071676B2 (en) | 2006-08-11 | 2018-09-11 | Magna Electronics Inc. | Vision system for vehicle |
| US11148583B2 (en) | 2006-08-11 | 2021-10-19 | Magna Electronics Inc. | Vehicular forward viewing image capture system |
| US11396257B2 (en) | 2006-08-11 | 2022-07-26 | Magna Electronics Inc. | Vehicular forward viewing image capture system |
| US11951900B2 (en) | 2006-08-11 | 2024-04-09 | Magna Electronics Inc. | Vehicular forward viewing image capture system |
| US10787116B2 (en) | 2006-08-11 | 2020-09-29 | Magna Electronics Inc. | Adaptive forward lighting system for vehicle comprising a control that adjusts the headlamp beam in response to processing of image data captured by a camera |
| US8417022B2 (en) | 2007-06-28 | 2013-04-09 | Fuji Jukogyo Kabushiki Kaisha | Stereo-image processing apparatus |
| JP2009008539A (en) * | 2007-06-28 | 2009-01-15 | Fuji Heavy Ind Ltd | Stereo image processing apparatus |
| DE102008030555A1 (en) | 2007-06-28 | 2009-01-29 | Fuji Jukogyo K.K. | Device for processing stereo images |
| DE102008030555B4 (en) * | 2007-06-28 | 2014-10-09 | Fuji Jukogyo K.K. | Device for processing stereo images |
| US20180260973A1 (en) * | 2007-08-17 | 2018-09-13 | Magna Electronics, Inc. | Vehicular imaging system with blockage determination and misalignment correction |
| US9972100B2 (en) | 2007-08-17 | 2018-05-15 | Magna Electronics Inc. | Vehicular imaging system comprising an imaging device with a single image sensor and image processor for determining a totally blocked state or partially blocked state of the single image sensor as well as an automatic correction for misalignment of the imaging device |
| US20220262038A1 (en) * | 2007-08-17 | 2022-08-18 | Magna Electronics, Inc. | Vehicular imaging system with misalignment correction of camera |
| US11328447B2 (en) | 2007-08-17 | 2022-05-10 | Magna Electronics Inc. | Method of blockage determination and misalignment correction for vehicular vision system |
| US10726578B2 (en) | 2007-08-17 | 2020-07-28 | Magna Electronics Inc. | Vehicular imaging system with blockage determination and misalignment correction |
| US11908166B2 (en) | 2007-08-17 | 2024-02-20 | Magna Electronics Inc. | Vehicular imaging system with misalignment correction of camera |
| US9018577B2 (en) * | 2007-08-17 | 2015-04-28 | Magna Electronics Inc. | Vehicular imaging system with camera misalignment correction and capturing image data at different resolution levels dependent on distance to object in field of view |
| US20150224917A1 (en) * | 2007-08-17 | 2015-08-13 | Magna Electronics, Inc. | Vehicular imaging system |
| US20110285850A1 (en) * | 2007-08-17 | 2011-11-24 | Yuesheng Lu | Vehicular imaging system |
| US11613209B2 (en) | 2007-09-11 | 2023-03-28 | Magna Electronics Inc. | System and method for guiding reversing of a vehicle toward a trailer hitch |
| US9796332B2 (en) | 2007-09-11 | 2017-10-24 | Magna Electronics Inc. | Imaging system for vehicle |
| US10766417B2 (en) | 2007-09-11 | 2020-09-08 | Magna Electronics Inc. | Imaging system for vehicle |
| US8451107B2 (en) | 2007-09-11 | 2013-05-28 | Magna Electronics, Inc. | Imaging system for vehicle |
| JP2009080113A (en) * | 2007-09-25 | 2009-04-16 | Seiko Epson Corp | Distance estimation method, distance estimation device, imaging device, and computer readable medium |
| US10003755B2 (en) | 2007-10-04 | 2018-06-19 | Magna Electronics Inc. | Imaging system for vehicle |
| US10616507B2 (en) | 2007-10-04 | 2020-04-07 | Magna Electronics Inc. | Imaging system for vehicle |
| US8446470B2 (en) | 2007-10-04 | 2013-05-21 | Magna Electronics, Inc. | Combined RGB and IR imaging sensor |
| US11165975B2 (en) | 2007-10-04 | 2021-11-02 | Magna Electronics Inc. | Imaging system for vehicle |
| US8908040B2 (en) | 2007-10-04 | 2014-12-09 | Magna Electronics Inc. | Imaging system for vehicle |
| US10875526B2 (en) | 2009-07-27 | 2020-12-29 | Magna Electronics Inc. | Vehicular vision system |
| US10106155B2 (en) | 2009-07-27 | 2018-10-23 | Magna Electronics Inc. | Vehicular camera with on-board microcontroller |
| US9495876B2 (en) | 2009-07-27 | 2016-11-15 | Magna Electronics Inc. | Vehicular camera with on-board microcontroller |
| US11518377B2 (en) | 2009-07-27 | 2022-12-06 | Magna Electronics Inc. | Vehicular vision system |
| US10875455B2 (en) | 2009-09-01 | 2020-12-29 | Magna Electronics Inc. | Vehicular vision system |
| US10053012B2 (en) | 2009-09-01 | 2018-08-21 | Magna Electronics Inc. | Imaging and display system for vehicle |
| US11794651B2 (en) | 2009-09-01 | 2023-10-24 | Magna Electronics Inc. | Vehicular vision system |
| US11285877B2 (en) | 2009-09-01 | 2022-03-29 | Magna Electronics Inc. | Vehicular vision system |
| US9041806B2 (en) | 2009-09-01 | 2015-05-26 | Magna Electronics Inc. | Imaging and display system for vehicle |
| US9789821B2 (en) | 2009-09-01 | 2017-10-17 | Magna Electronics Inc. | Imaging and display system for vehicle |
| US10300856B2 (en) | 2009-09-01 | 2019-05-28 | Magna Electronics Inc. | Vehicular display system |
| WO2011063785A1 (en) * | 2009-11-25 | 2011-06-03 | Conti Temic Microelectronic Gmbh Patente & Lizenzen | Method for estimating the roll angle in a travelling vehicle |
| US8824741B2 (en) | 2009-11-25 | 2014-09-02 | Conti Temic Microelectronic Gmbh | Method for estimating the roll angle in a travelling vehicle |
| US11553140B2 (en) | 2010-12-01 | 2023-01-10 | Magna Electronics Inc. | Vehicular vision system with multiple cameras |
| US10868974B2 (en) | 2010-12-01 | 2020-12-15 | Magna Electronics Inc. | Method for determining alignment of vehicular cameras |
| US9900522B2 (en) | 2010-12-01 | 2018-02-20 | Magna Electronics Inc. | System and method of establishing a multi-camera image using pixel remapping |
| US10486597B1 (en) | 2010-12-22 | 2019-11-26 | Magna Electronics Inc. | Vehicular vision system with rear backup video display |
| US10589678B1 (en) | 2010-12-22 | 2020-03-17 | Magna Electronics Inc. | Vehicular rear backup vision system with video display |
| US11708026B2 (en) | 2010-12-22 | 2023-07-25 | Magna Electronics Inc. | Vehicular rear backup system with video display |
| US10336255B2 (en) | 2010-12-22 | 2019-07-02 | Magna Electronics Inc. | Vehicular vision system with rear backup video display |
| US9731653B2 (en) | 2010-12-22 | 2017-08-15 | Magna Electronics Inc. | Vision display system for vehicle |
| US11155211B2 (en) | 2010-12-22 | 2021-10-26 | Magna Electronics Inc. | Vehicular multi-camera surround view system with video display |
| US10144352B2 (en) | 2010-12-22 | 2018-12-04 | Magna Electronics Inc. | Vision display system for vehicle |
| US9598014B2 (en) | 2010-12-22 | 2017-03-21 | Magna Electronics Inc. | Vision display system for vehicle |
| US11548444B2 (en) | 2010-12-22 | 2023-01-10 | Magna Electronics Inc. | Vehicular multi-camera surround view system with video display |
| US9469250B2 (en) | 2010-12-22 | 2016-10-18 | Magna Electronics Inc. | Vision display system for vehicle |
| US10814785B2 (en) | 2010-12-22 | 2020-10-27 | Magna Electronics Inc. | Vehicular rear backup vision system with video display |
| US9264672B2 (en) | 2010-12-22 | 2016-02-16 | Magna Mirrors Of America, Inc. | Vision display system for vehicle |
| US10858042B2 (en) | 2011-01-26 | 2020-12-08 | Magna Electronics Inc. | Trailering assist system with trailer angle detection |
| US11820424B2 (en) | 2011-01-26 | 2023-11-21 | Magna Electronics Inc. | Trailering assist system with trailer angle detection |
| US9085261B2 (en) | 2011-01-26 | 2015-07-21 | Magna Electronics Inc. | Rear vision system with trailer angle detection |
| US9950738B2 (en) | 2011-01-26 | 2018-04-24 | Magna Electronics Inc. | Trailering assist system with trailer angle detection |
| US10793067B2 (en) | 2011-07-26 | 2020-10-06 | Magna Electronics Inc. | Imaging system for vehicle |
| US11285873B2 (en) | 2011-07-26 | 2022-03-29 | Magna Electronics Inc. | Method for generating surround view images derived from image data captured by cameras of a vehicular surround view vision system |
| WO2013030932A1 (en) * | 2011-08-29 | 2013-03-07 | パイオニア株式会社 | Navigation device, image display device, server, adjustment device, and forward image display method |
| US9919705B2 (en) | 2011-10-27 | 2018-03-20 | Magna Electronics Inc. | Driver assist system with image processing and wireless communication |
| US11279343B2 (en) | 2011-10-27 | 2022-03-22 | Magna Electronics Inc. | Vehicular control system with image processing and wireless communication |
| US11673546B2 (en) | 2011-10-27 | 2023-06-13 | Magna Electronics Inc. | Vehicular control system with image processing and wireless communication |
| US9146898B2 (en) | 2011-10-27 | 2015-09-29 | Magna Electronics Inc. | Driver assist system with algorithm switching |
| US10457209B2 (en) | 2012-02-22 | 2019-10-29 | Magna Electronics Inc. | Vehicle vision system with multi-paned view |
| US11607995B2 (en) | 2012-02-22 | 2023-03-21 | Magna Electronics Inc. | Vehicular display system with multi-paned image display |
| US11007937B2 (en) | 2012-02-22 | 2021-05-18 | Magna Electronics Inc. | Vehicular display system with multi-paned image display |
| US10909393B2 (en) | 2012-09-26 | 2021-02-02 | Magna Electronics Inc. | Vehicular control system with trailering assist function |
| US11410431B2 (en) | 2012-09-26 | 2022-08-09 | Magna Electronics Inc. | Vehicular control system with trailering assist function |
| US10089541B2 (en) | 2012-09-26 | 2018-10-02 | Magna Electronics Inc. | Vehicular control system with trailering assist function |
| US9802542B2 (en) | 2012-09-26 | 2017-10-31 | Magna Electronics Inc. | Trailer angle detection system calibration |
| US10800332B2 (en) | 2012-09-26 | 2020-10-13 | Magna Electronics Inc. | Trailer driving assist system |
| US9558409B2 (en) | 2012-09-26 | 2017-01-31 | Magna Electronics Inc. | Vehicle vision system with trailer angle detection |
| US10586119B2 (en) | 2012-09-26 | 2020-03-10 | Magna Electronics Inc. | Vehicular control system with trailering assist function |
| US11285875B2 (en) | 2012-09-26 | 2022-03-29 | Magna Electronics Inc. | Method for dynamically calibrating a vehicular trailer angle detection system |
| US9779313B2 (en) | 2012-09-26 | 2017-10-03 | Magna Electronics Inc. | Vehicle vision system with trailer angle detection |
| US9446713B2 (en) | 2012-09-26 | 2016-09-20 | Magna Electronics Inc. | Trailer angle detection system |
| US11872939B2 (en) | 2012-09-26 | 2024-01-16 | Magna Electronics Inc. | Vehicular trailer angle detection system |
| US10300855B2 (en) | 2012-09-26 | 2019-05-28 | Magna Electronics Inc. | Trailer driving assist system |
| US9743002B2 (en) | 2012-11-19 | 2017-08-22 | Magna Electronics Inc. | Vehicle vision system with enhanced display functions |
| US10321064B2 (en) | 2012-11-19 | 2019-06-11 | Magna Electronics Inc. | Vehicular vision system with enhanced display functions |
| US10104298B2 (en) | 2012-11-19 | 2018-10-16 | Magna Electronics Inc. | Vehicle vision system with enhanced display functions |
| US10057489B2 (en) | 2013-05-06 | 2018-08-21 | Magna Electronics Inc. | Vehicular multi-camera vision system |
| US11616910B2 (en) | 2013-05-06 | 2023-03-28 | Magna Electronics Inc. | Vehicular vision system with video display |
| US11050934B2 (en) | 2013-05-06 | 2021-06-29 | Magna Electronics Inc. | Method for displaying video images for a vehicular vision system |
| US9769381B2 (en) | 2013-05-06 | 2017-09-19 | Magna Electronics Inc. | Vehicular multi-camera vision system |
| US9508014B2 (en) | 2013-05-06 | 2016-11-29 | Magna Electronics Inc. | Vehicular multi-camera vision system |
| US10574885B2 (en) | 2013-05-06 | 2020-02-25 | Magna Electronics Inc. | Method for displaying video images for a vehicular vision system |
| JP2015184121A (en) * | 2014-03-24 | 2015-10-22 | 国立大学法人東京工業大学 | distance calculation device |
| CN104949619A (en) * | 2014-03-28 | 2015-09-30 | 富士重工业株式会社 | Stereo image processing device for vehicle |
| JP2015190921A (en) * | 2014-03-28 | 2015-11-02 | 富士重工業株式会社 | Vehicle stereo-image processing apparatus |
| US9378553B2 (en) | 2014-03-28 | 2016-06-28 | Fuji Jukogyo Kabushiki Kaisha | Stereo image processing device for vehicle |
| KR101574020B1 (en) * | 2014-06-24 | 2015-12-03 | 국민대학교산학협력단 | Method and Apparatus for stereo rectification of left and right view images captured from stereo cameras |
| DE102015220884B4 (en) | 2014-10-27 | 2022-03-31 | Denso Corporation | image processing device |
| DE102015220884A1 (en) | 2014-10-27 | 2016-04-28 | Denso Corporation | Image processing device |
| US9852502B2 (en) | 2014-10-27 | 2017-12-26 | Denso Corporation | Image processing apparatus |
| WO2016203989A1 (en) * | 2015-06-17 | 2016-12-22 | ソニー株式会社 | Image processing device and image processing method |
| JPWO2017056484A1 (en) * | 2015-09-28 | 2018-04-19 | 京セラ株式会社 | Image processing apparatus, stereo camera apparatus, vehicle, and image processing method |
| WO2017056484A1 (en) * | 2015-09-28 | 2017-04-06 | 京セラ株式会社 | Image processing device, stereo camera device, vehicle, and image processing method |
| US10558867B2 (en) | 2015-09-28 | 2020-02-11 | Kyocera Corporation | Image processing apparatus, stereo camera apparatus, vehicle, and image processing method |
| US11277558B2 (en) | 2016-02-01 | 2022-03-15 | Magna Electronics Inc. | Vehicle vision system with master-slave camera configuration |
| JP2018005891A (en) * | 2016-06-28 | 2018-01-11 | キヤノン株式会社 | Image processing device, imaging device, image processing method, and program |
| WO2018173908A1 (en) | 2017-03-24 | 2018-09-27 | 日立オートモティブシステムズ株式会社 | Stereo image processing device |
| US11682131B2 (en) | 2017-10-27 | 2023-06-20 | Canon Kabushiki Kaisha | Image capturing apparatus and method of controlling image capturing apparatus |
| JP2019082468A (en) * | 2017-10-27 | 2019-05-30 | キヤノン株式会社 | Imaging device and control method for imaging device |
| JPWO2019163493A1 (en) * | 2018-02-20 | 2021-02-04 | 日立オートモティブシステムズ株式会社 | Imaging device |
| US11308337B2 (en) | 2018-02-20 | 2022-04-19 | Hitachi Astemo, Ltd. | Image capturing device |
| WO2019163493A1 (en) * | 2018-02-20 | 2019-08-29 | 日立オートモティブシステムズ株式会社 | Image capturing device |
| JP6999022B2 (en) | 2018-02-20 | 2022-01-18 | 日立Astemo株式会社 | Imaging device |
| WO2021215199A1 (en) * | 2020-04-22 | 2021-10-28 | ソニーセミコンダクタソリューションズ株式会社 | Information processing device, image capturing system, information processing method, and computer program |
Also Published As
| Publication number | Publication date |
|---|---|
| JP4803927B2 (en) | 2011-10-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP4803927B2 (en) | Distance correction apparatus and distance correction method for monitoring system | |
| JP4573977B2 (en) | Distance correction device for monitoring system and vanishing point correction device for monitoring system | |
| JP4363295B2 (en) | Plane estimation method using stereo images | |
| JP4032843B2 (en) | Monitoring system and monitoring method, distance correction device and distance correction method in the monitoring system | |
| JP2012075060A (en) | Image processing device, and imaging device using the same | |
| JP2001092970A (en) | Lane recognizing device | |
| JP2008033750A (en) | Object inclination detector | |
| JPH1166319A (en) | Method and device for detecting traveling object, method and device for recognizing traveling object, and method and device for detecting person | |
| JP4067340B2 (en) | Object recognition device and object recognition method | |
| JP4956099B2 (en) | Wall detector | |
| JP2000293693A (en) | Obstacle detecting method and device | |
| KR100574227B1 (en) | Apparatus and method for separating object motion from camera motion | |
| JP2003077099A (en) | Preceding vehicle recognition system | |
| JP4256992B2 (en) | Obstacle detection device | |
| JP2536986B2 (en) | Inter-vehicle distance detector | |
| JPH1144533A (en) | Preceding vehicle detector | |
| JP2006234682A (en) | Object discriminating device | |
| JP2002367077A (en) | Device and method for deciding traffic congestion | |
| JP3586938B2 (en) | In-vehicle distance measuring device | |
| JP5903901B2 (en) | Vehicle position calculation device | |
| JP4462533B2 (en) | Road lane detection device | |
| KR100844640B1 (en) | Method for object recognizing and distance measuring | |
| JP4018950B2 (en) | Stereo camera misalignment inspection apparatus and misalignment inspection method | |
| JP4171283B2 (en) | Terrain recognition device and terrain recognition method | |
| JP2003294422A (en) | Object recognition apparatus and method therefor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| A621 | Written request for application examination |
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20080822 |
|
| RD02 | Notification of acceptance of power of attorney |
Free format text: JAPANESE INTERMEDIATE CODE: A7422 Effective date: 20101019 |
|
| RD04 | Notification of resignation of power of attorney |
Free format text: JAPANESE INTERMEDIATE CODE: A7424 Effective date: 20101116 |
|
| A977 | Report on retrieval |
Free format text: JAPANESE INTERMEDIATE CODE: A971007 Effective date: 20110214 |
|
| A131 | Notification of reasons for refusal |
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20110222 |
|
| A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20110407 |
|
| TRDD | Decision of grant or rejection written | ||
| A01 | Written decision to grant a patent or to grant a registration (utility model) |
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20110809 |
|
| A01 | Written decision to grant a patent or to grant a registration (utility model) |
Free format text: JAPANESE INTERMEDIATE CODE: A01 |
|
| A61 | First payment of annual fees (during grant procedure) |
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20110809 |
|
| R150 | Certificate of patent or registration of utility model |
Ref document number: 4803927 Country of ref document: JP Free format text: JAPANESE INTERMEDIATE CODE: R150 Free format text: JAPANESE INTERMEDIATE CODE: R150 |
|
| FPAY | Renewal fee payment (event date is renewal date of database) |
Free format text: PAYMENT UNTIL: 20140819 Year of fee payment: 3 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| S531 | Written request for registration of change of domicile |
Free format text: JAPANESE INTERMEDIATE CODE: R313531 |
|
| R350 | Written notification of registration of transfer |
Free format text: JAPANESE INTERMEDIATE CODE: R350 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| S533 | Written request for registration of change of name |
Free format text: JAPANESE INTERMEDIATE CODE: R313533 |
|
| R350 | Written notification of registration of transfer |
Free format text: JAPANESE INTERMEDIATE CODE: R350 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| R250 | Receipt of annual fees |
Free format text: JAPANESE INTERMEDIATE CODE: R250 |
|
| LAPS | Cancellation because of no payment of annual fees |