JP3551920B2 - In-vehicle camera calibration device and calibration method - Google Patents

Description

この式を、具体的に表わすと次式となる。
【００２６】
【数２】

Ｐｐは斉次行列のため定数倍の不定性があり、パラメータ（本実施形態では、これがカメラパラメータとなる）の数は１２であるが、自由度は１１である。また、Ｐｐは、
【数３】

そこで、透視投影変換について述べると、例えば、（Ｔｘ，Ｔｙ，Ｔｚ）だけ平行移動後、Ｙ軸の廻りにθｙ（パン角）だけ回転した後、Ｘ軸の廻りにθｘ（チルト角）、Ｚ軸の廻りにθｚ（ロール角）だけ順次回転したとすると、並進と回転の各変換の成分は、次式の如く表現される。
【００２７】
【数４】

【数５】

一方、ワールド座標系（Ｘ，Ｙ，Ｚ）に対するカメラ座標系（Ｘ’，Ｙ’，Ｚ’）は、次式で表現される。
【００２８】
【数６】

【数７】

また、［Ｒ］は数８の式で表現され、［ＥＸＴ］は、数９の式で表現される。
【００２９】
【数８】

【数９】

よって、この式は、即ち、数１０の右辺の第１行列式となる。
【００３０】
【数１０】

これらのパラメータ（Ｐ_１１，Ｐ_１２，Ｐ_１３・・・Ｐ_３４）の内、どれだけを未知として校正するかにより具体的な算出方法は異なるが、１２個全てのパラメータ要素を未知として校正するためには、数２の式に関して１１元以上の連立方程式を解けば良い。
【００３１】
しかし、実用上ではいくつかを既知として扱って差し支えない場合や、所定数ｎのパラメータを未知として、数２の式に関してｎ元以上の連立方程式を解けば良いことになる。
【００３２】
実際、カメラを校正するには、既知の３次元上の複数の位置と、その各点に対応する画像上の点群から行列を算出する。このとき、複数のサンプル点に対してワールド座標系およびカメラ座標系での３次元座標と画像座標系の２次元画像座標との対応を取り、連立方程式を解いて要素を求めることができる。以降、３次元上の点：３次元点、２次元画像上の点：画像点と称す。
【００３３】
３次元点Ｘｉと画像点Ｘｉとの対応は、
【数１１】

で表現され、１個の対応点から２つの線形方程式が得られるため、Ｐの自由度（＝１１）：未知数に対して６個以上の対応点に関する連立方程式を解けば、全パラメータが求まりカメラ１７が校正される。よって、カメラ１７にどのようなカメラを採用するかが決まればカメラの内部パラメータＣは車両に搭載する前に測定しておけば既知となり、しかも、並進成分Ｔは車両のカメラ取付け場所により設計値を用いれば既知であることから、ロール角θｒ、チルト角θｔ、パン角θｐがわかればカメラ１７の校正が行える。即ち、未知のロール角θｒ、チルト角θｔ、パン角θｐを求めるのに、数１１の式から２つの対応点が得られた場合、未知のパラメータの校正ができることになる。
【００３４】
そこで、車両における車載カメラ１７の校正について、手順を説明する。本実施形態ではターゲットバーＴＢを床面上の指定した位置に配置する。ターゲットバーＴＢの両端の２点が校正上重要となる。
【００３５】
校正では、図３に示されるようにターゲットバーＴＢが、車両後方の所定位置（例えば、バンパーの後端から１ｍはなれた位置）に車両主軸（車両の中心を通る長手方向の軸）に対して、左右対称且つ垂直になるようにした状態から校正を始める。
【００３６】
この場合、校正時にはコネクタ８を介して調整ボタン７がＣＰＵ１１の信号入力端子に接続される。調整ボタン７は、図５に示される構成を取り、Ａボタン７ａ、Ｂボタン７ｂ、Ｃボタン７ｃの３つのボタンを備える。Ａボタン７ａはダイオードＤ１，Ｄ２を介して、Ｂボタン７ｂ，Ｃ７ｃの信号端子ＦＣＳ１，ＦＣＳ２とワイヤードオアになっており、一端がＧＮＤ共通で接地されている。また、調整ボタン７の他端はＣＰＵ１１のＦＣＳ１，ＳＣＦ２の入力端子に入力される。例えば、Ａボタン７ａを押すとＦＣＳ１，ＦＣＳ２が共に低電位（Ｌｏ）となる。一方、Ｂボタン７ｂのみを押すとＦＣＳ１が低電位となり、Ｃボタン７ｃのみを押すとＦＣＳ２が低電位となる。尚、Ａボタン７ａはワイヤードオアの構成を必ずしも取らなくても良い。
【００３７】
このＡボタン７ａを押すことで、図６の（ａ）〜（ｃ）に示されるように、ディスプレィ１３の画面上にウィンドウＷＤ（ＷＤｒ，ＷＤｔ，ＷＤｐのいずれか）が表示される。このウィンドウＷＤは、ターゲットバーＴＢが中に入る大きさで後方画像に対して電気的に重畳表示される。Ａボタン７ａによりモードの切換が可能となり、Ａボタン７ａを所定時間（例えば、０．５ｓｅｃ毎に連続して押すこと）により、ロール角調整モード、チルト角調整モード、パン角調整モード、調整終了モードと状態が遷移する。一方、Ｂボタン７ｂおよびＣボタン７ｃは表示されるウィンドウＷＤ（ＷＤｒ，ＷＤｔ，ＷＤｐ）位置を調整するためのボタンである。
【００３８】
尚、本実施形態では、チルト角＜±５．５度、パン角＜±５．５度、ロール角＜±６．０度以内であればターゲットバーＴＢがウインドウ内に完全に納まるよう設定されている。
【００３９】
調整方法に関して説明を図７に示すが、最初モードであるロール角調整モードではＢボタン７ｂを１回押す毎に、図６の（ａ）に示されるウィンドウＷＤｒ（表示色：黄色、ｘ方向の大きさ：（ｘｒｂ−ｘｒｓ）ドット、ｙ方向の大きさ：（ｙｒｂ−ｙｒｓ）ドット）を所定角度（例えば、０．５度だけ）画像中心に対して左回転させ、Ｃボタン７ｃを１回押す毎に、ウィンドウＷＤｒを所定角度（例えば、０．５度だけ）画像中心に対して右回転させる。
【００４０】
ここで、Ｂボタン７ｂ或いはＣボタン７ｃを１回操作ごとにロール角設定値を０．５度づつ移動させ、ロール角調整時のウインドウＷＤｒがターゲットバーＴＢと平行になるまで、Ｂボタン７ｂ（左回転用）、Ｃボタン７ｃ（右回転用）を操作する。尚、このモードではウィンドウＷＤｒの大きさがターゲットバーＴＢに対していくらか余裕がある。
【００４１】
ターゲットバーＴＢにウィンドウＷＤｒが回転操作を行って平行になったら、次に、Ａボタン７ａを所定時間（０．５ｓｅｃ）押し、今度はチルト角調整モードにする（図６の（ｂ）参照）。
【００４２】
このモードではチルト調整用のウインドウＷＤｔ（表示色：赤色）がディスプレィ画面上に表示される。このモードでのウィンドウＷＤｔの大きさは、ｘ方向ではウインドウＷＤｒと同じあるが、ｙ方向ではロール角調整用のウィンドウＷＤｒより小さくなり、１ｍ後方で±２ｃｍの許容誤差を含むように８ドットで表示される。このモードでも同様な操作を行うが、今度はＢボタン７ｂ、Ｃボタン７ｃを１回押す毎にチルト設定値を所定角度（例えば、０．５度）づつ変え、ウインドウＷＤｔの上下境界がターゲットバーＴＢに略外接するまで、Ｂボタン７ｂ（下方向移動用），Ｃボタン７ｃ（上方向移動用）を操作し、ウィンドウＷＤｔをｙ方向に動かしならが調整を行う。
【００４３】
その後、Ａボタン７ａを所定時間（例えば、０．５ｓｅｃ）押し続けると、今度はパン角調整モードになる（図６の（ｃ）参照）。このモードでは、ウインドウＷＤｐ（表示色：青色）が表示される。このウィンドウＷＤｐはｘ方向の大きさ：ロール角調整用のウィンドウＷＤｒより幅が狭くなる（ｘｐｂ−ｘｒｓ）ドット、ｙ方向の大きさ：１ｍ後方で左右方向±３ｃｍの許容誤差を含む。調整に関しては同様であるが、今度はＢボタン７ｂ，Ｃボタン７ｃを１回操作する毎にパン角設定値を所定角度（０．５度）づつ変え、ウインドウＷＤｐの左右境界がターゲットバーに略外接するまで、Ｂボタン７ｂ（左方向移動用）、Ｃボタン７ｃ（右方向移動用）を操作することでウィンドウＷＤｐがｘ方向に動き、ターゲットバーＴＢの両端が略外接する位置まで調整を行う。
【００４４】
このようにして、ロール角、チルト角、パン角の調整を行い、ターゲットバーＴＢがウィンドウＷＤｐと略一致した状態になった後、再び、Ａボタン７ａを所定時間（０．５ｓｅｃ）以上連続して押すと、調整モード終了となり、そのときのウィンドウ状態に基づく角度が決定される。この場合、３つのモードの調整によって得られた角度情報がそのときのカメラ１７のカメラパラメータとしてＣＰＵ内部のＥＥＰＲＯＭに書き込み、調整モードを終了する。
【００４５】
このような調整により、調整を終了すると、調整により設定されたカメラパラメータを基にして、ディスプレィ画面上に走行予想軌跡２０が表示される。
【００４６】
この場合、図８に示すように走行予想軌跡２０を所定距離線（０．５，１，２，３ｍ間隔ではしご状に表示させると良い。例えば、その軌跡上の１つの距離線（例えば、車両後方１ｍのライン）をターゲットバーＴＢとして用いれば、走行予想軌跡２０の一部（距離線（１ｍ）の両端の点ＴＢＲ，ＴＢＬ）から、正確なカメラパラメータが設定されているかの確認が同じ走行予想軌跡２０を用いて瞬時にして行える。一方、後方の所定距離の位置にターゲットバーＴＢを固定する場合、カメラパラメータを設定した後、走行予想軌跡２０を表示させると、校正によって得られたカメラパラメータが正確に設定されているのか否かを、走行予想軌跡２０の距離線（車両後方１ｍのライン）により容易に確認が行える。
【００４７】
このようにして、カメラパラメータの設定を行えば、駐車操作時にディスプレィ上に可変表示される走行予想軌跡２０は、カメラ１７の公差や車体への取付け公差の影響を受けなくなる。
【００４８】
本発明によれば、校正に平面に配置した平面状の校正指標を用いて、カメラ画像内に校正指標をディスプレィ上に表示させ、そこに所定のウィンドウを表示させる。ウィンドウの位置をカメラによって撮像される画像座標内で変化させ、校正指標の中の２点がウィンドウ内に入るよう位置関係を調整手段により調整し、校正指標がウィンドウ内に収まったときのロール角、チルト角、パン角を、カメラのカメラパラメータとして設定するようにしたので、カメラパラメータの設定は調整手段の調整だけで設定できる。この場合、校正指標は平面状であることから、校正指標は床面に置くか貼り付ければ校正を行う場合にあっては邪魔にならず、車両の製造現場等においても、簡単な方法でカメラの校正が行える装置が提供されると共に、カメラの校正が簡単に行える。
【００４９】
この場合、調整手段によって、カメラ画像に表示されるウィンドウを校正指標に対して回転または移動させて調整を行うようにすれば、ウィンドウの位置を調整手段の調整で回転または移動させ、簡単な方法でカメラパラメータを設定することが可能となる。よって、カメラパラメータの設定が行えるので、画面上の任意の点への描画が可能となる。
【００５０】
一方、車両に取付けられて車両の一方向を撮像して表示する表示手段に、車両の走行に係る可動画像を重畳表示させ、該可動画像の一部に対応する３次元空間上の２点を用いれば、その２点を用いてカメラパラメータの設定確認に用いることができる。
【００５１】
更に、可動画像は、ステアリング舵角によって変化する走行予想軌跡とすれば、走行予想軌跡の一部を用いて、正確なカメラパラメータが設定されているかの確認が瞬時にでき、設定確認が容易となる。
【図面の簡単な説明】
【図１】本発明の一実施形態における車載カメラの校正装置を駐車アシスト装置を例にとって説明する場合のシステム構成図である。
【図２】図１に示す駐車アシスト装置を車両へ取付けた場合の取付図である。
【図３】図１に示す車両に取り付けられたカメラとターゲットバーとの関係を示した図である。
【図４】本発明の３次元座標（ワールド座標系、カメラ座標系）から２次元座標（画像座標系）に変換する透視投影変換を説明するための説明図である。
【図５】図１に示す調整ボタンの構成図である。
【図６】図１に示すディスプレィ画面上で表示される表示画面であり、校正時のターゲットバーに対するウィンドウの移動を示しており、（ａ）はロール角調整モード、（ｂ）はチルト角調整モード、（ｃ）はパン角調整モードにおける表示画面を示す。
【図７】図１に示す調整ボタンの操作によって、モード遷移した場合のウィンドウ移動を示した説明図である。
【図８】本発明の一実施形態におけるターゲットバーを走行予想軌跡の一部とした例を示す。
【符号の説明】
１ 駐車アシスト装置
７（７ａ，７ｂ，７ｃ） 調整ボタン（調整手段）
９ スーパーインポーズ回路
１１ ＣＰＵ
１３ ディスプレィ（表示手段）
２０ 走行予想軌跡（可動画像）
ＴＢ ターゲットバー（校正指標）
ＷＤ（ＷＤｒ，ＷＤｔ，ＷＤｐ） ウィンドウ（ウィンドウ表示手段）[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a calibration device and a calibration method for calibrating a camera when an on-vehicle camera is attached to a vehicle, or for calibrating the camera when an attachment position of the on-vehicle camera is deviated from a prescribed position.ToIt is about.
[0002]
[Prior art]
Conventionally, with respect to camera calibration, various companies and research institutes have been conducting research to establish algorithms, and one of these methods is to use a three-dimensional object having a known shape in advance and a characteristic for calibration. There is known a method of calculating camera parameters based on how a feature point of a three-dimensional object is displayed in a three-dimensional space as an index in an image coordinate system of a camera. For example, a method of setting camera parameters in this manner is described in INRIA Camera Calibration with Feature Extraction Research Report No. 2204 (Feb. 1994), pages 1 to 21, and in the calibration method disclosed herein, camera parameters having 11 degrees of freedom are calculated using a characteristic solid object calibration index.
[0003]
It also has a camera that images the rear of the vehicle, and a signal generator that generates figures and characters on the distance scale.The distance scale can be moved up and down arbitrarily in the rear view projected on the monitor to display the real scale. Japanese Patent Application Laid-Open No. 4-103444 discloses calibration of a vehicle-mounted camera to be adjusted to an accurate position above.
[0004]
[Problems to be solved by the present invention]
In the camera calibration method shown in the former, in order to determine camera parameters, a characteristic three-dimensional object is placed at a position where a camera can image it, and at which point in the image coordinate system of the camera the three-dimensional object is displayed. A method of calculating geometrically and determining camera parameters from a three-dimensional space is used. As described above, when the camera is calibrated after the camera is mounted on the vehicle, when the object to be calibrated is the camera mounted on the vehicle, the calibration is performed on a vehicle assembly line such as a factory. In such a production site, good workability is a condition. At the production site, if a method is adopted in which a three-dimensional object is placed and calibrated, it is inefficient to place a three-dimensional object as a calibration index each time, and from the standpoint of disaster prevention, It is desired that a calibration index for calibrating the camera be an index that a worker can step on, such as a pattern drawn on a floor of a factory, avoiding placing a three-dimensional object.
[0005]
In the latter calibration method, the distance scale is moved up and down arbitrarily in the rear view projected on the monitor to adjust the distance scale to the exact position on the actual screen, and the sense of distance behind the vehicle is grasped. And no camera parameters are set. Therefore, on a monitor screen imaged by a camera of the vehicle, when a predicted driving trajectory assisting the driver's operation at the time of parking operation can be variably displayed by steering operation, when electrically superimposed and displayed in a rear image, Since the camera parameters have not been determined, it is impossible to draw the predicted travel locus at an arbitrary position on the display screen.
[0006]
Therefore, the present invention has been made in view of the above problems, and provides an apparatus and a method that can perform calibration by a simple method even when calibrating a camera at a vehicle manufacturing site or the like.ToTechnical issues.
[0007]
[Means for Solving the Problems]
The first technical measures taken to solve the above problems are:For a camera fixed at a predetermined position of the vehicle and having known internal parameters, calibration of the camera is performed based on a calibration index provided in a camera image captured by the camera, based on a calibration component provided by the camera. , Tilt angle, pan angle), a calibration device for a vehicle-mounted camera that performs adjustment of a calibration index in a plane and a calibration index in the camera image.Display means capable of displaying, window display means for displaying a predetermined window on the display means,The position of the windowAdjusting means for changing, and the adjusting meansInAdjust the positional relationship of the window so that two points fall within the window, and when the calibration index falls within the window,Roll angle, tilt angle, pan angleAnd a parameter setting means for setting as a camera parameter of the camera.
In this case, it is preferable that the calibration index has a linear shape, and the size of the window is set to a size such that the calibration index can be accommodated in the window.
[0008]
According to the first technical means, the calibration is performed using a planar calibration index placed on a plane (for example, the ground or floor), and the calibration index is set.Two points in theThe camera image is displayed on the display means, and a predetermined window is displayed there. Change the position of the window within the image coordinates captured by the camera, andInThe positional relationship is adjusted by the adjustment means so that the two points fall within the window. When the calibration index falls within the window,Roll angle, tilt angle, pan angleIs set as the camera parameter of the camera, so that the camera parameter can be set only by adjusting the adjusting means. In this case, since the calibration index is planar, the calibration index can be placed on or adhered to the floor without any hindrance when performing calibration. Is provided.
[0009]
Also, the second technical means taken to solve the above-mentioned problem is:For a camera fixed at a predetermined position of the vehicle and having known internal parameters, calibration of the camera is performed based on a calibration index provided in a camera image captured by the camera, based on a calibration component provided by the camera. , Tilt angle, pan angle) in a method of calibrating an on-vehicle camera, by adjusting a planar calibration index disposed on a plane and the calibration index in the camera image.Display means capable of displaying, window display means for displaying a predetermined window on the display means,The position of the windowAdjustment means to changemake use ofDisplaying the calibration index on the display means,InAdjust the positional relationship of the window so that two points fall within the window.Two pointsIn the windowRoll angle, tilt angle, pan angle when settledIs set as a camera parameter of the camera.
[0010]
According to the second technical means, the calibration uses a planar calibration index arranged on a plane,Calibration index in camera imageIt is displayed on the display means, and a predetermined window is displayed there. Change the position of the window in the image coordinate system projected on the camera, andInAdjustment of the positional relationship by the adjustment means so that the two points fall within the window, and the calibration indexTwo points inWhen it fits in the windowRoll angle, tilt angle, pan angleIs set as the camera parameter of the camera, the setting of the camera parameter can be performed only by the adjustment means. In this case, since the calibration index is planar, when calibration is performed, it may be fixed or attached to the floor surface in a factory or the like, and since it is not a three-dimensional object as in the past, it is obstructive to the operator. However, the camera can be calibrated by a simple method at a vehicle manufacturing site or the like.
[0011]
In this case, if the adjustment unit rotates or moves the window displayed on the camera image with respect to the calibration index to perform the adjustment, the position of the window is rotated or moved by the adjustment of the adjustment unit. Can set camera parameters. Therefore, since camera parameters can be set, drawing can be performed at an arbitrary point on the screen.
Preferably, the following method is performed. That is,
A mode transition button and a window move button are provided. The mode transition button is used to switch between roll angle adjustment, tilt angle adjustment, and pan angle adjustment, and the window move button preferably moves the window..
The calibration index may be fixed or affixed to the floor at any one of the manufacturing site, the factory, and the production line of the vehicle, and the camera may be calibrated with the distance between the calibration index and the vehicle being a predetermined distance.
The calibration index is a straight line, and the two points are preferably both ends of the calibration index.
The calibration index is a straight line, and the camera is preferably calibrated with the calibration index and the longitudinal main axis passing through the center of the vehicle being in a vertical state.
The camera may be calibrated using a calibration index symmetrical with respect to the longitudinal main axis passing through the center of the vehicle.
It is preferable that the camera parameters of the roll angle, the tilt angle, and the pan angle are sequentially adjusted, and the size of the window can be changed according to the adjustment.
It is preferable that an expected travel trajectory of a vehicle having a distance line indicating a position away from the vehicle by a predetermined distance is superimposed on the camera image on which the calibration index is displayed, and the distance line and the display of the calibration index coincide.
[0012]
On the other hand, a movable image related to the traveling of the vehicle is superimposed and displayed on a display unit attached to the vehicle and imaging and displaying one direction of the vehicle, and a part of the movable image(Two points in three-dimensional space) may be used to confirm the setting of the camera parameters..
[0013]
Furthermore, if the movable image is an expected travel trajectory that changes according to the steering angle, it is possible to obtain camera parameters from a part of the expected travel trajectory, and it is instantaneous to confirm whether the correct camera parameters are set. I can do it.
[0014]
In this case, the predicted traveling trajectory may be displayed in a ladder shape, and distance lines may be displayed at predetermined distance intervals.
[0015]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0016]
FIG. 1 is a diagram in which the calibration device of the present invention is applied to a device (hereinafter, referred to as a parking assist device) 1 that assists parking in a vehicle. The present apparatus 1 electrically superimposes a predicted traveling trajectory 20 that a vehicle will pass when the driver intends to perform a parking operation (at the time of a parking operation) on a display in a back view on a rear image. The controller 16 includes a CCD camera (hereinafter, referred to as a camera) 17 for photographing the rear of the vehicle, a steering sensor 2 for detecting a steering angle of a steering wheel (hereinafter, referred to as a steering) 21, and a reverse (reverse) of a transmission shift lever. Signals from a shift lever reverse switch 3 for detecting a state, a parking switch 4 for operating a parking assist function at the time of a parking operation, and wheel speed sensors 5 and 6 for detecting left and right wheel speeds of driven wheels are input. Based on these signals, the controller 16 displays the rearward image of the vehicle and the predicted traveling trajectory 20 on the display 13 so as to be electrically overlapped. The controller 16 further includes an external controller for correcting the camera position at the time of manufacture and shipment of the vehicle, for example, when a mounting position of the camera 17 is deviated from a normal position due to a vehicle collision or vibration during traveling of the vehicle. A connector 8 is provided so that calibration can be performed by attaching an external device. By connecting the adjustment button 7 to the connector 8 from the outside and operating the adjustment button 7, the camera 17 can be calibrated by changing a preset camera parameter value at a normal position of the camera 17. It is possible.
[0017]
Inside the controller, a CPU 11 for controlling, a graphics drawing circuit 12 for drawing graphics on a display 13, a superimposing circuit 9 for superimposing a graphics signal and a rear image from the camera 17, and extracting a synchronization signal from a camera image And a synchronization separation circuit 10 for supplying the data to the graphics drawing circuit 12.
[0018]
FIG. 2 shows a mounting diagram when the device 1 is mounted on a vehicle. The camera 17 for imaging the rear is mounted at a position slightly deviated from the center on the license plate behind the vehicle, is installed with the optical axis directed downward, and mounted downward (about 30 degrees) at the center behind the vehicle. In addition, the camera itself can secure a field of view of 113 degrees left and right in the horizontal direction by a wide-angle lens, and can capture an area up to about 8 m behind.
[0019]
In the cabin of the vehicle, a display 13 is provided on the panel surface of the center console at an angle that is easy for the driver to see, and a controller 16 is mounted above the glove box. The parking switch 4 for requesting parking assistance at the time of parking operation is provided near the center console which is easy for the driver to operate.
[0020]
When the shift reverse switch 3 is in the reverse state, the parking assist device 1 presses the parking switch 4 that assists parking by the driver (when the driver intends to park, the parking assist If necessary), the neutral point of the steering angle is determined based on the information from the right wheel speed sensor 5 and the left wheel speed sensor 6, and the neutral point is used as the steering angle reference point to steer the vehicle right or left from the reference point. Based on the information of the pulse signal from the steering sensor 2 as to how much the steering wheel 21 has been turned off, it is detected how much the vehicle has been steered, and the vehicle that the vehicle will travel when the vehicle goes back Is superimposed and displayed on the display in accordance with the rear image. The predicted travel locus 20 is a movable image, and a distance line is displayed in a ladder shape at a display width and a predetermined distance behind (for example, a position 0.5, 1, 3 m behind the vehicle) based on the vehicle data ( The display of the trajectory moves according to the steered direction and the steered amount according to the steering angle. When the parking switch 4 is pressed by the predicted travel locus 20, even a driver unfamiliar with parking displays a locus 20 indicating which locus the vehicle will follow during the parking operation, and the distance interval is known, so that the parking operation is facilitated. . A display method and a display form of the predicted traveling locus 20 are known (for example, see Japanese Patent Application Laid-Open No. H11-334470), and can be calculated and displayed by a geometric calculation. The detailed explanation is omitted.
[0021]
As described above, the predicted traveling trajectory 20 displayed when assisting the parking operation is for the driver to grasp the sense of distance behind, but in order to accurately display the predicted traveling trajectory on the display screen. The camera 17 needs to be calibrated due to the mounting error of the camera 17.
[0022]
Therefore, calibration of the camera 17 will be described next. Calibration of the camera 17 is performed at a repair shop when the camera 17 is mounted on a vehicle on a production line or when the mounting position of the camera 17 is deviated from a normal position due to a collision or vibration during traveling of the vehicle. It is.
[0023]
Hereinafter, a method of calibrating the camera 17 on a production line in a factory will be described as an example. The production line is performed in a state where the vehicle is stopped at a predetermined position (predetermined position). When stopping the vehicle at a predetermined position, the vehicle can be stopped at an accurate position by moving the vehicle backward or forward so that the wheels fit into the tire grooves. In this way, with the vehicle stopped at a predetermined position, a flat adjustment serving as an index at the time of camera calibration is provided at a position behind the vehicle (for example, 1 m behind the bumper of the vehicle) so as to be substantially parallel to the vehicle width. Target bar (calibration index) TB is fixed or pasted on the flat floor surface of the production line in advance. This adjustment target bar (hereinafter, referred to as a target bar) TB may not be fixed to the floor of the production line, but may be arranged directly on the ground, and the fixing method is not limited to this. The target bar TB used in the present embodiment uses a white plastic plate having a size of, for example, 200 cm × 5 cm × 0.2 cm, and has a thickness that is not thick and has no problem even when handled in a two-dimensional space. . Therefore, since the target bar TB is not a three-dimensional object, it does not hinder a worker on the production line performing the calibration, and there is no problem even if the target bar TB is stepped on during the operation.
[0024]
Regarding the calibration of the camera 17, a perspective projection transformation for transforming the world coordinate system into the camera coordinate system and the image coordinate system will be described with reference to FIG. By defining in this diagram, the general world space is the world coordinate system: (X, Y, Z), the coordinate system at the camera mounting position is the camera coordinate system: (X ', Y', Z '), and the display screen Assuming that a coordinate system in which a camera image is captured is an image coordinate system: (x, y), the correspondence between the image coordinate system (image space) and the three-dimensional coordinate system (three-dimensional space) can be expressed by the following equation. .
[0025]
(Equation 1)

This equation is specifically expressed as follows.
[0026]
(Equation 2)

Since Pp is a homogeneous matrix, it has an indeterminacy of a constant multiple, and the number of parameters (this is a camera parameter in the present embodiment) is 12, but the degree of freedom is 11. Also, Pp is
(Equation 3)

Therefore, the perspective projection transformation will be described. For example, after translation by (Tx, Ty, Tz), rotation around the Y axis by θy (pan angle), rotation around the X axis, θx (tilt angle), and Z Assuming that the rotation is sequentially performed about the axis by θz (roll angle), the components of the translation and the rotation are expressed as follows.
[0027]
(Equation 4)

(Equation 5)

On the other hand, the camera coordinate system (X ', Y', Z ') with respect to the world coordinate system (X, Y, Z) is expressed by the following equation.
[0028]
(Equation 6)

(Equation 7)

[R] is expressed by the equation (8), and [EXT] is expressed by the equation (9).
[0029]
(Equation 8)

(Equation 9)

Therefore, this expression is the first determinant on the right side of Expression 10.
[0030]
(Equation 10)

These parameters (P₁₁, P₁₂, P_Thirteen... P₃₄)), The specific calculation method differs depending on how much is to be calibrated as unknown. However, in order to calibrate all 12 parameter elements as unknown, solve a simultaneous equation of 11 or more elements in the equation (2). Good.
[0031]
However, in practice, some of them may be treated as known, or a predetermined number n of parameters may be unknown and a simultaneous equation of n or more elements may be solved with respect to the expression of Expression 2.
[0032]
Actually, to calibrate the camera, a matrix is calculated from a plurality of known three-dimensional positions and a point group on the image corresponding to each of the positions. At this time, three-dimensional coordinates in the world coordinate system and the camera coordinate system and two-dimensional image coordinates in the image coordinate system are set for a plurality of sample points, and elements can be obtained by solving simultaneous equations. Hereinafter, points on three dimensions: three-dimensional points, points on two-dimensional images: image points.
[0033]
The correspondence between the three-dimensional point Xi and the image point Xi is
(Equation 11)

Since two linear equations can be obtained from one corresponding point, the degree of freedom of P (= 11): by solving a simultaneous equation relating to six or more corresponding points with respect to the unknown, all parameters can be obtained and the camera 17 is calibrated. Therefore, when it is determined what kind of camera is to be used as the camera 17, the internal parameter C of the camera becomes known if it is measured before being mounted on the vehicle, and the translation component T is a design value depending on the camera mounting position of the vehicle. Is used, the camera 17 can be calibrated if the roll angle θr, tilt angle θt, and pan angle θp are known. That is, if two corresponding points are obtained from the equation (11) to obtain the unknown roll angle θr, tilt angle θt, and pan angle θp, the unknown parameters can be calibrated.
[0034]
Therefore, a procedure for calibrating the vehicle-mounted camera 17 in the vehicle will be described. In the present embodiment, the target bar TB is arranged at a designated position on the floor. Two points at both ends of the target bar TB are important for calibration.
[0035]
In the calibration, as shown in FIG. 3, the target bar TB is positioned at a predetermined position behind the vehicle (for example, a position 1 m away from the rear end of the bumper) with respect to the vehicle main shaft (a longitudinal axis passing through the center of the vehicle). Calibration is started from a state where it is symmetrical and vertical.
[0036]
In this case, during calibration, the adjustment button 7 is connected to the signal input terminal of the CPU 11 via the connector 8. The adjustment button 7 has the configuration shown in FIG. 5 and includes three buttons: an A button 7a, a B button 7b, and a C button 7c. The A button 7a is wired-OR with the signal terminals FCS1 and FCS2 of the B buttons 7b and C7c via diodes D1 and D2, and one end is grounded in common to GND. The other end of the adjustment button 7 is input to input terminals of FCS1 and SCF2 of the CPU 11. For example, when the A button 7a is pressed, both FCS1 and FCS2 become low potential (Lo). On the other hand, when only the B button 7b is pressed, the FCS1 becomes low potential, and when only the C button 7c is pressed, the FCS2 becomes low potential. Note that the A button 7a does not necessarily have to take a wired OR configuration.
[0037]
By pressing the A button 7a, a window WD (any one of WDr, WDt, and WDp) is displayed on the screen of the display 13, as shown in (a) to (c) of FIG. This window WD is electrically superimposed and displayed on the rear image in such a size that the target bar TB enters. The mode can be switched by the A button 7a, and the roll angle adjustment mode, the tilt angle adjustment mode, the pan angle adjustment mode, and the adjustment end are performed by pressing the A button 7a for a predetermined time (for example, by continuously pressing the button every 0.5 seconds). Mode and state transition. On the other hand, the B button 7b and the C button 7c are buttons for adjusting the position of the displayed window WD (WDr, WDt, WDp).
[0038]
In the present embodiment, the target bar TB is set to be completely contained within the window if the tilt angle is within ± 5.5 degrees, the pan angle is within ± 5.5 degrees, and the roll angle is within ± 6.0 degrees. ing.
[0039]
FIG. 7 illustrates the adjustment method. In the roll angle adjustment mode, which is the first mode, every time the B button 7b is pressed once, a window WDr (display color: yellow, x-direction) shown in FIG. The size: (xrb-xrs) dots, the size in the y-direction: (yrb-yrs) dots) are rotated counterclockwise by a predetermined angle (for example, 0.5 degrees) to the image center, and the C button 7c is pressed once. Each time the button is pressed, the window WDr is rotated clockwise by a predetermined angle (for example, by 0.5 degrees) with respect to the image center.
[0040]
Here, the roll angle set value is moved by 0.5 degrees each time the B button 7b or the C button 7c is operated once, and the B button 7b (is moved until the window WDr at the time of the roll angle adjustment becomes parallel to the target bar TB. The user operates the C button 7c (for right rotation) and the C button 7c (for right rotation). In this mode, the size of the window WDr has some allowance with respect to the target bar TB.
[0041]
When the window WDr is rotated in parallel with the target bar TB, the A button 7a is pressed for a predetermined time (0.5 sec), and then the tilt angle adjustment mode is set (see FIG. 6B). .
[0042]
In this mode, a window WDt (display color: red) for tilt adjustment is displayed on the display screen. The size of the window WDt in this mode is the same as that of the window WDr in the x direction, but is smaller than the roll angle adjustment window WDr in the y direction, and is 8 dots so as to include a tolerance of ± 2 cm one meter behind. Is displayed. In this mode, the same operation is performed, but each time the B button 7b and the C button 7c are pressed once, the tilt setting value is changed by a predetermined angle (for example, 0.5 degrees), and the upper and lower boundaries of the window WDt are displayed on the target bar. The B button 7b (for downward movement) and the C button 7c (for upward movement) are operated until the window WDt is moved in the y direction until the window WDt is substantially circumscribed.
[0043]
Thereafter, when the A button 7a is kept pressed for a predetermined time (for example, 0.5 sec), the pan angle adjustment mode is set (see FIG. 6C). In this mode, a window WDp (display color: blue) is displayed. This window WDp has a size in the x direction: a dot (xpb-xrs) narrower in width than the roll angle adjustment window WDr, and a size in the y direction: 1 m behind and includes a tolerance of ± 3 cm in the left-right direction. The adjustment is the same, except that each time the B button 7b and the C button 7c are operated once, the pan angle set value is changed by a predetermined angle (0.5 degrees), and the left and right borders of the window WDp are approximately on the target bar. By operating the B button 7b (for moving in the left direction) and the C button 7c (for moving in the right direction) until the circumscribing, the window WDp moves in the x direction, and the adjustment is performed until both ends of the target bar TB substantially circumscribe. .
[0044]
In this manner, the roll angle, the tilt angle, and the pan angle are adjusted, and after the target bar TB substantially matches the window WDp, the A button 7a is continuously pressed again for a predetermined time (0.5 sec) or more. When pressed, the adjustment mode ends, and the angle based on the window state at that time is determined. In this case, the angle information obtained by the adjustment of the three modes is written into the EEPROM inside the CPU as the camera parameters of the camera 17 at that time, and the adjustment mode ends.
[0045]
When the adjustment is completed by such adjustment, the predicted travel locus 20 is displayed on the display screen based on the camera parameters set by the adjustment.
[0046]
In this case, as shown in FIG. 8, the predicted travel locus 20 may be displayed as a predetermined distance line (a ladder at intervals of 0.5, 1, 2, 3 m. For example, one distance line on the locus (for example, If a line 1 m behind the vehicle is used as the target bar TB, a part of the predicted traveling trajectory 20 (points TBR and TBL at both ends of the distance line (1 m)),It is possible to instantaneously check whether an accurate camera parameter is set by using the same predicted travel locus 20. On the other hand, when the target bar TB is fixed at a position at a predetermined distance behind, setting the camera parameters and then displaying the predicted travel trajectory 20 will determine whether the camera parameters obtained by the calibration are correctly set. Can be easily confirmed from the distance line (line 1 m behind the vehicle) of the predicted traveling locus 20.
[0047]
If the camera parameters are set in this manner, the predicted travel locus 20 variably displayed on the display at the time of parking operation is not affected by the tolerance of the camera 17 or the mounting tolerance to the vehicle body.
[0048]
According to the present invention, using a planar calibration index arranged on a plane for calibration, camera imageCalibration index withinIt is displayed on the display, and a predetermined window is displayed there. Change the position of the window within the image coordinates captured by the camera, andInThe positional relationship is adjusted by the adjustment means so that the two points fall within the window. When the calibration index falls within the window,Roll angle, tilt angle, pan angleIs set as the camera parameter of the camera, so that the camera parameter can be set only by adjusting the adjusting means. In this case, since the calibration index is flat, the calibration index is placed on the floor surface or pasted on it, so that it does not hinder the calibration when it is performed. In addition to the provision of an apparatus capable of calibrating the camera, the camera can be easily calibrated.
[0049]
In this case, if the adjustment unit rotates or moves the window displayed on the camera image with respect to the calibration index to perform the adjustment, the position of the window is rotated or moved by the adjustment of the adjustment unit. Can set camera parameters. Therefore, camera parameters can be set.Can be drawn on.
[0050]
On the other hand, a movable image related to the traveling of the vehicle is superimposed and displayed on a display unit that is attached to the vehicle and captures and displays one direction of the vehicle, and a three-dimensional space corresponding to a part of the movable image is displayed.If two points are used, the two points can be used to confirm camera parameter settings..
[0051]
Furthermore, if the movable image is an expected traveling trajectory that changes according to the steering angle, a part of the expected traveling trajectory can be obtained.Using,You can instantly confirm that the correct camera parameters are set, and setVerificationBecomes easier.
[Brief description of the drawings]
FIG. 1 is a system configuration diagram illustrating a calibration device for a vehicle-mounted camera according to an embodiment of the present invention, taking a parking assist device as an example.
FIG. 2 is a mounting diagram when the parking assist device shown in FIG. 1 is mounted on a vehicle.
FIG. 3 is a diagram showing a relationship between a camera attached to the vehicle shown in FIG. 1 and a target bar.
FIG. 4 is an explanatory diagram for explaining perspective projection transformation for transforming three-dimensional coordinates (world coordinate system, camera coordinate system) into two-dimensional coordinates (image coordinate system) according to the present invention.
FIG. 5 is a configuration diagram of an adjustment button shown in FIG. 1;
FIG. 6 is a display screen displayed on the display screen shown in FIG. 1, showing movement of a window with respect to a target bar at the time of calibration, where (a) shows a roll angle adjustment mode, and (b) shows a tilt angle adjustment. (C) shows a display screen in the pan angle adjustment mode.
FIG. 7 is an explanatory diagram showing window movement when the mode is changed by operating the adjustment button shown in FIG. 1;
FIG. 8 shows an example in which a target bar according to an embodiment of the present invention is set as a part of a predicted traveling locus.
[Explanation of symbols]
1 parking assist device
7 (7a, 7b, 7c) adjustment button (adjustment means)
9 Superimpose circuit
11 CPU
13 Display (display means)
20 Travel trajectory (movable image)
TB target bar (calibration index)
WD (WDr, WDt, WDp) window (window display means)

Claims (10)

For a camera fixed at a predetermined position of the vehicle and having known internal parameters , calibration of the camera is performed based on a calibration index provided in a camera image captured by the camera, based on a calibration component provided by the camera. , Tilt angle, pan angle) by adjusting the in- vehicle camera.
A planar calibration index arranged on a plane,
Display means capable of displaying the calibration index in the camera image ,
Window display means for displaying a predetermined window on the display means;
Adjusting means for changing the position of the window ;
The adjusting means adjusts the positional relationship of the window so that two points in the calibration index fall within the window, and sets the roll angle, tilt angle, and pan angle when the calibration index falls within the window, Parameter setting means for setting as camera parameters of the camera,
A calibration device for a vehicle-mounted camera, comprising: 2. The calibrating device for a vehicle-mounted camera according to claim 1, wherein the calibration index has a linear shape, and the size of the window is set to a size such that the calibration index fits in the window . 3. For a camera fixed at a predetermined position of the vehicle and having known internal parameters, calibration of the camera is performed based on a calibration index provided in a camera image captured by the camera, based on a calibration component provided by the camera. In the method of calibrating the in-vehicle camera performed by adjusting the tilt angle and pan angle),
A planar calibration index disposed on a plane, display means capable of displaying the calibration index in the camera image, window display means for displaying a predetermined window on the display means, and changing a position of the window. Using the adjusting means
Displaying the calibration index on the display means, adjusting the positional relationship of the window so that the calibration index falls within the window, and the roll angle when two points in the calibration index fall within the window. , A tilt angle and a pan angle are set as camera parameters of the camera . The adjusting means includes a mode transition button and a window moving button. The mode transition button switches between adjustment of a roll angle, adjustment of a tilt angle, and adjustment of a pan angle, and movement of the window is performed by a window moving button. 4. The method for calibrating a vehicle-mounted camera according to claim 3, wherein: The calibration index is fixed or affixed to the floor at any one of a manufacturing site, a factory, and a production line of the vehicle, and calibration of the camera is performed using a distance between the calibration index and the vehicle as a predetermined distance. The calibration method for a vehicle-mounted camera according to claim 3, wherein the calibration is performed. 4. The method according to claim 3, wherein the calibration index is a straight line, and the two points are both ends of the calibration index. The calibration index of the vehicle-mounted camera according to claim 3, wherein the calibration index is a straight line, and the camera is calibrated by setting the calibration index and a vehicle main axis in a longitudinal direction passing through the center of the vehicle in a vertical state. Calibration method. 4. The method of calibrating an on-vehicle camera according to claim 3, wherein the calibration of the camera is performed using a calibration index that is bilaterally symmetric with respect to a longitudinal main axis passing through the center of the vehicle. The method according to claim 3, wherein the camera parameters of the roll angle, the tilt angle, and the pan angle are sequentially adjusted, and the size of the window is variable according to the adjustment. For the camera image in which the calibration index is displayed, a predicted travel trajectory of the vehicle having a distance line indicating a position away from the vehicle by a predetermined distance is displayed in an overlapping manner, and the display of the distance line and the calibration index 4. The in-vehicle turtle according to claim 3, wherein LA calibration method.

Images