JP2004040523A - Surveillance apparatus for vehicle surroundings - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a surveillance apparatus for vehicle surroundings for informing a driver of the current condition accurately, by monitoring a picture of vehicle surroundings by a camera mounted on a vehicle, detecting shifting objects as an obstacle, when the shifting object shifts to a region, which cannot be displayed by the current image, and indicating the shifting object on a display. <P>SOLUTION: In order to achieve the above-mentioned purpose, a complementing region which cannot be displayed in the present image is segmented from a past picture, and the complementing image is compounded with the present image. At the same time, an image picked up by a vehicle mounted camera from a prescribed view point is converted to an image picked up by a camera from a view point higher than that and displayed the converted image. Then, an obstacle in the complementing region is detected by laser radar, and when the obstacle is detected, the complementing region is subjected to hatching process and displayed to inform the presence of obstacle. <P>COPYRIGHT: (C)2004,JPO

Description

以上の変換式によって車両を中心とした視点ａから車載カメラの視点ａ’に変換することができる。
【００３０】
次に図３は画像処理器Ａ４の処理のうち、上記で説明した車載カメラ１のカメラ座標（ｘ′，ｙ′，ｚ′）から撮像面座標（ｘ″，ｚ″）への座標に変換するための説明図である。
図３において撮像面座標（ｘ″，ｚ″）は車載カメラ１の撮像面ｂの中心を原点Ｏ’とし、水平方向をｘ″軸、垂直方向をｚ″軸とする。カメラ座標から撮像面座標への座標変換は焦点距離をｆとすると次式となる。

以上の変換式によってカメラ座標から撮像面座標に変換することができる。
【００３１】
次に図４は画像処理器Ａ４の処理のうち、車載カメラ１から撮像した画像を車載カメラ１よりも高い視点にある仮想カメラから真下の地面を撮像した仮想カメラ画像に変換する処理の説明図である。
【００３２】
図４（Ａ）においてディスプレイ２０の表示領域に表示される画面の座標を（ｈ，ｖ）とする。表示領域の中心を原点とし、水平方向をｈ軸、垂直方向をｖ軸とする。この座標への変換方向は撮像面の座標（ｘ″，ｚ″）をワールド座標の地面上の座標系、すなわちｚ＝０とした（ｘ，ｙ）座標で表す。式（１）より逆変換は、

となる。
【００３３】
式（３）と式（２）及びｚ＝０からｘ′，ｙ′，ｚ′を消去すると、

となり、（ｘ″，ｚ″）平面から（ｘ，ｙ）平面に変換できる。
【００３４】
以上の変換式から、仮想カメラ画像に変換することができる。
【００３５】
次に上記（ｘ，ｙ）平面の中から任意の範囲を画面上に表示させるための処理を説明する。　任意の範囲は、例えば図４（Ｂ）に示すように車両後部０．５［ｍ］を含み、ｘ軸が幅４［ｍ］、ｙ軸が幅３［ｍ］のエリアとし、この任意の範囲を仮想カメラで撮影したものとして抽出し、図４（Ａ）のような画面座標に変換する。状況に応じては上記抽出範囲を運転者が可変できるようにすれば、より利便性が向上する。
【００３６】
図４（Ａ）のディスプレイ２０の表示領域は、補完領域の画像ｍと撮像画像ｎから成る。ここで仮想カメラフレームメモリ６の画素数を例えば水平を６４０画素、垂直を４８０画素とすると、（ｘ，ｙ）平面から（ｈ，ｖ）平面への変換式は次式となる。

ここで（ｈ，ｖ）平面における補完領域の画像の不等式ｖ＜ｆ（ｈ）を求める。撮像する範囲はカメラに近いほどｘ軸での幅が狭く、遠くなると広くなる。よって撮像面の左端のｘ″座標を−ｘ″_０、右端のｘ″座標をｘ″_０とすると、この値より絶対値の大きい範囲が撮像されない補完領域の画像となるため、式（４）よりｚ″を消去しｙ＜ｇ（ｘ）で表すと、

となる。
【００３７】
これを式（５）によりｈ、ｖに変換するとｖ＜ｆ（ｈ）は次式の通りとなる。

そしてサンプルタイムｔ_１のタイミングで撮像し、以上の処理を行った画像が仮想カメラ画像フレームメモリ６に記録される。
【００３８】
次に画像処理部Ｂ５の処理について図５を用いて説明する。
【００３９】
サンプルタイムｔ_１で撮像された画像の任意の点ｐが画面上では（ｈ_１，ｖ_１）に位置していたと仮定し、サンプルタイムｔ_１から車両が左側方向となるサンプルタイムｔ_２の（ｈ_２，ｖ_２）に後退した場合を説明する。
【００４０】
操舵角θ_ｓ　（右操舵が正）、ホイールベースＬ、ステアリングギア比ｎとすると、左右後輪の中心の回転半径Ｒ（右旋回が正）は次式となる。

尚、旋回時にはタイヤの横滑りによる影響が車速の２乗で発生するが、本装置が使用されるのは約１０［Ｋｍ／Ｈ］以下の低速域であるため、この影響は無視できる。　更に車速をｖとすると１周期の回転角は次式となる。

そして回転半径Ｒ、操舵角θ_１及びリアオーバーハング−Ｌ_０により次式の座標変換式が成立し、点ｐは（ｈ_１，ｖ_１）から（ｈ_２，ｖ_２）へ移動する。

前回のサンプルタイムｔ_１で記録した表示画像フレームメモリ１１の画素を式（１０）により移動し、これを補完画像フレームメモリ７に記録する。
【００４１】
画像合成器１０では撮像画像ｎに関しては仮想カメラ画像フレームメモリ６で記録した仮想カメラの画像を用いて、補完領域の画像ｍに関しては補完画像用フレームメモリ７で記録した補完画像を用いて合成する。この合成画像はスーパーインポーザー１５へ送られるとともに、次回のサンプルタイムｔ_３での補完処理のために画像処理器へと出力される。
以上の処理により、車両に搭載されたカメラによって撮像した過去の画像から現在、撮像できない領域に画像を補完することができる。
【００４２】
次に図６はレーザレーダ１２で障害物を検出する検出方法についての説明図である。補完領域を検出範囲とするためにレーザレーダを車両の右後端部と左後端部に設置し、発射したレーザ光が物体に反射して戻るまでの時間で距離を計測する。またレーザ光を回転スキャンさせることで、検出物体の角度情報を得ることができる。
【００４３】
（ｘ_Ｌ，０）に配置された左レーダが捕らえた障害物の距離Ｌ_Ｌと角度θ_Ｌから障害物のワールド座標における位置（ｘ_ｒ，ｙ_ｒ）を次式で求める。

上記説明は左側の例であるが、右側の処理も同等に行う。ここでは一例としてレーザレーダの場合で説明したが、ミリ波レーダを使用してもよいし、また安価で小型であるという特徴を有するソナーを使用してもよい。
【００４４】
次に図７を用いて、本実施の形態によって、ディスプレイ２０に表示される画像について説明する。
【００４５】
ディスプレイ２０の表示領域には、自車両の後端を示す自車両マーカー２１と、自車両の予想進路を示す予想進路マーカー２２とが、車載カメラ１によって撮像された画像に加えて表示されている。
【００４６】
まず図７に示したディスプレイ２０の表示領域の右側を用いて、障害物が検出されていない場合について説明する。
【００４７】
２３は車載カメラ１によって撮像された現在（例えばサンプルタイムｔ_２）の状況を示す現在画像で、２４は前回のサンプルタイムｔ_１に撮像された画像に基づいて、補完処理された補完画像であり、レーザレーダ１２によって障害物が検出されていない場合には、この現在画像２３と補完画像２４とは、視認者には一つの繋がった画像として認識される。すなわち、視認者にとっては、現在画像２３と補完画像２４も車両周囲の現在の画像として認識される。なお、点線２５は、現在画像２３と補完画像２４との境界を便宜的に示す線であり、実際のディスプレイ２０には表示されていない。
【００４８】
次にディスプレイ２０の表示領域の左側を用いて、障害物が検出された場合について説明する。
【００４９】
補完画像２４は、過去の画像を用いて補完された画像であって、現在の画像ではないので、瞬間的に三輪車２６が補完画像２４が表示される車両周囲の領域へと移動してきた場合、この三輪車２６の存在をディスプレイ２０へと表示することができない。従って、レーザレーダ１２によって障害物の存在を検出した場合には、補完画像２４による表示画像部分に、ハッチング処理２７を行うことで、現在画像２３よりも強調した表示を行う。この表示を視認者が認識することで、補完画像２４部分に障害物が存在することを報知することができ、車両周囲の状況を正確に報知することができる。視認者は、この報知に基づいて、目視による確認を行うことができる。
【００５０】
以上説明したように本実施の形態においては、所定の視点から車載カメラ１によって撮像した画像を、所定の視点よりも高いカメラから撮像した画像に変換して、表示するようにしたので、視認者が車両周囲の状況を見易いという効果を有する。
【００５１】
また、過去の画像から、現在の画像では表示できない補完領域を切り出し、この補完画像を現在の画像に合成すると共に、この合成画像を仮想カメラ位置の画像に変換しているので、ディスプレイ２０に表示される画像が、表示領域の全領域に表示されることになり、違和感の少ない表示を行うことができるという効果を有する。
【００５２】
また、補完領域に存在する障害物をレーザレーダ１２で検出し、障害物が検出された場合には、補完領域にハッチング処理を行って表示し、障害物の存在を報知するようにしたので、車両周囲の状況を正確に報知することができるという効果を有する。
【００５３】
以上説明した実施の形態では、障害物が存在した場合には、補完画像２４にハッチング処理を行うようにしたが、「障害物がいます。目視して下さい。」というような文字表示を行ったり、障害物の存在をブザーによって警報音として発生させたりしてもよい。またハッチング処理に変えて補完画像を表示しない、すなわち黒色や白色の表示を行っても良い。また、障害物の存在を示す矢印を表示しても良い。
【００５４】
また上述した実施の形態では、車載カメラ１をピンホールカメラとして説明したが、広角レンズなどを使用しても同様の効果が得られる。
【００５５】
また、図８に示すように、撮像した画像内に存在する障害物、例えば駐車場に置かれたブロック２８を検出し、マーカーを表示させるようにしてもよい。また、自車両の予想軌跡内に、障害物が存在する場合には、マーカーを赤色２９として、強調させて表示してもよい。
【００５６】
また、図９に示すように、撮像する画像内に存在する障害物が車両などの移動物体３０である場合には、移動物体の移動方向をマーカー３１によって表示させるようにしても良い。この場合には、前述に画像処理器Ａ４によって、障害物の存在を検出するようにする。
【００５７】
なお上述した実施の形態では、車載カメラによって車両後方を撮像するようにしたが、車両の前方や、側方などの周囲に用いても同様の効果を有することはいうまでもない。
【００５８】
特許請求の範囲の構成要素と実施の形態の対応関係は次のとおりである。
すなわち、車載カメラ１が撮像手段を、画像処理器Ｂ５が画像補完手段を、画像処理器Ａ４が画像変換手段を、画像合成器１０が画像合成手段を、レーザレーダ１２が障害物検出手段を、スーパーインポーザー１５が重畳表示手段を、ディスプレイ２０が表示手段を、スーパーインポーザー１５およびディスプレイ２０が報知手段を、ブザーが警報手段を、画像処理器Ａ４が車両検出手段をそれぞれ構成する。尚、本発明の特徴的な機能を損なわない限り、各構成要素は上記構成に限定されるものではない。
【図面の簡単な説明】
【図１】本実施の形態において車両周囲監視装置の構成を示す図。
【図２】本実施の形態においてワールド座標からカメラ座標へ変換する説明図。
【図３】本実施の形態においてカメラ座標から撮像面座標へ変換する説明図。
【図４】本実施の形態において車載カメラで撮像した画像を車載カメラより高い視点に変換する説明図。
【図５】本実施の形態において過去の画像から現在、撮像できない領域に画像を補完する説明図。
【図６】本実施の形態においてレーザレーダによる障害物の検出方法の説明図。
【図７】本実施の形態においてディスプレイに撮像した補完領域内に存在する障害物をハッチングで強調させる説明図。
【図８】本実施の形態においてディスプレイに撮像した画像内に存在する障害物をマーカーで表示させる説明図。
【図９】本実施の形態においてディスプレイに撮像した画像内に存在する障害物の移動方向をマーカーで表示させる説明図。
【符号の説明】
１　・・・車載カメラ
２　・・・バックモニタＥＣＵ
３　・・・カメラ画像フレームメモリ
４　・・・画像処理器Ａ
５　・・・画像処理器Ｂ
６　・・・仮想カメラ画像フレームメモリ
７　・・・補完画像フレームメモリ
８　・・・車速センタ
９　・・・操舵角センサ
１０・・・画像合成器
１１・・・表示画像フレームメモリ
１２・・・レーザレーダ
１３・・・マイコン
１４・・・マーカー発生器
１５・・・スーパーインポーザ
１６・・・カーナビゲーションＥＣＵ
１７・・・カーナビゲーションモジュール
１８・・・シフトポジションスイッチ
１９・・・画像切替え機
２０・・・ディスプレイ
２１・・・自車両マーカー
２２・・・予想進路マーカー
２３・・・現在画像
２４・・・補完画像
２５・・・点線
２６・・・三輪車
２７・・・ハッチング
２８・・・ブロック
２９・・・赤色マーカー
３０・・・移動物体
３１・・・移動方向マーカー[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a vehicle periphery monitoring device.
[0002]
[Prior art]
As a conventional technique, a technique disclosed in JP-A-2002-29314 is known.
[0003]
The above-described conventional technique is configured to cut out an image that cannot be picked up in a current image pickup area from a past image picked up by a camera mounted on a vehicle, and complement the cut-out past picked-up image with a current image to be displayed. A technique for displaying an image having a small portion is disclosed.
[0004]
[Problems to be solved by the invention]
However, in the conventional technology, the current image is supplemented with the past image, so that the displayed image of the complemented area does not always correspond to the current situation, and the current image around the vehicle is not always coincident with the current situation. May not be able to accurately report the status.
[0005]
The present invention has been made in view of the above circumstances, and has as its object to accurately notify a current state around a vehicle.
[0006]
[Means for Solving the Problems]
In order to solve the above problem, the present invention is based on a first image captured at a first time and a second image captured at a second time after a lapse of time from the first time. An image complementing unit that generates a third image in which an image of the first surrounding area of the vehicle, which is not captured in the second image, of the one image is complemented by the second image;
Image conversion means for converting a third image generated by the image complementing means into a display image from a second viewpoint higher than the first viewpoint;
Display means for displaying a display image converted by the image conversion means,
Obstacle detecting means for detecting whether an obstacle is present in the first surrounding area, and notifying means for notifying the presence of the obstacle when an obstacle is detected in the first surrounding area by the obstacle detecting means. The above object is achieved by providing the following.
[0007]
【The invention's effect】
According to the present invention, the third image obtained by complementing the first image with the second image is converted into a display image from the second viewpoint, displayed by the display means, and an obstacle is detected in the first surrounding area. In such a case, the notification is made by the notification means, so that the current state can be accurately notified.
[0008]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, an embodiment of a vehicle periphery monitoring device of the present invention will be described in detail with reference to FIGS.
[0009]
FIG. 1 is a diagram showing a configuration of a vehicle periphery monitoring device according to the present invention.
[0010]
Reference numeral 1 denotes an in-vehicle camera arranged near the rear center of the vehicle, which is a CCD camera that captures an image of a predetermined area (constant area) behind the vehicle from a predetermined viewpoint. Here, for simplicity of description, the vehicle-mounted camera 1 is treated as having no lens aberration like a pinhole camera.
[0011]
Reference numeral 8 denotes a vehicle speed sensor that detects the speed of the vehicle, and 9 denotes a steering angle sensor that detects the steering angle of the vehicle.
[0012]
Reference numeral 12 denotes a laser radar that detects whether an obstacle exists in a complementary area described later by transmitting and receiving infrared rays.
The image captured by the on-vehicle camera 1, the vehicle speed detected by the vehicle speed sensor 8, the steering angle detected by the steering angle sensor 9, and the presence or absence of an obstacle detected by the laser radar 12 are respectively determined by a back monitor ECU (electrical). Control unit 2).
[0013]
Hereinafter, the back monitor ECU 2 will be described. Reference numeral 3 denotes an in-vehicle camera image frame memory, which records an image captured by the in-vehicle camera 1 at a predetermined timing (hereinafter, referred to as a sample time). The recorded image is output to the image processor A4.
[0014]
The image processor A4 captures the ground from a virtual camera position arranged at a higher viewpoint than the vehicle-mounted camera 1 with respect to the image output from the vehicle-mounted camera image frame memory 3 (hereinafter, referred to as a virtual camera image). The pixels are rearranged so that Here, it is assumed that the pixels of the virtual camera image are rearranged as if looking down from directly above the sky.
[0015]
This virtual camera image is output to the virtual camera image frame memory 6.
[0016]
The virtual camera image frame memory 6 stores a virtual camera image, and the image recorded here is output to the image synthesizer 10. The image synthesizer 10 performs a synthesis process described later, and the synthesized image is recorded in the display image frame memory 11.
[0017]
The composite image recorded in the display image frame memory 11 is output to the superimposer 15 and the image processor B5.
[0018]
Based on the outputs from the vehicle speed sensor 8 and the steering angle sensor 9, the microcomputer 13 calculates the moving position and distance of the vehicle between the previous sample time and the current sample time. Based on the moving position and distance of the vehicle, the microcomputer 13 determines an area to be complemented and outputs it to the image processing unit B5.
[0019]
The image processor B <b> 5 performs a process of cutting out the image of the portion of the complementary region output from the microcomputer 13 from the composite image output from the display image frame memory 11. The cut-out image (hereinafter, referred to as a complementary image) is output to the complementary image frame memory 7.
[0020]
The complementary image frame memory 7 records the complementary image output from the image processor B5. The virtual camera image recorded in the virtual camera image frame memory 6 and the complementary image recorded in the complementary image frame memory 7 are output to the image synthesizer 10.
[0021]
The image synthesizer 10 synthesizes the virtual camera image and the complementary image into one image. The synthesized image is recorded in the display image frame memory 11, and the recorded image is output to the superimposer 15.
[0022]
A marker generator 14 generates various markers based on the processing of the microcomputer 13. The marker generated by the marker generator 14 is output to the super imposer 15.
[0023]
The superimposer 15 completes the display image for the back monitor by imposing the marker generated by the marker generator 14 on the composite image recorded in the display image frame memory 11.
[0024]
Reference numeral 18 denotes a shift position switch for detecting which position the shift position is selected. The position of the shift position detected by the shift position switch 18 and the image displayed by the superimposer 15 are output to a car navigation ECU (electric control unit).
[0025]
A car navigation module 17 performs various processes such as map display and intersection guidance to create a navigation image. The created navigation screen is output to the image switch 19.
[0026]
The image switch 19 selects a display image output from the superimposer 15 when the shift position switch 18 has selected the reverse position, and the car navigation module when the shift position switch 18 is not at the reverse position. The navigation image output from 17 is selected.
[0027]
Reference numeral 20 denotes a display, which is arranged near the center console in the vehicle compartment and displays an image selected by the image switch 19.
[0028]
FIG. 2 is an explanatory diagram for converting an image from the viewpoint a centering on the vehicle to the viewpoint a 'of the vehicle-mounted camera in the processing of the image processor A4. In FIG. 2, the world coordinates (x, y, z) centered on the vehicle have the origin on the road surface at the center of the rearmost portion of the vehicle, the x-axis is in the vehicle left-right direction, x> 0 in the passenger seat, and the y-axis is in the vehicle front-rear direction It is assumed that the rear direction is y> 0, the vertical direction of the vehicle is the z axis, and the upward direction is z> 0. Here, the on-vehicle camera 1 sets the lens position of the on-vehicle camera 1 to (0, 0, z ₀ ), the optical axis is a pitch angle around the x-axis θp (the upper side is positive), the yaw angle around the z-axis, and the optical axis. The surrounding roll angle is set to 0 degree. The camera coordinates have the origin at the center of the (x ', y', z ') lens of the camera, and the y' axis is the camera optical axis and the image taking is positive. At this time, the determinant of the coordinate transformation from the world coordinates to the camera coordinates is as follows.
[0029]

With the above conversion formula, the viewpoint a centered on the vehicle can be converted to the viewpoint a 'of the vehicle-mounted camera.
[0030]
Next, FIG. 3 shows a process of converting the camera coordinates (x ', y', z ') of the on-vehicle camera 1 into the coordinates of the imaging plane coordinates (x ", z") in the processing of the image processor A4. FIG.
In FIG. 3, the coordinates of the imaging plane (x ″, z ″) are such that the center of the imaging plane b of the vehicle-mounted camera 1 is the origin O ′, the horizontal direction is the x ″ axis, and the vertical direction is the z ″ axis. The coordinate conversion from the camera coordinates to the imaging plane coordinates is given by the following equation, where f is the focal length.

With the above conversion formula, it is possible to convert from the camera coordinates to the imaging plane coordinates.
[0031]
Next, FIG. 4 is an explanatory diagram of a process of converting an image captured from the on-vehicle camera 1 into a virtual camera image capturing the ground immediately below from a virtual camera at a higher viewpoint than the on-vehicle camera 1 among the processes of the image processor A4. It is.
[0032]
In FIG. 4A, the coordinates of the screen displayed in the display area of the display 20 are (h, v). The center of the display area is the origin, the horizontal direction is the h-axis, and the vertical direction is the v-axis. The direction of conversion to these coordinates is represented by the coordinates (x ″, z ″) of the imaging surface in the world coordinate system on the ground, that is, (x, y) coordinates where z = 0. From equation (1), the inverse transform is

It becomes.
[0033]
Eliminating x ', y', z 'from equations (3) and (2) and z = 0 gives

Thus, the (x ″, z ″) plane can be converted to the (x, y) plane.
[0034]
From the above conversion formula, it is possible to convert to a virtual camera image.
[0035]
Next, a process for displaying an arbitrary range on the screen from the (x, y) plane will be described. The arbitrary range includes, for example, the rear portion 0.5 [m] of the vehicle as shown in FIG. 4B, the x-axis is an area having a width of 4 [m], and the y-axis is an area having a width of 3 [m]. The range is extracted as that captured by the virtual camera, and is converted into screen coordinates as shown in FIG. If the driver can change the extraction range depending on the situation, the convenience is further improved.
[0036]
The display area of the display 20 in FIG. 4A includes an image m of the complementary area and a captured image n. Here, assuming that the number of pixels of the virtual camera frame memory 6 is, for example, 640 in the horizontal direction and 480 in the vertical direction, the conversion formula from the (x, y) plane to the (h, v) plane is as follows.

Here, the inequality v <f (h) of the image of the complementary area on the (h, v) plane is obtained. The width of the x-axis becomes narrower as the camera is closer to the camera, and becomes wider as the camera is farther from the camera. Therefore, if the x-coordinate at the left end of the imaging surface is -x " ₀ and the x-coordinate at the right end is x" ₀ , a range having an absolute value larger than this value is an image of the complementary region that is not imaged. When z ″ is eliminated and y <g (x) is expressed,

It becomes.
[0037]
When this is converted into h and v by equation (5), v <f (h) becomes as follows.

Then, an image captured at the timing of the sample time t ₁ and subjected to the above processing is recorded in the virtual camera image frame memory 6.
[0038]
Next, the processing of the image processing unit B5 will be described with reference to FIG.
[0039]
The arbitrary point p of the captured image on the screen at the sample time t ₁ Suppose were located (h _{1, v 1),} the sampling time t ₁ the vehicle is a sample time t ₂ which is a left direction ( h ₂ , v ₂ ).
[0040]
Assuming the steering angle θ _s (right steering is positive), the wheel base L, and the steering gear ratio n, the turning radius R of the center of the left and right rear wheels (right turning is positive) is as follows.

Note that, when turning, the effect of the side slip of the tire occurs at the square of the vehicle speed, but since this device is used in a low speed region of about 10 [Km / H] or less, this effect can be ignored. Further, assuming that the vehicle speed is v, the rotation angle in one cycle is given by the following equation.

The turning radius R, the coordinate conversion formula of the following equation is established by the steering angle theta ₁ and rear overhang -L _0, the point p is moved to _{(h 1, v} 1) from _{(h 2, v} 2).

The pixel of the display image frame memory 11 recorded at the previous sample time t ₁ is moved by the equation (10), and is moved to the complementary image frame memory 7.
[0041]
The image synthesizer 10 synthesizes the captured image n using the image of the virtual camera recorded in the virtual camera image frame memory 6 and the image m in the complementary area using the complementary image recorded in the complementary image frame memory 7. . This composite image with is sent to super-in interposer 15, and output to the image processor for complementing for a next sampling time t _3.
Through the above processing, an image can be complemented from a past image captured by a camera mounted on a vehicle to an area that cannot be captured at present.
[0042]
Next, FIG. 6 is an explanatory diagram of a detection method for detecting an obstacle with the laser radar 12. Laser radars are installed at the right rear end and the left rear end of the vehicle in order to set the complementary region as the detection range, and the distance is measured by the time required for the emitted laser light to reflect off the object and return. Further, by rotating and scanning the laser beam, angle information of the detected object can be obtained.
[0043]
(X _{L, 0)} Distance arranged obstacle left radar caught the L _L and the angle θ position in world coordinates of the obstacle from the _L (x _{r, y r)} the calculated by the following equation.

The above description is an example on the left side, but the processing on the right side is performed similarly. Here, the case of a laser radar has been described as an example, but a millimeter-wave radar may be used, or a sonar having features of being inexpensive and compact may be used.
[0044]
Next, an image displayed on the display 20 according to the present embodiment will be described with reference to FIG.
[0045]
In the display area of the display 20, an own vehicle marker 21 indicating the rear end of the own vehicle and an expected course marker 22 indicating the expected course of the own vehicle are displayed in addition to the image captured by the on-vehicle camera 1. .
[0046]
First, a case where no obstacle is detected will be described using the right side of the display area of the display 20 shown in FIG.
[0047]
Reference numeral 23 denotes a current image indicating the current state (for example, sample time t ₂ ) captured by the vehicle-mounted camera 1, and reference numeral 24 denotes a complementary image that has been subjected to a complementary process based on the image captured at the previous sample time t ₁ . When no obstacle is detected by the laser radar 12, the current image 23 and the complementary image 24 are recognized as one connected image by the viewer. That is, for the viewer, the current image 23 and the complementary image 24 are also recognized as the current images around the vehicle. Note that the dotted line 25 is a line that indicates the boundary between the current image 23 and the complementary image 24 for convenience, and is not displayed on the actual display 20.
[0048]
Next, a case where an obstacle is detected using the left side of the display area of the display 20 will be described.
[0049]
Since the complementary image 24 is an image complemented using a past image and is not the current image, when the tricycle 26 momentarily moves to a region around the vehicle where the complementary image 24 is displayed, The existence of the tricycle 26 cannot be displayed on the display 20. Therefore, when the presence of an obstacle is detected by the laser radar 12, the display image portion of the complementary image 24 is subjected to hatching 27 so that the display is emphasized more than the current image 23. By recognizing the display, the presence of an obstacle in the complementary image 24 can be notified, and the situation around the vehicle can be accurately notified. The viewer can visually confirm based on this notification.
[0050]
As described above, in the present embodiment, an image captured by the in-vehicle camera 1 from a predetermined viewpoint is converted into an image captured from a camera higher than the predetermined viewpoint, and is displayed. Has an effect that it is easy to see the situation around the vehicle.
[0051]
In addition, a complementary area that cannot be displayed in the current image is cut out from the past image, and the complementary image is combined with the current image, and the combined image is converted into an image at the virtual camera position. The image to be displayed is displayed in the entire display area, which has an effect that a display with less discomfort can be performed.
[0052]
In addition, the obstacle existing in the complementary area is detected by the laser radar 12, and when the obstacle is detected, the complementary area is hatched and displayed, so that the presence of the obstacle is notified. This has the effect that the situation around the vehicle can be accurately reported.
[0053]
In the above-described embodiment, when an obstacle is present, the supplementary image 24 is hatched. However, a character display such as “There is an obstacle. Alternatively, the presence of an obstacle may be generated as a warning sound by a buzzer. Further, instead of the hatching processing, the complementary image may not be displayed, that is, black or white may be displayed. Also, an arrow indicating the presence of an obstacle may be displayed.
[0054]
In the above-described embodiment, the in-vehicle camera 1 is described as a pinhole camera, but the same effect can be obtained by using a wide-angle lens or the like.
[0055]
As shown in FIG. 8, an obstacle existing in the captured image, for example, a block 28 placed in a parking lot may be detected and a marker may be displayed. Further, when an obstacle is present in the expected trajectory of the own vehicle, the marker may be highlighted as red 29.
[0056]
Further, as shown in FIG. 9, when the obstacle existing in the captured image is the moving object 30 such as a vehicle, the moving direction of the moving object may be displayed by the marker 31. In this case, the presence of the obstacle is detected by the image processor A4 as described above.
[0057]
In the above-described embodiment, the rear side of the vehicle is imaged by the on-vehicle camera. However, it goes without saying that the same effect can be obtained even when the vehicle is used in front of the vehicle or around the side.
[0058]
The correspondence between the components of the claims and the embodiments is as follows.
That is, the on-vehicle camera 1 is an imaging unit, the image processor B5 is an image complementing unit, the image processor A4 is an image converting unit, the image synthesizing unit 10 is an image synthesizing unit, the laser radar 12 is an obstacle detecting unit, The superimposer 15 constitutes superimposing display means, the display 20 constitutes display means, the superimposer 15 and the display 20 constitute notification means, the buzzer constitutes alarm means, and the image processor A4 constitutes vehicle detection means. Each component is not limited to the above configuration as long as the characteristic functions of the present invention are not impaired.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of a vehicle periphery monitoring device in the present embodiment.
FIG. 2 is an explanatory diagram for converting world coordinates into camera coordinates in the present embodiment.
FIG. 3 is an explanatory diagram for converting camera coordinates to imaging plane coordinates in the present embodiment.
FIG. 4 is an explanatory diagram for converting an image captured by a vehicle-mounted camera to a higher viewpoint than the vehicle-mounted camera in the embodiment.
FIG. 5 is an explanatory diagram for complementing an image from a past image to an area that cannot be captured at the present time in the present embodiment.
FIG. 6 is an explanatory diagram of a method for detecting an obstacle using a laser radar in the present embodiment.
FIG. 7 is an explanatory diagram in which an obstacle present in a complementary region imaged on a display is emphasized by hatching in the present embodiment;
FIG. 8 is an explanatory diagram of displaying an obstacle present in an image captured on a display by a marker in the present embodiment;
FIG. 9 is an explanatory diagram in which a moving direction of an obstacle present in an image captured on a display is displayed by a marker in the present embodiment.
[Explanation of symbols]
1 ... on-board camera 2 ... back monitor ECU
3 Camera camera frame memory 4 Image processor A
5 ... image processor B
6 virtual camera image frame memory 7 complementary image frame memory 8 vehicle speed center 9 steering angle sensor 10 image synthesizer 11 display image frame memory 12 laser Radar 13 ··· microcomputer 14 ··· marker generator 15 ··· superimposer 16 ··· car navigation ECU
17 Car navigation module 18 Shift position switch 19 Image switcher 20 Display 21 Own vehicle marker 22 Expected course marker 23 Current image 24 Complementary image 25: dotted line 26: tricycle 27: hatching 28: block 29: red marker 30: moving object 31: moving direction marker

Claims (11)

Imaging means mounted on the vehicle to image a certain area around the vehicle from a first viewpoint;
The image capturing means includes a first image captured at a first time and a second image captured at a second time after a lapse of time from the first time. An image complementing unit configured to generate a third image obtained by complementing an image of the first surrounding area of the vehicle that is not captured by the second image with the second image;
Image conversion means for converting the third image generated by the image complementing means into a display image from a second viewpoint higher than the first viewpoint;
Display means for displaying a display image converted by the image conversion means,
Obstacle detection means for detecting whether an obstacle is present in the first surrounding area;
And a notifying unit for notifying the presence of the obstacle when the obstacle detecting unit detects an obstacle in the first surrounding area. The vehicle surroundings monitoring device according to claim 1,
The vehicle surroundings monitoring device, wherein the notifying unit includes a superimposing display unit that superimposes an obstacle image indicating the presence of the obstacle on the display image when an obstacle is detected by the obstacle detecting unit. The vehicle surrounding monitoring device according to claim 2,
The obstacle surrounding image which is superimposed by the superimposing display means is an image which is emphasized more than the third image. The vehicle surrounding monitoring device according to claim 3,
A vehicle surroundings monitoring device, wherein the emphasized image is an image subjected to hatching processing. The vehicle surrounding monitoring device according to claim 3,
The vehicle surroundings monitoring device, wherein the emphasized image is a marker. The vehicle surrounding monitoring device according to claim 3,
The vehicle surroundings monitoring device according to claim 1, wherein the highlighted image prohibits display of the image of the first surrounding area. The vehicle surroundings monitoring device according to any one of claims 1 to 6,
The vehicle surroundings monitoring device, wherein the obstacle detecting means is at least one of a laser radar, a millimeter-wave radar, and a sonar. The vehicle surroundings monitoring device according to any one of claims 1 to 7,
The vehicle surroundings monitoring device, wherein the display means displays an expected course of the vehicle. The vehicle surrounding monitoring device according to any one of claims 1 to 8,
The vehicle surroundings monitoring device, wherein the display unit displays a part of the vehicle. The vehicle periphery monitoring device according to any one of claims 1 to 9,
A vehicle detection unit that detects another vehicle based on the second image;
The vehicle surroundings monitoring device, wherein the display means displays a marker indicating another vehicle when the other vehicle is detected by the vehicle detection means. The vehicle surroundings monitoring device according to any one of claims 1 to 10,
The vehicle surroundings monitoring device, wherein the notifying unit includes an alarm unit that issues an alarm when an obstacle is detected by the obstacle detecting unit.

Images