JP2004312523A - On-vehicle image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To grasp the statuses of other lanes up to a more distant place in accordance with the increase of vehicle speed. <P>SOLUTION: This on-vehicle image processor is provided with side cameras 13L, 13R and a rear camera 13B, images are extracted within the ranges of specific extraction field angles α, β from images by the side cameras 13L, 13R and an image by the rear camera 13B and the images are stuck together. Especially, the status of the adjacent lane is grasped up to the more distant place in the case of changing the lanes on a highway by increasing the angle of the extraction field angle β and reducing the angle of the extraction field angle α according to the increase of the vehicle speed. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

画角の変更の必要がない場合には、ステップＳ６０５をスキップし、ステップＳ６０７を実行する。
切り出し画角の変更の必要があると判断した場合には、切り出し画角又は使用する画像の変更（ステップＳ６０５）をする。つまり、ステップＳ６０１、ステップＳ６０３の判断に基づいて、変更がある場合は設定値を変更する。ステップＳ６０１〜Ｓ６０５によって、画像領域決定手段が構成される。
【００３４】
入力装置からの指示があるか否かを判断する（ステップＳ６０７）。例えば、予め複数の設定値を合成パターンのユーザ設定として登録しておき、運転中はスイッチの切換操作により、合成パターンを選択できるようにしておくことができる。
【００３５】
例えば、短縮ボタンのようなものを複数設置し、Ａボタンはデフォルト、Ｂボタンは運転者が設定した左側方の画像、Ｃボタンは運転者が設定した右側方の画像、といったものである。指示がない場合には、図４のフローチャートにリターンされる。
【００３６】
合成画像用設定値を読み出す（ステップＳ６０９）。入力装置１８から、表示すべき画像の画角などの変更の指示がある場合、ユーザからの入力が優先されるため、入力装置１８からの指示に従って設定値を変更する必要がある。
【００３７】
入力装置から変更される設定値としては、各カメラの画角、合成画像の表示方向、表示する側（右側／左側）などである。設定値の読み出しの後、図４のフローチャートにリターンされる。
【００３８】
画像切り出し処理を行う（ステップＳ４０３）。合成パターンの選択（ステップＳ４０１）により決定された設定値に従って、記憶装置１０に保存されている各魚眼画像を読み出し、必要な画像領域が切り出される（切り出す位置はリアルタイムにユーザが操作できるように構成してもよい。）。上記ステップＳ４０１、Ｓ４０３によって、後方画像抽出手段と、側方画像抽出手段が構成される。
【００３９】
次に、正像変換処理を行う（ステップＳ４０５）。
【００４０】
切り出された画像は魚眼レンズにより撮像された画像なので、そのレンズ特性を基に歪みを補正する正像変換処理が行われる。ステップＳ４０５によって、正像変換手段が構成される。
【００４１】
さらに、視点変換処理を行う（ステップＳ４０７）。
【００４２】
次に正像変換した画像を運転者の視点位置より見た画像とするため、各画像に対して視点変換処理が行われる。視点変換処理における視点移動量は、初期化処理で設定されているカメラ設置位置と合成画像の視点位置とから決まる。視点変換処理において、前後方向の変換は画像の縮小処理で対応する。上下左右方向の変換については、予め規定された値に基づいて視点変換先（運転者視点位置）が画像中心となるように、正射影、立体射影、等距離射影、等立体射影等の魚眼レンズの射影方式によって決まる変換式を使用して、座標変換を行う。こうして生成される画像が視点変換画像である。視点変換された各画像は記憶装置１０の視点変換画像データメモリ１７Ｓに記憶される。このステップＳ４０７によって視点変換手段が構成される。
【００４３】
視点変換処理後、各画像が合成される（ステップＳ４０９）。必要ならばここで、記憶装置１０に記憶されている車体の外形を表す輪郭線（ワイヤーフレーム）が合成される。合成された画像は記憶装置１０の合成画像データメモリ１７Ｃに保存される。ステップＳ４０９により画像合成手段が構成される。複数の画像を同一視点位置に視点変換した後、合成されるので、死角位置する対象物（他車両）と自車両との位置関係が、視覚で認識した場合に近い状態で、単一の画面に表示され、状況の把握が容易にできる。
【００４４】
合成画像の画像表示を行う（ステップＳ１０９）。側方画像と、後方画像は、視点変換された後、合成されるので、死角領域に位置する隣の車線の他車両と、後方車との位置関係が、視覚で視認した場合に近い状態で、単一の画面内に合成された表示されるので、運転者は把握が容易である。
【００４５】
ここで、測距センサ等によって車両周囲に物体を検知した場合は、警告表示を行ったり、画面のフラッシュを行ったりしてもよい。また、画像を表示する際に、後方画像の場合は、バックミラーに映されていると同じように画像を左右反転させることも可能である。運転者がバックミラーやサイドミラーを視認した時と同じ感覚で、後方の画像を認識でき、画像の内容を把握することが容易となる。この画像表示処理は、図６に示されているフローチャートに基づき説明する。
【００４６】
警告表示の要否を判断する（ステップＳ５０１）。
【００４７】
システムに搭載されている、測距センサ１９（本実施形態では、超音波センサ）により自車両周辺に障害物や他車両などが検知されたか否かの情報を取得し、警告表示が必要か否かを判断する。
【００４８】
警告表示データを準備する（ステップＳ５０３）。
【００４９】
ステップＳ５０１において、自車両周辺（例えば半径２ｍ以内等）に障害物などが検知された場合（警告が必要であると判断した場合）、警告表示をする準備を行う。なお、自車両周辺の定義は速度によって変更されてもよい。つまり、車速が速くなるにつれて、警告する範囲も広くする。例えば車速が０ｋｍ〜３０ｋｍまでは自車両半径２ｍだが、３０ｋｍ〜６０ｋｍまでは自車両半径５ｍと変更するなどである。
【００５０】
警告表示方法は、警告文字を画面に表示、画面色を通常の色から別の色に変える、フラッシュを行う、警告音を鳴らすなどである。
【００５１】
画像表示装置へ出力する（ステップＳ５０５）。以上のステップを経て決定された設定における画像を表示装置１４へ出力する。このステップＳ５０５の処理により表示手段が構成される。この処理の後、メインルーチンにリターンされる。
【００５２】
設定値の変更があるか判断する（ステップＳ１１１）。
【００５３】
画像の表示が終わると、設定値の変更があるかどうかを確認し（操作スイッチの入力等、運転者から新たな入力がないかをチェックする）、変更がある場合は設定値の変更を受付ける。変更がない場合には、ステップＳ１１３はスキップし、ステップＳ１１５を実行する。
【００５４】
設定値の変更をする（ステップＳ１１３）。なお、この設定値の変更は、ステップＳ６０９で行うことも可能である。
【００５５】
ステップＳ１１１にて、変更ありと判断された場合、変更すべき設定値を変更する。設定値変更時は画像処理を一旦中断しても、継続していても構わない（また、継続させる場合は別ルーチンで設定変更監視を行う構成とすることができる。）。
【００５６】
ここで変更する設定値は、合成パターンの選択時に変更される以外の設定値、つまり画像解像度や画像レート等である。
【００５７】
終了トリガを判断する（ステップＳ１１５）
続いてシステム終了トリガを確認し、トリガがなければデータ取得以後の処理を繰り返す。尚、システム終了のトリガはイグニッションＯＦＦ、シフトチェンジ（例えばＤポジション→Ｐポジション）、車速（例えば５ｋｍ／ｈ以下に減速したとき）、あるいはシステム終了用スイッチ等が考えられる。
【００５８】
なお、この判断は、別に終了トリガを監視するルーチンを使用して、終了トリガを確認したと同時に、すぐに画像処理を中断してしまう構成とすることもできる。
【００５９】
以上説明した実施形態の他、後退時において、後方カメラ１３Ｂの画像を用い、視野の広いバックモニタとして使用することもできる。魚眼カメラは、画角が１８０°に近いため、カメラの視点から車両の両側が同時に撮影できる。このため、見通しの悪い狭い場所から道路に後進で出る場合など、車両後部が道路に臨んだ位置で、見えない道路の状況を把握することができ、安全確認をするために有効に利用できる。また、側方カメラ１３Ｌ、１３Ｒを利用すれば、車両後方側の画像を利用して車体両側を同時に確認しながら後進移動できる。
【００６０】
さらに、カーナビゲーションシステムが有するデータを用いて、画像合成のために、一層精密な制御をすることもできる。カーナビゲーションシステムは、ユーザが所望する目的地までの案内を行うための地図データ（図示せず）を備えており、本願発明の道路情報記憶手段として機能する。この道路情報記憶手段としての地図データと、現在位置検出手段として機能する現在位置検出装置１５が検出する現在位置とを使用することにより、車両が位置する道路の種類、道路の形状、道路幅などの道路に関する状況を検出する道路状況検出手段を備えている。
【００６１】
このようなカーナビゲーションの機能によって、現在、自車両が走行している道路の状況、例えば、高速道路と一般道路の区別、交差点、合流地点、駐車場、道路の幅などの判別が可能となる。
【００６２】
例えば、高速道路を走行している場合には、車線変更、合流あるいは渋滞などへの注意が必要である。この場合、自車に隣接する車線の車の挙動に注意を払わなければならない。つまり側方カメラからの画像が重要となるので、そのような場所に位置することを検出した場合には、側方カメラの切り出し画角を多く取り、或は、側方カメラの画像のみで合成画像を構成するようにすることができる。
【００６３】
具体的には、自車両が高速道路の本線を走行中、左からの進入路線が検出された場合、後方撮影用カメラ１３Ｂは３０°、左側方撮影用カメラ１３Ｌは１２０°、右側方撮影用カメラ１３Ｒは３０°のような画角が設定され、車両速度の上昇に伴い、左側方カメラの画角を広げていく。このような画角にすることにより、左側進入車の確認が行えると共に、右側後方の走行車の確認も同時に行うことができる。
【００６４】
また、車両の現在位置と地図データとから車両が高速道路の進入路を走行していることが検出された場合、後方撮影用カメラ１３Ｂは３０°、左側方撮影用カメラ１３Ｌは０°、右側方撮影用カメラ１３Ｒは１５０°のような画角に設定され、車両速度の上昇に伴い、右側方カメラの画角を広げていく。この場合、高速道路本線を走行中の車に細心の注意を配ることができる。このような画角設定は、高速道路に限らず、一般道路の合流地点が検出された場合においても応用可能である。
【００６５】
また、車両の現在位置と地図データとから車両が交差点付近に位置することが検出され、車速の減速により交差点において自車両が曲がると推測された場合、方向指示器からの信号に基づいて、曲がる方向の周囲を表示し、減速に基づいて後方カメラの画角を狭めると共に側方カメラの画角を広くしていくことも可能である。この場合、側方カメラのみを使用することも可能である。このような画角にすると、前方の安全が確認できると共に、巻き込み防止の確認も可能となる。
【００６６】
また、地図データから現在位置が駐車場であると検出された場合、車速が０ｋｍ／ｈ付近への減速を検出したら、車庫入れと判断し、後方撮影用カメラ１３Ｂのみを使用してもよい。この場合、広範囲を確認可能なバックモニタとして利用することが可能となる。さらに、駐車の際にはステアリング切角から車の内輪側と外輪側で表示する周囲の方向を変更して表示することも可能である。
【００６７】
また、道幅の狭い道路を走行している場合には、対向車、駐車車両、軽車両、歩行者への注意等が必要である。接触事故防止のためにも、自車との位置関係が把握できる画像が望まれる。この場合にも、側方カメラによる画像を用いて表示画像を生成する構成とすることができる。
【００６８】
【発明の効果】
請求項１に記載の発明によれば、車速に応じて複数画像を張り合わせる割合を調整できるので、車速に応じて生じる死角領域に対して、これを十分に補完する合成画像を得ることでき、運転者の死角に対する不安感を減らすことができる。
【００６９】
請求項２に記載の発明によれば、車速が高くなる程、後方画像では把握できない領域が広がるので、側方画像取得手段によって後方画像取得手段による画像では把握できない領域の画像を補完し、後方から接近する死角領域の他車両を把握することが容易となる。
【００７０】
請求項３に記載の発明によれば、さらに走行中の道路に関する情報を取得することで、周囲の状況に応じた、自車両の周囲の画像を選択して合成することが可能となり、一層死角領域の削減を図ることができる。
【００７１】
請求項４に記載の発明によれば、自車の位置する道路状況の変化に応じて画像領域が決定されるので、道路状況の変化に応じて生じる死角領域の画像や、その他運転操作をするにあたって必要となる方向の画像を取得することができる。
【図面の簡単な説明】
【図１】本発明の画像処理装置の構成を示すブロック図である。
【図２】本発明の画像処理装置を搭載した車両と、画像取得手段の撮像範囲を示す平面図である。
【図３】本発明の画像処理装置の作用を示すフローチャートである。
【図４】本発明の画像処理装置の作用を示すフローチャートである。
【図５】本発明の画像処理装置の作用を示すフローチャートである。
【図６】本発明の画像処理装置の作用を示すフローチャートである。
【符号の説明】
１ 画像処理装置
１０ 記憶装置
１２ 演算処理装置
１３ カメラ
１４ 画像表示装置
１５ 現在位置検出装置
１６ 入力データメモリ
１７Ｓ 視点変換画像データメモリ
１８ 入力装置
１９ 測距センサ
２１ 自車[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing apparatus for a vehicle, and more particularly to an in-vehicle image processing apparatus that generates an image of a blind spot generated by a surrounding vehicle and assists a driver in confirming the surroundings of the vehicle.
[0002]
[Prior art]
The driver visually confirms the situation around the vehicle when operating the vehicle, but a blind spot area that cannot be visually recognized by the driver occurs due to the body of the own vehicle or due to the presence of obstacles. . Conventionally, in order to assist the driver in recognizing such a blind spot area, an image of the blind spot area is collected by a camera mounted on the vehicle body, and the image is displayed on a screen in the vehicle to assist the driver. Has been proposed. For example, such techniques include the following.
[0003]
[Patent Document 1]
JP-A-2002-166802
[0004]
[Problems to be solved by the invention]
On the other hand, when traveling on a highway or the like, it is necessary to turn the rudder while the vehicle speed is relatively high, and the relative speed with other vehicles may be high. In such a case, to change lanes, it is necessary to confirm whether there is a traveling vehicle in the lane to be entered. However, if a vehicle exists behind the vehicle, there is an area (blind spot area) behind the vehicle behind the vehicle. Occurs. When another vehicle approaching at high speed exists in the blind spot area, it is difficult to reliably recognize the vehicle. In particular, on a highway with a high speed limit, the relative speed with other vehicles may be high, and the blind spot area requiring confirmation of other vehicles is expanding.
The invention described in Patent Document 1 does not assume that the target moves at high speed in the blind spot area. That is, in the blind spot area, it may be difficult to recognize another vehicle approaching at high speed.
[0005]
SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing apparatus that recognizes a blind spot area of a driver and that can change an area for recognizing a blind spot area in accordance with a traveling speed of a vehicle. .
[0006]
[Means for Solving the Problems]
The above object is achieved by the present invention described below.
(1) a rear image acquisition unit that captures an image toward the rear of the vehicle and obtains an image of the rear;
An image is taken toward the side of the vehicle, and a side image obtaining unit that obtains an image of the side,
Vehicle speed detecting means for detecting the speed of the vehicle,
From the acquired rear and side images, an image area determining unit that determines an image area to be extracted according to the detected vehicle speed,
Combining means for combining the determined rear image area and the lateral image area;
Display means for displaying the combined image on a display device in the vehicle,
An in-vehicle image processing apparatus comprising:
[0007]
(2) The image region determining means increases the extraction ratio of the lateral image region as the speed of the vehicle increases.
The in-vehicle image processing device according to the above (1), wherein:
[0008]
(3) current position detecting means for detecting the current position of the vehicle;
Road information storage means in which road information is stored;
Road condition detecting means for detecting a road condition at the current position based on the current position of the vehicle and the road information,
The image area determination unit further determines an image area to be extracted from the rear image and the side image based on a change in road conditions at the current position.
The in-vehicle image processing device according to the above (1) or (2), wherein
[0009]
(4) a rearward image acquisition unit that captures an image toward the rear of the vehicle and obtains an image of the rear;
An image is taken toward the side of the vehicle, and a side image obtaining unit that obtains an image of the side,
Current position detecting means for detecting the current position of the vehicle,
Road information storage means in which road information is stored;
Road condition detecting means for detecting a road condition at a current position based on the current position of the vehicle and the road information;
Image area determining means for determining an image area to be extracted from the rear image and the side image based on a change in road conditions at the current position,
Combining means for combining the determined rear image area and the lateral image area;
Display means for displaying the synthesized image on a display device in the vehicle.
[0010]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a block diagram showing a configuration of an in-vehicle image processing device 1 according to the present invention. The in-vehicle image processing device 1 includes an arithmetic processing device 12 that performs image processing, cameras 13B, 13R, and 13L as three image acquisition units attached to the vehicle body, and a display device 14 as a display unit that displays images. A current position detecting device 15, an input device 18, a distance measuring sensor 19 as a distance measuring means, and a storage device 10 are provided, and these devices are interconnected via a system bus 11.
[0011]
FIG. 2 is an overall plan view of a car 21 equipped with the on-vehicle image processing device 1 of the present invention. The vehicles 21 are provided with cameras 13R and 13L as side image acquisition means provided toward the side of the vehicle body, and a camera 13B as rear image acquisition means at the rear end of the vehicle body. . Each of the cameras 13R, 13L, and 13B uses a fish-eye lens, and has a configuration for converting a light (image) signal into an electric signal by, for example, a CCD (Charge-Coupled Devices). The cameras 13R, 13L, and 13B are connected to the system bus 11 via camera A / D converters 131R, 131L, and 131B, respectively. The camera 13R uses the right side of the car 21 as the optical axis of the lens and acquires the surrounding scenery as an image. The camera 13L acquires the scenery around the left side of the car 21 as the optical axis of the lens as an image. The camera 13B uses the rear of the vehicle 21 as the optical axis of the lens and acquires the surrounding scenery as an image. In this embodiment, the angle of view of the fisheye lens is 180 degrees.
[0012]
The image signals output from the cameras 13R, 13L, 13B are converted into digital signals by A / D converters 131R, 131L, 131B, respectively. If the connected cameras 13R, 13L, 13B are capable of outputting digital signals, they are not necessary for the A / D converters 131R, 131L, 131B.
[0013]
The current position detecting device 15 functioning as current position detecting means includes a steering angle sensor 151, a vehicle speed sensor 152, a GPS receiving device 153, a direction sensor 154, and a distance sensor 155. The steering angle sensor 151 detects a steering angle of the vehicle. The steering angle is obtained by detecting the rotation angle of the steering wheel or the angle of the driving wheel. A vehicle speed sensor 152 as a vehicle speed detecting means detects a running speed of the vehicle. The GPS receiver 153 detects the absolute position of the vehicle. The direction sensor 154 detects the direction of the vehicle. The distance sensor 155 detects the moving distance of the vehicle.
[0014]
The image display device 14 is composed of, for example, a liquid crystal display or the like, and displays a composite image stored in the composite image data memory 17C. Data sent to the image display device 14 is converted into an analog signal via the D / A converter 141. When the image display device 14 can input digital signals, the D / A converter becomes unnecessary.
[0015]
The distance measuring sensor 19 is a sensor that measures the distance to the vehicle behind, and uses, for example, an ultrasonic wave, a microwave, a millimeter-wave radar, and the like, and uses an ultrasonic wave in this embodiment. The transmitted ultrasonic wave is reflected by the rear vehicle, and the distance from the rear vehicle is measured by the reflected ultrasonic wave. The distance measuring sensor 19 is connected to the system bus 11 via the interface 191. The input device 18 includes a keyboard, a pointing device, a touch screen configured by arranging a transparent panel in which pressure-sensitive elements are arranged in a matrix on the screen of the image display device 14, and the like. The input device 18 is connected to the bus line 11 via an interface 181. Through the input device 18, the driver performs input operations such as initial value input and mode conversion.
[0016]
The storage device 10 includes an input data memory 16 that stores data input from the cameras 13R, 13L, and 13B, the current position detection device 15, the input device 18, the distance measurement sensor 19, and the like, and a viewpoint conversion that stores an image obtained by performing a viewpoint conversion. An image data memory 17S, a combined image data memory 17C for storing a combined image, and an image processing setting data memory 17A are provided.
[0017]
The input data memory 16 includes a rear image data memory 161, a left image data memory 162, a right image data memory 163, a position data memory 164, and a time data memory 165. The rear image data memory 161 stores the image obtained by the camera 13B, the left image data memory 162 stores the image obtained by the camera 13L, and the right image data memory 163 stores the image obtained by the camera 13R. The stored image is stored. The position data memory 164 stores the position data detected by the current position detection device 15 (for example, the absolute position of the vehicle detected by the GPS reception device 153, the azimuth detected by the azimuth sensor 154, and the like). The time when the position data is detected is stored in the memory 165 using the GPS receiver 153 or a real-time clock (not shown).
[0018]
The viewpoint conversion image data memory 17S includes a rear image data memory 17B, a left image data memory 17L, and a right image memory 17R. In the rear image data memory 17B, a rear image obtained by converting a rear image acquired first to an image viewed from the viewpoint position of the driver is recorded. The left-side image data memory 17L stores a left-side image obtained by converting the first-acquired left-side image into an image when viewed from the driver's viewpoint position. In the right side image data memory 17R, a right side image obtained by converting the first acquired right side image into an image when the driver visually recognizes the right side image from the viewpoint position is recorded.
[0019]
The combined image data memory 17C stores an image obtained by combining images cut out from the side image and the rear image at a predetermined angle of view. The setting data memory 17A for image processing includes an extraction process for determining an image area to be extracted to create a composite image from each image, a viewpoint conversion process, and corrects a distortion of an image acquired by a fish-eye lens, thereby obtaining a normal image. Data necessary for normal image conversion processing, synthesis processing, and the like, and an outline (wire frame) representing the outer shape of the vehicle body are recorded.
[0020]
The arithmetic processing unit 12 includes a central processing unit [CPU (Central Processing Unit)] 121, a read-only memory [ROM] 122, and a random access memory [RAM] 123. The central processing unit 121 calculates, for example, information on the relative position between the preceding vehicle and the own vehicle from the position data and the direction of the vehicle obtained from the current position detecting device 15 and the received position data on the preceding vehicle, Alternatively, the image data of the own vehicle and the received image data of the preceding vehicle are acquired from the storage device 10 and the combined image data is generated from these data. The ROM 122 stores, for example, software for the central processing unit 121 to perform image processing, and the RAM 123 is used, for example, as a working area as a place for performing various image processing and the like.
[0021]
The image processing device of the present invention operates as follows. From the images acquired from the cameras 13B, 13L, 13R, an image area corresponding to a predetermined angle of view is cut out. A normal image conversion process is performed on the cut-out image. A viewpoint conversion process is performed on the image on which the normal image conversion process has been performed. The images subjected to the viewpoint conversion processing are combined to form a display image.
[0022]
Here, the image displayed on the display screen is formed by pasting and synthesizing images respectively cut out from the rear image and the left and right side images. The angle of view set for clipping the image from the side image and the rear image is set each time according to the road condition at which the vehicle is located. As shown in FIG. 2, when the vehicle is traveling on a highway and there is a vehicle behind, the vehicle C traveling in the overtaking lane is shaded by the vehicle B and is grasped only by the rear camera 13B. I can't. In this case, an image from which the car C can be grasped is created by combining the images of the right camera 13R. As the angle of view of the image to be cut out, as the vehicle speed increases, the angle of view α of the image cut out from the rearward image becomes smaller, and the angle of view β of the image cut out from the side image becomes larger. The reason that the angle of view β of each of the side cameras 13L and 13R is increased as the vehicle speed increases is that the image is easy to see with respect to another vehicle (in the case of FIG. 2, the vehicle C) running in the adjacent lane, and is unlikely to be a blind spot. This is because the use of many peripheral portions of the fisheye image reduces the overall image processing load and improves the drawing response during high-speed running.
[0023]
Also, when traveling at high speed, it is necessary to check for the presence of a vehicle on the side (adjacent lane) of the vehicle behind, and at low speeds, many of the surrounding vehicles are also at low speed. Is likely to be able to safely change lanes, but as the vehicle speed increases, the time required for another vehicle in the rear to approach the position of the own vehicle increases, so that the vehicle may contact another vehicle in the lane change. Therefore, it is necessary to take a particularly wide extraction angle of view of the side cameras 13L and 13R that can more surely project the far side of the adjacent lane.
[0024]
In particular, when driving at high speed, it is necessary to check the next lane before turning the steering wheel. For this reason, the image of the next lane is determined before the lane change is started (before turning the steering wheel) by the driver turning on the changeover switch or turning on the turn signal, and the extracted image angle is determined. Is displayed.
[0025]
The operation of the image processing apparatus 1 according to the present invention configured as described above will be described. FIG. 3 is a flowchart showing the operation of the arithmetic processing unit 12.
The process is started by, for example, an operation such as turning on an ignition or switching on a system. In addition, the trigger for switching on may be when the shift change lever is set to the D position. Alternatively, the switch may be turned on depending on the vehicle speed (for example, when the vehicle is accelerated to 10 km / h or more).
[0026]
When the system is started, an initialization process is performed (step S101). This is a process for reading the set values necessary for the system and preparing for image processing. That is, the data stored in the storage device is read out so that image data and the like can be written to the storage area after reading the setting values required for the processing recorded in the image processing setting data memory 17A of the storage device 10. If there is no area that can be stored, prepare to enable overwriting of the next data to be acquired, and delete the data that will not be used in the future by referring to the position and time stored with the image data I do.
[0027]
The set values include an image capture resolution, a range of extraction (extraction) of a fisheye image, an image capture rate, a camera installation position, a synthesized image viewpoint position, a display image (normal image or mirror image), and the like. Note that the viewpoint movement amount at the time of viewpoint conversion is obtained from the camera setting position and the synthesized image viewpoint position. Also, the user can change the set values of the combined pattern and the like. Such a change of the set value is performed via the input device 18.
[0028]
Next, an image is obtained from each camera (step S103). In the image acquisition process, all cameras 13R, 13L, and 13B are synchronized to acquire images at the same time (based on the set capture resolution and capture rate). At the same time as the image is acquired, the position and time are acquired by the GPS receiver 153. The time may be separately acquired using a real-time clock (not shown).
[0029]
The acquired image is stored (Step S105). Each captured image is stored in the input data memory 16 of the storage device 10 together with the position data and time when the image was acquired. That is, the rear image obtained by the camera 13B is stored in the rear image data memory 161, the left image obtained by the camera 13L is stored in the left image data memory 162, and the right image obtained by the camera 13R is stored in the right image data. In the memory 163, the position data is recorded in the position data memory 164, and the time is recorded in the time data memory 165.
[0030]
The stored image is processed (step S107). This image processing will be described based on a subroutine shown in FIG.
First, a synthesis pattern is selected (step S401).
A vehicle signal such as a vehicle speed and a shift position, a signal of an input device and the like (for example, a button press operation, a rotation operation of a dial SW, and the like) are obtained, and a composite image is formed from the images obtained from the cameras 13R, 13L, and 13B. An image area (hereinafter referred to as a cutout angle of view) and direction to be created are determined. This synthesis pattern selection process will be described based on a subroutine shown in FIG.
[0031]
Various information about the vehicle is obtained (step S601). Specifically, the signal from the direction indicator and the information on the vehicle speed from the vehicle speed sensor 152 are acquired. When the direction indicator is operating, the driver determines that the lane change is to be performed, and the direction indicator is output. Prepare to display a side image in the direction of Further, information of the vehicle speed for setting the cutout angle of view of the image area is received, and the process proceeds to the next step.
[0032]
It is determined whether to change the cutout view angle (step S603). Based on the information obtained in step S601, the angle of view to be used for each of the left, right, and rear fisheye images (the cutout angle of view of each fisheye image) is determined. The determination of the angle of view is performed such that, as the vehicle speed increases, the angle of the cutout angle of view β of the side image is increased, and the cutout angle of view α of the rear image is reduced. For example, the angle of view is determined according to the table shown in Table 1. The angle of view α is an angle opened right and left about the optical axis of the lens of the camera 13B, and is set in a range of 0 to 180 °. The angle of view β is an angle measured from the reference position toward the axis of the lens with respect to the end position that reflects the rear of the vehicle in the image of the fisheye lens, and is set in the range of 0 to 90 °.
[0033]
[Table 1]

If there is no need to change the angle of view, step S605 is skipped and step S607 is executed.
If it is determined that the cutout view angle needs to be changed, the cutout view angle or the image to be used is changed (step S605). That is, if there is a change, the set value is changed based on the determinations in steps S601 and S603. Steps S601 to S605 constitute an image area determining unit.
[0034]
It is determined whether or not there is an instruction from the input device (step S607). For example, a plurality of set values may be registered in advance as user settings of a composite pattern, and a composite pattern may be selected by a switch operation during operation.
[0035]
For example, a plurality of buttons such as abbreviated buttons are installed, the A button is a default, the B button is a left image set by the driver, the C button is a right image set by the driver, and the like. If there is no instruction, the process returns to the flowchart of FIG.
[0036]
The setting value for the composite image is read (step S609). If there is an instruction from the input device 18 to change the angle of view of the image to be displayed, etc., the input from the user has priority, so the setting value needs to be changed according to the instruction from the input device 18.
[0037]
The setting values changed from the input device include the angle of view of each camera, the display direction of the composite image, and the display side (right / left). After reading the set value, the process returns to the flowchart of FIG.
[0038]
An image clipping process is performed (step S403). Each fish-eye image stored in the storage device 10 is read out according to the setting value determined by the selection of the composite pattern (step S401), and a necessary image area is cut out (the position to be cut out can be operated by the user in real time). May be configured). The steps S401 and S403 constitute a rear image extracting unit and a side image extracting unit.
[0039]
Next, normal image conversion processing is performed (step S405).
[0040]
Since the clipped image is an image captured by a fisheye lens, a normal image conversion process for correcting distortion based on the lens characteristics is performed. Step S405 constitutes a normal image conversion unit.
[0041]
Further, a viewpoint conversion process is performed (step S407).
[0042]
Next, a viewpoint conversion process is performed on each image in order to make the image subjected to normal image conversion into an image viewed from the viewpoint position of the driver. The amount of viewpoint movement in the viewpoint conversion process is determined by the camera installation position set in the initialization process and the viewpoint position of the synthesized image. In the viewpoint conversion processing, the conversion in the front-rear direction corresponds to the image reduction processing. For vertical and horizontal conversion, the fisheye lens such as orthographic projection, stereoscopic projection, equidistant projection, and stereoscopic projection is used so that the viewpoint conversion destination (driver's viewpoint position) becomes the image center based on a predetermined value. Coordinate conversion is performed using a conversion formula determined by the projection method. The image generated in this way is a viewpoint-converted image. Each image whose viewpoint has been converted is stored in the viewpoint converted image data memory 17S of the storage device 10. This step S407 constitutes a viewpoint conversion unit.
[0043]
After the viewpoint conversion process, the images are combined (step S409). If necessary, a contour (wire frame) representing the outer shape of the vehicle body stored in the storage device 10 is synthesized. The synthesized image is stored in the synthesized image data memory 17C of the storage device 10. Step S409 constitutes an image synthesizing unit. Since a plurality of images are viewpoint-converted to the same viewpoint position and then synthesized, a single screen is displayed in a state where the positional relationship between the target object (other vehicle) located at the blind spot and the own vehicle is close to that visually recognized. Is displayed on the screen, and the situation can be easily grasped.
[0044]
Image display of the composite image is performed (step S109). The side image and the rear image are combined after the viewpoint is changed, so that the positional relationship between the other vehicle in the adjacent lane located in the blind spot area and the rear vehicle is close to that when visually recognized. , Are displayed in a single screen, so that the driver can easily grasp.
[0045]
Here, when an object is detected around the vehicle by a distance measurement sensor or the like, a warning display may be performed or a screen may be flashed. When displaying an image, in the case of a rear image, it is also possible to flip the image left and right in the same way as in the rearview mirror. The rear image can be recognized with the same feeling as when the driver visually recognizes the rearview mirror and the side mirror, and the content of the image can be easily grasped. This image display processing will be described based on the flowchart shown in FIG.
[0046]
It is determined whether a warning display is necessary (step S501).
[0047]
Information about whether an obstacle or another vehicle is detected around the own vehicle by a distance measurement sensor 19 (in the present embodiment, an ultrasonic sensor) mounted on the system is acquired, and whether a warning display is necessary is obtained. Judge.
[0048]
The warning display data is prepared (step S503).
[0049]
In step S501, when an obstacle or the like is detected around the own vehicle (for example, within a radius of 2 m) (when it is determined that a warning is necessary), preparation for displaying a warning is performed. Note that the definition of the area around the host vehicle may be changed according to the speed. That is, as the vehicle speed increases, the range of the warning is increased. For example, the radius of the own vehicle is 2 m when the vehicle speed is 0 km to 30 km, but the radius of the own vehicle is 5 m when the vehicle speed is 30 km to 60 km.
[0050]
The warning display method includes displaying a warning character on a screen, changing a screen color from a normal color to another color, performing a flash, and sounding a warning sound.
[0051]
Output to the image display device (step S505). An image with the settings determined through the above steps is output to the display device 14. The processing in step S505 constitutes a display unit. After this processing, the process returns to the main routine.
[0052]
It is determined whether there is a change in the set value (step S111).
[0053]
When the display of the image is completed, it is checked whether there is a change in the set value (check whether there is a new input from the driver, such as an input of an operation switch), and if there is a change, the change of the set value is accepted. . If there is no change, step S113 is skipped and step S115 is executed.
[0054]
The setting value is changed (step S113). Note that the change of the set value can be performed in step S609.
[0055]
If it is determined in step S111 that there is a change, the setting value to be changed is changed. When the setting value is changed, the image processing may be temporarily interrupted or may be continued (if the processing is continued, the setting change may be monitored by another routine).
[0056]
The setting value to be changed here is a setting value other than the value that is changed when the composite pattern is selected, that is, an image resolution, an image rate, and the like.
[0057]
A termination trigger is determined (step S115).
Subsequently, the system end trigger is confirmed, and if there is no trigger, the processing after data acquisition is repeated. The system end trigger may be an ignition OFF, a shift change (for example, from the D position to the P position), a vehicle speed (for example, when the speed is reduced to 5 km / h or less), or a system end switch.
[0058]
Note that this determination may be made by using a routine for separately monitoring the end trigger, confirming the end trigger, and immediately interrupting the image processing.
[0059]
In addition to the embodiment described above, when the vehicle retreats, it can be used as a back monitor having a wide field of view by using the image of the rear camera 13B. Since the angle of view of the fisheye camera is close to 180 °, both sides of the vehicle can be simultaneously photographed from the viewpoint of the camera. For this reason, the situation of the invisible road can be grasped at the position where the rear portion of the vehicle faces the road, such as when the vehicle is traveling backward from a narrow place with poor visibility, and can be effectively used to confirm safety. In addition, if the side cameras 13L and 13R are used, the vehicle can move backward while simultaneously checking both sides of the vehicle body using the image on the rear side of the vehicle.
[0060]
Further, more precise control for image composition can be performed using data of the car navigation system. The car navigation system includes map data (not shown) for providing guidance to a destination desired by the user, and functions as a road information storage unit of the present invention. By using the map data as the road information storage means and the current position detected by the current position detection device 15 functioning as the current position detection means, the type of road on which the vehicle is located, the shape of the road, the road width, etc. Road condition detecting means for detecting a condition related to the road.
[0061]
With such a car navigation function, it is possible to determine the state of the road on which the vehicle is currently traveling, for example, the distinction between an expressway and a general road, an intersection, a junction, a parking lot, the width of a road, and the like. .
[0062]
For example, when traveling on a highway, attention must be paid to changing lanes, merging, or congestion. In this case, attention must be paid to the behavior of the vehicle in the lane adjacent to the own vehicle. In other words, since the image from the side camera is important, if it is detected that the camera is located in such a place, the cutout angle of view of the side camera is increased, or the image is synthesized only with the image of the side camera. An image can be configured.
[0063]
Specifically, while the own vehicle is traveling on the main road of the highway, if an approach route from the left is detected, the rear camera 13B is 30 °, the left camera 13L is 120 °, and the right camera 13L is An angle of view such as 30 ° is set for the camera 13R, and the angle of view of the left camera is widened as the vehicle speed increases. With such an angle of view, it is possible to confirm a vehicle entering on the left side and simultaneously confirm a traveling vehicle on the right rear side.
[0064]
When it is detected from the current position of the vehicle and the map data that the vehicle is traveling on the approach road of the highway, the rear camera 13B is 30 °, the left camera 13L is 0 °, and the right camera 13L is right. The angle of view of the camera 13R is set to an angle of view such as 150 °, and the angle of view of the right camera is increased as the vehicle speed increases. In this case, it is possible to pay close attention to a car traveling on the main expressway. Such an angle of view setting is applicable not only to a highway but also to a case where a merging point of a general road is detected.
[0065]
In addition, when it is detected that the vehicle is located near the intersection from the current position of the vehicle and the map data, and it is estimated that the vehicle turns at the intersection due to a decrease in vehicle speed, the vehicle turns based on a signal from the turn signal. It is also possible to display the surroundings in the direction and to narrow the angle of view of the rear camera and widen the angle of view of the side camera based on the deceleration. In this case, it is also possible to use only the side camera. With such an angle of view, safety in the front can be confirmed, and prevention of entanglement can be confirmed.
[0066]
When it is detected from the map data that the current position is the parking lot, when the vehicle speed is detected to decrease to around 0 km / h, it is determined that the vehicle is to be put in the garage, and only the rear camera 13B may be used. In this case, it can be used as a back monitor that can confirm a wide range. Further, at the time of parking, it is also possible to change and display the surrounding directions displayed on the inner wheel side and the outer wheel side of the vehicle from the steering angle.
[0067]
When traveling on a narrow road, attention must be paid to oncoming vehicles, parked vehicles, light vehicles, pedestrians, and the like. In order to prevent a contact accident, an image that can grasp the positional relationship with the own vehicle is desired. Also in this case, a configuration in which a display image is generated using an image from the side camera can be adopted.
[0068]
【The invention's effect】
According to the first aspect of the present invention, since the rate at which a plurality of images are stuck together can be adjusted according to the vehicle speed, it is possible to obtain a composite image that sufficiently complements a blind spot generated according to the vehicle speed, It is possible to reduce the driver's anxiety about the blind spot.
[0069]
According to the second aspect of the present invention, as the vehicle speed increases, the area that cannot be grasped by the rear image increases, so that the image of the area that cannot be grasped by the image obtained by the rear image acquisition means is complemented by the side image acquisition means. It is easy to grasp other vehicles approaching from the blind spot area.
[0070]
According to the third aspect of the present invention, by further acquiring information on the road on which the vehicle is traveling, it becomes possible to select and synthesize an image around the own vehicle according to the surrounding situation, thereby further reducing the blind spot. The area can be reduced.
[0071]
According to the fourth aspect of the present invention, since the image area is determined according to the change in the road condition where the vehicle is located, an image of the blind spot area generated according to the change in the road condition and other driving operations are performed. It is possible to acquire an image in a direction required for this.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an image processing apparatus according to the present invention.
FIG. 2 is a plan view showing a vehicle equipped with the image processing apparatus of the present invention and an imaging range of an image acquisition unit.
FIG. 3 is a flowchart illustrating an operation of the image processing apparatus according to the present invention.
FIG. 4 is a flowchart illustrating an operation of the image processing apparatus according to the present invention.
FIG. 5 is a flowchart illustrating the operation of the image processing apparatus according to the present invention.
FIG. 6 is a flowchart illustrating the operation of the image processing apparatus according to the present invention.
[Explanation of symbols]
1 Image processing device
10 Storage device
12 Arithmetic processing unit
13 Camera
14 Image display device
15 Current position detection device
16 Input data memory
17S Viewpoint conversion image data memory
18 Input device
19 Distance sensor
21 Own car

Claims (4)

Backward image acquisition means for imaging toward the rear of the vehicle, and acquiring an image of the rear,
An image is taken toward the side of the vehicle, and a side image obtaining unit that obtains an image of the side,
Vehicle speed detecting means for detecting the speed of the vehicle,
From the acquired rear and side images, an image area determining unit that determines an image area to be extracted according to the detected vehicle speed,
Combining means for combining the determined rear image area and the lateral image area;
Display means for displaying the combined image on a display device in the vehicle,
An in-vehicle image processing apparatus comprising: 2. The in-vehicle image processing apparatus according to claim 1, wherein the image area determination unit increases an extraction ratio of the side image area as the speed of the vehicle increases. 3. Current position detecting means for detecting the current position of the vehicle,
Road information storage means in which road information is stored;
Road condition detecting means for detecting a road condition at the current position based on the current position of the vehicle and the road information,
3. The image area determining unit further determines an image area to be extracted from the rear image and the side image based on a change in road conditions at the current position. 2. The in-vehicle image processing device according to claim 1. Backward image acquisition means for imaging toward the rear of the vehicle, and acquiring an image of the rear,
An image is taken toward the side of the vehicle, and a side image obtaining unit that obtains an image of the side,
Current position detecting means for detecting the current position of the vehicle,
Road information storage means in which road information is stored;
Road condition detecting means for detecting a road condition at a current position based on the current position of the vehicle and the road information;
Image area determining means for determining an image area to be extracted from the rear image and the side image based on a change in road conditions at the current position,
Combining means for combining the determined rear image area and the lateral image area;
Display means for displaying the combined image on a display device in the vehicle,
An in-vehicle image processing apparatus comprising:

Images