JP2004212184A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of suitably removing distortion of images and simplifying an overall constitution in the case of controlling various apparatuses of a motor vehicle. <P>SOLUTION: The vicinity of the motor vehicle is photographed by a photographing part 10 at every prescribed time. On the basis of photographed images acquired by the continuous photographing, the history of fall of rain and snow etc. in the images are determined by a characteristic amount computation part 32. On the basis of the history of fall etc. in the images and the observed characteristics of fall speed, the actual history of fall of rain and snow are determined by a characteristic amount determining part 33. The locations of adhesion and the time of adhesion are estimated by an adhesion data estimation part 34 on the basis of the actual history of fall of rain and snow 50 etc. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

とすることができる。
【００３１】
また、画像は撮影部１０により一定時間ごとに取得されている。今、図４（ｂ）に示すように、時刻ｔにおける画像上の雨雪５０が、座標（ｘ１，ｙ１）の位置ｐ１であるとする。そして、Δｔ秒後の画像上の雨雪５０が、座標（ｘ２，ｙ２）の位置ｐ２であるとする。そして、雨雪５０の水平方向の移動を考慮しないことからｘ１＝ｘ２とすると、実際の雨雪５０の落下速度ｖは、
【数２】

とすることができる。
【００３２】
この式（２）に、式（１）を代入すると、実空間での雨雪５０の落下速度ｖは、
【数３】

とすることができる。ここで、Ｌ１は、時刻ｔにおける画像上の雨雪５０の大きさである。
【００３３】
また、式（３）において、ｙ１，ｙ２，Δｔ，Ｌ１は既知の数であるため、実際の雨雪５０の落下速度ｖは、
【数４】

とすることができる。この式（４）が、画像上から算出される実空間の雨雪５０の落下速度関数である。なお、落下速度関数は、実空間の雨雪５０までの距離Ｚが不明であるため、Ｓの１次式となっている。すなわち、距離Ｚが不明であると、図５に示すように、撮影部１０に近い側（距離ｚ１）の雨雪５０の画像上での大きさが遠い側（距離ｚ２）の雨雪５０の画像上での大きさのｚ１／ｚ２倍である場合、両者は同じ大きさとして撮影されるからであり、画像上からでは、小さい雨雪５０が近くにある場合と、大きい雨雪５０が遠くにある場合との見分けが付かないからである。このため、落下速度関数は、Ｓの１次式となってしまう。
【００３４】
よって、特徴量決定部３３は、実空間の特徴量を求めるために、落下速度関数だけでなく、実測落下速度特性も用いて、両者の交点αの値を実際の雨雪５０の落下速度等とするようにしている（図４（ｃ））。
【００３５】
また、上記は車両停止中においての交点αの判断であったが、車両走行中においては、自車両が走行しているため、次のようにして交点αの判断を行う。
【００３６】
車両が速度ｖｓで走行している場合、Δｔ秒後における実空間の雨雪５０の位置は、Ｚ−ｖｓ・Δｔとなるので、これを上記式（２）に当てはめると、
【数５】

とすることができる。
【００３７】
そして、式（１）を用いると、式（５）は、
【数６】

とすることができる。ここで、ｙ１，ｙ２，Ｌ１，ｖｓ，Δｔは既知の数であるため、式（６）は、
【数７】

とすることができる。この式（７）が車両走行中の落下速度関数である。
【００３８】
特徴量決定部３３は、車両が走行している場合、ステップＳＴ４において、この落下速度関数と、実測落下速度特性との交点を車両停車時と同様に求める（図４（ｃ））。
【００３９】
再び、図２を参照して説明する。落下速度関数と実測落下速度特性との交点がないと判断した場合（ＳＴ４：ＮＯ）、処理は終了する。一方、落下速度関数と実測落下速度特性との交点があると判断した場合（ＳＴ４：ＹＥＳ）、特徴量決定部３３は、雨雪５０の実空間の特徴量を決定する（ＳＴ５）。すなわち、上記式（４）または式（７）と実測落下速度特性との交点αから実空間の雨雪５０の落下速度ｖおよび大きさＳを求める。
【００４０】
そして、特徴量決定部３３は、実空間の雨雪５０の大きさＳを式（１）に代入して、撮影部１０内のレンズから雨雪５０までの距離Ｚを求め、雨雪５０の位置を決定する。これにより、特徴量決定部３３は、実空間における雨雪の特徴量（位置、落下速度、大きさ）を決定する。
【００４１】
このように、特徴量決定部３３は、画像上の雨雪の特徴量（大きさや落下速度）に基づいて、雨雪の実空間の特徴量（位置や落下速度）を求めていることになる。詳しくは、特徴量決定部３３は、画像上の雨雪の特徴量（大きさや落下速度）に基づいて、実空間における雨雪の大きさと落下速度との相関を示す落下速度関数を求め、さらに、雨雪の大きさと落下速度との実際の相関を示す実測落下速度特性を用いて、雨雪の実空間の特徴量（位置や落下速度）を求めている。
【００４２】
その後、付着データ推定部３４は、求められた雨雪の特徴量に基づいて、ウインド部５１への付着位置と付着時刻とを推定する（ＳＴ６）。ウインド部５１への付着位置は、車両停止中の場合、雨雪５０が垂直に落下しているとすると、雨雪５０の実空間での位置により推定することができる。また、ウインド部５１への付着時刻は、車両停止中の場合、雨雪５０が垂直に落下しているとすると、雨雪５０の実空間での位置と落下速度とにより推定できる。さらに、車両走行時においても、車速取得部２０により取得された車速に関する情報に基づき、付着位置と付着時刻との推定が可能となる。
【００４３】
その後、画像補正部３５は、雨雪が雨滴か雪かを判断する（ＳＴ７）。この判断は、例えば交点αに基づいて判断することができる。すなわち、画像補正部３５は、交点αが落下速度関数と雨滴の実測落下速度特性との交点である場合に雨雪を雨滴であると判断し、雪の実測落下速度特性との交点である場合に雨雪を雪であると判断する。
【００４４】
雪であると判断した場合、画像補正部３５は、雪の付着位置の画像を補正する（ＳＴ８）。具体的に画像補正部３５は、雪の付着位置の画像を削除する。雪が付着した場合に画像を削除するのは、雪がウインド部５１に付着して直ぐに溶けない場合などには、光路の変化による画像の歪み自体が発生しておらず、単に車両内部から外界が見えない状態になっているからである。そこで、ウインド部５１に付着したものが雪である場合、画像補正部３５は、撮影画像の雪付着部分を削除する補正を行うようにしている。そして、処理は終了する。
【００４５】
一方、雨滴であると判断した場合、画像補正部３５は、後述する拡大処理を行うか画像削除処理を行うかを判断する（ＳＴ９）。具体的に、画像補正部３５は、ウインド部５１に付着したにより、後述する拡大率が算出可能である場合には拡大処理を行うと判断し、拡大率が算出不可能である場合には削除処理を行うと判断する。削除処理を行うと判断した場合、処理は、ステップＳＴ８を経て終了する。ここで、ステップＳＴ７において雨滴であると判断されたのにもかかわらず、ステップＳＴ９において拡大率の算出不可能と判断される場合とは、例えば落下してくる雨雪が大きすぎて、拡大率が算出不可能な場合をいう。またこれに加えて、雨滴に塵などが入り交じり、非常に透明度が損なわれている場合なども拡大率が算出不可能として処理されることが考えられる。
【００４６】
一方、拡大処理を行うと判断した場合、画像補正部３５は、補正パラメータを算出し（ＳＴ１０）、雨滴の付着位置の画像を拡大し（ＳＴ１１）、処理は終了する。
【００４７】
ここで、画像削除処理（ＳＴ８）および拡大処理（ＳＴ１１）について詳説する。図６は、画像削除処理（ＳＴ８）および拡大処理（ＳＴ１１）の説明図であり、（ａ）は画像削除の範囲と雨雪５０の大きさとの関係を示しており、（ｂ）は雨滴付着時の光路の様子を示しており、（ｃ）は拡大処理（ＳＴ１１）にて求められる拡大率と雨雪の大きさとの関係を示している。
【００４８】
画像補正部３５は、画像削除処理（ＳＴ８）において、雨雪付着部分の撮影画像をマスクするようにしている。マスクの範囲は、雨雪の大きさが大きいほど広くなるようにされている。具体的には、図６（ａ）に示すように、雨雪の大きさに比例するようにマスクの範囲が広くなっている。
【００４９】
また、画像補正部３５は、画像補正処理（ＳＴ１１）において、雨滴付着部分の画像を拡大するようにしている。雨滴がウインド部５１に付着すると、図６（ｂ）に示すように、凸レンズと同じように作用して光を集光するようになる。このため、雨滴の付着部分については、車両外部の景色等は縮小して見えるようになる。従って、画像補正部３５は、雨滴の付着部分を拡大するように補正する必要がある。
【００５０】
この拡大する際、画像補正部３５は、雨滴の大きさが大きいほど画像の拡大率を大きくしている。具体的には、図６（ｃ）に示すように、雨滴の大きさに比例するように拡大率が大きくなっている。これは、雨滴の大きさが大きいほど、光を集光し雨滴付着部分の画像が小さく見える傾向にあるからである。
【００５１】
なお、画像補正部３５は、付着データ推定部３４により付着時刻に基づいて、上記画像補正（削除および拡大の双方）を雨雪５０のウインド部５１への近接時または付着時に行うことが望ましい。近接時または付着時に行うことで、画像の補正を行うタイミングが早すぎることによる画像への悪影響を防止することができるからである。
【００５２】
このようにして、本実施形態に係る画像処理装置１では、連続的に車両周辺を撮影して得られた撮影画像に基づいて、雨雪５０の画像上での特徴量（大きさ、落下速度など）を算出し、これと予め測定された雨雪の実測落下速度特性とに基づいて、実空間での雨雪５０の特徴量（位置、落下速度等）を決定する。そして、決定された実空間での雨雪５０の特徴量（位置、落下速度）に基づいて、雨雪５０のウインド部５１への付着位置を推定する。つまり、本実施形態では、画像上における雨雪５０の落下課程等を求め、画像上の落下課程等と、実測の落下速度特性とに基づき、実際の雨雪５０の落下課程等を得ている。そして、実際の雨雪５０の落下課程等から付着位置を推定している。
【００５３】
このように、本実施形態では雨雪５０の付着位置を推定することが可能となっている。このため、雨雪５０が実際にウインド部５１に付着する前であっても、推定された付着位置に基づく歪み除去を行うことで、除去に遅れが生じることなく、推定された付着位置の歪みを適切に除去することができる。
【００５４】
また、本実施形態では、撮影部１０によって得られた撮影画像から付着位置を推定している。このため、雨雪５０の検出用に他の要素を設ける必要がなくなっている。
【００５５】
従って、画像の歪みを好適に除去することが可能で、車両の各種機器を制御する場合には、全体としての構成の簡素化を図ることが可能となる。
【００５６】
また、車速に関する情報を取得するので、車両停止時のみでなく、車両走行時においても雨雪５０の付着位置および付着時刻を推定することが可能となり、利便性を向上させることができる。
【００５７】
また、雨雪５０の付着前に画像補正を行うということは、未だ雨雪が付着していないときの正常な画像に補正をかけることとなる。このため、画像補正のタイミングが早すぎると、画像に悪影響を及ぼしてしまう。ところが、本実施形態では、付着時刻に基づいて、画像補正のタイミングを雨雪５０がウインド部５１に近接したとき、または付着したときとするので、タイミングが早すぎることによる画像への悪影響を防止することができる。
【００５８】
なお、本発明は、上記した実施形態に限られるものではない。例えば、付着した雨滴上に雨滴が飛来してくることもある。この場合、新たに飛来してきた雨滴の大きさは上記実施形態の手法により容易に決定できるので、すでに付着している雨滴に新たに飛来した雨滴の大きさを加算するようにしてもよい。
【００５９】
また、本実施形態では、飛来物検出部３１は、検出した飛来物をすべて雨雪５０と推定するものであるが、これに限られるものではなく、例えば、テンプレートマッチングすることにより雨雪５０を推定するようにしてもよい。すなわち、小さな雨雪５０は空気抵抗が少なく撮影された雨雪５０の画像は円形に近くなる。このため、飛来物検出部３１に予め円形の画像を記憶しておき、テンプレートマッチングすることにより小さな雨雪５０の推定を精度よく行うようにすることもできる。また、大きな雨雪５０については、落下時に楕円形状となる傾向があることから、飛来物検出部３１に予め楕円形状の雨雪の画像を記憶しておき、テンプレートマッチングするようにしてもよい。
【図面の簡単な説明】
【図１】本発明の実施形態に係る画像処理装置の構成図である。
【図２】本実施形態に係る画像処理装置のフローチャートである。
【図３】雨雪がウインド部に付着するときの様子を示す説明図である。
【図４】落下速度関数と実測落下速度特性との交点を求める処理（ＳＴ４）の説明図であり、（ａ）は撮影部１０から実空間の雨雪５０までの距離等の関係を示し、（ｂ）は撮影部１０により連続的に得られる撮影画像の一例を示し、（ｃ）は落下速度関数と実測落下速度特性との交点等を示している。
【図５】雨雪が落下する際の様子を示す説明図である。
【図６】画像削除処理（ＳＴ８）および拡大処理（ＳＴ１１）の説明図であり、（ａ）は画像削除の範囲と雨雪の大きさとの関係を示しており、（ｂ）は雨滴付着時の光路の様子を示しており、（ｃ）は拡大処理（ＳＴ１１）にて求められる拡大率と雨雪の大きさとの関係を示している。
【符号の説明】
１ 画像処理装置
１０ 撮影部（撮影手段）
２０ 車速取得部（車速取得手段）
３１ 飛来物検出部（雨雪推定手段）
３２ 特徴量算出部（特徴量算出手段）
３３ 特徴量決定部（特徴量決定手段）
３４ 付着データ推定部（付着データ推定手段）
３５ 画像補正部（画像補正手段）
５０ 雨雪
５１ ウインド部[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing device.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, there has been known an image processing apparatus having a function of photographing an area around a vehicle by a camera or the like, performing image processing on an obtained photographed image, and controlling various devices of the vehicle based on a result of the image processing ( See, for example, Patent Document 1.
[0003]
However, in this device, when rain or snow is falling, if a raindrop or snow adheres to a lens of the camera or a window portion in front of the camera, the captured image is distorted. That is, a change in the optical path occurs due to, for example, raindrops acting as a convex lens, and the captured image is distorted.
[0004]
In order to remove the distortion of the captured image, it is necessary to detect rain and snow before the rain or snow adheres to the lens or the window, and to know the position of the lens or window where the rain or snow adheres. The reason that rain / snow is detected before attachment is that if rain / snow is detected after the attachment of rain / snow, the image has already been distorted due to raindrops, etc. at the time of rain / snow detection, causing a delay in removing the distortion. is there. Knowing the attachment position is important for specifying the distortion position in the image.
[0005]
On the other hand, as a detection device that detects raindrops and the like before attachment and specifies the attachment position, a light-emitting unit and a light-receiving unit that receives band-shaped light projected from the light-emitting unit are provided, and when raindrops or snow passes between them, Utilizing the fact that the characteristics of the band-shaped light changes, an apparatus has been proposed that can detect raindrops and snow before the raindrops adhere and also know the adhesion position of the raindrops and the like to some extent (for example, see Patent Document 2). ).
[0006]
Therefore, by using this apparatus, it is not necessary to add a function of removing distortion of a captured image to the image processing apparatus, and it is possible to preferably remove distortion.
[0007]
[Patent Document 1]
JP 2001-147278 A
[Patent Document 2]
JP-A-10-82741
[Problems to be solved by the invention]
However, when the image distortion is removed using the device described in Patent Document 2, elements such as a light projecting unit and a light receiving unit are required, and a function of removing the image distortion is incorporated in the image processing device. Compared with the case, the configuration as a whole becomes complicated.
[0010]
The present invention has been made to solve such a conventional problem, and an object thereof is to appropriately remove image distortion and to control various devices of a vehicle. An object of the present invention is to provide an image processing apparatus capable of simplifying the configuration as a whole.
[0011]
[Means for Solving the Problems]
In order to achieve the above object, according to the present invention, the photographing unit continuously photographs the periphery of the vehicle, and the rain / snow estimation unit detects a flying object from a photographed image photographed by the photographing unit and estimates it as rain / snow. A feature amount calculating unit calculates a feature amount on the image of the rain and snow estimated by the rain and snow estimating unit based on the photographed images continuously photographed by the photographing unit; The feature amount on the image of the rain and snow calculated by the amount calculating means, and the correlation between the size and the falling speed of the rain and snow in the real space and based on the actually measured falling speed characteristics of the rain and snow measured in advance, The characteristic amount of rain and snow in the real space is determined, and the adhesion data estimating means determines the position of the rain and snow to be attached to the window based on the characteristic amount of rain and snow in the real space determined by the characteristic amount determining means. presume.
[0012]
【The invention's effect】
According to the present invention, a feature amount (for example, a position, a size, and a falling speed of rain and snow) on a rain and snow image is calculated based on a photographed image obtained by continuously photographing the periphery of a vehicle. Based on this and the actually measured falling snow velocity characteristics of rain and snow, the characteristic amount of rain and snow in the real space (for example, the position, size and falling speed of rain and snow) is determined. Then, based on the determined characteristic amount of rain and snow in the real space, the position where rain and snow adhere to the window is estimated. That is, in the present invention, the rainfall process of rain and snow on the image is obtained, and the actual rainfall process of rain and snow is obtained based on the fall process on the image and the actually measured fall velocity characteristics. Then, the attachment position is estimated from the actual falling process of rain and snow.
[0013]
As described above, according to the present invention, it is possible to estimate the attachment position of rain and snow. Therefore, even before rain or snow actually adheres to the window portion, by performing distortion removal based on the estimated attachment position, the distortion of the estimated attachment position can be appropriately adjusted without delay. Can be removed.
[0014]
Further, in the present invention, the attachment position is estimated from the photographed image obtained by the photographing means. Therefore, it is not necessary to provide another element for detecting rain and snow.
[0015]
Therefore, it is possible to preferably remove image distortion, and to control various devices of the vehicle, it is possible to simplify the overall configuration.
[0016]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, a preferred embodiment of the present invention will be described with reference to the drawings.
[0017]
FIG. 1 is a configuration diagram of an image processing apparatus according to an embodiment of the present invention. As shown in FIG. 1, the image processing apparatus 1 includes a photographing unit (photographing means) 10 which is installed inside the vehicle and photographs the periphery of the vehicle every predetermined time. Further, the image processing apparatus 1 is configured to acquire information on a vehicle speed based on a vehicle speed acquisition unit (vehicle speed acquisition unit) 20, an image photographed by the photographing unit 10, and vehicle speed information acquired by the vehicle speed acquisition unit 20. A processing unit 30 for performing image processing.
[0018]
Here, the photographing unit 10 is a high frame rate photographing unit that can photograph, for example, about 1000 images per second. The vehicle speed acquisition unit 20 detects a traveling distance and a traveling direction of the own vehicle by using a method such as dead reckoning using a wheel speed sensor or a steering angle sensor. In addition, the vehicle speed acquisition unit 20 may be one that detects the position and speed of the own vehicle by GPS. The vehicle speed acquisition unit 20 acquires various information, but transmits only information relating to the vehicle speed to the processing unit 30.
[0019]
The processing unit 30 detects a flying object from the photographed image acquired by the photographing unit 10 and estimates it as rain / snow, and a flying object detection unit (rain / snow estimation unit) 31 and a photographing continuously photographed by the photographing unit 10. A feature amount calculation unit (feature amount calculation unit) 32 that calculates a feature amount on the image of rain and snow estimated by the flying object detection unit 31 based on the image is provided. Here, the characteristic amount indicates a characteristic relating to the state or movement of rain and snow, and includes, for example, at least one of the size of rain and snow, the position of rain and snow, and the falling speed. Also, rain and snow are raindrops, snow, or a mixture of them.
[0020]
Further, the processing unit 30 determines a feature amount of rain and snow in the real space based on the feature amount of rain and snow on the image calculated by the feature amount calculation unit 32 (a feature amount determination unit) 33. And an adhesion data estimating unit (adhesion data estimating means) 34 for estimating the position and time of attachment of rain and snow to the window based on the characteristic amount of the real space determined by the characteristic amount determining unit 33. I have. Further, the processing unit 30 includes an image correction unit (image correction unit) 35 that corrects an image based on the result of estimation by the adhesion data estimation unit 34.
[0021]
Further, an external device control unit 40 that controls an external device based on the image corrected by the image correction unit 35 is connected to the image processing device 1. The external device control unit 40 controls, for example, the wiper and the driving operation itself based on the image data from the image correction unit 35.
[0022]
FIG. 2 is a flowchart of the image processing apparatus 1 according to the present embodiment. First, the image capturing unit 10 captures an image around the vehicle to acquire an image (ST1). FIG. 3 is an explanatory diagram showing how rain and snow adhere to the window. As shown in FIG. 3, the rain / snow 50 falls with a substantially constant-velocity linear motion (time t1 to t4 are all equally spaced). Then, the rain / snow 50 adheres to the window 51 at time t5. The photographing unit 10 photographs about 1000 drops of the constant velocity linear motion (including the attachment to the window unit 51) per second, and sequentially outputs photographed image data to the processing unit 30.
[0023]
The rain / snow 50 at the time of falling is usually substantially circular, but when it exceeds a certain size or more, it receives an air resistance and forms an elliptical shape which spreads horizontally. Further, the actual rain / snow 50 moves in the horizontal direction to receive wind or the like while falling, but in the present embodiment, the horizontal movement of the rain / snow 50 is not considered. This is because the influence of the rain / snow 50 in the horizontal direction is extremely small because the photographing unit 10 having a high frame rate is used as the photographing unit 10. In addition, during traveling of the vehicle, the amount of movement of the rain / snow 50 in the horizontal direction is extremely smaller than the traveling speed of the vehicle.
[0024]
Description will be made with reference to FIG. 2 again. After acquiring the image, the flying object detection unit 31 detects the flying object in the image and estimates it as rain / snow 50 (ST2). Specifically, the flying object detection unit 31 detects a flying object and estimates all detected objects as raindrops or snow. Then, the flying object detection unit 31 transmits the estimation result to the feature amount calculation unit 32. In addition, the flying object detection unit 31 outputs a captured image continuously acquired by the imaging unit 10 to the feature amount calculation unit 32 together with the estimation result.
[0025]
Thereafter, the feature amount calculating unit 32 determines the position and size of the feature amount of the rain / snow 50 on the image based on the captured images continuously captured by the capturing unit 10 and the detection result of the flying object detection unit 31. The height and the falling speed are calculated (ST3). For example, the feature amount calculation unit 32 obtains the size and position of the rain / snow 50 on the image (corresponding to the size and position of the image formed by the image sensor 12) from the image captured this time, The moving amount of the position is obtained from the photographed image, and the falling speed (the moving amount of the rain / snow 50 imaged on the image sensor 12 per predetermined time) is calculated based on the moving amount and the photographing interval Δt of the photographing unit 10. Equivalent) is calculated.
[0026]
At this time, the feature amount calculation unit 32 obtains the size of the rain / snow 50 on the image from, for example, the width of the outer shape of the rain / snow 50. In addition, the feature amount calculation unit 32 specifies the position of the rain / snow 50 on the image as a position that becomes the center of gravity of the rain / snow 50 such as a circular shape or an elliptical shape. If it is clear that the target flying object has not fallen on the image, the flying object is excluded from processing as objects other than rain and snow.
[0027]
After the calculation of the characteristic amount, the characteristic amount calculation unit 32 outputs information on the calculated characteristic amount and information on the vehicle speed from the vehicle speed acquisition unit 20 to the characteristic amount determination unit 33.
[0028]
Then, the feature amount determining unit 33 receives the information and calculates a falling speed function indicating a correlation between the size of the rain / snow 50 and the falling speed. After the calculation of the function, the feature amount determination unit 33 determines the correlation between the falling speed function of the rain / snow 50 and the actually measured falling speed characteristic of the rain / snow 50 that indicates the correlation between the size and the falling speed of the rain / snow 50. It is determined whether there is an intersection (ST4). Hereinafter, this processing will be described in detail.
[0029]
FIG. 4 is an explanatory diagram of a process (ST4) for obtaining an intersection between the falling speed function and the actually measured falling speed characteristic, and FIG. 4 (a) shows a relationship such as a distance from the photographing unit 10 to the rain / snow 50 in the real space. (B) shows an example of a photographed image continuously obtained by the photographing unit 10, and (c) shows an intersection or the like between the fall velocity function and the measured fall velocity characteristic. Note that FIG. 4B shows an image obtained by superimposing continuously obtained captured images into one image for ease of explanation.
[0030]
The image capturing unit 10 captures an image of rain and snow 50 in real space with the image sensor 12 via the lens 11. At this time, as shown in FIG. 4A, the size of the actual rain / snow 50 is S, the distance from the lens 11 to the actual rain / snow 50 is Z, and the distance from the image sensor 12 to the lens 11 is f, and the size of the rain / snow 50 imaged on the image sensor 12 is L, the actual size S of the rain / snow 50 is
(Equation 1)

It can be.
[0031]
The images are acquired by the photographing unit 10 at regular intervals. Now, as shown in FIG. 4B, it is assumed that the rain / snow 50 on the image at the time t is the position p1 of the coordinates (x1, y1). Then, it is assumed that the rainy snow 50 on the image after Δt seconds is the position p2 of the coordinates (x2, y2). Then, since x1 = x2 because the horizontal movement of the rain / snow 50 is not considered, the actual falling speed v of the rain / snow 50 is:
(Equation 2)

It can be.
[0032]
By substituting equation (1) into equation (2), the falling speed v of the rain / snow 50 in the real space becomes
[Equation 3]

It can be. Here, L1 is the size of rain / snow 50 on the image at time t.
[0033]
Further, in equation (3), since y1, y2, Δt, and L1 are known numbers, the actual falling speed v of the rain / snow 50 is
(Equation 4)

It can be. Equation (4) is a falling speed function of the rain / snow 50 in the real space calculated from the image. The falling velocity function is a linear expression of S because the distance Z to the rain / snow 50 in the real space is unknown. In other words, if the distance Z is unknown, as shown in FIG. 5, the size of the rain / snow 50 on the far side (distance z2) on the image of the rain / snow 50 on the side closer to the imaging unit 10 (distance z1) is larger. This is because when the size is z1 / z2 times the size on the image, the two are photographed as the same size. From the image, when the small rainy snow 50 is close and when the large rainy snow 50 is far, Because it cannot be distinguished from the case in Therefore, the falling speed function is a linear expression of S.
[0034]
Therefore, the feature value determination unit 33 uses the measured fall speed characteristics as well as the fall speed function to determine the value of the intersection α between the actual rain speed and the fall speed of the rain / snow 50 in order to obtain the feature value of the real space. (FIG. 4C).
[0035]
In the above description, the intersection α is determined while the vehicle is stopped. Since the own vehicle is running while the vehicle is traveling, the intersection α is determined as follows.
[0036]
When the vehicle is traveling at the speed vs, the position of the rain / snow 50 in the real space after Δt seconds is Z−vsΔt, and when this is applied to the above equation (2),
(Equation 5)

It can be.
[0037]
Then, using equation (1), equation (5) becomes
(Equation 6)

It can be. Here, since y1, y2, L1, vs, and Δt are known numbers, equation (6) is
(Equation 7)

It can be. This equation (7) is a falling speed function during the running of the vehicle.
[0038]
When the vehicle is running, in step ST4, the characteristic amount determination unit 33 obtains an intersection between the falling speed function and the measured falling speed characteristic in the same manner as when the vehicle is stopped (FIG. 4C).
[0039]
Description will be made again with reference to FIG. If it is determined that there is no intersection between the falling speed function and the measured falling speed characteristic (ST4: NO), the process ends. On the other hand, when it is determined that there is an intersection between the falling speed function and the measured falling speed characteristic (ST4: YES), the feature amount determining unit 33 determines the feature amount of the rain / snow 50 in the real space (ST5). That is, the falling speed v and the size S of the rain / snow 50 in the real space are obtained from the intersection α between the above formula (4) or the formula (7) and the actually measured falling speed characteristic.
[0040]
Then, the feature quantity determining unit 33 substitutes the size S of the rain / snow 50 in the real space into the equation (1), obtains a distance Z from the lens in the photographing unit 10 to the rain / snow 50, and calculates the distance Z of the rain / snow 50. Determine the position. Thereby, the characteristic amount determination unit 33 determines the characteristic amounts (position, falling speed, size) of rain and snow in the real space.
[0041]
As described above, the feature amount determination unit 33 obtains the feature amount (position and fall speed) of the real space of rain and snow based on the feature amount (size and fall speed) of rain and snow on the image. . More specifically, the feature amount determination unit 33 obtains a falling speed function indicating a correlation between the size and the falling speed of the rain and snow in the real space based on the feature amounts (the size and the falling speed) of the rain and snow on the image. In addition, the characteristic amounts (positions and falling speeds) of the real space of rain and snow are obtained by using the actually measured falling speed characteristics indicating the actual correlation between the size of rain and snow and the falling speed.
[0042]
Thereafter, the attachment data estimation unit 34 estimates the attachment position and the attachment time to the window 51 based on the obtained rain / snow feature amounts (ST6). When the vehicle is stopped, the rainy snow 50 can be estimated from the position of the rainy snow 50 in the real space assuming that the rainy snow 50 is vertically falling when the vehicle is stopped. Further, when the rainy snow 50 is falling vertically when the vehicle is stopped, the time of attachment to the window 51 can be estimated from the position of the rainy snow 50 in the real space and the falling speed. Further, even during traveling of the vehicle, it is possible to estimate the attachment position and the attachment time based on the information on the vehicle speed acquired by the vehicle speed acquisition unit 20.
[0043]
Thereafter, the image correction unit 35 determines whether the rain or snow is raindrops or snow (ST7). This determination can be made, for example, based on the intersection α. That is, the image correction unit 35 determines that rain or snow is a raindrop when the intersection α is the intersection of the falling velocity function and the measured falling velocity characteristic of raindrops, and determines that the intersection α is the intersection with the measured falling velocity characteristic of snowdrops. It is determined that rain is snow.
[0044]
If it is determined that the snow is present, the image correction unit 35 corrects the image at the position where the snow is attached (ST8). Specifically, the image correction unit 35 deletes the image of the position where snow is attached. The reason why the image is deleted when snow adheres is that when the snow adheres to the window portion 51 and does not melt immediately, the image itself is not distorted due to a change in the optical path, and the outside world is simply seen from inside the vehicle. This is because there is no state. Therefore, when the thing attached to the window 51 is snow, the image correcting unit 35 performs a correction for deleting a snow-attached part of the captured image. Then, the process ends.
[0045]
On the other hand, when determining that the image is a raindrop, the image correction unit 35 determines whether to perform an enlargement process or an image deletion process described later (ST9). Specifically, the image correction unit 35 determines that the enlargement process is to be performed when the enlargement ratio described later can be calculated due to the attachment to the window unit 51, and deletes the image when the enlargement ratio cannot be calculated. It is determined that processing will be performed. If it is determined that the deletion process is to be performed, the process ends after step ST8. Here, the case where it is determined in step ST9 that the enlargement ratio cannot be calculated in spite of the fact that it is determined in step ST7 that it is a raindrop means that, for example, the rain snow falling is too large and the enlargement ratio is large. Means that it is not possible to calculate. In addition to this, it is conceivable that the enlargement factor is processed as being uncalculable even when dust or the like mixes with the raindrops and the transparency is extremely impaired.
[0046]
On the other hand, if it is determined that the enlargement process is to be performed, the image correction unit 35 calculates a correction parameter (ST10), enlarges the image of the position where the raindrop is attached (ST11), and ends the process.
[0047]
Here, the image deletion processing (ST8) and the enlargement processing (ST11) will be described in detail. FIGS. 6A and 6B are explanatory diagrams of the image deletion process (ST8) and the enlargement process (ST11). FIG. 6A shows the relationship between the range of image deletion and the size of the rain / snow 50, and FIG. The state of the optical path at the time is shown, and (c) shows the relationship between the enlargement ratio obtained in the enlargement processing (ST11) and the size of rain and snow.
[0048]
The image correction unit 35 masks the photographed image of the rain-and-snow attached portion in the image deletion process (ST8). The range of the mask is made wider as the size of rain and snow increases. Specifically, as shown in FIG. 6A, the range of the mask is increased in proportion to the size of rain and snow.
[0049]
The image correction section 35 enlarges the image of the raindrop-attached portion in the image correction processing (ST11). When the raindrops adhere to the window portion 51, as shown in FIG. 6 (b), they act like a convex lens to condense light. For this reason, in the portion where the raindrops adhere, the scenery and the like outside the vehicle appear to be reduced. Therefore, the image correction unit 35 needs to perform correction so as to enlarge the portion where the raindrop is attached.
[0050]
At the time of enlargement, the image correction unit 35 increases the enlargement ratio of the image as the size of the raindrop increases. Specifically, as shown in FIG. 6C, the enlargement ratio increases in proportion to the size of the raindrop. This is because the larger the size of the raindrop, the more the light is condensed and the image of the portion where the raindrop is attached tends to appear smaller.
[0051]
It is desirable that the image correction unit 35 performs the above-described image correction (both deletion and enlargement) when the rain / snow 50 approaches or adheres to the window 51 based on the adhesion time by the adhesion data estimation unit 34. This is because by performing the process at the time of approach or at the time of adhesion, it is possible to prevent an adverse effect on the image due to too early a timing of performing the image correction.
[0052]
Thus, in the image processing apparatus 1 according to the present embodiment, the characteristic amounts (the size, the falling speed, etc.) on the image of the rain / snow 50 are based on the captured image obtained by continuously capturing the surroundings of the vehicle. And the like, and the characteristic amount (position, falling speed, etc.) of the rain / snow 50 in the real space is determined based on this and the actually measured falling speed characteristics of the rain / snow. Then, based on the determined feature amounts (positions, falling speeds) of the rain / snow 50 in the real space, the position where the rain / snow 50 is attached to the window 51 is estimated. That is, in the present embodiment, the fall process of the rain / snow 50 on the image is obtained, and the actual fall process of the rain / snow 50 is obtained based on the fall process on the image and the actually measured fall speed characteristics. . Then, the attachment position is estimated from the actual falling process of the rain / snow 50.
[0053]
As described above, in the present embodiment, it is possible to estimate the attachment position of the rain and snow 50. For this reason, even before the rain / snow 50 actually adheres to the window portion 51, by performing the distortion removal based on the estimated adhesion position, the removal of the estimated adhesion position can be performed without delay. Can be appropriately removed.
[0054]
Further, in the present embodiment, the attachment position is estimated from the photographed image obtained by the photographing unit 10. Therefore, it is not necessary to provide another element for detecting the rain / snow 50.
[0055]
Therefore, it is possible to preferably remove image distortion, and to control various devices of the vehicle, it is possible to simplify the overall configuration.
[0056]
Further, since the information on the vehicle speed is acquired, it is possible to estimate the attachment position and the attachment time of the rain / snow 50 not only when the vehicle is stopped but also when the vehicle is running, and the convenience can be improved.
[0057]
In addition, performing image correction before the attachment of rain and snow 50 means that a normal image when rain and snow is not attached yet is corrected. Therefore, if the timing of the image correction is too early, the image is adversely affected. However, in the present embodiment, based on the attachment time, the timing of the image correction is set when the rain / snow 50 approaches the window portion 51 or when the attachment occurs, so that an adverse effect on the image due to too early timing is prevented. can do.
[0058]
Note that the present invention is not limited to the above embodiment. For example, a raindrop may fly on the attached raindrop. In this case, since the size of the newly arriving raindrop can be easily determined by the method of the above embodiment, the size of the newly arriving raindrop may be added to the already attached raindrop.
[0059]
In the present embodiment, the flying object detection unit 31 estimates all the detected flying objects as rain and snow 50. However, the present invention is not limited to this. For example, the flying object 50 is detected by performing template matching. It may be estimated. That is, the image of the rain and snow 50 photographed by the small rain and snow 50 has low air resistance and is close to a circle. For this reason, it is also possible to store a circular image in the flying object detection unit 31 in advance, and accurately estimate the small rain / snow 50 by performing template matching. Further, since the large rain / snow 50 tends to have an elliptical shape when dropped, an oval rain / snow image may be stored in the flying object detection unit 31 in advance, and template matching may be performed.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of an image processing apparatus according to an embodiment of the present invention.
FIG. 2 is a flowchart of the image processing apparatus according to the embodiment.
FIG. 3 is an explanatory diagram showing a state when rain and snow adhere to a window portion.
FIG. 4 is an explanatory diagram of a process (ST4) for obtaining an intersection between a falling speed function and an actually measured falling speed characteristic, where (a) shows a relationship such as a distance from the photographing unit 10 to rain / snow 50 in a real space; (B) shows an example of a photographed image continuously obtained by the photographing unit 10, and (c) shows an intersection or the like between the fall velocity function and the measured fall velocity characteristic.
FIG. 5 is an explanatory view showing a state when rain and snow fall.
6A and 6B are explanatory diagrams of an image deletion process (ST8) and an enlargement process (ST11), wherein FIG. 6A shows a relationship between a range of image deletion and the size of rain and snow, and FIG. (C) shows the relationship between the enlargement ratio obtained in the enlargement process (ST11) and the size of rain and snow.
[Explanation of symbols]
1 image processing apparatus 10 photographing unit (photographing means)
20 Vehicle speed acquisition unit (vehicle speed acquisition means)
31 Flying object detector (rain / snow estimation means)
32 feature amount calculation unit (feature amount calculation means)
33 feature amount determination unit (feature amount determination means)
34 adhesion data estimation unit (adhesion data estimation means)
35 Image Correction Unit (Image Correction Means)
50 Rain and snow 51 Wind section

Claims (4)

Photographing means for continuously photographing around the vehicle;
Rain / snow estimation means for detecting a flying object from a photographed image taken by the photographing means and estimating rain / snow,
A feature amount calculating unit that calculates a feature amount on the image of the rain and snow estimated by the rain and snow estimating unit, based on the photographed images continuously photographed by the photographing unit;
The feature amount on the image of the rain and snow calculated by the feature amount calculating means, and the correlation between the size and the falling speed of the rain and snow in the real space, and based on the measured falling speed characteristics of the rain and snow measured in advance. Feature amount determining means for determining a feature amount in the real space of rain and snow;
An adhesion data estimating means for estimating an adhesion position of rain and snow on a window portion based on the characteristic amount in the real space of rain and snow determined by the characteristic amount determining means,
An image processing apparatus comprising: Further provided is a vehicle speed acquisition means for acquiring information on the vehicle speed,
The adhesion data estimating means includes, in addition to the characteristic amount of the rain and snow in the real space determined by the characteristic amount determining means, based on the information about the vehicle speed acquired by the vehicle speed acquiring means, and The image processing apparatus according to claim 1, wherein a position of attachment to the window is estimated. An image correcting unit that corrects a rain-and-snow-adhered portion in a photographed image taken by the photographing unit, based on the attachment position of the rain / snow estimated by the attachment data estimating unit;
The adhesion data estimation unit estimates the time of attachment of the rain and snow to the window based on the feature amount in the real space of rain and snow determined by the feature amount determination unit,
2. The image processing apparatus according to claim 1, wherein the image correction unit performs correction based on the adhesion time when the raindrop approaches or adheres to the window. The size of rain and snow is used as the feature amount on the image of the rain snow and the feature amount of the rain snow in the real space, and the feature amount includes at least one of the position of the rain snow and the falling speed. The image processing apparatus according to claim 1, wherein:

Images