JP5891723B2 - Imaging method and imaging unit - Google Patents

Description

  The present invention relates to an imaging method and an imaging unit suitable for detecting foreign matter adhering to the surface of a windshield of a vehicle.

Conventionally, as an image processing system for detecting foreign matter adhering to the surface of a windshield of a vehicle, those described in Patent Documents 1 to 4 are known.
Patent Document 1 discloses a raindrop sensor that is attached to the inner wall of a front windshield of a vehicle and optically detects the presence or absence of raindrops. This raindrop sensor is configured by arranging a light emitting element and a light receiving element on both sides in the longitudinal direction of a prism. The light from the light emitting element is converted into parallel light and incident on the prism body. The light incident on the prism main body is reflected between the outer wall of the front windshield and the upper central wall of the prism main body, and then condenses the light from the lens portion and enters the light receiving element.
In this method, the presence or absence of raindrops is determined by measuring the change in the amount of light generated in the light receiving element in accordance with the presence or absence of raindrops on the outer wall of the front windshield.

Patent Document 2 discloses a raindrop detection device that determines the presence or absence of raindrops using an imaging device that can capture a P-polarized image and an S-polarized image. A parallel light beam having an incident angle of Brewster's angle is irradiated from the light source to the front screen of the vehicle, and the reflected light of the light beam irradiated to the front screen is received by the imaging device to pick up an S-polarized image and a P-polarized image, thereby detecting raindrop The processing unit determines whether or not raindrops are attached to the front screen from the difference in reflectance between the captured S-polarized image and P-polarized image, and the amount of raindrops when raindrops are attached to the front screen has a simple configuration. To detect.
Patent Document 3 discloses a lens that can take a first focal length for a short distance for imaging raindrops attached to a vehicle and a second focal length for a long distance for imaging the periphery of the vehicle. A vehicle-mounted monitoring device provided is disclosed.
Patent Document 4 discloses an image processing system that includes an image processing device and a light source, and images raindrops outside the glass by illuminating the light source.

As described above, it is known that raindrops attached to the windshield surface can be detected by using a raindrop sensor composed of a light emitting element and a light receiving element as disclosed in Patent Document 1.
However, it is necessary to closely contact the windshield surface, and the detection area is limited. When trying to enlarge the detection area, it is necessary to enlarge the optical system of the raindrop sensor, so there is a trade-off between sensor size and detection accuracy.
Further, in recent vehicles, an imaging device is installed on the rear side of the rear-view mirror on the windshield to capture the vehicle running conditions and obstacle information to warn the driver and to control the vehicle. The buffering with the attachment position of such a raindrop sensor is an issue.

On the other hand, a method using an imaging device including an imaging lens and an imaging element as disclosed in Patent Document 2 is disclosed. In Patent Document 2, the detection area can be made larger by adjusting the lens position and the lens angle of view as compared with the dedicated sensor as in Patent Document 1, but in Patent Document 2, the focal point of the lens is the windshield surface. Therefore, it is described that it is necessary to install an imaging device for vehicle periphery monitoring in order to capture the vehicle periphery situation.
If a lens having a focal point for both near and near as in Patent Document 3 is used, it is possible to focus on raindrops on the windshield surface and vehicle periphery information.
However, when raindrops are detected by focusing on the windshield surface, the background contrast is reflected in the raindrops, and the position of the bright spot changes according to the position of the light source, making it difficult to extract the raindrop area. It is. The inventors have confirmed through experiments that there is such a problem.

In addition, as in Patent Document 3, it is possible to form two focal points in the distance by dividing the lens area, but there is a possibility that it is a single lens, but generally there are about 4 to 6 imaging lenses mounted on a vehicle. In addition, it is extremely difficult to divide the region in such a plurality of lenses.
According to Patent Document 4, it is described that the focus position is shifted and images around the vehicle are taken together. In addition, the content which estimates some of raindrops is not described.
The present invention has been made in view of the above, and an object of the present invention is to provide an imaging method capable of detecting an attachment state of raindrops from a captured image in a light source irradiation area on a windshield surface irradiated with light from a light source. And providing an imaging unit.

In order to solve the above-mentioned problem, the invention according to claim 1 irradiates light from a light source toward the windshield, and includes light including reflected light reflected by raindrops attached to the surface of the windshield. In the imaging method in which light is collected by an imaging lens arranged on the same side as the light source with respect to the windshield and imaged by an imaging device, the focal point of the imaging lens is set farther than the position of the windshield, Based on pixels included in a spot corresponding to the reflected light included in the plurality of region images, by dividing the captured image in the light source irradiation region on the windshield surface irradiated with light from the light source into a plurality of regions, A detection step of detecting a raindrop adhesion state on the windshield surface is performed, and the detection step includes brightness of pixels in the spot in each region image. Based, and wherein the imaging method for detecting the raindrop adhesion state based on the variance of the average luminance of each region image.

  ADVANTAGE OF THE INVENTION According to this invention, the adhesion state of a raindrop is detectable from the captured image in the light source irradiation area | region on the windshield surface which irradiated the light from a light source.

Image processing that controls the driving state of a vehicle, controls headlights, controls wipers, and performs various types of driving support using information from an imaging device mounted on a vehicle such as an automobile The figure which showed the system typically. The figure explaining the structure of the imaging unit in FIG. 1 is a diagram illustrating a configuration of an imaging device. The figure explaining the division | segmentation of the filter area | region of an optical filter. The figure which shows the spectral filter provided in the optical filter in order to divide | segment an optical filter according to a characteristic. The enlarged view of a sensor board | substrate part. The figure which shows the image image | photographed with the imaging unit shown in FIG. 2 with a photograph. The figure explaining the bright spot detection of a raindrop. The figure explaining the light source installation angle in a raindrop detection function. The figure which shows the example at the time of arrange | positioning three light sources. The figure which shows the light source arrangement | positioning which considered the angle of view of the imaging device. The figure which shows light source arrangement | positioning containing a folding mirror. The figure explaining the pattern illumination optical system for illuminating a spatial intermittent pattern. The figure explaining the process of the disturbance light component in raindrop detection. The figure which shows the characteristic of an infrared-light cut filter. The figure explaining the method of calculating a raindrop characteristic from a captured image. The flowchart explaining calculation of raindrop characteristics. The flowchart for demonstrating calculation of the raindrop characteristic by the imaging unit which concerns on 2nd Embodiment. The figure which shows the illumination area | region grasped | ascertained beforehand. The figure for demonstrating the subdivision of a detection area. 12 is a flowchart for explaining calculation of raindrop characteristics by the imaging unit according to the third embodiment. The figure which shows the raindrop spot distribution in an illumination area | region. The figure which shows the relationship between the number of raindrops and a dispersion | distribution value with a graph, and the figure which shows the relationship between a dispersion | distribution value and the whole area brightness | luminance average value with a graph. The figure which shows the relationship between the number of raindrops, and the total illumination area brightness | luminance average value with a graph. The figure which shows the characteristic of a spectral filter layer with a graph. The figure which shows the filter transmittance | permeability and the relationship of a wavelength. The figure which shows arrangement | positioning of the pixel of an optical filter and a pixel sensor.

Embodiments of the present invention will be described below with reference to the drawings.
<First Embodiment>
An image processing system according to a first embodiment of the present invention will be described.
FIG. 1 shows the use of information from an imaging device mounted on a vehicle 100 such as an automobile to control the running state of the vehicle, to control headlights, to control wipers, It is the figure which showed typically the image processing system which performs support.
The image processing system according to the present embodiment includes an imaging unit 101 and an image analysis unit 102.

The imaging unit 101 is installed at a rearview mirror position of the seat so that an image in front of the vehicle 100 traveling can be captured. An image in front of the vehicle captured by the imaging unit 101 is converted into an image signal and input to the image analysis unit 102. The image analysis unit 102 analyzes the image signal output from the imaging unit 101.
As a specific example, the image analysis unit 102 calculates a position, a distance, and an angle to another vehicle existing in front of the vehicle 100. The headlight control unit 103 receives an output signal from the image analysis unit 102 and generates a control signal for controlling the headlight 104 from the distance value calculated by the image analysis unit 102. The control signal generated by the headlight control unit 103 is sent to the headlight 104. The headlight control unit 103 switches the high beam and the low beam of the headlight 104 or partially blocks the light to ensure the driver's field of view while preventing dazzling of the preceding vehicle and the oncoming vehicle.

Further, the image analysis unit 102 detects foreign matter attached to the windshield 105 of the vehicle.
The wiper control unit 106 receives an output signal from the image analysis unit 102 and generates a control signal for controlling the wiper 107 based on the foreign matter or rain amount calculated by the image analysis unit 102.
The control signal generated by the wiper control unit 106 is sent to the wiper 107. The wiper control unit 106 operates the wiper 107 in order to ensure the driver's field of view.

Further, the image analysis unit 102 detects a road surface area or a white line on a road on which the vehicle travels.
The image analysis unit 102 calculates the road surface area where the vehicle 100 travels and the coordinate information of the white line, and then the vehicle travel control unit 108 receives the output signal from the image analysis unit 102 and receives the image analysis unit 102. A warning signal to the driver is generated from the coordinate value calculated by 102. The vehicle travel control unit 108 can also control the steering wheel and the brake.

FIG. 2 is a diagram illustrating the configuration of the imaging unit 101 in FIG.
The imaging unit 101 includes an imaging device 201 and a light source 202 as shown in FIG.
In addition, the imaging unit 101 is installed on the indoor side of the vehicle 100 so as to face the windshield 105.
In addition, the imaging apparatus 201 includes an imaging lens 204, an optical filter 205, and an imaging element 206, as shown in FIG.

A light source 202 shown in FIG. 2A is arranged to emit light toward the windshield 105 and irradiates the windshield 105.
As shown in FIG. 2A, the light source 202 is installed to detect foreign matter and raindrops on the surface of the windshield 105. When raindrops 203 are attached to the outside of the windshield 105, the light emitted from the light source 202 is reflected at the interface between the raindrops and the air, and the light enters the imaging device 201.
Note that the imaging unit 101 covers the imaging device 201 and the light source 202 with a case 207 as shown in FIG. If the imaging unit 101 is surrounded in this manner, even if fogging occurs in other parts of the windshield 105, the windshield 105 in the vicinity of the imaging apparatus 201 is prevented from being fogged, and the imaging apparatus 201 misrecognizes. It is possible to operate without. On the other hand, if it is desired to detect cloudiness using the imaging device 201 and control the air conditioner, a passage through which air flows may be provided in a part of the case 207 so as to be in the same environment as the windshield 105. Good.

FIG. 3 is a diagram illustrating a configuration of the imaging apparatus 201.
An imaging apparatus 201 illustrated in FIG. 3 includes an imaging lens 204, an optical filter 205, a sensor substrate 601 including an imaging element 206, and a signal processing unit 602.
Light from the test object passes through the imaging lens 204, passes through the optical filter 205, and is converted into an electrical signal by the imaging device 206.
The signal processing unit 602 receives an electrical signal output from the image sensor 206 and generates an image signal described later. Then, the imaging apparatus 201 outputs, as image data, a digital signal indicating the brightness (luminance) for each pixel of the captured image to the subsequent output device together with the horizontal / vertical synchronization signal of the image.
The focal position of the imaging lens 204 is set at infinity or between infinity and the windshield 105. As a result, it is possible to detect the preceding vehicle and the oncoming vehicle, and to detect information such as the road surface area and the white line.

By the way, in addition to the lens configuration for focusing at infinity or far away from the windshield 105, the optical filter 205 is divided into regions having different optical characteristics by the spectral filter provided on the imaging element side, and a spectral filter is provided. By assigning a raindrop detection function to an area and assigning another detection function (vehicle surrounding information detection function) used for other functions such as light distribution control to an area where a spectral filter is not provided, the imaging unit 101 can be used as a plurality of functions. I can do it.
In the present embodiment described below, an image area (raindrop detection area) for detecting foreign matter and raindrops on the surface of the windshield 105, and an image area (vehicle surrounding information detection area) for detecting a preceding vehicle, an oncoming vehicle, a road surface, and a white line. Both. What is necessary is just to use what the said spectral filter was partially formed for coexistence.

4A and 4B are diagrams for explaining division of the optical filter. FIG. 4A is a diagram illustrating an example of an optical filter having different optical characteristics depending on regions, and FIG. 4B is a photograph taken using such an imaging device. It is a figure which shows the example of the image of a vehicle and raindrops.
For example, as shown in FIG. 4A, the optical filter 205 is divided, and the area in the center half of the screen is set as a vehicle peripheral information detection area 52 for photographing an image for light distribution control or vehicle control. The lower and upper quarter areas of the screen are set as raindrop detection areas 51 for performing image processing by transmitting only infrared light for reasons described later so as to detect raindrops.
FIG. 4B shows an example of a photographed image of a vehicle and raindrops. A portion 31 having a bonnet in the raindrop detection area 51 at the bottom of the screen is indicated by a broken line, and raindrops 203 are shown. Further, the road surface 32 and the preceding vehicle 33 are shown in the area 52 in the center of the screen.

FIG. 5 is a diagram for explaining a spectral filter provided in the optical filter in order to divide the optical filter according to characteristics.
In the present embodiment, as shown in FIG. 5, spectral filter layers 205 </ b> A and 205 </ b> B are formed on both surfaces (the imaging lens side 205 a and the imaging element side 205 b) of the optical filter 205.
The spectral filter 205B is provided at a location corresponding to the raindrop detection region 51 described above. A portion where the spectral filter 205 </ b> A is not provided corresponds to the vehicle surrounding information detection region 52.

The characteristics of each filter layer will be described in detail below.
The reason why the raindrop detection area 51 is provided at the top and bottom of the screen and the vehicle peripheral information detection area 52 is provided at the center is that the subject is often reflected mainly in the center of the screen for detection functions other than raindrop detection. can give.
For example, in the light distribution control, the headlight of the oncoming vehicle and the tail lamp of the preceding vehicle are usually displayed in the center of the screen, and the road surface near the own vehicle is displayed in the lower part of the screen. The information at the bottom is important, and the information at the bottom of the screen is not so important. Similarly, since it is normal that the sky appears in the upper part of the screen, information at the center of the screen is important in light distribution control and vehicle control, and information at the upper part of the screen is not so important.
Therefore, when both the raindrop detection function and the other detection functions are compatible, the configuration shown in FIG. 4A is preferable in which the upper and lower portions of the screen are infrared light transmission filter regions (raindrop detection regions 51) for raindrop detection filters. It is.

In addition, when the camera is tilted downward, the hood of the vehicle appears, and the sunlight reflected by the hood of the vehicle and the tail lamp of the preceding vehicle become disturbances, which is disadvantageous for lane detection and light distribution control. It becomes a necessary condition. However, when a spectral filter as shown in FIG. 4A is used, disturbance light is removed and the area is assigned to an area for performing raindrop detection image processing (raindrop detection area 51) due to the bandpass characteristics. Can do.
Of course, when only raindrop detection is performed, a raindrop detection filter (spectral filter 205B) may be formed on the entire surface of the image sensor side 205b of the optical filter 205.
Note that due to the characteristics of the imaging lens 204, the top and bottom are reversed between the actual scene to be imaged and the image condensed on the imaging device 30. Therefore, in order to make the lower part of the screen an infrared light transmission region for detecting raindrops, a spectral filter 205B is provided on the top side of the optical filter 205 to form an infrared transmission filter region (raindrop detection region) 51.

Next, the bonding between the optical filter and the image sensor will be described.
FIG. 6 is an enlarged view of the sensor substrate section.
The imaging element 206 is a light receiving element such as a CCD or CMOS, and a photodiode 206b is arranged in an array for each pixel so as to form a two-dimensional image.
A microlens 206a is formed on the surface of the photodiode 206b to increase the light collection efficiency of the photodiode 206b. Such an image sensor 206 is bonded to the PWB substrate 207 by a method such as wire bonding, and information is transmitted to the signal processing unit 602. An optical filter 205 is disposed in front of the sensor substrate 207. An optical filter 205 is disposed close to the surface of the image sensor on the microlens side.
The optical filter 205 and the image sensor 206 may be joined with a UV adhesive, or UV bonding or thermocompression may be performed in the four-side region outside the effective pixel while the outside of the effective pixel range used for photographing is supported by a spacer or the like.
By tightly joining the optical filter 205 and the image sensor 206, the boundary between the raindrop detection area 51 (FIG. 4) and the area 52 (vehicle periphery information detection area) for detecting vehicle periphery information is clarified. The discrimination accuracy is increased.

Further, a pattern in which raindrop detection areas 51 are provided at the upper and lower parts of the screen as shown in FIG. 4 is desirable. When the raindrop detection region 51 is provided only on either the upper part or the lower part of the screen, it is difficult to bond the optical filter 205 and the image sensor 206 in parallel, and the optical filter 205 is inclined by the thickness of the optical filter 205. If the adhesive is tilted, the optical path length is changed between the upper part of the screen and the lower part of the screen, which causes deterioration of recognition accuracy such as reading error of white line coordinates when detecting vehicle peripheral information, for example, white line coordinates.
Note that the filter and the image sensor 206 are bonded in parallel even if a raindrop detection pixel and a vehicle peripheral information detection pixel, which will be described later, are formed in a checkered pattern or a stripe pattern on the entire screen. Is possible.
Since the raindrop detection area 51 can be made larger than the conventional example by forming the upper and lower portions of the screen, or the checkered pattern or stripe pattern over the entire image, it is possible to improve the raindrop detection accuracy.

FIG. 7 is a diagram showing a photograph of an image taken by the imaging unit 101 shown in FIG.
In addition, the photograph shown in FIG. 7 is an image in the raindrop detection area 51 of FIG.
As an example of the imaging state of raindrops, as shown in FIG. 7A, the bright spot of each raindrop is visible. By adjusting the focus to infinity, the image becomes blurred.
FIG. 7B shows an imaging state when the windshield surface is in focus, and the background image is reflected on the water drop surface and the bright spot is an arcuate image.
The reflected background image changes depending on the traveling scene, and the shape of the bright spot also changes according to changes in various disturbance lights such as sunlight and street lights. In order to cope with such various changes, the recognition processing becomes enormous and the recognition rate is lowered.
On the other hand, when some blurring occurs as shown in FIG. 7A, a change due to background disturbance light can be greatly reduced. By generating some blurring, the recognition rate as a reference shape used in image recognition, for example, a circular shape is increased, and raindrop detection performance is improved.

The above-described detection of the bright spot will be described.
FIG. 8 is a diagram for explaining the detection of bright spots of raindrops.
As shown in FIG. 8, the light beam A-1 emitted from the light source 202 is refracted on the surface of the windshield 105 (the surface on the light source 202 side) and enters the windshield 105, and from the other windshield 105 surface. , And enters the raindrop 203 attached.
In this case, reflected light is generated at a wide range of angles at the interface between the raindrop 203 and the air. However, the incident light C-2 corresponding to the angle of view of the imaging device 201 and the light C having the same refraction angle on the front glass 105 surface. Only the -1 component can be imaged.
This light ray is reflected light from an extremely small area in the entire interface between the raindrop 203 and the air, and as a result, it is imaged as a bright “point”.

FIG. 9 is a diagram illustrating the light source installation angle in the raindrop detection function.
FIG. 9A is a diagram for explaining the arrangement of the light source 202, the windshield 105, and the imaging device 201.
The imaging device 201 is arranged horizontally, and the elevation angle with respect to the normal 105a of the windshield 105 is θa.
On the other hand, the light source 202 is disposed at an elevation angle θ with respect to the normal line 105 a of the windshield 105, and light emitted from the light source 202 enters the windshield 105.
The elevation angle θ is set in the following range.
θa-50deg <θ <θa + 20deg

FIG. 9B is a diagram for explaining the relationship between the angle of the light source and the amount of imaged light.
As shown in FIG. 9B, the imaging light quantity has a maximum value when the elevation angle of the light source 202 is near θ or slightly lower than θ, and the imaging light quantity can be obtained in the above range. Become.
The above range fluctuates due to the influence of the contact angle of raindrops, but is an effective numerical range particularly when the contact angle is 60 degrees or more.
By installing in this range, the reflected light from the raindrops emitted from the light source 202 and attached to the surface of the windshield 105 can be imaged as bright spots as shown in FIG.

Next, the light source installation angle in the horizontal direction will be described.
FIG. 10 is a diagram showing an example in which three light sources are arranged, (a) is a side view showing an example in which three light sources 202-R, 202-C, 202-L are arranged, and (b). Is a top view showing the light source arrangement setting in consideration of the horizontal angle of view of the imaging apparatus. FIG. 10C is a graph showing the elevation angle of the light source and the imaging light quantity of the raindrop reflected light.
FIG. 11 is a diagram illustrating a light source arrangement set in consideration of the angle of view of the imaging apparatus.
In the side view shown in FIG. 10 (a), the raindrop 203 is illuminated with light emitted from the light source 202-R and imaged by the imaging device. At this time, the light source with respect to the imaging field angle θu in the raindrop 203 region The exit angle (glass incident angle) θLS of 202-R is set in the following range.
θu−20 ≦ θLS ≦ θu + 20 deg
Thereby, the reflected light from the interface between raindrops and air can be detected and imaged with high efficiency.

Further, as shown in FIG. 10C, when the elevation angle θLS of the light source with respect to the normal 105a of the windshield 105 is equal to or close to the imaging field angle Qu, the value of the imaging light quantity of the raindrop reflected light is maximum. It becomes.
The above range fluctuates due to the influence of the contact angle of raindrops, but is an effective numerical range particularly when the contact angle is 60 degrees or more.
The plurality of light sources arranged in FIG. 10 (a) may have an angular arrangement such that the light beams intersect with each other as shown in FIG. 11 as long as the above relationship is satisfied.

FIG. 12 is a diagram showing a light source arrangement including a folding mirror.
The irradiation directions of the light sources 202-L, C, and R are not limited to those facing the glass surface shown so far, but may be arranged in the reverse direction including the folding mirror 208 as shown in FIG.
By including the mirror 208, it is possible to contribute to improvement in the degree of freedom of light source arrangement and miniaturization.

Next, the light source will be described.
As the light source 202, a light emitting diode (LED), a semiconductor laser (LD), or the like may be used.
The form of the emitted light beam is preferably substantially parallel light. Parallel light can be formed and emitted by a collimating lens or the like disposed immediately after the light source 202 on the optical path. The light source 202 includes components such as a lens capable of adjusting the form of the light flux as described above.

Further, visible light or infrared light may be used as the emission wavelength of the light source 202. In order to avoid dazzling oncoming vehicles and pedestrians especially at night, a wavelength that is longer than visible light and within a range that the light receiving sensitivity of the image sensor 206 extends (for example, a wavelength in the range of 800 to 1000 nm) may be selected. In particular, in order to reduce the influence of light from the outside such as direct sunlight, it is effective to select a wavelength near 940 nm.
Moreover, in this embodiment, it is necessary to match the wavelength range with the filter mentioned later.
As a filter arrangement, the wavelength lambda ₁ to [lambda] _2, and _{λ 3 ~λ 4 (λ 1 <} λ 2 <λ 3 <λ 4) separation filter A that transmits only light of the wavelength components in the range of the optical filter 205 some areas of the effective image pickup area in the plane of the image pickup element 206 side of the substrate which constitutes the wavelength _{λ 3 ~λ 5 (λ 3 <} λ 5) region division type spectral filter layer transmits only the wavelength components in the range of B when having a oscillation wavelength range, lambda ₃ to [lambda] _4, or lambda ₃ to [lambda] _5, or width needs to be included in narrower.
The oscillation wavelength range described here is desirably the full width at half maximum of the intensity spectrum or the full width at a threshold value less than or equal to the half value.

Next, pattern illumination and its light source arrangement will be described.
In the light source 202 shown in FIG. 8, the means for forming and emitting parallel light by a collimator lens or the like has been described. However, as the illumination to the window glass (front glass) 105, in addition to the means for illuminating with a substantially uniform intensity. There is a method of modulating the presence or absence of illumination depending on the region.
As described above with reference to FIG. 7, since the reflected light of the raindrop is captured as a blurred image, depending on the density of the raindrop, an image in which adjacent raindrop images are superimposed is captured, and the light of each raindrop is captured. It becomes difficult to separate the spots.
In order to solve this problem, the influence of superposition may be reduced by illuminating a spatial intermittent pattern and imaging individual raindrops.

FIGS. 13A and 13B are diagrams for explaining a pattern illumination optical system for illuminating a spatial pattern. FIG. 13A is a diagram illustrating the configuration, and FIG. 13B is a diagram illustrating an example of an irradiation pattern.
As a configuration for performing pattern illumination, there is the following FIG. Two light emitters (light sources) 21 a and 21 b are arranged close to each other, and are collimated by a collimator lens 22 and incident on a cylindrical lens array 23.
The light emitter, the collimating lens, and the cylindrical lens are arranged so that the light emitted from the cylindrical lens array 23 is substantially condensed on the window glass 105.
At this time, the light emitted from each light emitter and emitted as parallel light by the collimator lens 22 has a slightly different emission direction due to the shift of the position of the light source, and they are collected on the window through the cylindrical lens array 23. It is designed so that the pattern positions are shifted from each other when illuminated.

For example, as shown in FIG. 13B, in the pattern on the horizontal line, the bright and dark positions are reversed. When superimposed, it becomes a spatially uniform pattern as shown on the right. Here, FIG. 13B shows the illumination pattern 25a of the light emitter 21a, the illumination pattern 25b of the light emitter 21b, and the superimposed illumination pattern 26. FIG.
The light emitters 21a and 21b are alternately lit at different light emission timings, and can capture individual raindrop detection images by capturing an image synchronized with the light.
Here, an example using a spatial modulation pattern in only one direction using the cylindrical lens 23 is shown, but a two-dimensional pattern, for example, an illumination pattern having a staggered pattern may be used. In this case, the above arrangement can be realized by replacing the cylindrical lens array 23 with a microlens array and designing the arrangement appropriately.
In addition, the configuration using two light emitters 21a and 21b is shown, but three or more configurations may be used as the light emitter.

With the above configuration, it is possible to obtain light spots of individual raindrops that are more independent regardless of the density of the raindrops.
The light source irradiation method of the present embodiment is not limited to the above spatial modulation pattern, and an optical element that changes the beam profile may be provided. Here, the beam profile is a cross-sectional shape of light from the light source, an LD or LED is an elliptical shape, and its intensity distribution is an elliptical shape with high intensity on the optical axis, such as a Gaussian beam shape. And has a circular intensity distribution.

  Such an elliptical shape may be converted into a rectangular pattern, or the intensity distribution may be converted into a top hat-like intensity distribution. For example, if the beam shape is converted to a rectangular pattern, it becomes easier to divide the image of the light source irradiation area, and if it is a top hat intensity distribution shape, the light irradiation intensity of each divided area is made uniform it can. By making the intensity distribution uniform, it contributes to higher accuracy of raindrop size and quantity estimation calculation described later. These changes in shape and intensity distribution can be realized by using a well-known diffractive optical element.

Next, disturbance light components in raindrop detection will be described.
FIG. 14 is a diagram for explaining the processing of disturbance light components in raindrop detection. FIG. 14A is a diagram showing the characteristics of a cut filter as an example of a spectral filter 205B used for raindrop detection, and FIG. It is a figure which shows the characteristic of the band pass filter as an example of the spectral filter 205B used for raindrop detection.
If an infrared wavelength light reflected from a water droplet is detected as it is, the light source 202 that irradiates the infrared wavelength light must brighten the light emitted from disturbance light having a large amount of light such as outdoor direct sunlight. There is a problem that must be.

Next, the required filter characteristics (infrared light transmission filter) in the raindrop detection function will be described.
For example, as shown in FIG. 14A, a filter that cuts light having a wavelength shorter than the light emission wavelength of the light source 202 or a transmittance peak as shown in FIG. A band pass filter substantially matched with the emission wavelength may be formed on the image sensor side 205b of the optical filter 205.
Thereby, light other than the light emission wavelength of the required light source 202 can be removed, and the light amount of the detected light source 202 can be relatively increased.

Next, an infrared light cut filter will be described as required filter characteristics in other detection functions.
FIG. 12 is a diagram showing the characteristics of the infrared light cut filter.
As a filter related to a detection function other than raindrop detection including the light distribution control described above as an example, it is desirable that infrared light can be cut in addition to transmitting visible light. A typical wavelength characteristic is shown in FIG. In this figure, as an example of the visible light region, it has a short-pass filter characteristic that transmits 400 to 670 nm and cuts an infrared light region longer than 670 nm.
The imaging range in which the headlight and tail lamp of a distant vehicle and the headlight and tail lamp of a nearby vehicle are reflected is an image processing range necessary for realizing the orientation control function.
Here, when recognizing the preceding vehicle, it is necessary to determine the presence of the preceding vehicle with the taillight of the preceding vehicle, but the amount of light is small compared to the headlight of the oncoming vehicle, and there are many disturbing lights such as street lights For this reason, it is difficult to detect the tail lamp only from the luminance, and it is necessary to recognize the red color of the tail lamp.
In the present embodiment, a spectral filter 205A that cuts infrared light other than the raindrop detection wavelength may be formed on the 205a side. In general, an image sensor has sensitivity in the infrared region, so if an image is captured only by the image sensor, the resulting image data will be entirely red, making it difficult to extract the red part of the tail lamp. It may become. Therefore, if a filter having the characteristic of cutting infrared light is formed, the light of other colors that becomes a disturbance can be removed, so that the detection accuracy of the tail lamp can be improved.

Next, countermeasures against unnecessary reflected light from the front glass surface of the light source will be described.
A part of the light source 202 is transmitted through the surface of the windshield 105, but the rest is reflected and becomes disturbance light.
For such disturbance light, a polarization component parallel to a plane formed by two optical axes of an optical axis of light emitted toward the windshield 105 of the light source and an optical axis of the imaging lens. A polarizing filter (not shown) that transmits only the light is disposed.
FIG. 16 is a diagram for explaining a method of calculating raindrop characteristics from a captured image, (a) to (c) are diagrams illustrating an example of a raindrop image, and (d) is a relationship between raindrop size and rainfall spot luminance. It is a figure which shows an example.
As shown in FIG. 16A, the raindrop image picked up by the above means is an image that generally includes a plurality of circular patterns. This circular pattern corresponds to the bright spot of each raindrop.
In particular, the circular size shown in FIG. 16C (D in the figure is the diameter of the pattern) depends on the f value of the imaging lens and the setting of the focal position set at any position from the window position to the infinity position. It is determined. The diameter value and the number of pixels included in the single circular pattern are obtained in advance through experiments or the like.
Further, the luminance value of the circular pattern is related to the raindrop size, and the luminance value tends to increase as the size increases, as shown in the graph of FIG. 16D. The relationship between the raindrop size and the luminance value is also grasped in advance by experiments or the like.

FIG. 17 is a flowchart illustrating calculation of raindrop characteristics.
In addition, the flowchart shown in FIG. 17 is characterized by calculating raindrop characteristics corresponding to uniform illumination. Note that the signal processing unit 602 includes a ROM, a RAM, and a CPU, and the CPU reads and executes a program stored in the ROM.
In FIG. 17, in step S <b> 5, the signal processing unit 602 causes the image pickup unit 101 to pick up an image including raindrops, and image data output from the image pickup unit 101 is provided in the signal processing unit 602 (not shown). To remember.
Next, in step S10, the signal processing unit 602 performs pixel extraction of raindrop spots on the image stored in the memory.
This is because threshold processing is performed on a pixel G inside a lighting area (for example, a frame 250 shown in FIG. 19) that is grasped in advance, and a plurality of threshold values G _th or more that are specific luminance values. Are extracted and stored in the memory.
Next, in step S15, the signal processing unit 602 reads out the extracted pixels from the memory, calculates the number of pixels by counting a plurality of pixels G that are equal to or more than the extracted specific threshold value _Gth , The number of pixels N as a result value is stored in the memory.
Next, in step S20, the signal processing unit 602 divides the number of pixels N obtained by calculating the number of pixels by the number of pixels N _cp included in one single circular pattern serving as a reference that has been grasped in advance. Thus, the raindrop quantity estimation value M indicating the quantity of raindrops is calculated.

On the other hand, in step S30, the signal processing unit 602 reads out a plurality of pixels G or above the threshold G _th from memory, subtract the number of pixels N and adding all the luminance values of the threshold G _th or more pixels G Thus, the average value G _av is calculated, and the frequently used value G _fmax is obtained as the luminance value, thereby calculating the spot luminance representative value. Here, it is assumed that G _fmax = G _av and the average value G _av is treated as a spot luminance representative value.
Next, in step S35, the signal processing unit 602 uses the graph showing the relationship between the raindrop size and the brightness value that has been grasped in advance in FIG. _{16D with} respect to the calculated spot brightness representative value G _fmax . Find the raindrop size estimate. In addition, it is preferable to memorize | store the graph which shows the relationship between raindrop size and a luminance value as a conversion table.
This makes it possible to estimate the number of raindrops and the size of the raindrops regardless of the density of the raindrop distribution, and to reflect it in precise operation control of the subsequent wiper operation.

Second Embodiment
An image processing system according to a second embodiment of the present invention will be described.
In the case of the flowchart shown in FIG. 17, when the “spot luminance representative value calculation” is performed, when the target image is an image in which spots of a plurality of raindrops are superimposed on each other, the number is high or the luminance of each spot is high. In some cases, it is difficult to calculate the representative value because it is difficult to determine whether or not.
In order to solve this problem, in the case of the flowchart shown in FIG. 18, by using the individual images obtained by the pattern illumination described in the above-mentioned “pattern illumination and its light source arrangement”, the influence of overlapping of individual raindrop spots This makes it easy to individually estimate the number of raindrops and the brightness (size) of each spot.
Next, calculation of raindrop characteristics will be described with reference to the flowchart shown in FIG. First, FIG. 18 will be described. The flow chart shown in FIG. 18 is characterized in that raindrop characteristics are calculated corresponding to illumination having a pattern. Note that the signal processing unit 602 includes a ROM, a RAM, and a CPU, and the CPU reads and executes a program stored in the ROM.

First, in step S105, the signal processing unit 602 causes the imaging unit 101 to capture a plurality of images by each illumination, and image data output from the imaging unit 101 is provided in a memory (not shown) provided in the signal processing unit 602. To remember.
Next, in step S110, the signal processing unit 602 performs raindrop spot pixel extraction on each image of each illumination stored in the memory. This is because threshold processing is performed on a pixel G inside a lighting area (for example, a frame 250 shown in FIG. 19) that is grasped in advance, and a plurality of threshold values G _th or more that are specific luminance values. Are extracted and stored in the memory.
Next, in step S115, the signal processing unit 602 superimposes a plurality of images by each illumination by adding the luminance of each pixel, and stores the superimposed result in the memory.
Next, in step S120, the signal processing unit 602 reads the superimposed pixels from the memory, calculates the number of pixels by counting a plurality of pixels G that are equal to or more than the extracted specific threshold value _Gth , The number of pixels N as a result value is stored in the memory.
Next, in step S125, the signal processing unit 602 divides the number of pixels N obtained by calculating the number of pixels by the number of pixels N _cp included in one single circular pattern serving as a reference that has been grasped in advance. Thus, the raindrop quantity estimation value M indicating the quantity of raindrops is calculated.

On the other hand, in step S140, the signal processing unit 602 reads, from the memory, a plurality of pixels G that are equal to or greater than the threshold value _Gth for each image of each illumination, and sets the luminance values of the pixels G that are equal to or greater than the threshold value _Gth. The average value G _av is calculated by adding all and subtracting by the number of pixels N, and the spot luminance representative value is calculated by _{obtaining the} frequently used value G _fmax as the luminance value. Here, it is assumed that G _fmax = G _av and the average value G _av is treated as a spot luminance representative value.
Next, in step S145, the signal processing unit 602 uses a graph indicating the relationship between the raindrop size and the brightness value that has been previously grasped in FIG. _{16D with} respect to the calculated spot brightness representative value G _fmax . Find the raindrop size estimate. In addition, it is preferable to memorize | store the graph which shows the relationship between raindrop size and a luminance value as a conversion table.
This makes it possible to estimate the number of raindrops and the size of the raindrops regardless of the density of the raindrop distribution, and to reflect it in precise operation control of the subsequent wiper operation.

<Third Embodiment>
An image processing system according to the third embodiment of the present invention will be described.
In the present embodiment, a method for calculating raindrop characteristics from a captured image will be described. In addition, the raindrop characteristic calculation method different from FIG. 17, FIG. 18 is demonstrated.
Here, the illumination area indicated by the frame 250 shown in FIG. 19 is subdivided into a plurality of areas, and differences in luminance characteristics of the respective areas are detected.

FIG. 20 is a diagram for explaining subdivision of the detection area.
In FIG. 20, 9 horizontal and 6 vertical rectangles indicated by a frame 251 are subdivided areas. These area settings are made in advance within a known illumination area range (frame 250).
FIG. 21 is a flowchart illustrating a method for calculating raindrop characteristics using luminance information of a subdivided area.
First, in step S205, the signal processing unit 602 causes the imaging unit 101 to capture an image including raindrops, and stores image data output from the imaging unit 101 in a memory (not shown) provided in the signal processing unit 602. To do.
Next, in step S210, the signal processing unit 602 reads the image stored in the memory, subdivides the image into M subdivided regions, and averages the luminance values G _{av1 to} G a for each subdivided region. _avM is calculated and stored in the memory. These average values reflect the presence and degree of raindrop overlap in each region.
Next, in step S215, the signal processing unit 602 calculates a variance value for the average values G _{av1 to} G _avM of each subdivided region and stores them in the memory. The variance value is an index representing the density of the raindrop distribution as shown in FIG.

FIGS. 22A and 22B are diagrams showing the raindrop spot distribution in the illumination area. FIG. 22A shows the number of raindrops: small, FIG. 22B shows the number of raindrops: medium, and FIG. 22C shows the number of raindrops: many.
On the other hand, in step S215, the signal processing unit 602 calculates an average value G _av for all the illumination areas before performing the segmentation process, and stores it in the memory.
FIG. 23 (a) is a graph showing the relationship between the number of raindrops and the variance value, and FIG. 23 (b) is a graph showing the relationship between the variance value and the average luminance of the entire area.
Due to the change in the number of raindrops, the variance value changes as shown in FIG. Although the maximum value varies depending on the raindrop size, the maximum value is taken when the number of raindrops is medium.

Further, the relationship between this variance value and the average brightness value G _av of the entire illumination area is as shown in FIG.
From this figure, the raindrop size can be estimated by measuring and calculating the luminance average value and the variance value. In the present embodiment, the relationship of FIG. 23B is also calculated in advance as a data table using the size of the raindrop spot and the relationship between the raindrop size and spot luminance shown in the previous section, and is shown in FIG. Used in the steps of the flowchart.
Therefore, in step S225, the luminance average value _Gav and the variance value of all the illumination areas are read from the memory, and information on the estimated raindrop size is obtained from the relationship between the luminance average value, the variance value, and the raindrop size shown in FIG. Is read.

The relationship shown in FIG. 24 is used for estimating the number of raindrops.
FIG. 24 is a graph showing the relationship between the number of raindrops and the average luminance value of the entire illumination area.
As the number of raindrops increases, the average brightness value of all illumination areas tends to increase. As for the raindrop size, as shown in the previous section, the larger the raindrop size, the higher the luminance. This relationship is also grasped in advance by experiments.
Therefore, in step S230, the luminance average value G _av of the entire illumination area, and estimated raindrop size information, the relationship between the brightness average value G _av of the raindrop quantity and total illumination region in the graph of FIG. 24, a raindrop quantity Can be estimated.
Thereby, the picked-up raindrop image becomes an image including a plurality of substantially circular patterns as shown in FIG. This circular pattern corresponds to the bright spot of each raindrop. The size of the circle (D in the figure is the diameter of the pattern) is determined by the f value of the imaging lens and the setting of the focal position set from the window position to the infinity position. The diameter value and the number of pixels included in the single circular pattern are obtained in advance through experiments or the like.

In the flowchart shown in FIG. 21, “average value of each subdivision area” is calculated in step S210 and then “calculation of variance” is calculated in step S215. However, the present invention is limited to such an order. It is not a thing. That is, similar results can be obtained by calculating “dispersion within each subdivided region” and then performing “calculation of average value”.
This makes it possible to estimate the number of raindrops and the size of the raindrops regardless of the density of the raindrop distribution, and to reflect it in precise operation control of the subsequent wiper operation.

Next, the configuration of the spectral filter shown in FIG. 5 will be described.
An embodiment in consideration of the required filter characteristics described in the previous section of the imaging unit configuration will be described.
FIG. 25 is a graph showing the characteristics of the spectral filter layer. (A) is the spectral filter layer 205A (Example A-1), (b) is the spectral filter layer 205B (Example B-1), ( c) is a diagram showing superposition filter characteristics of (Examples A-1, B-1).

<Example A-1>
It is Example A-1 for demonstrating the combination of a filter characteristic.
A spectral filter layer 205A is formed on a surface 205a of the optical filter 205 on the imaging lens 204 side. The characteristic example A-1 of the spectral filter layer 205A transmits a so-called visible light band having a wavelength range of 400 nm to 670 nm and a light source wavelength band having a wavelength range of 920 to 960 nm as shown in FIG. Here, it is assumed that the center wavelength of the light source for raindrop detection is 940 nm and the full width at half maximum is 10 nm.
Here, the correspondence between λ _{1 to} λ ₄ is as follows.
λ ₁ : 400 nm (limit value on the short wavelength side of image sensor sensitivity)
λ ₂ : 670 nm
λ ₃ : 920 nm
λ ₄ : 960 nm
The raindrop detection function uses transmission in the light source wavelength band.
For other detection functions, transmission in the visible light band is used. Further, it is desirable that light in the wavelength range of 700 to 920 nm does not pass through or has a transmittance of 5% or less. The reason for suppressing the transmission of this wavelength is that, as described above as the required filter characteristics, when this wavelength range is taken in, the obtained image data will be entirely red, such as the portion indicating the red color of the tail lamp. It may be difficult to extract.
Therefore, if a filter having the characteristic of cutting infrared light is formed, light of other colors that becomes a disturbance can be removed, so that, for example, detection accuracy of a tail lamp can be improved. From this point of view, it is desirable not to transmit the light source wavelength band of 920 to 960 nm, but the band is narrower than the visible light band, and relative to visible light image sensors such as CMOS. Since the sensitivity is low, the amount of light is sufficiently small and the influence can be ignored.

<Example B-1>
A spectral filter layer 205B is formed in the raindrop detection function region on the surface 205b of the optical filter 205 on the image sensor 206 side.
An example B-1 of the characteristics of the spectral filter layer 205B is shown in FIG.
As shown in FIG. 25B, an infrared light region having a wavelength range of 880 to 1100 nm is used as a transmission band, and a wavelength range of 920 to 960 nm is obtained by combination with the spectral filter layer 205A on the surface 205a on the imaging lens 204 side. Only light of the above will be transmitted. It is desirable that the center value of this wavelength range and the light emission wavelength of the light source 202 be substantially the same.
Here, the correspondence between λ ₅ and λ ₆ is as follows.
λ ₅ : 880 nm
λ ₆ : 1100 nm (limit value on the long wavelength side of image sensor sensitivity)
Due to the arrangement of these filters A-1 and B-1, the filter characteristics of the raindrop detection function region pass only through the hatched region in FIG. 25C, that is, the range of 920 to 960 nm.
As described above, since the light source for raindrop detection has a center wavelength of 940 nm and a full width at half maximum of 10 nm, the light emitted from the light source for raindrop detection can pass through the hatched band of 920 to 960 nm.
Also, in the filter area of this raindrop detection function, disturbance light components such as direct sunlight enter only a small component that passes through the 920 to 960 nm band, so that the disturbance light can be greatly reduced. This will improve the accuracy of raindrop detection.

<Example B-2>
Next, Example B-2 of combinations of filter characteristics will be described.
As for the spectral filter B, Example B-2 shown in FIG. 26 can be used instead of Example B-1.
FIG. 26 is a diagram showing the relationship between the filter transmittance and the wavelength.
In FIG. 26, an infrared light region having a wavelength range of 925 to 965 nm is used as a transmission band, and only light in the above wavelength range of 925 to 960 nm is transmitted through a combination with the spectral filter layer A on the 205a side. It becomes.
Here, the correspondence between λ ₅ and λ ₆ is as follows.
λ ₅ : 925 nm
λ ₆ : 965 nm
Next, Example B-2 of the spectral filter layer B shown in FIG. 26 will be described.
Due to this characteristic, the transmission band is further narrowed compared to the combination of Examples A-1 and B-1, and disturbance light components such as direct sunlight can be further reduced. Of course, the light emitted from the light source for raindrop detection can be transmitted.
In this embodiment, the effect of the combination of the spectral filters 205A and 205B is shown. However, if a slight decrease in the effect is allowed, it is also effective in using the spectral filters 205A and 205B alone.
In designing the above characteristics, a light beam angle when entering the filter from the lens is taken into consideration, and examination is performed in accordance with the light beam.
As will be described later, a polarizing filter for suppressing unnecessary reflected light reflected by the surface of the windshield inside the vehicle may be formed between the spectral filter layer B and the substrate of the optical filter.

Next, a case where an area division pattern is used as the configuration of the filter unit will be described.
FIG. 27 is a diagram showing the arrangement of the pixels of the optical filter and the pixel sensor.
An area-divided pattern is formed on the surface of the optical filter 205 on the image sensor 206 side. For example, as shown in FIG. 27, a checkered pattern corresponding to each photodiode, that is, each pixel G is formed. It arrange | positions so that it may correspond to the pixel of an image pick-up element.
Although there may be a configuration in which there is a gap between the optical filter 205 and the image sensor 206, the configuration in which the optical filter 205 is closely attached to the image sensor 206 and the boundary between the regions on the optical filter 205 and the image sensor 206. This makes it easier to match the boundaries of the regions.
Here, description will be made on the assumption that the image pickup device is a monochrome image pickup device. A color image sensor may be used. When a color image sensor is used, the transmission characteristics of each pattern of the optical filter may be adjusted according to the characteristics of the color filter attached to each pixel of the image sensor.
Various images described later are formed from these pieces of information different in pixel units. Such image formation is performed by a signal processing unit 602 inside the camera.

<Example C>
Next, Example C of the filter area division pattern will be described.
The filter area division pattern is not limited to the area division corresponding to the pixel size, and is formed in the upper 1/4 and lower 1/4 areas of the entire screen as shown in FIGS. It may be.

DESCRIPTION OF SYMBOLS 101 Imaging unit, 102 Image analysis unit, 105 Windshield, 201 Imaging device, 202, 202 'Light source, 204 Imaging lens, 205 Optical filter, 206 Imaging element, 207 Sensor substrate, 602 Signal processing unit

Japanese Patent No. 353638 JP 2010-204059 A JP 2005-225250 A JP-A-2005-195 566

Claims (4)

Light is emitted from a light source toward the windshield, and light including reflected light reflected by raindrops attached to the surface of the windshield is collected by an imaging lens disposed on the same side of the windshield as the light source. In an imaging method in which light is used to capture an image with an image sensor,
The focal point of the imaging lens is set farther than the position of the windshield,
By dividing the captured image of the light source irradiation area on the front glass surface was irradiated with light from the light source into a plurality of regions, based on pixels included in spots corresponding to the reflected light included in the plurality of area images Performing a detection step of detecting a raindrop adhesion state on the windshield surface,
The imaging method according to claim 1, wherein the detecting step detects the raindrop adhesion state based on a variance of average luminance of each area image based on luminance of pixels in the spot in each area image. A light source that emits light toward the windshield;
An imaging device that is provided on the same side as the light source with respect to the windshield, and that captures light including reflected light in which light from the light source is reflected by raindrops attached to the surface of the windshield,
The imaging device
An imaging lens whose focal point is set farther from the position of the windshield, an imaging device having a plurality of pixel arrays, and the reflected light provided selectively between the imaging lens and the imaging device. And an optical filter
Based on pixels included in a spot corresponding to the reflected light in the plurality of region images, by dividing a captured image in a light source irradiation region on the windshield surface irradiated with light from the light source into a plurality of regions, An adhesion state detection means for detecting a raindrop adhesion state on the windshield surface;
The image pickup unit, wherein the attachment state detection unit detects the raindrop attachment state based on a variance of average luminance of each area image based on luminance of pixels in the spot in each area image.   The imaging unit according to claim 2, further comprising an optical element that converts light from the light source into a predetermined beam profile.   The imaging unit according to claim 2, wherein the light source includes at least two illumination optical systems having different light distributions on the windshield.

Images