JP6287769B2 - Image processing apparatus, vehicle, and program - Google Patents

Description

The present invention relates to an image processing device, a vehicle , and a program.

  2. Description of the Related Art Conventionally, an attached matter detection device that is mounted on a vehicle such as an automobile and detects an attached matter such as raindrops attached to a windshield using an imaging unit is known.

  As an example, the raindrop detection area is divided into a predetermined number of blocks, and the illuminance for each block is calculated from the image data, and foreign matter adheres to the raindrop detection area when the illuminance variation between the blocks is greater than or equal to a predetermined reference value. The apparatus which judges that it has carried out can be mentioned (for example, refer patent document 1).

  As another example, the raindrop detection area is divided into a predetermined number of blocks, and the illuminance change rate for each block is calculated by adding a sign from the image data, and the raindrop is based on the difference in the interval at which the sign of the illuminance change rate changes. An apparatus for identifying the deposit in the detection region can be given (for example, see Patent Document 2).

  However, in the apparatus exemplified above, noise that fluctuates due to environmental changes, detector variations, etc. is not fully considered, so the detection sensitivity is low, and small deposits (for example, one drop of rain) and noise It was difficult to distinguish.

SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus with improved detection sensitivity.

The image processing apparatus is an image processing apparatus that detects an adhering substance adhering to the transparent body based on an image obtained by an imaging unit that performs imaging through a transparent body. In each of the plurality of regions, the region-specific deposit information is calculated, the region-specific deposit information is compared with a predetermined threshold value, and the region-specific deposit information of the region exceeding the predetermined threshold value is compared. And calculating means for calculating the deposit information at a predetermined time, averaging the deposit information at the predetermined time and the deposit information before the predetermined time, and calculating the average deposit information. And an averaging processing means for calculating.

According to the disclosed technology, an image processing apparatus with improved detection sensitivity can be provided.

It is a schematic diagram which illustrates schematic structure of the mobile body apparatus control system which concerns on this Embodiment. It is a schematic diagram which illustrates schematic structure of the raindrop detection apparatus which concerns on this Embodiment. FIG. 3 is a schematic view illustrating a part of FIG. 2 in an enlarged manner. It is a figure which illustrates the optical filter which concerns on this Embodiment. It is a figure which illustrates a captured image. It is a figure which illustrates a cut filter. It is a figure which illustrates a band pass filter. It is a schematic diagram which illustrates schematic structure of the raindrop detection apparatus which concerns on the modification of this Embodiment. It is a figure which illustrates the composition of an image analysis unit. It is a figure which shows the example which mounts the raindrop amount information calculation means in the imaging unit. It is a figure which shows the example of a connection of a raindrop amount information calculation means and an imaging device. It is a figure which illustrates the signal brightness | luminance of the raindrop detected in a raindrop detection area. It is a figure which illustrates the sum total of the pixel value in each reference area | region of the raindrop detected in a raindrop detection area. It is a figure which shows the example of the raindrop detection area | region provided in a windshield. It is a figure which shows the example which divides | segments the image area | region corresponding to a raindrop detection area | region into a some reference area. It is an example of the flowchart explaining operation | movement of the raindrop amount information calculation means. FIG. 6 is a diagram (part 1) illustrating a temporal change in the sum of pixel values in each reference region of raindrops detected in a raindrop detection region; FIG. 6 is a diagram (part 1) illustrating the relationship between the sum of pixel values and time; It is a block diagram which shows an example of a moving average process. FIG. 6 is a diagram (part 2) illustrating a temporal change in the sum of pixel values in each reference region of raindrops detected in the raindrop detection region; FIG. 6 is a diagram (part 2) illustrating the relationship between the sum of pixel values and time; It is a block diagram which illustrates the specific process for responding to the rapid change of raindrop information. It is a figure which illustrates the result of the difference calculation process in a differential calculation means. It is a figure which illustrates the relationship between the sum total of a pixel value at the time of performing the process of FIG. 22, and time. FIG. 23 is an example (part 1) of a flowchart for explaining the processing of FIG. 22; FIG. FIG. 23 is an example (part 2) of a flowchart for explaining the processing of FIG. 22; FIG. It is the graph which simulated the output of the raindrop detection process with respect to the amount of raindrops adhering to the windshield. It is a block diagram which illustrates a weighted average process. It is a graph at the time of removing the noise of a direct current component by performing a weighted average difference process. It is a block diagram which illustrates weighted average difference processing. It is a graph at the time of adding a moving average process with respect to an output pixel value. It is a graph (the 1) which shows the simulation result with respect to the rapid change (splash) of raindrop information. It is a graph (the 2) which shows the simulation result with respect to the rapid change (splash) of raindrop information. It is the example (the 1) of the flowchart explaining the specific process for responding to the rapid change of raindrop information. It is the example (the 2) of the flowchart explaining the specific process for responding to the rapid change of raindrop information. It is a block diagram which illustrates the specific process for responding to the rapid change of raindrop information. It is the graph which simulated the case where the amount of raindrops adhering to a windshield fell rapidly. It is an example (the 1) of the flowchart explaining a clamp process. It is an example (the 2) of the flowchart explaining a clamp process. It is the graph which simulated the case where the amount of raindrops adhering to a windshield at the time of clamp processing introduction fell rapidly.

  Hereinafter, embodiments for carrying out the invention will be described with reference to the drawings. In the drawings, the same components are denoted by the same reference numerals, and redundant description may be omitted.

  In the following embodiments, the mobile object device is mounted on a mobile object such as an automobile, detects mobile objects such as raindrops attached to the windshield of the mobile object, and controls the mobile object device. Although an example of a control system is shown, it is not limited to this. For example, it is possible to detect an adhering matter such as raindrops adhering to glass of a building using an adhering matter detecting device. The mobile device refers to a predetermined device mounted on the mobile body. For example, if the mobile body is an automobile, a wiper, a headlamp, or the like is the mobile device.

[Mobile equipment control system]
First, the mobile device control system according to the present embodiment will be described. FIG. 1 is a schematic diagram illustrating a schematic configuration of a mobile device control system according to the present embodiment. A mobile device control system 300 shown in FIG. 1 is an example of a vehicle (automobile) as an example of a mobile body. Referring to FIG. 1, the mobile device control system 300 includes a deposit detection device 10, a headlamp control unit 310, a wiper control unit 320, a vehicle travel control unit 330, a headlamp 350, and a wiper 360. Have In FIG. 1, reference numeral 400 denotes a host vehicle such as an automobile.

  The mobile device control system 300 uses the captured image data of the front area (imaging area) in the traveling direction of the host vehicle captured by the imaging device 40 (see FIGS. 2 and 3) of the attached matter detection apparatus 10 mounted on the host vehicle 400. Utilizing it, it has a function of performing predetermined control. Examples of the predetermined control include light distribution control of the headlamp, drive control of the wiper, control of other devices mounted on the host vehicle 400, and the like.

  The captured image data captured by the imaging device 40 is input to the image analysis unit 50. The image analysis unit 50 analyzes the captured image data transmitted from the imaging device 40, thereby calculating, for example, the position, direction, distance, and the like of another vehicle existing in front of the host vehicle 400.

  Further, the image analysis unit 50 detects the preceding vehicle traveling in the same traveling direction as the own vehicle 400 by identifying the tail lamp of the other vehicle, or is opposite to the own vehicle 400 by identifying the headlamp of the other vehicle. Or detecting an oncoming vehicle traveling in the direction. Further, the image analysis unit 50 detects an adhering matter such as raindrops or foreign matters adhering to the windshield 101, or a detection target such as a white line (partition line) on the road surface existing in the imaging region. .

  The analysis result of the image analysis unit 50 is sent to the headlamp control unit 310. For example, the headlamp control unit 310 generates a control signal for controlling the headlamp 350 that is a device mounted on the host vehicle 400 from the distance data calculated by the image analysis unit 50.

  Specifically, for example, while avoiding the strong light of the headlamp of the own vehicle 400 entering the eyes of the driver of the preceding vehicle or the oncoming vehicle, the driver of the other vehicle is prevented from being dazzled. Ensure driver visibility. For this purpose, the switching of the high beam and the low beam of the headlamp 350 is controlled, or partial light shielding control of the headlamp 350 is performed.

  The analysis result of the image analysis unit 50 is also sent to the wiper control unit 320. The wiper control unit 320 controls the wiper 360 to remove deposits such as raindrops and foreign matters attached to the windshield 101 of the host vehicle 400. The wiper control unit 320 receives the attached matter detection result detected by the image analysis unit 50 and generates a control signal for controlling the wiper 360. When the control signal generated by the wiper control unit 320 is sent to the wiper 360, the wiper 360 is operated to ensure the visibility of the driver of the host vehicle 400.

  The analysis result of the image analysis unit 50 is also sent to the vehicle travel control unit 330. The vehicle travel control unit 330, for example, to the driver of the host vehicle 400 when the host vehicle 400 is out of the lane area defined by the white line based on the white line detection result detected by the image analysis unit 50. Announce a warning. In addition, the vehicle travel control unit 330 performs travel support control such as controlling a handle and a brake of the host vehicle 400, for example.

[Adherent detection device]
Next, the deposit detection apparatus according to the present embodiment will be described. The adhering matter detection device according to the present embodiment can detect adhering matters such as raindrops and foreign matters, but raindrops are given as a typical example of the adhering matter, and raindrop detection is given as a typical example of the adhering matter detection device. The following description will be given with an example of the apparatus. Therefore, the attached matter detection device 10 may be referred to as the raindrop detection device 10.

  FIG. 2 is a schematic view illustrating the schematic configuration of the raindrop detection apparatus according to this embodiment. FIG. 3 is a schematic view illustrating a part of FIG. 2 in an enlarged manner. With reference to FIGS. 2 and 3, the raindrop detection apparatus 10 includes an imaging unit 20 and an image analysis unit 50. The windshield 101 is a typical example of an object to which an attached object according to the present invention is attached.

  The imaging unit 20 is installed, for example, in the vicinity of the vehicle windshield 101 (for example, in the vicinity of a room mirror (not shown)). For example, the imaging unit 20 may be installed in the vicinity of the windshield 101 in a state in which the imaging unit 20 is accommodated in the cover 104 having at least the windshield 101 side.

  Alternatively, the cover 104 having the front glass 101 side opened may be brought into close contact with the windshield 101 so that the imaging unit 20 is covered with the windshield 101 and the cover 104. In this case, even if the inner wall surface of the windshield 101 is clouded, the windshield 101 in the portion covered with the cover 104 can be prevented from being fogged. As a result, the image analysis unit 50 can be prevented from being erroneously analyzed due to fogging of the windshield 101, and various control operations based on the analysis result of the image analysis unit 50 can be appropriately performed.

  However, for example, when the imaging unit 20 detects cloudiness of the windshield 101 and controls the air conditioning equipment of the host vehicle, the portion of the windshield 101 facing the imaging unit 20 is the same as the other parts. An air flow path may be formed in the cover 104 so that

  The imaging unit 20 includes a light source 30 and an imaging device 40. The light source 30 is a light source provided to detect raindrops attached to the windshield 101 and has a function of irradiating light from one surface side (inner wall surface side) of the windshield 101 toward the windshield 101. .

  The light source 30 is disposed at a position where at least the raindrop detection region 102 for detecting the raindrop 103 attached to the windshield 101 can be irradiated. For example, if the raindrop detection region 102 is on the lower end side of the windshield 101, the light source 30 is disposed at a position where at least the lower end side of the windshield 101 can be irradiated. If the raindrop detection area 102 is the upper end side and the lower end side of the windshield 101, the light source 30 is arranged at a position where at least the upper end side and the lower end side of the windshield 101 can be irradiated. If the raindrop detection region 102 is the entire surface of the windshield 101, the light source 30 is disposed at a position where at least the entire surface of the windshield 101 can be irradiated. In order to irradiate the raindrop detection region 102 with certainty, a plurality of light sources 30 may be arranged.

  As the light source 30, for example, a light emitting diode (LED), a semiconductor laser (LD), or the like can be used. The light source 30 emits light in a predetermined wavelength band. As the emission wavelength of the light source 30, for example, visible light, infrared light, or the like can be used.

  However, it is necessary to avoid dazzling the driver or pedestrian of the oncoming vehicle with the light of the light source 30. Therefore, as the light emission wavelength of the light source 30, a wavelength that is longer than the visible light and within a range in which the light receiving sensitivity of the image sensor 43 described later reaches (for example, an infrared light wavelength of 800 nm to 1000 nm) is selected. It is preferable.

  If necessary, it is desirable to provide a collimating lens downstream of the light source 30 so that the irradiation light from the light source 30 is substantially parallel light. The light source 30 is a typical example of the light irradiation means according to the present invention. However, the light irradiation means may include a component such as a lens capable of adjusting the form of irradiation light from the light source 30 such as a collimating lens. The raindrop detection area 102 is a typical example of the predetermined area according to the present invention.

  The imaging device 40 is disposed on the same side as the light source 30 with respect to the windshield 101, and has a function of imaging reflected light of light emitted from the light source 30 to the raindrop 103 and incident light from outside the vehicle. In the present embodiment, the imaging lens 41 is arranged so that the optical axis thereof faces in the horizontal direction, but the present invention is not limited to this. The light source 30 is disposed at a position where the reflected light of the light emitted from the light source 30 to the raindrop 103 enters the imaging device 40.

  The imaging device 40 includes an imaging lens 41, an optical filter 42, an image sensor 43, a sensor substrate 44, and a signal processing unit 45. The imaging lens 41 is composed of, for example, a plurality of lenses, and the focal point is set farther than the position of the windshield 101. The focal position of the imaging lens 41 can be set, for example, between infinity or infinity and the windshield 101.

  The optical filter 42 is disposed in front of the image sensor 43 and has a function of limiting the wavelength band of light incident on the image sensor 43. The optical filter 42 can be formed by appropriately selecting a material that can transmit light in the emission wavelength band of the light source 30. For example, when the light emission wavelength band of the light source 30 is a visible light region or an infrared region, glass, sapphire, crystal, or the like can be used as the material of the optical filter 42. The optical filter 42 may be formed so as to transmit only a predetermined polarization component (for example, P polarization component) of the incident light.

  4A and 4B are diagrams illustrating the optical filter 42 according to the present embodiment, in which FIG. 4A is a side view and FIG. 4B is a front view (viewed from the image sensor 43 side). As shown in FIG. 4, the optical filter 42 has a substrate 42a, and a spectral filter layer 42b that transmits infrared light and visible light is provided on the surface of the substrate 42a on the imaging lens 41 side (surface on the A side). Is formed. A polarizing filter layer 42c, an SOG (Spin On Glass) layer 42d, and an infrared light transmission filter 42e are formed in this order on the surface of the substrate 42a on the image sensor 43 side (surface on the B side). In FIG. 4, reference numeral 111 denotes a vehicle detection image area.

  In the present embodiment, the polarizing filter layer 42c is formed of a wire grid polarizer. The wire grid is formed by arranging conductor wires made of metal such as aluminum in a lattice pattern at a specific pitch. If the specific pitch of the wire grid is considerably smaller than the incident light (for example, visible light wavelength of 400 nm to 800 nm) (for example, half or less), the electric field vector component light that oscillates in parallel to the conductor line. Almost reflective. Then, most of the electric field vector component perpendicular to the conductor line is transmitted. Therefore, it can be used as a polarizer that produces single polarized light.

  FIG. 5 is a diagram illustrating a captured image. As shown in FIG. 5, the headlamp (not shown) of the oncoming vehicle, the tail lamp of the preceding vehicle 115, and the image of the white line 116 often exist mainly in the center of the captured image. Further, it is normal that an image of the road surface 117 immediately in front of the host vehicle exists at the lower part of the captured image and an image of the sky 118 exists at the upper part of the captured image. That is, information necessary for identifying the headlamp (not shown) of the oncoming vehicle, the tail lamp of the preceding vehicle 115, and the white line 116 is concentrated in the central portion of the captured image, and in the identification, information on the upper portion of the captured image and the lower portion of the captured image. Is not very important.

  Therefore, when both the detection of the oncoming vehicle, the preceding vehicle 115 or the white line 116 and the detection of the raindrop 103 are performed simultaneously from a single captured image data, as shown in FIG. The image area 111 is preferably used. Then, it is preferable that the upper part of the captured image and the lower part of the captured image are the raindrop detection image regions 112 and 113, respectively.

  Therefore, as shown in FIG. 4 described above, the red image is divided into an upper part of the captured image corresponding to the raindrop detection image area 112 in FIG. 5 and a lower part of the captured image corresponding to the raindrop detection image area 113 in FIG. An external light transmission filter 42e is provided. As the infrared light transmission filter 42e, for example, a cut filter that substantially transmits the emission wavelength of the light source 30 and shields the shorter wavelength side than the emission wavelength of the light source 30 as shown in FIG. Alternatively, a band pass filter that transmits only the vicinity of the emission wavelength of the light source 30 as shown in FIG. 7 may be used. In the present embodiment, the upper portion of the captured image is a portion corresponding to a height ¼ above the imaging region, and the lower portion of the captured image is a portion corresponding to height ¼ below the imaging region. .

  2 and 3, the image sensor 43 includes a plurality of two-dimensionally arranged pixels that receive light transmitted through the optical filter 42, and has a function of photoelectrically converting incident light for each pixel. The image sensor 43 is mounted on the sensor substrate 44. The image sensor 43 is composed of, for example, about several hundred thousand pixels arranged two-dimensionally. As the image sensor 43, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like can be used. For example, a microlens or the like corresponding to each pixel may be provided on the incident side of the image sensor 43 to increase the light collection efficiency to the image sensor 43.

  A signal processing unit 45 is provided following the image sensor 43. The signal processing unit 45 indicates the brightness (luminance) for each pixel of the captured image of the analog electrical signal photoelectrically converted by the image sensor 43 (electrical signal corresponding to the amount of light incident on each pixel of the image sensor 43). Convert to digital electrical signal. And it has the function to output to the image analysis unit 50 of a back | latter stage with the horizontal / vertical synchronizing signal of an image as image data. The image sensor 43 and the signal processing unit 45 are a representative example of the imaging unit according to the present invention.

  The image analysis unit 50 includes raindrop amount information calculation means 60. The raindrop amount information calculating means 60 has a function of processing the image data obtained from the signal processing unit 45 and calculating the amount of raindrop 103 attached to the windshield 101 and the like.

  The processing by the raindrop amount information calculating unit 60 is executed by a computer as a program, for example. The raindrop amount information calculating means 60 can be constituted by, for example, a CPU, RAM, ROM, FROM, etc. (not shown). The CPU may realize a function as the raindrop amount information calculating unit 60 and may have a function of controlling a peripheral IC that drives the light source 30 and the image sensor 43. The RAM has a function of holding imaging information, calculation results, and the like. The ROM has a function of storing a control program and the like. FROM is a non-volatile memory for holding adjustment values and the like. Note that the CPU, RAM, ROM, FROM, peripheral IC, and the like can transmit and receive signals via a bus, for example. The raindrop amount information calculating unit 60 is a typical example of the adhesion amount information calculating unit according to the present invention.

  Here, the raindrop detection apparatus 10 will be described in more detail. When raindrops 103 are not attached to the outer wall surface of the windshield 101, the light emitted from the light source 30 is reflected at the interface between the outer wall surface of the windshield 101 and the outside air, and the specularly reflected light is incident on the imaging device 40. It is configured not to. On the other hand, as shown in FIG. 2, when raindrops 103 are attached to the outer wall surface of the windshield 101, the refractive index difference between the outer wall surface of the windshield 101 and the raindrop 103 is the same as that of the outer wall surface of the windshield 101. It becomes smaller than the refractive index difference from the outside air.

  Therefore, the light emitted from the light source 30 passes through the interface between the outer wall surface of the windshield 101 and the raindrop 103 and enters the raindrop 103. The light incident on the raindrop 103 is reflected at the interface between the raindrop 103 and the air, and the diffusely reflected light is incident on the imaging device 40. Accordingly, the amount of light incident on the imaging device 40 increases due to the presence of the raindrop 103. Due to such a difference depending on the presence / absence of the raindrop 103, the raindrop amount information calculation means 60 can detect the presence / absence of the raindrop 103 attached to the windshield 101 from the captured image data transmitted from the imaging device 40.

  Here, a raindrop detection apparatus 10A, which is another configuration of the raindrop detection apparatus, will be described. FIG. 8 is a schematic view illustrating a schematic configuration of a raindrop detection apparatus according to a modification of the present embodiment. As shown in FIG. 8, the raindrop detection apparatus 10 </ b> A is different from the raindrop detection apparatus 10 (see FIG. 1 and the like) in that a light guide member 65 is added inside the windshield 101.

  In the raindrop detection apparatus 10A, when the raindrop 103 is not attached to the outer wall surface of the windshield 101 (not shown), the light emitted from the light source 30 is guided by a light guide member 65 such as a prism, Total reflection is performed at the interface between the outer wall surface of the windshield 101 and the outside air, and the reflected light enters the imaging device 40. On the other hand, as shown in FIG. 8, when raindrops 103 are attached to the outer wall surface of the windshield 101, the refractive index difference between the outer wall surface of the windshield 101 and the raindrop 103 is the same as that of the outer wall surface of the windshield 101. It becomes smaller than the refractive index difference from the outside air.

  Therefore, the light emitted from the light source 30 passes through the interface between the outer wall surface of the windshield 101 and the raindrop 103 without being totally reflected by the outer wall surface of the windshield 101. Therefore, the amount of light incident on the imaging device 40 is reduced due to the presence of the raindrop 103. Due to such a difference depending on the presence / absence of the raindrop 103, the raindrop amount information calculation means 60 can detect the presence / absence of the raindrop 103 attached to the windshield 101 from the captured image data transmitted from the imaging device 40.

  In the present embodiment, the focal position of the imaging lens 41 is set between infinity or infinity and the windshield 101. Thereby, the image analysis unit 50 not only detects when the raindrop amount information calculation means 60 detects the raindrop 103, but also other parts in the image analysis unit 50 detect the preceding vehicle and the oncoming vehicle, detect the white line, and the like. Also when performing, appropriate information can be acquired from the captured image data of the imaging device 40.

  However, when the imaging lens 41 is in focus at infinity, when identifying the tail lamp of a preceding vehicle traveling far away, the image sensor 43 may have about one pixel that receives light from the tail lamp. is there. In this case, there is a possibility that the tail lamp cannot be recognized and the preceding vehicle cannot be detected.

  In order to avoid such a problem, it is preferable to focus the imaging lens 41 closer to infinity than infinity. As a result, the tail lamp of the preceding vehicle traveling far is out of focus, so the number of pixels that receive light from the tail lamp can be increased. As a result, since the tail lamp recognition accuracy is improved, the detection accuracy of the preceding vehicle can be improved.

  The light source 30 can be disposed so as to irradiate only the region of the infrared light transmission filter 42e. Thereby, the noise component from the vehicle detection area can be avoided. A plurality of light sources 30 may be provided. When a plurality of light sources 30 are provided, it is preferable to set the polarizer pattern in each region of the polarizing filter layer 42c so as to transmit only a predetermined polarization component. The predetermined polarization component refers to two optical axes, that is, an optical axis of light emitted from the light source having the largest incident light quantity to the polarizer pattern toward the windshield 101 among the plurality of light sources 30 and an optical axis of the imaging lens. Is a polarization component parallel to the plane formed by

  As a light emitting method of the light source 30, in addition to continuous light emission (also referred to as CW light emission), pulse light emission may be performed at a specific timing. By synchronizing the timing of light emission and the timing of photographing, the influence of disturbance light can be further reduced. When a plurality of light sources 30 are installed, the plurality of light sources 30 may emit light simultaneously or sequentially. When light is emitted sequentially, the influence of disturbance light can be further reduced by synchronizing the light emission timing with the photographing timing.

  FIG. 9 is a diagram illustrating the configuration of the image analysis unit. Referring to FIG. 9, the image analysis unit 50 includes a CPU 51, a ROM 52, a RAM 53, a hardware logic 54, a digital signal processor 55, an I / O port 56, and a communication interface 57.

  The processing by the raindrop amount information calculating means 60 is mounted on the image analysis unit 50 as, for example, a program of the CPU 51, as the hardware logic 54, or as a mixed form thereof. The I / O port 56 and the communication interface 57 are connected to each unit shown in FIG. 1 according to necessary information.

  The raindrop amount information calculation means 60 receives the captured image data transmitted from the imaging device 40 of the imaging unit 20, detects the raindrop 103 adhering to the windshield 101, and sends the detection result to the wiper control unit 320. The wiper control unit 320 operates the wiper 360 based on the result.

  Each unit shown in FIG. 1 may be mounted on the vehicle as independent hardware, or each unit may be mounted on, for example, one substrate. Further, the image analysis unit 50 has a plurality of functions such as raindrop amount calculation, headlamp control, front image recognition, etc., but each function is distributed to provide a raindrop amount calculation function (raindrop amount information calculation means 60). ) May be mounted on the imaging unit 20 or the wiper control unit 320. The mounting form can be changed as appropriate according to the mounting space and cost.

  FIG. 10 is a diagram illustrating an example in which the raindrop amount information calculating unit is mounted on the imaging unit. In the example of FIG. 10, the raindrop amount information calculation means 60 is physically integrated with the imaging device 40 (for example, the CPU and the like constituting the imaging device 40 and the raindrop amount information calculation means 60 are on one substrate. Installed).

  FIG. 11 is a diagram illustrating a connection example between the raindrop amount information calculating unit and the imaging apparatus. Referring to FIG. 11, the raindrop amount information calculation means 60 includes a CPU 61, a ROM 62, a RAM 63, a hardware logic 64, I / O ports 66a and 66b, and communication interfaces 67a and 67b. Thus, the process in the raindrop amount information calculating means 60 can be realized as a program of the CPU 61, as the hardware logic 64, or as a mixed form thereof.

  The raindrop amount information calculation means 60 transmits / receives image data to / from the imaging device 40 via the I / O port 66a, and transmits / receives control commands for the imaging device 40 via the communication interface 67a. The I / O port 66b and the communication interface 67b are connected to the wiper control unit 320 shown in FIG. 1 according to necessary information. The connection to the wiper control unit 320 may be made via the image analysis unit 50.

[Raindrop amount information calculation means]
Next, the raindrop amount information calculating means according to the present embodiment will be described. In the present embodiment, the raindrop amount information calculating means 60 is configured so that the raindrop 103 can be detected with high detection sensitivity. Before explaining the details of the processing by the raindrop amount information calculating means 60, the basic idea will be explained.

  FIG. 12 is a diagram illustrating the signal luminance of raindrops detected in the raindrop detection region. First, as shown in FIG. 12A, consider a case where the raindrop detection area 102 is not divided into a plurality of reference areas. Here, it is assumed that the noise N exists uniformly in the raindrop detection area 102 and that the signal S detected from the raindrop 103 exists only in one reference area. In this case, the sum of pixel values (sum of luminance) is S + N, and the SN ratio is S / N.

  In the present application, the sum of all pixel values in an arbitrary reference region is referred to as a “sum of pixel values”, and the sum of the pixel values in each reference region is referred to as “the sum of pixel values”. Called. That is, when there are p reference areas (p is a natural number of 2 or more), the sum of the pixel values = the sum of the pixel values of the first reference area + the sum of the pixel values of the second reference area +. This is the sum of the pixel values of the p-th reference area. However, in the above example, since only one signal S detected from the raindrop 103 exists in the reference region, the sum of the pixel values in the reference region = the sum of the pixel values. Further, since the pixel value indicates luminance, the sum of the pixel values is synonymous with the luminance sum, and the sum of the pixel values is synonymous with the luminance sum.

Next, as shown in FIG. 12B, consider a case where the raindrop detection area 102 is divided into _p reference areas 102 _{1 to} 102 _P (p is a natural number of 2 or more). In this case, assuming that the noise N exists uniformly in the raindrop detection region 102, the noise in each of the reference regions 102 _{1 to} 102 _P is N / p.

Figure 12 (b), the signal S detected in the reference region 102 ₄ raindrop 103 is present. In reference area 102 _4, the sum of the pixel values S + N / p becomes, SN ratio is S / (N / p) = pS / N. As described above, by dividing the raindrop detection region 102 into p reference regions 102 _{1 to} 102 _P , the sensitivity becomes p times.

That is, by dividing the raindrop detection area 102 into p reference areas 102 _{1 to} 102 _P , it is possible to detect a small amount of signal S from the raindrop 103 with high sensitivity without adding an excessive amount of noise. Become. Note that the size of each reference region is preferably a size including the diameter of the raindrop 103. This is because the signal S from one raindrop 103 is not missed.

  Note that the number of reference regions can be set as appropriate, but may be a number corresponding to the light source. That is, for example, when there are eight light sources (for example, light emitting diodes (LEDs)), the number of reference regions can be eight. Thereby, a raindrop signal can be detected with higher sensitivity corresponding to each light source.

Here, a method for determining that the raindrop 103 is attached to the windshield 101 will be described by taking the case where the number of reference regions is 8 (p = 8) as an example. FIG. 13 is a diagram illustrating the sum of pixel values in each reference region of raindrops detected in the raindrop detection region. As shown in FIG. 13, when it starts to rain and the raindrop 103 starts to adhere to the windshield 101, the sum of the pixel values of the reference regions 102 _{1 to} 102 ₈ (or the average value of the pixel values) may be random. Begin to increase.

As means for determining that it is raining, a threshold Th0 for candidate determination is determined in advance. When the sum of the pixel values (sum of luminance) of each of the reference areas 102 _{1 to} 102 ₈ exceeds the threshold value Th0, it is determined that the raindrop 103 is attached to the windshield 101 in the reference area, and the threshold value Th0 is set. The sum of the pixel values of the reference areas that have been exceeded is added to obtain the sum of the pixel values, which is used as raindrop amount information. In other words, the raindrop amount information in this case is a characteristic amount that replaces the rainfall by representing only the sum of the raindrop signals that have exceeded the threshold Th0.

However, an average value of each pixel value of the reference region ₁₀₂ 1-102 _8, when the average value of each pixel value of the reference region ₁₀₂ 1-102 ₈ exceeds a threshold value Th0, with its reference area raindrop You may determine with 103 adhering to the windshield 101. FIG. In this case, the sum of the average values of the pixel values in the reference area exceeding the threshold Th0 can be summed to obtain the sum of the average values of the pixel values, which can be used as raindrop amount information. In the case of FIG. 13, the reference region _{102 2,} 102 4, 102 _5, and employs a sum of each pixel value of 102 ₈ their total can be calculated amount of raindrops information.

  The reason why the threshold Th0 is provided is as follows. The sum of pixel values detected from each reference area is not necessarily information based on the attachment of raindrops. For example, information based on adhesion of foreign matter other than raindrops such as sand, dust, pollen, and bird droppings is also included. However, the sum of pixel values based on the adhesion of these foreign substances is usually not as great as the sum of pixel values based on the adhesion of raindrops.

  Therefore, if a threshold Th0 is provided so that the sum of pixel values based on the adhesion of foreign matter and the sum of pixel values based on the attachment of raindrops can be separated, information considered to be based on the adhesion of foreign matter is judged as noise and removed. it can. As a result, the raindrop information in each reference region can be detected with high accuracy, and the raindrop amount information for the entire raindrop detection region 102 based on the raindrop information can be calculated with high accuracy.

  In the present application, the raindrop information is information on the rainfall in each reference area, and the raindrop information is information on the rainfall in the entire raindrop detection area 102 (all reference areas). That is, raindrop information corresponds to the sum of pixel values, and raindrop amount information corresponds to the sum of pixel values. The raindrop information (sum of pixel values) is a typical example of the deposit information by region according to the present invention, and the raindrop amount information (total sum of pixel values) is representative of the deposit amount information according to the present invention. This is an example.

  The threshold value Th0 can be determined, for example, by acquiring a lot of data regarding each of the sum of pixel values based on the attachment of raindrops and the sum of pixel values based on the attachment of various foreign substances, and analyzing the acquired data. A different threshold may be provided for each reference area.

  In some cases, a signal from the raindrop 103 appears across the boundary between the reference areas spread over each other. When the signal from the raindrop 103 crosses the boundary of the reference region, the signal is decomposed into two adjacent reference regions, and the sum of the pixel values does not reach the threshold Th0 in each of the two adjacent reference regions. May not be detected. In this case, a reference area that covers the boundary of the reference area may be further installed. As a result, no matter where the raindrop 103 is attached, it can be detected in any reference area.

  In addition, when acquiring raindrop information in each reference area, the sum of the pixel values of the image obtained when the light source 30 is turned off is subtracted from the sum of the pixel values of the image obtained when the light source 30 is turned on. preferable. This is because the background (background) included in the image obtained when the light source 30 is turned on is removed, and only the raindrop signal is extracted.

  Further, it is preferable to multiply the sum of pixel values (raindrop information) inside each reference area by a gain before comparing with the threshold Th0. This is to correct the variation of light emitting diodes (LEDs) used as the light source 30 and the peripheral light amount of the collimating lens so that the same raindrop can be obtained with the same pixel value in each reference region. Furthermore, in order to remove in advance the noise that is already known, such as flare that occurs optically in each reference area, the offset after multiplying the sum of the pixel values (raindrop information) inside each reference area by a gain Is preferably reduced.

  Here, a specific operation of the raindrop amount information calculating unit 60 will be described with reference to a flowchart. As a premise, in the present embodiment, as an example, a raindrop detection region 102 is provided on the lower end side of the image sensor 43 as shown in FIG.

  The number of pixels of the image sensor 43 may be arbitrary, but may be a general VGA size (640 × 480 pixels), for example. In that case, if the raindrop detection area is about 640 × 30 pixels, the balance between normal imaging area (for front imaging of the vehicle) and securing of the raindrop detection area is good. Further, in consideration of ease of calculation processing, for example, it may be 512 × 32, 512 × 64 pixels or the like so as to be a power of 2.

In the present embodiment, as an example, as shown in FIG. 15, the image area of the image sensor 43 corresponding to the raindrop detection area 102 is divided into _eight reference areas 102 _{1 to} 102 8 that are spread without gaps. To do. Furthermore, it is assumed that seven reference regions 102 _{9 to} 102 ₁₅ including the boundary between adjacent reference regions are provided. Each reference area corresponds to a 30 × 80 pixel area of the image sensor 43. In FIG. 15, for the sake of clarity, the reference regions 102 _{1 to} 102 ₈ and the reference regions 102 _{9 to} 102 ₁₅ are shifted up and down for convenience.

  As described above, in the present embodiment, as an example, the image area of the image sensor 43 corresponding to the raindrop detection area 102 is divided into reference areas that are spread without gaps. However, if the minimum raindrop can be detected, reference is made. There may be a gap between the regions.

  FIG. 16 is an example of a flowchart for explaining the operation of the raindrop amount information calculating means. The program corresponding to the flowchart shown in FIG. 16 can be stored in, for example, the ROM 62 of the raindrop amount information calculating unit 60 and can be executed by the CPU 61 of the raindrop amount information calculating unit 60. Alternatively, the raindrop amount information calculation means 60 is provided in the image analysis unit 50, and the processing by the raindrop amount information calculation means 60 is performed, for example, as a program of the CPU 51 of the image analysis unit 50, as the hardware logic 54, or You may implement as a mixed form.

First, in step S <b> 101, the raindrop amount information calculating unit 60 turns on the light source 30 when the vehicle is running, for example, acquires imaging information of the raindrop detection region 102 irradiated with the light source 30, and stores the reference regions 102 _{1 to} 102 ₁₅ . In each case, a sum Pi (i = 1 to 15) of pixel values is calculated. Here, the pixel value (luminance) per pixel (pixel) is represented by 10 bits (0 to 1023). That is, the pixel value (luminance) of each pixel is expressed as a numerical value between 0 and 1023.

Next, in step S102, the raindrop amount information calculating unit 60 turns off the light source 30, for example, when the vehicle is running, acquires the imaging information of the raindrop detection area 102, and sets the pixel value in each of the reference areas 102 _{1 to} 102 ₁₅ . The sum Qi (i = 1 to 15) is calculated.

  Next, in step S103, the raindrop amount information calculation unit 60 calculates the sum of pixel values when the light source 30 is turned off in step S102 from the sum Pi (i = 1 to 15) of pixel values when the light source 30 is turned on in step S101. Qi (i = 1 to 15) is subtracted to calculate raindrop information Ai (i = 1 to 15). At this time, if Ai <0, it is handled as Ai = 0. Here, since the pixel value (luminance) per pixel (pixel) is represented by 10 bits (0 to 1023), 1023 × 30 × 80 = 22 bits are required to represent each Ai.

  Next, in step S104, the raindrop amount information calculation unit 60 multiplies each Ai calculated in step S103 by a gain to reduce the offset. The range of gains to be multiplied by each Ai can be determined as appropriate. For example, a gain of 8 bits (0 to 255) can be multiplied using a multiplier.

  When each Ai is 22 bits, gain adjustment can be performed in a range of substantially 0 times to about 4 times by multiplying the gain of 8 bits and removing the lower 6 bits. The gain can be set for each reference area. The range of offsets to be subtracted from each Ai after multiplication by the gain can be determined as appropriate. For example, a 24-bit (0-16777216) offset can be subtracted using a subtractor.

Next, in step S105, Ai (i = 1 to 15) obtained by multiplying the gain in step S104 and subtracting the offset is sequentially compared with a preset threshold value Th0, and only Ai when Ai> Th0 is added. A total sum A of pixel values is calculated. For example, when only the raindrop information A ₂ , A ₄ , A ₅ , and A ₈ of each of the reference regions 102 ₂ , 102 ₄ , 102 ₅ , and 102 ₈ exceeds Th0, the sum of pixel values A = A ₂ + A ₄ + A ₅ + A ₈ If each Ai is 22 bits, 1023 × 30 × 80 × 15 × 4 = 28 bits are required to represent the sum A of pixel values. Here, 15 is the number of reference areas, and 4 is the maximum value of the gain.

  In this way, the sum A of pixel values representing the raindrop amount information can be calculated. Note that the level of raindrop amount can be known by further comparing the sum A of pixel values with a predetermined threshold (for example, five thresholds Th1 to Th5).

Thus, in the present embodiment, a plurality of reference regions 102 _{1 to} 102 102 in which the image region (for example, a region of 30 pixels × 640 pixels) of the image sensor 43 corresponding to the raindrop detection region 102 is laid without gaps. Divide into ₈ . Further, reference areas 102 _{9 to} 102 ₁₅ that include borders between adjacent reference areas among the reference areas 102 _{1 to} 102 ₈ are provided, and raindrop information in each reference area is calculated. Then, the amount of raindrop information is calculated based on only the raindrop information that exceeds the predetermined threshold Th0 by comparing the raindrop information in each reference region with the predetermined threshold Th0.

  Thereby, raindrop information that does not exceed the threshold Th0, that is, information that is considered to be based on adhesion of foreign matters other than raindrops such as sand, dust, pollen, and bird droppings can be determined as noise and removed. As a result, accurate raindrop information and raindrop amount information based on the raindrop information can be calculated. That is, a raindrop detection device and a raindrop detection method that can improve detection sensitivity can be realized.

Further, by providing the reference region ₁₀₂ 9-102 ₁₅ includes boundary of adjacent reference area of the reference region ₁₀₂ 1-102 _8, across the boundary of the reference region ₁₀₂ 1-102 ₈ paved mutually raindrop Even when a signal from appears, it is possible to detect raindrops. That is, no matter which raindrop is attached to the raindrop detection area 102, it is possible to detect it in any reference area.

  In the present embodiment, when acquiring raindrop information in each reference region, the pixel value of the image obtained when the light source 30 is turned off is calculated from the sum of the pixel values of the image obtained when the light source 30 is turned on. The sum is subtracted. Thereby, the background (background) included in the image obtained when the light source 30 is turned on can be removed, and only the signal from the raindrop can be extracted.

  In the present embodiment, before comparing the raindrop information in each reference area with a predetermined threshold Th0, the raindrop information is multiplied by a gain to reduce the offset. By multiplying the gain, variations in the light emitting diodes (LEDs) used as the light source 30 and the peripheral light amount of the collimating lens, etc. are corrected, and the same raindrop can be obtained with the same pixel value in each reference area. Further, by reducing the offset, noise such as flare optically generated in each reference region can be removed in advance.

[Response to splash]
Here, an example will be described in which the raindrop amount information calculation means enables more stable raindrop detection and also allows raindrop detection without reducing the reaction speed against a sudden change (splash) in raindrop information.

FIG. 17 illustrates the state from the time t = T0 to T4 of the sum of the pixel values of each reference area when the raindrop detection area 102 is divided into _eight reference areas 102 _{1 to} 102 8 as shown in FIG. It is a schematic diagram. Times t = T0 to T4 indicate the timing at which an image for detecting raindrops is captured and the final calculation result (raindrop amount information) is obtained. The time interval at which images are captured can be set to about 100 ms, for example.

FIG. 17A shows a state where there is no rain at time t = T0, and Kn0 is noise that is constantly generated. Causes of noise include those caused by sunlight entering from the windshield and electrical noise. As described above, Th0 is a threshold value for the pixel value, and when the sum of pixel values (luminance sum) in the reference regions 102 _{1 to} 102 ₈ exceeds the threshold value Th0, the raindrop 103 is present in the reference region. It determines with adhering to the windshield 101, and uses it for calculation of raindrop amount information.

FIG. 17 (b), at time t = T1, represents a state in which raindrop appears in reference area 102 _1. FIG. 17C shows a state in which the noise level has increased at time t = T2. Further, FIG. 17 (d), at time t = T3, and represents a state in which raindrop appears in reference area 102 _8. Further, FIG. 17 (e) is the noise which the sum of pixel values appearing in the reference area 102 ₈ at time t = T3 occurs only time t = T3, a case in which the noise disappears at time t = T4 Show.

FIG. 18 is a diagram illustrating the relationship between the sum of pixel values and time. FIG. 18A shows the sum of pixel values (luminance values) output by the image sensor 43 when there is a certain amount of rainfall (attachment of raindrops) per unit time when the processing shown in FIG. 16 is not performed. It shows the relationship between the sum) and time. Note that Kn0 × 8 indicated by a broken line is the total sum of noises that are constantly generated in the reference regions 102 _{1 to} 102 ₈ .

FIG. 18B is a reference when the rainfall shown in FIG. 17 changes from the time shown in FIG. 17A to the time shown in FIG. 17D after the processing shown in FIG. is a graph showing the temporal change in the sum of the pixel values of region ₁₀₂ 1-102 _8.

In FIG. 18B, since the sum of the pixel values does not exceed the threshold Th0 in any reference region until time t = T1, the sum of the pixel values is zero. Since the sum of the pixel value exceeds the threshold value Th0 in the reference region 102 ₁ at time t = T1, the sum of pixel values in the reference region 102 ₁ is output as the sum of the pixel values. At time t = T2, there is no change in output (total pixel value).

Since the sum of the pixel value exceeds the threshold value Th0 in the reference region 102 ₈ at time t = T3, a sum of pixel values in the reference region 102 ₈ to the sum of the pixel values in the reference region 102 ₁ is added, as the sum of the pixel values Is output. Although the subsequent states are not illustrated, when the sum of the pixel values increases step by step for each of the reference regions 102 _{1 to} 102 ₈ , when the sum of the pixel values exceeds the threshold Th0 in all the reference regions, FIG. ), The sum of the pixel values changes. As described above, by performing the processing shown in FIG. 16, it is possible to detect pixel values due to raindrops buried in the noise level (Kn0 × 8 region).

However, as explained in FIG. 17 (d), the sum of the pixel values in the reference region 102 ₈ at time t = T3 is not a raindrop information may be a noise generated by at time t = T3. In this case, as described with reference to FIG. 17 (e), the noise at time t = T4 in reference area 102 ₈ disappears, Figure 18 (c), the sum of the pixel values at time t = T2 Return to the same level as the time.

  In the processing shown in FIG. 16, since the sum of the pixel values for each reference region does not become a calculation target until the threshold Th0 is exceeded, and immediately after the threshold Th0 is exceeded, it becomes a calculation target. . When the vehicle wiper is operated based on such a result, the wiper may move quickly in response to the noise at time t = T3, and may become unstable operation such as stopping immediately thereafter.

  Such a fear of unstable operation can be eliminated by performing a calculation that also takes into account the current and past situations to suppress a rapid change. Specifically, this can be eliminated by performing an averaging process on the sum of pixel values. An example of the averaging process is a moving average process. When averaging processing such as moving average processing is performed, for example, the change in the sum of pixel values at time t = T4 as in the relationship between the sum of pixel values and time shown in FIG. This can be suppressed more than in the case of (c). In FIG. 18 (d), the rise at time t = T1 is somewhat reduced, but since raindrop detection has been performed, there is no problem with minimum raindrop detection if the threshold Th0 is lowered.

  FIG. 19 shows an example of the moving average process. In the example of FIG. 19, the sum KS (t) of pixel values at time t, the sum KS (t−1) of the previous pixel value, and the sum KS (t−2) of the previous pixel value are used. Thus, KH (t) = 1/3 × KS (t−2) + 1/3 × KS (t−1) + 1/3 × KS (t) is calculated. However, the total number of pixel values to be averaged may be changed according to the noise situation. Further, the coefficient may be other than 1/3, 1/3, or 1/3, and may be set to, for example, 1/4, 1/2, 1/4, or the like in consideration of the characteristics of the filter to be realized. Moreover, you may use the well-known averaging process (IIR filter etc.) which is not illustrated.

  Next, let us consider a case in which changes occur in a short time as shown in FIGS. FIG. 20A shows a state where there is no rain at time t = T4, and Kn0 is noise that is constantly generated. FIG. 20B shows a case where raindrops appear in all reference areas at time t = T5. FIG. 20C shows a case where the amount of raindrops appearing in all the reference areas further increases at time t = T6.

  After FIG. 20C, the state may return to the state at time t = T4 shown in FIG. 20A again, or the amount of raindrops appearing in all reference regions may further increase. When returning to the state at time t = T4 shown in FIG. 20 (a) after FIG. 20 (c), for example, water splashes on the windshield 101 when the oncoming vehicle travels on a deep puddle, A steep evening can be considered.

  FIG. 21 is a diagram illustrating the relationship between the sum of pixel values and time in the above case (when returning to the state at time t = T4). However, FIG. 21A shows the sum of pixel values (raindrop amount information) when the moving average process (filter process) shown in FIG. 19 is not performed, and FIG. 21B shows the movement shown in FIG. This is the sum of pixel values (average raindrop amount information) when average processing (filter processing) is performed. The average raindrop amount information is a typical example of the average adhesion amount information according to the present invention.

  As shown in FIG. 21 (a), when the moving average process is not performed, the sum of the pixel values increases at a time at time t = T5 and time t = T6. Therefore, from time t = T4 to time t = T5 and The sum of the pixel values changes (rises) rapidly until T6. In such a case, since a large amount of raindrops adhere to the windshield 101 and obstructs the field of view, it is necessary to operate the wiper immediately (splash handling operation).

  On the other hand, as shown in FIG. 21B, when the moving average process is performed, the waveform (graph) with a rising edge is lost, and the wiper may not be able to be operated immediately. Therefore, when the sum of pixel values changes abruptly, it is preferable that a waveform (graph) indicating the relationship between the sum of pixel values and time is not blurred.

  FIG. 22 is a block diagram exemplifying a specific process (to prevent a waveform (graph) from becoming distorted) to cope with a sudden change in raindrop information. In FIG. 22, first, the moving average processing means 71 performs the above-described moving average processing, and the differential calculation means 72 performs difference calculation processing. The difference calculation process is to calculate a differential signal indicating the degree of change of the adhesion amount information with respect to time.

  Specifically, in the differential calculation means 72, KD, which is a difference (a change in unit time) between the sum KS (t) of pixel values at time t and the sum KS (t−1) of the previous pixel value. Calculate (t) = − 1 × KS (t−1) + KS (t). The calculation result in the differential calculation means 72 is input to the A terminal of the comparison means 73 and compared with the second threshold THs input to the B terminal of the comparison means 73.

  The comparison means 73 outputs “1” from the C terminal to the S terminal of the selection means 74 functioning as a selector when the calculation result by the differentiation calculation means 72 is equal to or greater than the second threshold THs, and is less than the second threshold THs. In this case, “0” is output from the C terminal to the S terminal of the selection means 74. Note that a moving average value KH (t), which is a sum of pixel values after moving average processing (average raindrop amount information), is input to the A terminal of the selection means 74, and pixels before moving average processing are input to the B terminal. The sum KS (t) of pixel values at time t, which is the sum of values, is input.

  When the signal input from the comparison unit 73 to the S terminal is “0”, the selection unit 74 outputs the moving average value KH (t) as the output KO (t) from the terminal C. Further, when the signal inputted from the comparison means 73 to the S terminal is “1”, the selection means 74 outputs the sum KS (t) of the pixel values at time t as the output KO (t) from the terminal C. . Thus, in the present embodiment, when the amount of change in the unit time of rainfall is equal to or greater than the second threshold THs, the pixel value before the moving average process is not selected without selecting the sum of the pixel values after the moving average process. Is selected (sum of current pixel values).

  FIG. 23 is a diagram illustrating a result of the difference calculation process in the differential calculation means. As shown in FIG. 23, by setting the second threshold value THs so high that it does not affect the moving average process, at time t = T5 and T6, instead of the sum of the pixel values in FIG. Since the sum of the pixel values of 21 (a) is selected, the waveform (graph) shown by the solid line in FIG. 24 is obtained. That is, an abrupt change that does not occur is output. In addition, the broken line of FIG. 24 has shown the waveform (graph) of FIG.21 (b) for the comparison.

  Note that the difference calculation processing in the differential calculation means 72 in FIG. 22 may change the coefficient according to the change to be detected or widen the range of pixels to be used. Further, instead of the difference calculation process, second-order differential detection such as a well-known Laplacian filter (not shown) may be performed.

  The above processing is summarized in FIG. 25 and FIG. As shown in FIG. 25, first, in step S201, the raindrop amount information calculation means 60 reads an image frame to be processed every predetermined time (time t = T0, T1, T2,...). Next, in step S <b> 202, the raindrop amount information calculation unit 60 cuts out pixels in the raindrop detection area 102 of the image sensor 43.

  Next, the raindrop amount information calculation means 60 calculates the sum of the pixel values of each reference area in the same manner as the processing shown in FIG. 16, further obtains the sum of the pixel values (step S203), and the t-th image frame. Is stored (step S204).

  Next, in step S205, the raindrop amount information calculation means 60 executes a moving average process. For example, as shown in step S301 of FIG. 26 (a), the moving average processing means 71 performs summation KS (t-3), KS (t-2), and KS (t-1) of pixel values of four image frames. ) And the average of KS (t) to calculate the moving average value KH (t) (Formula 1). However, calculating the moving average value from the past four image frames is an example, and the moving average value may be calculated from the past two or three, or the past five or more image frames.

KH (t) = 1 / 4KS (t-3) + 1 / 4KS (t-2) + 1 / 4KS (t-1) + 1 / 4KS (t)...
Next, in step S206, the raindrop amount information calculation unit 60 executes difference calculation processing, comparison, and selection. For example, as shown in step S401 of FIG. 26B, the differential calculation means 72 performs a difference calculation process. Specifically, as shown in Equation 2, the sum KS (t−1) of pixel values at time t−1 is subtracted from the sum KS (t) of pixel values at time t, and a difference KD (t ) Is calculated. However, in Expression 2, t> = 2 and KD (1) = 0.

KD (t) = KS (t) −KS (t−1).
For example, as shown in step S402 in FIG. 26B, the comparison unit 73 compares the difference KD (t) with the second threshold THs, and the difference KD (t) is equal to or greater than the second threshold THs. "1" is output as the comparison result. When the difference KD (t) is less than the second threshold value THs, “0” is output as the comparison result.

  Then, when “1” is input from the comparison unit 73, the selection unit 74 outputs the sum KS (t) of the pixel values at time t as the output KO (t). When “0” is input from the comparison means 73, the moving average value KH (t) is output as the output KO (t). That is, KO (t) = KS (t) when KD (t) ≧ THs, and KO (t) = KH (t) when KD (t) <THs.

  Next, in step S207, the raindrop amount information calculation means 60 determines whether or not reading of all image frames has been completed. If not completed (in the case of No), the processing of steps S201 to S206 is performed. repeat. If all the image frames have been read (Yes), the process ends.

  As described above, by introducing the moving average process for the sum of the pixel values, it is possible to remove the noise component that fluctuates with the time of the raindrop information and to perform stable raindrop detection. In addition, by appropriately selecting the sum of pixel values before moving average processing and the sum of pixel values after moving average processing according to the amount of rainfall, it reacts to sudden changes (splash) in raindrop information. Raindrop detection is possible without slowing down.

[Response to splash and reduction of noise effects]
By the way, in FIG. 18A, the noise level in each reference area is set to Kn0 which is a constant value, but the actual noise level does not become a constant value because it fluctuates due to variations in the environment and detectors. Thus, for example, a fluctuating noise level is measured, its maximum value is predicted, and the predicted maximum noise level is set. For this reason, there is a problem in that the amount of raindrops below the preset maximum noise level cannot be detected even in the actual use environment even when the noise level is low.

  That is, the noise level Kn0 in each reference region shown in FIG. 18A is the maximum noise level predicted in advance, and even if the actual noise level is smaller than Kn0, the raindrop amount below Kn0 cannot be detected. There was a problem. This problem can be solved by introducing a weighted average process (weighted addition average process). Hereinafter, a method for solving this problem will be described in detail.

  FIG. 27 is a graph simulating the output of raindrop detection processing for the amount of raindrops attached to the windshield. In FIG. 27, the vertical axis represents the sum of pixel values, and the horizontal axis represents unit time. The unit time is the time for which the image sensor 43 samples an image (time for each frame), and can be about 100 ms, for example. When the unit time is 100 ms, the maximum value on the horizontal axis in FIG. 27 is 31 × 100 ms = 3.1 s.

  In FIG. 27, KS represents the sum of pixel values, N represents a noise pixel value per unit time, and S represents a pixel value per unit time. Here, as described above, the sum of the pixel values is the sum of the values obtained by converting the deposition amount (area) information of the raindrops on the windshield photoelectrically converted by the image sensor 43 into pixel data, and each reference region. This is a value obtained by summing the region-specific deposit information (sum of pixel values) indicating the amount of deposits at. Therefore, the sum of the pixel values may be referred to as summed deposit information.

  The noise pixel value per unit time represents random noise such as electrical noise and external light noise that appears every unit time. The pixel value per unit time represents the pixel value for each frame sampled from the image sensor 43 per unit time. The sum KS (t) of pixel values at time t is a cumulative sum of the pixel value S (t) per unit time and the noise pixel value N (t) per unit time. However, in practice, the noise pixel value N per unit time is unknown and is the sum of the pixel value S per unit time, and therefore cannot be separated as shown in the graph of FIG.

  Note that when the raindrop 103 adheres to the entire raindrop detection area 102, the pixel value does not increase any more, so the sum of the pixel values is saturated. In FIG. 27, the sum KS of pixel values is saturated around time t = 9.

  The weighted average value KJ can be calculated by, for example, a block diagram related to the weighted average process shown in FIG.

KJ (t) = F * KS (t) + (1-F) * KS (t-1) (t ≧ 2, KJ (1) = KS (1), F: 0 to 1) 3
It should be noted that the weighted average value KJ can be calculated using information after a predetermined time by previously storing the sum KS of the pixel values at each time in the memory. The weighted average value KJ may be calculated from the sum of pixel values at three or more times.

  FIG. 29 is a graph when the noise of the DC component is removed by performing the weighted average difference process. In FIG. 29, in order to reduce the influence of fluctuations in the sum KS (t) of pixel values, F = 1/32 in Expression 3. In FIG. 29, KS represents the sum of pixel values, KJ represents a weighted average pixel value (weighted average value), and KOS represents an output pixel value.

  In FIG. 29, it can be seen that the DC noise is removed from the output pixel value KOS. Although the output pixel value KOS gradually decreases as the pixel value sum KS continues to be saturated, the wiper 360 normally operates to remove the raindrops 103 attached to the windshield 101, and the amount of raindrops attached. Will not be a problem.

  The output pixel value KOS (weighted average difference value) can be calculated by, for example, the block diagram of the weighted average difference process shown in FIG. 30, and is expressed as the following Expression 4.

KOS (t) = KS (t) −KJ (t) (When KOS (t) <0, KOS (t) = 0).
FIG. 31 is a graph when the moving average process is further applied to the output pixel value in the state of FIG. In the output pixel value KOS (moving average value of the weighted average difference value) in FIG. 31, it can be seen that fine noise (high frequency noise) is attenuated as compared with the output pixel value KOS (weighted average difference value) in FIG. .

  32 and 33 are graphs showing simulation results for a sudden change (splash) in raindrop information. FIG. 32 is a graph when the moving average process is added to the output pixel value, but the difference calculation process described with reference to FIG. 22 is not introduced. FIG. 33 is a graph when the moving average process is added to the output pixel value and the difference calculation process described with reference to FIG. 22 is further introduced.

  In FIG. 32, since the difference calculation process is not introduced, the output pixel value KOS is gentle even though the pixel value S per unit time has increased to just before saturation at the times t = 11, 12, and 13. It has changed.

  On the other hand, in FIG. 33, since the difference calculation process is introduced, when the result of the difference calculation process is equal to or greater than a predetermined threshold, the sum of the pixel values after the moving average process is not selected, and before the moving average process. The sum of pixel values (total sum of current pixel values) is selected. Therefore, in FIG. 33, the waveform rounding of the output pixel value KOS near the time t = 10 and 11 is eliminated compared to FIG. 32, and the output pixel value KOS is the sum of the pixel values that are the input pixel values at the time t = 11. Following KS.

  In FIG. 33, after time t = 12, 13, the output pixel value KOS returns to the same state as in FIG. 32, but normally the wiper 360 operates to remove the raindrop 103 attached to the windshield 101. This is not a problem because the amount of raindrop adhesion is reset.

  34 and 35 are examples of flowcharts for explaining the processing of FIG. 33 (specific processing for responding to a sudden change in raindrop information). FIG. 36 is a block diagram illustrating the process of FIG. 34, the same reference numerals are given to the same processing portions as those in FIG. 25, and the description thereof is omitted. 36, the same reference numerals are given to the same processing portions as those in FIG. 22, and the description thereof is omitted.

  In the flowchart shown in FIG. 34, step S501 is added between step S204 and step S205 of the flowchart shown in FIG. The specific process of step S501 of FIG. 34 is shown in step S601 of FIG. In step S601, the weighted average difference processing means 75 shown in FIG. 36 calculates the weighted average value KJ (t) as shown in FIGS. 28 and 30, and equations 3 and 4, and further calculates the sum KS ( The output pixel value KOS (t) is calculated by subtracting the weighted average value KJ (t) from t). The processes before and after step S501 in FIG. 34 are the same as those described with reference to FIGS.

  By the way, when the wiper is operated or when raindrops are wiped by wiping the window glass, the amount of raindrops rapidly decreases, and the sum of the pixel values also correspondingly decreases accordingly. FIG. 37 is a graph simulating a case where the amount of raindrops adhering to the windshield is drastically decreased, and shows temporal changes in the output pixel value and the weighted average value when the sum of pixel values becomes zero. Yes.

  In FIG. 37, the output pixel value KOS and the weighted average value KJ do not immediately become zero even though the sum KS of pixel values becomes zero at time t = 11. At time t = 15, the total pixel value KS is not zero, but the output pixel value KOS is zero. This is because the weighted average value and the output pixel value are calculated with reference to the sum of the past pixel values.

  In this case, at time t = 11 to 13, the output pixel value KOS outputs that there is raindrops (rainfall state) even though the sum KS of pixel values is zero and there are no raindrops (not rainy). At time t = 15-17, the reverse output is performed. Then, problems such as the wiper operating when not intended can occur.

  This problem can be solved by resetting the output pixel value KOS when the raindrop amount suddenly decreases. The reset means, for example, a clamping process for clamping the output pixel value KOS to a predetermined value. 38 and 39 are examples of flowcharts for explaining the clamping process. Note that, in FIG. 38, the same processing portions as those in FIG. 34 are denoted by the same reference numerals, and description thereof is omitted.

  The flowchart shown in FIG. 38 is obtained by adding step S701 between step S206 and step S207 of the flowchart shown in FIG. Specific processing in step S701 in FIG. 38 is shown in steps S801 to S803 in FIG. First, in step S801, the raindrop amount information calculation unit 60 determines whether or not raindrops have been removed. For example, when the CPU 61 shown in FIG. 11 transmits a signal or a command for operating the wiper 360 to the wiper control unit 320 shown in FIG. 1, it is determined that raindrops have been wiped.

  Next, when it is determined in step S801 that raindrops have been removed (in the case of YES), in step S802, the raindrop amount information calculation unit 60 resets the moving average process for a predetermined time. For example, the output of the moving average process is held at zero for a predetermined time. The predetermined time is preferably equal to or longer than the number of taps (the number of multipliers). This is because the influence of the past time (the influence of times t = 9 and 10 in FIG. 40 described later) is reset. In the example of FIG. 19, the number of taps is 3.

  Next, in step S803, the raindrop amount information calculation unit 60 resets the weighted average process. For example, the output of the weighted average process is held at zero. The processing before step S701 in FIG. 38 and the subsequent processing are as described in FIG. If it is determined in step S801 that raindrops have not been removed (NO), the processes in steps S802 and S803 are not executed.

By executing the clamping process shown in FIGS. 38 and 39, the graph of FIG. 37 changes as shown in FIG. In FIG. 40, the total pixel value KS becomes zero at time t = 11, and the output pixel value KOS and the weighted average value KJ also become zero following this. From time t = 13, the output pixel value KOS rises following the sum KS of pixel values. As a result, problems such as the wiper operating when not intended can be solved.

The program corresponding to the processing by the raindrop amount information calculating means 60 shown in FIGS. 34, 35, 38 and 39 can be stored in, for example, the ROM 62 of the raindrop amount information calculating means 60 shown in FIG. It can be executed by the CPU 61. Alternatively, the raindrop amount information calculation means 60 is provided in the image analysis unit 50, and the processing by the raindrop amount information calculation means 60 is performed as a program of the CPU 51 of the image analysis unit 50, as the hardware logic 54, or a mixed form thereof. May be executed as

  Alternatively, the program may be provided by being recorded in a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD in an installable or executable format file. It may be configured. Alternatively, the program may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program may be configured to be provided or distributed via a network such as the Internet.

  In this way, by introducing the weighted average difference process for the sum of pixel values, it is possible to detect raindrops with reduced noise influence (S / N improved). In addition, by executing the moving average process on the sum of pixel values (output pixel values) after the weighted average difference process, it is possible to remove the noise component that fluctuates with the time of the raindrop information and perform stable raindrop detection. .

  Further, the sum of the pixel values after the weighted average difference processing (output pixel value) after executing the moving average processing and the sum of the pixel values after the weighted average difference processing (output pixel value) without executing the moving average processing , Select according to the amount of rainfall. As a result, it is possible to detect raindrops without reducing the reaction speed even for a sudden change (splash) in raindrop information.

  The preferred embodiment has been described in detail above. However, the present invention is not limited to the above-described embodiment, and various modifications and replacements are made to the above-described embodiment without departing from the scope described in the claims. Can be added.

  For example, in the above embodiment, the raindrop information is calculated based on the sum of the pixel values of each reference area. However, the raindrop information may be calculated based on the average value of the pixel values of each reference area. . Also in this case, the same effect as the above embodiment can be obtained.

  In the above embodiment, when the sum of the pixel values of the reference area exceeds the threshold Th0, it is determined that raindrops are attached to the windshield in the reference area. However, it goes without saying that when the sum of the pixel values of the reference area is equal to or greater than the threshold Th0, it may be determined that raindrops are attached to the windshield in the reference area.

DESCRIPTION OF SYMBOLS 10, 10A Raindrop detection apparatus 20 Imaging unit 30 Light source 40 Imaging apparatus 41 Imaging lens 42 Optical filter 42a Substrate 42b Spectral filter layer 42c Polarization filter layer 42d SOG (Spin On Glass) layer 42e Infrared light transmission filter 43 Image sensor 44 Sensor substrate 45 Signal processor 50 Image analysis unit 51, 61 CPU
52, 62 ROM
53, 63 RAM
54, 64 Hardware logic 55 Digital signal processor 56, 66a, 66b I / O ports 57, 67a, 67b Communication interface 60 Raindrop amount information calculation means 65 Light guide member 71 Moving average processing means 72 Differential calculation means 73 Comparison means 74 Selection Means 75 Weighted average difference processing means 101 windshield 102 raindrop detection area 102 _{1 to} 102 _P reference area 103 raindrop 104 cover 111 image area for vehicle detection 112, 113 image area for raindrop detection 115 preceding vehicle 116 white line 117 road surface 118 sky 300 movement Body equipment control system 310 Head lamp control unit 320 Wiper control unit 330 Vehicle travel control unit 350 Head lamp 360 Wiper 400 Own vehicle

Japanese Patent Laid-Open No. 10-148618 JP-A-10-261064

Claims (10)

An image processing apparatus that detects an adhering substance adhering to the transparent body based on an image obtained by an imaging unit that performs imaging through a transparent body,
At each predetermined time, the region-specific deposit information is calculated for each of the plurality of regions in the image, and each region-specific deposit information is compared with a predetermined threshold value, and the predetermined threshold value is exceeded. A calculation means for calculating the deposit information at a predetermined time by adding the region-specific deposit information of the region ;
An image processing apparatus comprising: averaging processing means for performing average processing on the deposit information at a predetermined time and the deposit information before the predetermined time to calculate average deposit information.   The image processing apparatus according to claim 1, further comprising a selection unit that selects and outputs either the deposit information at the predetermined time or the average deposit information.   The image processing apparatus according to claim 1, further comprising a comparison unit that compares at least one of the deposit information and the average deposit information at the predetermined time with a plurality of threshold values.   The image processing apparatus according to claim 1, wherein the averaging process is a moving average process or a process using an IIR filter. An image processing apparatus that detects an adhering substance adhering to the transparent body based on an image obtained by an imaging unit that performs imaging through a transparent body,
At each predetermined time, the region-specific deposit information is calculated for each of the plurality of regions in the image, and each region-specific deposit information is compared with a predetermined threshold value, and the predetermined threshold value is exceeded. A calculating means for calculating the total deposit information by adding the region-specific deposit information of the region ;
An image processing apparatus comprising: weighted average processing means for performing a weighted average process on the combined deposit information at a predetermined time and the combined deposit information in a predetermined time range to calculate a weighted average value.   The image processing apparatus according to claim 5, further comprising a weighted average difference processing unit that calculates a weighted average difference value obtained by subtracting the weighted average value from the combined deposit information at a predetermined time.   The image processing apparatus according to claim 5, wherein the combined deposit information in the predetermined time range is combined deposit information before the predetermined time.   A vehicle comprising the image processing device according to claim 1, the imaging unit, and a windshield as the transparent body. A program for causing a computer to detect an adhering substance adhering to the transparent body based on an image obtained by an imaging means for imaging through the transparent body,
At each predetermined time, the region-specific deposit information is calculated for each of the plurality of regions in the image, and each region-specific deposit information is compared with a predetermined threshold, and the predetermined threshold is exceeded. The deposit information at a predetermined time is calculated by adding the region-specific deposit information of the region ,
A program for calculating average deposit information by performing an averaging process on deposit information at a predetermined time and deposit information before the predetermined time. A program for causing a computer to detect an adhering substance adhering to the transparent body based on an image obtained by an imaging means for imaging through the transparent body,
At each predetermined time, the region-specific deposit information is calculated for each of the plurality of regions in the image, and each region-specific deposit information is compared with a predetermined threshold, and the predetermined threshold is exceeded. The total deposit information is calculated by adding the region-specific deposit information of the region ,
A program for calculating a weighted average value by performing a weighted average process on the combined deposit information at a predetermined time and the combined deposit information in a predetermined time range.

Images