JP2014232026A - Attachment detection device, moving body apparatus control system, and moving body - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress a reduction in detection accuracy of attachments due to the saturation state of a light receiving unit.SOLUTION: An attachment detection device specifies, from a photographed image obtained through the irradiation with light from a light irradiation unit 202, a photographed image part corresponding to a light receiving unit in an imaging unit 201 receiving light with the intensity equal to or higher than a predetermined threshold, as an unused image part, and detects, on the basis of two photographed images photographed in photographing periods different from each other, an image area having a difference in pixel value equal to or more than a prescribed value between both photographed images, as an attachment image area, within a photographed image part excluding the specified unused image part as a processing object.

Description

  The present invention relates to an adhering matter detection device that detects an adhering matter adhering to a light transmitting member based on a captured image obtained by imaging a light transmitting member such as glass irradiated with light from a light source with an imaging means. The present invention relates to a mobile device control system and a mobile body.

  As this kind of deposit detection apparatus, for example, an image processing system described in Patent Document 1 is known. In this image processing system, a window glass (light transmissive member) of a vehicle or the like is irradiated with light from a light source, and reflected light from the window glass is imaged by an imaging means in which a plurality of light receiving portions are arranged in a two-dimensional direction. Then, an adhering matter such as raindrops adhering to the window glass is detected based on the captured image. In this image processing system, first, an image signal when a light source is turned on is input from an image sensor to an image processor. Then, edge detection processing is performed on the input image signal to create an edge image in which the boundary between the raindrop image region and the non-raindrop image region is emphasized. Thereafter, a circle detection process is performed on the edge image to detect a circular area, and the detected circular area is detected as a raindrop image area (attachment image area).

  Patent Document 2 discloses an image pickup apparatus having a function of detecting adhering substances such as water droplets attached to a lens cover-like light guide (light transmitting member) that covers a lens of an image pickup apparatus attached outside a vehicle such as an automobile. Is disclosed. In this imaging apparatus, an image obtained when the light source is turned on and an image obtained when the light source is turned off using an imaging means in which a plurality of light receiving units are arranged in a two-dimensional direction. A density difference image is generated. In the image pickup apparatus of Patent Document 2, the image density of the attached region where the attached matter is attached is relatively brighter (lower) than the image density of the non-attached region where the attached matter is not attached. Therefore, in the imaging device of Patent Document 2, it is determined that a pixel area having a predetermined image density or less in the density difference image is an attachment image area.

  In the case where an object is detected by imaging light from the illumination range of the light transmitting member illuminated by the light source, the light from the attachment region to which the object is attached is detected in any of the techniques disclosed in any of the above-mentioned patent documents. Using the fact that there is a difference between the amount of received light (pixel value on the captured image) and the amount of received light (pixel value on the captured image) from the non-attached area where the attached matter is not attached. Is detected. At this time, if the illuminance within the illumination range illuminated by the light source is uniform, there is a pixel value difference (contrast) greater than or equal to a specified value with respect to the non-adherent image region that projects the non-adherent region in a single captured image. It is possible to detect a certain image area as the deposit image area.

  However, normally, illuminance unevenness occurs in the illumination range on the light transmission member by light emitted from a single light source. Moreover, even when illuminating the light transmissive member using a plurality of light sources, since illuminance unevenness exists in each light source, it is difficult to eliminate the illuminance unevenness in the illumination range on the light transmissive member. Therefore, even when a single light source is used or when a plurality of light sources are used, when the light transmitting member is illuminated, illuminance unevenness appears within the illumination range. When there is such illuminance unevenness, it is distinguished whether the pixel value difference (contrast) in a single captured image is due to the difference between the adhered image area and the non-attached object image area or due to uneven illuminance. Difficult to do. As a result, it is difficult to obtain high deposit detection accuracy.

  On the other hand, as a method for detecting an adhering substance, there is a method for detecting an adhering substance by observing a pixel value difference at the same location between two captured images having different imaging times. In this method, for a location where the difference in pixel value at the same location between two captured images is greater than or equal to a specified value, the imaging timing of the previous captured image and the imaging timing of the subsequent captured image between the two captured images It can be detected that the deposit is attached at the time between. According to this method, even if illuminance unevenness exists within the illumination range of the light source, the illuminance by the light source is the same at the same location between both captured images, and therefore the influence of the illuminance unevenness on the detection accuracy of the attached matter is small. Therefore, it is possible to detect a deposit with high detection accuracy.

  However, if the light receiving intensity of the light source is too high or strong disturbance light is incident and a portion (light receiving portion) where the amount of received light is saturated appears in the image sensor, the image portion corresponding to that portion For, it is not possible to obtain an appropriate pixel value difference between two captured images. When a numerical example is given and explained, for example, when an adhering material adheres to a position (non-adhering region) corresponding to a non-adhering material image region where the received light amount (pixel value) was 2, the received light amount (pixel value) ) Is changed to 9. In this case, the difference in received light amount (pixel value difference) is calculated as 9−2 = 7, and a relatively large difference is obtained. Therefore, the image area is appropriately detected as an attached object image area. At this time, if the saturation amount (saturated pixel value) of each light receiving unit constituting the image sensor is 10, the received light amount can be appropriately detected until the received light amount (pixel value) becomes 10, In the case of the amount of received light exceeding this, this cannot be detected appropriately and is detected as 10 which is the saturation amount. Therefore, for example, when an adhering material adheres to a portion (non-adhering region) corresponding to the non-adhering material image region where the received light amount (pixel value) is 8, the received light amount (pixel value) is originally detected as 15. The place to be detected is detected as 10 which is the saturation amount. In this case, the difference in received light amount (pixel value difference) should be calculated as 15−8 = 7. However, this is calculated as 10−8 = 2, and the luminance difference is calculated smaller than the original. The As a result, the image area is not detected as a deposit image area. As described above, an appropriate pixel value difference between the two captured images cannot be obtained for the image region in which the light receiving unit constituting the imaging element is saturated, and the attached matter cannot be appropriately detected. For this reason, there is a problem that the detection accuracy of the attached matter is lowered.

  To solve this problem, for example, a countermeasure can be considered in which the light irradiation intensity of the light source is adjusted to be small so that the number of light receiving portions that are saturated is reduced. However, as described above, the light source causes unevenness in illuminance. Therefore, when the light irradiation intensity of the light source is adjusted to be small, the ratio of the low illuminance area in the illumination range is relatively increased. The area with low illuminance cannot take a large pixel value difference (difference in received light amount) between the attached image area and the non-attached object image area, and therefore the detection accuracy of the attached substance is lower than the area with high illuminance. . Therefore, if the light irradiation intensity of the light source is adjusted to be small, the ratio of the area with low attached matter detection accuracy is increased, and the attached matter detection accuracy is lowered.

  The present invention has been made in view of the above problems, and the object of the present invention is to improve the accuracy of detection of an adhering substance due to the light receiving unit becoming saturated without adjusting the light irradiation intensity of the light source to be small. An object is to provide a deposit detection device, a mobile device control system, and a deposit detection program capable of suppressing a decrease.

  In order to achieve the above object, the present invention images an adhering observation portion of a light transmissive member irradiated with light from a light source, using an imaging means in which a plurality of light receiving portions are arranged in a two-dimensional direction. In the attached matter detection apparatus provided with the detection processing means for executing the attached matter detection process for detecting the attached matter attached to the attached matter observation portion from the captured image obtained in the above, when the light is irradiated from the light source Non-use image part specifying means for specifying as a non-use image part a picked-up image part corresponding to a light receiving part that receives a received light amount equal to or greater than a predetermined threshold in the image pickup means from among the picked-up image observation part. And the detection processing means uses the captured image portion excluding the unused image portion specified by the unused image portion specifying means as the target of the attached matter detection processing, and the attached matter observation portion by the imaging means. And detecting a deposit based on the pixel value difference between two captured images captured in different imaging times are.

  According to the present invention, it is possible to reduce the erroneous detection of the adhering matter for the captured image portion corresponding to the light receiving unit that is saturated, so that the light receiving unit is saturated without adjusting the light irradiation intensity of the light source to be small. The outstanding effect that the fall of the detection accuracy of the deposit | attachment by this can be suppressed is acquired.

It is a mimetic diagram showing a schematic structure of an in-vehicle device control system in an embodiment. It is a schematic diagram which shows an example of schematic structure of the imaging unit and image analysis unit which comprise the raindrop detection apparatus in the same vehicle equipment control system. It is a schematic diagram which shows the other example of schematic structure of the imaging unit and image analysis unit which comprise the raindrop detection apparatus in the same vehicle equipment control system. It is explanatory drawing which shows an example when an image area for raindrop detection is illuminated with one light source. It is explanatory drawing which shows an example when an image area for raindrop detection is illuminated with eight light sources. It is a graph which shows the filter characteristic of the cut filter applicable to the picked-up image data for raindrop detection. It is a graph which shows the filter characteristic of the band pass filter applicable to the picked-up image data for raindrop detection. It is a front view of the optical filter provided in an imaging part. It is explanatory drawing which shows the image example of the captured image data of the imaging part. It is a flowchart which shows the flow of the raindrop amount calculation process in embodiment. It is a functional block diagram of a raindrop amount calculation processing unit that executes a raindrop amount calculation process in the image analysis unit. (A) shows an example of a light source image in which attention is paid to an illumination range by one light source in a situation where there is no disturbance light and no deposits. (B) shows an example of a lighting image in a situation where ambient light exists. (C) shows an example of an image at the time of extinction in a situation where disturbance light exists. (D) is explanatory drawing which illustrated having excluded the unused image part shown by hatching. (A) is explanatory drawing which showed the image at the time of lighting in the condition where disturbance light does not exist, and the received light amount distribution corresponding to this. (B) is explanatory drawing which showed the image at the time of lighting in the condition where disturbance light exists, and received light amount distribution corresponding to this. It is a flowchart which shows the flow of the saturated pixel map production | generation process in embodiment. It is a flowchart which shows the flow of the raindrop amount calculation process in a modification.

Hereinafter, an embodiment in which the deposit detection apparatus according to the present invention is used in an in-vehicle device control system that is a mobile device control system will be described.
In addition, the deposit | attachment detection apparatus which concerns on this invention is applicable not only to a vehicle equipment control system but another system.

FIG. 1 is a schematic diagram illustrating a schematic configuration of an in-vehicle device control system according to the present embodiment.
This in-vehicle device control system uses a captured image data of a forward region (imaging region) in the traveling direction of the host vehicle captured by an imaging unit mounted on the host vehicle 100 such as an automobile that is a moving body, It controls the light distribution of the headlamps and other in-vehicle devices.

  The imaging unit provided in the in-vehicle device control system of the present embodiment is provided in the imaging unit 101, and images an area in the traveling direction of the traveling vehicle 100 as an imaging area. The imaging unit 101 is installed near a room mirror (not shown) of the windshield 105 of the host vehicle 100, for example. The captured image data captured by the imaging unit of the imaging unit 101 is input to the image analysis unit 102. The image analysis unit 102 analyzes the captured image data transmitted from the imaging unit, calculates the position, direction, and distance of another vehicle existing ahead of the host vehicle 100 in the captured image data, or is a light transmitting member. For example, an adhering matter such as raindrops or foreign matters adhering to the windshield 105 is detected, or a detection target such as a white line (partition line) on the road surface existing in the imaging region is detected. In the detection of other vehicles, a preceding vehicle traveling in the same traveling direction as the own vehicle 100 is detected by identifying the tail lamp of the other vehicle, and traveling in the opposite direction to the own vehicle 100 by identifying the headlamp of the other vehicle. An oncoming vehicle is detected.

  The calculation result of the image analysis unit 102 is sent to a headlamp control unit 103 as lamp control means. For example, the headlamp control unit 103 generates a control signal for controlling the headlamp 104 that is an in-vehicle device of the host vehicle 100 from the distance data calculated by the image analysis unit 102. Specifically, for example, while avoiding that the strong light of the headlamp of the own vehicle 100 is incident on the driver of the preceding vehicle or the oncoming vehicle, the driver of the other vehicle is prevented from being dazzled. The switching of the high beam and the low beam of the headlamp 104 is controlled, and partial shading control of the headlamp 104 is performed so that the driver's visibility can be secured.

  The calculation result of the image analysis unit 102 is also sent to the wiper control unit 106 as a wiper control means. The wiper control unit 106 controls the wiper 107 to remove deposits such as raindrops and foreign matters attached to the windshield 105 of the host vehicle 100. The wiper control unit 106 receives the attached matter detection result detected by the image analysis unit 102 and generates a control signal for controlling the wiper 107. When the control signal generated by the wiper control unit 106 is sent to a wiper drive unit (not shown) of the wiper 107, the wiper 107 is activated to ensure the visibility of the driver of the host vehicle 100.

  The calculation result of the image analysis unit 102 is also sent to the vehicle travel control unit 108 as vehicle travel control means. Based on the white line detection result detected by the image analysis unit 102, the vehicle travel control unit 108 warns the driver of the host vehicle 100 when the host vehicle 100 is out of the lane area defined by the white line. Notification is performed, and driving support control such as controlling the steering wheel and brake of the host vehicle is performed.

FIG. 2 is a schematic diagram illustrating an example of a schematic configuration of the imaging unit 101 and the image analysis unit 102 that configure the raindrop detection device 200 as the attached matter detection device.
The raindrop detection apparatus 200 includes an imaging unit 201 and a light irradiation unit 202 provided in the imaging unit 101, and a raindrop amount information calculation unit 203 provided in the image analysis unit 102. In this embodiment, the light irradiation part 202 is for detecting the deposit | attachment adhering to the outer wall surface of the windshield 105 (Hereinafter, it demonstrates as an example when the deposit | attachment is raindrop Rd). .

  In the raindrop detection apparatus 200 shown in FIG. 2, the light irradiation unit 202 is arranged so that the light irradiated from the light irradiation unit 202 enters the inside from the inner wall surface of the windshield 105. Most of the light incident on the inside of the windshield 105 from the light irradiation unit 202 is transmitted through the outer wall surface of the windshield in the non-attached region where the raindrop Rd is not attached on the outer wall surface of the windshield 105. Is not received. On the other hand, most of the outer wall surface of the windshield 105 where raindrops Rd adhere is reflected by the windshield outer wall surface, and the specularly reflected light is received by the imaging unit 201. Therefore, the amount of light received by the imaging unit 201 is small from the non-attached area and large from the attached area. Therefore, on the captured image, the pixel value (luminance) of the raindrop image area that is an attachment image area that projects the adhesion area is higher than the pixel value (luminance) of the non-raindrop image area that is the non-attachment image area that projects the non-attachment area. ) Is higher.

FIG. 3 is a schematic diagram illustrating another example of a schematic configuration of the imaging unit 101 and the image analysis unit 102 that configure the raindrop detection apparatus 200.
The raindrop detection apparatus 200 includes an imaging unit 201, a light irradiation unit 202, and a light guide unit 204 provided in the imaging unit 101, and a raindrop amount information calculation unit 203 provided in the image analysis unit 102. In the raindrop detection apparatus 200 shown in FIG. 3, the light irradiated from the light irradiation unit 202 enters the light guide unit 204 configured by a prism or the like. The light guide unit 204 has a close contact surface that is in close contact with the inner wall surface of the windshield 105, and the light incident from the light irradiation unit 202 into the light guide unit 204 is in contact with the close contact surface of the light guide unit 204 and the inner wall surface of the windshield. And enters the windshield 105 through the boundary surface. Most of the light incident on the inside of the windshield 105 is reflected on the outer surface of the windshield 105 in the non-attached area where the raindrop Rd is not attached, and the specularly reflected light is reflected by the imaging unit 201. Is received. On the other hand, in the attachment region where raindrops Rd are attached on the outer wall surface of the windshield 105, most of the region passes through the outer wall surface of the windshield and is not received by the imaging unit 201. Therefore, the amount of light received by the imaging unit 201 is large from the non-attached area and small from the attached area. Therefore, on the captured image, the pixel value (luminance) of the raindrop image area that projects the attached area is lower than the pixel value (luminance) of the non-raindrop image area that projects the non-attached area.

  In the present embodiment, the raindrop detection device 200 shown in FIG. 2 or the raindrop detection device 200 shown in FIG. 3 can be similarly adopted. However, in the following description, the raindrop detection device 200 shown in FIG. 3 is taken as an example. I will give you a description.

  In the present embodiment, the focal point of the imaging lens provided in the imaging unit 201 is set between infinity or infinity and the outer wall surface of the windshield 105. Thereby, not only when detecting the raindrop Rd adhering to the windshield 105, but also when detecting the preceding vehicle, the oncoming vehicle, and the white line, appropriate information is obtained from the captured image data of the imaging unit 201. Can be acquired.

  The light irradiation unit 202 is equipped with one or more light sources. When the windshield 105 is illuminated with only one light source, an image as shown in FIG. 4 is displayed in the raindrop detection image area 214. This image shows a light source image when the windshield 105 is illuminated with one light source in a situation where there is no disturbance light and no deposits. As shown in FIG. 4, unevenness in illuminance exists within the illumination range on the windshield 105 due to light emitted from one light source. When the windshield 105 is illuminated with two or more light sources, an image as shown in FIG. 5 is displayed in the raindrop detection image area 214. This image shows a light source image in which the windshield 105 is illuminated with eight light sources in a situation where no disturbance light and adhering matter exist. As shown in FIG. 5, illuminance unevenness also exists in the illumination range on the windshield 105 by the light emitted from the eight light sources.

  As a light source provided in the light irradiation part 202, LED (light emitting diode) and LD (laser diode) can be used suitably. In the present embodiment, since a small LED is used as a light source in terms of cost, power consumption, and space saving, a light irradiation unit 202 using a plurality of LEDs is employed to ensure a windshield illumination range. . Therefore, in the present embodiment, the light source image appearing on the raindrop detection image area 214 is as shown in FIG. Note that the light irradiation unit 202 may be provided with a collimating lens or the like that collimates the light rays in order to reduce light divergence and increase the light efficiency within the illumination range.

  As the emission wavelength of the light irradiation unit 202, for example, visible light or infrared light can be used. However, in order to avoid dazzling the driver or pedestrian of the oncoming vehicle with the light of the light irradiation unit 202, the wavelength is longer than the visible light and the light receiving sensitivity of the image sensor reaches, for example, 800 nm. The infrared light region of 1000 nm or less is preferable. The light irradiation unit 202 of the present embodiment irradiates light having a wavelength in the infrared light region.

  In particular, in order to reduce the influence of light from the outside such as direct sunlight, it is effective to select a wavelength centered on 950 nm. Here, when imaging the infrared wavelength light from the light irradiation unit 202 reflected by the outer wall surface of the windshield 105 with the imaging unit 201, the image sensor of the imaging unit 201 uses the infrared wavelength light from the light irradiation unit 202. In addition, a large amount of disturbance light including infrared wavelength light such as sunlight is also received. Therefore, in order to distinguish the infrared wavelength light from the light irradiation unit 202 from such a large amount of disturbance light, the light emission amount of the light irradiation unit 202 needs to be sufficiently larger than the disturbance light. It is often difficult to use such a light emitting unit 202 with a large light emission amount.

  Therefore, in the present embodiment, for example, a cut filter that cuts light having a wavelength shorter than the light emission wavelength of the light irradiation unit 202 as shown in FIG. 6, or a transmittance peak as shown in FIG. Is configured such that the light from the light irradiation unit 202 is received by the image sensor through a bandpass filter that substantially matches the emission wavelength of the light irradiation unit 202. As a result, light other than the emission wavelength of the light irradiation unit 202 can be removed and received, so that the amount of light from the light irradiation unit 202 received by the image sensor is relatively large with respect to disturbance light. As a result, it is possible to distinguish the light from the light irradiation unit 202 from the disturbance light even if the light irradiation unit 202 does not have a large light emission amount.

  However, in the present embodiment, not only the raindrop Rd on the windshield 105 is detected from the captured image data, but also the preceding vehicle and the oncoming vehicle and the white line are detected. Therefore, if the wavelength band other than the infrared wavelength light irradiated by the light irradiation unit 202 is removed from the entire captured image, light in the wavelength band necessary for detection of the preceding vehicle and the oncoming vehicle and detection of the white line is detected by the image sensor. The light cannot be received, and this hinders detection. Therefore, in the present embodiment, the image area of the captured image data, the image area for raindrop detection for detecting the raindrop Rd on the windshield 105, and the vehicle for detecting the preceding vehicle and the oncoming vehicle and detecting the white line. An optical filter is used that is divided into detection image areas and removes a wavelength band other than the infrared wavelength light irradiated by the light irradiation unit 202 only on a portion corresponding to the raindrop detection image area.

FIG. 8 is a front view showing an example of the optical filter.
FIG. 9 is an explanatory diagram illustrating an example of captured image data.
As shown in FIG. 9, the optical filter 210 of the present embodiment includes an infrared light cut filter region 211 disposed at a position corresponding to the upper image 2/3 of the captured image, which is the vehicle detection image region 213, and raindrop detection. The region is divided into an infrared light transmission filter region 212 disposed at a position corresponding to the lower third of the captured image that is the image region 214. For the infrared light transmission filter region 212, the cut filter shown in FIG. 6 or the band-pass filter shown in FIG. 7 is used.

  The headlamp of the oncoming vehicle, the tail lamp of the preceding vehicle, and the image of the road edge and white line are often present mainly at the center of the captured image, and the image of the closest road surface in front of the host vehicle is present at the bottom of the captured image. It is normal. Therefore, the information necessary for identifying the headlamp of the oncoming vehicle, the tail lamp of the preceding vehicle, the road edge, and the white line is concentrated in the center of the captured image, and the information below the captured image is not so important in the identification. Therefore, when both detection of an oncoming vehicle, a preceding vehicle, or a road edge or white line and raindrop detection are performed simultaneously from a single captured image data, the lower part of the captured image is used for raindrop detection as shown in FIG. It is preferable that the image area 214 is formed, and the upper part of the remaining captured image including the central portion of the captured image is the vehicle detection image area 213, and the optical filter 210 is divided into areas corresponding thereto.

  The information above the captured image usually has an image of the sky in front of the host vehicle. Similar to the information at the bottom of the captured image, the headlamp of the oncoming vehicle, the tail lamp of the preceding vehicle, and the road edge and white line are identified. Is not very important. Therefore, the raindrop detection image area 214 may be provided in the upper part of the captured image, or may be provided in both the upper part and the lower part of the captured image.

  In the present embodiment, the bonnet of the host vehicle may enter the lower part of the imaging area. In this case, sunlight reflected by the bonnet of the host vehicle, tail lamps of the preceding vehicle, etc. become disturbance light, and this is included in the captured image data, thereby causing raindrop detection accuracy to deteriorate. Even in such a case, in this embodiment, the cut filter shown in FIG. 6 and the bandpass filter shown in FIG. Disturbance light such as a tail lamp of a vehicle is removed without being included in the raindrop detection image region 214, and deterioration of raindrop detection accuracy is suppressed.

  In addition, an infrared light cut filter region 211 is disposed at a location corresponding to the vehicle detection image region 213 in the optical filter 210 of the present embodiment. Since the filter region at this point only needs to be able to transmit visible light, it may be a non-filter region that transmits the entire wavelength band. However, even if infrared wavelength light from the light irradiation unit 202 is incident, noise due to this is reduced. Thus, it is desirable to be able to cut infrared wavelengths.

  Here, when the preceding vehicle is detected, the preceding vehicle is detected by identifying the tail lamp on the captured image. However, the tail lamp has a smaller amount of light compared to the headlamp of the oncoming vehicle, and disturbance such as street lights. Since there is a lot of light, it is difficult to detect the tail lamp with high accuracy only from mere luminance data. Therefore, spectral information may be used to identify the tail lamp, and the tail lamp may be identified based on the amount of received red light.

  The light from the imaging region passes through the imaging lens of the imaging unit 201, passes through the optical filter 210, and is converted into an electrical signal corresponding to the light intensity by the image sensor. The electrical signal (analog signal) output from the image sensor is converted into a digital signal indicating the brightness (luminance) of each pixel on the image sensor, and the image analysis unit 102 as captured image data together with the horizontal / vertical synchronization signal of the image. Is output.

  The image sensor is an image sensor using a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, and a photodiode is used as a light receiving element (light receiving portion). The photodiodes are arranged in a two-dimensional direction for each pixel, and a microlens is provided on the incident side of each photodiode in order to increase the light collection efficiency of the photodiode. This image sensor is bonded to a printed wiring board (PWB) by a method such as wire bonding to form a sensor substrate.

Next, a raindrop amount calculation process that is an attached matter detection process in the present embodiment will be described.
FIG. 10 is a flowchart showing the flow of raindrop amount calculation processing in the present embodiment.
FIG. 11 is a functional block diagram of the raindrop amount information calculation unit 203 that executes the raindrop amount calculation processing in the image analysis unit 102.
When the predetermined raindrop amount detection timing arrives, first, the light irradiation unit 202 is turned on (S1), and the imaging unit 201 captures a lighting image (illuminated captured image) (S2). As a result, the captured image (lighted image) of the deposit observation part (raindrop observation part) of the windshield 105 irradiated with light from the light irradiation unit 202 is displayed in the raindrop detection image area 214 below the captured image. Data is stored in the image storage memory 301 in synchronization with the horizontal synchronization signal. Subsequently, the light irradiation unit 202 is turned off (S3), and an image at the time of extinction (a non-illuminated captured image) is captured by the imaging unit (S4). Thereby, the data of the picked-up image (image when extinguished) in which the raindrop observation portion of the windshield 105 in a state where no light is irradiated from the light irradiation unit 202 is displayed in the raindrop detection image region 214 below the picked-up image is horizontal. It is stored in the image storage memory 301 in synchronization with the synchronization signal. It is preferable that the time when the image is turned on and the time when the image is turned off are closer to each other. Therefore, in the present embodiment, images that are captured in two consecutive frames are used as the on-image and the off-image.

  Note that the vertical synchronization signal input to the raindrop amount information calculation unit 203 is used for selecting a lighting image and a lighting image, permission for writing to the image storage memory 301, and the like. In addition, when the number of pixels of the on-image or off-image is large, it may be thinned out as appropriate and stored in the image storage memory 301.

  Next, the difference image generation unit 303 reads out a saturated pixel map generated by a saturation pixel map generation process, which will be described later, from the memory 305, and identifies unused pixels that are not used for calculating the raindrop amount (S5). Although not described in detail later, this unused pixel corresponds to a light receiving element (saturated light receiving element) that has a high possibility of receiving a light receiving amount exceeding the saturation amount among light receiving elements constituting the image sensor of the imaging unit 201. It is a pixel on a captured image. In the present embodiment, in order to exclude such unused pixels from the calculation of the amount of raindrops, the difference image generating unit 303 performs processing on the processing target area in which the unused pixels are excluded from the raindrop detection image area 214. A difference image between the on-image and the off-image is created (S6). This difference image is an image in which the difference value between the pixel value of the on-image and the pixel value of the off-image is used as the pixel value. The on-image and the off-image in the present embodiment are luminance images for a specific wavelength band (a wavelength band including the emission wavelength of the light irradiation unit 202), and the difference image is a luminance difference image obtained by taking a luminance difference. . Note that the on-image and the off-image may be special images such as a polarized image obtained by capturing only a specific polarization component.

  FIG. 12A shows an example of a light source image in which attention is paid to an illumination range by one light source in a situation where there is no disturbance light and no deposits. In this example, the center of the substantially circular illumination range has the highest illuminance, and has an illuminance distribution in which the illuminance decreases as it goes outward in the radial direction. Corresponds to a saturated pixel portion in a saturated state. The raindrop detection image area 214 imaged in a state where the light irradiation unit 202 turns on such a light source reflects both disturbance light and light from the light irradiation unit 202. For this reason, as shown in the lower diagram of FIG. 13A, if there is no disturbance light, even if there are a few saturated pixel portions exceeding the saturation value, as shown in the lower diagram of FIG. In the situation where light is present, the illuminance is raised as a whole, and the saturated pixel portion A ′ exceeding the saturation value increases. As a result, due to the influence of disturbance light, the saturated pixel portion shown in FIG. 12B becomes larger than the saturated pixel portion due to light from the light irradiation unit 202 shown in FIG.

  On the other hand, as shown in FIG. 12C, the raindrop detection image area 214 imaged in a state where the light irradiation unit 202 is turned off displays only disturbance light not including light from the light irradiation unit 202. It will be a thing. Since the lighting image captured under actual imaging conditions includes light from the light irradiation unit 202 and disturbance light (FIG. 12B), the light source image shown in FIG. An image having a pixel value obtained by adding the pixel value and the pixel value of the off-time image shown in FIG. 12C should be obtained. In this case, the pixel value of the luminance difference image obtained by calculating the luminance difference between the on-image and the off-image corresponds to the light from the light irradiation unit 202 by appropriately excluding the luminance of disturbance light. It will be a thing.

  However, since the pixel value cannot take a value exceeding the saturation value, all the pixel values of the pixels whose combined pixel value exceeds the saturation value indicate the saturation value. Therefore, if a saturated pixel portion exists in the lighting image, the luminance component of the disturbance light cannot be appropriately excluded even if the luminance difference between the pixel portion and the unlit image is calculated. More specifically, pixel values exceeding the original luminance of ambient light are excluded. Therefore, it corresponds appropriately to the light from the light irradiation unit 202. The luminance difference image does not appropriately correspond to the light from the light irradiation unit 202, and raindrop detection described later cannot be appropriately performed.

  Therefore, in the present embodiment, a saturated pixel map indicating pixels that can become saturated pixels when disturbance light within an assumed range is incident is generated by a saturated pixel map generation process described later, and is specified by the saturated pixel map. The pixel is excluded from the target of the luminance difference image creation process as an unused pixel. Thereby, the area B indicated by hatching in FIG. 12D is excluded from the target of the luminance difference image creation process as an unused pixel.

  The brightness difference image data created in this way is stored in the memory 305 (S7). Thereafter, the difference image generation unit 303 reads out the data of the brightness difference image previously stored in the memory 305 and creates a time difference image by taking the difference between the previous brightness difference image and the current brightness difference image (S8). . The difference image at this time is an image in which the difference value between the pixel value of the previous luminance difference image and the pixel value of the current luminance difference image is used as the pixel value. The time difference image created in this way is sent from the difference image generation unit 303 to the raindrop amount calculation unit 304.

  In this embodiment, the image area having a pixel value greater than or equal to the specified value in the difference image at this time is between the time when the previous luminance difference image is created and the time when the current luminance difference image is created between the two luminance difference images. It is the place where raindrops adhered at the time of. According to such a method of detecting raindrops from the pixel values of the time difference image, even if illuminance unevenness exists within the illumination range of the light source, the illuminance by the light source is the same at the same location between both images. There is little influence of illuminance unevenness on the detection accuracy of Rd. Therefore, it is possible to detect the raindrop Rd with high detection accuracy.

  In the present embodiment, as described above, of the light from the light irradiation unit 202, the light that enters the attachment region on the windshield 105 to which the raindrop Rd is attached does not enter the imaging unit 201, and the raindrop Rd. The light that has entered the non-attached area where no is attached enters the imaging unit 201. Therefore, the raindrop image that displays the raindrop Rd is displayed as a low-brightness image on the raindrop detection image area 214. Therefore, the raindrop image area that displays the raindrop Rd has a large pixel value in the time difference image.

  Therefore, in the present embodiment, the raindrop amount calculation unit 304 extracts an image region having a pixel value exceeding a predetermined specified value from the time difference image, and the extracted image region is a raindrop on which the raindrop Rd is projected. The candidate area is specified as an image area candidate area. Specifically, the raindrop amount calculation unit 304 first performs binarization processing by comparing the pixel value of the time difference image with a predetermined specified value. In this binarization process, for example, a binary image is created by assigning “1” to pixels having a pixel value equal to or greater than a predetermined specified value and assigning “0” to pixels that are not. Next, in the binarized image, when pixels assigned with “1” are close to each other, a labeling process for recognizing them as one image region is performed. Thereby, a set of a plurality of adjacent pixels having a large pixel value of the time difference image is extracted as one image region.

  Next, the raindrop image area is specified by performing shape identification processing on the candidate raindrop image area specified in this way. Since the shape of the raindrop image on the captured image is often circular, shape identification processing is performed to determine whether the candidate raindrop image region is a circular shape, and the raindrop image region is identified based on the result. Then, the result of counting the number of raindrop image areas specified in this way is calculated as the raindrop amount (S9).

  For example, the wiper control unit 106 determines that the calculation result of the raindrop amount is a predetermined condition (for example, a condition that the count value of the raindrop amount is 10 or more for each of the 20 time difference images created in succession). When the above condition is satisfied, the drive control of the wiper 107 and the discharge control of the washer liquid are performed. In the present embodiment, the portion that can be a saturated pixel, that is, the unused pixel portion B is not subjected to the raindrop Rd detection process, so that the raindrop Rd cannot be detected. Rd false detection will not occur. As a result, a decrease in the detection accuracy of the raindrop Rd due to the saturated pixels is suppressed.

Next, a saturated pixel map generation process for generating the above-described saturated pixel map will be described.
FIG. 14 is a flowchart showing the flow of the saturated pixel map generation process in this embodiment.
The saturated pixel map generation process is executed when a predetermined process start timing arrives, for example, when the power is turned on or when a certain time has passed. When the predetermined processing start timing arrives, first, the light irradiation unit 202 is turned on (S11), and an image at lighting is captured by the imaging unit 201 (S12). Thereby, for example, a lighting image including a saturated pixel portion A ′ as shown in FIG. 12B is obtained. As shown in FIG. 11, the lighting-time image data is sent to an unused pixel specifying unit 302 as an unused image portion specifying means. When the lighting-time image data for the raindrop detection image area 214 is input, the unused pixel specifying unit 302 initializes various parameters used in the saturated pixel map generation process (S13). Specifically, the width w (number of pixels in the x direction) of the raindrop detection image area 214, the height h (number of pixels in the y direction length) of the raindrop detection image area 214, past saturated pixel map data, and the like. initialize.

  Thereafter, the unused pixel specifying unit 302 acquires the brightness value P (x, y) for the first pixel coordinate (x, y) in the raindrop detection image region 214 from the data of the captured lighting image (S14). ). Then, it is determined whether or not the acquired luminance value P (x, y) is equal to or greater than a predetermined specified value (S15). At this time, if it is determined that the luminance value P (x, y) is equal to or greater than the predetermined specified value (Yes in S15), the pixel coordinate (x, y) is set to the pixel coordinate (x, y) to make the pixel an unused pixel. A flag is set (S16).

  The processing (S14 to S16) from the acquisition of the luminance value P (x, y) to the setting of the flag is sequentially performed for each pixel in the raindrop detection image area 214 in a predetermined order from the first pixel coordinate. Is called. When the processing is finished up to the final pixel coordinates (w, h) in the raindrop detection image area 214 (Yes in S17), the unused pixel specifying unit 302 creates a saturated pixel map indicating the pixel coordinates for which the flag is set. Data is stored in the memory 305 (S18).

  In this embodiment, pixels that are saturated pixels in the lighting image captured during the saturation pixel map generation process are used as unused pixels, but the unused pixels match the saturated pixels in the lighting image. There is no need. For example, a range larger than the saturated pixel portion in the lighting image may be used as an unused pixel, or a range narrower than the saturated pixel portion in the lighting image may be used as an unused pixel. However, even when a range narrower than the saturated pixel portion in the lighting image is used as a non-use pixel, the saturated pixel portion that becomes a saturated pixel when the light irradiation unit 202 is lit in a situation where ambient light does not exist Is preferably an unused pixel. This is because such a pixel is a saturated pixel regardless of the intensity of disturbance light.

[Modification]
Next, a modification of the raindrop amount calculation process in the embodiment will be described.
In the raindrop amount calculation process, it is only necessary to be able to calculate the amount of raindrops, and information on the location of raindrops is not used. In this modification, instead of the luminance difference image data, a simple luminance total difference value obtained by subtracting the total pixel value (luminance value) of the unlit image from the total pixel value (luminance value) of the lit image is used. This is a raindrop amount calculation process.

FIG. 15 is a flowchart showing a flow of raindrop amount calculation processing in the present modification.
When the predetermined raindrop amount detection timing arrives, first, the light irradiation unit 202 is turned on (S21), and an image at the time of lighting is picked up by the image pickup unit (S22), and generated by a saturated pixel map generation process performed in advance. The obtained saturated pixel map is acquired (S23). Thereafter, the brightness value P (x, y) for the first pixel coordinates (x, y) in the raindrop detection image area 214 is acquired from the data of the captured lighting image (S24). Then, referring to the saturated pixel map, whether or not a flag is set in the pixel coordinates (x, y) of the acquired luminance value P (x, y), that is, corresponding to the pixel coordinates (x, y). It is determined whether or not the value of the flag F (x, y) is 1 (S25).

  In this determination, when the flag F (x, y) is 1 (Yes in S25), it is determined that the pixel is an unused pixel, and the luminance value P (x, y) of the pixel coordinate (x, y) is determined. y) is not used. On the other hand, when the flag F (x, y) is zero (No in S25), the luminance value P (x, y) of the pixel coordinate (x, y) is converted into Sum data that becomes the luminance sum difference value. Processing for addition is performed (S26). The process (S24 to S26) from the acquisition of the luminance value P (x, y) to the process of adding to the Sum data (S24 to S26) starts from the first pixel coordinates for each pixel in the raindrop detection image area 214 in the lighting image. It is sequentially performed in a predetermined order. The Sum data obtained at this time indicates the total sum of all pixel values excluding unused pixels (pixels that can be saturated pixels) in the raindrop detection image area 214 of the lighting image.

  When the process is completed up to the final pixel coordinate (w, h) in the raindrop detection image area 214 for the lighting image (Yes in S27), the light irradiation unit 202 is then turned off (S28), and the imaging unit turns off the light. A time image is captured (S29). Then, the brightness value P (x, y) for the first pixel coordinate (x, y) in the raindrop detection image area 214 is acquired from the data of the captured unlit image (S30). Thereafter, referring to the saturated pixel map, whether or not a flag is set in the pixel coordinates (x, y) of the acquired luminance value P (x, y), that is, corresponding to the pixel coordinates (x, y). It is determined whether or not the value of the flag F (x, y) is 1 (S31).

  In this determination, when the flag F (x, y) is 1 (Yes in S31), it is determined that the pixel is an unused pixel, and the luminance value P (x, y) of the pixel coordinate (x, y) is determined. y) is not used. On the other hand, when the flag F (x, y) is zero (No in S31), the luminance value P (x, y) of the pixel coordinate (x, y) is subtracted from the Sum data ( S32). The processing (S30 to S32) from when the luminance value P (x, y) is acquired to when the sum data is subtracted (S30 to S32) is the first pixel coordinate for each pixel in the raindrop detection image area 214 in the unlit image. Are sequentially performed in a predetermined order. As a result, the obtained Sum data is obtained by removing unused pixels in the raindrop detection image area 214 of the unlit image from the sum of all pixel values excluding unused pixels in the raindrop detection image area 214 of the lighting image. This indicates a luminance sum difference value obtained by subtracting the sum of all pixel values.

  The Sum data calculated in this way, that is, the luminance sum total difference value Sum is corrected in accordance with the number of unused pixels, which will be described later, and then stored in the memory 305 (S32). Then, a difference value is calculated when the difference between the current corrected luminance total difference value Sum and the previous corrected luminance total difference value Sum is taken (S33). The difference value at this time shows a high correlation with the image area of the raindrops attached at the time between the previous time and the current time, and the amount of raindrops is calculated from the difference value at this time (S34).

  In the present modification, the number of unused pixels (pixels that can be saturated pixels) in the raindrop detection image region 214 is not a fixed value, and thus the number of unused pixels may vary. In this case, even if the amount of raindrops is the same, the value of the sum total difference value Sum changes, so when calculating the amount of raindrops using a common correlation even if the number of unused pixels varies. The luminance sum total difference value Sum is preferably corrected according to the number of unused pixels. Therefore, in this modification, in the processing step S32, the calculated luminance sum difference value Sum is corrected according to the number of unused pixels. Specifically, for example, for the calculated luminance sum difference value Sum, (total number of pixels in the raindrop detection image area 214) / {(total number of pixels in the raindrop detection image area 214) − (number of unused pixels) )} Multiplied by the correction coefficient is stored in the memory 305 as the corrected luminance sum difference value Sum. Thereby, even if the number of unused pixels fluctuates, it is possible to calculate the raindrop amount with high accuracy using the common correlation.

  According to the present modification, the data stored in the memory 305 is numerical data called the luminance sum total difference value Sum, so that the memory capacity can be significantly reduced compared to the case of the embodiment in which the luminance difference image data is stored in the memory 305. .

In the present embodiment, the order of capturing the lighting-time image and the lighting-time image has been described with respect to the example in which the lighting-time image is first, but the same applies to the case where the lighting-time image is captured first. .
Further, in the present embodiment, in order to suppress the influence of disturbance light, the case where the raindrop amount calculation processing is performed using the luminance difference image obtained by calculating the luminance difference between the lighting image and the lighting image is described. The raindrop amount calculation process may be performed using only the time image. In this case, the amount of raindrops can be calculated from the time difference images or the time difference values between the lighting images by the same method as in the above-described embodiment and modification. Even in this case, the amount of raindrops can be appropriately calculated in a situation where there is little change in ambient light.

What has been described above is merely an example, and the present invention has a specific effect for each of the following modes.
(Aspect A)
A windshield 105 on which light from a light source such as a light irradiation unit 202 is irradiated using an imaging unit such as an imaging unit 201 provided with an image sensor in which light receiving units such as a plurality of light receiving elements are arranged in a two-dimensional direction. The attached matter detection process such as the raindrop amount calculation process for detecting the attached matter such as the raindrop Rd adhering to the adhered observation part from the captured image obtained by imaging the attached observation part of the light transmitting member such as In the adhering matter detection device such as the raindrop detection device 200 having detection processing means such as the difference image generation unit 303 and the raindrop amount calculation unit 304 to be executed, the captured image of the adhering observation portion when light is emitted from the light source An unused image such as an unused pixel specifying unit 302 that specifies a captured image portion corresponding to a light receiving unit that receives a received light amount equal to or greater than a predetermined threshold, such as a saturation value, in the imaging unit. portion And the detection processing unit sets the captured image portion excluding the unused image portion specified by the unused image portion specifying unit as a target of the attached matter detection processing, and the attached matter by the imaging unit. It is characterized in that an adhering matter is detected based on a pixel value difference (a pixel value or a time difference value of a time difference image) between two captured images such as two luminance difference images obtained by imaging the observation part at different imaging times. To do.
According to this, the captured image part corresponding to the light receiving part that receives a light receiving amount equal to or greater than a predetermined threshold is set as an unused image part, and the adhered object is detected using the captured image part excluding this part. Accordingly, by appropriately setting the predetermined threshold value, it is possible to reduce false detection of attached matter in the captured image portion corresponding to the light receiving unit that is saturated without adjusting the light irradiation intensity of the light source to be small. Therefore, the fall of the detection accuracy of the deposit | attachment by a light-receiving part becoming saturated can be suppressed. In addition, since it is not necessary to adjust the light irradiation intensity of the light source to be small, the adhesion detection accuracy does not decrease due to the increase in the ratio of the low illuminance area in the illumination range by the light source.

(Aspect B)
In the aspect A, the two captured images are respectively before and after the illumination-time captured image such as a lighting-up image obtained by imaging the adhering portion observed when light is irradiated from the light source and the imaging timing of the illumination-captured image. Thus, the image is a difference image (luminance difference image) from a non-illuminated captured image such as an unlit image obtained by capturing an adhering observation portion when no light is irradiated from the light source.
According to this, it is possible to detect an attached substance that is less affected by disturbance light.

(Aspect C)
In the aspect A or B, the two captured images are captured through an optical filter such as an infrared light transmission filter region 212 that selectively transmits light emitted from the light source. And
According to this, it is possible to detect an adhering substance with little influence of disturbance light.

(Aspect D)
Based on the detection result of the adhering matter detecting means for detecting adhering matter such as raindrops Rd adhering to the adhering observation portion of the light transmitting member such as the windshield 105 in the moving body such as the own vehicle 100, etc. In a mobile device control system comprising a mobile device control means such as a wiper control unit 106 for controlling a predetermined device such as a wiper 107 mounted on the mobile body, the attached matter detection means is the mode A A deposit detection apparatus according to any one of the aspects C to C is used.
According to this, since the fall of the detection accuracy of the deposit | attachment by a light-receiving part becoming a saturated state is suppressed, the predetermined | prescribed apparatus mounted in the mobile body can be controlled appropriately.

(Aspect E)
A moving body control system according to the aspect D is used as a means for controlling the predetermined apparatus in a moving body such as the own vehicle 100 that moves by mounting the predetermined apparatus such as the wiper 107. .
According to this, since the fall of the detection accuracy of the deposit | attachment by a light-receiving part becoming a saturated state is suppressed, the mobile body which can control a predetermined | prescribed apparatus appropriately can be provided.

DESCRIPTION OF SYMBOLS 100 Own vehicle 101 Imaging unit 102 Image analysis unit 105 Windshield 106 Wiper control unit 107 Wiper 200 Raindrop detection apparatus 201 Imaging part 202 Light irradiation part 203 Raindrop amount information calculation part 204 Light guide part 210 Optical filter 211 Infrared light cut filter area | region 212 Infrared light transmission filter region 213 Vehicle detection image region 214 Raindrop detection image region 301 Image storage memory 302 Unused pixel specifying unit 303 Difference image generation unit 304 Raindrop amount calculation unit 305 Memory

JP-A-2005-195 566 JP 2008-64630 A

Claims (5)

Using an imaging means in which a plurality of light receiving parts are arranged in a two-dimensional direction, from the captured image obtained by imaging the adhered observation part of the light transmitting member irradiated with light from the light source, the adhered observation part In the adhering matter detection apparatus provided with the detection processing means for executing the adhering matter detection process for detecting the adhering matter adhering to the
Among the captured images of the adhering matter observation portion when light is irradiated from the light source, the captured image portion corresponding to the light receiving portion that receives the received light amount equal to or larger than the predetermined threshold in the imaging means is set as an unused image portion. A non-use image part specifying means for specifying,
The detection processing means takes the imaged image portion excluding the unused image portion specified by the unused image portion specifying means as the target for the attached matter detection processing, and the imaging means picks up the different attachment observation portions from each other. An adhering matter detection apparatus that detects an adhering matter based on a pixel value difference between two captured images taken at a time. The deposit detection apparatus according to claim 1,
The two picked-up images are respectively irradiated with light from the light source before and after the picked-up image of the image of the observed portion of the deposit when the light is irradiated from the light source A deposit detection apparatus, which is a difference image from a non-illuminated captured image obtained by imaging a deposit observation portion when there is not. In the deposit detection apparatus according to claim 1 or 2,
The two picked-up images are picked-up through an optical filter that selectively transmits light emitted from the light source. An adhering matter detecting means for detecting an adhering matter adhering to the adhering matter observation portion of the light transmitting member in the moving body;
In a mobile device control system comprising a mobile device control means for controlling a predetermined device mounted on the mobile based on the detection result of the attached matter detection means,
A mobile device control system using the deposit detection device according to claim 1 as the deposit detection means. In a moving body that carries predetermined equipment and moves,
A moving body using the moving body apparatus control system according to claim 4 as means for controlling the predetermined apparatus.