JP6817104B2 - Adhesion detection device, deposit detection method - Google Patents

Description

The present invention relates to a deposit detection device and a deposit detection method.

Conventionally, for example, there is an deposit detection device that detects water droplets such as raindrops adhering to the lens of a camera attached to a vehicle. Such a deposit detecting device extracts the edge of the deposit from the captured image and detects water droplets adhering to the lens (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2010-14494

However, in the prior art, for example, when light is reflected by the attached water droplet or the shape of the water droplet may reduce the detection accuracy of the water droplet, there is room for further improvement in improving the detection accuracy of the water droplet. there were.

The present invention has been made in view of the above, and an object of the present invention is to provide a deposit detection device and a deposit detection method capable of improving the detection accuracy of water droplets.

In order to solve the above-mentioned problems and achieve the object, the present invention includes an extraction unit, a conversion unit, a matching unit, and a detection unit in the deposit detection device. The extraction unit extracts the edge information of each pixel included in the captured image captured by the imaging device. The conversion unit converts each pixel into a predetermined data format based on the edge information extracted by the extraction unit. The matching unit performs matching processing between each pixel converted into the data format by the conversion unit and a template of the data format indicating water droplets. The detection unit detects water droplets adhering to the image pickup apparatus based on the matching result of the matching unit.

According to the deposit detection device according to the present invention, the accuracy of detecting water droplets can be improved.

FIG. 1 is a diagram showing an outline of a deposit detection method. FIG. 2 is a block diagram of the deposit detection device according to the first embodiment. FIG. 3 is a diagram showing a binarization threshold. FIG. 4 is a diagram showing an example of a template according to the first embodiment. FIG. 5 is a diagram showing scanning positions of the template. FIG. 6 is a diagram showing an example of matching processing by the matching unit. FIG. 7A is a diagram (No. 1) showing the detection process by the detection unit. FIG. 7B is a diagram (No. 2) showing the detection process by the detection unit. FIG. 7C is a diagram (No. 3) showing the detection process by the detection unit. FIG. 7D is a diagram (No. 4) showing the detection process by the detection unit. FIG. 8 is a flowchart showing a processing procedure executed by the deposit detection device according to the first embodiment. FIG. 9 is a block diagram of the deposit detection device according to the second embodiment. FIG. 10 is a diagram showing an extraction range. FIG. 11A is a diagram showing a vector calculation method. FIG. 11B is a diagram showing an example of parameter information. FIG. 12 is a diagram showing an example of a template according to the second embodiment. FIG. 13 is a diagram showing a detection threshold value according to the second embodiment. FIG. 14 is a flowchart showing a processing procedure executed by the deposit detection device according to the second embodiment. FIG. 15 is a block diagram of the deposit detection device according to the third embodiment. FIG. 16A is a diagram (No. 1) for explaining a method of calculating a representative value. FIG. 16B is a diagram (No. 2) for explaining a method of calculating a representative value. FIG. 17A is a diagram showing an example of a template according to the third embodiment. FIG. 17B is a diagram showing an example of matching processing by the matching unit according to the third embodiment. FIG. 18 is a diagram showing a detection process by the detection unit according to the third embodiment. FIG. 19 is a diagram showing an example of exclusion of the detection process by the detection unit according to the third embodiment. FIG. 20 is a flowchart showing a processing procedure executed by the deposit detection device according to the third embodiment. FIG. 21 is a block diagram of the deposit removing system according to the fourth embodiment. FIG. 22A is a diagram showing an example of the content of notification from the deposit detection unit. FIG. 22B is a diagram showing an example of data contents relating to the detection area included in the detection information DB. FIG. 22C is an explanatory diagram of the state of the detection area. FIG. 23A is a processing explanatory diagram (No. 1) of the inter-algorithm overlap determination unit. FIG. 23B is a processing explanatory diagram (No. 2) of the inter-algorithm overlap determination unit. FIG. 23C is a processing explanatory diagram (No. 3) of the inter-algorithm overlap determination unit. FIG. 23D is a processing explanatory view of the frame-to-frame overlap determination unit and the adhesion determination unit. FIG. 24A is a processing explanatory diagram (No. 1) of the frame-to-frame overlap determination unit. FIG. 24B is a processing explanatory diagram (No. 2) of the frame-to-frame overlap determination unit. FIG. 24C is a processing explanatory diagram (No. 3) of the frame-to-frame overlap determination unit. FIG. 24D is a processing explanatory view of the frame-to-frame overlap determination unit and the adhesion determination unit. FIG. 24E is a processing explanatory view of the removal necessity determination unit. FIG. 25 is a flowchart showing a processing procedure executed by the deposit removing system according to the fourth embodiment.

Hereinafter, embodiments of the deposit detection device and the deposit detection method disclosed in the present application will be described in detail with reference to the accompanying drawings. The present invention is not limited to this embodiment. Further, a case of detecting water droplets adhering to a camera, which is an imaging device, as adhering matter will be described below.

First, the outline of the deposit detection method disclosed in the present application will be described with reference to FIG. FIG. 1 is a diagram showing an outline of a deposit detection method.

As shown in the figure, in the deposit detection method, first, edge information is extracted from the camera image L (step S1). Here, the edge information indicates, for example, the gradient of the brightness of each pixel in the camera image L in the horizontal direction (X-axis direction) and the vertical direction (Y-axis direction).

After that, in the deposit detection method, each pixel in the camera image L is converted into a predetermined data format based on the edge information (step S2). Here, in the deposit detection method disclosed in the present application, the accuracy of detecting water droplets is improved by converting each pixel into a predetermined data format based on the edge information.

Specifically, each pixel is binarized based on the edge strength of each pixel in the camera image L. As a result, it is possible to reduce the influence of uneven brightness of water droplets captured in the camera image L. That is, it is possible to accurately detect water droplets that reflect light. Details of this point will be described as the first embodiment with reference to FIGS. 2 to 8.

Further, in the deposit detection method disclosed in the present application, each pixel is converted into a predetermined data format by using a parameter in which the edge orientation opposite to the edge orientation of each pixel in the camera image L is a one's complement relationship.

As a result, the difference in edge orientation of each pixel can be made clear. Therefore, it is possible to improve the recognition accuracy in the matching process. Details of this point will be described as a second embodiment with reference to FIGS. 9 to 14.

Further, in the deposit detection method disclosed in the present application, each pixel is encoded by assigning a corresponding code to the edge direction of each pixel in the camera image L.

In such a case, a matching process using a regular expression with a code string indicating a water droplet is performed. As a result, in the camera image L, for example, it is possible to extract the code string of each side of the rectangle included in the water droplet.

Further, by detecting water droplets adhering to the camera by combining the extracted code strings, it is possible to detect water droplets having an irregular shape such as water droplets that have been cut off from the camera image L. Details of this point will be described as a third embodiment with reference to FIGS. 15 to 20.

Then, in the deposit detection method, matching processing is performed between each converted pixel and the template showing the water droplet (step S3), and the water droplet attached to the camera is detected based on the matching result (step S4). In the camera image L1 shown in the figure, a mark M is attached to a portion where water droplets are detected by the deposit detection method.

As described above, in the deposit detection method disclosed in the present application, each pixel in the camera image L is converted into a predetermined data format based on the edge information, and water droplets adhered to the camera by the matching process using the template of the data format. Is detected.

Therefore, according to the deposit detection method disclosed in the present application, the accuracy of detecting water droplets can be improved.

In the deposit detection method disclosed in the present application, a template showing a partial shape of a portion of water droplets different depending on the scanning position of the matching process is used. Details of this point will be described later with reference to FIGS. 4 and 5.

Further, the detection accuracy can be further improved by detecting water droplets by combining the detection results by the deposit detection device according to the first to third embodiments. The details of this point will be described in detail in the fourth embodiment with reference to FIGS. 21 to 25.

(First Embodiment)
Next, the configuration of the deposit detection device 1 according to the first embodiment will be described. FIG. 2 is a block diagram of the deposit detection device 1 according to the first embodiment. Note that FIG. 2 also shows the camera 10 and the deposit removing device 11.

As shown in the figure, the deposit detecting device 1 is connected to, for example, the deposit removing device 11 and the four cameras 10.

Each of the four cameras 10 is equipped with an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and is attached to a position for photographing the front, rear, right side, and left side of the vehicle, respectively. Be done. It should be noted that the deposit detection device 1 may be provided for each camera 10.

The deposit removing device 11 removes water droplets adhering to the camera 10 based on the detection result of the deposit detecting device 1. The deposit removing device 11 ejects compressed air, for example, toward the camera 10 to remove water droplets. The deposit removing device 11 may, for example, eject the washer fluid toward the camera 10 or wipe the camera 10 with a camera wiper.

The deposit detection device 1 according to the first embodiment includes a control unit 2 and a storage unit 3. The control unit 2 includes an image acquisition unit 21, an extraction unit 22, a conversion unit 23, a matching unit 24, and a detection unit 25. Further, the storage unit 3 stores the binarization threshold information 31, the template information 32, and the detection information 33.

The control unit 2 includes, for example, a computer having a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), an input / output port, and various circuits.

The CPU of the computer functions as an image acquisition unit 21, an extraction unit 22, a conversion unit 23, a matching unit 24, and a detection unit 25 of the control unit 2, for example, by reading and executing a program stored in the ROM.

Further, at least one or all of the image acquisition unit 21, the extraction unit 22, the conversion unit 23, the matching unit 24, and the detection unit 25 of the control unit 2 are ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). It can also be configured with hardware such as.

Further, the storage unit 3 corresponds to, for example, a RAM or an HDD. The RAM or HDD can store the binarization threshold information 31, the template information 32, the detection information 33, and information on various programs.

The deposit detection device 1 may acquire the above-mentioned program and various information via another computer or a portable recording medium connected by a wired or wireless network.

The image acquisition unit 21 acquires a camera image from the camera 10 and converts the camera image into a grayscale image by grayscale it. Further, the image acquisition unit 21 outputs a grayscale image to the extraction unit 22.

The grayscale conversion is a process of converting each pixel in the camera image so as to be expressed in each gradation from white to black according to the brightness. The grayscale processing may be omitted.

The extraction unit 22 extracts the edge information of each pixel in the grayscale image by using a sobel filter for the grayscale image input from the image acquisition unit 21. Here, the edge information refers to the edge strength of each pixel in the X-axis direction and the Y-axis direction.

Further, the extraction unit 22 associates the extracted edge information with the grayscale image and outputs it to the conversion unit 23. The extraction unit 22 may use another edge extraction method such as a Laplacian filter instead of the sobel filter.

The conversion unit 23 binarizes the grayscale image based on the edge information of each pixel input from the extraction unit 22. Specifically, first, the conversion unit 23 calculates the edge amount of each pixel as the value obtained by squaring the edge strengths in the X-axis direction and the Y-axis direction, which are edge information, and then adding them.

Subsequently, the conversion unit 23 sets the pixel in which the calculated edge amount is equal to or greater than the binarization threshold THa, which will be described later, to be “1”, and sets the pixel in which the calculated edge amount is equal to or less than the binarization threshold to “0” to grayscale. Binarize the image.

In this way, the conversion unit 23 can eliminate the influence of noise by binarizing each pixel based on the edge amount. Therefore, the accuracy of detecting water droplets can be improved. Also, by binarizing, all the edges of water droplets are made equivalent. Therefore, the processing load of the matching unit 24 can be suppressed.

In the following, the case of binarization based on the edge amount will be described, but the edge strength and the edge amount have a correlation. Therefore, binarization based on the edge amount is synonymous with binarization based on the edge strength.

Then, the conversion unit 23 outputs a binarized image of each pixel (hereinafter referred to as a binarized image) to the matching unit 24. Here, in the deposit detection device 1 according to the first embodiment, the value of the binarization threshold value THa is dynamically set according to the surrounding environment of the camera 10. Details of this point will be described later with reference to FIG.

In addition to "black" and "white", the conversion unit 23 may use "1", "0", other characters, figures, and the like to binarize the grayscale image.

The matching unit 24 calculates the degree of similarity between the binarized image and the template by matching processing between the binarized image input from the conversion unit 23 and the template showing the characteristics of water droplets. Further, the matching unit 24 outputs the calculated similarity value to the detection unit 25 in association with each pixel in the binarized image.

The details of the processing by the matching unit 24 will be described later with reference to FIG. Further, the template is stored in the storage unit 3 as template information 32. Details of such a template will be described later with reference to FIG.

The detection unit 25 detects water droplets adhering to the camera 10 based on the similarity input from the matching unit 24. Then, when the detection unit 25 detects water droplets, it notifies the deposit removing device 11 and the vehicle control device (not shown) that automatically drives the vehicle.

As a result, for example, the deposit removing device 11 removes water droplets adhering to the camera 10. Further, in the vehicle control device, for example, a white line or the like is recognized while avoiding such an attached region. The details of the detection process by the detection unit 25 will be described later with reference to FIGS. 7A to 7D.

Next, a method of setting the binarization threshold value THa by the conversion unit 23 will be described with reference to FIG. FIG. 3 is a diagram showing a binarization threshold value THa. The vertical axis of FIG. 3 indicates the above-mentioned edge amount, and the horizontal axis indicates the ambient illuminance.

As shown in FIG. 3, the binarization threshold value THa is dynamically set according to the illuminance as the surrounding environment, for example. Specifically, the binarization threshold value THa is set so that the edge amount Sa is the upper limit and the edge amount Sb is the lower limit, for example, the higher the ambient illuminance, the higher the binarization threshold value THa.

Pixels whose edge amount is higher than the binarization threshold THa are represented by "white" in the binarized image, and pixels whose edge amount is equal to or less than the binarization threshold THa are represented by "black" in the binarized image. To.

Here, the light of an ambient light source may be reflected by water droplets. In such a case, the amount of edges of the pixels indicating the edges of the water droplets increases. At this time, when the ambient illuminance is high, that is, when the surroundings are bright, in addition to water droplets, background structures and white lines on the road surface appear in the grayscale image.

Therefore, by setting the binarization threshold value THa to a high value, only the pixel indicating the edge of the water droplet becomes “white”. In other words, by setting the binarization threshold THa to a high value, the edges of unnecessary objects are efficiently removed.

On the other hand, when the illuminance is low, that is, when the surroundings are dark, it is difficult for unnecessary objects to appear in the grayscale image. Further, when light is reflected by the water droplet, the edge amount of the pixel indicating the edge of the water droplet tends to increase as compared with the pixel indicating the edge of the unnecessary object.

Further, depending on the intensity of the light source, the edge amount of the water droplet may not exceed the edge amount Sa. In such a case, if the binarization threshold Tha when the surroundings are dark is set to the same binarization threshold Tha as when the illuminance is high, the edge amount of the water droplet may be lower than the binarization threshold THa, which is not preferable.

From these facts, when the surroundings are dark, the binarization threshold THa is set to a value lower than when the illuminance is high so that only the pixel indicating the edge of the water droplet on which light is reflected becomes "white". There is. In other words, when it is dark, the edges of water droplets can be efficiently extracted regardless of the intensity of the light source.

As described above, in the deposit detection device 1 according to the first embodiment, by dynamically setting the binarization threshold value THa according to the surrounding conditions, only the edge of the water droplet can be efficiently extracted. ..

This makes it possible to accurately detect water droplets reflected by light at night, which was difficult to detect in the past, and water droplets when a strong light source is present in the daytime. As shown in the figure, the conversion unit 23 sets the value of the binarization threshold value THa to the edge amount Sa and the edge amount Sb, respectively, in the daytime and at nighttime, respectively.

As such a method of discriminating between daytime and nighttime, for example, a method of discriminating between daytime and nighttime according to the time zone, and a method of discriminating between daytime and nighttime in conjunction with the headlight of the vehicle can be used.

For example, the conversion unit 23 sets the value of the binarization threshold value THa to the edge amount Sb when the headlight is ON, and sets the value of the binarization threshold value THa to the edge amount Sa when the headlight is OFF. Can be set to.

Further, for example, when the vehicle is provided with an illuminance sensor, the conversion unit 23 can set the value of the binarization threshold value THa based on the sensor value of the illuminance sensor.

In such a case, the conversion unit 23 can sequentially set the value of the binarization threshold value THa according to the illuminance, for example, as shown by the broken line in the figure. Further, the binarization threshold value THa may be switched according to a user operation on an operation unit (not shown).

Here, the case where the binarization threshold value THa is switched according to the illuminance as the ambient environment has been illustrated, but the present invention is not limited to this. That is, the binarization threshold value THa may be switched based on the position information as the surrounding environment.

For example, the conversion unit 23 may acquire the position information from the navigation device and set the binarization threshold value THa to a high value when the vehicle is indoors, for example, in a multi-storey car park.

Next, the template according to the first embodiment will be described with reference to FIG. FIG. 4 is a diagram showing an example of a template according to the first embodiment. As shown in the figure, in the deposit detection device 1 according to the first embodiment, a template in which the outline of water droplets is white is used on a black background.

Further, as shown in the figure, the deposit detection device 1 according to the first embodiment has a plurality of templates showing the characteristics of the contour of the water droplet. Specifically, as shown in a of the figure, a template in which the outline of the water droplet is substantially circular, and as shown in b to e of the figure, a template in which the water droplet is partially arcuate. Have.

Further, as shown in f in the figure, the template is not limited to a substantially perfect circle shape or a substantially semi-arc shape, and may be a substantially elliptical shape. This is because when a convex lens such as a fisheye lens is used for the camera 10, water droplets adhering to the convex lens tend to have an elliptical shape.

As described above, in the deposit detection device 1 according to the first embodiment, the accuracy of detecting water droplets can be improved by performing the matching process using a plurality of templates. Further, by using the template showing the partial shape of the water droplet shown in FIGS. 4 to e, it is possible to detect the water droplet even when the water droplet is cut off from the image.

Further, in the deposit detection device 1 according to the first embodiment, the area for performing the matching process is set for each type of template (a to f in the figure). Details of this point will be described later with reference to FIG.

The template shown in FIG. 4 is an example and is not limited thereto. For example, the optimum template can be derived by simulation, statistics, etc. in consideration of the shape of the camera 10 and the dripping of water droplets.

Further, the deposit detection device 1 according to the first embodiment has a plurality of templates having different scales of the templates shown in FIGS. 4A to 4F. As a result, water droplets of different sizes can be detected with high accuracy. The matching unit 24 can also perform matching processing from templates having different sizes depending on the surrounding environment of the camera 10.

For example, when it rains heavily, a large amount of rain tends to adhere to the camera 10, so the matching unit 24 starts the matching process from a large template for large water droplets, and gradually performs the matching process using a small template.

Further, when it is raining lightly, small water droplets are likely to adhere to the camera 10, so the matching process is started from a small template, and the matching process is gradually performed using a large template.

By doing so, water droplets can be efficiently detected according to the surrounding conditions of the vehicle. Further, in the deposit detection device 1 according to the first embodiment, different detection threshold values are set according to the size of the template. Details of this point will be described later with reference to FIG. 7A.

Next, the relationship between the template and the scanning position will be described with reference to FIG. FIG. 5 is a diagram showing the relationship between the template and the scanning position. The regions Ra to Re shown in FIG. 5 correspond to the templates shown in a to e of FIG. 4, respectively.

Specifically, the matching unit 24 is used when scanning the region Ra located at the center of the binarized image L2 using the substantially perfect circular template shown in FIG. 4a. Further, the template in which the arc shown in FIG. 4b is directed downward in the figure is used when scanning the region Rb located at the upper part of the binarized image L2.

Further, the template in which the arc shown in c of FIG. 4 points upward in the figure is used when scanning the region Rc located at the lower part of the binarized image L2. Similarly, the templates in which the arcs shown in d and e in FIG. 4 are oriented to the left or right in the figure are used when scanning the regions Rd and Re located on the left and right sides of the binarized image L2, respectively.

In this way, the matching unit 24 can perform matching processing using different templates depending on the scanning position of the binarized image L2. As a result, it is possible to efficiently detect water droplets having a shape that easily adheres to each region in the binarized image L2.

The matching unit 24 may scan all types of templates in the entire region of the binarized image L2. In such a case, it is possible to suppress the detection omission of water droplets.

Further, when the camera 10 is a wide-angle lens, water droplets are distorted and displayed toward the edge of the camera image. For this reason, a template may be used that shows the characteristics of the contour of the distorted water droplet toward the edge.

Next, the template and the matching process by the matching unit 24 will be described with reference to FIG. FIG. 6 is a diagram showing an example of matching processing.

As shown in the figure, first, the matching unit 24 is arranged, for example, in the pixel P1 in which the upper left of the template G is located in the upper left of the binarized image L2.

Subsequently, the matching unit 24 calculates the degree of similarity between the binarized image L2 and the template G at such a position. Since the method of calculating the similarity will be described in the second embodiment, the description thereof will be omitted here.

Subsequently, the matching unit 24 stores the calculated similarity value in, for example, the pixel P1 located at the upper left of the template G. After that, the template G is shifted to the right by one pixel to calculate the similarity, and the calculated similarity value is stored in the pixel P2.

The matching unit 24 repeats the calculation of the similarity, and when the calculation of the similarity is completed up to the right end, the matching unit 24 calculates the similarity for all the pixels by shifting the template G downward by one pixel and repeating the calculation process. .. As a result, the matching unit 24 obtains the similarity value for all the pixels.

Further, the matching unit 24 shall similarly calculate the degree of similarity for templates having different types and sizes. Then, the matching unit 24 outputs the calculated similarity value together with the pixel coordinates to the detection unit 25.

Here, the matching unit 24 does not necessarily have to calculate the similarity value for all the pixels, and the similarity calculation process is simplified, for example, the similarity is calculated at predetermined intervals. You may. By doing so, the processing load by the matching unit 24 can be reduced.

Further, for example, the similarity value may be calculated from a region having a high priority for detecting water droplets, such as a region Rg located at the center of the binarized image L2 shown in the figure. By doing so, it is possible to quickly detect the adhering water droplet in the high priority region.

At this time, the matching unit 24 may not calculate the similarity for the region having a low priority such as the region above the binarized image L2. As a result, the processing load of the matching unit 24 can be further reduced.

Further, for example, the matching unit 24 may calculate the similarity for all the pixels in the high priority region, and simplify the similarity calculation process for the low priority region.

That is, in the region with high priority, the accuracy may be increased to detect water droplets, and in the region with low priority, the accuracy may be coarsened to detect water droplets. As a result, it is possible to suppress omission of detection of water droplets in a region having a high priority while suppressing the processing load.

Next, the determination process by the detection unit 25 will be described with reference to FIGS. 7A to 7D. 7A to 7D are diagrams for explaining the determination process by the detection unit 25.

First, the relationship between the template and the detection threshold will be described with reference to FIG. 7A. As shown in the figure, the detection unit 25 sets different detection threshold values according to the size of the template.

The detection threshold is compared with the similarity value input from the matching unit 24 by the detection unit 25. Then, if the value of the similarity is equal to or higher than the detection threshold value, the detection unit 25 determines that water droplets are attached.

As shown in the figure, in the deposit detection device 1 according to the first embodiment, the larger the template, the lower the detection threshold value, while the smaller the template, the higher the detection threshold value is set.

This is because the smaller the template, the more likely it is that unnecessary edges other than water droplets will be erroneously detected as edges of water droplets. That is, by setting the detection threshold value according to the size of the template, erroneous detection of water droplets can be suppressed.

Although the case where the template has a substantially perfect circular shape corresponding to a in FIG. 4 is shown in the figure, the same applies to the other templates shown in b to f in FIG. Further, different detection threshold values may be provided depending on the type of template, or the same detection threshold value may be used for all templates.

Next, a case where the detection unit 25 sets a different detection threshold value for each region of the binarized image L2 will be described with reference to FIG. 7B. As described above, the priority for detecting water droplets differs depending on the region of the binarized image L2.

Therefore, in the deposit detection device 1 according to the first embodiment, different detection threshold values can be set depending on the region of the binarized image L2. Specifically, for example, in the region Rl where a position close to the vehicle is captured, the priority of water droplet detection is set high, and the detection threshold is set lower than in other regions. As a result, it is possible to suppress omission of detection of water droplets in the high-priority region RI.

On the other hand, for the region Rh in which a position far from the vehicle is captured, the priority of water droplet detection is set low and the detection threshold is set high. Further, for the region Rm located between the regions Rl and Rh, for example, the detection threshold value is set to be an intermediate value between the regions Rl and the region Rh.

In this way, by setting different detection threshold values according to the region of the binarized image L2, erroneous detection of water droplets in the low priority region Rh is reduced, and water droplets are reliably generated in the high priority region Rl. Can be detected.

Although the case where the binarized image L2 is divided into three regions has been described with reference to FIG. 7B, the divided regions may be two or four or more. Further, the binarized image L2 may be divided into a horizontal direction, an oblique direction, or a concentric circle to set a detection threshold value.

Further, the priority of each of the above-mentioned regions is an example, and the priority may be changed according to the purpose of detecting water droplets. Further, the detection unit 25 may set different detection threshold values in the daytime and at night, for example.

Next, a case where the detection unit 25 detects water droplets based on the binarized image L2 of a plurality of frames will be described with reference to FIG. 7C. Here, for example, the detection unit 25 shows a scene in which the detection process already described with reference to FIGS. 7A and 7B has been completed.

Further, in the figure, it is shown that the similarity with the template corresponding to the size of each frame is larger than the detection threshold value in the region where the frames F1 to F3 exist.

Here, when there is an area having a high degree of similarity with templates having different sizes, such as the frame F1 and the frame F2, and there is an area in which the frame F2 exists in the area surrounded by the frame F1, it is detected. The unit 25 determines that such a region has water droplets attached to it. This is because when one template has a high degree of similarity, the degree of similarity tends to be high in other templates having similar sizes.

Further, when the detection unit 25 does not satisfy the above conditions when there is only one frame in the adjacent region, for example, the frame F3, it is said that water droplets do not adhere to the region of the frame F3. judge. As mentioned above, if one template has a high degree of similarity, the other templates that are close in size also tend to have a high degree of similarity. Therefore, only one template has a high degree of similarity and the other (size). This is because if the similarity is not high in the (close) template, it can be judged that the similarity is high in one template due to the influence of noise or the like.

That is, the detection unit 25 determines that water droplets are attached to the region having a high degree of similarity to the plurality of templates having different sizes. In this way, the deposit detection device 1 can improve the reliability of the water droplet detection accuracy while reducing the false detection of the water droplet by detecting the water droplet based on the similarity with the plurality of templates. ..

In such a case, the detection unit 25 may determine, for example, that water droplets are attached if at least a part of the frame F1 and the frame F2 overlap. Further, the detection unit 25 may determine, for example, that water droplets are attached when a plurality of frames are present at adjacent positions.

Next, a case where the detection unit 25 detects water droplets based on a plurality of binarized images L2 will be described with reference to FIG. 7D. In the figure, it is shown that the front side of the paper surface is a newer binarized image L2. That is, it is shown that the binarized image L2e is the new binarized image L2 in the order of the binarized image L2e to the binarized image L2a.

Further, it is assumed that the frame F4 and the frame F5 shown in the figure have a high degree of similarity with the template corresponding to the size of the frame in the area of the frame as in FIG. 7C.

As shown in the figure, the binarized image L2a, the binarized image L2b, and the binarized image L2c have a frame F4, and the binarized image L2a, the binarized image L2c, and the binarized image L2e There is a frame F5 in.

At this time, the detection unit 25 determines that, for example, when three frames are continuously present at the same position, water droplets are attached to such a region. In the example shown in the figure, since the frame F4 is continuously detected over three frames, the detection unit 25 determines that water droplets are attached to the region of the frame F4.

Note that the detection unit 25 may determine that water droplets are attached to the region of the frame F5 if the detection unit 25 is detected three times during the five frames, for example, the frame F5.

In this way, the detection unit 25 can improve the detection accuracy while suppressing erroneous detection of water droplets by determining the adhesion of water droplets using a plurality of frames. It should be noted that the positions and sizes of the frames F4 and F5 do not have to be exactly the same in each frame, and a predetermined error is allowed.

Further, in the figure, when the detection unit 25 detects the same frame for three consecutive frames, the water droplet is detected, but the present invention is not limited to this, and even if the number of frames is 2 or less, 4 It may be more than a frame.

The detection unit 25 may change the number of frames used for detecting water droplets according to the priority of the above-mentioned region and the purpose of detecting water droplets. For example, the number of frames can be increased in the region where the detection accuracy is prioritized, and the number of frames can be decreased in the region where the detection speed is prioritized.

Next, the processing procedure executed by the deposit detection device 1 according to the first embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing a processing procedure executed by the deposit detection device 1 according to the first embodiment. The processing shown below is repeatedly executed by the control unit 2.

As shown in the figure, first, the image acquisition unit 21 acquires a camera image from the camera 10 and grayscales the camera image (step S101). Subsequently, the extraction unit 22 extracts the edge information of each pixel from the grayscale image (step S102).

After that, the conversion unit 23 sets the binarization threshold value THa as shown in FIG. 3 (step S103), and binarizes the grayscale image (step S104).

Subsequently, the matching unit 24 performs a matching process using the binarized image binarized by the conversion unit 23 as shown in FIG. 6 (step S105). Then, the detection unit 25 detects water droplets adhering to the camera 10 as shown in FIGS. 7A to 7D (step S106).

As described above, the deposit detection device 1 according to the first embodiment can accurately detect shining water droplets at night or in the daytime by binarizing based on the edge amount of each pixel.

By the way, as described above, the conversion process by the conversion unit 23 is not limited to the binarization process. Therefore, next, as a second embodiment, a case where each pixel is converted into a parameter according to the direction of the edge will be described.

(Second Embodiment)
First, the configuration of the deposit detection device 1B according to the second embodiment will be described with reference to FIG. FIG. 9 is a block diagram of the deposit detection device 1B according to the second embodiment. In the following description, the same parts as those already described will be designated by the same reference numerals as those already described, and duplicate description will be omitted.

As shown in FIG. 9, the deposit detection device 1B according to the second embodiment includes a control unit 2B and a storage unit 3B. The control unit 2B includes a conversion unit 23B, a matching unit 24B, and a detection unit 25B in place of the conversion unit 23, the matching unit 24, and the detection unit 25 shown in FIG.

Further, the storage unit 3B stores the range information 34, the parameter information 31B, the template information 32B, and the detection information 33B. Here, the description of the image acquisition unit 21 and the extraction unit 22 will be omitted, and the description will be given from the conversion unit 23B.

The conversion unit 23B calculates the direction of the edge of each pixel based on the edge information of each pixel input from the extraction unit 22, and assigns a three-dimensional parameter to the edge direction to parameterize each pixel. Details of this point will be described later with reference to FIGS. 11A and 11B.

Further, the conversion unit 23B calculates the above-mentioned edge amount and performs a filter process on the calculated edge amount prior to the calculation of the edge direction. Details of this point will be described later with reference to FIG.

The matching unit 24B performs a matching process between the parameterized grayscale image input from the conversion unit 23B and the template showing the characteristics of the water droplets. Since the matching process is the same as the matching process already described with reference to FIG. 6, the description thereof is omitted here.

Here, the matching unit 24B calculates the degree of similarity using Zero-mean Normalized Cross-Correlation. In such a zero mean normalization cross-correlation, the similarity ranges from -1 to +1.

The matching unit 24B may calculate the similarity by using another calculation method such as SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference).

The detection unit 25B detects water droplets adhering to the camera 10 based on the similarity value input from the matching unit 24B. The details of the detection process by the detection unit 25B will be described later with reference to FIG.

Next, the details of the filtering process by the conversion unit 23B will be described with reference to FIG. FIG. 10 is a diagram showing an extraction range W. The vertical axis of the figure shows the amount of edges. Further, in the figure, the extraction range W is shown by being hatched.

As shown in the figure, in the deposit detection device 1B according to the second embodiment, for example, different extraction ranges W are set at night and in the daytime. Specifically, in the case of nighttime, the range from the edge amount Sc to the edge amount Se is set as the extraction range W1.

In the daytime, the range from the edge amount Sd lower than the edge amount Sc to the edge amount Sf is set as the extraction range W2. This is because, as described above, only the edges of water droplets are efficiently extracted regardless of the ambient brightness.

By performing the filtering process in this way, unnecessary edges other than water droplets can be removed. As a result, the accuracy of detecting water droplets can be improved.

The optimum values of the extraction range W1 and the extraction range W2 can be derived by using simulation, statistics, or the like. Further, the conversion unit 23B may set the extraction range W by feeding back the detection result of the water droplet by the detection unit 25B.

For example, if water droplets are not detected by the detection unit 25B, the extraction range W may not be set appropriately. Therefore, the conversion unit 23B may change the extraction range W and re-extract each pixel. As a result, it is possible to suppress the detection omission of water droplets, so that the accuracy of detecting water droplets can be improved.

Further, in the storage unit 3B, statistical values such as the distribution of the edge amount and the average value in each pixel are stored in each of the nighttime and the daytime. Then, the conversion unit 23B can set the extraction range W by discriminating between nighttime and daytime by comparing the statistical value with the extracted edge amount.

As a result, the extraction range W can be appropriately set according to the surrounding conditions. The control unit 2B may be provided with a learning function, and the control unit 2B may derive such a statistical value.

Next, the processing by the conversion unit 23B will be described with reference to FIGS. 11A and 11B. FIG. 11A is a diagram showing an edge vector. FIG. 11B is a diagram showing an example of parameter information 31B.

Note that FIG. 11A shows the edge strength in the X-axis direction and the Y-axis direction. As described above, the conversion unit 23B calculates the vector of each pixel based on the edge strength in the X-axis direction and the edge strength in the Y-axis direction input from the extraction unit 22.

Specifically, the vector of each pixel is calculated by using a trigonometric function based on the edge strength in the X-axis direction and the Y-axis direction. In the following, the angle θ formed by the calculated vector shown in the figure and the X axis on the positive direction side is referred to as the edge direction, and the length L of the vector is referred to as the edge strength of each pixel. The edge strength is used in the third embodiment.

It should be noted that the conversion unit 23B does not need to calculate the edge orientation for all the pixels, and may simplify the process by calculating the edge orientation for each pixel at a predetermined interval in the low priority region. Good.

Subsequently, the conversion unit 23B parameterizes the edge orientation calculated using the parameter information 31B shown in FIG. 11B. Here, a case where the conversion unit 23B parameters the edge orientation using the color vector of the 12 hue circle will be described.

Here, the color vector is a vector defined according to the color elements of RGB, and has each parameter of three components composed of RGB.

In the 12 hue circle, each parameter is represented by three values of "0", "128" or "127", and "255" when red, green, and blue are expressed in decimal numbers. Here, in the 12-hue circle, each RGB value having a complementary color relationship has a one's complement relationship in binary notation such as a hexadecimal number.

That is, the conversion unit 23B according to the second embodiment converts each pixel using a parameter in which each of the RGB values of the opposing angle ranges satisfies the one's complement relationship. It should be noted that such a parameter does not have to satisfy the relationship of exactly 1's complement, and a predetermined error is allowed.

Specifically, the conversion unit 23B assigns RGB parameters in the angle range corresponding to the calculated edge orientation to the pixels. For example, when the edge orientation is in the angle range of 75 ° to 105 °, RGB (0, 255, 255) parameters corresponding to light blue are assigned to the edge orientation parameters.

Further, when the edge orientation exists in the angle range of −75 ° to −105 ° facing the angle range, RGB (255, 0, 0) corresponding to red, which is the complementary color of light blue, is assigned. ..

Further, the conversion unit 23B uses RGB (255, 255, 255) parameters corresponding to white for pixels whose edge amount is out of the extraction range W.

As described above, in the deposit detection device 1B according to the second embodiment, each pixel is converted using the parameter in which the opposite edge directions are in the one's complement relationship. This makes it possible to clarify the difference in the orientation of the opposing edges.

Further, by using the color vector of the 12 hue circle, it is possible to clarify the difference in each angle range corresponding to the edge direction. Therefore, the recognition accuracy by the matching unit 24B can be improved.

Further, in the deposit detection device 1B according to the second embodiment, the same parameters are assigned as long as the edge directions are close to each other, regardless of the edge strength. Therefore, even a water droplet having a blurred edge, that is, a water droplet having a weak edge strength can be detected with the same accuracy as a water droplet having a strong edge strength.

Based on these facts, the deposit detection device 1B according to the second embodiment accurately detects even water droplets having a weak edge strength, that is, water droplets that appear blurred in a camera image, which was difficult to detect in the past. It becomes possible.

Further, the table referred to by the conversion unit 23B is not limited to the example shown in FIG. 11B, and for example, in addition to 12 colors, the angle range such as 24 colors may be further divided, or an angle such as 9 colors may be used. The range may be divided into larger parts.

Further, the present invention is not limited to RGB, and other parameters may be used, such as using a matrix as a parameter, in which opposite edge directions satisfy the one's complement relationship.

Next, the template according to the second embodiment will be described with reference to FIG. FIG. 12 is a schematic diagram of the template according to the second embodiment. In the figure, the template is simplified for the sake of simplicity.

As shown in the figure, in the template according to the second embodiment, the circle simulating a water droplet is divided into a plurality of fan-shaped regions. Then, each fan-shaped region is assigned a parameter corresponding to the above-mentioned angle range in which the value of the region facing the center of the circle is a complement of 1.

Further, as shown in a in the figure, RGB parameters indicating white and gray are assigned in addition to the region indicating the characteristics of the water droplet. Specifically, RGB (255, 255, 255) parameters corresponding to white are assigned to the central part of the template, and RGB (128, 128, 128) parameters corresponding to gray are assigned to the outside of the water droplet. Assigned.

The parameters of the central portion and the outer regions of the water droplet are merely examples, and parameters corresponding to other colors may be assigned.

Further, as shown in b of the figure, it is also possible to use a template in which each parameter is assigned radially from the central portion to all the regions. Such a template is effective when detecting water droplets which are thin and appear blurry in a camera image.

Further, as shown in c of the figure, the white region in the center of the template may be enlarged. When such a template is used, it is possible to efficiently detect water droplets having bright edges. Further, as shown in d in the figure, the shape indicating the water droplet may be an elliptical shape.

As described above, in the deposit detection device 1B according to the second embodiment, the template showing water droplets with such parameters is obtained for the grayscale image converted into the parameters in which the opposite edge directions are in the one's complement relationship. Is used.

Therefore, the difference in the direction of each edge becomes clear for each pixel and each template, and the recognition accuracy by the matching unit 24B can be improved.

Further, the deposit detection device 1B according to the second embodiment may use a semi-arc-shaped template as already described with reference to FIGS. 4b to e. In such a case, as already described with reference to FIG. 5, a matching process of showing a partial shape of water droplets according to the scanning position can be performed.

In the deposit detection device 1B according to the second embodiment, a template showing the characteristics of water droplets when the brightness increases from the end to the center, that is, when the edge direction is from the end to the center is used. Performs matching processing.

However, the present invention is not limited to this, and a template showing the characteristics of water droplets when the brightness increases from the center to the edge, that is, when the edge direction is from the center to the edge may be used.

Next, the detection process by the detection unit 25B will be described with reference to FIG. FIG. 13 is a diagram showing a specific example of the detection threshold value. The horizontal axis in the figure shows the value of similarity.

As described above, in the deposit detection device 1B according to the second embodiment, the similarity with the template is calculated by a value of -1 to +1. Here, when the similarity is close to +1 it indicates that it is similar to the template. That is, it is similar to a water droplet whose brightness increases from the end to the center.

Further, when the similarity is close to -1, it indicates that the template is similar to the so-called negative / positive inverted template. That is, in this embodiment, water droplets whose brightness increases from the center to the edges are shown.

Therefore, as shown in FIG. 13, in the second embodiment, the detection thresholds are set for both positive and negative similarities. Specifically, as shown in the figure, for example, the positive detection threshold is +0.7 and the negative detection threshold is −0.8.

Then, in the detection unit 25B, when the value of the similarity input from the matching unit 24B is equal to or more than the positive detection threshold value, that is, +0.7 or more, or less than or equal to the negative detection threshold value, that is, −0.8 or less. In a certain case, it is determined that water droplets are attached to the camera 10.

As described above, in the deposit detection device 1B according to the second embodiment, positive and negative detection threshold values are set for the similarity value. Thereby, by the matching process using one type of template, it is possible to detect both water droplets whose brightness increases from the center to the end and water droplets whose brightness increases from the center to the end.

In other words, various water droplets can be detected while maintaining the processing load. The detection unit 25B may detect water droplets in combination with the processes already described with reference to FIGS. 7A to 7D.

Further, in the deposit detection device 1B according to the second embodiment, the positive detection threshold value is set to a value whose absolute value is lower than the negative detection threshold value as described above. This is because when the similarity is negative, the number of false detections of water droplets tends to be higher than when the similarity is positive.

Each value of the detection threshold value shown in the figure is an example and is not limited to this. For example, the absolute value of the positive detection threshold and the negative detection threshold may be the same, or the absolute value of the positive detection threshold may be larger than the absolute value of the negative detection threshold. ..

Next, the processing procedure executed by the deposit detection device 1B according to the second embodiment will be described with reference to FIG. FIG. 14 is a flowchart showing a processing procedure executed by the deposit detection device 1B according to the second embodiment.

Here, since the description has already been given in the first embodiment, the description of step S101 and step S102 will be omitted, and the description will be given from step S201 shown in the figure.

First, the conversion unit 23B calculates the edge vector of each pixel based on the edge information input from the extraction unit 22 (step S201), and parameterizes each pixel based on the edge orientation as described in FIG. 11B. (Step S202).

Subsequently, the matching unit 24B performs a matching process between the parameterized grayscale image and the template (step S203). Then, the detection unit 25B detects water droplets based on the detection threshold value shown in FIG. 13 (step S204).

As described above, the deposit detection device 1B according to the second embodiment can accurately detect water droplets by using each RGB parameter of the 12 color wheel corresponding to the edge direction of each pixel. ..

In addition, the deposit detection device 1B according to the second embodiment can detect both water droplets whose center is shining and water droplets whose edges are shining by one matching process.

(Third Embodiment)
Next, the deposit detection device 1C according to the third embodiment will be described with reference to FIGS. 15 to 20. The deposit detection device 1C according to the third embodiment converts each pixel into a code according to the direction of the edge of each pixel, and performs a matching process using a regular expression.

First, the configuration of the deposit detection device 1C according to the third embodiment will be described with reference to FIG. FIG. 15 is a block diagram of the deposit detection device 1C according to the third embodiment.

As shown in FIG. 15, the deposit detection device 1C according to the third embodiment includes a control unit 2C and a storage unit 3C. The control unit 2C includes a conversion unit 23C, a matching unit 24C, and a detection unit 25C in place of the conversion unit 23, the matching unit 24, and the detection unit 25 shown in FIG. Further, the storage unit 3C stores the code information 31C, the template information 32C, and the detection information 33C.

Here, since the image acquisition unit 21 and the extraction unit 22 have already been described with reference to FIGS. 2 and 9, they will be omitted and will be described from the conversion unit 23C.

The conversion unit 23C calculates the edge vector of each pixel based on the edge strength of each pixel in the X-axis direction and the Y-axis direction input from the extraction unit 22, and encodes the edge direction, respectively. Since the method of calculating such a vector has already been described with reference to FIG. 11A, the description thereof will be omitted here.

Then, the conversion unit 23C outputs a grayscale image in which each pixel is encoded to the matching unit 24C. Here, in the deposit detection device 1C according to the third embodiment, for example, a representative value of an edge in a plurality of pixels is obtained, and the representative value is encoded. Details of this point will be described later with reference to FIGS. 16A and 16B.

The matching unit 24C performs a matching process using a regular expression of the coded grayscale image input from the conversion unit 23C and the code pattern indicating the characteristics of the water droplets. Here, the regular expression is a set of code strings represented by a single code.

Since the matching unit 24C performs matching processing using a regular expression, it does not require complicated processing such as calculation of similarity as described above. Therefore, water droplets can be detected while suppressing the processing load.

The code pattern indicating the characteristics of the water droplet is stored in the template information 32C. The details of such a code pattern will be described later with reference to FIG. 17A. The details of the processing by the matching unit 24C will be described later with reference to FIG. 17B.

The detection unit 25C detects water droplets adhering to the camera 10 based on the code pattern extracted by the matching unit 24C. The detection process by the detection unit 25C will be described later with reference to FIGS. 18 and 19.

Next, the coding process by the conversion unit 23C will be described with reference to FIGS. 16A and 16B. 16A and 16B are diagrams illustrating processing by the conversion unit 23C.

First, a pixel for which a representative value is calculated will be described with reference to FIG. 16A. Here, a pixel of 8 × 8 pixels is called a cell, and a 3 × 3 cell is called a block. The cell in the center of the block is called the cell of interest.

The conversion unit 23C creates a histogram showing the edge orientation and edge strength of each pixel for each block. Such a histogram will be described with reference to FIG. 16B. Here, the conversion unit 23C derives the edge orientation of the central coordinate in the cell of interest from the histogram in the block.

Then, when the conversion unit 23C derives the representative value of the cell of interest in one block, the conversion unit 23C shifts the block by one cell to create a histogram, and calculates the representative value of the cell of interest in the block.

That is, in the deposit detection device 1C according to the third embodiment, the amount of data can be reduced by calculating the representative value for each of a plurality of pixels. Therefore, the matching process by the matching unit 24C can be simplified. In the example shown in the figure, since the cell is 8 × 8 pixels, the amount of data is reduced to 1/64.

The number of pixels of the block and the cell shown in FIG. 16A is an example, and the number of pixels of the cell and the block can be arbitrarily set. At this time, the number of pixels of each cell can be changed according to the size of the water droplet to be detected.

For example, if you want to detect small water droplets, set a small number of pixels in the cell, and if you want to detect large water droplets, set a large number of pixels in the cell. This makes it possible to efficiently detect water droplets of the size to be detected.

Further, the conversion unit 23C may simply create a histogram for each cell and calculate the representative value of each cell based on the histogram. Note that the conversion unit 23C may encode all the pixels without calculating the representative value.

Next, the histogram will be described with reference to FIG. 16B. In the figure, the vertical axis shows the edge strength, and the horizontal axis shows the class for the edge. As shown in the figure, in the deposit detection device 1C according to the third embodiment, for example, the edge orientation is assigned to each class of 18 stages every 20 ° to create a histogram.

Specifically, the conversion unit 23C creates a histogram in the block by adding the edge strength of each pixel in the block to the class corresponding to the edge direction. Subsequently, the conversion unit 23C obtains the class that maximizes the sum of the edge intensities from the created histogram.

In the example shown in the figure, the case where the class of 80 to 100 ° takes the maximum value is shown. At this time, the conversion unit 23C uses such a class as a representative value when the sum of the edge intensities is equal to or greater than the threshold value in such a class.

In the example shown in the figure, since the sum of the edge intensities exceeds the threshold value in the class of 80 to 100 °, the above condition is satisfied. Therefore, the class of the cell of interest in such a block is 80 to 100 °.

Subsequently, the conversion unit 23C converts the cell of interest into a code assigned according to the class. Here, 18 types of codes 0 to 9 and A to H are assigned to each class. Note that 0 to 9 and A to H are codes assigned to each class from 0 ° to 360 ° in increments of 20 degrees. Further, when the representative value does not exceed the threshold value, that is, the cell having a low edge strength is assigned the sign Z.

In this way, the conversion unit 23C encodes all the cells. As a result, the codes are arranged in a grid pattern in the coded grayscale image. In addition to calculating the representative value described above, the conversion unit 23C may calculate the representative value by using another statistical calculation method.

Further, in the figure, the case where the edge orientation is classified into 18 classes has been described, but the present invention is not limited to this, and the number may be less than or more than 18 classes. Further, in the figure, the cases where the symbols are A to H and Z are shown, but other characters such as hiragana and numbers, figures, and the like may be used as the symbols.

Further, for example, when there are a plurality of classes exceeding the threshold value in one block, the conversion unit 23C may output the codes corresponding to the plurality of classes to the matching unit 24C in association with the grayscale image. ..

In other words, the conversion unit 23C may associate a plurality of edge orientation information with the grayscale image. In such a case, since the amount of data for detecting water droplets increases, it becomes possible to detect water droplets more accurately.

Next, the processing by the matching unit 24C according to the third embodiment will be described with reference to FIGS. 17A and 17B. FIG. 17A is a schematic view showing an example of the template according to the third embodiment. FIG. 17B is a diagram showing an example of matching processing by the matching unit 24C.

In addition, in FIG. 17A, in order to make it visually easy to understand, the template is schematically shown with the actual edge orientation instead of the above-mentioned reference numerals. As shown in FIG. 17A, the deposit detection device 1C according to the third embodiment has a code pattern which is a code string indicating the characteristics of water droplets as a template. Specifically, it has, for example, an upper side pattern, a lower side pattern, a left side pattern, and a right side pattern.

Here, the pattern of each side shown in the figure shows each side of a rectangle containing water droplets or externally water droplets. Further, in the figure, the case where the edge direction of each side pattern is directed toward the center is shown. In this case, the brightness of the water droplet increases from the end to the center, that is, the characteristic of the water droplet is that the center is bright and the end is dark.

The deposit detection device 1C according to the third embodiment includes a pattern on each side showing the characteristics of water droplets whose brightness increases from the center to the edges, that is, the center is dark and the edges are bright. You may. By doing so, it becomes possible to detect various water droplets.

In the figure, four patterns of up, down, left, and right are illustrated, but a pattern including an oblique direction can also be used. By doing so, the accuracy of detecting water droplets can be improved.

Further, the code string indicating the characteristics of the water droplet may be, for example, an array of codes arranged in an arc shape. Further, the matching unit 24C may limit the area where the regular expression is performed according to the pattern of each side.

Next, the matching process by the matching unit 24C will be described with reference to FIG. 17B. Here, for convenience of explanation, the upper side patterns shown in FIG. 17A are shown using reference numerals A to F, respectively. Further, a and b in the figure schematically show a part of the grayscale image encoded by the conversion unit 23C.

As shown in a in the figure, if the code patterns are arranged in the order of A to F, the matching unit 24C determines that the code pattern matches the upper side pattern.

Specifically, as shown in a in the figure, the matching unit 24C has an array in which A is repeated three times, B, C, D and E are repeated twice, and F is repeated three times. If the arrangement order of each code of the pattern is satisfied, it is extracted as the upper side pattern.

This is because the number of times the code is repeated differs depending on the size of the water droplet. That is, the larger the water droplet, the longer the length of each code string. By allowing the repetition of such a code in this way, it is possible to extract a code string indicating a plurality of water droplets having different sizes in one matching process.

Therefore, water droplets can be detected while reducing the processing load. The matching unit 24C may prepare a plurality of patterns on each side having different lengths of the code strings according to the size of the water droplets, and extract the code strings using all such patterns.

Also, since water droplets are generally spherical, the number of repetitions of each code should be line-symmetrical from the center. Therefore, the matching unit 24C decides to exclude the unbalanced code string from the extracted code strings.

Specifically, as shown in b in the figure, the matching unit 24C examines, for example, the balance between A and F located at both ends. Here, the figure shows a case where A is repeated 3 times and F is repeated 10 times.

At this time, the matching unit 24C excludes such a code string pattern even when the arrangement order is satisfied when the numbers of A and F are different by two times or more. By doing so, it is possible to prevent erroneous extraction of unnecessary code patterns other than water droplets, and it is possible to suppress erroneous detection of water droplets.

Further, in the matching unit 24C, for example, when the extracted code string is longer than the threshold value, such a code string can be excluded from the matching. This is because if the code string is long, it is unlikely that it is a water droplet. Therefore, erroneous detection of water droplets can be suppressed. It should be noted that the optimum value of such a threshold value shall be derived in advance by statistics or the like.

Next, the detection process by the detection unit 25C according to the third embodiment will be described with reference to FIG. FIG. 18 is a diagram illustrating a detection process by the detection unit 25C according to the third embodiment. In the same figure, as in FIG. 17A, the actual edge orientation is schematically shown instead of the reference numerals.

Further, here, the case where the upper side pattern is first extracted by the matching unit 24C will be described. First, the detection unit 25C sets a substantially square detection region R1 based on the width of the upper side pattern.

Subsequently, it is assumed that the right-hand side pattern is extracted by the matching unit 24C at a position deviating from the detection region R1. At this time, if the center coordinates of the detection region R2 of the right-hand side pattern are within the detection region R1, the detection unit 25C performs a process of integrating both detection regions R1 and R2.

After that, for example, when the lower side pattern or the left side pattern is extracted in the integrated detection area R3, the detection unit 25C detects water droplets in the integrated detection area R3. In other words, the detection unit 25C detects water droplets under a detection condition (hereinafter, referred to as a direction condition) that a pattern indicating each side of three or more different directions is extracted in the detection region R3.

In addition to the directional condition, the detection unit 25C detects that, for example, in the integrated detection region R3, a pattern indicating each side is extracted a predetermined number of times (for example, four times including up, down, left, and right) or more. It may be a condition (hereinafter referred to as a number-of-times condition).

In this way, by setting the detection conditions as the direction condition of three or more directions and the number of times condition, water droplets can be detected even if all the upper, lower, left, and right sides are not extracted. That is, for example, a semicircular water droplet that can be seen off from the camera image can be detected.

In addition, for example, the directional condition may be changed according to the region where water droplets are detected. For example, for the central region of the camera image, the direction conditions are set to four directions. As a result, the accuracy of detecting water droplets can be improved.

Further, for example, for the regions at the four corners of the camera image, the direction condition is set to twice. This makes it possible to detect the fan-shaped water droplets that are cut off at the four corners of the camera image.

In the figure, a case where the detection area is integrated when the center coordinates of the detection area R2 are within the detection area R1 of the upper side pattern is shown, but the present invention is not limited to this. That is, if at least a part of the detection area R1 and the detection area R2 overlap, both detection areas may be integrated.

Further, the integrated detection area R3 may be the logical product of the detection area R1 and the detection area R2, or may be the logical sum of the detection areas. Further, in the figure, the case where the detection area R1 and the detection area R2 have a rectangular shape is shown, but the present invention is not limited to this, and the detection area may have another shape such as a circular shape.

Note that the detection unit 25C may detect water droplets based on the detection results in a plurality of frames, as already described in FIG. 7D.

Next, the exclusion process of the detection condition by the detection unit 25C will be described with reference to FIG. FIG. 19 is a diagram showing conditions to be excluded from the detection conditions. Further, in the figure, a part of the grayscale image is enlarged to schematically show the edge orientation.

The length a1 shown in the figure is, for example, 1.5 times the length a2. That is, the length a2 is a value of 2/3 of the length a1. Here, for example, in the left two-thirds region of the grayscale image (the region on the left side of the broken line shown in the figure), the left side pattern tends to be easily extracted even if water droplets are not attached to the camera 10.

Therefore, the detection unit 25C is exceptionally a detection target even when the above-mentioned number of times condition is satisfied when a plurality of left side patterns are extracted in such a region and patterns indicating other sides are not extracted. Exclude from.

As a result, erroneous detection of water droplets can be suppressed. Although the left side pattern has been described as an example here, the same applies to the patterns on the other sides.

Next, the treatment procedure executed by the deposit detection device 1C according to the third embodiment will be described with reference to FIG. FIG. 20 is a flowchart showing a processing procedure executed by the deposit detection device 1C according to the third embodiment. Here, steps S101 and S102 will be omitted because they have already been described, and will be described from step S301 shown in the figure.

First, the conversion unit 23C creates a histogram as shown in FIGS. 16A and 16B based on the edge information extracted by the extraction unit 22, and calculates a representative value (step S301). After that, the conversion unit 23C performs a coding process for coding the representative value for each cell (step S302).

Subsequently, the matching unit 24C performs a matching process on the encoded grayscale image using a regular expression (step S303). Then, the detection unit 25C detects water droplets as shown in FIG. 18 (step S304).

As described above, in the deposit detection device 1C according to the third embodiment, each pixel is encoded and a matching process is performed using a regular expression. As a result, the matching process can be simplified. That is, water droplets can be detected accurately while suppressing the processing load.

Further, in the deposit detection device 1C according to the third embodiment, by using a regular expression for the matching process, it is possible to improve the detection accuracy of water droplets having different sizes and water droplets that can be seen from the camera image.

It should be noted that the deposit detection devices 1, 1B and 1C according to the first to third embodiments can also be used in appropriate combinations. For example, the method of calculating the representative value shown in FIG. 16A may be used in the first and second embodiments.

Further, in the deposit detection devices 1, 1B and 1C according to the first to third embodiments, the frame interval for acquiring a camera image from the camera 10 can be changed according to the purpose of detecting water droplets. For example, when presenting a camera image to the driver when the vehicle is moving backward, it is necessary to detect water droplets as soon as possible.

Therefore, in such a case, all the frames imaged by the camera 10 are acquired, and water droplets are detected from all the frames. On the other hand, for example, when the detection purpose is a sensing application such as automatic parking, the camera image may be acquired every few frames, for example.

Further, in such a case, the camera 10 for acquiring the camera image such as the rear camera → the front camera → the right camera → the left camera may be switched and acquired for each frame.

Further, in the deposit detection devices 1, 1B and 1C according to the first to third embodiments, the resolution of the camera image may be changed to perform the water droplet detection process. For example, when the resolution is lowered to detect water droplets, the processing load of the detection process can be reduced. The resolution can be changed according to the purpose of detecting water droplets.

(Fourth Embodiment)
Next, as a fourth embodiment, the removal is configured to include the functions of the deposit detection devices 1, 1B and 1C described so far, and controls the entire process from the detection of the deposit to the removal of the deposit. The configuration of the control device 50 will be described with reference to FIGS. 21 to 25.

FIG. 21 is a block diagram of the deposit removing system 100 according to the present embodiment. In FIG. 21, only the components necessary for explaining the features of the present embodiment are represented by functional blocks, and the description of general components is omitted.

In other words, each component shown in FIG. 21 is a functional concept and does not necessarily have to be physically configured as shown. For example, the specific form of distribution / integration of each functional block is not limited to the one shown in the figure, and all or part of the functional blocks are functionally or physically distributed in arbitrary units according to various loads and usage conditions. -It is possible to integrate and configure.

Further, FIG. 22A is a diagram showing an example of the content of notification from the deposit detection unit 61. Further, FIG. 22B is a diagram showing an example of data contents related to the detection area Da included in the detection information DB (database) 71. Further, FIG. 22C is an explanatory diagram of the state of the detection area Da.

As shown in FIG. 21, the deposit removing system 100 includes a camera 10, a deposit removing device 11, and a removal control device 50. Since the camera 10 and the deposit removing device 11 have already been described, the description thereof will be omitted here.

The removal control device 50 includes a control unit 60 and a storage unit 70. The control unit 60 instructs the plurality of deposit detection units 61, the exclusion unit 62, the algorithm-to-algorithm overlap determination unit 63, the inter-frame overlap determination unit 64, the adhesion determination unit 65, and the removal necessity determination unit 66. A unit 67 is provided.

The storage unit 70 is a storage device such as a hard disk drive, a non-volatile memory, or a register, and stores the detection information DB 71.

The control unit 60 controls the entire removal control device 50. Each of the plurality of deposit detection units 61 acquires a camera image for one frame from the camera 10, and uses a detection algorithm associated with each of the detection areas Da in which it is estimated that deposits are present in the camera image. Is extracted. Further, the deposit detection unit 61 notifies the exclusion unit 62 of the extracted detection area Da.

Here, as shown in FIG. 22A, the notification content from the deposit detection unit 61 includes, for example, the upper left coordinates (x, y) of the detection area Da extracted in a rectangular shape, the width w, and the height h. included.

The exclusion unit 62 performs image analysis on each of the detection areas Da notified from the deposit detection unit 61, and determines whether or not the deposits presumed to exist in the detection area Da are really deposits.

Further, the exclusion unit 62 notifies the inter-algorithm overlap determination unit 63 of the detection area Da that is determined to be an deposit based on the determination result. On the other hand, the exclusion unit 62 does not notify the inter-algorithm overlap determination unit 63 of the detection area Da that is determined to be non-adhesive based on the determination result, and excludes the detection area Da from the processing target in the subsequent stage. By excluding the useless image area in this way, the accuracy of detecting the deposits can be improved and the processing load in the subsequent stage can be reduced.

For example, the exclusion unit 62 generates a histogram classified into three classes of “weak”, “medium”, and “strong” for each of the edge intensity, brightness, and saturation of the detection area Da. Then, the exclusion unit 62 determines whether or not it is a deposit based on the ratio of the frequency of each class in each of the generated histograms, and excludes the detection area Da that is determined not to be a deposit.

The inter-algorithm overlap determination unit 63 determines the overlap of the detection areas Da between the plurality of algorithms in the current frame, that is, the existence or nonexistence of the overlap portion between the detection areas Da extracted by each of the deposit detection units 61. Further, the inter-algorithm overlap determination unit 63 reflects the determination result as a “score” of each detection area Da. The reflection result is managed on the detection information DB 71. The details of the determination process executed by the inter-algorithm overlap determination unit 63 will be described later with reference to FIGS. 23A to 23D.

The inter-frame overlap determination unit 64 determines whether or not there is an overlap with the detection area Da extracted in the past frame for all the processing results of the algorithm inter-frame overlap determination unit 63 related to the current frame. Further, the inter-frame overlap determination unit 64 reflects the determination result in the "score" and "state" of each detection area Da. The reflection result is managed on the detection information DB 71.

Here, as shown in FIG. 22B, the detection information DB 71 includes, for example, a "detection area ID" item, an "area information" item, a "score" item, and a "state" item. The identifier of the detection area Da is stored in the "detection area ID" item, and the detection information DB 71 is managed for each such detection area ID.

In the "area information" item, the upper left coordinates (x, y) of the detection area Da shown in FIG. 22A, the width w, the height h, and the like are stored. In the "score" item, the current score of each detection area Da is stored. The "state" item stores the current state of each detection area Da.

As shown in FIG. 22C as a state machine diagram, each detection area Da can transition to four states of "IDLE", "latent", "observation", and "penalty". "IDLE" refers to an "undetected state", that is, a state in which no deposits are attached. "Hidden" refers to the state of "possible deposits".

"Observation" refers to the "observation state after the removal process" in which the deposit removal operation is performed by the deposit removal device 11. The "penalty" refers to "a state in which deposits are continuously detected in the area even after the removal process", that is, a state of poor removal or false detection.

The inter-frame overlap determination unit 64 updates the "score" of each detection area Da on the detection information DB 71 according to the determination result of the determination, and transitions the "state".

Returning to the description of FIG. The adhesion determination unit 65 determines the "adhesion confirmation" of the adhered matter according to the "state" and the "score" of the detection area Da of the detection information DB 71.

The removal necessity determination unit 66 determines whether or not to actually cause the deposit removal device 11 to perform the deposit removal operation when the adhesion determination unit 65 determines that the adhesion is confirmed. Details of the processes executed by the frame-to-frame overlap determination unit 64, the adhesion determination unit 65, and the removal necessity determination unit 66 will be described later with reference to FIGS. 24A to 24E.

When the removal necessity determination unit 66 determines that the removal of the deposit is necessary, the instruction unit 67 generates an instruction signal for causing the deposit removal device 11 to perform the removal operation, and outputs the instruction signal to the deposit. It is sent to the removal device 11.

Next, the details of the determination process executed by the inter-algorithm overlap determination unit 63 will be described with reference to FIGS. 23A to 23D. 23A to 23D are processing explanatory views (No. 1) to (No. 4) of the inter-algorithm overlap determination unit 63.

As already described above, as shown in FIG. 23A, the inter-algorithm overlap determination unit 63 determines whether or not the detection area Da overlaps between a plurality of algorithms in the current frame. Specifically, as shown in FIG. 23A, for example, the overlap of all the detection areas Da-1 of the deposit detection algorithm-1 and all the detection areas Da-2 of the deposit detection algorithm-2 is determined. This also applies between the deposit detection algorithm-1 and the deposit detection algorithm-3, or between the deposit detection algorithm-2 and the deposit detection algorithm-3.

As shown in FIG. 23B, for example, the overlap between the detection area Da-1 and the detection area Da-2 is determined by the distance d from the respective center of gravity.

Then, as shown in FIG. 23C, when the inter-algorithm overlap determination unit 63 determines that there is an overlap between the detection area Da-1 and the detection area Da-2, for example, the detection area Da-1 and the detection area Da-2. Add points to each Da-2 score.

As a result, it can be shown that the detection area Da-1 and the detection area Da-2 having the overlap are more likely to have the deposits than the detection area Da where the overlap does not exist.

Further, as shown in FIG. 23D, for example, when the detection area Da-1 and the detection area Da-2 have overlaps, the inter-algorithm overlap determination unit 63 of the detection area Da-1 and the detection area Da-2 Update area information.

For example, as shown in FIG. 23D (a), the detection area Da-1 is prioritized and the area information is integrated into the detection area Da-1. Further, as shown in FIG. 23D (b), conversely, the detection area Da-2 is prioritized and the area information is integrated into the detection area Da-2.

Further, as shown in FIG. 23D (c), the area information is integrated into the detection area Da-A having only the overlapping portion by taking a logical product. Further, as shown in FIG. 23D (d), the area information may be integrated into the detection area Da-S corresponding to the logical sum of the detection area Da-1 and the detection area Da-2.

Further, as shown in FIG. 23D (e), the area information may be integrated into the detection area Da-E expanded so as to include both the detection area Da-1 and the detection area Da-2.

Next, the details of the processes executed by the frame-to-frame overlap determination unit 64, the adhesion determination unit 65, and the removal necessity determination unit 66 will be described with reference to FIGS. 24A to 24E. 24A to 24C are processing explanatory views (No. 1) to (No. 3) of the frame-to-frame overlap determination unit 64.

Further, FIG. 24D is a processing explanatory view of the frame-to-frame overlap determination unit 64 and the adhesion determination unit 65. Further, FIG. 24E is a processing explanatory view of the removal necessity determination unit 66.

As already described above, as shown in FIG. 24A, the inter-frame overlap determination unit 64 sets each detection area Da that has been extracted in the past frame for all the processing results of the inter-algorithm overlap determination unit 63 related to the current frame. Judge the existence or nonexistence of the overlap of.

Specifically, as shown in FIG. 24B, it is determined that all of the detection areas Da-C of the current frame and all of the detection areas Da-P of the past frame overlap. The concept of overlap may be the same as in the case of the inter-algorithm overlap determination unit 63.

Then, as shown in FIG. 24B, when the inter-frame overlap determination unit 64 determines that there is an overlap between the detection area Da-C of the current frame and the detection area Da-P of the past frame, the detection area Da Points are added to the scores of -C and detection area Da-P.

As a result, the frame-to-frame overlap determination unit 64 can indicate, for example, deposits existing in substantially the same region of the camera 10 across the frames.

On the other hand, as shown in FIG. 24C, the inter-frame overlap determination unit 64 deducts points for the detection area Da-P of the past frame if there is no overlap with any of the detection areas Da-C of the current frame. ..

Further, the inter-frame overlap determination unit 64 considers that the detection area Da-C of the current frame has no overlap with any of the detection areas Da-P of the past frame as a new detection area Da, and detects the detection information. Newly register in DB71.

As shown in FIG. 24D, the newly registered detection area Da is in the "latent" state and is given a predetermined score. Then, according to the score of the detection area Da that changes from the "latent" state through the above-mentioned "point addition" and "point deduction", the frame-to-frame overlap determination unit 64 and the adhesion determination unit 65 are in the state of the detection area Da. To transition.

For example, as shown in FIG. 24D, when the score of the detection area Da in the “latent” state becomes equal to or less than a predetermined point, the inter-frame overlap determination unit 64 changes the state of the detection area Da from “latent” to “IDLE”. (Step S11). As a result, it is possible to prevent an erroneous reaction in which the deposits such as raindrops that move by flowing down and need not be removed are removed.

Further, when the score of the detection area Da in the "latent" state becomes a predetermined point or more, the adhesion determination unit 65 confirms (determines the adhesion) the adhesion of the deposit to the area (step S12).

Further, the adhesion determination unit 65 shifts all the detection area Das in the “latent” state to the “observation” state after the adhesion is confirmed (step S13). This means that if the removal process is performed according to the confirmation of adhesion of one detection area Da, the deposits are normally removed even for the other detection area Da in the "latent" state where the adhesion has not been confirmed. This is because it is presumed that it was done.

The inter-frame overlap determination unit 64 shifts the detection area Da to the "penalty" state when the removal process is performed and the score of the detection area Da in the "observation" state becomes a predetermined point or more (step S14). ). As a result, it is possible to grasp the removal failure or false detection in which the deposits are continuously detected even after the removal treatment.

Further, when the score of the detection area Da in the “observation” state or the “penalty” state becomes equal to or less than a predetermined point, the frame-to-frame overlap determination unit 64 shifts the detection area Da to the “IDLE” state (step S15). ..

The reaction rate until the adhesion is confirmed may be controlled by adjusting the slopes of the arrows indicating "point addition" and "point deduction" in FIG. 24D. For example, by increasing the amount of points added and the amount of points deducted and making the inclination of the arrow steep, the reaction speed from the detection of the deposit to the removal process can be increased.

Further, even in the detection area Da where the adhesion is confirmed, the removal operation may be intentionally not performed. For example, as shown in FIG. 24E, the removal necessity determination unit 66 determines that the removal process does not need to be executed if the detection area Da for which the adhesion is confirmed exists in the skip area substantially along the outer circumference of the screen. can do.

In this way, the processing load of the entire system can be reduced by skipping the removal processing itself for the deposits adhering to the image area that have little influence on the passenger's visual recognition and driving operation.

Next, the processing procedure executed by the deposit removing system 100 according to the present embodiment will be described with reference to FIG. FIG. 25 is a flowchart showing a processing procedure executed by the deposit removing system 100.

First, the plurality of deposit detection units 61 each acquire a camera image for one frame (step S401). Then, for example, the deposit detection unit 61a extracts the detection area Da-1 using the deposit detection algorithm-1 (step S402).

Further, for example, the deposit detection unit 61b extracts the detection area Da-2 using the deposit detection algorithm-2 (step S403). Further, for example, the deposit detection unit 61c extracts the detection area Da-3 using the deposit detection algorithm-3 (step S404).

Then, the exclusion unit 62 performs an exclusion process for each of the detection areas Da extracted and notified by the deposit detection unit 61 (step S405). That is, the exclusion unit 62 determines whether or not the deposit presumed to exist in the detection area Da is really a deposit, and if it is not a deposit, the detection area Da corresponding to the deposit is removed from the processing target in the subsequent stage. exclude.

The exclusion process itself may be omitted, for example. As a result, the processing load on the entire system can be reduced.

Subsequently, the inter-algorithm overlap determination unit 63 performs the inter-algorithm overlap determination process (step S406). That is, the inter-algorithm overlap determination unit 63 determines whether or not the detection area Da overlaps between the plurality of algorithms in the current frame, and updates the score of the detection area Da according to the determination result.

Then, the inter-frame overlap determination unit 64 performs the inter-frame overlap determination process (step S407). That is, the inter-frame overlap determination unit 64 determines whether or not there is an overlap with each detection area Da extracted in the past frame for all the processing results of the inter-algorithm overlap determination unit 63, and detects according to the determination result. Update the score and status of Area Da.

Then, the adhesion determination unit 65 performs the adhesion determination process (step S408). That is, the adhesion determination unit 65 determines the adhesion determination of the adhered matter according to the score and the state of the detection area Da of the detection information DB 71 updated by the frame-to-frame overlap determination unit 64.

Then, when the removal necessity determination unit 66 determines that the adhesion determination unit 65 has “determined adhesion”, it determines whether or not it is actually necessary to remove the deposits by the deposit removal device 11 (step S409). ..

Here, when it is determined that "removal is required" (step S409, Yes), the instruction unit 67 sends an instruction signal to the deposit removal device 11 to cause the deposit removal device 11 to perform the removal operation. The removal process is executed (step S410). On the other hand, if it is determined that "removal is not required" (step S409, No), the instruction unit 67 does not execute the removal process.

Then, the control unit 60 determines whether or not there is a processing end event (step S411). The processing end event corresponds to, for example, IG off or ACC off. Here, when it is determined that there is no processing end event (steps S411, No), the processing from step S401 is repeated. If it is determined that there is a processing end event (steps S411, Yes), the deposit removing system 100 ends the processing.

As described above, in the fourth embodiment, a plurality of deposit detection algorithms are used to determine water droplets adhering to the camera 10. Therefore, it is possible to suppress erroneous detection of water droplets and improve the reliability of water droplet detection accuracy.

By the way, in the above-described embodiment, the case where the deposit detection devices 1, 1B and 1C extract the gradient of the brightness of each pixel of the camera image L as edge information has been described, but the present invention is not limited to this.

The deposit detection devices 1, 1B and 1C can also extract the saturation gradient of each pixel of the camera image L as edge information and detect water droplets adhering to the camera 10 based on the edge information. In such a case, the deposit detection devices 1, 1B and 1C can accurately detect turbid water droplets mixed with mud, sand and the like adhering to the camera 10.

Specifically, for example, when the extraction unit 22 of the deposit detection devices 1, 1B and 1C is based on the HSV color space, the maximum values of R, G and B of each pixel of the camera image L are set to Imax and the minimum values are set to the minimum value. When Imin is set, the saturation can be extracted by saturation (S) = (Imax-Imin) / Imax.

Further, when based on the HSL color space, the extraction unit 22 sets the brightness (L) = (Imax + Imin) / 2, and when L ≦ 0.5, the saturation (S) = (Imax-Imin) / (Imax + Imin). , L> 0.5, the saturation can be extracted by saturation (S) = (Imax-Imin) / (2-Imax-Imin).

Subsequently, the conversion unit 23 according to the first embodiment calculates the value obtained by squaring the saturations in the X-axis direction and the Y-axis direction and then adding them as the edge amount based on the saturation of each pixel. Then, the conversion unit 23 can binarize each pixel of the camera image L by comparing the edge amount with the binarization threshold value THa as shown in FIG. As the binarization threshold value THa, one optimized for saturation can be used.

After that, the deposit detection device 1 can accurately detect the turbid water droplets adhering to the camera 10 by performing the processes already described by the matching unit 24 and the detection unit 25.

Further, the conversion unit 23B according to the second embodiment calculates the direction of the edge based on the saturation of each pixel, and assigns the three-dimensional parameters shown in FIGS. 11A and 11B to the edge direction. Pixels can be parameterized.

Subsequently, the matching unit 24B performs a matching process using the template shown in FIG. At this time, if the water droplets are muddy, the saturation increases toward the center. Therefore, the detection unit 25B can detect turbid water droplets only when the value of the zero average normalized cross-correlation calculated by the matching unit 24B exceeds the positive detection threshold shown in FIG.

That is, the deposit detection device 1B according to the second embodiment can suppress erroneous detection of turbid water droplets by setting only a positive detection threshold value when detecting turbid water droplets.

Further, the conversion unit 23C according to the third embodiment encodes each pixel of the camera image L with respect to the direction of the saturation gradient as shown in FIGS. 16A and 16B. Then, the matching unit 24C extracts a code string indicating a water droplet by a matching process using a regular expression.

Then, the detection unit 25C detects turbid water droplets adhering to the camera 10 based on the code string extracted by the matching unit 24C. At this time, as described above, if the water droplet is turbid, the saturation increases toward the center.

Therefore, the detection unit 25C can accurately detect turbid water droplets by detecting the pattern of the code string whose saturation increases toward the center based on the code string extracted by the matching unit 24C. ..

As described above, the deposit detection devices 1, 1B and 1C can accurately detect turbid water droplets by using saturation instead of the brightness of each pixel as edge information. The deposit detection devices 1, 1B and 1C simultaneously detect, for example, water droplets whose brightness increases toward the center and turbid water droplets whose saturation increases toward the center by a single matching process. It may be.

Further, in the above-described embodiment, the case where the deposit detection devices 1, 1B and 1C and the deposit removal system 100 are all applied to the in-vehicle camera 10 has been shown, but for example, they are set inside or outside the building or in an alley. It may be applied to other types of cameras such as surveillance / security cameras.

Further effects and variations can be easily derived by those skilled in the art. For this reason, the broader aspects of the invention are not limited to the particular details and representative embodiments described and described above. Therefore, various changes are possible without departing from the spirit or scope of the overall concept of the invention as defined by the appended claims and their equivalents.

1,1B, 1C deposit detection device
10 Camera (imaging device)
11 Adhesion removal device
21 Image acquisition section
22 Extraction unit 23, 23B, 23C Conversion unit 24, 24B, 24C Matching unit 25, 25B, 25C Detection unit
50 Removal control device
61 Adhesion detector
62 Exclusion part
63 Overlap judgment unit between algorithms
64 Frame overlap judgment unit
65 Adhesion judgment unit
66 Removal necessity judgment unit
67 Indicator
100 deposit removal system

Claims (3)

Extracts the edge information of each pixel contained in an image taken in an ambient environment where the ambient illuminance changes by an image sensor having an image sensor composed of a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Extraction unit and
A conversion unit that converts each pixel into a predetermined data format based on the edge information extracted by the extraction unit, and a conversion unit.
A matching unit that performs matching processing between each pixel converted into the data format by the conversion unit and a template of the data format indicating water droplets.
It is provided with a detection unit that detects water droplets adhering to the image pickup apparatus based on the matching result by the matching unit.
The conversion unit
The direction of the edge of each pixel is calculated based on the edge information extracted by the extraction unit, and each pixel is converted into a code by assigning a code corresponding to the direction of the edge.
The matching unit
A deposit detection device, characterized in that a matching process is performed using a regular expression of each pixel converted into the code by the conversion unit and a code string indicating the water droplet. The matching unit
A claim characterized in that an array of the codes satisfying the arrangement order defined by the code strings and containing at least one of the codes is extracted as the code strings from the pixels encoded by the conversion unit. Item 2. The deposit detection device according to Item 1 . Extracts the edge information of each pixel contained in an image taken in an ambient environment where the ambient illuminance changes by an image sensor having an image sensor composed of a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). Extraction process and
A conversion step of converting each pixel into a predetermined data format based on the edge information extracted by the extraction step, and a conversion step.
A matching step of performing matching processing between each pixel converted into the data format by the conversion step and a template of the data format indicating water droplets.
Including a detection step of detecting water droplets adhering to the image pickup apparatus based on the matching result of the matching step.
The conversion step is
By calculating the edge strength of each pixel based on the edge information extracted by the extraction step and comparing the edge strength with a binarization threshold that takes a different value according to the surrounding environment of the imaging device. A method for detecting deposits, which comprises binarizing each pixel.