JP6755161B2 - Adhesion detection device and deposit detection method - Google Patents

Description

The disclosed embodiments relate to a deposit detection device and a deposit detection method.

Conventionally, an in-vehicle camera that is mounted on a vehicle and images the surroundings of the vehicle has been known. The image captured by the in-vehicle camera is displayed on a monitor to assist the driver's field of vision, or is used for sensing to detect a white line on the road or an approaching object to the vehicle.

It should be noted that, for example, raindrops, snowflakes, dust, mud, and other deposits may adhere to the lens of the in-vehicle camera, which may interfere with the above-mentioned visual field assistance and sensing. Therefore, a technique for removing deposits by injecting cleaning water or compressed air toward the lens of an in-vehicle camera has been proposed. For such a technique, for example, a detection algorithm for detecting deposits on a lens by image analysis of an image captured by an in-vehicle camera can be used (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2001-141838

However, there is room for further improvement in the above-mentioned prior art in terms of further improving the detection accuracy of deposits.

For example, the above-mentioned detection algorithm may detect an edge from a captured image and cut out the outline of the deposit based on the edge. However, for example, raindrops may bleed and the outline may be blurred, so that the outline is accurate. It may not be detected.

Further, even if the outline of the raindrop is clear, for example, a structure having a shape similar to that of the raindrop may be erroneously detected as a raindrop.

One aspect of the embodiment is made in view of the above, and an object of the present invention is to provide a deposit detection device and a deposit detection method capable of further improving the detection accuracy of deposits.

The deposit detection device according to one aspect of the embodiment includes an acquisition unit, a generation unit, and a determination unit. The acquisition unit acquires an area to be determined for deposits in the captured image. The generation unit generates at least a histogram of edge intensity, brightness, and saturation for the determination target area acquired by the acquisition unit. The determination unit determines the presence or absence of the deposit in the determination target area based on the ratio of the frequency of each class of each of the histograms generated by the generation unit.

According to one aspect of the embodiment, the accuracy of detecting deposits can be further improved.

FIG. 1A is a schematic explanatory view (No. 1) of the deposit detection method according to the embodiment. FIG. 1B is a schematic explanatory view (No. 2) of the deposit detection method according to the embodiment. FIG. 1C is a schematic explanatory view (No. 3) of the deposit detection method according to the embodiment. FIG. 1D is a schematic explanatory view (No. 4) of the deposit detection method according to the embodiment. FIG. 2A is a schematic view of the description of the first embodiment. FIG. 2B is a schematic view of the description of the second embodiment. FIG. 3 is a block diagram of the deposit removing system according to the first embodiment. FIG. 4A is a diagram (No. 1) showing a specific example of the exclusion condition. FIG. 4B is a diagram (No. 2) showing a specific example of the exclusion condition. FIG. 4C is a diagram (No. 3) showing a specific example of the exclusion condition. FIG. 5A is an explanatory diagram (No. 1) of a modified example of the exclusion condition. FIG. 5B is an explanatory diagram (No. 2) of a modified example of the exclusion condition. FIG. 5C is an explanatory diagram (No. 3) of a modified example of the exclusion condition. FIG. 6 is a diagram showing a specific example of adjusting the exclusion condition. FIG. 7 is a flowchart showing a processing procedure executed by the deposit detection device according to the first embodiment. FIG. 8A is a diagram (No. 1) showing a setting example of the divided area. FIG. 8B is a diagram (No. 2) showing an example of setting the divided area. FIG. 8C is a diagram (No. 3) showing a setting example of the divided area. FIG. 8D is a diagram (No. 4) showing a setting example of the divided area. FIG. 9 is a diagram showing a specific example of the retained information according to the second embodiment. FIG. 10 is a diagram showing a specific example of detection conditions. FIG. 11 is a diagram showing an example of a parameter setting screen. FIG. 12A is a diagram (No. 1) showing a specific example of the determination method when the amount of change before the past two generations is included. FIG. 12B is a diagram (No. 2) showing a specific example of the determination method when the amount of change before the past two generations is included. FIG. 13 is a diagram showing a specific example of adjusting the detection conditions. FIG. 14 is a flowchart showing a processing procedure executed by the deposit detection device according to the second embodiment. FIG. 15 is a block diagram of the deposit removing system according to the third embodiment. FIG. 16A is a diagram showing an example of the content of notification from the deposit detection unit. FIG. 16B is a diagram showing an example of data contents relating to the detection area included in the detection information DB. FIG. 16C is an explanatory diagram of the state of the detection area. FIG. 17A is a processing explanatory diagram (No. 1) of the inter-algorithm overlap determination unit. FIG. 17B is a processing explanatory diagram (No. 2) of the inter-algorithm overlap determination unit. FIG. 17C is a processing explanatory diagram (No. 3) of the inter-algorithm overlap determination unit. FIG. 17D is a processing explanatory diagram (No. 4) of the inter-algorithm overlap determination unit. FIG. 18A is a processing explanatory view (No. 1) of the frame-to-frame overlap determination unit. FIG. 18B is a processing explanatory view (No. 2) of the frame-to-frame overlap determination unit. FIG. 18C is a processing explanatory view (No. 3) of the frame-to-frame overlap determination unit. FIG. 18D is a processing explanatory view of the frame-to-frame overlap determination unit and the adhesion determination unit. FIG. 18E is a processing explanatory view of the removal necessity determination unit. FIG. 19 is a flowchart showing a processing procedure executed by the deposit removing system according to the third embodiment.

Hereinafter, embodiments of the deposit detection device and the deposit detection method disclosed in the present application will be described in detail with reference to the accompanying drawings. The present invention is not limited to the embodiments shown below.

Further, in the following, after explaining the outline of the deposit detection method according to the present embodiment with reference to FIGS. 1A to 1D, the deposit detection device 10 to which the deposit detection method according to the embodiment is applied will be described in FIGS. 2A and later. Will be described using.

First, the outline of the deposit detection method according to the present embodiment will be described with reference to FIGS. 1A to 1D. 1A to 1D are schematic explanatory views (No. 1) to (No. 4) of the deposit detection method according to the embodiment.

As shown in FIG. 1A, the vehicle C is equipped with an in-vehicle camera such as a front camera 21, a back camera 22, a right side camera 23, and a left side camera 24 in order to image the periphery of the vehicle C. In the following, when these in-vehicle cameras are collectively referred to, they are referred to as "camera 2".

The camera 2 includes an image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and the image pickup element captures the periphery of the vehicle C. Then, the camera 2 outputs the captured image to, for example, the deposit removing system 1 including the deposit detecting device 10 according to the present embodiment.

A wide-angle lens such as a fisheye lens is used for the lens 2a of the camera 2 (see FIG. 1B), and each camera 2 has an angle of view of 180 degrees or more, and by using these, the entire circumference of the vehicle C is imaged. It is possible to do.

Further, as shown in FIG. 1B, the deposit removing system 1 according to the present embodiment includes a deposit removing device 3 for removing deposits such as raindrops, snowflakes, dust, and mud adhering to the lens 2a of the camera 2. ..

The deposit removing device 3 includes a nozzle 3a. The nozzle 3a is provided with a discharge port facing the lens 2a. For example, the compressed air supplied through the compressed air supply source 3b and the valve 3c and the cleaning liquid supplied via the cleaning liquid supply source 3d and the valve 3e are lensed. The deposits are removed by spraying toward 2a.

The operation control of the deposit removing device 3 is performed by the removal determining device 5 included in the deposit removing system 1. The removal determination device 5 automatically determines whether or not the deposits are attached to the lens 2a and it is necessary to remove the deposits based on the detection result by the deposit detection device 10, and the removal is necessary. If this is the case, the deposit removing device 3 is made to perform the removing operation.

Then, in order to further improve the detection accuracy of the deposits contributing to the automatic determination, the deposit detection device 10 according to the present embodiment has at least the edge strength, the brightness and the saturation of each pixel in the area to be determined for the presence or absence of the deposits. It was decided to generate a histogram of the above and determine the deposits based on the frequency of each class of the histogram.

Specifically, as shown in FIG. 1C, the deposit detection device 10 is an area where the presence of deposits is estimated, which is detected from the camera image of the camera 2, for example, by using a plurality of detection algorithms. Acquire the area (step S1).

Then, the deposit detection device 10 generates a histogram classified into three classes, for example, "weak", "medium", and "strong" for each of the edge strength, brightness, and saturation of the acquired detection area (step). S2). A specific example of how to obtain the edge strength, the brightness, and the saturation will be described in the description of the histogram generation unit 11b (described later).

Then, the deposit detection device 10 determines whether or not the deposit presumed to exist in the detection area is really a deposit based on the “ratio” of the frequency of each class of each generated histogram (step S3). ). Specifically, if the "ratio" of the frequency satisfies a predetermined exclusion condition to be excluded as a deposit, it is determined that the deposit estimated in the detection area is not a deposit.

For example, FIG. 1C shows a case where the tire portion of another vehicle is detected as a detection area, but the deposit detection device 10 generates the above-mentioned histogram for the detection area, and each frequency of each histogram is generated. The "ratio" of is compared to the prescribed exclusion conditions.

As a result, the detection area, which is the tire portion, satisfies a predetermined exclusion condition. Therefore, the deposit detection device 10 determines that the deposit estimated in the detection area is not a deposit. Specific examples of the exclusion conditions will be described later with reference to FIGS. 4A to 4C.

Then, the deposit detection device 10 excludes the detection area determined not to be a deposit from the processing target of the subsequent removal determination device 5 because it is an erroneous detection portion. That is, the deposit detection device 10 does not notify the removal determination device 5 of the detection area.

As a result, the processing load of the removal determination process in the removal determination device 5 can be reduced. Further, by performing the false detection determination process by the deposit detection device 10 on the detection area detected by using the plurality of detection algorithms in this way, the false detection by each detection algorithm can be supplemented, and the deposit can be supplemented. It can contribute to further improving the detection accuracy of.

Here, among the elements of the detection area, an example focusing on edge intensity, brightness, and saturation is given, but hue and standard deviation may be used, and the elements for which the histogram is generated are not limited. .. Also, the class is not limited to "weak", "medium", and "strong".

By the way, in FIG. 1C, the case where the deposit detection device 10 is positioned as an auxiliary to each detection algorithm is taken as an example, but the deposit detection device 10 is also configured to execute one of the detection algorithms. can do.

In such a case, specifically, as shown in FIG. 1D, the deposit detecting device 10 sets a plurality of divided areas for one frame of the camera image of the camera 2 and acquires each of the divided areas (step). S1').

Then, the deposit detection device 10 generates a histogram of each of the acquired edge strength, brightness, and saturation of each divided area (step S2'). Information including such a histogram is retained not only for the current frame but also for the past frames of one generation or more.

Then, the deposit detection device 10 determines the presence or absence of deposits in each divided area based on the "change amount" between the frames for the current frame and the frames for the past frame (step S3'). Note that FIG. 1D illustrates an example of edge strength. The deposit detecting device 10 determines the presence or absence of a deposit based on the tendency shown by, for example, the change in the current edge strength, the change in the past one generation, and the change in the past two generations.

By making a judgment based on the "change amount" between the frames, for example, even in the case of raindrops whose contours are blurred and difficult to detect as edges, deposits are based on the characteristics of the raindrops appearing in the "change amount". Can be easily detected as. That is, it can contribute to further improving the detection accuracy of deposits. Further, by performing the determination for each divided area, it is possible to detect the deposits having higher resolution than the entire frame. Therefore, it can also contribute to further improving the detection accuracy of deposits.

Further, the detection condition based on the "change amount" can be set for each divided area. This makes it possible to make an appropriate determination according to the characteristics of each divided area, such as the susceptibility to change.

Details of the processing, such as the detection conditions when the "change amount" is used, will be described later with reference to FIGS. 9 to 13.

Hereinafter, an embodiment of the deposit detection device 10 to which the above-mentioned deposit detection method is applied will be described in more detail.

In the following, the example shown in FIG. 1C will be described as the first embodiment, and the example shown in FIG. 1D will be described as the second embodiment. However, from the viewpoint of making the explanation easy to understand, FIGS. FIG. 2B shows an outline of the following description.

As shown in FIG. 2A, in the description of the first embodiment, the deposit detection device 10 executes the deposit detection algorithms -1, -2, ...- N, and the external deposit detection device 4 (see FIG. 3). Is used as the first stage, and the removal judgment device 5 that executes the removal judgment processing is used as the second stage, and a configuration in which the false detection determination processing (exclusion of the false detection portion) is performed is given as an example (see FIG. 1C).

Further, as shown in FIG. 2B, in the description of the second embodiment, a configuration in which the deposit detection device 10 executes one of the deposit detection algorithms -1, -2, ...- n will be given as an example ( See FIG. 1D). The false detection determination process shown by the broken line may be performed in the configuration of the first embodiment or may be omitted.

(First Embodiment)
FIG. 3 is a block diagram of the deposit removing system 1 according to the present embodiment. Note that, in FIG. 3, only the components necessary for explaining the features of the present embodiment are represented by functional blocks, and the description of general components is omitted.

In other words, each component shown in FIG. 3 is a functional concept and does not necessarily have to be physically configured as shown. For example, the specific form of distribution / integration of each functional block is not limited to the one shown in the figure, and all or part of the functional blocks are functionally or physically distributed in arbitrary units according to various loads and usage conditions. -It is possible to integrate and configure.

As shown in FIG. 3, the deposit removing system 1 includes a camera 2, a deposit removing device 3, and one or more external deposit detecting devices 4 (for example, external deposit detecting devices 4-1 and -2, -3 ...), The removal determination device 5, and the deposit detection device 10.

As shown in FIG. 3, the external deposit detection device 4, the deposit detection device 10, and the removal determination device 5 are the removal control device 50 that controls the entire process from the detection of the deposit to the removal of the deposit. It can be a component. The specific configuration of the removal control device 50 will be described later in the third embodiment after describing the present embodiment and the second embodiment.

Since the camera 2 and the deposit removing device 3 have already been described, the description thereof will be omitted here. The external deposit detection device 4 acquires the camera image for each frame from the camera 2, and extracts the detection area where the deposit is presumed to exist from the camera image by using the detection algorithm associated with each frame. , The extracted detection area is notified to the deposit detection device 10.

As described above, the removal determination device 5 causes the deposit removing device 3 to perform the removing operation when it is necessary to remove the deposit based on the detection result by the deposit detecting device 10.

The deposit detection device 10 includes a control unit 11 and a storage unit 12. The control unit 11 includes a target area acquisition unit 11a, a histogram generation unit 11b, a deposit determination unit 11c, and a condition adjustment unit 11d.

The storage unit 12 is a storage device such as a hard disk drive, a non-volatile memory, or a register, and stores condition information 12a.

The control unit 11 controls the entire deposit detection device 10. The target area acquisition unit 11a acquires the detection area notified from the external deposit detection device 4 as the determination target area. In the second embodiment described later, the target area acquisition unit 11a acquires the divided area as the determination target area from the camera image of the camera 2.

The histogram generation unit 11b generates a histogram of at least edge intensity, brightness, and saturation of each pixel for each of the detection areas acquired by the target area acquisition unit 11a in a predetermined number of classes classified in advance. As already mentioned, the predetermined number of classes is, for example, "weak", "medium", and "strong".

Specifically, regarding the edge intensity, the histogram generation unit 11b converts the image corresponding to the detection area into a grayscale image by grayscale it. The grayscale conversion is a process of converting each pixel in the camera image so as to be expressed in each gradation from white to black according to the brightness.

Further, the histogram generation unit 11b extracts the edge information of each pixel in the grayscale image by using the sobel filter for the grayscale image. Here, the edge information refers to the edge strength of each pixel in the X-axis direction and the Y-axis direction. In addition, instead of the sobel filter, another edge extraction method such as a Laplacian filter may be used.

Further, the histogram generation unit 11b calculates the edge strength as a representative value of each pixel of the grayscale image based on the edge information of each extracted pixel. Specifically, the value obtained by squaring the edge strengths in the X-axis direction and the Y-axis direction, which are the edge information, and then adding them is calculated as a representative value of the edge strength of each pixel.

Then, the histogram generation unit 11b generates a histogram of the edge strength based on the calculated representative value of the edge strength of each pixel. Specifically, the histogram generation unit 11b normalizes each representative value of the calculated edge strength so as to take a value between 0 and 1, for example, and the predetermined class numbers are "weak" and "medium" as described above. , "Strong", for example, "weak" is 0 or more and less than 0.3, "medium" is 0.3 or more and less than 0.7, and "strong" is 0.7 or more and 1 or less. Histogram is generated.

Further, regarding the brightness, the histogram generation unit 11b calculates each pixel based on the RGB values (R: 0 to 255, G: 0 to 255, B: 0 to 255) of each pixel in the above-mentioned grayscale conversion. Use brightness. For example, the histogram generation unit 11b can extract only one element value of R, G, and B of each pixel as a representative value and use this as the brightness of each pixel.

Further, for example, the histogram generation unit 11b can calculate a simple average value of each element value of R, G, and B as the brightness. Further, for example, the histogram generation unit 11b is an average value weighted by using a so-called NTSC (National Television System Committee) weighted average method according to the formula “luminance = 0.298912 × R + 0.586611 × G + 0.114478 × B”. Can be calculated as the brightness.

Then, the histogram generation unit 11b generates a luminance histogram based on the calculated luminance of each pixel. The method of normalizing the brightness and dividing the predetermined number of classes is the same as in the case of the edge strength.

Regarding the saturation, for example, when the histogram generation unit 11b is based on the HSV color space, when the maximum values of R, G, and B are Imax and the minimum values are Imin, the brightness (V) = Imax and the saturation. Saturation can be calculated by (S) = (Imax-Imin) / Imax.

Further, for example, the histogram generation unit 11b sets the brightness (L) = (Imax + Imin) / 2 when based on the HSL color space, and when L ≦ 0.5, the saturation (S) = (Imax-Imin) /. When (Imax + Imin) and L> 0.5, the saturation can be calculated by the saturation (S) = (Imax-Imin) / (2-Imax-Imin). The brightness (V) or the brightness (L) may be used as the above-mentioned brightness.

Then, the histogram generation unit 11b generates a saturation histogram based on the calculated saturation of each pixel. The method of normalizing the saturation and dividing the predetermined number of classes is the same as in the case of edge strength and brightness.

The histogram generation unit 11b preferably scales the size of the detection area to a reference size when generating the histogram. As a result, it is possible to suppress variations in detection accuracy due to differences in size when comparing the ratio of the frequencies of each element in the detection area, which may have different sizes, to a predetermined exclusion condition. That is, it can contribute to further improving the detection accuracy of deposits.

The deposit determination unit 11c excludes whether or not the deposit estimated to be present in the detection area is really a deposit based on the frequency ratio of each class of the histogram generated by the histogram generation unit 11b. Judge in light of the conditions. The predetermined exclusion condition is a combination of the ratios of each of the three classes (“weak”, “medium”, and “strong”) of edge strength, brightness, and saturation so as not to match the characteristics of the deposit. , Is included in the condition information 12a in advance.

Here, FIGS. 4A to 4C show specific examples of predetermined exclusion conditions. 4A to 4C are diagrams (No. 1) to (No. 3) showing specific examples of exclusion conditions.

As shown in FIG. 4A, for example, as "exclusion condition-1", "strong" of "edge strength" is 50% or more, "weak" of "brightness" is 30% or more, and "saturation" is "saturation". The case where "strong" is 50% or more can be mentioned.

Further, as shown in FIG. 4B, for example, as "exclusion condition-2", "strong" of "edge strength" is 50% or more, "medium" of "brightness" is 50% or more, and "saturation". The case where the "weakness" of is 50% or more can be mentioned.

Further, as shown in FIG. 4C, for example, as "exclusion condition-3", a case where "medium" of "luminance" is 80% or more and "strong" of "saturation" is 80% or more is mentioned. be able to.

The exclusion conditions of FIGS. 4A to 4C are conditions obtained by verification tests and the like and do not conform to the characteristics of raindrops. By using these exclusion conditions of FIGS. 4A to 4C, among the deposits, In particular, it can contribute to further improving the detection accuracy of raindrops.

Returning to the description of FIG. Further, the deposit determination unit 11c satisfies the predetermined exclusion condition and excludes the detection area determined to be non-adhesion from the processing target in the subsequent stage. Further, the deposit determination unit 11c notifies the removal determination device 5 in order to set the detection area determined to be the deposit as the processing target in the subsequent stage without satisfying the predetermined exclusion condition.

By the way, so far, the case where the elements constituting the exclusion condition are "edge strength", "luminance" and "saturation" has been taken as an example, but as already mentioned, other elements other than these are used. It may be added.

A modified example of such an exclusion condition will be described with reference to FIGS. 5A to 5C. 5A to 5C are explanatory views (No. 1) to (No. 3) of a modified example of the exclusion condition.

For example, the exclusion condition can include the similarity obtained by matching the detection area with a predetermined template.

In such a modification, a case where attention is paid to the vector orientation of each pixel in the detection area will be given as an example. Specifically, as shown in FIG. 5A, in such a modification, for example, the histogram generation unit 11b converts the detection area Da into a vector-oriented image to generate a vector-oriented image Vd.

More specifically, the histogram generation unit 11b calculates the vector orientation of each pixel from the detection area Da, generates a vector orientation image Vd in which the vector orientation is colored based on, for example, the color wheel, and the generated vector orientation image. Notify the deposit determination unit 11c of Vd.

Then, the deposit determination unit 11c calculates, for example, the mutual correlation coefficient for all target pixels by template matching using the notified vector-oriented image Vd and the template Ti previously provided based on the hue circle. Then, the condition is determined including the similarity indicated by the mutual correlation coefficient.

The template Ti shown in FIG. 5A shows the characteristics of raindrops that are bright inward, for example. As shown in FIG. 5B, the template Ti includes the template Ti-1 of FIG. 5B (a) and the template Ti of the same (b) according to the size and shape of the detection area Da, the characteristics of how the raindrops shine, and the like. If various variations such as -2 are provided, it is possible to deal with various forms of deposits.

Then, when the above-mentioned intercorrelation coefficient is used, the similarity is shown in the range of -1 to 1, so that, for example, the exclusion condition of such a modification is "similar" as a requirement of + α as shown in FIG. 5C. Conditions such as degree <threshold value "and" similarity <0 "will be included.

As described above, by including not only the ratio of the frequency of the histogram but also the similarity of the template matching result in the exclusion condition, it is possible to contribute to further improving the detection accuracy of the deposit.

Returning to the description of FIG. The condition adjusting unit 11d adjusts the exclusion condition and appropriately updates the condition information 12a when a predetermined trigger suitable for adjusting the exclusion condition is generated.

Such a specific example will be described with reference to FIG. FIG. 6 is a diagram showing a specific example of adjusting the exclusion condition. As shown in FIG. 6, as a trigger for adjusting the exclusion condition, for example, "at the time of scene switching" can be mentioned.

When the scene is switched, for example, it corresponds to the case where a change in day / night or landscape that can be determined by analyzing the camera image from the camera 2 is detected.

In such a case, as an example of adjustment contents, for example, when a scene change to an urban area at night is detected, raindrops as deposits are strongly reflected by, for example, many light sources existing in the city, and the outline becomes clear (edge). It is possible that the proportion of strength "strong" will increase). Therefore, as an exclusion condition for excluding raindrops in such a case, for example, adjustment to reduce the ratio of the edge strength “strong” (see “↓” in the figure) can be mentioned as an example.

Further, as another exclusion condition adjustment trigger, for example, "detection area position" can be mentioned. This is because, for example, if the detection area is a position corresponding to the sky, it is considered that there is little influence on the driver's visual recognition during driving. Therefore, in this case, for example, the condition adjusting unit 11d is applied as an example of adjustment contents. The position detection area is adjusted to be excluded unconditionally.

Further, as yet another exclusion condition adjustment trigger, for example, "continuity between a plurality of frames" can be mentioned. This is because, for example, if the detection area is detected by one frame alone, the possibility of deposits is low. In this case, for example, the condition adjusting unit 11d may use one frame as an example of the adjustment content. An adjustment is made to unconditionally exclude a single detection area.

In this way, when a predetermined trigger suitable for adjusting the exclusion condition is generated, by adjusting the exclusion condition, it is possible to detect an appropriate deposit according to the traveling situation of the vehicle C. Is possible. That is, it can contribute to further improving the detection accuracy of deposits.

Next, the processing procedure executed by the deposit detection device 10 according to the present embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing a processing procedure executed by the deposit detection device 10 according to the first embodiment.

First, the target area acquisition unit 11a acquires the detection area Da of each detection algorithm of the external deposit detection device 4 (step S101). Then, the histogram generation unit 11b generates a histogram for each of the edge intensity, the brightness, and the saturation of the detection area Da acquired by the target area acquisition unit 11a (step S102).

Then, the deposit determination unit 11c determines whether or not the deposit is a deposit based on the frequency ratio of each class of each histogram generated by the histogram generation unit 11b (step S103).

Here, when it is determined that there is no deposit (step S104, Yes), the deposit determination unit 11c excludes the detection area Da (step S105). If it is determined that the deposit is a deposit (step S104, No), the deposit determination unit 11c notifies the removal determination device 5 of the detection area (step S106).

Then, the control unit 11 determines whether or not there is a processing end event (step S107). The processing end event corresponds to, for example, IG off or ACC off. Here, if it is determined that there is no processing end event (steps S107, No), the processing from step S101 is repeated. If it is determined that there is a processing end event (step S107, Yes), the deposit detection device 10 ends the processing.

As described above, the deposit detection device 10 according to the first embodiment corresponds to the target area acquisition unit 11a (corresponding to an example of the “acquisition unit”) and the histogram generation unit 11b (corresponding to an example of the “generation unit”). ) And the deposit determination unit 11c (corresponding to an example of the "determination unit").

The target area acquisition unit 11a acquires the detection area Da (corresponding to an example of the “determination target area”) of deposits in the camera image (corresponding to an example of the “captured image”). The histogram generation unit 11b generates a histogram of at least edge intensity, brightness, and saturation for the detection area Da acquired by the target area acquisition unit 11a. The deposit determination unit 11c determines the presence or absence of deposits in the detection area Da based on the frequency ratio of each class of the histogram generated by the histogram generation unit 11b.

Therefore, according to the deposit detection device 10 according to the present embodiment, the detection accuracy of the deposit can be further improved. Further, since the unnecessary detection area Da can be excluded from the processing target in the subsequent stage, the effect of suppressing the processing load of the entire system can be obtained.

(Second Embodiment)
Next, the second embodiment will be described with reference to FIGS. 3 and 8A to 14. As described above, in the second embodiment, the deposit detection device 10 executes one of the deposit detection algorithms -1, -2, ...- N (see FIG. 2B), and for each divided area. This is a case where the presence or absence of deposits is determined based on the “change amount” between frames in the histograms of edge strength, brightness, and saturation of each divided area (see FIG. 1D).

The block configuration of the deposit detection device 10 according to the second embodiment can be described using the block diagram of the first embodiment. Therefore, for convenience of explanation, the block diagram of FIG. 3 already shown. Will be used to mainly explain the parts different from the first embodiment.

As shown in FIG. 3, in the deposit detection device 10 according to the second embodiment, the target area acquisition unit 11a sets a plurality of divided areas for one frame of the camera image of the camera 2, and the divided areas are divided. Each is acquired as a judgment target area.

Here, a setting example of the divided area will be described with reference to FIGS. 8A to 8D. 8A to 8D are diagrams (No. 1) to (No. 4) showing a setting example of the divided area. As shown in FIG. 8A, the division area can be set by dividing the entire camera image for one frame into nine division areas of, for example, 3 × 3.

In the following, the case of FIG. 8A will be taken as the main example, and when the screen is referred to as "top", "middle", or "bottom", "top", "middle", or "bottom" in the figure. It shall refer to the upper, middle, and lower regions corresponding to "", respectively.

Further, although 9 regions of 3 × 3 are used here, the number of divisions is not limited, and 16 regions of 4 × 4 may be used. By making the divided area finer, it is possible to contribute to improving the detection accuracy of small raindrops and the like.

Further, as shown in FIG. 8B, the sizes of the divided areas do not have to be the same. For example, as shown in FIG. 8B, the divided area may be made finer in a region such as the vicinity of the center where it is considered that the driver's visual recognition is greatly affected when deposits are attached. This makes it possible to increase the sensitivity for detecting the amount of change in a region where high visibility is required from the viewpoint of safety.

Further, as shown in FIG. 8C, the division area is not limited to a rectangle, and can be set to, for example, a circle that matches the shape of raindrops according to the shape of the deposit. This makes it easier to capture the amount of change according to the shape.

Further, as shown in FIG. 8D, the divided area may be set in advance so as to exclude the area determined to have little influence on the visibility of the driver, for example. Further, the non-target area may be set variably according to changes in the traveling situation and the like.

Similarly, the number of divisions may be changed according to changes in traveling conditions. For example, between day and night, the amount of change in edge strength, brightness, etc. is considered to be smaller at night, so when it is detected by image analysis that it is night, the number of divisions is dynamically increased to change. The sensitivity for detecting the amount may be increased.

Returning to the description of FIG. In the deposit detection device 10 according to the second embodiment, the histogram generation unit 11b generates a histogram of each of the acquired edge strength, brightness, and saturation of each divided area. Information including such a histogram is retained not only for the current frame but also for the past frames of one generation or more.

The details of the retained information will be described with reference to FIG. FIG. 9 is a diagram showing a specific example of the retained information according to the second embodiment. As shown in FIG. 9, in the second embodiment, the generated histogram holds, for example, the current frame, the past one generation, and the past two generations. In addition, the amount of change calculated from the histogram is also retained.

More specifically, as shown as retained information in FIG. 9, the number of data for the current frame (each frequency), the number of data for each of the past two generations, the simple moving average (SMA), and the current frame. The amount of change from the simple moving average and the amount of change from each of the past two generations and the simple moving average are at least retained.

In the figure, the “*” mark indicates the concept of SMA according to the present embodiment, and the “**” mark indicates the concept of the amount of change according to the present embodiment.

Returning to the description of FIG. Then, in the deposit detection device 10 according to the second embodiment, the deposit determination unit 11c determines at least the transition of the current change amount and the change amount of the past one generation before each class of the histogram based on the above-mentioned holding information. In light of the detection conditions of, it is determined that the deposits are present when the detection conditions are satisfied. The predetermined detection condition is included in the condition information 12a in advance.

Here, FIG. 10 shows a specific example of a predetermined detection condition. FIG. 10 is a diagram showing a specific example of detection conditions.

As shown in FIG. 10, for example, as the "detection condition" according to the second embodiment, "strong" of "edge strength" decreases by a predetermined amount and "weak" increases by a predetermined amount, and "medium" of "brightness". "Is increased by a predetermined amount, and" weak "of" saturation "is increased by a predetermined amount.

Then, the increase / decrease thresholds and the like constituting the detection condition can be set as parameters for each divided area. For example, FIG. 11 is a diagram showing an example of a parameter setting screen. It should be noted that the parameter setting screen is shown for convenience to explain that the parameter can be set for each divided area, and is not essential in the system.

As shown in FIG. 11, on the parameter setting screen, widgets such as check boxes and sliders are used for each element of the histogram (“edge strength”, “brightness”, “saturation”) and for each class of such elements. It is possible to set a threshold value corresponding to the "predetermined amount".

For example, for "edge strength", check the "strong" and "weak" check boxes and move the slider left and right, and for "strong", set the threshold value to a negative value to set the "decrease" condition. It shows what you can do. Moreover, in the case of "weak", it means that the condition of "increase" can be set by setting the threshold value to a positive value.

Further, on the parameter setting screen, threshold values can be individually set in the areas corresponding to "top", "middle", and "bottom" (see FIG. 8A) of the screen. For example, FIG. 11 shows an example in which the area corresponding to the “bottom” of the screen surrounded by the closed curve of the broken line is set differently from the other “top” and “middle”.

This is because the camera image from the camera 2 mounted on the vehicle C tends to change more drastically at the "bottom" of the screen, so the parameters are individually set according to such characteristics. Is. In this way, by making it possible to set the parameter of the amount of change according to each characteristic of the divided area, it is possible to contribute to further improving the detection accuracy of the deposit.

In addition, the deposit determination unit 11c according to the second embodiment determines the transition including the past two generations ago change amount in addition to the transition of the current change amount and the past one generation previous change amount of each class of the histogram. By performing the above, the accuracy of detecting deposits can be further improved.

Specific examples of such cases will be described with reference to FIGS. 12A and 12B. 12A and 12B are diagrams (No. 1) and (No. 2) showing specific examples of the determination method when the amount of change before the past two generations is included. Note that FIG. 12A shows the transition of the amount of change in the edge strength in the case of deposits, and FIG. 12B shows the transition of the amount of change in the edge strength in the case of the white line.

In the case of FIG. 12A and the case of FIG. 12B, there is not much difference in the transition of the current change amount and the change amount of the past one generation before. In such a case, the accuracy of detecting deposits can be improved by further determining the transition including the past two generations ago change amount.

Specifically, looking at FIG. 12A, in the case of deposits, it can be said that there is almost no change in the changes in the past two generations and the changes in the past one generation shown in the portion surrounded by the broken line M1. Then, in FIG. 12A, it can be seen that the amount of change suddenly decreases due to the change amount of the past one generation before and the change amount of the present change amount through such a transition.

This is characteristic of raindrops that abruptly adhere and blur the camera image, reducing the "strong" edge strength. Therefore, when tracing the transition of the amount of change as shown in FIG. 12A, it can be determined that the deposit is attached.

On the other hand, looking at FIG. 12B, in the case of the white line, the transitions of the past two generations previous change amount and the past one generation previous change amount shown in the portion surrounded by the broken line closed curve M2 gradually decrease from the “+” side. You can see that there is. This transition is different from the characteristics in the case of raindrops shown in FIG. 12A. Therefore, when tracing the transition of the amount of change as shown in FIG. 12B, it can be determined that the deposit is not attached.

In this way, the accuracy of detecting deposits can be further improved by determining the transition of the amount of change including the amount of change before the generation in the past. In addition, since the adhesion of deposits is determined based on the temporal transition of the amount of change in the histogram including the edge strength, for example, the water-repellent coating of the lens 2a is lowered, and raindrops with unclear contours are adhered. Even if this happens, it can be detected with high accuracy.

Returning to the description of FIG. In the deposit detection device 10 according to the second embodiment, the condition adjusting unit 11d adjusts the detection condition when a predetermined trigger suitable for adjusting the detection condition is generated, and appropriately obtains the condition information 12a. Perform the update process.

Such a specific example will be described with reference to FIG. FIG. 13 is a diagram showing a specific example of adjusting the detection conditions. As shown in FIG. 13, as a trigger for adjusting the detection condition, for example, "division area position" can be mentioned.

In such a case, as an example of the adjustment content, as already described, the amount of change tends to be more drastic at the “bottom” of the screen. Therefore, for example, in the condition adjustment unit 11d, the “division area position” is the “bottom” of the screen. If this is the case, make adjustments to strengthen the conditions so that the sensitivity for detection does not become too high.

Further, as another detection condition adjustment trigger, for example, "running state" can be mentioned. In such a case, for example, if the vehicle C is stopped, it is considered that deposits such as raindrops are more likely to adhere than during traveling. In this case, for example, the condition adjusting unit 11d is used for detection as an example of the adjustment content. Make adjustments to relax the conditions so that the sensitivity of the.

Further, as another detection condition adjustment trigger, for example, "wiper operation", "rain sensor", "precipitation information reception" and the like can be mentioned. Since all of these indicate a situation in which deposits are likely to adhere due to rainfall or the like, in these cases, for example, the condition adjusting unit 11d is a condition for increasing the sensitivity for detection as an example of the adjustment content. Make adjustments to alleviate.

Further, as another detection condition adjustment trigger, for example, "sky color" can be mentioned. If cloudy or rainy weather is detected from the color of the sky by image analysis of the camera image, for example, the condition adjustment unit 11d may make adjustments to relax the conditions as an example of the adjustment contents, as in the above-mentioned "wiper operation". Do.

Further, as yet another exclusion condition adjustment trigger, for example, a "gyro sensor" can be mentioned. This is because, for example, if the gyro sensor detects that the vehicle C is traveling downhill, the condition adjusting unit 11d increases the sensitivity for detection, as in the case of the above-mentioned "wiper operation". Make adjustments to ease the conditions.

This is because, for example, the lens 2a of the back camera 22 faces upward from the normal state when traveling downhill, so it can be said that raindrops are likely to adhere if there is rainfall or the like.

Next, the processing procedure executed by the deposit detection device 10 according to the present embodiment will be described with reference to FIG. FIG. 14 is a flowchart showing a processing procedure executed by the deposit detection device 10 according to the second embodiment.

First, the target area acquisition unit 11a acquires each divided area from one frame of the camera image of the camera 2 (step S201). Then, the histogram generation unit 11b generates a histogram of each of the edge strength, the brightness, and the saturation of each divided area acquired by the target area acquisition unit 11a (step S202).

Then, the deposit determination unit 11c determines the presence or absence of deposits in each divided area based on the amount of change between the frames of each histogram generated by the histogram generation unit 11b (step S203).

Here, when it is determined that there is an deposit (step S204, Yes), the deposit determination unit 11c notifies, for example, the removal determination device 5 of the division area (step S205). If it is determined that there is no deposit (steps S204, No), the deposit determination unit 11c shifts control to step S206.

Then, the control unit 11 determines whether or not there is a processing end event (step S206). The processing end event corresponds to, for example, IG off or ACC off. Here, if it is determined that there is no processing end event (steps S206, No), the processing from step S201 is repeated. If it is determined that there is a processing end event (step S206, Yes), the deposit detection device 10 ends the processing.

As described above, in the deposit detection device 10 according to the second embodiment, the deposit determination unit 11c changes the amount of change between the current frame and the past frame of the histogram generated by the histogram generation unit 11b. The presence or absence of deposits is determined based on.

Therefore, according to the deposit detection device 10 according to the second embodiment, even when the deposit is, for example, a raindrop whose contour is blurred and is difficult to detect as an edge, the amount of change between frames is increased. Based on the characteristics of the raindrops that appear, it can be easily detected as deposits. That is, the accuracy of detecting deposits can be further improved.

(Third Embodiment)
Next, as a third embodiment, the removal control device 50 is configured to include the functions of the deposit detection device 10 described so far, and controls the entire process from the detection of the deposit to the removal of the deposit. The configuration will be described with reference to FIGS. 15 to 19.

FIG. 15 is a block diagram of the deposit removing system 1 according to the present embodiment. Note that, in FIG. 15, only the components necessary for explaining the features of the present embodiment are represented by functional blocks, and the description of general components is omitted.

In other words, each component shown in FIG. 15 is a functional concept and does not necessarily have to be physically configured as shown. For example, the specific form of distribution / integration of each functional block is not limited to the one shown in the figure, and all or part of it is functionally or physically distributed in arbitrary units according to various loads and usage conditions. -It is possible to integrate and configure.

Further, FIG. 16A is a diagram showing an example of the content of notification from the deposit detection unit 51a. Further, FIG. 16B is a diagram showing an example of data contents related to the detection area Da included in the detection information DB (database) 52a. Further, FIG. 16C is an explanatory diagram of the state of the detection area Da.

As shown in FIG. 15, the deposit removing system 1 includes a camera 2, a deposit removing device 3, and a removal control device 50. Since the camera 2 and the deposit removing device 3 have already been described, the description thereof will be omitted here.

The removal control device 50 includes a control unit 51 and a storage unit 52. The control unit 51 includes a plurality of deposit detection units 51a (for example, deposit detection units 51a-1, -2, -3 ...), an exclusion unit 51b, an algorithm-to-algorithm overlap determination unit 51c, and an inter-frame overlap determination unit 51d. The adhesion determination unit 51e, the removal necessity determination unit 51f, and the instruction unit 51g are provided.

The storage unit 52 is a storage device such as a hard disk drive, a non-volatile memory, or a register, and stores the detection information DB 52a.

The control unit 51 controls the entire removal control device 50. Each of the plurality of deposit detection units 51a acquires a camera image for one frame from the camera 2, and uses a detection algorithm associated with each of the detection areas Da, which are estimated to have deposits in the camera image. Is extracted. Further, the deposit detection unit 51a notifies the exclusion unit 51b of the extracted detection area Da.

The deposit detection device 10 according to the second embodiment described above corresponds to any of the plurality of deposit detection units 51a of the present embodiment.

Here, as shown in FIG. 16A, the notification content from the deposit detection unit 51a includes, for example, the upper left coordinates (x, y) of the detection area Da extracted in a rectangular shape, the width w, and the height h. included.

The exclusion unit 51b performs image analysis on each of the detection areas Da notified from the deposit detection unit 51a, and determines whether or not the deposits presumed to exist in the detection area Da are really deposits.

Further, the exclusion unit 51b notifies the inter-algorithm overlap determination unit 51c of the detection area Da that is determined to be an deposit based on the determination result. On the other hand, the exclusion unit 51b does not notify the inter-algorithm overlap determination unit 51c of the detection area Da that is determined to be non-adhesive based on the determination result, and excludes it from the processing target in the subsequent stage. By excluding the useless image area in this way, the accuracy of detecting the deposits can be improved, and the processing load of the subsequent processing can be reduced.

The deposit detection device 10 according to the first embodiment described above corresponds to the exclusion unit 51b of the present embodiment.

The inter-algorithm overlap determination unit 51c determines the overlap of the detection areas Da between the plurality of algorithms in the current frame, that is, the existence or nonexistence of the overlap portion between the detection areas Da extracted by each of the deposit detection units 51a. Further, the inter-algorithm overlap determination unit 51c reflects the determination result as a "score" of each detection area Da. The reflection result is managed on the detection information DB 52a. The details of the determination process executed by the inter-algorithm overlap determination unit 51c will be described later with reference to FIGS. 17A to 17D.

The inter-frame overlap determination unit 51d determines whether or not there is an overlap with the detection area Da extracted in the past frame for all the processing results of the inter-algorithm overlap determination unit 51c related to the current frame. Further, the frame-to-frame overlap determination unit 51d reflects the determination result in the “score” and “state” of each detection area Da. The reflection result is managed on the detection information DB 52a.

Here, as shown in FIG. 16B, the detection information DB 52a includes, for example, a "detection area ID" item, an "area information" item, a "score" item, and a "state" item. The identifier of the detection area Da is stored in the "detection area ID" item, and the detection information DB 52a is managed for each such detection area ID.

In the "area information" item, the upper left coordinates (x, y) of the detection area Da shown in FIG. 16A, the width w, the height h, and the like are stored. In the "score" item, the current score of each detection area Da is stored. The "state" item stores the current state of each detection area Da.

As shown in FIG. 16C as a state machine diagram, each detection area Da can transition to four states of "IDLE", "latent", "observation", and "penalty". "IDLE" refers to an "undetected state", that is, a state in which no deposits are attached. "Hidden" refers to the state of "possible deposits".

"Observation" refers to the "observation state after the removal process" in which the deposit removal operation of the deposit removal device 3 is performed. The "penalty" refers to "a state in which deposits are continuously detected in the area even after the removal process", that is, a state of poor removal or false detection.

The frame-to-frame overlap determination unit 51d updates the "score" of each detection area Da on the detection information DB 52a according to the determination result of the determination, and transitions the "state".

Returning to the description of FIG. The adhesion determination unit 51e determines the "adhesion confirmation" of the adhered matter according to the "state" and the "score" of the detection area Da of the detection information DB 52a.

When the adhesion determination unit 51e determines that "adhesion is confirmed", the removal necessity determination unit 51f determines whether or not the deposit removal device 3 actually performs the adhesion removal operation. Details of the processes executed by the frame-to-frame overlap determination unit 51d, the adhesion determination unit 51e, and the removal necessity determination unit 51f will be described later with reference to FIGS. 18A to 18E.

The instruction unit 51g generates an instruction signal for causing the deposit removal device 3 to perform a removal operation when the removal necessity determination unit 51f determines that the deposit needs to be removed, and outputs the instruction signal to the deposit. It is sent to the removal device 3.

The removal determination device 5 according to the first and second embodiments described above includes the algorithm-to-algorithm overlap determination unit 51c, the inter-frame overlap determination unit 51d, the adhesion determination unit 51e, the removal necessity determination unit 51f, and the removal necessity determination unit 51f. Corresponds to the indicator 51g.

Next, the details of the determination process executed by the inter-algorithm overlap determination unit 51c will be described with reference to FIGS. 17A to 17D. 17A to 17D are processing explanatory views (No. 1) to (No. 4) of the inter-algorithm overlap determination unit 51c.

As already described above, as shown in FIG. 17A, the inter-algorithm overlap determination unit 51c determines whether or not the detection area Da overlaps between a plurality of algorithms in the current frame. Specifically, as shown in FIG. 17A, for example, the overlap of all the detection areas Da-1 of the deposit detection algorithm-1 and all the detection areas Da-2 of the deposit detection algorithm-2 is determined. This also applies between the deposit detection algorithm-1 and the deposit detection algorithm -n, or between the deposit detection algorithm-2 and the deposit detection algorithm -n.

As shown in FIG. 17B, for example, the overlap between the detection area Da-1 and the detection area Da-2 is determined by the distance d from the respective center of gravity.

Then, as shown in FIG. 17C, when the inter-algorithm overlap determination unit 51c determines that there is an overlap between the detection area Da-1 and the detection area Da-2, for example, the detection area Da-1 and the detection area Da-1 Add points to each Da-2 score.

As a result, it can be shown that the detection area Da-1 and the detection area Da-2 having the overlap are more likely to have the deposits than the detection area Da where the overlap does not exist.

Further, as shown in FIG. 17D, for example, when there is an overlap in the detection area Da-1 and the detection area Da-2, the inter-algorithm overlap determination unit 51c of the detection area Da-1 and the detection area Da-2 Update area information.

For example, as shown in FIG. 17D (a), the detection area Da-1 is prioritized and the area information is integrated into the detection area Da-1. Further, as shown in FIG. 17D (b), conversely, the detection area Da-2 is prioritized and the area information is integrated into the detection area Da-2.

Further, as shown in FIG. 17D (c), the area information is integrated into the detection area Da-A having only the overlapping portion by taking a logical product. Further, as shown in FIG. 17D (d), the area information may be integrated into the detection area Da-S corresponding to the logical sum of the detection area Da-1 and the detection area Da-2.

Further, as shown in FIG. 17D (e), the area information may be integrated into the detection area Da-E expanded so as to include both the detection area Da-1 and the detection area Da-2.

Next, the details of the processes executed by the frame-to-frame overlap determination unit 51d, the adhesion determination unit 51e, and the removal necessity determination unit 51f will be described with reference to FIGS. 18A to 18E. 18A to 18C are processing explanatory views (No. 1) to (No. 3) of the frame-to-frame overlap determination unit 51d.

Further, FIG. 18D is a processing explanatory view of the frame-to-frame overlap determination unit 51d and the adhesion determination unit 51e. Further, FIG. 18E is a processing explanatory view of the removal necessity determination unit 51f.

As already described above, as shown in FIG. 18A, the inter-frame overlap determination unit 51d sets the detection area Da that has been extracted in the past frame for all the processing results of the inter-algorithm overlap determination unit 51c related to the current frame. Judge the existence or nonexistence of the overlap of.

Specifically, as shown in FIG. 18B, the overlap between all the detection areas Da-C of the current frame and all the detection areas Da-P of the past frame is determined. The concept of overlap may be the same as in the case of the inter-algorithm overlap determination unit 51c.

Then, as shown in FIG. 18B, when the inter-frame overlap determination unit 51d determines that there is an overlap between the detection area Da-C of the current frame and the detection area Da-P of the past frame, the detection area Da Points are added to the scores of -C and detection area Da-P.

As a result, the frame-to-frame overlap determination unit 51d can indicate, for example, deposits existing in substantially the same region of the lens 2a across the frame.

On the other hand, as shown in FIG. 18C, the inter-frame overlap determination unit 51d deducts points for the detection areas Da-P of the past frames if there is no overlap with any of the detection areas Da-C of the current frame. ..

Further, the inter-frame overlap determination unit 51d considers that the detection area Da-C of the current frame has no overlap with any of the detection areas Da-P of the past frame as a new detection area Da, and detects the detection information. Newly register in DB52a.

As shown in FIG. 18D, the newly registered detection area Da is in the "latent" state and is given a predetermined score. Then, according to the score of the detection area Da that changes from the "latent" state through the above-mentioned "point addition" and "point deduction", the frame-to-frame overlap determination unit 51d and the adhesion determination unit 51e are in the state of the detection area Da. To transition.

For example, as shown in FIG. 18D, when the score of the detection area Da in the “latent” state becomes equal to or less than a predetermined point, the inter-frame overlap determination unit 51d changes the state of the detection area Da from “latent” to “IDLE”. (Step S11). As a result, it is possible to prevent an erroneous reaction in which the deposits such as raindrops that move by flowing down and need not be removed are removed.

Further, when the score of the detection area Da in the "latent" state becomes a predetermined point or more, the adhesion determination unit 51e confirms (determines the adhesion) the adhesion of the deposit to the area (step S12).

Further, the adhesion determination unit 51e shifts all the detection areas Da in the “latent” state to the “observation” state after the adhesion is confirmed (step S13). This means that if the removal process is performed according to the confirmation of adhesion of one detection area Da, the deposits are normally removed even for the other detection area Da in the "latent" state where the adhesion has not been confirmed. This is because it is presumed that it was done.

The inter-frame overlap determination unit 51d shifts the detection area Da to the "penalty" state when the removal process is performed and the score of the detection area Da in the "observation" state becomes a predetermined point or more (step S14). ). As a result, it is possible to grasp the removal failure or false detection in which the deposits are continuously detected even after the removal treatment.

Further, the frame-to-frame overlap determination unit 51d shifts the detection area Da to the “IDLE” state when the score of the detection area Da in the “observation” state or the “penalty” state becomes a predetermined point or less (step S15). ..

The reaction rate until the adhesion is confirmed may be controlled by adjusting the slopes of the arrows indicating "point addition" and "point deduction" in FIG. 18D. For example, by increasing the amount of points added and the amount of points deducted and making the inclination of the arrow steep, the reaction speed from the detection of the deposit to the removal process can be increased.

Further, even in the detection area Da where the adhesion is confirmed, the removal operation may be intentionally not performed. For example, as shown in FIG. 18E, the removal necessity determination unit 51f determines that the removal process does not need to be executed if the detection area Da for which the adhesion is confirmed exists in the skip area substantially along the outer circumference of the screen. can do.

In this way, the processing load of the entire system can be reduced by skipping the removal processing itself for the deposits adhering to the image area that have little influence on the passenger's visual recognition and driving operation.

Next, the processing procedure executed by the deposit removing system 1 according to the present embodiment will be described with reference to FIG. FIG. 19 is a flowchart showing a processing procedure executed by the deposit removing system 1.

First, the plurality of deposit detection units 51a each acquire a camera image for one frame (step S301). Then, for example, the deposit detection unit 51a-1 extracts the detection area Da-1 using the deposit detection algorithm-1 (step S302).

Further, for example, the deposit detection unit 51a-2 extracts the detection area Da-2 using the deposit detection algorithm-2 (step S303). Further, for example, the deposit detection unit 51a-n extracts the detection area Dan by using the deposit detection algorithm-n (step S304).

Then, the exclusion unit 51b performs an exclusion process for each of the detection areas Da extracted and notified by the deposit detection unit 51a (step S305). That is, the exclusion unit 51b determines whether or not the deposit presumed to exist in the detection area Da is really a deposit, and if it is not a deposit, the detection area Da corresponding to the deposit is removed from the processing target in the subsequent stage. exclude.

The exclusion process itself may be omitted, for example. As a result, the processing load on the entire system can be reduced.

Subsequently, the inter-algorithm overlap determination unit 51c performs the inter-algorithm overlap determination process (step S306). That is, the inter-algorithm overlap determination unit 51c determines whether or not the detection area Da overlaps between the plurality of algorithms in the current frame, and updates the score of the detection area Da according to the determination result.

Then, the inter-frame overlap determination unit 51d performs the inter-frame overlap determination process (step S307). That is, the inter-frame overlap determination unit 51d determines whether or not there is an overlap with each detection area Da extracted in the past frame for all the processing results of the inter-algorithm overlap determination unit 51c, and detects according to the determination result. Update the score and status of Area Da.

Then, the adhesion determination unit 51e performs the adhesion determination process (step S308). That is, the adhesion determination unit 51e determines the adhesion determination of the adhered matter according to the score and the state of the detection area Da of the detection information DB 52a updated by the frame-to-frame overlap determination unit 51d.

Then, when the removal necessity determination unit 51f is determined by the adhesion determination unit 51e to be "adhesion confirmed", it is determined whether or not the adhesion removal device 3 actually needs to remove the deposits (step S309). ..

Here, when it is determined that "removal is required" (step S309, Yes), the instruction unit 51g sends an instruction signal to the deposit removal device 3 to cause the deposit removal device 3 to perform the removal operation. The removal process is executed (step S310). On the other hand, when it is determined that "removal is not required" (step S309, No), the indicating unit 51g does not execute the removal process.

Then, the control unit 51 determines whether or not there is a processing end event (step S311). The processing end event corresponds to, for example, IG off or ACC off. Here, when it is determined that there is no processing end event (steps S311 and No), the processing from step S301 is repeated. If it is determined that there is a processing end event (steps S311, Yes), the deposit removing system 1 ends the processing.

Further effects and variations can be easily derived by those skilled in the art. For this reason, the broader aspects of the invention are not limited to the particular details and representative embodiments expressed and described as described above. Therefore, various modifications can be made without departing from the spirit or scope of the general concept of the invention as defined by the appended claims and their equivalents.

1 Adhesive removal system 2 Camera 3 Adhesive removal device 5 Removal judgment device 10 Adhesion detection device 11a Target area acquisition unit 11b histogram generator 11c Adhesion judgment unit 11d Condition adjustment unit 12a Condition information 50 Removal control device 51a Adhesion detection Part 51b Exclusion part 51c Inter-algorithm overlap judgment part 51d Inter-frame overlap judgment part 51e Adhesion judgment part 51f Removal necessity judgment part 51g Indicator 52a Detection information DB

Claims (8)

An acquisition unit that acquires an area to be determined for deposits in the captured image captured by the imaging device, and an acquisition unit.
A generation unit that generates at least a histogram of edge intensity, brightness, and saturation for the determination target area acquired by the acquisition unit.
A determination unit for determining the presence or absence of the deposit in the determination target area is provided based on the frequency ratio of each class of the histogram generated by the generation unit.
The acquisition unit
Based on the captured image, a detection area estimated to have the deposits is acquired as the determination target area by executing a predetermined detection algorithm in advance.
The determination unit
When the ratio of the frequency of each class of the histogram generated by the generation unit satisfies a predetermined exclusion condition, it is determined that the deposit does not exist in the detection area, and the detection area is used to remove the deposit. Exclude from the processing target of the removal judgment process that judges the necessity ,
The exclusion condition is
A deposit detection device, characterized in that the condition is a combination of the ratios of the edge strength, the brightness, and the saturation for each class so as not to match the characteristics of the deposit. The generator
The first aspect of the present invention is to generate the histogram classified into at least three classes of weak, medium and strong for each of the edge strength, brightness and saturation classes of the determination target area. Adhesion detector. The determination unit
The deposit detection according to claim 1 or 2 , wherein the presence or absence of the deposit is determined based on the amount of change between the current frame portion and the past frame portion of the histogram generated by the generation unit. apparatus. The acquisition unit
A plurality of divided areas set for the captured image are acquired as the determination target areas, respectively.
The determination unit
The deposit detection device according to claim 3, wherein the change amount based on the histogram generated by the generation unit determines that the deposit exists in the divided area that satisfies a predetermined detection condition. The deposit detection device according to claim 4, wherein the detection conditions can be set for each of the divided areas. A condition adjusting unit for adjusting the detection condition is provided when a predetermined trigger suitable for adjusting the detection condition occurs.
The condition adjustment unit
The deposit detection device according to claim 4 or 5, wherein when the position of the determination target area is within the lower region of the captured image, the detection condition is adjusted so as to be strengthened. The generator
The deposit detection device according to any one of claims 1 to 6, wherein the size of the determination target area is scaled to a reference size when generating the histogram. The acquisition process for acquiring the area to be determined for deposits in the captured image, and
A generation step of generating at least a histogram of edge intensity, brightness, and saturation for the determination target area acquired by the acquisition step, and
A determination step of determining the presence or absence of the deposit in the determination target area based on the ratio of the frequency of each class of the histogram generated by the generation step is included.
The acquisition process is
Based on the captured image, a detection area estimated to have the deposits is acquired as the determination target area by executing a predetermined detection algorithm in advance.
The determination step is
When the ratio of the frequency of each class of the histogram generated by the generation step satisfies a predetermined exclusion condition, it is determined that the deposit does not exist in the detection area, and the detection area is used to remove the deposit. Exclude from the processing target of the removal judgment process that judges the necessity ,
The exclusion condition is
The condition is a combination of the ratios of the edge strength, brightness, and saturation for each class so as not to match the characteristics of the deposits.
A method for detecting deposits, which comprises.