JP2021050933A - Attached matter detector and attached matter detecting method - Google Patents

Abstract

To improve the accuracy of detecting attached matter.SOLUTION: An attached matter detector pertaining to an embodiment comprises a calculation unit, a detection unit, and a determination unit. The calculation unit calculates, for each unit area composed of a prescribed number of pixels included in an imaged image, an area feature quantity based on the edge vector of each pixel. The detection unit detects the attached state of attached matter to the unit area along with a first detection condition based on the area feature quantity. The determination unit determines the attachment rate of attached matter to a camera lens on the basis of the detection result of the detection unit. The detection unit also detects the attached state of attached matter to the unit area along with the first and second detection conditions based on the area feature quantity when it is determined by the determination unit that the attachment rate is greater than or equal to a given rate.SELECTED DRAWING: Figure 2

Description

The disclosed embodiments relate to a deposit detection device and a deposit detection method.

Conventionally, there is known a deposit detection device that detects deposits adhering to a camera lens based on an image captured by a camera mounted on a vehicle or the like. Some deposit detection devices detect deposits based on, for example, differences in captured images over time (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2012-038048

However, there is room for further improvement in the above-mentioned prior art in improving the detection accuracy of deposits.

One aspect of the embodiment is made in view of the above, and an object of the present invention is to provide a deposit detection device and a deposit detection method capable of improving the detection accuracy of deposits.

The deposit detection device according to one aspect of the embodiment includes a calculation unit, a detection unit, and a determination unit. The calculation unit calculates a region feature amount based on the edge vector of each pixel for each unit region composed of a predetermined number of pixels included in the captured image. The detection unit detects the state of adhesion of the deposit to the unit region according to the first detection condition based on the region feature amount. The determination unit determines the adhesion rate of deposits to the lens of the camera based on the detection result of the detection unit. Further, when the determination unit determines that the adhesion rate is equal to or higher than a certain level, the detection unit has an adhering substance according to the first detection condition and the second detection condition based on the region feature amount, respectively. Detects the adhesion state of.

According to one aspect of the embodiment, the accuracy of detecting deposits can be improved.

FIG. 1A is a schematic explanatory view (No. 1) of the deposit detection method according to the embodiment. FIG. 1B is a schematic explanatory view (No. 2) of the deposit detection method according to the embodiment. FIG. 1C is a schematic explanatory view (No. 3) of the deposit detection method according to the embodiment. FIG. 2 is a block diagram of the deposit detection device according to the embodiment. FIG. 3 is a diagram (No. 1) showing the processing contents of the calculation unit. FIG. 4 is a diagram (No. 2) showing the processing contents of the calculation unit. FIG. 5 is a diagram (No. 3) showing the processing contents of the calculation unit. FIG. 6 is a diagram (No. 4) showing the processing contents of the calculation unit. FIG. 7 is a diagram (No. 5) showing the processing contents of the calculation unit. FIG. 8 is a diagram (No. 6) showing the processing content of the calculation unit. FIG. 9 is a diagram (No. 7) showing the processing content of the calculation unit. FIG. 10 is a diagram (No. 8) showing the processing contents of the calculation unit. FIG. 11 is a diagram (No. 9) showing the processing content of the calculation unit. FIG. 12 is a diagram (No. 10) showing the processing contents of the calculation unit. FIG. 13 is a diagram (No. 11) showing the processing contents of the calculation unit. FIG. 14 is a diagram (No. 12) showing the processing contents of the calculation unit. FIG. 15 is a diagram (No. 13) showing the processing contents of the calculation unit. FIG. 16 is a diagram (No. 14) showing the processing contents of the calculation unit. FIG. 17 is a diagram showing the processing contents of the determination unit. FIG. 18 is a flowchart showing a processing procedure executed by the deposit detection device according to the embodiment.

Hereinafter, embodiments of the deposit detection device and the deposit detection method disclosed in the present application will be described in detail with reference to the accompanying drawings. The present invention is not limited to the embodiments shown below.

First, the outline of the deposit detection method according to the embodiment will be described with reference to FIGS. 1A to 1C. 1A to 1C are schematic explanatory views (No. 1) to (No. 3) of the deposit detection method according to the embodiment.

As shown in FIG. 1A, for example, it is assumed that there are captured images I1 to I4 captured in time series with snow attached to the lens surface of the in-vehicle camera. In the following, the deposit detection device 1 (see FIG. 2) to which the deposit detection method according to the embodiment is applied is referred to as a feature amount related to the brightness gradient of each pixel of the captured images I1 to I4 (hereinafter, “edge feature amount”). Based on (sometimes referred to as), the case of detecting a state in which most of the lenses of an in-vehicle camera are buried by snow is taken as an example. When the captured images I1 to I4 are generically referred to, they are simply referred to as captured images I.

Specifically, the deposit detection device 1 detects the snow adhesion state based on the edge feature amount of each pixel PX (see FIG. 4) calculated from the captured image I. The edge features include angular features and intensity features. The angular feature amount is the direction of the edge vector (luminance gradient) of each pixel PX (hereinafter, may be referred to as “edge direction”). The intensity feature amount is the magnitude of the edge vector of each pixel PX (hereinafter, may be referred to as “edge intensity”).

The deposit detection device 1 handles the edge feature amount in units of cells 100 (see FIG. 4) composed of a predetermined number of pixels PX in order to reduce the processing load in image processing. This can contribute to reducing the processing load in image processing. Further, the unit area UA shown in FIG. 1A is a collection of such cells 100.

Then, the deposit detection device 1 calculates the area feature amount, which is the feature amount for each unit area UA, based on the edge feature amount calculated for each of the cells 100. The area feature amount is, so to speak, a statistical feature amount of the edge feature amount for each unit area UA. For example, the brightness average, the average edge strength, the variance of the edge strength, the number of paired regions, the total edge strength of the paired regions, etc. including. Here, the pair area is a combination of cells 100 that are adjacent to each other and whose edge orientations are opposite to each other. Then, the deposit detection device 1 detects the deposit state for each unit region UA based on the region feature amount.

More specifically, as shown in FIG. 1A, the deposit detection device 1 is attached to the captured image I according to a predetermined detection condition for each unit region UA based on the calculated region feature amount (““ Detects "adhesion" or "non-adhesion" (see, for example, captured image I1 in the figure).

Then, for example, as shown in the captured image I2, when the adhesion rate exceeds a certain level, the deposit detection device 1 determines that most of the lenses of the vehicle-mounted camera are buried, and turns on the buried flag. Here, the adhesion rate is, for example, the area ratio of the adhesion portion detected as "adhesion" in a predetermined region of interest in the captured image I. The buried flag is a flag indicating whether or not the lens is buried, and is set to on when it is determined that the lens is buried and off when it is determined that the lens is not buried.

By the way, the mode of snow changes according to changes in the environment, such as melting, but it is difficult to detect snow under the same detection conditions because the characteristics of snow that has begun to adhere and snow that has melted are different. Therefore, for example, as shown by the rectangular portion of the broken line in the captured image I3, when a melt-off portion occurs, the melt-off portion is detected as the adhesion portion under the detection conditions in which the adhesion portion can be detected in the captured images I1 and I2. You can't.

Therefore, the number of non-adherent portions increases and the adhesion rate decreases, so that the deposit detection device 1 turns off the buried flag.

However, the originally desirable off-timing of the buried flag should be a timing at which the in-vehicle camera is secured to some extent as the unraveling progresses, as shown in, for example, the captured image I4.

Therefore, after the burial flag is turned on, stabilizing the burial flag by detecting the adhesion state of the melted snow also contributes to the improvement of the detection accuracy of the deposit.

Therefore, in the deposit detection method according to the embodiment, after it is determined that most of the lens is buried by the deposits detected according to the first detection condition, the first detection condition and the second detection condition It was decided to detect deposits along the lines.

Specifically, as shown in FIG. 1B, in the deposit detection method according to the embodiment, if the adhesion rate is equal to or higher than a certain level and the burial flag is turned on, the first detection condition and the first detection condition and the first detection condition for each unit region UA. The adhesion state is detected according to the detection conditions of 2.

Here, the first detection condition is a detection condition for detecting snow before it adheres and begins to melt, which is shown as a predetermined detection condition in FIG. 1A. The second detection condition is a detection condition for detecting snow that has adhered but has begun to melt.

Then, in the deposit detection method according to the embodiment, the above-mentioned adhesion rate is calculated based on the total value of the adhesion locations detected according to the first detection condition and the second detection condition, respectively. As a result, it is possible to suppress a decrease in the adhesion rate after the burial flag is turned on, stabilize the burial flag, and turn off the burial flag at a desired timing when the burial flag has progressed to some extent.

Therefore, according to the deposit detection method according to the embodiment, the deposit detection accuracy can be improved.

As shown in FIG. 1C, the first detection condition and the second detection condition are set based on various elements included in the region feature amount calculated for each unit region UA. The first detection condition is the characteristics of the normal attachment point, for example, in the unit region UA, the weak edge, the cohesiveness of the angle classification is large, the angle classification is biased, and the appearance frequency of the pair region is low. It is preset so that it can be detected.

On the other hand, the second detection condition is a feature of the unraveled portion, for example, a strong edge in the unit region UA, a fine cohesiveness of the angle classification, a small bias of the angle classification, and a high frequency of appearance of the pair region. Is preset so that it can be detected.

The angle classification corresponds to the edge orientation of the cell 100, which is a representative value of the vector orientation of each pixel PX classified in a predetermined angle range. The calculation process of the edge feature amount and the area feature amount including the angle classification will be described later with reference to FIGS. 3 to 16.

Hereinafter, a configuration example of the deposit detection device 1 to which the deposit detection method according to the above-described embodiment is applied will be described in more detail.

FIG. 2 is a block diagram of the deposit detection device 1 according to the embodiment. Note that, in FIG. 2, only the components necessary for explaining the features of the present embodiment are represented by functional blocks, and the description of general components is omitted.

In other words, each component shown in FIG. 2 is a functional concept and does not necessarily have to be physically configured as shown in the figure. For example, the specific form of distribution / integration of each functional block is not limited to the one shown in the figure, and all or part of it is functionally or physically distributed in arbitrary units according to various loads and usage conditions. -It is possible to integrate and configure.

As shown in FIG. 2, the deposit detection device 1 according to the embodiment includes a storage unit 2 and a control unit 3. Further, the deposit detection device 1 is connected to the camera 10 and various devices 50.

Note that FIG. 2 shows a case where the deposit detection device 1 is configured separately from the camera 10 and the various devices 50, but the present invention is not limited to this, and the deposit detection device 1 is integrally integrated with at least one of the camera 10 and the various devices 50. It may be configured.

The camera 10 is, for example, an in-vehicle camera including a lens such as a fisheye lens and an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is provided, for example, at a position where the front and rear of the vehicle and the side of the vehicle can be imaged, and the captured image I is output to the deposit detection device 1.

The various devices 50 are devices that acquire the detection results of the deposit detection device 1 and perform various controls on the vehicle. The various devices 50, for example, have a display device that notifies the user that deposits are attached to the lens of the camera 10 and an instruction to wipe off the deposits, and jets fluid, gas, or the like toward the lens to remove the deposits. It includes a removal device for removing, a vehicle control device for controlling automatic driving, and the like.

The storage unit 2 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), or a storage device such as a hard disk or an optical disk. The estimation information 22, the condition information 23, and the determination history information 24 are stored.

The template information 21 is information about a template used in the matching process executed by the calculation unit 32, which will be described later. The estimation information 22 is information about a region feature space having each element of the region feature described later as each dimension. The estimation information 22 may be rephrased as an estimation model that estimates the adhesion state based on the region feature amount calculated by the calculation unit 32 described later.

The condition information 23 is information about the detection conditions used in the detection process executed by the detection unit 33, which will be described later, and includes, for example, the first detection condition and the second detection condition described above. The determination history information 24 is information regarding the determination history of deposit detection in the captured image I for a predetermined past n frames.

The control unit 3 is a controller, and for example, various programs stored in the storage device inside the deposit detection device 1 work on the RAM by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like. It is realized by executing as an area. Further, the control unit 3 can be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 3 has an acquisition unit 31, a calculation unit 32, a detection unit 33, and a determination unit 34, and realizes or executes an information processing function or operation described below.

The acquisition unit 31 acquires the captured image I captured by the camera 10. The acquisition unit 31 performs grayscale processing for converting each pixel in the acquired captured image I into each gradation from white to black according to the brightness, and also performs smoothing processing for each pixel to the calculation unit 32. Output. For the smoothing process, for example, an averaging filter, an arbitrary smoothing filter such as a Gaussian filter, or the like can be used. Further, the grayscale processing and the smoothing process may be omitted.

The calculation unit 32 calculates the edge feature amount for each cell 100 of the captured image I acquired from the acquisition unit 31. Here, the calculation process of the edge feature amount by the calculation unit 32 will be specifically described with reference to FIGS. 3 and 4.

3 and 4 are diagrams (No. 1) and (No. 2) showing the processing contents of the calculation unit 32. As shown in FIG. 3, the calculation unit 32 first performs edge detection processing for each pixel PX to obtain the strength of the edge ex in the X-axis direction (horizontal direction of the captured image I) and the Y-axis direction (captured image I). The strength of the edge eye (in the vertical direction) is detected. For the edge detection process, for example, an arbitrary edge detection filter such as a Sobel filter or a Prewitt filter can be used.

Subsequently, the calculation unit 32 calculates the edge vector V by using a trigonometric function based on the detected intensity of the edge ex in the X-axis direction and the intensity of the edge ey in the Y-axis direction. The edge direction, which is the angle θ formed by the X-axis, and the edge strength, which is the length L of the edge vector V, are calculated.

Subsequently, the calculation unit 32 extracts a representative value for the edge in the cell 100 based on the calculated edge vector V of each pixel PX. Specifically, as shown in the upper part of FIG. 4, the calculation unit 32 sets the edge directions −180 ° to 180 ° of the edge vector V of each pixel PX in the cell 100 in four directions of up / down / left / right at 90 ° intervals. It is classified into angle classifications (0) to (3) (hereinafter, may be referred to as "upper, lower, left, and right 4 classifications").

More specifically, when the edge orientation of the pixel PX is in the angle range of −45 ° or more and less than 45 °, the calculation unit 32 classifies it into the angle classification (0), and the angle of 45 ° or more and less than 135 °. If it is a range, it is classified into angle classification (1), and if it is 135 ° or more and less than 180 °, or if it is an angle range of -180 ° or more and less than -135 °, it is classified into angle classification (2) and -135. If the angle range is greater than or equal to ° and less than −45 °, it is classified into angle classification (3).

Then, as shown in the lower part of FIG. 4, the calculation unit 32 generates a histogram for each cell 100 with the angle classifications (0) to (3) as each class. Then, when the frequency of the class having the highest frequency is equal to or higher than the predetermined threshold value THa in the generated histogram, the calculation unit 32 indicates the angle classification corresponding to the class (in the example of FIG. 4, the angle classification (1)). Is extracted as a representative value for the edge orientation in the cell 100.

The frequency of the above-mentioned histogram is calculated by adding the edge intensities of the pixel PXs classified in the same angle range among the pixel PXs in the cell 100. Specifically, consider the power of the histogram in the class of angle classification (0). For example, it is assumed that there are three pixel PXs classified in the angle classification (0), and the edge intensities in each pixel PX are 10, 20, and 30. In this case, the frequency of the histogram in the class of angle classification (0) is calculated as 10 + 20 + 30 = 60.

Based on the histogram calculated in this way, the calculation unit 32 calculates a representative value of the edge strength in the cell 100. Specifically, as a representative value of such an edge strength, when the frequency of the class having the highest frequency in the histogram is equal to or higher than a predetermined threshold value THa, the frequency corresponding to the class is defined as the edge strength of the cell 100. That is, the process of calculating the representative value of the edge strength in the calculation unit 32 can be said to be the process of calculating the feature related to the edge strength in the cell 100 corresponding to the representative value for the edge.

On the other hand, when the frequency of the highest frequency class is less than the predetermined threshold value THa, the calculation unit 32 is "invalid" for the edge orientation of the cell 100, in other words, "no representative value for the edge orientation". Treat as. This makes it possible to prevent the calculation of a specific edge orientation as a representative value when the variation in the edge orientation of each pixel PX is large.

The processing content of the calculation unit 32 shown in FIGS. 3 and 4 is only an example, and the processing content may be arbitrary as long as the representative value for the edge can be calculated. For example, the average value of each pixel PX in the cell 100 in the edge direction may be calculated, and the angle classifications (0) to (3) corresponding to the average value may be used as the representative value in the edge direction.

Further, in FIG. 4, a case where a total of 16 pixel PXs of 4 × 4 is regarded as one cell 100, but the number of pixel PXs in the cell 100 may be arbitrarily set, and 3 × 5 Etc., the number of pixels PX in the vertical direction and the horizontal direction may be different.

Returning to the description of FIG. Further, the calculation unit 32 calculates the area feature amount for each unit area UA based on the calculated edge feature amount for each cell 100.

First, the calculation unit 32 calculates the brightness average, the average edge strength of the cell 100, and the variance in each unit area UA as the area feature amount. Further, the calculation unit 32 calculates the total number of pair regions 200 and the edge strength as the region features.

Here, a case of calculating the total number of pair regions 200 and the edge strength will be described with reference to FIGS. 5 and 6. 5 and 6 are diagrams (No. 3) and (No. 4) showing the processing contents of the calculation unit 32.

Note that FIG. 5 shows a case where the two pair areas 200 do not share the cell 100, and FIG. 6 shows a case where the two pair areas 200 share the cell 100.

As shown in FIG. 5, the calculation unit 32 scans a plurality of cells 100 arranged in the left-right direction and the up-down direction of the unit area UA in the left-right direction and the up-down direction to search for the pair area 200. That is, the calculation unit 32 extracts the cells 100 that are adjacent to each other and whose edge directions are opposite to each other as the pair area 200 among the cells 100 in the unit area UA.

Then, the calculation unit 32 calculates the number of the extracted pair regions 200 and the total edge strength in the pair regions 200. As shown in FIG. 5, when the extracted two pair areas 200 do not share the cell 100, the calculation unit 32 calculates the number of the pair areas 200 as two and sums the edge strengths. It is calculated as the total value of the edge intensities of the four cells 100 included in the two pair areas 200.

Further, as shown in FIG. 6, when the extracted two pair areas 200 share the cell 100, the calculation unit 32 calculates the number of the pair areas 200 as two and sums the edge strengths. It is calculated as the total value of the edge intensities of the three cells 100 included in the two pair areas 200.

In addition, the calculation unit 32 calculates representative values for two or more types of edges for one cell 100 based on, for example, not only the angle classification of the above-mentioned "up / down / left / right 4 classifications" but also the angle classification of the "diagonal 4 classifications". Allocation and area feature amount may be calculated. This point will be described with reference to FIGS. 7 and 8. 7 and 8 are diagrams (No. 5) and (No. 6) showing the processing contents of the calculation unit 32.

If the calculation unit 32 uses "up, down, left, and right 4 classifications" as the first angle classification and the representative value for the edge direction based on this as the first representative value, as shown in FIG. 7, the "diagonal 4 classification" is the first. The angle classification of 2 can be used, and the representative value for the edge direction based on this can be calculated as the second representative value.

In such a case, the calculation unit 32 classifies the edge direction −180 ° to 180 ° of the edge vector V of each pixel PX in the cell 100 by the second angle classification into four diagonal directions of 90 ° (4). Classify into ~ (7).

More specifically, when the edge orientation of the pixel PX is in the angle range of 0 ° or more and less than 90 °, the calculation unit 32 classifies it into the angle classification (4), and the angle range of 90 ° or more and less than 180 °. If it is, it is classified into angle classification (5), if it is in the angle range of -180 ° or more and less than -90 °, it is classified in angle classification (6), and in the angle range of -90 ° or more and less than 0 °. If there is, it is classified into angle classification (7).

Then, as shown in the lower part of FIG. 4, the calculation unit 32 generates a histogram for each cell 100 with the angle classifications (4) to (7) as each class. Then, when the frequency of the class having the highest frequency is equal to or higher than the predetermined threshold value THa in the generated histogram, the calculation unit 32 uses the angle classification corresponding to the class as the second representative value for the edge in the cell 100. calculate.

As a result, as shown in FIG. 8, two edge-oriented representative values can be assigned to each cell 100. Then, as shown in FIG. 8, in the adjacent cell 100, when at least one of the first representative value and the second representative value in the edge direction are opposite to each other, the adjacent cell 100 Is extracted as the pair area 200.

That is, the calculation unit 32 can extract the pair area 200 that could not be extracted only by one type of edge orientation by calculating the first representative value and the second representative value for the edge in each cell 100. It becomes.

For example, a pixel PX having an edge orientation of 140 ° and a pixel PX having an edge orientation of −40 ° are not opposite in the first angle range, but are opposite in the second angle range, so that the cell 100 It becomes possible to detect the change in the edge orientation in more accurately.

In addition, the calculation unit 32 maps the area feature amount for each unit area UA calculated in this way to the area feature amount space in which each element of the area feature amount is a dimension to the determination unit 34 described later, and the area. It is possible to estimate the state of attachment of snow, which is an deposit, based on the position of the area feature amount in the feature amount space.

This point will be described with reference to FIGS. 9 and 10. 9 and 10 are diagrams (No. 7) and (No. 8) showing the processing contents of the calculation unit 32.

For example, as shown in FIG. 9, the calculation unit 32 determines the snow adhesion state as "adhesion" and "adhesion" based on the average edge strength and the position in the two-dimensional space in which the dispersion of the edge strength is one dimension. It can be estimated as either "non-adhesive" or "difficult to judge".

Further, for example, as shown in FIG. 10, the calculation unit 32 is based on the number of pair regions 200 and the position in the two-dimensional space in which the sum of the edge strengths of the pair regions 200 is one dimension, and the snow adhesion state. Can be estimated as either "adherent" or "non-adhered".

Here, "adhesion" is a state in which the background cannot be seen and is buried in snow. In addition, "non-adhesive" is a state in which the background is visible. In addition, "difficult to judge" is a state in which it cannot be seen due to overexposure or the like.

In the example of each region feature space shown in FIGS. 9 and 10, the region feature of each unit region UA of the sample data is calculated in advance based on a large amount of sample data of the captured image I, and the actual region feature is calculated. The sample points are color-coded according to the adhesion state and mapped to each space. Therefore, each threshold value (see the dotted line in the figure) that separates each adhesion state is defined according to, for example, the color division of the sample points. Although the threshold values are defined by separating them with straight lines in FIGS. 9 and 10, they may be defined by a curve having a shape along the color division of the sample points for convenience of explanation.

For example, in the "non-adherent" state, a relatively large amount of the pair area 200 is extracted due to white lines on the road, guardrails, contours of buildings, and the like, and the edge strength of the cell 100 is also increased. Therefore, the total edge strength of the pair region 200 is also relatively large. On the other hand, in the "adhered" state, the brightness of the unit region UA is uniform and the edge strength of the cell 100 is also small, so that the number of pair regions 200 to be extracted is relatively small, and the pair region is also relatively small. The total edge strength of 200 is also relatively small.

Therefore, paying attention to this point, as shown in FIG. 10, a region feature in which the number of pair regions 200 and the sum of the edge strengths of the pair regions 200 are set as each dimension based on the sample data of the captured image I in advance. If a quantity space is generated and the calculated region feature amount is mapped to such a space, the adhesion state of the deposit for each unit region UA is estimated as either "adhesion" or "non-adhesion" based on the position. be able to.

Further, the average edge strength and the dispersion of the edge strength shown in FIG. 9 are based on statistical viewpoints. In other words, it can be said that the average and variance of the edge strength corresponding to the actual adhesion state for each unit region UA are learned based on a large amount of sample data, and the state is determined based on the result of such learning.

Therefore, paying attention to this point, as shown in FIG. 9, based on the sample data of the captured image I in advance, a region feature space having the average edge strength and the dispersion of the edge strength as each dimension is generated. If the calculated region feature amount is mapped to such a space, the adhered state of the adhered matter for each unit region UA is set as either "adhered", "non-adhered", or "difficult to judge" based on the position. Can be estimated.

Therefore, according to the calculation unit 32 according to the embodiment, it is possible to improve the detection accuracy of the deposits.

Further, the calculation unit 32 calculates the number of intersections at the time of pattern matching as the area feature amount. Here, the case of calculating the number of intersections at the time of pattern matching will be described with reference to FIGS. 11 to 14.

11 to 14 are diagrams (No. 9) to (No. 12) showing the processing contents of the calculation unit 32. It is assumed that the amorphous hatched portions shown in FIGS. 11, 13 and 14 are pattern portions having a predetermined edge feature amount in the captured image I.

The calculation unit 32 searches for a predetermined search pattern that matches a predetermined template by using the edge orientation of the calculated edge feature amounts of the cell 100. As shown in FIG. 11, the search directions are the horizontal direction and the vertical direction.

For example, the calculation unit 32 searches for a search pattern on the condition that "opposite angle classifications do not appear on both sides of the angle classification of interest". Specifically, when the angle classification of interest is searched in the left-right direction as the angle classification (1), as shown in FIG. 12, for example, the start position is the cell of the angle classification (2) in which the angle classification is not in the opposite direction. The starting position is cell 100-2 of the angle classification (1) adjacent to 100-1.

Then, when the array of the angle classification (1) continues and the cell 100-4 of the angle classification (0) in which "the angle classification is not in the opposite direction" appears, the cell of the angle classification (1) adjacent to the cell 100-4 appears. 100-3 is the end position. In such a case, the match length is "8" in the example of FIG. When there is a match with the search pattern in this way, the calculation unit 32 holds the position and the match length.

Further, when there is a match in the search pattern shown in FIG. 12, the brightness changes between the adjacent classifications among the four types of angle classifications can be seen at the start position and the end position.

Then, as shown in FIG. 13, when the search patterns match in the left-right direction and the up-down direction, the calculation unit 32 horizontally matches each angle classification as storage information corresponding to the unit area UA corresponding to the intersection. Cumulatively add the product of length and top and bottom match lengths.

Specifically, according to the examples of FIGS. 12 and 13, as shown in FIG. 14, the calculation unit 32 cumulatively adds 5 × 8 in association with the angle classification (1) of the unit area. Further, although not shown, the calculation unit 32 also cumulatively adds the number of intersections in association with the angle classification (1) of the unit area.

By repeating such a matching process, the detection unit 33, which will be described later, determines, for example, three or more of the cumulative addition results associated with the angle classifications (0) to (3) of the unit region UA based on the predetermined detection conditions. If it is equal to or more than the threshold value of, the adhesion state of the unit region UA is determined to be “adhesion”. Further, if the determination condition is not satisfied, it is determined to be "non-adherent". The processing contents of the matching processing shown in FIGS. 11 to 14 are merely examples, and the processing contents are not limited.

Further, the calculation unit 32 calculates the total number of changes in the angle classification as the area feature amount. Here, the case of calculating the total number of changes in the angle classification will be described with reference to FIGS. 15 and 16. 15 and 16 are diagrams (No. 13) and (No. 14) showing the processing contents of the calculation unit 32.

As shown in FIG. 15, the calculation unit 32 calculates the number of changes in the angle classification of the cells 100 arranged in the horizontal direction and the vertical direction in the unit area UA for each unit area UA of the angle classification image.

Specifically, as shown in the figure, there is an array of cells 100-1, 100-2, 100-3, 100-4, and the angle classifications for the edges are (0), (1), and (1), in order. It is assumed that it was 1) and (2).

In such a case, when the calculation unit 32 scans the cells 100-1 to 100-2, the angle classification changes from (0) to (1), so that the number of changes in the angle classifications (0) and (1) is calculated. Add +1 to each. Further, in the same arrangement, the calculation unit 32 changes the angle classification from (1) to (2) when scanning from cell 100-3 to cell 100-4, so that the angle classifications (1) and (2) are Increase the number of changes by +1.

In this way, the calculation unit 32 counts the number of changes in the angle classification of the cell 100 for each unit area UA, and as shown in FIG. 16, classifies the angles in each of the "upper, lower, left, and right 4 classifications" and "diagonal 4 classifications". Calculate the total number of changes in.

Returning to the description of FIG. Further, the calculation unit 32 outputs the calculated area feature amount for each unit area UA to the detection unit 33.

The detection unit 33 detects the adhesion state of each unit area UA according to a predetermined detection condition included in the condition information based on the area feature amount for each unit area UA calculated by the calculation unit 32.

Specifically, when the buried flag is off, that is, when it is determined that the lens is not buried in the previous frame, the detection unit 33 has an adhesion state for each unit region UA according to the first detection condition. Is detected.

Further, when the buried flag is not turned off, that is, when it is determined that the lens is buried in the previous frame, the detection unit 33 sets each unit region UA according to the first detection condition and the second detection condition. Detects the adhesion state of.

The detection unit 33 comprehensively describes each unit area UA based on the estimation result using the estimation information 22, the matching result by the calculation unit 32, and the detection result for the past frames included in the determination history information 24. Adhesion state can be detected.

For example, the detection unit 33 performs a detection process in which the matching result by the calculation unit 32 is added to the estimation result obtained by inputting the region feature amount calculated by the calculation unit 32 into the estimation information 22.

Specifically, when the estimation result is "non-adhesive", the detection unit 33 detects it as "non-adhesive" regardless of the matching result. As a result, it is possible to suppress erroneous detection when the matching result determines "adhesion".

Further, when the estimation result is "adhesion" and the matching result is "adhesion", the detection unit 33 detects it as "adhesion".

Further, when the estimation result is "adhesion" but the matching result is "non-adhesion", the detection unit 33 has detected "adhesion" in the past n frames, and after such detection, "adhesion" is once detected. If there is no detection result of "non-adhesion", it is detected as "adhesion".

Further, when the estimation result is "difficult to determine", the detection unit 33 has a detection result of "adhesion" in the past n frames regardless of the matching result, and after such detection, it is said to be "non-adhesion" even once. If there is no detection result of, it is detected as "adhesion".

In this way, the accuracy of detecting the adhered matter can be improved by comprehensively detecting the adhered state for each unit region UA.

Then, the detection unit 33 outputs the detection result to the determination unit 34. The determination unit 34 calculates the adhesion rate based on the detection result of the detection unit 33, and determines whether or not the lens of the camera 10 is buried based on the adhesion rate.

Here, FIG. 17 is a diagram showing the processing contents of the determination unit 34. As shown in FIG. 17, the determination unit 34 determines the number of adhesion points detected according to the first detection condition and the number of adhesion points detected according to the second detection condition when calculating the adhesion rate. And add up.

Then, the determination unit 34 calculates the area ratio of the adhesion portion in the predetermined region of interest, that is, the adhesion ratio, based on the total value. Then, when the adhesion rate is equal to or higher than a certain level, the determination unit 34 determines that the lens is buried and turns on the buried flag. Further, when the adhesion rate is not more than a certain level, the determination unit 34 determines that the lens is not buried and turns off the buried flag.

Returning to the description of FIG. Then, the determination unit 34 notifies the various devices 50 of the determination result of the determination.

Next, the processing procedure executed by the deposit detection device 1 according to the embodiment will be described with reference to FIG. FIG. 18 is a flowchart showing a processing procedure executed by the deposit detection device 1 according to the embodiment. Note that FIG. 18 shows a processing procedure for the captured image I for one frame.

As shown in FIG. 18, first, the acquisition unit 31 acquires the captured image I (step S101). At the same time, the acquisition unit 31 performs grayscale processing and smoothing processing on the captured image I.

Subsequently, the calculation unit 32 calculates the edge feature amount for each cell 100 of the captured image I, and calculates the area feature amount for each unit area UA based on the calculated edge feature amount (step S102).

Then, the detection unit 33 determines whether or not the buried flag is off (step S103). Here, when the buried flag is off (step S103, Yes), that is, when it is determined that the lens is not buried in the previous frame, the detection unit 33 determines the unit region UA according to the first detection condition. Each adhesion state is detected (step S104).

On the other hand, when the buried flag is not turned off (step S103, No), that is, when it is determined that the lens is buried in the previous frame, the detection unit 33 conforms to the first detection condition and the second detection condition. The adhesion state of each unit region UA is detected (step S105).

Then, the determination unit 34 determines whether or not the adhesion rate is equal to or higher than a certain level based on the detection result of the detection unit 33 (step S106). Here, when the adhesion rate is equal to or higher than a certain level (step S106, Yes), the determination unit 34 determines that the lens is buried and turns on the buried flag (step S107).

If the adhesion rate is not equal to or higher than a certain level (step S106, No), the determination unit 34 determines that the lens is not buried and turns off the buried flag (step S108).

Then, the determination unit 34 notifies the various devices 50 of the determination result (step S109), and ends the process.

As described above, the deposit detection device 1 according to the embodiment includes a calculation unit 32, a detection unit 33, and a determination unit 34. The calculation unit 32 calculates a region feature amount based on the edge vector of each pixel PX for each unit region UA composed of a predetermined number of pixel PX included in the captured image I. The detection unit 33 detects the state of adhesion of the deposit to the unit region UA according to the first detection condition based on the region feature amount. The determination unit 34 determines the adhesion rate of the deposits to the lens of the camera 10 based on the detection result of the detection unit 33. Further, when the determination unit 34 determines that the adhesion rate is equal to or higher than a certain level, the detection unit 33 adheres the adhered matter according to the first detection condition and the second detection condition based on the above-mentioned region feature amount. Detect the condition.

Therefore, according to the deposit detection device 1 according to the embodiment, it is possible to improve the detection accuracy of the deposit.

Further, the determination unit 34 has a unit region UA detected as an adhering portion according to the first detection condition and a unit region UA detected as an adhering portion according to the second detection condition. Is added up to calculate the adhesion rate.

Therefore, according to the deposit detection device 1 according to the embodiment, it is possible to suppress a decrease in the adhesion rate due to a change in the mode of the deposit and stabilize the buried flag.

Further, the deposit is snow, and the detection unit 33 detects the state of snow adhesion before it adheres to the lens and begins to melt according to the first detection condition, and although it adheres to the lens, it begins to melt. The state of snow adhesion after the lens is detected according to the second detection condition.

Therefore, according to the deposit detection device 1 according to the embodiment, it is possible to suppress a decrease in the adhesion rate due to the snow adhering but starting to thaw, and to stabilize the buried flag.

In the above-described embodiment, the case where −180 ° to 180 ° is divided into four directions divided by an angle range of 90 ° is shown, but the angle range is not limited to 90 °, for example, every 60 °. The angle may be classified into 6 directions divided by the angle range of.

Further, the width of each angle range may be different between the first angle classification and the second angle classification. For example, in the first angle classification, the angle may be classified by 90 °, and in the second angle classification, the angle may be classified by 60 °. Further, in the first angle classification and the second angle classification, the boundary of the angle in the angle range is shifted by 45 °, but the shifting angle may be more than 45 ° or less than 45 °.

Further, in the above-described embodiment, the captured image I captured by the in-vehicle camera is taken as an example, but the captured image I is, for example, the captured image I captured by a security camera, a camera installed in a street light, or the like. There may be. That is, it may be the captured image I captured by the camera in which deposits may adhere to the lens of the camera.

Further effects and variations can be easily derived by those skilled in the art. For this reason, the broader aspects of the invention are not limited to the particular details and representative embodiments expressed and described as described above. Therefore, various modifications can be made without departing from the spirit or scope of the general concept of the invention as defined by the appended claims and their equivalents.

1 Adhesion detection device 2 Storage unit 3 Control unit 10 Camera 21 Template information 22 Estimated information 23 Condition information 24 Judgment history information 31 Acquisition unit 32 Calculation unit 33 Detection unit 34 Judgment unit 50 Various devices 100 Cell 200 Pair area I Captured image PX Pixel UA unit area

Claims (4)

A calculation unit that calculates a region feature amount based on the edge vector of each pixel for each unit region consisting of a predetermined number of pixels included in the captured image.
A detection unit that detects the state of adhesion of deposits to the unit region according to the first detection condition based on the region feature amount, and
A determination unit for determining the adhesion rate of deposits to the lens of the camera based on the detection result of the detection unit is provided.
The detection unit
When the determination unit determines that the adhesion rate is equal to or higher than a certain level, the adhesion state of the deposit is detected according to the first detection condition and the second detection condition based on the region feature amount, respectively. A deposit detection device characterized by. The determination unit
The unit region detected as an adhering portion of the deposit according to the first detection condition and the unit region detected as the adhering portion of the deposit according to the second detection condition are added together. The deposit detection device according to claim 1, wherein the adhesion rate is calculated. The deposit is snow,
The detection unit
The snow adhesion state before the lens adheres to the lens and begins to melt is detected according to the first detection condition, and the snow adhesion state after the lens adheres to the lens but begins to melt is detected by the second detection condition. The deposit detection device according to claim 1 or 2, wherein the detection is performed according to the detection conditions. A calculation process for calculating a region feature amount based on the edge vector of each pixel for each unit region consisting of a predetermined number of pixels included in the captured image, and a calculation process.
A detection step of detecting the state of adhesion of deposits to the unit region according to the first detection condition based on the region feature amount, and
Including a determination step of determining the adhesion rate of deposits to the lens of the camera based on the detection result of the detection step.
The detection step is
When it is determined by the determination step that the adhesion rate is equal to or higher than a certain level, the adhesion state of the deposit is detected according to the first detection condition and the second detection condition based on the region feature amount, respectively. A method for detecting deposits, which comprises.

Images