JP7274867B2 - Attached matter detection device and attached matter detection method - Google Patents

Description

The disclosed embodiments relate to a deposit detection device and a deposit detection method.

2. Description of the Related Art Conventionally, there has been known an adhering matter detection device that detects an adhering matter adhering to a camera lens based on an image captured by a camera mounted on a vehicle or the like. 2. Description of the Related Art For example, there is an attached matter detection device that detects an attached matter based on a difference between time-series captured images (see, for example, Patent Document 1).

JP 2012-038048 A

However, the conventional technology described above has room for further improvement in terms of improving the detection accuracy of adhering matter.

One aspect of the embodiments has been made in view of the above, and an object thereof is to provide an adhering matter detection device and an adhering matter detection method capable of improving the detection accuracy of an adhering matter.

An attached matter detection device according to an aspect of an embodiment includes a control unit . For each cell consisting of a predetermined number of pixels included in the captured image, the control unit classifies an edge direction obtained by classifying an angle feature amount based on an edge vector of each pixel included in the cell for each predetermined angle range for each cell. Extract to Further, the control unit controls the number of combinations of the cells adjacent to each other and having the opposite edge directions as a region feature amount for each unit region, which is a predetermined region composed of a predetermined number of the cells, and A sum of edge strengths of each cell included in the combination is calculated. Further, the control unit maps the calculated area feature amount to a feature amount space having the number and the sum as each dimension, and performs mapping based on the position of the mapped area feature amount in the feature amount space. to estimate the adhesion state of the adhering matter in the unit area.

According to one aspect of the embodiment, it is possible to improve the detection accuracy of adhering matter.

FIG. 1A is a schematic explanatory diagram (part 1) of an attached matter detection method according to an embodiment; FIG. 1B is a schematic explanatory diagram (part 2) of the attached matter detection method according to the embodiment; FIG. 1C is a schematic explanatory diagram (part 3) of the attached matter detection method according to the embodiment; FIG. 1D is a schematic explanatory diagram (part 4) of the attached matter detection method according to the embodiment; FIG. 2 is a block diagram of the attached matter detection device according to the embodiment. FIG. 3 is a diagram (part 1) illustrating the processing contents of an extraction unit; FIG. 4 is a diagram (part 2) showing the processing contents of the extraction unit. FIG. 5 is a diagram (part 1) illustrating the processing contents of the calculation unit; FIG. 6 is a diagram (part 2) illustrating the processing contents of the calculation unit; FIG. 7A is a diagram (part 1) showing the processing contents of a matching unit; FIG. 7B is a diagram (part 2) showing the processing contents of the matching unit; FIG. 7C is a diagram (part 3) showing the processing contents of the matching unit; FIG. 7D is a diagram (part 4) showing the processing contents of the matching unit; FIG. 7E is a diagram (No. 5) showing the processing contents of the matching unit; FIG. 8A is a diagram (part 1) showing the processing contents of the determination unit; FIG. 8B is a diagram (part 2) showing the processing contents of the determination unit; FIG. 9 is a flowchart illustrating a processing procedure executed by the adhering matter detection device according to the embodiment. FIG. 10 is a diagram (part 1) illustrating the processing contents of the extraction unit according to the modification; FIG. 11 is a diagram (part 2) illustrating the processing contents of the extraction unit according to the modification;

DETAILED DESCRIPTION OF THE INVENTION Embodiments of the attached matter detection device and the attached matter detection method disclosed in the present application will be described in detail below with reference to the accompanying drawings. In addition, this invention is not limited by embodiment shown below.

First, with reference to FIGS. 1A to 1D, an outline of an attached matter detection method according to an embodiment will be described. 1A to 1D are schematic explanatory diagrams (Part 1) to (Part 4) of an attached matter detection method according to an embodiment.

As shown in FIG. 1A, for example, it is assumed that there is a captured image I captured in a state where most of the surface of the lens of an in-vehicle camera is covered with snow. Below, the attached matter detection apparatus 1 (see FIG. 2) to which the attached matter detection method according to the embodiment is applied detects a feature amount (hereinafter referred to as an "edge feature amount") related to the luminance gradient of each pixel PX of the captured image I. ), the case of detecting the snow adhesion state for each unit area UA divided by a predetermined number of pixels PX in the captured image I will be described.

Note that the adhering matter is not limited to snow, and may be, for example, light-colored mud. In other words, the adhering matter blocks the object in the captured image I to such an extent that it does not appear in the image, but allows a certain amount of light to pass therethrough and causes a slight change in luminance depending on how much light is transmitted.

By the way, as an attached matter detection device according to a comparative example, there is one that detects attached matter based on the difference between time-series captured images. However, according to this technique, when the entire lens is buried in snow, for example, it becomes difficult to generate a time-series difference, so there is a possibility that the adhering matter cannot be detected.

In addition, according to this technique, in a captured image buried in snow, for example, while driving in a tunnel, the brightness of a portion of the captured image I temporarily increases due to the light from the light inside the tunnel. In such a case, the strength of the edge of each pixel increases in a high-brightness region, and there is a risk of erroneous detection that there is no adhering matter.

Therefore, in the adhering matter detection method according to the embodiment, the adhering matter is detected based on the feature amount of each pixel PX edge extracted from the captured image I. FIG. The edge features include angle features and strength features. The angle feature amount is the direction of the edge vector (brightness gradient) of each pixel (hereinafter sometimes referred to as "edge direction"). The intensity feature amount is the magnitude of the edge vector of each pixel (hereinafter sometimes referred to as "edge intensity").

In addition, in the adhering matter detection method according to the embodiment, such an edge feature amount is handled in units of cells 100 each having a predetermined number of pixels PX. This can help reduce the processing load in image processing.

Also, the unit area UA described above is a collection of such cells 100 . Then, in the attached matter detection method according to the embodiment, based on the edge feature amount extracted for each cell 100, the area feature amount, which is the feature amount for each unit area UA, is calculated. The area feature amount is, in other words, a statistical feature amount of the edge feature amount for each unit area UA. In addition, in the adhering matter detection method according to the embodiment, the state of adhering snow in each unit area UA is estimated based on the position of the calculated area feature amount in the area feature amount space.

Specifically, as shown in FIG. 1A, in the attached matter detection method according to the embodiment, in a captured image I, a unit area UA, which is a predetermined area in which a predetermined number of cells 100 are arranged in the vertical direction and the horizontal direction, is detected. Each time, the edge feature amount is first extracted in units of 100 cells (step S1). The edge feature amounts are the edge direction and the edge strength as described above.

The edge orientation of the cell 100 is, as shown in FIG. 1A, the representative value of the vector orientation of each pixel PX classified by a predetermined angular range. Either up, down, left, or right is determined. Also, the edge intensity of the cell 100 is the representative value of the vector intensity of each pixel PX. Note that the edge feature amount extraction processing for each cell 100 will be described later with reference to FIGS. 3 and 4. FIG.

Subsequently, in the attached matter detection method according to the embodiment, an area feature amount for each unit area UA is calculated based on the edge feature amount for each cell 100 extracted in step S1 (step S2). Specifically, in step S2, first, the average and variance of the edge intensity of the cell 100 are calculated for each unit area UA (step S2-1).

In step S2, the number of paired areas and the sum of the edge strengths of the paired areas are calculated for each unit area UA (step S2-2). Here, a pair region is a combination of adjacent cells 100 whose edge directions are opposite to each other.

The example of FIG. 1A shows a pair region 200a in which two cells 100 are adjacent in the vertical direction and a pair region 200b in which the cells are adjacent in the horizontal direction. In the following, when the pair region 200a and the pair region 200b are not particularly distinguished, they are collectively referred to as the pair region 200. FIG.

Then, in the attached matter detection method according to the embodiment, the calculated area feature amount for each unit area UA is mapped to an area feature amount space having each element of the area feature amount as each dimension, and the area feature amount in the area feature amount space is mapped. Based on the position of the amount, the adhesion state of the snow, which is the adhesion matter, is estimated (step S3).

For example, in the attached matter detection method according to the embodiment, as shown in FIG. Estimated as either "adhesive", "non-adherent", or "difficult to judge".

Further, for example, in the attached matter detection method according to the embodiment, as shown in FIG. , estimates the snow adhesion state as either "adhered" or "non-adhered".

Here, as shown in FIG. 1D, "adhesion" is the state of "buried in snow" seen in the unit area UA(7,8). "Non-attachment" is the state of "the background is visible" seen in the unit area UA (4, 11). "Difficult to determine" is the state of "not visible due to overexposure, etc." seen in the unit area UA(3,7).

1B and 1C, based on a large amount of sample data of the captured image I, the area feature amount of each unit area UA of the sample data is calculated in advance, and the actual The sample points are color-coded according to the adhesion state and mapped in each space. Therefore, each threshold value (see the dotted line in the figure) that separates each adhesion state is defined according to, for example, the separation of the colors of the sample points. In FIGS. 1B and 1C, the thresholds are defined by separating them with straight lines, but for convenience of explanation, they may be defined by curved lines that follow the color divisions of the sample points.

The number of paired regions 200 shown in FIG. 1C is the sum of the number of vertically adjacent paired regions 200a and the number of horizontally adjacent paired regions 200b. The sum of the edge strengths of the paired regions 200 is the sum of the edge strengths of all the cells 100 included in the paired regions 200 . The number of pair regions 200 and the method of calculating the sum of edge strengths will be described later with reference to FIGS. 5 and 6. FIG.

For example, in the "non-attached" state, a relatively large number of pair regions 200 are extracted due to white lines on roads, guardrails, outlines of buildings, etc., and the edge strength of cells 100 is also high. Therefore, the sum of the edge strengths of the paired regions 200 is also relatively large. On the other hand, in the “adhered” state, the brightness of the unit area UA is uniform and the edge strength of the cell 100 is small, so the number of pair areas 200 to be extracted is relatively small, and the pair areas 200 are extracted. The sum of 200 edge strengths is also relatively small.

Focusing on this point, as shown in FIG. 1C, based on the sample data of the captured image I, region feature By generating an amount space and mapping the area feature amount calculated in step S2 in this space, the adherence state of the adhering matter for each unit area UA can be either "adhered" or "non-adhered" based on the position. can be estimated as

Also, the average edge strength and the variance of edge strength shown in FIG. 1B are based on statistical considerations. In other words, based on a large amount of sample data, the mean and variance of edge strength corresponding to the actual adhesion state of each unit area UA are learned, and the state is discriminated based on the learning result.

Therefore, focusing on this point, as shown in FIG. 1B, based on the sample data of the captured image I, a region feature amount space is generated in which each dimension is the average edge strength and the variance of the edge strength. If the area feature amount calculated in step S2 is mapped in this space, the adherence state of the adhering matter for each unit area UA can be classified as "adhered", "non-adhered", or "difficult to determine" based on the position. can be estimated as either

Therefore, according to the adhering matter detection method according to the embodiment, the adhering matter detection accuracy can be improved.

It should be noted that the estimation result estimated in the above-described adhering matter detection method may be subjected to a determination that takes into consideration the matching result of the matching process executed based on the above-described edge feature amount, for example, to further improve the detection accuracy. . Further, the judgment results for the past frames of the captured image I may be added. Details of these points will be described later with reference to FIGS. 7A to 8B.

FIG. 1A shows a case where one type of edge orientation representative value is extracted for each cell 100. However, for example, by further extracting edge orientation representative values with different angle ranges, two A type or more than one type of edge orientation may be assigned. A case of extracting representative values of two types of edge directions will be described later with reference to FIGS. 10 and 11. FIG.

Hereinafter, a configuration example of the adhering matter detection device 1 to which the adhering matter detection method according to the above-described embodiment is applied will be described more specifically.

FIG. 2 is a block diagram of the adhering matter detection device 1 according to the embodiment. In addition, in FIG. 2, only the components necessary for explaining the features of the present embodiment are represented by functional blocks, and the description of general components is omitted.

In other words, each component illustrated in FIG. 2 is functionally conceptual and does not necessarily need to be physically configured as illustrated. For example, the specific forms of distribution and integration of each functional block are not limited to those shown in the figure, and all or part of them can be functionally or physically distributed in arbitrary units according to various loads and usage conditions.・It is possible to integrate and configure.

As shown in FIG. 2 , the attached matter detection device 1 according to the embodiment includes a storage unit 2 and a control unit 3 . Also, the adhering matter detection device 1 is connected to the camera 10 and various devices 50 .

Although FIG. 2 shows the case where the adhering matter detection device 1 is configured separately from the camera 10 and the various devices 50, the present invention is not limited to this. may be configured.

The camera 10 is, for example, an in-vehicle camera that includes a lens such as a fisheye lens and an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is provided at a position capable of capturing images of the front, rear, and sides of the vehicle, for example, and outputs the captured image I to the adhering matter detection device 1 .

The various devices 50 are devices that acquire detection results of the adhering matter detection device 1 and perform various vehicle controls. The various devices 50 include, for example, a display device that notifies the user that there is a deposit on the lens of the camera 10 and instructs the user to wipe off the deposit; It includes a removal device that removes, and a vehicle control device that controls automatic driving and the like.

The storage unit 2 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disc. In the example of FIG. Template information 22 and determination history information 23 are stored.

The threshold information 21 is information about a threshold used in estimation processing executed by the estimation unit 34, which will be described later. The threshold referred to here corresponds to the threshold in each region feature amount space indicated by dashed lines in FIGS. 1B and 1C.

The template information 22 is information about templates used in matching processing executed by the matching unit 35, which will be described later. The determination history information 23 is information relating to the determination history of adhering matter detection in the captured image I for predetermined past n frames.

The control unit 3 is a controller, and various programs stored in a storage device inside the adhering matter detection apparatus 1 operate in the RAM by means of, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). It is realized by being executed as a region. Also, the control unit 3 can be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 3 includes an acquisition unit 31, an extraction unit 32, a calculation unit 33, an estimation unit 34, a matching unit 35, and a determination unit 36, and implements the information processing functions and actions described below. or run.

The acquisition unit 31 acquires the captured image I captured by the camera 10 . The acquisition unit 31 performs grayscaling processing for converting each pixel in the acquired captured image I into each gradation from white to black according to the luminance, and performs smoothing processing on each pixel, and outputs the data to the extraction unit 32. Output. Any smoothing filter such as an averaging filter or a Gaussian filter can be used for the smoothing process. Also, the grayscaling process and the smoothing process may be omitted.

The extraction unit 32 extracts an edge feature amount for each cell 100 of the captured image I acquired from the acquisition unit 31 . Here, extraction processing by the extraction unit 32 will be specifically described with reference to FIGS. 3 and 4. FIG.

3 and 4 are diagrams (part 1) and (part 2) showing the processing contents of the extraction unit 32. FIG. As shown in FIG. 3, the extraction unit 32 first performs edge detection processing for each pixel PX, and determines the intensity of the edge ex in the X-axis direction (horizontal direction of the captured image I) and the intensity of the edge ex in the Y-axis direction (the captured image I and the strength of the edge ey in the vertical direction of ). Any edge detection filter such as a Sobel filter or a Prewitt filter can be used for edge detection processing.

Subsequently, the extraction unit 32 calculates an edge vector V using a trigonometric function based on the detected intensity of the edge ex in the X-axis direction and the intensity of the detected edge ey in the Y-axis direction, and calculates the edge vector V and The edge direction, which is the angle θ formed with the X axis, and the edge strength, which is the length L of the edge vector V, are extracted.

Subsequently, the extraction unit 32 extracts the representative value of the edge direction in the cell 100 based on the calculated edge vector V of each pixel PX. Specifically, as shown in the upper part of FIG. 4, the extracting unit 32 extracts the edge direction of the edge vector V of each pixel PX in the cell 100 from −180° to 180° in four directions of up, down, left, and right at every 90°. It is classified into angle classifications (0) to (3).

More specifically, the extraction unit 32 classifies the edge direction of the pixel PX into the angle category (0) when the angle range is −45° or more and less than 45°, and the angle of 45° or more and less than 135° If it is in the range, it is classified into angle classification (1), and if it is in the angle range of 135° or more and less than 180°, or -180° or more and less than -135°, it is classified into angle classification (2), and -135 If the angle range is greater than or equal to degrees and less than -45 degrees, it is classified as angle classification (3).

Then, as shown in the lower part of FIG. 4, the extracting unit 32 generates a histogram with angle classifications (0) to (3) for each class for each cell 100. FIG. Then, when the frequency of the class with the highest frequency in the generated histogram is equal to or greater than a predetermined threshold value THa, the extraction unit 32 performs an angle classification corresponding to the class (angle classification (1) in the example of FIG. 4). is extracted as the representative value of the edge orientation in the cell 100 .

The frequency of the histogram described above is calculated by adding together the edge intensities of the pixels PX classified into the same angular range among the pixels PX in the cell 100 . Specifically, consider the histogram frequency in the class of angle classification (0). For example, assume that there are three pixels PX classified into angle classification (0), and the edge strengths of the pixels PX are 10, 20, and 30, respectively. In this case, the frequency of the histogram in the class of angle classification (0) is calculated as 10+20+30=60.

Based on the histogram calculated in this way, the extraction unit 32 extracts the representative value of the edge strength in the cell 100 . Specifically, when the frequency of the class with the highest frequency in the histogram is equal to or greater than a predetermined threshold THa, the representative value of the edge strength is the edge strength of the cell 100 corresponding to the class. That is, the process of extracting the representative value of the edge strength in the extraction unit 32 can be said to be the process of extracting the feature related to the strength of the edge in the cell 100 corresponding to the representative value of the edge direction.

On the other hand, if the frequency of the class with the highest frequency is less than the predetermined threshold value THa, the extraction unit 32 determines that the edge direction of the cell 100 is “invalid”, in other words, “there is no edge direction representative value”. treated as As a result, it is possible to prevent a specific edge orientation from being extracted as a representative value when there is a large variation in the edge orientation of each pixel PX.

The processing contents of the extraction unit 32 shown in FIGS. 3 and 4 are merely examples, and the processing contents may be arbitrary as long as the representative value of the edge direction can be extracted. For example, the average value of the edge orientation of each pixel PX in the cell 100 may be calculated, and the angle classifications (0) to (3) corresponding to the average value may be used as the representative values of the edge orientation.

In addition, although FIG. 4 shows a case where a total of 16 pixels PX (4×4) are defined as one cell 100, the number of pixels PX in the cell 100 may be set arbitrarily. For example, the number of pixels PX in the vertical direction and the horizontal direction may be different.

Returning to FIG. 2, the calculation unit 33 will be described. The calculation unit 33 calculates the area feature amount for each unit area UA based on the edge feature amount for each cell 100 extracted by the extraction unit 32 .

Specifically, the calculation unit 33 calculates the average value and variance value of the edge strength for each unit area UA. The calculation unit 33 also calculates the number of pair regions 200 and the sum of edge strengths for each unit region UA.

Calculation processing of the number of pair regions 200 and the sum of edge strengths by the calculation unit 33 will now be described with reference to FIGS. 5 and 6. FIG. 5 and 6 are diagrams (part 1) and (part 2) showing the processing contents of the calculation unit 33. FIG.

5 shows the case where the two pair regions 200 do not share the cell 100, and FIG. 6 shows the case where the two pair regions 200 share the cell 100. As shown in FIG.

As shown in FIG. 5 , the calculation unit 33 scans the plurality of cells 100 arranged in the horizontal direction and the vertical direction of the unit area UA in the horizontal direction and the vertical direction to search for the pair area 200 . That is, the calculation unit 33 extracts, as the pair area 200, the cells 100 that are adjacent to each other and whose edge directions are opposite to each other among the cells 100 in the unit area UA.

Calculation unit 33 then calculates the number of extracted pair regions 200 and the sum of edge strengths in pair regions 200 . Note that, as shown in FIG. 5, the calculation unit 33 calculates the number of the pair regions 200 as two when, for example, the two pair regions 200 extracted do not share the cell 100, and the sum of the edge strengths is calculated as follows. It is calculated as a sum of the edge strengths of the four cells 100 included in the two pair regions 200 .

Further, as shown in FIG. 6 , if, for example, two paired regions 200 that are extracted share a cell 100, the calculation unit 33 calculates the number of paired regions 200 as two, and calculates the sum of the edge strengths as follows: It is calculated as a sum of the edge strengths of the three cells 100 included in the two pair regions 200 .

Returning to the description of FIG. Then, the calculation unit 33 outputs the calculated area feature amount for each unit area UA to the estimation unit 34 . The estimating unit 34 maps the area feature amount for each unit area UA calculated by the calculating unit 33 to the above-described area feature amount space, and based on the position of the mapped area feature amount and the threshold information 21, for each unit area UA Estimate the adhesion state of the deposits. Since the attachment state has been described above with reference to FIGS. 1B to 1D, description thereof will be omitted here. The estimation unit 34 also outputs the estimated result to the determination unit 36 .

The matching unit 35 performs matching processing for matching the edge feature amount of the cell 100 extracted by the extracting unit 32 and a predetermined template included in the template information 22 .

Here, the processing contents of the matching processing by the matching unit 35 will be described with reference to FIGS. 7A to 7E. 7A to 7E are diagrams (part 1) to (part 5) showing the processing contents of the matching unit 35. FIG. 7A and 7C to 7E are assumed to be pattern portions having a predetermined edge feature amount in the captured image I. FIG.

The matching unit 35 searches for a predetermined search pattern that matches a predetermined template, using the edge direction of the edge feature amount of the cell 100 extracted by the extraction unit 32 . As shown in FIG. 7A, the search directions are the horizontal direction and the vertical direction.

For example, the matching unit 35 searches for a search pattern under the condition that "an opposite angle classification does not appear on both sides of the angle classification of interest". Specifically, when the target angle classification is angle classification (1) and the search is performed in the horizontal direction, as shown in FIG. Cell 100-2 of angle classification (1) adjacent to 100-1 is the starting position.

Then, when the array of angle classification (1) continues and a cell 100-4 of angle classification (0) "angle classification is not reversed" appears, a cell of angle classification (1) adjacent to the cell 100-4 appears. 100-3 is the end position. In such a case, the match length is "8" in the example of FIG. 7B. Incidentally, when there is a match with the search pattern in this way, the matching unit 35 holds the position and the match length.

Also, if there is a match in the search pattern shown in FIG. 7B, a change in luminance between adjacent classifications among the four angle classifications will be seen at the start position and the end position.

Then, as shown in FIG. 7C, when matching search patterns intersect in the left-right direction and the up-down direction, the matching unit 35 stores horizontal matching data for each angle classification as stored information corresponding to the unit area UA corresponding to the crossing point. Accumulate the product of the length and the upper and lower match lengths.

Specifically, in accordance with the examples of FIGS. 7B and 7C, the matching unit 35 cumulatively adds 5×8 in association with the angle classification (1) of the unit area, as shown in FIG. 7D. Also, although not shown, the matching unit 35 also cumulatively adds the number of intersections in association with the angle classification (1) of the unit area.

By repeating such matching processing, the matching unit 35 determines that three or more of the cumulative addition results associated with the angle classifications (0) to (3) of the unit area UA are equal to or greater than a predetermined threshold, as shown in FIG. 7E. If there is, the adhesion state of the unit area UA is determined to be "adhesion". Also, if the determination condition is not satisfied, it is determined as "non-adhesion". Note that the processing contents of the matching processing shown in FIGS. 7A to 7E are merely examples, and the processing contents are not limited.

Returning to the description of FIG. The matching section 35 outputs the matching result to the determination section 36 . The determination unit 36 comprehensively determines the adhesion state for each unit area UA based on the estimation result by the estimation unit 34 and the matching result by the matching unit 35 , and outputs the determination result to various devices 50 .

Here, the processing contents of the determination processing by the determination unit 36 will be described with reference to FIGS. 8A and 8B. 8A and 8B are diagrams (part 1) and (part 2) showing the processing contents of the determination unit 36. FIG.

As shown in FIG. 8A , the determining unit 36 adds the matching result of the matching unit 35 to the estimation result of the estimating unit 34, and further adds the matching result of the matching unit 35 to the captured image I being processed. Determination processing is performed in which the determination result of the image I' is further taken into consideration.

Specifically, as shown in FIG. 8B , when the estimation result by the estimation unit 34 is “non-adhesion”, the determination unit 36 determines “non-adhesion” regardless of the matching result by the matching unit 35 . As a result, it is possible to suppress erroneous detection when it is determined as "attachment" in the matching result.

Further, when the estimation result by the estimation unit 34 is "attachment" and the matching result by the matching unit 35 is "attachment", the determination unit 36 determines "attachment".

Further, when the estimation result by the estimation unit 34 is "attachment" but the matching result by the matching unit 35 is "non-attachment", the determination unit 36 determines "attachment" in the past n frames. If there is no determination of "non-adherence" after that, it is determined to be "adherence".

Further, when the estimation result by the estimation unit 34 is “difficult to determine”, the determination unit 36 determines “attachment” in the past n frames regardless of the matching result by the matching unit 35. If there is no determination of "non-adhesion" in either case, it is determined as "adhesion".

In this way, by taking into consideration the determination result of the past captured image I', even if the determination is difficult with the captured image I being processed, the area once determined to be "attached" is denied. If there is no determination of "non-adhesion", for example, it is possible to determine "adhesion" in the same manner as other surrounding unit areas UA. Thereby, non-detection can be suppressed and the detection accuracy can be improved.

Next, a processing procedure executed by the adhering matter detection device 1 according to the embodiment will be described with reference to FIG. 9 . FIG. 9 is a flow chart showing a processing procedure executed by the adhering matter detection device 1 according to the embodiment. Note that FIG. 9 shows the processing procedure for one frame of camera image.

As shown in FIG. 9, first, the acquisition unit 31 acquires a captured image I (step S101). In addition, the acquisition unit 31 performs grayscaling processing and smoothing processing on the captured image I. FIG.

Subsequently, the extraction unit 32 extracts the edge feature amount for each cell 100 of the captured image I (step S102). Then, the calculation unit 33 calculates the area feature amount for each unit area UA based on the edge feature amount extracted by the extraction unit 32 (step S103).

Then, the estimation unit 34 maps the area feature amount calculated by the calculation unit 33 to the area feature amount space, and determines the position of the area feature amount (step S104). Here, if the position indicates "non-adhesion" (step S104, "non-adhesion"), the determination unit 36 determines "non-adhesion" based on this (step S105), and the process is terminated.

Further, when the position of the region feature value indicates "attachment" (step S104, "attachment"), the determination unit 36 determines whether or not the matching result of the matching unit 35 indicates "attachment" (step S106). .

Here, if the matching result is "adhesion" (step S106, "adhesion"), the determination unit 36 determines "adhesion" (step S107), and ends the process. On the other hand, if the matching result is "non-adhesion" (step S106, "non-adhesion"), control is transferred to step S108. Further, when the position of the area feature value indicates "difficult to determine" (step S104, "difficult to determine"), the control is transferred to step S108.

In step S108, the determination unit 36 determines whether or not the condition including the past frame is met (see FIG. 8B). Here, if applicable (step S108, Yes), the determination unit 36 determines "attachment" (step S109), and terminates the process. On the other hand, if not applicable (step S108, No), the determination unit 36 determines "non-adhesion" (step S110), and terminates the process.

As described above, the attached matter detection device 1 according to the embodiment includes the extraction unit 32, the calculation unit 33, and the estimation unit . The extraction unit 32 extracts an edge feature amount based on the edge vector V of each pixel PX included in each cell 100 including a predetermined number of pixels PX included in the captured image I. The calculation unit 33 calculates an area feature amount for each unit area UA, which is a predetermined area including a predetermined number of cells 100, based on the edge feature amount extracted by the extraction unit 32. FIG. The estimation unit 34 maps the area feature amount calculated by the calculation unit 33 to a feature amount space having each element of the area feature amount as each dimension, and based on the position of the mapped area feature amount in the unit area UA Estimate the state of adherence of adherents.

Therefore, according to the adhering matter detection device 1 according to the embodiment, the adhering matter detection accuracy can be improved.

The extracting unit 32 also extracts, for each cell 100, the edge direction obtained by classifying the angle feature amount included in the edge feature amount for each predetermined angle range. The calculation unit 33 calculates the number of pair regions 200 (corresponding to an example of a “combination”) of cells 100 that are adjacent to each other and have opposite edge directions, and the edge strength of each cell 100 included in the pair regions 200. is calculated as the area feature amount. The estimating unit 34 estimates the adherence state of the adhering matter based on the position of the region feature amount in the feature amount space whose dimensions are the number and the sum.

Therefore, according to the adhering matter detection device 1 according to the embodiment, it is possible to improve the detection accuracy of, for example, snow adhering to the lens of the camera 10 .

The calculation unit 33 also calculates the average and variance of the edge strength of each cell 100 included in the unit area UA as area feature amounts. The estimating unit 34 also estimates the attachment state of the adhering matter based on the position of the region feature amount in the feature amount space whose dimensions are the average and the variance.

Therefore, according to the adhering matter detection device 1 according to the embodiment, it is possible to improve the detection accuracy of, for example, snow adhering to the lens of the camera 10 based on the statistical viewpoint.

Also, the estimation unit 34 classifies the attachment state of the attached matter into attached, non-attached, or difficult to determine, based on the position of the region feature amount.

Therefore, according to the adhering matter detection device 1 according to the embodiment, it is possible to appropriately classify the adhering state when snow or the like adheres to the lens of the camera 10 .

Further, the adhering matter detection device 1 according to the embodiment further includes a determination unit 36 . The determination unit 36 determines the attachment state of the adhering matter based on the estimation result by the estimation unit 34 and the matching result by the matching unit 35 (corresponding to an example of "determination result by an algorithm different from that of the estimation unit"). Further, when the estimation result indicates non-adhesion, the determination unit 36 determines non-adhesion regardless of the matching result.

Therefore, according to the adhering matter detection device 1 according to the embodiment, it is possible to suppress erroneous detection of adhering matter by comprehensive determination combining the estimation result of the estimating unit 34 and the matching result of the matching unit 35. .

In addition, when the estimation result indicates adhesion and the matching result indicates non-adhesion, the judgment unit 36 determines that adhesion has occurred during a predetermined period in the past, and after such judgment, non-adhesion has been judged. If not, it is judged as adhered.

Therefore, according to the adhering matter detection device 1 according to the embodiment, it is possible to suppress the case where an object is not detected only by the matching result based on the captured image I being processed.

Further, when the determination result is difficult to determine, the determination unit 36 determines that there is adhesion when there is determination of adhesion during a predetermined period in the past and there is no determination of non-adhesion after this determination.

Therefore, according to the adhering matter detection device 1 according to the embodiment, it is difficult to make a determination based only on the estimation result, and it is possible to suppress cases in which an object is not detected.

(Extraction unit 32 according to modification)
In the above-described embodiment, the case of extracting one type of edge orientation representative value for one cell 100 has been described, but two or more types of edge orientation representative values may be assigned to one cell 100. good. This point will be described with reference to FIGS. 10 and 11. FIG.

10 and 11 are diagrams (part 1) and (part 2) showing the processing contents of the extraction unit 32 according to the modification. In addition, FIG. 10 and FIG. 11 explain the processing contents in the case of extracting the representative values of two types of edge orientations with different angular ranges.

As shown in FIG. 10, the extraction unit 32 extracts a first representative value of edge orientations classified by the first angle range, and a second representative value of edge orientations classified by a second angle range different from the first angle range. A representative value is determined for each cell 100 .

Specifically, the extracting unit 32 divides the edge direction of the edge vector V of each pixel PX in the cell 100 from −180° to 180° into first angle ranges of 90° each, and divides the angle into four directions, namely, up, down, left, and right. In addition to being classified into (0) to (3), it is classified into angle classifications (4) to (7), which are four oblique directions divided by a second angle range different from the first angle range.

More specifically, the extraction unit 32 classifies the edge orientation of the pixel PX into the angle category (0) when the angle range is −45° or more and less than 45°, If it is in the range, it is classified into angle classification (1), and if it is in the angle range of 135° or more and less than 180°, or -180° or more and less than -135°, it is classified into angle classification (2), and -135 If the angle range is greater than or equal to degrees and less than -45 degrees, it is classified as angle classification (3).

Furthermore, the extraction unit 32 classifies the edge direction of the pixel PX into the angle classification (4) when it is in the angle range of 0° or more and less than 90°, and when it is in the angle range of 90° or more and less than 180° is classified into angle classification (5), if it is in the angle range of 180° or more and less than -90°, it is classified into angle classification (6), and if it is in the angle range of -90° or more and less than 0°, the angle It is classified into classification (7).

Then, as shown in the lower part of FIG. 10, for each cell 100, the extracting unit 32 creates a histogram with angle classifications (0) to (3) in each class, and angle classifications (4) to (7) in each class. and generate a histogram with Then, when the frequency of the class with the highest frequency in the generated histogram is equal to or greater than a predetermined threshold value THa, the extraction unit 32 sets, for example, the angle classification (0) corresponding to this class as the first representative value of the edge direction. , for example, the angle classification (4) is extracted as the second representative value of the edge orientation.

Then, as shown in FIG. 11 , when at least one of the edge orientation first representative value and the edge orientation second representative value is opposite to each other in adjacent cells 100, the extraction unit 32 are extracted as the pair region 200 .

In other words, the extraction unit 32 extracts the first representative value and the second representative value of the edge orientation in each cell 100, thereby extracting the pair region 200 that could not be extracted with only one type of edge orientation. becomes.

That is, for example, for a pixel PX with an edge orientation of 140° and a pixel PX with an edge orientation of −40°, the orientation is not opposite in the first angle range, but the orientation is opposite in the second angle range. It becomes possible to detect a change in edge direction in the cell 100 with higher accuracy.

In addition, in the above-described embodiment and modification, the case of classifying the angles into four directions obtained by dividing −180° to 180° into angular ranges of 90° is shown, but the angular range is not limited to 90°. The angles may be classified into six directions divided by angle ranges of 60°.

Also, the width of each angle range may be different between the first angle range and the second angle range. For example, in the first angle range, the angles may be classified in units of 90°, and in the second angle range, the angles may be classified in units of 60°. Also, in the first angle range and the second angle range, the angle boundaries of the angle ranges are shifted by 45°, but the shifted angle may be more than 45° or less than 45°.

Further, in the above-described embodiment, the captured image I captured by an in-vehicle camera is taken as an example. There may be. In other words, any captured image I captured by a camera in which there is a possibility that an adhering substance may adhere to the lens of the camera may be used.

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments so shown and described. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

1 adhering matter detection device 2 storage unit 3 control unit 10 camera 21 threshold information 22 template information 23 determination history information 31 acquisition unit 32 extraction unit 33 calculation unit 34 estimation unit 35 matching unit 36 determination unit 50 various devices 100 cell 200 pair region I Captured image PX Pixel UA Unit area

Claims (7)

A deposit detection device comprising a control unit,
The control unit
extracting , for each cell composed of a predetermined number of pixels included in a captured image, an edge orientation obtained by classifying an angle feature amount based on an edge vector of each pixel included in the cell for each predetermined angle range, and
The number of combinations of the cells adjacent to each other and having opposite edge directions, and each cell included in the combination, as a region feature amount for each unit region, which is a predetermined region composed of a predetermined number of the cells. Calculate the sum of the edge strengths of
The calculated area feature amount is mapped onto a feature amount space having dimensions of the number and the sum total , and the attached matter in the unit area is determined based on the position of the mapped area feature amount in the feature amount space. Estimate the adhesion state of
An adhering matter detection device characterized by: The control unit
calculating the average and variance of the edge strength of each cell included in the unit area as the area feature amount;
2. The adhering matter detection apparatus according to claim 1 , wherein the adhering state of the adhering matter is estimated based on the position of the area feature amount in the feature amount space having the mean and the variance as respective dimensions. . The control unit
3. The adhering matter detection device according to claim 1 , wherein the adhering state of the adhering matter is classified into adhering, non-adhering, or difficult to determine based on the position of the region feature amount. The control unit further
Determining the adhesion state of the adhesion matter based on the estimation result of the adhesion state of the adhesion matter and the determination result by an algorithm different from the estimation of the adhesion state of the adhesion matter ,
4. The adhering matter detection device according to claim 1, wherein when the estimation result indicates non-adherence, the non-adherence is determined regardless of the determination result. The control unit
If the estimation result indicates adhesion and the determination result indicates non-adhesion, and there was an adhesion determination during a predetermined period in the past, and there is no non-adhesion determination after the determination, it is determined that there is adhesion. The adhering matter detection device according to claim 4 , characterized in that: The control unit
5. If it is difficult to determine in the estimation result, there is a determination of adhesion during a predetermined period in the past, and if there is no determination of non-adhesion after the determination, it is determined that there is adhesion. 6. The attachment detection device according to 5 . extracting , for each cell composed of a predetermined number of pixels included in a captured image, an edge direction obtained by classifying an angle feature amount based on an edge vector of each pixel included in the cell by a predetermined angle range for each cell; ,
The number of combinations of the cells adjacent to each other and having opposite edge directions, and each cell included in the combination, as a region feature amount for each unit region, which is a predetermined region composed of a predetermined number of the cells. calculating the sum of the edge strengths of
The calculated area feature amount is mapped onto a feature amount space having dimensions of the number and the sum total , and the attached matter in the unit area is determined based on the position of the mapped area feature amount in the feature amount space. A deposit detection method, comprising : estimating the adhesion state of

Images