JP7234885B2 - Attached matter detection device and attached matter detection method - Google Patents

Description

The present invention relates to an adhering matter detection device and an adhering matter detection method.

Conventionally, there is an adhering matter detection apparatus that detects a candidate area having characteristics similar to the shape of a raindrop or the like based on edges detected from each pixel of a captured image, and detects an adhering matter area based on the pixel values of the candidate area. known (see, for example, Patent Document 1).

JP 2019-106644 A

However, in the prior art, there is room for improvement in terms of detecting deposits with high accuracy. For example, since the shape of raindrops is similar to the shape of reflections on the road surface under indoor lighting, there is a risk that the reflections on the road surface will be erroneously detected as an adhering matter area.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an adhering matter detection device and an adhering matter detection method capable of detecting adhering matter with high accuracy.

In order to solve the above-described problems and achieve the object, an attached matter detection device according to the present invention includes an extraction unit, a calculation unit, and a determination unit . The extraction unit extracts a candidate area for an adhering matter area corresponding to an adhering matter adhering to the imaging device based on the edge detected from each pixel of the image captured by the imaging device . The calculation unit calculates luminance change amounts of a plurality of pixels consecutive in a predetermined direction in a predetermined internal region in the candidate region extracted by the extraction unit . The determination unit determines the number of luminance variations in the first range calculated by the calculation unit , and the number of luminance variations in the second range in which the luminance variation is greater than the first range. It is determined whether or not the candidate area is a deposit area based on the number of luminance change amounts. The determination unit determines that the candidate area is not a deposit area when the number of luminance variations in the first range is a predetermined number or more and the number of luminance variations in the second range is a predetermined number or more. judge.

According to the present invention, adhering matter can be detected with high accuracy.

FIG. 1 is a diagram showing an outline of an attached matter detection method according to an embodiment. FIG. 2 is a block diagram showing the configuration of the attached matter detection device according to the embodiment. FIG. 3A is a diagram illustrating processing contents of an extraction unit; FIG. 3B is a diagram illustrating processing contents of an extraction unit; FIG. 4A is a diagram illustrating the processing contents of an extraction unit; FIG. 4B is a diagram illustrating the processing contents of the extraction unit; FIG. 5 is a diagram illustrating exclusion processing by the extraction unit. FIG. 6 is a diagram showing determination processing of the extraction unit. FIG. 7 is a diagram illustrating calculation processing by a calculation unit; FIG. 8 is a diagram showing determination processing by the determination unit. FIG. 9 is a diagram illustrating continuity determination processing by the determination unit. FIG. 10 is a diagram showing area calculation processing by the flag output unit. FIG. 11 is a flowchart illustrating a procedure of overall processing of attached matter executed by the attached matter detection apparatus according to the embodiment. FIG. 12 is a flowchart illustrating a procedure of an attached matter determination process executed by the attached matter detection device according to the embodiment.

DETAILED DESCRIPTION OF THE INVENTION Embodiments of the attached matter detection device and the attached matter detection method disclosed in the present application will be described in detail below with reference to the accompanying drawings. In addition, this invention is not limited by embodiment shown below.

First, with reference to FIG. 1, an outline of an attached matter detection method according to an embodiment will be described. FIG. 1 is a diagram showing an outline of an attached matter detection method according to an embodiment. Note that FIG. 1 shows a captured image I captured with water droplets such as raindrops adhering to the lens of an in-vehicle camera (an example of an imaging device), for example.

When dirt, dust, raindrops, snowflakes, and other substances adhere to the lens of the camera, information about the vehicle surroundings, such as parking frames, other vehicles, and people, cannot be obtained from the captured image I. There is a risk that it will not be possible to accurately detect frames, other vehicles, people, and the like. Note that the attached matter is not limited to mud, dust, raindrops, and snowflakes, and any attached matter may be used as long as the area of the attached matter becomes a blurred area.

Conventionally, however, there is a risk of erroneous detection of adhering matter even though no adhering matter adheres to the lens. One such case is erroneous detection of road surface reflections illuminated by interior lighting. Specifically, since the characteristics of the brightness of the road surface reflection (particularly when the road surface is smooth with little unevenness) in the captured image are similar to the brightness characteristics of the area of blurred deposits such as raindrops, There is a possibility that the area is erroneously detected as an adhering matter area.

Therefore, the adhering matter detection device 1 (see FIG. 2) according to the embodiment performs the adhering matter detection method to distinguish between the road surface reflection area and the adhering matter area with high accuracy.

Specifically, first, in the adhering matter detection method according to the embodiment, based on the edge detected from each pixel of the captured image I, candidates for the adhering matter area corresponding to the adhering matter adhering to the lens of the camera are detected. A region 100 is extracted (step S1).

In the attached matter detection method according to the embodiment, for example, a rectangular area including a circular contour such as a raindrop is extracted as the candidate area 100 by matching processing such as pattern matching.

Subsequently, in the attached matter detection method according to the embodiment, luminance change amounts of a plurality of pixels that are continuous in a predetermined direction in a predetermined internal region 200 in the extracted candidate region 100 are calculated (step S2).

In FIG. 1, in the candidate area 100, an internal area 200 is set in the central area of the captured image I, and the luminance change amount of three pixels that are consecutive in the vertical direction is calculated. Also, the luminance change amount is a luminance difference value in a plurality of pixels. It should be noted that the luminance change amount is obtained by performing two difference calculations for three pixels, but this point will be described later.

Subsequently, in the adhered substance detection method according to the embodiment, a histogram of the calculated luminance change amount is created, and based on the histogram, it is determined whether or not the candidate region 100 is a adhered substance region (step S3).

The lower part of FIG. 1 shows a histogram in which the class on the horizontal axis is the luminance change amount and the frequency on the vertical axis is the area frequency. Note that the area frequency is a number corresponding to the number of pixels, and one frequency corresponds to one pixel. In other words, one frequency corresponds to a luminance change amount (ie, one luminance change amount) in one plurality of pixels. Also, the area frequency is a frequency normalized so that the sum of the frequencies is a predetermined value.

In the attached matter detection method according to the embodiment, in the histogram, the number of luminance change amounts whose luminance change amount is in the first range and the luminance change amount in the second range where the luminance change amount is larger than that of the first range are counted. Determination of the adhering matter region is performed based on the number of . Note that the number of luminance change amounts can also be rephrased as the number of units of a plurality of pixels, with one plurality of pixels being one unit.

For example, the higher the area frequency (the number of luminance change amounts) in the first range, the more the candidate area 100 is in a dull state with fewer luminance change locations in the image, and the second range area frequency (the number of luminance change amounts). number), the candidate area 100 will be in a rougher state where there are more places where the luminance changes in the image.

Although the details will be described later, the higher the area frequency of the first range, the more similar it is to the feature of a deposit area such as raindrops, and the higher the area frequency of the second range, the more similar it is to the feature of road surface reflection. This is because the area frequency of the second range increases due to scratches, stains, etc. on the road surface. less.

As described above, in the method for detecting an adhering matter according to the embodiment, the adhering matter area is determined based on the number of luminance variations in the first range and the number of luminance variations in the second range. A reflection area and an adhering matter area can be discriminated with high accuracy. That is, according to the attached matter detection method according to the embodiment, the attached matter can be detected with high accuracy.

Next, the configuration of the adhering matter detection device 1 according to the embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the adhering matter detection device 1 according to the embodiment. As shown in FIG. 2 , the attached matter detection device 1 according to the embodiment is connected to a camera 10 , a vehicle speed sensor 11 and various devices 50 . Although FIG. 2 shows a case where the adhering matter detection device 1 is configured separately from the camera 10 and the various devices 50, it is not limited to this, and may be integrated with at least one of the camera 10 and the various devices 50. may consist of

The camera 10 is, for example, an in-vehicle camera that includes a lens such as a fisheye lens and an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The cameras 10 are provided, for example, at positions capable of capturing images of the front, rear, and sides of the vehicle, and output captured images I to the adhering matter detection device 1 .

The vehicle speed sensor 11 is a sensor that detects the speed of the vehicle (vehicle speed). The vehicle speed sensor 11 outputs information on the detected vehicle speed to the adhering matter detection device 1 . Note that the vehicle speed sensor 11 may be omitted and, for example, the adhering matter detection device 1 may calculate vehicle speed information based on time-series images from the camera 10 .

The various devices 50 are devices that acquire detection results of the adhering matter detection device 1 and perform various vehicle controls. The various devices 50 include, for example, a display device that notifies that an adhering substance is attached to the lens of the camera 10 and a user's instruction to wipe off the adhering matter, and a display device that injects fluid, gas, or the like toward the lens to remove the adhering matter. It includes a removal device that removes dust, a vehicle control device that controls automatic driving, etc.

As shown in FIG. 2 , the adhering matter detection device 1 according to the embodiment includes a control unit 2 and a storage unit 3 . The control unit 2 includes an image acquisition unit 21 , an extraction unit 22 , a calculation unit 23 , a determination unit 24 and a flag output unit 25 . The storage unit 3 stores undulation condition information 31 .

Here, the adhering matter detection device 1 includes, for example, a computer and various circuits having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), data flash, input/output ports, and the like.

The CPU of the computer functions as the image acquisition section 21, the extraction section 22, the calculation section 23, the determination section 24, and the flag output section 25 of the control section 2 by reading and executing programs stored in the ROM, for example.

Further, at least one or all of the image acquisition unit 21, the extraction unit 22, the calculation unit 23, the determination unit 24, and the flag output unit 25 of the control unit 2 may be an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). ) or the like.

Also, the storage unit 3 corresponds to, for example, a RAM or a data flash. The RAM and data flash can store the undulation condition information 31, various program information, and the like. Note that the adhering matter detection apparatus 1 may acquire the above-described programs and various information via another computer or portable recording medium connected via a wired or wireless network.

The undulation condition information 31 stored in the storage unit 3 is information including conditions that serve as a reference for processing by the extraction unit 22 described later, and includes, for example, undulation pattern conditions of luminance distribution (distribution of average luminance). It should be noted that the pattern condition is a change pattern of undulation change in the average luminance distribution, or a luminance threshold range that each pixel (unit region R) in the luminance distribution can take. Processing using the undulation condition information 31 will be described later.

The control unit 2 extracts a candidate area 100 for the adhering matter area based on the edge detected from each pixel of the captured image I obtained from the camera 10, and extracts a plurality of candidate areas 100 included in the internal area 200 set as the extracted candidate area 100. Whether or not the candidate area 100 is a deposit area is determined based on the luminance change amount of the pixel.

The image acquisition unit 21 acquires an image captured by the camera 10 and generates (acquires) a current frame, which is the current captured image I. Specifically, the image acquisition unit 21 performs grayscaling processing for converting each pixel in the acquired image into each gradation from white to black according to the luminance.

In addition, the image acquisition unit 21 performs pixel thinning processing on the acquired image to generate an image smaller in size than the acquired image. The image acquisition unit 21 also generates the current frame, which is an integral image of the sum of the pixel values and the sum of the squares of the pixel values of each pixel, based on the thinned image. Note that the pixel value is information corresponding to the luminance and edge of the pixel.

In this way, the adhering matter detection apparatus thins out the acquired image and generates an integral image, thereby speeding up the processing calculations in the latter stage. can be shortened.

Note that the image acquisition unit 21 may perform smoothing processing on each pixel using a smoothing filter such as an averaging filter. Alternatively, the image acquiring unit 21 may generate a current frame of the same size as the acquired image without performing the thinning process. Also, the current frame may be referred to as a captured image I below.

The extracting unit 22 extracts a candidate area 100 for the adhering matter area from the captured image I based on the edge detected from each pixel of the captured image I acquired by the image acquiring unit 21 . Specifically, first, the extraction unit 22 extracts the luminance and edge information of each pixel in the captured image I. FIG. The brightness of each pixel is represented by a parameter from 0 to 255, for example.

In addition, the extraction unit 22 performs edge detection processing based on the luminance of each pixel, and detects an edge in the X-axis direction (horizontal direction of the captured image I) and an edge in the Y-axis direction (vertical direction of the captured image I) for each pixel. ) edges. Any edge detection filter such as a Sobel filter or a Prewitt filter can be used for edge detection processing.

Then, the extraction unit 22 detects, as edge information, a vector including information on the edge angle and edge strength of the pixel by using a trigonometric function based on the edge in the X-axis direction and the edge in the Y-axis direction. Specifically, the edge angle is represented by the orientation of the vector, and the edge strength is represented by the length of the vector.

Then, the extraction unit 22 performs a matching process (template matching) between the template information created in advance indicating the outline of the adhering matter and the detected edge information, and extracts edge information similar to the template information. Then, the extracting unit 22 extracts the area of the extracted edge information, that is, the candidate area 100 which is a rectangular area including the outline of the adhering matter.

Note that the extraction unit 22 may output information on the candidate region 100 extracted by the matching process to the calculation unit 23, and output information on the candidate region 100 further narrowed down by the extraction process described below to the calculation unit 23. may

Subsequently, the extracting unit 22 divides a plurality of continuous pixel rows in the extracted candidate region 100 into unit regions R each having a predetermined number of pixels as a unit, and calculates the average luminance value and the standard deviation of luminance for each unit region R. Calculate

Here, processing contents of the extraction unit 22 will be described with reference to FIGS. 3A to 4B. 3A to 4B are diagrams showing the processing contents of the extraction unit 22. FIG.

As shown in FIGS. 3A and 3B , first, the extraction unit 22 sets a plurality of continuous pixel columns in the candidate region 100 as the band region 110 . The band area 110 is an area to be subjected to the subsequent process of dividing the unit area R and the process of calculating luminance information (average value and standard deviation).

Specifically, as shown in FIG. 3A, first, the extraction unit 22 selects three horizontal pixel columns H1 to H3 and three vertical pixel columns V1 to V3 in the candidate region 100. do. Then, the extracting unit 22 sets a plurality of pixel rows including the selected pixel row as the strip region 110 .

FIG. 3B shows a strip region 110 containing a horizontal row of pixels H1. Note that the setting method of the band region 110 including the vertical pixel columns V1 to V3 is substantially the same as the setting method of the band region 110 including the horizontal pixel column H1 described below (if the pixel column is horizontal or vertical). direction), so the description is omitted.

As shown in FIG. 3B , the extracting unit 22 sets a plurality of vertically continuous pixel rows as a band region 110 centering on the selected pixel row H1. In other words, the extraction unit 22 sets the selected pixel line H1 and a plurality of pixel lines parallel to the selected pixel line H1 and adjacent to each other as the band region 110 .

In FIG. 3B, a total of 7 pixel columns including 3 pixel columns (total of 6 pixel columns) continuous to the vertical upper side and lower side of the horizontal pixel column H1 and 1 pixel column H1. A row of pixels is set as a band region 110 .

Note that the position of the pixel row H1 in the strip region 110 may not be the center of the strip region 110, and may be located at either the upper side or the lower side in the vertical direction. In addition, the number of pixel columns that are continuous above and below the pixel column H1 in the vertical direction may be two or less, or four or more.

The pixel columns H1 to H3 and V1 to V3 to be selected may be pixel columns in either one of the horizontal pixel columns H1 to H3 and the vertical pixel columns V1 to V3. Further, the number of pixel columns H1 to H3 and V1 to V3 selected in each direction is not limited to three, and may be two or less or four or more.

Subsequently, as shown in FIG. 4A, the extraction unit 22 divides the set band region 110 into unit regions R1 to R8 each having a predetermined number of pixels as a unit. Calculate the standard deviation of

Note that FIG. 4A shows a case where the extraction unit 22 sets eight unit regions R1 to R8 obtained by dividing the band region 110 at equal intervals (same number of pixels), but the number of unit regions R is six. It may be less than or equal to nine or more. Also, the number of pixels included in each unit region R may be the same or different.

Then, when the calculated standard deviation of luminance is less than a predetermined value, the extraction unit 22 sets the average luminance value as the representative value of the unit regions R in the band region 110 .

On the other hand, as shown in FIG. 4B , when the calculated standard deviation of luminance is equal to or greater than a predetermined value, the extraction unit 22 sets the distribution of the luminance average in the band region 110 to a predetermined An abnormal value is set to a representative value of the unit region R.

In the example shown in FIG. 4B , when the standard deviation of the brightness of the unit region R8 is equal to or greater than a predetermined value, the calculation unit 23 replaces the average value of the brightness of the unit region R8 with a predetermined abnormal value (for example, the minimum value , zero, etc.) is set as a representative value of the unit area R8.

This abnormal value is a value for determining that the area is not an adhering matter area in the post-processing. For example, when the extracting unit 22 determines an adhering matter area based on whether or not the average luminance distribution matches a predetermined pattern in the post-processing, it sets an abnormal value that does not match the predetermined pattern.

In this way, by setting an abnormal value that is determined to be not an adhering matter region in the post-processing, the candidate region 100 that is not an adhering matter region can be removed without adding new processing. Complications can be suppressed.

Although the extraction unit 22 uses the average luminance value as the representative value of the unit regions R, for example, a luminance histogram is created for each of the unit regions R1 to R8, and the mode, median, and average A value or the like may be set as a representative value.

Subsequently, the extraction unit 22 further narrows down the candidate regions 100 based on the distribution of the luminance average calculated for each unit region R. FIG.

Specifically, first, if the unevenness of the luminance distribution (distribution of average luminance) of the pixels included in the candidate region 100 satisfies a predetermined exclusion condition, the extraction unit 22 regards the candidate region 100 as the adhering matter region determination process. Exclude from coverage.

Here, the exclusion processing of the candidate region 100 by the extraction unit 22 will be described with reference to FIG. FIG. 5 is a diagram showing exclusion processing by the extraction unit 22. As shown in FIG. FIG. 5 shows the average luminance distribution of each band area 110 calculated from the predetermined candidate area 100 . FIG. 5 shows the average luminance distribution for each band region 110 corresponding to each pixel row H1 to H3 and V1 to V3 set in the candidate region 100. FIG. For example, the graph of "H1" in FIG. 5 shows the distribution of the luminance average of the unit area R in the band area 110 including the pixel row H1.

If the undulations in the distribution of the average brightness of each band region 110 corresponding to each of the pixel rows H1 to H3 and V1 to V3 satisfy a predetermined exclusion condition, the extraction unit 22 selects the candidate region 100 as the target of the adhering matter region determination processing. Exclude from

For example, if the representative value of the unit regions R included in the band region 110 includes an abnormal value, the extraction unit 22 excludes the candidate region 100 from the target of the adhering matter region determination processing. In other words, when the difference between the representative values of two adjacent unit regions R in the band region 110 is greater than or equal to a predetermined value, the extraction unit 22 excludes the candidate region 100 from the adhering matter region determination processing.

In other words, if the unit regions R in the band region 110 include a unit region R in which the standard deviation of the representative value is equal to or greater than a predetermined value (that is, the unit region R in which an abnormal value is set) , the candidate region 100 is excluded from the adhering matter region determination processing.

That is, if part of the undulations in the distribution of the average luminance has a shape that is abruptly uneven due to an abnormal value, the extraction unit 22 determines that the exclusion condition is satisfied, and removes the candidate region 100 from the target of the adhering matter region determination processing. exclude. In this way, the extraction unit 22 can perform the exclusion process based on the abnormal value and remove the non-attachment area with high accuracy.

Further, as shown in the graphs of the pixel columns V1 to V3 in FIG. 5, the extracting unit 22 detects that the undulations in the average luminance distribution of the unit region R are flat in all three graphs, that is, in the pixel columns V1 to V3 in the vertical direction. If the undulation change of the luminance average for all three corresponding band regions 110 is within a predetermined range, the candidate region 100 is excluded from the object region determination processing. Specifically, it has been found by experiments that when the candidate region 100 reflects the light from the back lamp on the wall, the average luminance distribution of the unit region R tends to be flat. Therefore, if the undulation change of the luminance average is within a predetermined range, it is determined that the candidate area 100 is caused by the reflection of the back lamp, and the object area 400 is excluded from the object of determination processing.

Note that FIG. 5 shows the case where the three luminance distributions in the vertical pixel rows V1 to V3 are flat. If the six luminance distributions in the horizontal and horizontal pixel rows H1 to H3 and V1 to V3 are flat, the candidate area 100 may be excluded from the adhering matter area determination process. In other words, the extraction unit 22 determines that the exclusion condition is satisfied when at least one of the vertical and horizontal pixel rows in the candidate region 100 has an undulating change in luminance distribution within a predetermined range, and the candidate region 100 satisfies the exclusion condition. 100 is excluded from the target of the adhering matter region determination processing.

In this manner, when the undulations of the luminance distribution in the candidate region 100 satisfy a predetermined exclusion condition, the extraction unit 22 excludes the candidate region 100 from the target of the adhering matter region determination processing, thereby removing the adhering matter region. Misjudgments can be reduced. That is, adhering matter can be detected with high accuracy.

Subsequently, the extraction unit 22 performs processing for further narrowing down the candidate regions 100 other than the candidate regions 100 excluded by the exclusion condition. Here, processing by the extraction unit 22 will be described with reference to FIG.

FIG. 6 is a diagram showing determination processing of the extraction unit 22. As shown in FIG. First, as shown in FIG. 6, the extraction unit 22 first calculates, for each band region 110, the amounts of change D1 to D7 in the luminance average of adjacent unit regions R1 to R8.

Then, the extraction unit 22 extracts the candidate region 100 in which the pattern of undulation change in the distribution of the luminance average of the unit region R satisfies a predetermined change pattern from among the plurality of band regions 110 in the candidate region 100 . For example, the extraction unit 22 compares each value of the variation amounts D1 to D7 with each value of the variation pattern included in the undulation condition information 31 stored in the storage unit 3 to perform determination processing. The variation pattern included in the undulation condition information 31 is the threshold range of the maximum and minimum values that the variations D1 to D7 can take.

In other words, the extracting unit 22 extracts the candidate regions 100 in which the calculated values of the amounts of change D1 to D7 are within the respective threshold ranges of the amounts of change D1 to D7 included in the undulation condition information 31. FIG.

In other words, the extracting unit 22 extracts the candidate region 100 in which the patterns of the amounts of change D1 to D7 in the average brightness of the adjacent unit regions R1 to R8 satisfy the change pattern that is the threshold range set by the undulation condition information 31. do.

In addition, by setting the maximum and minimum values of the variation amounts D1 to D7 in the undulation condition information 31 to provide a range, it is possible to extract the candidate area 100 even if the undulations of the luminance distribution are somewhat uneven.

FIG. 6 shows a case where threshold ranges for all the variation amounts D1 to D7 are set for the undulation condition information 31. Of D7, only the threshold range of a part of the amount of change may be set.

The extraction unit 22 outputs the extracted information of the candidate region 100 to the calculation unit 23 .

The calculation unit 23 calculates luminance change amounts of a plurality of pixels that are continuous in a predetermined direction in a predetermined internal region 200 in the candidate region 100 extracted by the extraction unit 22 .

Here, the calculation processing of the luminance change amount in the internal region 200 by the calculator 23 will be described with reference to FIG. 7 . FIG. 7 is a diagram showing calculation processing by the calculation unit 23. As shown in FIG.

As shown in FIG. 7 , the calculator 23 first sets an internal region 200 for the extracted candidate region 100 . For example, the calculation unit 23 divides the candidate area 100 into a predetermined number in each of the vertical and horizontal directions. FIG. 7 shows a case where the candidate area 100 is divided into four in each of the vertical direction and the horizontal direction, but may be divided into three or less, or five or more.

Then, the calculation unit 23 sets the central region of the predetermined number of divided regions as the internal region 200 . In FIG. 7 , of the 16 divided areas generated by dividing into four in each of the vertical and horizontal directions, the four central areas are set as the internal areas 200 .

Then, the calculation unit 23 calculates the amount of change in luminance of multiple pixels in the internal region 200 . In FIG. 7, the calculator 23 calculates the amount of change in brightness of three pixels that are consecutive in the vertical direction. This is because the area of road surface reflection often extends in the vertical direction. As a result, the determination unit 24 in the subsequent stage can determine the area of the road surface reflection with high accuracy. Note that the calculation unit 23 may calculate the amount of change in brightness of a plurality of pixels that are consecutive in the horizontal direction.

First, the calculator 23 calculates a luminance difference value between two vertically adjacent pixels as a first difference. In the middle part of FIG. 7, one first difference is defined as one cell, which is arranged two-dimensionally in correspondence with the arrangement of pixels. That is, in the middle part of FIG. 7, one square represents the luminance difference value of two vertically adjacent pixels, that is, the first difference, and two vertically adjacent squares represent two second values calculated from three pixels. 1 difference.

Subsequently, the calculator 23 calculates a second difference, which is a difference value between two first differences that are consecutive in the vertical direction. In the lower part of FIG. 7, one second difference converted into an absolute value is set as one cell and arranged two-dimensionally in correspondence with the arrangement of pixels. That is, in the lower part of FIG. 7, one square is one second difference (luminance change amount) calculated from three pixels. In other words, the second difference is the amount of change in the first difference.

When the luminance fluctuates sharply in this manner, it is likely that the luminance is caused by a small scratch on the road surface, and is highly likely to be a reflection area of the road surface. Therefore, the calculation unit 23 calculates the second difference as the luminance change amount of a plurality of pixels, so that the subsequent determination unit 24 can easily determine whether the area is an adherent area or a road surface reflection area. can. Specifically, for example, in a region where the luminance of three consecutive pixels rapidly rises and falls due to a small scratch on the road surface ("180"→"160"→"180" shown in FIG. 7), the second difference is higher. That is, by calculating the second difference, it becomes possible to easily identify such a region in which the brightness rises and falls rapidly. can be easily determined.

The calculation unit 23 outputs the second difference, which is the calculation result, to the determination unit 24 as the luminance change amount of the plurality of pixels.

In addition, although the calculation unit 23 uses the amount of luminance change of a plurality of pixels as the second difference, the amount of luminance change of a plurality of pixels may be used as the first difference. When the first difference is the luminance change amount of a plurality of pixels, the number of pixels in the plurality of pixels should be 2 pixels or more. Specifically, when calculating the luminance change amount of two pixels, one calculated first difference may be used as the luminance change amount. may be used as the luminance change amount.

The determining unit 24 determines whether or not the candidate area 100 is a deposit area by forming a histogram of the luminance change amounts of the plurality of pixels calculated by the calculating unit 23 . Here, determination processing by the determination unit 24 will be described with reference to FIG. 8 .

FIG. 8 is a diagram showing determination processing by the determination unit 24. As shown in FIG. FIG. 8 shows a histogram of luminance change amounts for a plurality of pixels included in one internal region 200 . In the histogram shown in FIG. 8, the horizontal axis represents the luminance change amount, and the vertical axis represents the area frequency. In addition, FIG. 8 shows a histogram when the candidate area 100 is the deposit area and a histogram when the candidate area 100 is the road surface reflection area (area other than the deposit).

Note that the area frequency is a number corresponding to the number of pixels, and one frequency corresponds to one pixel. In other words, one frequency corresponds to a luminance change amount (ie, one luminance change amount) in one plurality of pixels. Also, the area frequency is a frequency normalized so that the sum of the frequencies is a predetermined value. In other words, the sum of the area frequencies of the histogram shown in FIG. 8 is the same between the internal areas 200 having different sizes.

As shown in FIG. 8, when the histogram of the adhering matter area and the histogram of the road surface reflection area are compared, the area frequencies in the first range are similar, but in the second range, the road surface reflection , the total area frequency is slightly higher than that of the deposit area. This is because scratches, stains, etc. on the road surface affect the area frequency of the second range in the area of the road surface reflection.

Focusing on this point, the determination unit 24 determines the number of luminance variations (region frequency) in which the luminance variation is in the first range, and the number of luminance variations in the second range in which the luminance variation is larger than that in the first range. It is determined whether or not the candidate area 100 is a deposit area based on the number of luminance variations.

Specifically, the determining unit 24 determines the sum of the area frequencies in the first range whose minimum value is the amount of change in brightness of "0", and the second range whose minimum value is greater than the maximum value of the first range. It is determined whether or not the candidate region 100 is a deposit region based on the sum of the region frequencies in .

For example, when the sum of the area frequencies in the first range is equal to or greater than a predetermined value and the sum of the area frequencies in the second range is less than the predetermined value, the determination unit 24 determines that the candidate area 100 is the adherent area. Determine that there is.

On the other hand, if the sum of the area frequencies in the first range is a predetermined value or more and the sum of the area frequencies in the second range is a predetermined value or more, the determination unit 24 determines that the candidate area 100 is a deposit area. It is determined that there is no road surface reflection area.

As a result, the determining unit 24 can exclude the area corresponding to the road surface reflection area from the candidate area 100, thereby suppressing erroneous detection of the road surface reflection area as an adhering matter area.

Further, if the sum of the area frequencies in the first range is less than a predetermined value, the determination unit 24 determines that the candidate area 100 is not a deposit area regardless of the sum of the area frequencies in the second range. I judge.

Then, the determination unit 24 performs continuity determination processing to determine whether or not the candidate area 100 has been continuously determined as an adhering matter area. Here, the continuity determination processing by the determination unit 24 will be described.

FIG. 9 is a diagram showing continuity determination processing by the determination unit 24 . FIG. 9 shows a diagram of a state machine for managing the state of each candidate area 100, and shows state transitions regarding adhering matter in the candidate area 100. In FIG.

As shown in FIG. 9, candidate region 100 can transition to four states: "IDLE," "latent," "attached," and "move attached."

"IDLE" indicates a state in which the area is not determined as an adhering matter area. In other words, "IDLE" indicates that the candidate area 100 is not continued as an adhering matter area.

"Latent" is a state indicating that there is a possibility that deposits are attached. In other words, "latency" indicates that the candidate area 100 has been determined to be an adhering matter area but has been continuously determined as an adhering matter area for a period of less than a predetermined period of time.

“Attachment” is a state indicating that an attachment is attached. That is, "adhesion" indicates that the candidate area 100 has been continuously determined as an adhering matter area for a predetermined period or longer.

"Moving attachment" is a state indicating that the attached matter continues to adhere even when the vehicle is moving (vehicle speed is equal to or higher than a predetermined value). That is, "moving attachment" indicates that the candidate area 100 is continuously determined as an attachment area for a predetermined period or longer even while the vehicle is moving. In the following description, the candidate area 100 (attachment area) in the "moving attachment" state may be referred to as a moving attachment area.

The transition between states will be described below. Note that the state transition can be performed based on the score indicating the continuity of the determination process.

<"IDLE" → "Conceal">
For example, the determination unit 24 transitions the candidate area 100, which is determined to be an adhering matter area for the first time, from "IDLE" to "latent".

<"Concealment" → "IDLE">
For example, if the determination unit 24 determines that the candidate region 100, which has been continuously determined as an adhering matter region for a period of time less than a predetermined period, is not an adhering matter region in the current captured image I, the determination unit 24 changes "latency" to " IDLE”.

<"Concealment" → "Attachment">
For example, the determining unit 24 makes a transition from "latency" to "adherence" when the period during which the adhering matter area is continuously determined is equal to or longer than a predetermined period. The determination unit 24 outputs, to the flag output unit 25, the information of the attached matter area that is associated with the state of the “attachment” when the transition from “latent” to “attachment” occurs.

<“Attachment” → “IDLE”>
For example, if the determination unit 24 determines that the candidate area 100, which has been continuously determined as an adhering matter area for a predetermined period or longer, is not an adhering matter area in the current captured image I, the determining unit 24 changes "adhering" to " IDLE”.

<“Attachment” → “Moving attachment”>
For example, after transitioning to the "adhesion" state, the determination unit 24 changes the state from "adhesion" to "moving adhesion" when the period during which the adhering matter area is continuously determined even while the vehicle is moving is equal to or longer than a predetermined period. transition. When the state of "attachment" is changed to "moving attachment", the determination unit 24 outputs to the flag output unit 25 the information of the attachment area (moving attachment area) associated with the state of "moving attachment". .

<“Moving attachment” → “IDLE”>
For example, if the determining unit 24 determines that the current captured image I acquired while the vehicle is moving is not an attached object area in the state of "moving attachment", the determination unit 24 causes a transition from "moving attachment" to "IDLE". .

The flag output unit 25 outputs an adhering matter flag indicating whether or not an adhering matter is adhered based on the determination result of the determining unit 24 to the various devices 50 . First, the flag output unit 25 performs a predetermined area calculation process based on the area determined by the determination unit 24 to be an adhering matter area.

Here, with reference to FIG. 10, the area calculation processing of the adhering matter area by the flag output unit 25 will be described.

FIG. 10 is a diagram showing area calculation processing by the flag output unit 25. As shown in FIG. As shown in FIG. 10, first, the flag output unit 25 divides the entire area of the captured image I into predetermined small areas 500, and divides each small area 500 into three areas 510, 520a, and 520b.

Specifically, the first area 510 is an area corresponding to the road surface. The second region 520a is a region (body region) corresponding to the vehicle body. The third area 520b is an area corresponding to the sky.

For example, the flag output unit 25 detects a horizontal line in the captured image I, and sets an area above the horizontal line as the third area 520b. Also, the second area 520a is an area preset according to the mounting position of the camera 10 . In addition, the flag output unit 25 sets the area of the entire area of the captured image I excluding the second area 520 a and the third area 520 b as the first area 510 .

Then, the flag output unit 25 converts the area determined to be the deposit area by the determination unit 24 into units of small areas 500, and determines the deposit areas (small areas 500) in each of the three areas 510, 520a, and 520b. Calculate the area ratio of For example, the flag output unit 25 calculates the area ratio of the deposit areas located in each of the areas 510, 520a, and 520b, with the entire deposit area being 1 (100%).

For example, when the area ratios of the areas 510, 520a, and 520b are approximately the same (the difference in area ratio between the areas 510, 520a, and 520b is less than a predetermined value), the flag output unit 25 outputs the three areas 510, 520a. , 520b is the calculation result of the area calculation process.

On the other hand, if the area ratio of the first region 510 among the regions 510, 520a, 520b is larger than that of the other regions 520a, 520b by a predetermined value or more, the flag output unit 25 The total area of the regions is used as the calculation result of the area calculation process.

This is because raindrops and other deposits often adhere evenly over the entire captured image I, while the area of the road surface reflection is often concentrated on the road surface area of the captured image I. ing.

In other words, when the adhering matter area is concentrated in the first area 510 which is the road surface area, the flag output unit 25 determines that the adhering matter area is the area of road surface reflection, is used as the calculation result of the area calculation process.

It should be noted that even if the adhering matter areas are concentrated in the first area 510, which is the road surface area, the flag output unit 25 detects that the adhering matter area is a moving adhering matter area. Do not subtract the deposit area of the first area 510 from the total.

This is for detecting that a small amount of moving adhering matter has adhered as a determination result with high accuracy. In other words, when the moving deposit areas are concentrated in the first area 510, which is the road surface area, there is a high possibility that the moving deposit areas are not the road surface reflection areas but the deposit areas.

Then, if the area calculated as the result of the area calculation process is equal to or greater than a predetermined value, the flag output unit 25 outputs the adhering substance flag ON to the various devices 50, and if the area is less than the predetermined value. , the attachment flag OFF is output to the various devices 50 .

Specifically, the flag output unit 25 outputs the attached matter flag ON when the area of the moving attached matter region is equal to or larger than the first threshold value, and outputs the attached matter flag ON when the area of the moving attached matter region is less than the first threshold value. Kimono flag OFF is output.

Further, when the area of the adhering matter region (“adhering” and “moving adhering”) is equal to or larger than a second threshold larger than the first threshold, the flag output unit 25 outputs an adhering matter flag ON, and the area is less than the second threshold. If so, the adhering matter flag OFF is output.

In this manner, the flag output unit 25 sets the first threshold smaller than the second threshold in the case of the moving deposit area, so that even if a small amount of the moving deposit area with high accuracy of the determination result adheres, the flag output unit 25 It can be detected with precision.

Next, the processing procedure of the overall processing executed by the adhering matter detection device 1 according to the embodiment will be described with reference to FIG. 11 . FIG. 11 is a flow chart showing the procedure of the overall processing of the attached matter executed by the attached matter detection device 1 according to the embodiment.

As shown in FIG. 11, first, the image acquisition unit 21 acquires a captured image I captured by the camera 10, performs grayscale processing and thinning processing on the acquired captured image I, and then reduces the captured image. An integral image generated based on the pixel values of I is obtained as a captured image I (step S101).

Subsequently, the extracting unit 22 extracts a candidate area 100 of the attached matter area corresponding to the attached matter attached to the camera 10 based on the edge detected from each pixel of the captured image I acquired by the image acquiring unit 21 ( step S102).

Subsequently, the extraction unit 22 divides the band region 110 including a plurality of continuous pixel rows in the candidate region 100 into unit regions R each having a predetermined number of pixels as a unit (step S103).

Subsequently, the extraction unit 22 calculates luminance information of the unit region R (step S104). The brightness information is, for example, the average value of brightness and the standard deviation of brightness.

Subsequently, the extraction unit 22 determines whether or not the calculated standard deviation of luminance is less than a predetermined value (step S105).

When the standard deviation of luminance is less than the predetermined value (step S105: Yes), the extraction unit 22 sets the average value of luminance (luminance average) as the representative value of the unit region R (step S106).

On the other hand, when the standard deviation of luminance is equal to or greater than the predetermined value (step S105: No), the extraction unit 22 sets an abnormal value as the representative value of the unit region R (step S107).

Subsequently, the extraction unit 22 performs an exclusion process for excluding the candidate area 100 from the target of the adhering matter area determination process when the distribution of the luminance averages of the unit areas R in the candidate area 100 satisfies a predetermined exclusion condition (( step S108).

Subsequently, the extraction unit 22 extracts the candidate region 100 in which the undulation change in the distribution of the luminance average of the unit region R in the candidate region 100 matches a predetermined change pattern. Based on this, determination processing is performed to determine whether or not the candidate area 100 is an adhering matter area (step S109).

Subsequently, the flag output unit 25 performs a predetermined area calculation process based on the area of the area continuously determined as the adhering matter area by the determination unit 24, and the area calculated as the calculation result is equal to or larger than the predetermined value. It is determined whether or not there is (step S110).

When the calculated area is equal to or larger than the predetermined value (step S110: Yes), the flag output unit 25 outputs the adhering matter flag ON to the various devices 50 (step S111), and ends the process.

On the other hand, when the calculated area is less than the predetermined value (step S110: No), the flag output unit 25 outputs the adhering matter flag OFF to the various devices 50 (step S112), and ends the process.

Next, with reference to FIG. 12, the processing procedure of the adhering matter determination processing executed by the adhering matter detection apparatus 1 according to the embodiment will be described. FIG. 12 is a flow chart showing the procedure of the adhering matter determination process executed by the adhering matter detection device 1 according to the embodiment.

As shown in FIG. 12, the calculator 23 first sets the internal region 200 as the candidate region 100 (step S201).

Subsequently, the calculation unit 23 calculates a first difference, which is a luminance difference value between adjacent pixels in a plurality of pixels in the internal region 200 (step S202).

Subsequently, the calculator 23 calculates a second difference, which is a difference value between adjacent first differences (step S203).

Subsequently, the determination unit 24 determines whether the frequency of regions (the number of luminance change amounts) in which the luminance change amount, which is the second difference calculated by the calculation unit 23, is in the first range is equal to or greater than a predetermined number. (step S204).

If the region frequency in the first range is equal to or greater than the predetermined number (step S204: Yes), the determination unit 24 determines whether the region frequency in the second range is less than the predetermined number (step S205). ).

If the area frequency in the second range is less than the predetermined number (step S205: Yes), the determination unit 24 determines that the candidate area 100 is an adhering matter area (step S206), and ends the process.

On the other hand, in step S204, if the area frequency in the first range is less than the predetermined number (step S204: No), the determination unit 24 determines that the candidate area 100 is not an adhering matter area (step S207), End the process.

Further, in step S205, when the area frequency in the second range is equal to or greater than the predetermined number (step S205: No), the determination unit 24 determines that the candidate area 100 is not an adhering matter area (a road surface reflection area). (step S207), and the process ends.

As described above, the attached matter detection device 1 according to the embodiment includes the extraction unit 22 , the calculation unit 23 and the determination unit 24 . The extracting unit 22 extracts a candidate area 100 for an adhering matter area corresponding to an adhering matter adhering to the imaging device (camera 10) based on the edge detected from each pixel of the captured image I captured by the imaging device (camera 10). The calculation unit 23 calculates luminance change amounts of a plurality of pixels that are continuous in a predetermined direction in a predetermined internal region 200 in the candidate region 100 extracted by the extraction unit 22 . The determination unit 24 determines the number of luminance change amounts in the first range calculated by the calculation unit 23, and the number of luminance change amounts in the second range in which the luminance change amount is larger than the first range. It is determined whether or not the candidate area 100 is an adhering matter area based on the number. As a result, adhering matter can be detected with high accuracy.

Further, in the above-described embodiment, the captured image I captured by a camera mounted on a vehicle is used, but the captured image I is, for example, a captured image captured by a security camera or a camera installed on a streetlight or the like. It may be I. In other words, any captured image I captured by a camera in which there is a possibility that an adhering substance may adhere to the lens of the camera may be used.

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments so shown and described. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

Reference Signs List 1 adhering matter detection device 2 control unit 3 storage unit 10 camera 11 vehicle speed sensor 21 image acquisition unit 22 extraction unit 23 calculation unit 24 determination unit 25 flag output unit 31 undulation condition information 50 various devices 100 candidate area 200 internal area I captured image

Claims (4)

an extraction unit for extracting a candidate area for an adhering matter area corresponding to an adhering matter adhering to the imaging device based on edges detected from pixels of an image captured by the imaging device;
a calculation unit that calculates a luminance change amount of a plurality of pixels consecutive in a predetermined direction in a predetermined internal region in the candidate region extracted by the extraction unit;
The number of luminance change amounts in a first range calculated by the calculation unit, and the number of the luminance change amounts in a second range in which the luminance change amount is larger than the first range. a determination unit that determines whether the candidate area is the adhering matter area based on the number of
The determination unit is
When the number of luminance change amounts in the first range is a predetermined number or more and the number of the luminance change amounts in the second range is a predetermined number or more, the candidate area is the adhering matter area. A deposit detection device characterized in that it determines that there is no attached matter . The calculation unit
calculating a luminance difference value of two pixels adjacent in the predetermined direction as a first difference, and calculating a second difference, which is a difference value between the two first differences consecutive in the predetermined direction, as the luminance change amount; The attached matter detection device according to claim 1, characterized by: The calculation unit
3. The adhering matter detection apparatus according to claim 1 , wherein the amount of change in luminance of the plurality of pixels that are continuous in the vertical direction of the image is calculated . an extracting step of extracting a candidate area for an adhering matter area corresponding to an adhering matter adhering to the imaging device based on edges detected from each pixel of an image captured by the imaging device;
a calculation step of calculating a luminance change amount of a plurality of pixels consecutive in a predetermined direction in a predetermined internal region in the candidate region extracted by the extraction step;
The number of luminance change amounts in a first range calculated by the calculating step, and the number of luminance change amounts in a second range in which the luminance change amount is larger than the first range. determining whether or not the candidate area is the adhering matter area based on the number of the candidate areas, and determining whether the number of the luminance change amounts in the first range is equal to or greater than a predetermined number and in the second range. a determination step of determining that the candidate area is not the adhering matter area when the number of luminance change amounts is equal to or greater than a predetermined number;
A deposit detection method , comprising :

Images