JP7234884B2 - Attached matter detection device and attached matter detection method - Google Patents

Description

The present invention relates to an adhering matter detection device and an adhering matter detection method.

Conventionally, an average brightness value and a standard deviation of brightness are calculated for each small area obtained by dividing an area of a captured image. is known (see, for example, Patent Document 1).

JP 2019-106644 A

However, in the prior art, there is room for improvement in terms of detecting deposits with high accuracy.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an adhering matter detection device and an adhering matter detection method capable of detecting adhering matter with high accuracy.

In order to solve the above-described problems and achieve the object, an attached matter detection device according to the present invention includes an extraction section, a determination section, and a calculation section . The extracting unit extracts a candidate area for an adhering matter area corresponding to an adhering matter adhering to the imaging device based on edges detected from each pixel of an image captured by the imaging device. The determining unit divides a plurality of continuous pixel rows in the candidate region extracted by the extracting unit into unit regions each having a predetermined number of pixels as a unit, and calculates undulations in the average luminance distribution for each unit region. It is determined whether or not the candidate area is the adhering matter area based on. The calculator calculates a standard deviation of brightness for each unit area. The determination unit determines that the candidate area is not the adhering matter area when the standard deviation of the unit area is equal to or greater than a predetermined value. When the standard deviation of the unit area is equal to or greater than a predetermined value, the determination unit sets a predetermined abnormal value to the unit area instead of the luminance average, and determines the candidate area based on the abnormal value. It is determined whether or not there is an adhering matter area.
An attached matter detection method according to the present invention includes an extraction step of extracting a candidate area of an attached matter area corresponding to an attached matter attached to an imaging device based on edges detected from pixels of an image captured by the imaging device. Then, with respect to a plurality of continuous pixel rows in the candidate region extracted by the extraction step, it is divided into unit regions each having a predetermined number of pixels as a unit. and determining whether or not the candidate area is the adhering matter area. The determining step determines that the candidate area is not the adhering matter area when the standard deviation calculated for the brightness for each unit area is equal to or greater than a predetermined value. In the determination step, when the standard deviation of the unit area is equal to or greater than a predetermined value, a predetermined abnormal value is set in the unit area instead of the luminance average, and the candidate area is determined based on the abnormal value. It is determined whether or not there is an adhering matter area.

According to the present invention, adhering matter can be detected with high accuracy.

FIG. 1 is a diagram showing an outline of an attached matter detection method according to an embodiment. FIG. 2 is a block diagram showing the configuration of the attached matter detection device according to the embodiment. FIG. 3A is a diagram illustrating processing contents of a calculation unit; FIG. 3B is a diagram illustrating the processing content of the calculation unit; FIG. 4A is a diagram illustrating processing contents of a calculation unit; FIG. 4B is a diagram illustrating the processing content of the calculation unit; FIG. 5 is a diagram illustrating exclusion processing by the determination unit. FIG. 6 is a diagram showing determination processing of the determination unit. FIG. 7 is a diagram showing confirmation processing by the determination unit. FIG. 8 is a diagram showing confirmation processing by the determination unit. FIG. 9 is a diagram illustrating continuity determination processing by the determination unit. FIG. 10 is a diagram showing area calculation processing by the flag output unit. FIG. 11 is a flowchart illustrating a procedure of an attached matter detection process performed by the attached matter detection apparatus according to the embodiment.

DETAILED DESCRIPTION OF THE INVENTION Embodiments of the attached matter detection device and the attached matter detection method disclosed in the present application will be described in detail below with reference to the accompanying drawings. In addition, this invention is not limited by embodiment shown below.

First, with reference to FIG. 1, an outline of an attached matter detection method according to an embodiment will be described. FIG. 1 is a diagram showing an outline of an attached matter detection method according to an embodiment. Note that FIG. 1 shows a captured image I captured with water droplets such as raindrops adhering to the lens of an in-vehicle camera (an example of an imaging device), for example.

When dirt, dust, raindrops, snowflakes, and other substances adhere to the lens of the camera, information about the vehicle surroundings, such as parking frames, other vehicles, and people, cannot be obtained from the captured image I. There is a risk that it will not be possible to accurately detect frames, other vehicles, people, and the like. Note that the attached matter is not limited to mud, dust, raindrops, and snowflakes, and any attached matter may be used as long as the area of the attached matter becomes a blurred area.

The attached matter detection apparatus 1 (see FIG. 2) according to the embodiment detects the attached matter attached to the lens of the camera based on the captured image I captured by the camera by executing the attached matter detection method.

Specifically, first, in the adhering matter detection method according to the embodiment, based on the edge detected from each pixel of the captured image I, candidates for the adhering matter area corresponding to the adhering matter adhering to the lens of the camera are detected. A region 100 is extracted (step S1).

In the attached matter detection method according to the embodiment, for example, a rectangular area including a circular contour such as a raindrop is extracted as the candidate area 100 by matching processing such as pattern matching.

Subsequently, in the attached matter detection method according to the embodiment, a plurality of continuous pixel rows in the extracted candidate region 100 are divided into unit regions R1 to R8 each having a predetermined number of pixels as a unit, and each of the unit regions R1 to R8 An average value of luminance (luminance average) is calculated (step S2).

In FIG. 1, in a candidate region 100, a plurality of vertically continuous pixel columns of a captured image I are set as a band region 110, and the band region 110 is horizontally divided into eight unit regions R1 to R8. ing. A detailed setting method for the band region 110 and the unit regions R1 to R8 will be described later. Further, the unit regions R1 to R8 may be collectively referred to as unit regions R hereinafter.

Subsequently, in the attached matter detection method according to the embodiment, it is determined whether or not the candidate area 100 is an attached matter area based on the undulations in the distribution of the luminance average calculated for each unit area R (step S3). For example, in the adhering matter detection method according to the embodiment, when the undulating change in the average luminance distribution matches a predetermined change pattern, it is determined that the candidate region 100 is an adhering matter region. More specifically, in the adhering matter detection method according to the embodiment, when the undulation shape in the average luminance distribution is convex or concave, it is determined to be an adhering matter region. The details of the adhering matter region determination processing will be described later.

In this way, by determining the adhering matter area based on the undulations in the average luminance distribution of the unit area R obtained by dividing a plurality of continuous pixel rows, it is possible to suppress erroneous determination due to luminance noise in each pixel. . That is, according to the attached matter detection method according to the embodiment, the attached matter can be detected with high accuracy.

In addition, in the attached matter detection method according to the embodiment, the standard deviation of luminance is further calculated for each unit region R. Then, when the calculated standard deviation of luminance is equal to or greater than a predetermined threshold value, an abnormal value is set instead of the average luminance of the unit area, so that the candidate area 100 is determined not to be a deposit area. However, this point will be described later.

Next, the configuration of the adhering matter detection device 1 according to the embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing the configuration of the adhering matter detection device 1 according to the embodiment. As shown in FIG. 2 , the attached matter detection device 1 according to the embodiment is connected to a camera 10 , a vehicle speed sensor 11 and various devices 50 . Although FIG. 2 shows a case where the adhering matter detection device 1 is configured separately from the camera 10 and the various devices 50, it is not limited to this, and may be integrated with at least one of the camera 10 and the various devices 50. may consist of

The camera 10 is, for example, an in-vehicle camera that includes a lens such as a fisheye lens and an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The cameras 10 are provided, for example, at positions capable of capturing images of the front, rear, and sides of the vehicle, and output captured images I to the adhering matter detection device 1 .

The vehicle speed sensor 11 is a sensor that detects the speed of the vehicle (vehicle speed). The vehicle speed sensor 11 outputs information on the detected vehicle speed to the adhering matter detection device 1 . Note that the vehicle speed sensor 11 may be omitted and, for example, the adhering matter detection device 1 may calculate vehicle speed information based on time-series images from the camera 10 .

The various devices 50 are devices that acquire detection results of the adhering matter detection device 1 and perform various vehicle controls. The various devices 50 include, for example, a display device that notifies that an adhering substance is attached to the lens of the camera 10 and a user's instruction to wipe off the adhering matter, and a display device that injects fluid, gas, or the like toward the lens to remove the adhering matter. It includes a removal device that removes dust, a vehicle control device that controls automatic driving, etc.

As shown in FIG. 2 , the adhering matter detection device 1 according to the embodiment includes a control unit 2 and a storage unit 3 . The control unit 2 includes an image acquisition unit 21 , an extraction unit 22 , a calculation unit 23 , a determination unit 24 and a flag output unit 25 . The storage unit 3 stores undulation condition information 31 .

Here, the adhering matter detection device 1 includes, for example, a computer and various circuits having a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), data flash, input/output ports, and the like.

The CPU of the computer functions as the image acquisition section 21, the extraction section 22, the calculation section 23, the determination section 24, and the flag output section 25 of the control section 2 by reading and executing programs stored in the ROM, for example.

Further, at least one or all of the image acquisition unit 21, the extraction unit 22, the calculation unit 23, the determination unit 24, and the flag output unit 25 of the control unit 2 may be an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). ) or the like.

Also, the storage unit 3 corresponds to, for example, a RAM or a data flash. The RAM and data flash can store the undulation condition information 31, various program information, and the like. Note that the adhering matter detection apparatus 1 may acquire the above-described programs and various information via another computer or portable recording medium connected via a wired or wireless network.

The undulation condition information 31 stored in the storage unit 3 is information including conditions that serve as criteria for processing by the determination unit 24, which will be described later, and includes, for example, undulation pattern conditions of luminance distribution (distribution of average luminance). It should be noted that the pattern condition is a change pattern of undulation change in the average luminance distribution, or a luminance threshold range that each pixel (unit region R) in the luminance distribution can take. Processing using the undulation condition information 31 will be described later.

The control unit 2 extracts a candidate area 100 for the adhering matter area based on the edge detected from each pixel of the captured image I obtained from the camera 10, and calculates the distribution of the luminance average for each unit area R in the extracted candidate area 100. It is determined whether or not the candidate area 100 is an adhering matter area.

The image acquisition unit 21 acquires an image captured by the camera 10 and generates (acquires) a current frame, which is the current captured image I. Specifically, the image acquisition unit 21 performs grayscaling processing for converting each pixel in the acquired image into each gradation from white to black according to the luminance.

In addition, the image acquisition unit 21 performs pixel thinning processing on the acquired image to generate an image smaller in size than the acquired image. The image acquisition unit 21 also generates the current frame, which is an integral image of the sum of the pixel values and the sum of the squares of the pixel values of each pixel, based on the thinned image. Note that the pixel value is information corresponding to the luminance and edge of the pixel.

In this way, the adhering matter detection apparatus thins out the acquired image and generates an integral image, thereby speeding up the processing calculations in the latter stage. can be shortened.

Note that the image acquisition unit 21 may perform smoothing processing on each pixel using a smoothing filter such as an averaging filter. Alternatively, the image acquiring unit 21 may generate a current frame of the same size as the acquired image without performing the thinning process. Also, the current frame may be referred to as a captured image I below.

The extracting unit 22 extracts a candidate area 100 for the adhering matter area from the captured image I based on the edge detected from each pixel of the captured image I acquired by the image acquiring unit 21 . Specifically, first, the extraction unit 22 extracts the luminance and edge information of each pixel in the captured image I. FIG. The brightness of each pixel is represented by a parameter from 0 to 255, for example.

In addition, the extraction unit 22 performs edge detection processing based on the luminance of each pixel, and detects an edge in the X-axis direction (horizontal direction of the captured image I) and an edge in the Y-axis direction (vertical direction of the captured image I) for each pixel. ) edges. Any edge detection filter such as a Sobel filter or a Prewitt filter can be used for edge detection processing.

Then, the extraction unit 22 detects, as edge information, a vector including information on the edge angle and edge strength of the pixel by using a trigonometric function based on the edge in the X-axis direction and the edge in the Y-axis direction. Specifically, the edge angle is represented by the orientation of the vector, and the edge strength is represented by the length of the vector.

Then, the extraction unit 22 performs a matching process (template matching) between the template information created in advance indicating the outline of the adhering matter and the detected edge information, and extracts edge information similar to the template information. Then, the extracting unit 22 extracts the area of the extracted edge information, that is, the candidate area 100 which is a rectangular area including the outline of the adhering matter.

The extraction unit 22 outputs the extracted information of the candidate region 100 to the calculation unit 23 .

The calculation unit 23 divides a plurality of continuous pixel rows in the candidate region 100 extracted by the extraction unit 22 into unit regions R each having a predetermined number of pixels as a unit, and calculates an average luminance value and a luminance value for each unit region R. Calculate the standard deviation.

Here, processing contents of the calculation unit 23 will be described with reference to FIGS. 3A to 4B. 3A to 4B are diagrams showing the processing contents of the calculator 23. FIG.

As shown in FIGS. 3A and 3B, first, the calculator 23 sets a plurality of consecutive pixel columns in the candidate region 100 as the band region 110. FIG. The band area 110 is an area to be subjected to the subsequent process of dividing the unit area R and the process of calculating luminance information (average value and standard deviation).

Specifically, as shown in FIG. 3A, first, the calculation unit 23 selects three horizontal pixel columns H1 to H3 and three vertical pixel columns V1 to V3 in the candidate region 100. do. Then, the calculation unit 23 sets a plurality of pixel rows including the selected pixel row as the band region 110 for each pixel row selected.

FIG. 3B shows a strip region 110 containing a horizontal row of pixels H1. Note that the method of setting the band region 110 including the vertical pixel column is substantially the same as the method of setting the band region 110 including the horizontal pixel column H1 described below (it is possible to determine whether the pixel column is horizontal or vertical). only difference), so the description is omitted.

As shown in FIG. 3B, the calculation unit 23 sets a plurality of pixel rows that are continuous in the vertical direction as the band region 110 with the selected pixel row H1 as the center. In other words, the selected pixel row H1 and a plurality of pixel rows parallel to the selected pixel row H1 and adjacent to each other are set as the band region 110 .

In FIG. 3B, a total of 7 pixel columns including 3 pixel columns (total of 6 pixel columns) continuous to the vertical upper side and lower side of the horizontal pixel column H1 and 1 pixel column H1. A row of pixels is set as a band region 110 .

Note that the position of the pixel row H1 in the strip region 110 may not be the center of the strip region 110, and may be located at either the upper side or the lower side in the vertical direction. In addition, the number of pixel columns that are continuous above and below the pixel column H1 in the vertical direction may be two or less, or four or more.

The pixel columns H1 to H3 and V1 to V3 to be selected may be pixel columns in either one of the horizontal pixel columns H1 to H3 and the vertical pixel columns V1 to V3. Further, the number of pixel columns H1 to H3 and V1 to V3 selected in each direction is not limited to three, and may be two or less or four or more.

Subsequently, as shown in FIG. 4A, the calculation unit 23 divides the set band region 110 into unit regions R1 to R8 each having a predetermined number of pixels as a unit, and calculates an average luminance value and a luminance value for each of the unit regions R1 to R8. Calculate the standard deviation of

Note that FIG. 4A shows a case where the calculation unit 23 sets eight unit regions R1 to R8 obtained by dividing the band region 110 at equal intervals (same number of pixels), but the number of unit regions R is six. It may be less than or equal to nine or more. Also, the number of pixels included in each unit region R may be the same or different.

Then, when the calculated standard deviation of luminance is less than a predetermined value, the calculation unit 23 sets the average luminance value as the representative value of the unit regions R in the band region 110 .

On the other hand, as shown in FIG. 4B, when the calculated standard deviation of luminance is equal to or greater than a predetermined value, the calculation unit 23 sets the distribution of the luminance average in the band region 110 to a predetermined value instead of the luminance average of the unit region R. An abnormal value is set to a representative value of the unit region R.

In the example shown in FIG. 4B , when the standard deviation of the brightness of the unit region R8 is equal to or greater than a predetermined value, the calculation unit 23 replaces the average value of the brightness of the unit region R8 with a predetermined abnormal value (for example, the minimum value , zero, etc.) is set as a representative value of the unit area R8.

This abnormal value is a value that is determined as not being an adhering matter area by the determination unit 24 in the subsequent stage. For example, when the determination unit 24, which will be described later, determines whether or not the distribution of the average brightness matches a predetermined pattern to determine whether or not there is an adhering matter region, the calculation unit 23 sets an abnormal value that does not match the predetermined pattern. . That is, when the standard deviation of the unit region R is equal to or greater than a predetermined value, the determination unit 24 at the subsequent stage determines that the candidate region 100 is not a deposit region. As a result, it is possible to suppress erroneous determination of the candidate area 100 containing noise components as an adhering matter area.

Furthermore, by setting an abnormal value that is determined to be not an adhering matter region by the determination unit 24 at a later stage, the candidate region 100 that is not an adhering matter region can be removed without adding new processing. complication can be suppressed.

Although the calculation unit 23 uses the average luminance value as the representative value of the unit regions R, for example, a luminance histogram is created for each of the unit regions R1 to R8, and the mode, median, and average A value or the like may be set as a representative value.

The determination unit 24 determines whether or not the candidate region 100 is a deposit region based on the undulations in the average luminance distribution calculated for each unit region R by the calculation unit 23 .

First, if the unevenness of the luminance distribution (distribution of luminance average) of the pixels included in the candidate region 100 satisfies a predetermined exclusion condition, the determination unit 24 excludes the candidate region 100 from the target of the adhering matter region determination processing.

Here, the exclusion processing of the candidate area 100 by the determination unit 24 will be described with reference to FIG. FIG. 5 is a diagram showing exclusion processing by the determination unit 24. As shown in FIG. FIG. 5 shows the average luminance distribution of each band area 110 calculated from the predetermined candidate area 100 . FIG. 5 shows the average luminance distribution for each band region 110 corresponding to each pixel row H1 to H3 and V1 to V3 set in the candidate region 100. FIG. For example, the graph of "H1" in FIG. 5 shows the distribution of the luminance average of the unit area R in the band area 110 including the pixel row H1.

If the undulations in the average luminance distribution of each band region 110 corresponding to each of the pixel columns H1 to H3 and V1 to V3 satisfy a predetermined exclusion condition, the determination unit 24 determines the candidate region 100 as the target of the adhering matter region determination processing. Exclude from

For example, if the representative value of the unit regions R included in the band region 110 includes an abnormal value, the determination unit 24 excludes the candidate region 100 from the target of the adhering matter region determination processing. In other words, if the difference between the representative values of two adjacent unit regions R in the band region 110 is greater than or equal to a predetermined value, the determination unit 24 excludes the candidate region 100 from the adhering matter region determination processing target.

In other words, if the unit regions R in the band region 110 include a unit region R whose standard deviation of the representative value is equal to or greater than a predetermined value (that is, the unit region R for which an abnormal value is set) , the candidate region 100 is excluded from the adhering matter region determination processing.

That is, when part of the undulations in the distribution of the average luminance has a shape that is abruptly uneven due to an abnormal value, the determination unit 24 determines that the exclusion condition is satisfied and excludes the candidate region 100 from the target of the adhering matter region determination processing. exclude. In this manner, the determination unit 24 can perform exclusion processing (which can also be referred to as processing for determining that there is no deposit area) based on the abnormal value, and can remove the non-attachment area with high accuracy.

Further, as shown in the graphs of the pixel columns V1 to V3 in FIG. 5, the determination unit 24 determines that the undulations in the average luminance distribution of the unit region R are flat in all three graphs, that is, in the pixel columns V1 to V3 in the vertical direction. If the undulation change of the luminance average for all three corresponding band regions 110 is within a predetermined range, the candidate region 100 is excluded from the object region determination processing. Specifically, it has been found by experiments that when the candidate region 100 reflects the light from the back lamp on the wall, the average luminance distribution of the unit region R tends to be flat. Therefore, if the undulation change of the luminance average is within a predetermined range, it is determined that the candidate area 100 is caused by the reflection of the back lamp, and the object area 400 is excluded from the object of determination processing.

Note that FIG. 5 shows the case where the three luminance distributions in the vertical pixel rows V1 to V3 are flat. If the six luminance distributions in the horizontal and horizontal pixel rows H1 to H3 and V1 to V3 are flat, the candidate area 100 may be excluded from the adhering matter area determination process. In other words, the determination unit 24 determines that the exclusion condition is satisfied when at least one of the pixel rows in the vertical direction and the horizontal direction in the candidate region 100 has an undulation change in the luminance distribution within a predetermined range, and the candidate region 100 satisfies the exclusion condition. 100 is excluded from the target of the adhering matter region determination processing.

In this manner, when the unevenness of the luminance distribution in the candidate region 100 satisfies a predetermined exclusion condition, the determining unit 24 excludes the candidate region 100 from the target of the adhering matter region determination processing. Misjudgments can be reduced. That is, adhering matter can be detected with high accuracy.

Subsequently, the determination unit 24 performs determination processing of whether or not the candidate areas 100 other than the candidate areas 100 excluded by the exclusion condition are adhering matter areas. Here, determination processing by the determination unit 24 will be described with reference to FIG. 6 .

FIG. 6 is a diagram showing determination processing of the determination unit 24. As shown in FIG. First, as shown in FIG. 6, the determination unit 24 first calculates the variation amounts D1 to D7 of the brightness averages of the adjacent unit regions R1 to R8 for each band region 110 .

Then, if there is a band region 110 among the plurality of band regions 110 in the candidate region 100 where the undulation change pattern in the luminance average distribution of the unit region R satisfies a predetermined change pattern, the determination unit 24 determines that the candidate region 100 is determined to be the deposit area. For example, the determination unit 24 performs determination processing by comparing each value of the variation amounts D1 to D7 with each value of the variation pattern included in the undulation condition information 31 stored in the storage unit 3 . The variation pattern included in the undulation condition information 31 is the threshold range of the maximum and minimum values that the variations D1 to D7 can take.

That is, when each value of the calculated variation amounts D1 to D7 is within the threshold range of each of the variation amounts D1 to D7 included in the undulation condition information 31, the determination unit 24 determines that the candidate area 100 is a deposit area. judge.

In other words, the determination unit 24 determines that the pattern of the variation amounts D1 to D7 of the luminance averages of the adjacent unit regions R1 to R8 satisfies the variation pattern, which is the threshold range set in the undulation condition information 31. It is determined to be an area.

In addition, by setting the maximum and minimum values of the variation amounts D1 to D7 in the undulation condition information 31 to provide a width, even if the undulations of the luminance distribution vary somewhat, it can be detected as an adhering matter area.

FIG. 6 shows a case where threshold ranges for all the variation amounts D1 to D7 are set for the undulation condition information 31. Of D7, only the threshold range of a part of the amount of change may be set.

Note that the determining unit 24 calculates the amount of change in brightness in a predetermined area for the candidate area 100 determined to be an adhering matter area, and performs confirmation processing for confirming the determination result of the attached matter area based on the calculated amount of change in luminance. may This point will be described with reference to FIGS. 7 and 8. FIG.

FIG. 7 and FIG. 8 are diagrams showing determination processing of an adhering matter region by the determination unit 24. FIG. In the confirmation process, the determining unit 24 first calculates the amount of change in brightness of a plurality of pixels that are consecutive in a predetermined direction in a predetermined inner region 200 in the candidate region 100 that is determined to be an adhering matter region.

Specifically, as shown in FIG. 7 , the determination unit 24 first sets an internal region 200 for the candidate region 100 . For example, the determination unit 24 divides the candidate region 100 into a predetermined number of segments in each of the vertical and horizontal directions. FIG. 7 shows a case where the candidate area 100 is divided into four in each of the vertical direction and the horizontal direction, but may be divided into three or less, or five or more.

Then, the determination unit 24 sets the central region as the internal region 200 among the divided regions divided into the predetermined number. In FIG. 7 , of the 16 divided areas generated by dividing into four in each of the vertical and horizontal directions, the four central areas are set as the internal areas 200 .

Then, the determination unit 24 calculates the amount of change in brightness of the plurality of pixels in the internal region 200 . In FIG. 7, the determination unit 24 calculates the amount of change in brightness of three pixels that are consecutive in the vertical direction. This is because the area of road surface reflection often extends in the vertical direction. As a result, the area of the road surface reflection can be determined with high accuracy by the subsequent determination processing. Note that the calculation unit 23 may calculate the amount of change in brightness of a plurality of pixels that are consecutive in the horizontal direction.

First, the determination unit 24 calculates a luminance difference value between two vertically adjacent pixels as a first difference. In the middle part of FIG. 7, one first difference is defined as one cell, which is arranged two-dimensionally in correspondence with the arrangement of pixels. That is, in the middle part of FIG. 7, one square represents the luminance difference value of two vertically adjacent pixels, that is, the first difference, and two vertically adjacent squares represent two second values calculated from three pixels. 1 difference.

Subsequently, the determination unit 24 calculates a second difference, which is a difference value between two first differences that are consecutive in the vertical direction. In the lower part of FIG. 7, one second difference converted into an absolute value is set as one cell and arranged two-dimensionally in correspondence with the arrangement of pixels. That is, in the lower part of FIG. 7, one square is one second difference (luminance change amount) calculated from three pixels. In other words, the second difference is the amount of change in the first difference.

When the luminance fluctuates sharply in this manner, it is likely that the luminance is caused by a small scratch on the road surface, and is highly likely to be a reflection area of the road surface. Therefore, the determination unit 24 calculates the second difference as the luminance change amount of a plurality of pixels, so that it can be easily determined in the post-processing whether the area is an adhering matter area or a road surface reflection area. Specifically, for example, in a region where the luminance of three consecutive pixels rapidly rises and falls due to a small scratch on the road surface ("180"→"160"→"180" shown in FIG. 7), the second difference is higher. That is, by calculating the second difference, it becomes possible to easily identify such a region where the luminance fluctuates sharply. can be determined.

Although the determination unit 24 sets the second difference as the amount of luminance change of a plurality of pixels, the first difference may be set as the amount of luminance change of a plurality of pixels. When the first difference is the luminance change amount of a plurality of pixels, the number of pixels in the plurality of pixels should be 2 pixels or more. Specifically, when calculating the luminance change amount of two pixels, one calculated first difference may be used as the luminance change amount. may be used as the luminance change amount.

Subsequently, the determination unit 24 performs determination processing using the second difference, which is the calculation result, as the amount of change in brightness of a plurality of pixels. Specifically, as shown in FIG. 8, the determining unit 24 determines that the candidate area 100 is the adherent area by forming a histogram of the calculated luminance change amounts of the plurality of pixels.

FIG. 8 shows a histogram of luminance change amounts for a plurality of pixels included in one internal region 200 . In the histogram shown in FIG. 8, the horizontal axis represents the luminance change amount, and the vertical axis represents the area frequency. In addition, FIG. 8 shows a histogram when the candidate area 100 is the deposit area and a histogram when the candidate area 100 is the road surface reflection area (area other than the deposit).

Note that the area frequency is a number corresponding to the number of pixels, and one frequency corresponds to one pixel. In other words, one frequency corresponds to a luminance change amount (ie, one luminance change amount) in one plurality of pixels. Also, the area frequency is a frequency normalized so that the sum of the frequencies is a predetermined value. In other words, the sum of the area frequencies of the histogram shown in FIG. 8 is the same between the internal areas 200 having different sizes.

As shown in FIG. 8, when the histogram of the adhering matter area and the histogram of the road surface reflection area are compared, the area frequencies in the first range are similar, but in the second range, the road surface reflection , the total area frequency is slightly higher than that of the deposit area. This is because scratches, stains, etc. on the road surface affect the area frequency of the second range in the area of the road surface reflection.

Focusing on this point, the determination unit 24 determines the number of luminance change amounts whose luminance change amount is in the first range and the luminance change amount in the second range where the luminance change amount is larger than that of the first range. The candidate area 100 is determined to be a fouling area based on the numbers.

Specifically, the determining unit 24 determines the sum of the area frequencies in the first range whose minimum value is the amount of change in brightness of "0", and the second range whose minimum value is greater than the maximum value of the first range. The candidate region 100 is determined to be a deposit region based on the sum of the region frequencies in .

For example, when the sum of the area frequencies in the first range is equal to or greater than a predetermined value and the sum of the area frequencies in the second range is less than the predetermined value, the determination unit 24 determines that the candidate area 100 is the adherent area. Confirm that there is.

On the other hand, if the sum of the area frequencies in the first range is a predetermined value or more and the sum of the area frequencies in the second range is a predetermined value or more, the determination unit 24 determines that the candidate area 100 is a deposit area. Determine that there is no (road surface reflection area).

As a result, the determining unit 24 can exclude the area corresponding to the road surface reflection area from the candidate area 100, thereby suppressing erroneous detection of the road surface reflection area as an adhering matter area.

Further, if the sum of the area frequencies in the first range is less than a predetermined value, the determination unit 24 determines that the candidate area 100 is not a deposit area regardless of the sum of the area frequencies in the second range. and confirm.

Then, the determination unit 24 performs continuity determination processing to determine whether or not the candidate area 100 has been continuously determined (confirmed) as an adhering matter area. Here, the continuity determination processing by the determination unit 24 will be described.

FIG. 9 is a diagram showing continuity determination processing by the determination unit 24 . FIG. 9 shows a diagram of a state machine for managing the state of each candidate area 100, and shows state transitions regarding adhering matter in the candidate area 100. In FIG.

As shown in FIG. 9, candidate region 100 can transition to four states: "IDLE," "latent," "attached," and "move attached."

"IDLE" indicates a state in which the area is not determined as an adhering matter area. In other words, "IDLE" indicates that the candidate area 100 is not continued as an adhering matter area.

"Latent" is a state indicating that there is a possibility that deposits are attached. In other words, "latency" indicates that the candidate area 100 has been determined to be an adhering matter area but has been continuously determined as an adhering matter area for a period of less than a predetermined period of time.

“Attachment” is a state indicating that an attachment is attached. That is, "adhesion" indicates that the candidate area 100 has been continuously determined as an adhering matter area for a predetermined period or longer.

"Moving attachment" is a state indicating that the attached matter continues to adhere even when the vehicle is moving (vehicle speed is equal to or higher than a predetermined value). That is, "moving attachment" indicates that the candidate area 100 is continuously determined as an attachment area for a predetermined period or longer even while the vehicle is moving. In the following description, the candidate area 100 (attachment area) in the "moving attachment" state may be referred to as a moving attachment area.

The transition between states will be described below. Note that the state transition can be performed based on the score indicating the continuity of the determination process.

<"IDLE" → "Conceal">
For example, the determination unit 24 transitions the candidate area 100, which is determined to be an adhering matter area for the first time, from "IDLE" to "latent".

<"Concealment" → "IDLE">
For example, if the determination unit 24 determines that the candidate region 100, which has been continuously determined as an adhering matter region for a period of time less than a predetermined period, is not an adhering matter region in the current captured image I, the determination unit 24 changes "latency" to " IDLE”.

<"Concealment" → "Attachment">
For example, the determining unit 24 makes a transition from "latency" to "adherence" when the period during which the adhering matter area is continuously determined is equal to or longer than a predetermined period. The determination unit 24 outputs, to the flag output unit 25, the information of the attached matter area that is associated with the state of the “attachment” when the transition from “latent” to “attachment” occurs.

<“Attachment” → “IDLE”>
For example, if the determination unit 24 determines that the candidate area 100, which has been continuously determined as an adhering matter area for a predetermined period or longer, is not an adhering matter area in the current captured image I, the determining unit 24 changes "adhering" to " IDLE”.

<“Attachment” → “Moving attachment”>
For example, after transitioning to the "adhesion" state, the determination unit 24 changes the state from "adhesion" to "moving adhesion" when the period during which the adhering matter area is continuously determined even while the vehicle is moving is equal to or longer than a predetermined period. transition. When the state of "attachment" is changed to "moving attachment", the determination unit 24 outputs to the flag output unit 25 the information of the attachment area (moving attachment area) associated with the state of "moving attachment". .

<“Moving attachment” → “IDLE”>
For example, if the determining unit 24 determines that the current captured image I acquired while the vehicle is moving is not an attached object area in the state of "moving attachment", the determination unit 24 causes a transition from "moving attachment" to "IDLE". .

The flag output unit 25 outputs an adhering matter flag indicating whether or not an adhering matter is adhered based on the determination result of the determining unit 24 to the various devices 50 . First, the flag output unit 25 performs a predetermined area calculation process based on the area determined by the determination unit 24 to be an adhering matter area.

Here, with reference to FIG. 10, the area calculation processing of the adhering matter area by the flag output unit 25 will be described.

FIG. 10 is a diagram showing area calculation processing by the flag output unit 25. As shown in FIG. As shown in FIG. 10, first, the flag output unit 25 divides the entire area of the captured image I into predetermined small areas 500, and divides each small area 500 into three areas 510, 520a, and 520b.

Specifically, the first area 510 is an area corresponding to the road surface. The second region 520a is a region (body region) corresponding to the vehicle body. The third area 520b is an area corresponding to the sky.

For example, the flag output unit 25 detects a horizontal line in the captured image I, and sets an area above the horizontal line as the third area 520b. Also, the second area 520a is an area preset according to the mounting position of the camera 10 . In addition, the flag output unit 25 sets the area of the entire area of the captured image I excluding the second area 520 a and the third area 520 b as the first area 510 .

Then, the flag output unit 25 converts the area determined to be the deposit area by the determination unit 24 into units of small areas 500, and determines the deposit areas (small areas 500) in each of the three areas 510, 520a, and 520b. Calculate the area ratio of For example, the flag output unit 25 calculates the area ratio of the deposit areas located in each of the areas 510, 520a, and 520b, with the entire deposit area being 1 (100%).

For example, when the area ratios of the areas 510, 520a, and 520b are approximately the same (the difference in area ratio between the areas 510, 520a, and 520b is less than a predetermined value), the flag output unit 25 outputs the three areas 510, 520a. , 520b is the calculation result of the area calculation process.

On the other hand, if the area ratio of the first region 510 among the regions 510, 520a, 520b is larger than that of the other regions 520a, 520b by a predetermined value or more, the flag output unit 25 The total area of the regions is used as the calculation result of the area calculation process.

This is because raindrops and other deposits often adhere evenly over the entire captured image I, while the area of the road surface reflection is often concentrated on the road surface area of the captured image I. ing.

In other words, when the adhering matter area is concentrated in the first area 510 which is the road surface area, the flag output unit 25 determines that the adhering matter area is the area of road surface reflection, is used as the calculation result of the area calculation process.

It should be noted that even if the adhering matter areas are concentrated in the first area 510, which is the road surface area, the flag output unit 25 detects that the adhering matter area is a moving adhering matter area. Do not subtract the deposit area of the first area 510 from the total.

This is for detecting that a small amount of moving adhering matter has adhered as a determination result with high accuracy. In other words, when the moving deposit areas are concentrated in the first area 510, which is the road surface area, there is a high possibility that the moving deposit areas are not the road surface reflection areas but the deposit areas.

Then, if the area calculated as the result of the area calculation process is equal to or greater than a predetermined value, the flag output unit 25 outputs the adhering substance flag ON to the various devices 50, and if the area is less than the predetermined value. , the attachment flag OFF is output to the various devices 50 .

Specifically, the flag output unit 25 outputs the attached matter flag ON when the area of the moving attached matter region is equal to or larger than the first threshold value, and outputs the attached matter flag ON when the area of the moving attached matter region is less than the first threshold value. Kimono flag OFF is output.

Further, when the area of the adhering matter region (“adhering” and “moving adhering”) is equal to or larger than a second threshold larger than the first threshold, the flag output unit 25 outputs an adhering matter flag ON, and the area is less than the second threshold. If so, the adhering matter flag OFF is output.

In this manner, the flag output unit 25 sets the first threshold smaller than the second threshold in the case of the moving deposit area, so that even if a small amount of the moving deposit area with high accuracy of the determination result adheres, the flag output unit 25 It can be detected with precision.

Next, with reference to FIG. 11, a processing procedure of processing executed by the adhering matter detection device 1 according to the embodiment will be described. FIG. 11 is a flow chart showing the procedure of the adhering matter detection process executed by the adhering matter detection device 1 according to the embodiment.

As shown in FIG. 9, first, the image acquisition unit 21 acquires a captured image I captured by the camera 10, performs grayscale processing and thinning processing on the acquired captured image I, and then reduces the captured image. An integral image generated based on the pixel values of I is acquired as a captured image I (step S101).

Subsequently, the extracting unit 22 extracts a candidate area 100 of the attached matter area corresponding to the attached matter attached to the camera 10 based on the edge detected from each pixel of the captured image I acquired by the image acquiring unit 21 ( step S102).

Subsequently, the calculation unit 23 divides the band region 110 including a plurality of continuous pixel rows in the candidate region 100 into unit regions R each having a predetermined number of pixels as a unit (step S103).

Subsequently, the calculator 23 calculates luminance information of the unit region R (step S104). The brightness information is, for example, the average value of brightness and the standard deviation of brightness.

Subsequently, the calculation unit 23 determines whether or not the calculated standard deviation of luminance is less than a predetermined value (step S105).

If the standard deviation of luminance is less than the predetermined value (step S105: Yes), the calculation unit 23 sets the average value of luminance (luminance average) as the representative value of the unit region R (step S106).

On the other hand, if the standard deviation of luminance is equal to or greater than the predetermined value (step S105: No), the calculator 23 sets an abnormal value as the representative value of the unit region R (step S107).

Subsequently, the determining unit 24 performs an exclusion process for excluding the candidate area 100 from the target of the adhering matter area determination process when the distribution of the luminance averages of the unit areas R in the candidate area 100 satisfies a predetermined exclusion condition (( step S108).

Subsequently, the determination unit 24 performs determination processing for determining the candidate region 100 as an adhering matter region when the undulation change in the distribution of the luminance average of the unit region R in the candidate region 100 matches a predetermined change pattern ( step S109).

Subsequently, the flag output unit 25 performs a predetermined area calculation process based on the area of the area determined to be the continuous adhering matter area by the determination unit 24, and the area calculated as the calculation result is equal to or larger than the predetermined value. It is determined whether or not (step S110).

When the calculated area is equal to or larger than the predetermined value (step S110: Yes), the flag output unit 25 outputs the adhering matter flag ON to the various devices 50 (step S111), and ends the process.

On the other hand, when the calculated area is less than the predetermined value (step S110: No), the flag output unit 25 outputs the adhering matter flag OFF to the various devices 50 (step S112), and ends the process.

As described above, the adhering matter detection device 1 according to the embodiment includes the extraction 22 section and the determination section 24 . The extracting unit 22 extracts a candidate area 100 of an adhering matter area corresponding to an adhering matter adhering to the imaging device (camera 10) based on the edge detected from each pixel of the captured image I captured by the imaging device (camera 10). The determination unit 24 divides a plurality of continuous pixel rows in the candidate region 100 extracted by the extraction unit 22 into unit regions R each having a predetermined number of pixels as a unit, and calculates the distribution of the luminance average for each unit region R. It is determined whether or not the candidate area 100 is an adhering matter area based on the undulations at . As a result, adhering matter can be detected with high accuracy.

Further, in the above-described embodiment, the captured image I captured by a camera mounted on a vehicle is used, but the captured image I is, for example, a captured image captured by a security camera or a camera installed on a streetlight or the like. It may be I. In other words, any captured image I captured by a camera in which there is a possibility that an adhering substance may adhere to the lens of the camera may be used.

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments so shown and described. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

Reference Signs List 1 adhering matter detection device 2 control unit 3 storage unit 10 camera 11 vehicle speed sensor 21 image acquisition unit 22 extraction unit 23 calculation unit 24 determination unit 25 flag output unit 31 undulation condition information 50 various devices 100 candidate area 200 internal area I captured image

Claims (2)

an extraction unit for extracting a candidate area for an adhering matter area corresponding to an adhering matter adhering to the imaging device based on edges detected from pixels of an image captured by the imaging device;
A plurality of continuous pixel rows in the candidate region extracted by the extraction unit are divided into unit regions each having a predetermined number of pixels as a unit, and the candidates are calculated based on undulations in the average luminance distribution calculated for each unit region. a determination unit that determines whether or not an area is the adhering matter area ;
Further comprising a calculation unit for calculating the standard deviation of luminance for each unit area,
The determination unit is
determining that the candidate area is not the adhering matter area when the standard deviation of the unit area is equal to or greater than a predetermined value;
The determination unit is
When the standard deviation of the unit area is equal to or greater than a predetermined value, a predetermined abnormal value is set in the unit area instead of the luminance average, and the candidate area is the adhering matter area based on the abnormal value. An adhering matter detection device characterized by determining whether or not . an extracting step of extracting a candidate area for an adhering matter area corresponding to an adhering matter adhering to the imaging device based on edges detected from each pixel of an image captured by the imaging device;
A plurality of continuous pixel rows in the candidate region extracted by the extraction step is divided into unit regions each having a predetermined number of pixels as a unit, and the candidate is calculated based on the undulations in the average luminance distribution calculated for each unit region. a determination step of determining whether the area is the adhering matter area;
including
the determining step determines that the candidate area is not the adhering matter area when the standard deviation calculated for the brightness of each unit area is equal to or greater than a predetermined value;
In the determination step, when the standard deviation of the unit area is equal to or greater than a predetermined value, a predetermined abnormal value is set in the unit area instead of the luminance average, and the candidate area is determined based on the abnormal value. A deposit detection method , comprising determining whether or not there is a deposit area .

Images