JP7210882B2 - Attached matter detection device and attached matter detection method - Google Patents

Description

The disclosed embodiments relate to a deposit detection device and a deposit detection method.

Conventionally, an in-vehicle camera that is mounted on a vehicle and captures an image of the surroundings of the vehicle is known. Images captured by an in-vehicle camera are displayed on a monitor to assist the driver's vision, or are used for sensing to detect white lines on the road, objects approaching the vehicle, and the like.

It should be noted that, for example, raindrops, snowflakes, dust, mud, and other deposits may adhere to the lens of the vehicle-mounted camera, interfering with the vision assistance and sensing described above. Therefore, a technology has been proposed to remove the deposits by injecting cleaning water or compressed air toward the lens of the vehicle-mounted camera. In such a technique, for example, an object attached to a lens is detected by image analysis of an image captured by an in-vehicle camera (see Patent Document 1, for example).

Japanese Patent Application Laid-Open No. 2001-141838

However, the conventional technology described above has room for further improvement in terms of suppressing failures in detection of adhering matter.

For example, if the adhering matter is water droplets such as raindrops, the appearance of the water droplets changes depending on the background that changes as the vehicle moves and the thickness of the water droplets. could have occurred.

One aspect of the embodiments has been made in view of the above, and an object thereof is to provide an adhering matter detection device and an adhering matter detection method capable of suppressing detection omissions of adhering matter.

An attached matter detection device according to an aspect of an embodiment includes a detection unit and a determination unit. The detection unit detects a position in a captured image of an object attached to the imaging device based on a predetermined first condition. The determination unit determines whether or not the adhering matter exists at the position based on information regarding the position detected a plurality of times with respect to the captured images arranged in time series . Further, when the position is detected by the detection section, the determination section causes the detection section to detect the position based on a second condition that is relaxed compared to the first condition.

According to one aspect of the embodiment, it is possible to suppress detection omissions of adhering matter.

FIG. 1A is a schematic explanatory diagram of a conventional adhering matter detection method. FIG. 1B is a schematic explanatory diagram of the attached matter detection method according to the first embodiment. FIG. 2 is a block diagram of the automatic parking system according to the first embodiment. FIG. 3 is a diagram showing a vector calculation method. FIG. 4A is an explanatory diagram (Part 1) of a method of calculating a representative value. FIG. 4B is an explanatory diagram (part 2) of the method of calculating the representative value. FIG. 5A is a diagram showing an example of a template. FIG. 5B is a diagram illustrating an example of matching processing. FIG. 6 is a diagram illustrating an example of detection area extraction processing. FIG. 7A is a diagram illustrating an example of notification content from a detection unit; FIG. 7B is a diagram showing an example of the data content regarding the detection area included in the detection information. FIG. 7C is an explanatory diagram of the state of the detection area. FIG. 8A is a diagram (part 1) for explaining the processing of the determination unit; FIG. 8B is a process explanatory diagram (part 2) of the determination unit; FIG. 8C is a diagram (part 3) for explaining the processing of the determination unit; FIG. 8D is a diagram (part 4) for explaining the processing of the determination unit; FIG. 9 is a process explanatory diagram of the detection unit after feedback. FIG. 10 is a flow chart showing a processing procedure executed by the adhering matter detection device according to the first embodiment. FIG. 11A is a diagram showing the flow of processing according to the first embodiment. FIG. 11B is a diagram showing the flow of processing according to the first modification. FIG. 11C is a diagram showing the flow of processing according to the second modification. FIG. 12 is a block diagram of a deposit removing system according to the second embodiment.

DETAILED DESCRIPTION OF THE INVENTION Embodiments of the attached matter detection device and the attached matter detection method disclosed in the present application will be described in detail below with reference to the accompanying drawings. In addition, this invention is not limited by embodiment shown below.

Moreover, below, the case where the waterdrop which adhered to the vehicle-mounted camera mounted in the vehicle in order to image the surroundings of the own vehicle is detected as an attachment is mentioned as an example, and is demonstrated.

Further, in the following, after an overview of the attached matter detection method according to the first embodiment is described with reference to FIGS. 1A and 1B, an attached matter detection apparatus 10 to which the attached matter detection method according to the first embodiment is applied will be described with reference to FIGS. 2 to 11C. Also, with reference to FIG. 12, a deposit removal system 1A according to the second embodiment will be described.

(First embodiment)
First, an overview of the attached matter detection method according to the present embodiment will be described with reference to FIGS. 1A and 1B. FIG. 1A is a schematic explanatory diagram of a conventional adhering matter detection method. Moreover, FIG. 1B is a schematic explanatory diagram of the attached matter detection method according to the present embodiment.

Vehicles are equipped with in-vehicle cameras such as a front camera, a back camera, a right side camera, and a left side camera, in order to capture images of the surroundings of the vehicle. In addition, below, such a vehicle-mounted camera is described as "camera 2."

The camera 2 includes an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and images the surroundings of the vehicle with the imaging element. A wide-angle lens such as a fish-eye lens is adopted as the lens of the camera 2, and each camera 2 has an angle of view of 180 degrees or more.

An image captured by the camera 2 is output to an attached matter detection device (not shown) mounted on the vehicle.

The adhering substance detection device analyzes the image acquired from the camera 2 frame by frame, and detects the presence area of the adhering substance in the image by using a technique such as template matching.

For example, as shown in FIG. 1A, there are temporally consecutive images of frames F1 to F5. shall be detected. In this case, since the detection area Da-1 can be stably detected, the attached matter detection device can easily determine that the attached matter is attached to the detection area Da-1 (hereinafter referred to as "attachment confirmation ”) can be done.

However, when the adhering matter is water droplets, the appearance of the water droplets in the image changes depending on the background that changes with the movement of the vehicle and the thickness of the water droplets. Detection tends to be unstable, being detected and not detected depending on the frame. In other words, the adhering matter detection device cannot determine whether the adhering matter is "determined" in the detection area Da-2, and there is a possibility that the adhering matter detection failure occurs.

Therefore, in the adhering matter detection method according to the present embodiment, a detection area in which an adhering matter is detected based on a predetermined condition in a certain frame is inherited in temporally subsequent frames. In addition, since it is estimated that there is a high possibility that an adhering matter exists in the inherited detection area, the presence or absence of an adhering matter is determined under a condition that is less severe than the predetermined condition.

Specifically, as shown in FIG. 1B, in the attached matter detection method according to the present embodiment, for example, when the detection area Da-2 is detected under a predetermined first condition in the frame F1 (step S1), immediately after The detection area Da-2 is inherited to the frame F2 of (step S2).

Then, in the attached matter detection method according to the present embodiment, in the inherited detection area Da-2, the presence or absence of the attached matter is determined under the second condition, which is more relaxed than the first condition (step S3). Then, as a result of the determination in step S3, if it is determined that an adhering matter exists in the detection area Da-2 in the frame F2 as well, the detection area Da-2 in the frame F2 is inherited in the frame F3 as well, After that, the frames F4 and F5 will be taken over in the same way.

This can eliminate the instability of detection when the adhering matter is water droplets or the like. Therefore, according to the adhering matter detection method according to the present embodiment, it is possible to suppress detection omissions of adhering matter.

In addition, according to the attached matter detection method according to the present embodiment, the detection results of the detection area and the like that have been detected once can be effectively used in subsequent analysis processing, etc., so that the processing load of the entire system can be reduced. At the same time, improvement in processing speed and detection accuracy can be expected.

An embodiment of the adhering matter detection device 10 to which the above-described adhering matter detection method is applied will be described below in more detail.

FIG. 2 is a block diagram of the automatic parking system 1 according to this embodiment. In addition, in FIG. 2, only the components necessary for explaining the features of the present embodiment are represented by functional blocks, and the description of general components is omitted.

In other words, each component illustrated in FIG. 2 is functionally conceptual and does not necessarily need to be physically configured as illustrated. For example, the specific forms of distribution and integration of each functional block are not limited to those shown in the figure, and all or part of them can be functionally or physically distributed in arbitrary units according to various loads and usage conditions.・It is possible to integrate and configure.

As shown in FIG. 2, the automatic parking system 1 includes a camera 2, a parking control device 3, and an adhering matter detection device 10. As shown in FIG.

Since the camera 2 has already been described, its description is omitted here. The parking control device 3 automatically controls the parking of the vehicle based on the image captured by the camera 2 when, for example, an empty parking space is recognized in the parking lot. Further, the parking control device 3 stops the parking control of the vehicle for safety when the determination unit 11c confirms the adhesion of the deposit and the instruction unit 11d instructs the position of the deposit.

The adhering matter detection device 10 includes a control section 11 and a storage section 12 . The control unit 11 includes an acquisition unit 11a, a detection unit 11b, a determination unit 11c, and an instruction unit 11d. The detection unit 11b includes an extraction unit 11ba and an exclusion unit 11bb.

The storage unit 12 is a storage device such as a hard disk drive, a nonvolatile memory, or a register, and stores template information 12a, condition information 12b, and detection information 12c.

The control unit 11 controls the attached matter detection device 10 as a whole. The acquiring unit 11a acquires a camera image frame by frame from the camera 2, and performs preprocessing necessary for image analysis.

Acquisition unit 11a grayscales a camera image, for example, as preprocessing. It should be noted that grayscaling is a process of converting each pixel in a camera image so as to be expressed in each gradation from white to black according to luminance. Such grayscaling may be omitted. In addition, as another preprocessing, the acquisition unit 11a changes the size of the camera image to a predetermined size, for example. The acquisition unit 11a also outputs the preprocessed camera image to the detection unit 11b.

Based on a predetermined detection algorithm, the detection unit 11b detects the detection area Da in which the adhering matter adheres in the camera image, and notifies the determination unit 11c.

The extraction unit 11ba extracts a detection area Da in which an adhering matter is estimated to exist from the camera image using the detection algorithm described above, and notifies the extraction unit 11bb of the extracted detection area Da.

Here, an example of the detection algorithm executed by the extraction unit 11ba will be specifically described with reference to FIGS. 3 to 6. FIG. FIG. 3 is a diagram showing a vector calculation method. 4A and 4B are explanatory diagrams (part 1) and (part 2) of the method of calculating the representative value.

Also, FIG. 5A is a diagram showing an example of a template. FIG. 5B is a diagram showing an example of matching processing. Moreover, FIG. 6 is a figure which shows an example of an extraction process.

The extraction unit 11ba extracts edge information of each pixel in the camera image by using, for example, a Sobel filter for the camera image input from the acquisition unit 11a. Here, edge information refers to the edge intensity of each pixel in the X-axis direction and the Y-axis direction.

Note that the extraction unit 11ba may use another edge extraction method such as a Laplacian filter instead of the Sobel filter.

The extraction unit 11ba also calculates the vector of the edge of each pixel based on the extracted edge strength in the X-axis direction and the Y-axis direction of each pixel. Specifically, as shown in FIG. 3, the extraction unit 11ba calculates the vector of each pixel using a trigonometric function based on the edge strength in the X-axis direction and the Y-axis direction.

In the following description, the angle θ between the calculated vector shown in FIG. 3 and the X-axis in the positive direction is referred to as "edge direction", and the length L of the vector is referred to as "edge strength" of each pixel.

In addition, the extraction unit 11ba does not need to calculate the edge orientation for all pixels, and may simplify the process by calculating the edge orientation for each pixel at a predetermined interval for regions with low priority. .

The extraction unit 11ba also encodes the calculated edge directions. For example, the extraction unit 11ba obtains a representative value of edges in a plurality of pixels and encodes the representative value. Specifically, as shown in FIG. 4A, the extraction unit 11ba handles, for example, a pixel unit of 8×8 pixels as one “cell” and a pixel unit of 3×3 cells as one “block”. Also, the center cell of one block is treated as a "target cell".

The extraction unit 11ba creates a histogram indicating the edge direction and edge strength of each pixel for each block. Such a histogram will be explained using FIG. 4B. Here, the extracting unit 11ba derives the edge direction of the coordinates of the center of the cell of interest from the histogram of the block.

After deriving the representative value of the cell of interest in one block, the extraction unit 11ba creates a histogram by shifting the block by one cell, and calculates the representative value of the cell of interest in the block.

That is, the extraction unit 11ba can reduce the amount of data by calculating the representative value for each of a plurality of pixels. In addition, in the example shown in FIG. 4A, since the cell is 8×8 pixels, the data amount is reduced to 1/64.

Note that the numbers of pixels in the blocks and cells shown in FIG. 4A are examples, and the numbers of pixels in the cells and blocks can be set arbitrarily. At this time, the number of pixels in each cell can be changed according to the size of the water droplet to be detected.

For example, to detect small water droplets, the number of cell pixels is set to be small, and to detect large water droplets, the number of cell pixels is set to be large. As a result, it is possible to efficiently detect water droplets of a size desired to be detected.

Alternatively, the extraction unit 11ba may simply create a histogram for each cell and calculate the representative value of each cell based on the histogram. Note that the extraction unit 11ba may encode all the pixels without calculating the representative value.

Next, the histogram will be explained using FIG. 4B. In FIG. 4B, the vertical axis indicates the edge strength, and the horizontal axis indicates the class of the edge direction. As shown in FIG. 4B , the extraction unit 11ba creates a histogram by assigning edge orientations to 18-level classes every 20°, for example.

Specifically, the extraction unit 11ba creates a histogram for the block by adding the edge intensity of each pixel in the block to the class corresponding to the edge direction. Subsequently, the extraction unit 11ba obtains a class that maximizes the sum of edge strengths from the created histogram.

The example shown in FIG. 4B shows the case where the class of 80 to 100° takes the maximum value. At this time, if the sum of edge strengths in this class is equal to or greater than the threshold, the extraction unit 11ba takes this class as the representative value.

In the example shown in FIG. 4B, the sum of the edge strengths exceeds the threshold in the class of 80 to 100 degrees, so the above conditions are met. Therefore, the class of the cell of interest in such a block is 80 to 100 degrees.

Subsequently, the extraction unit 11ba converts the cell of interest into a code assigned according to the class. Here, 18 types of codes 0 to 9 and A to H, for example, are assigned to each class. 0 to 9 and A to H are codes assigned to each class from 0° to 360° in 20° increments. Also, when the representative value does not exceed the threshold, that is, the cell with low edge strength is assigned a code of Z, for example.

In this manner, the extraction unit 11ba encodes all cells. As a result, each code is arranged in a grid pattern in the encoded camera image. Note that the extraction unit 11ba may calculate the representative value using another statistical calculation method other than calculating the representative value described above.

Also, FIG. 4B describes the case where the edge orientations are classified into 18 levels, but the invention is not limited to this, and the number may be less or more than 18 levels. In addition, although FIG. 4B shows the case where the codes are A to H and Z, other characters such as hiragana and numbers, or graphics may be used as the codes.

Subsequently, the extraction unit 11ba performs matching processing using a regular expression between the encoded camera image and the code pattern indicating the characteristics of water droplets. Here, a regular expression represents a set of code strings with one code.

A template, which is a code pattern indicating the characteristics of water droplets, is stored in the storage unit 12 as template information 12a. An example of such a template is shown in FIG. 5A. In addition, in FIG. 5A, for the sake of visual clarity, the actual edge orientation is schematically illustrated using pin-shaped symbols instead of the above symbols for the templates.

As shown in FIG. 5A, the template information 12a has a code pattern, which is a code string representing the characteristics of water droplets, as a template. Specifically, for example, it has an upper side pattern, a lower side pattern, a left side pattern, and a right side pattern.

Here, the pattern of each side shown in FIG. 5A corresponds to each side of a rectangle that contains water droplets or is contained inside water droplets. Here, "water droplets are contained inside" includes the case where water droplets are inscribed in a rectangle. Also, "included inside the water droplet" includes the case where the rectangle is inscribed in the water droplet. In addition, in this embodiment, a "rectangle" includes a square. Also, FIG. 5A shows the case where the edge direction of the pattern on each side is directed toward the center. In this case, the brightness of the droplet increases from the edge to the center, ie, it exhibits the characteristics of a droplet that is bright in the center and dark at the edges.

Conversely, the template information 12a may have a pattern on each side showing the characteristics of water droplets in which the brightness of the water droplets increases from the center to the ends, that is, the water droplets are dark at the center and bright at the ends. good. By doing so, it is possible to detect various water droplets.

In addition, in the same figure, although the pattern of four directions of up-and-down and right-and-left was illustrated, the pattern including an oblique direction can also be used. By doing so, it is possible to improve the detection accuracy of water droplets.

FIG. 5B shows an example of matching processing using the pattern of each side of the extraction unit 11ba. Here, for convenience of explanation, the upper side patterns shown in FIG. 5A are indicated using symbols A to F, respectively. Parts of the encoded camera image are schematically shown in a and b of FIG.

As shown in FIG. 5B, the extracting unit 11ba determines that the code pattern matches the upper side pattern if the code patterns are arranged in the order of A to F in good order.

Specifically, as shown in a of FIG. 5B, the extraction unit 11ba repeats, for example, A three times, B, C, D and E twice, and F three times. If the arrangement satisfies the arrangement order of each code of the upper side pattern, it is extracted as the upper side pattern.

This is because the number of repetitions of the code differs depending on the size of the water droplet. That is, the larger the water droplet, the longer the length of each code string. In this way, by permitting the repetition of such codes, it is possible to extract a code string indicating a plurality of water droplets of different sizes in a single matching process.

Therefore, it is possible to detect water droplets while reducing the processing load. Note that the extraction unit 11ba may prepare a plurality of patterns for each side having code strings of different lengths according to the size of the water droplet, and extract the code strings using all such patterns.

Also, since water droplets are generally spherical, the number of repetitions of each code should be line symmetrical from the center. For this reason, the extraction unit 11ba excludes poorly balanced code strings from the extracted code strings.

Specifically, as shown in b of FIG. 5B , the extraction unit 11ba scrutinizes the balance between A and F positioned at both ends, for example. Here, in the figure, A is repeated three times and F is repeated ten times.

At this time, when the numbers of A and F differ by two times or more, the extraction unit 11ba excludes such a code string pattern even if the arrangement order is satisfied. By doing so, erroneous extraction of unnecessary code patterns other than water droplets can be prevented, and erroneous detection of water droplets can be suppressed.

Moreover, in the extraction unit 11ba, for example, when an extracted code string is longer than a threshold, such a code string can be excluded from matching. This is because a long code string is less likely to be a water droplet. As a result, erroneous detection of water droplets can be suppressed. It should be noted that such a threshold is assumed to derive an optimum value in advance from statistics or the like.

Next, FIG. 6 shows an example of extraction processing of the detection area Da through matching processing. In FIG. 6, like FIG. 5A, actual edge orientations are schematically illustrated instead of reference numerals.

Also, here, the case where the upper side pattern is first extracted by the matching process will be described. The extraction unit 11ba first sets a substantially square-shaped detection area Da1 based on the extracted width of the upper side pattern.

Next, it is assumed that the right side pattern is extracted at a position deviating from the detection area Da1. At this time, if the center coordinates of the detection area Da2 of the right-side pattern are within the detection area Da1, the extraction unit 11ba performs a process of integrating both detection areas Da1 and Da2.

After that, for example, when the lower side pattern or the left side pattern is extracted in the integrated detection area Da3, the extraction unit 11ba extracts the detection area Da3 as an area in which water droplets are estimated to exist. In other words, the extracting unit 11ba extracts the detection area Da in which water droplets are estimated to exist under the detection condition that a pattern indicating each side in three or more different directions is extracted (hereinafter referred to as "direction condition"). do.

In addition to this direction condition, the extracting unit 11ba may also set a detection condition (e.g., a detection condition that a pattern indicating each side is extracted a predetermined number of times (for example, four times including up, down, left, and right) in the integrated detection area Da3). hereinafter referred to as "number of times condition").

In this way, by setting direction conditions of three or more directions and number of times conditions as detection conditions, it is possible to extract the detection area Da in which water droplets are estimated to exist even if the patterns of all the top, bottom, left, and right sides are not extracted. can be done. That is, it is possible to detect, for example, semicircular water droplets that are not visible from the camera image. These detection conditions are stored in the storage unit 12 as condition information 12b.

Note that FIG. 6 shows the case where the detection areas Da are integrated when the center coordinates of the detection areas Da2 fall within the detection area Da1 of the top pattern, but the present invention is not limited to this. That is, if at least a part of detection area Da1 and detection area Da2 overlap, both may be integrated.

Further, the integrated detection area Da3 may be the logical product or the logical sum of the detection areas Da1 and Da2. In addition, although FIG. 6 shows the case where the detection area Da1 and the detection area Da2 are rectangular, the shape is not limited to this, and other shapes such as a circular shape may be used.

Returning to the description of FIG. 2, the exclusion unit 11bb will be described. The exclusion unit 11bb performs image analysis on each of the detection areas Da notified from the extraction unit 11ba, and determines whether or not the adhering matter presumed to exist in the detection area Da is really an adhering matter.

In addition, the exclusion unit 11bb notifies the determination unit 11c of the detection area Da that is determined to be adhering matter based on the determination result. On the other hand, the exclusion unit 11bb does not notify the determination unit 11c about the detection area Da that is determined not to be an adhering matter based on the determination result, and excludes it from subsequent processing targets. That is, the exclusion unit 11bb excludes the detection area Da with low reliability. By excluding useless image areas in this way, the accuracy of adhering matter detection can be improved, and the processing load in the subsequent stage can be reduced.

For example, the exclusion unit 11bb generates a histogram classified into three classes of "weak", "middle", and "strong" for each of the edge intensity, luminance and saturation of the detection area Da. Then, the exclusion unit 11bb determines whether or not there is an adhering matter based on the frequency ratio of each class in each generated histogram, and excludes the detection area Da determined as not being an adhering matter.

The determination unit 11c manages the state transition between frames of each detection area Da notified from the exclusion unit 11bb of the detection unit 11b, and determines the "determination of adherence" of the adhering matter in each detection area Da according to the state transition. judge.

Here, the determination processing executed by the determination section 11c will be specifically described with reference to FIGS. 7A to 8D. FIG. 7A is a diagram showing an example of notification contents from the detection unit 11b. Further, FIG. 7B is a diagram showing an example of data contents regarding the detection area Da included in the detection information 12c. Moreover, FIG. 7C is an explanatory diagram of the state of the detection area Da. 8A to 8D are explanatory diagrams (part 1) to (part 4) of processing of the determination unit 11c.

As shown in FIG. 7A, for example, upper left coordinates (x, y), width w, and height h and are included.

The determination unit 11c determines whether or not all of the detection areas Da extracted in the current frame as having attached matter overlap with the detection areas Da already extracted in the previous frame. Further, the determination unit 11c reflects the determination result on the "score" and "state" of each detection area Da. The reflection result is managed by the detection information 12c.

Here, the detection information 12c includes, for example, a "detection area ID" item, an "area information" item, a "score" item, a "state" item, and a "mitigation" item, as shown in FIG. 7B. . The "detection area ID" item stores the identifier of the detection area Da, and the detection information 12c is managed for each detection area ID.

The "area information" item stores the upper left coordinates (x, y), width w, height h, etc. of the detection area Da shown in FIG. 7A. The "score" item stores the current score of each detection area Da. The "state" item stores the current state of each detection area Da.

As shown as a state machine diagram in FIG. 7C, each detection area Da can transition to four states: "IDLE", "Hide", "Observe", and "Penalty". "IDLE" refers to an "undetected state", ie, a state in which no deposit is attached. "Latent" refers to the state of "possibility of attachment of deposits".

"Observation" refers to "after removal processing" in which an operation of removing deposits by the deposit removing device is performed when the vehicle is equipped with a deposit removing device capable of removing deposits by using compressed air, a wiper, or the like, for example. "observation state". A deposit removing device will be described later in a second embodiment. "Penalty" refers to "a state in which the adhering matter continues to be detected in the area even after removal processing", that is, a state of defective removal or erroneous detection.

The determination unit 11c updates the "score" of each detection area Da on the detection information 12c according to the determination result of whether or not the detection areas Da overlap between frames, and changes the "state". Then, the determination unit 11c determines whether the adhering matter is "determined" according to the "state" and "score" of the detection area Da of the detection information 12c. More specifically, it will be described later with reference to FIGS. 8A to 8D.

Returning to FIG. 7B, the “Relief” item will now be described. The “relaxation” item stores a flag value indicating whether or not the current detection conditions for each detection area Da should be relaxed. In the example of FIG. 7B, the detection area Da with the detection area ID “xx2” with a check mark in the “relaxed” item should be extracted under the second condition (see FIG. 1B) that is more relaxed than the first condition. is shown. The extraction unit 11ba of the detection unit 11b described above extracts the detection area Da with the detection area ID "xx2" using the second condition instead of the first condition. This point will also be described later with reference to FIG.

Next, the determination processing executed by the determination unit 11c will be described more specifically. As already described above, as shown in FIG. 8A, the determination unit 11c determines whether all the detection areas Da extracted in the current frame as having attached matter are detected as detection areas Da already extracted in the past frame. It is determined whether or not there is an overlap of .

Specifically, as shown in FIG. 8B, it is determined whether all of the detection areas Da-C of the current frame overlap with all of the detection areas Da-P of the past frame. For example, the overlap between detection area Da-C and detection area Da-P is determined based on the distance between their centroids.

Then, as shown in FIG. 8B, when the determination unit 11c determines that there is an overlap between the detection area Da-C of the current frame and the detection area Da-P of the previous frame, the detection area Da-C and A point is added to the score of each of the detection areas Da-P.

As a result, the determination unit 11c can indicate, for example, deposits that are present in substantially the same area of the lens even across the frame. Note that the added point is, for example, "+20" added to the score.

On the other hand, as shown in FIG. 8C, the determination unit 11c deducts the points of the detection areas Da-P of the past frame that do not overlap any of the detection areas Da-C of the current frame. It should be noted that, for example, "-10" is subtracted from the score.

Further, the determination unit 11c regards detection areas Da-C of the current frame that do not overlap with any of the detection areas Da-P of the past frames as new detection areas Da, and adds them to the detection information 12c. register.

As shown in FIG. 8D, the newly registered detection area Da is in a "latent" state and given a predetermined score. Then, the determination unit 11c causes the state of the detection area Da to transition from the "hidden" state according to the score of the detection area Da that changes through the above-described "addition" and "deduction".

For example, as shown in FIG. 8D, when the score of the detection area Da in the "hiding" state becomes equal to or less than a predetermined point, the determination unit 11c changes the state of the detection area Da from "hiding" to "IDLE". (step S11). As a result, it is possible to prevent an erroneous reaction to an adhering matter, such as a raindrop, which moves by running down, for example, and does not need to be determined as an adhering matter.

Further, when the score of the detection area Da in the "latent" state is equal to or higher than a predetermined point, the determination unit 11c confirms adhesion of the adhering matter to the area (adhesion confirmation) (step S12). For example, when the score is "100" or more, the determination unit 11c confirms attachment of the area.

Further, after determination of adhesion, the determination unit 11c causes all the detection areas Da in the "latent" state to transition to the "observation" state (step S13). This is because, if removal processing is performed by the above-described adhering matter removing device in response to confirmation of adherence of one detection area Da, other detection areas Da in the "latent" state for which adherence has not been confirmed will also normally be removed. This is because it is presumed that the adhering matter has been removed if so.

Note that when the score of the detection area Da in the "observation" state becomes a predetermined point or more due to the removal processing by the adhering substance removal device, the determination unit 11c shifts the detection area Da to the "penalty" state ( step S14). As a result, removal failure or erroneous detection in which the adhering matter continues to be detected even after removal processing can be grasped.

When the score of the detection area Da in the "observation" state or the "penalty" state becomes equal to or less than a predetermined point, the determination unit 11c transitions the detection area Da to the "IDLE" state (step S15).

It should be noted that the reaction speed up to determination of adhesion may be controlled by adjusting the inclination of the arrows indicating "additional points" and "deductive points" in FIG. 8D. For example, by increasing the amount of added points and the amount of deducted points to steepen the slope of the arrow, it is possible to increase the reaction speed until adhesion is determined.

Further, when there is a detection area Da for which adherence is confirmed, the determination unit 11c causes the instruction unit 11d to instruct the parking control device 3 about the position of the detection area Da.

In addition, the determination unit 11c sets the "relief" item of the detection information 12c to indicate that the detection condition should be relaxed for the detection area Da that is extracted at least in the current frame, is in the "latent" state, and has not reached the point where adhesion is confirmed. The flag value shown is stored (see FIG. 7B). Further, the determination unit 11c notifies the above-described detection unit 11b of the detection area Da that relaxes the detection conditions. In other words, the determination unit 11c feeds back the detection area Da once detected as having an adhering matter to the detection unit 11b so that it is passed on to subsequent frames in terms of time (see FIG. 1B).

Based on the detection information 12c, the detection unit 11b relaxes the detection conditions and performs detection processing for the detection area Da fed back from the determination unit 11c. FIG. 9 is a process explanatory diagram of the detection unit 11b after feedback.

FIG. 9 corresponds to the detection area Da3 shown at the bottom in FIG. 6. As shown in the left diagram of FIG. 9, in FIG. , the detection area Da3 is extracted as an area in which the presence of water droplets is estimated when the lower side pattern or the left side pattern is extracted. In other words, the extracting unit 11ba extracts the detection area Da in which the presence of water droplets is estimated under the directional condition in which the pattern indicating each side in three or more different directions is extracted. The directional condition in such a case is the "first condition" (see FIG. 1B).

On the other hand, as shown in the right diagram of FIG. 9, after the feedback described above, the extraction unit 11ba of the detection unit 11b extracts patterns indicating sides in two or more different directions, such as an upper side pattern and a right side pattern. A detection area Da in which the presence of a water droplet is estimated is extracted under the directional condition. That is, the detection unit 11b after the feedback extracts the detection area Da under the "second condition" (see FIG. 1B) that is "loosen" than the first condition. For example, the detection unit 11b extracts the detection area Da using the "second condition" included in the condition information 12b for the detection area Da flagged in the "relief" item of the detection information 12c.

This can eliminate the instability of detection when the adhering matter is water droplets or the like. Therefore, according to the adhering matter detection device 10 according to the present embodiment, it is possible to suppress detection failure of adhering matter.

Further, according to the adhering matter detection apparatus 10 according to the present embodiment, resources such as the detection area Da once detected can be effectively used in subsequent processing, so that the processing load of the entire system can be reduced. , improvement in processing speed and detection accuracy can be expected.

Returning to the description of FIG. 2, the instruction unit 11d will be described. The instruction unit 11 d generates an instruction signal for instructing the position of the adhered matter to the parking control device 3 and outputs the instruction signal to the parking control device 3 when the determination unit 11 c confirms the adhering matter.

Next, a processing procedure executed by the adhering matter detection device 10 according to the present embodiment will be described with reference to FIG. 10 . FIG. 10 is a flow chart showing a processing procedure executed by the adhering matter detection device 10 according to the present embodiment.

First, the acquisition unit 11a acquires a camera image for one frame (step S101). Then, the extraction unit 11ba of the detection unit 11b extracts a detection area in which the presence of water droplets is estimated (Step S102).

Then, the exclusion unit 11bb of the detection unit 11b excludes the detection area with low reliability (step S103).

Note that the exclusion process itself may be omitted, for example. Thereby, the processing load on the entire system can be reduced.

Subsequently, the determination unit 11c determines the state of the detection area between frames (step S104). That is, the determination unit 11c determines whether there is an overlap with each detection area Da extracted in the past frame, and updates the score and state of the detection area Da according to the determination result.

Then, the determination unit 11c determines whether or not adhesion is confirmed for each detection area Da according to the updated score and state of the detection area Da (step S105). Here, if the adhesion is not confirmed (step S105, No), the determination unit 11c relaxes the detection condition by setting a flag for the "relief" item of the detection information 12c for the corresponding detection area Da (step S106). . Then, the processing from step S101 is repeated.

On the other hand, if the adherence is confirmed (step S105, Yes), the instructing unit 11d instructs the parking control device 3 about the position of the adherent (step S107).

Then, the control unit 11 determines whether or not there is a process end event (step S108). The processing end event corresponds to, for example, IG off or ACC off of the vehicle. Here, if it is determined that there is no processing end event (step S108, No), the processing from step S101 is repeated. If it is determined that there is a process end event (step S108, Yes), the adhering matter detection device 10 ends the process.

Next, a modified example of the flow of processing according to this embodiment will be described with reference to FIGS. 11A to 11C. FIG. 11A is a diagram showing the flow of processing according to this embodiment. FIG. 11B is a diagram showing the flow of processing according to the first modification. FIG. 11C is a diagram showing the flow of processing according to the second modification.

Until now, as shown in FIG. 11A, the attached matter detection apparatus 10 has the acquisition unit 11a perform acquisition processing based on the camera image of the camera 2, the extraction unit 11ba of the detection unit 11b performs extraction processing, A case has been described where the exclusion unit 11bb of the detection unit 11b executes the exclusion process and the determination unit 11c executes the determination process. In addition, a case will be described where the determination unit 11c feeds back the detection area Da, which is in the "latent" state in the above example, before the adherence is confirmed, so as to be passed on to the later extraction process while relaxing the detection conditions. bottom.

However, the flow of processing is not limited to this example. For example, as shown in FIG. 11B, for the detection area Da fed back from the determination process to the extraction process, the detection area Da for which the exclusion process has been cleared once, the process flow is controlled so that the second and subsequent exclusion processes are omitted. You may

This is because the detection area Da that has cleared the exclusion process at least once is considered not to have low reliability. As a result, the processing load required for the exclusion process can be suppressed, and the processing load of the entire system can be reduced.

Further, for example, as shown in FIG. 11C, the detection area Da detected through the extraction process and the exclusion process is not fed back to the extraction process, and the process flow is adjusted so that the loop is closed between the exclusion process and the determination process. may be controlled.

This is based on the idea that the detection area Da, which has been detected at least once through the extraction process and the exclusion process, will not disappear immediately in terms of time. That is, in this case, the once detected detection area Da is not extracted again using the detection algorithm in the extraction process, and the state of the detection area Da is managed by repeating the exclusion process and the determination process. As a result, the processing load required for the extraction process can be suppressed, and the processing load of the entire system can be reduced.

As described above, the adhering matter detection device 10 according to the present embodiment includes the detection section 11b and the determination section 11c. The detection unit 11b detects the position in the captured image of the adhering matter adhering to the camera 2 (corresponding to an example of the “imaging device”) based on a predetermined first condition. The determination unit 11c determines the presence or absence of the adhering matter at the position based on the information regarding the position detected a plurality of times. Further, when the position is detected by the detection unit 11b, the determination unit 11c causes the detection unit 11b to detect the position based on a second condition that is relaxed compared to the first condition.

Therefore, according to the adhering matter detection device 10 according to the present embodiment, it is possible to suppress detection failure of adhering matter. In addition, since the position detected under the first condition is detected under the second condition, which is more relaxed than the first condition, the detection accuracy is maintained, and the processing load of the entire system is reduced and the processing speed is increased. can contribute to

The determination unit 11c determines the presence or absence of an adhering matter at the position based on the state transition of the position detected a plurality of times.

Therefore, according to the attached matter detection device 10 according to the present embodiment, it is possible to accurately determine the presence or absence of attached matter without unnecessarily confirming attachment of temporarily detected attached matter.

The determination unit 11c adds points to the score indicating the state of the position when the position is detected by the detection unit 11b, and subtracts points from the score when the position is not detected by the detection unit 11b. The state transition is managed, and when the score reaches or exceeds a predetermined point, attachment of the adhering matter to the position is determined.

Therefore, according to the adhering matter detection device 10 according to the present embodiment, it is possible to manage the state transition of the position and determine adherence confirmation according to the state by a simple method.

The detection unit 11b converts each pixel into a predetermined data format based on the edge information extracted from each pixel in the captured image, and outputs each pixel converted into this data format and the data indicating the adhering matter. The position is detected based on the matching result with the format template.

Therefore, according to the adhering matter detection device 10 according to the present embodiment, it is possible to detect various adhering matter according to the template indicating the characteristics of the adhering matter.

The detection unit 11b detects the position as a rectangular detection area Da (corresponding to an example of a “detection area”) that includes the adhering matter or is contained inside the adhering matter, and detects the above position as each side of the detection area Da: The position is detected under the first condition that each side indicating three or more different directions is extracted depending on the template, and the second condition that each side indicating two or more different directions is extracted depending on the template.

Therefore, according to the attached matter detection device 10 according to the present embodiment, the attached matter can be detected by a simple method by using the rectangular detection area Da. Further, since the detection area Da detected under the first condition is subsequently detected under the second condition, which is more relaxed than the first condition, the detection accuracy is maintained, the processing load of the entire system is reduced, and the processing speed is increased. It can contribute to speeding up.

In addition, the camera 2 is mounted on a vehicle, and the vehicle is provided with a parking control device 3 (equivalent to an example of a "parking control unit") that performs parking control of the vehicle based on an image captured by the camera 2. The parking control device 3 performs determination Parking control of the vehicle is stopped when the unit 11c confirms that the adhering matter is adhered to the above position. Therefore, according to the automatic parking system 1 according to the present embodiment, an automatic parking system with high safety can be realized.

In the first embodiment described above, the case where the extraction unit 11ba extracts the detection area Da using a predetermined detection algorithm based on the edge intensity of each pixel was taken as an example. Algorithms may be executed in parallel. Detection algorithms based on factors other than edge intensity include, for example, extracting the gradient of luminance and gradient of saturation of each pixel as edge information, and extracting the detection area Da based on this.

When the extraction unit 11ba executes the detection algorithms in parallel, the exclusion unit 11bb executes the exclusion process for each processing result of the detection algorithm. Also, the determination unit 11c preferably determines overlap between algorithms before determining overlap between frames.

(Second embodiment)
Next, a second embodiment will be described with reference to FIG. 12. FIG. FIG. 12 is a block diagram of a deposit removing system 1A according to the second embodiment. FIG. 12 corresponds to FIG. 2, and the deposit removing system 1A differs from the automatic parking system 1 in that it includes a deposit removing device 4 instead of the parking control device 3. FIG.

The deposit removing device 4 removes deposits adhering to the lens of the camera 2 based on the detection result of the deposit detecting device 10 . For example, the deposit remover 4 has a nozzle (not shown) with an ejection opening directed toward the lens of the camera 2, and removes deposits by ejecting compressed air from the nozzle toward the lens. Note that the attached matter removing device 4 is not limited to such a mode, and may, for example, spray cleaning liquid together with compressed air from a nozzle, or may wipe the lens of the camera 2 with a wiper.

In the attached matter removing system 1A, the determination unit 11c instructs the instructing unit 11d to perform the removing operation by the attached matter removing device 4 when there is a detection area Da for which adhesion is confirmed. Note that the determination unit 11c may dare not perform the removing operation even if there is a detection area Da for which adherence is confirmed.

For example, the determination unit 11c determines that execution of the removal process is not necessary if the detection area Da for which adhesion has been confirmed exists in an area substantially along the outer periphery of the screen that is unlikely to hinder the viewing of the camera image. be able to. Further, for example, the determination unit 11c can determine that execution of the removal process is unnecessary if the number of detection areas Da for which adherence is confirmed is less than a predetermined number.

In this way, the processing load of the entire system can be reduced by skipping the removal processing itself for attached matter that is judged to have little effect on the passenger's visual recognition and driving behavior.

Further, when the determination unit 11c determines that the removal of the attached matter is necessary, the instructing unit 11d generates an instruction signal for causing the attached matter removing device 4 to perform a removing operation, and the instructing unit 11d generates the instruction signal to perform the removing operation of the attached matter removal device. Output to device 4 . That is, taking FIG. 10 already shown as an example, in the second embodiment, the instructing unit 11d does not "instruct the position of the adhering matter" in step S107, but instructs the adhering matter removing device 4 to perform the "removal operation." "instruct". The deposit removing device 4 performs a removing operation based on such an instruction to remove the deposit.

Therefore, according to the attached matter removing system 1A according to the present embodiment, by including the attached matter detection device 10, it is possible to detect the attached matter while suppressing detection omissions, and the detected attached matter can be detected by the attached matter removing device. 4 can be removed.

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments so shown and described. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

1 automatic parking system 2 camera 3 parking control device 4 adhering substance removal device 10 adhering matter detection device 11 control unit 11a acquisition unit 11b detection unit 11ba extraction unit 11bb exclusion unit 11c determination unit 11d instruction unit 12 storage unit 12a template information 12b condition information 12c detection information

Claims (6)

a detection unit that detects the position in the captured image of the adhering matter adhering to the imaging device based on a predetermined first condition;
a determination unit that determines the presence or absence of the attached matter at the position based on information about the position detected a plurality of times with respect to the captured images arranged in time series ,
The determination unit is
An adhering matter detection device, wherein when the position is detected by the detection unit, the detection unit detects the position based on a second condition that is relaxed compared to the first condition. The determination unit is
2. The adhering matter detection device according to claim 1, wherein presence or absence of the adhering matter at the position is determined based on the state transition of the position detected a plurality of times. The determination unit is
The state transition is managed by adding points to a score indicating the state of the position when the position is detected by the detection unit, and subtracting points from the score when the position is not detected by the detection unit. 3. The adhering matter detection device according to claim 2, further comprising determining attachment of the adhering matter at the position when the score reaches or exceeds a predetermined point. The detection unit is
The attachment, which converts each pixel into a predetermined data format based on the edge information extracted for each pixel in the captured image, and shows the characteristics of each pixel converted into the data format and the adhering matter. Based on the result of matching with the template in the data format as a pattern of pixels corresponding to each side of a rectangle containing the object inside or a rectangle containing the object inside, a rectangle containing the object inside is determined . The position is detected as a detection area or a rectangular detection area included inside the adhering matter, and out of the sides of the detection area, the sides indicating three or more different directions depending on the template are extracted. 4. The attachment according to claim 1, 2, or 3, wherein the position is detected under the first condition and the second condition that each side indicating two or more different directions is extracted by the template. Kimono detection device. The imaging device is mounted on a vehicle,
The vehicle includes a parking control unit that performs parking control of the vehicle based on the captured image,
The deposit according to claim 3 or 4 , wherein the parking control unit stops parking control of the vehicle when the determination unit determines that the deposit is adhered at the position. detection device. a detection step of detecting the position in the captured image of the adhering matter adhering to the imaging device based on a predetermined first condition;
a determination step of determining the presence or absence of the attached matter at the position based on information about the position detected a plurality of times with respect to the captured images arranged in time series ,
The determination step includes
An adhering matter detection method, wherein when the position is detected by the detection step, the detection step detects the position based on a second condition that is relaxed compared to the first condition.

Images