WO2018150661A1 - Onboard image-capture device - Google Patents

Abstract

Provided is an onboard image-capture device including a lens dirt elimination diagnosis device with which the probability of an erroneous determination can be reduced. The onboard image-capture device is provided with: an onboard camera for capturing an image via a lens; a dirt sensing unit which, on the basis of captured images sequentially captured by means of the onboard camera, detects the presence or absence of dirt or a dirt attached area on the lens; a stable state determination unit which, on the basis of a temporal change in the captured images sequentially captured by the onboard camera, determines the presence or absence of a stable state; an eliminated area extraction unit which extracts a dirt eliminated area on the basis of the temporal change in captured images, among the captured images sequentially captured by the onboard camera, when the result of determination by the stable state determination unit indicates a stable state; and a dirt state management unit in which the presence or absence of dirt or the dirt attached area detected by the dirt sensing unit is stored, and which manages dirt on the lens by reflecting the dirt eliminated area extracted by the eliminated area extraction unit on the presence or absence of dirt or the dirt attached area that is stored.

Description

In-vehicle imaging device

The present invention relates to an in-vehicle image pickup apparatus that is mounted on a car body of an automobile and can image the outside of the automobile.

In recent years, an in-vehicle camera mounted on the body of an automobile is used to image at least one of the front, rear, and side directions of the host vehicle, or the entire peripheral direction, and the captured image is displayed on a monitor screen in the vehicle. In-vehicle imaging devices are becoming widespread. There is also an alarm device that detects other vehicles and pedestrians approaching the host vehicle from the captured image of the in-vehicle camera and notifies the driver.

By the way, except for some parts, the in-vehicle camera is attached to the outside of the vehicle body. Therefore, dirt such as raindrops and mud tends to adhere to the surface of the lens (including lens protection glass) of the in-vehicle camera. If dirt adheres to the lens surface, the dirt is reflected in the captured image of the in-vehicle camera, and the scenery is shielded.

Therefore, in order to notify the user that the lens of the in-vehicle camera is dirty, a technique for detecting the dirt attached to the lens surface of the in-vehicle camera from the captured images sequentially captured by the in-vehicle camera is disclosed (for example, JP, 2012-38048, A).

Conventionally, a lens dirt removing device (washer, blower) that blows water or air onto the lens surface of an in-vehicle camera and removes dirt adhering to the lens surface is known. Japanese Patent Laid-Open No. 2014-11785 discloses this A lens dirt removal diagnostic technique for diagnosing whether the lens dirt removal apparatus has functioned normally is disclosed. In this technique, before and after the operation of the lens dirt removing device, the captured images of the in-vehicle camera are compared, and it is diagnosed that dirt has been removed from changes in contrast and edge strength.

However, there are many in-vehicle cameras that do not have a lens dirt removal device, and in this case, the user manually wipes the lens surface of the in-vehicle camera with a cloth or finger to remove the dirt on the lens surface. Since the timing of performing the dirt removal work is unknown, it is impossible to diagnose the dirt removal by comparing the captured images before and after the dirt removal work as disclosed in Japanese Patent Application Laid-Open No. 2014-11785.

Correspondingly, Japanese Patent Laid-Open No. 2016-15583 discloses a lens dirt removal sequential diagnosis technique as a method capable of diagnosing dirt removal even when the timing of dirt removal work execution is unknown. In this lens dirt removal sequential diagnosis technology, a captured image captured by an in-vehicle camera immediately after the host vehicle stops is used as a reference image, and a difference image between the reference image and a captured image captured by the in-vehicle camera thereafter is sequentially determined. The difference image is generated and compared with a previously detected dirt adhesion region, and if a difference is detected in the majority of the dirt adhesion region, it is diagnosed that the dirt has been wiped away.

JP 2012-38048 A JP 2014-11785 A Japanese Unexamined Patent Publication No. 2016-15583

However, if the lens surface of the in-vehicle camera is manually cleaned, the cleaner will be reflected closer to the camera, causing a large change in the entire image. For this reason, in the method of detecting the wiping of the dirt from the time change of the picked-up image as in JP-A-2016-15583, there is a concern that the change due to the reflection of the cleaner is erroneously determined to be due to the removal of the dirt. .

In addition, when a captured image at the time of stopping is used as a reference image as in Japanese Patent Application Laid-Open No. 2016-15583, the time from when the vehicle is stopped until the lens surface of the in-vehicle camera is cleaned becomes long, and two images that take a difference are taken. It is assumed that the imaging time of is separated. In this case, changes in ambient light and surrounding moving bodies (pedestrians, vehicles, etc.) become large, which also contributes to erroneous determination.

The object of the present invention is to reduce the possibility of erroneous determination when diagnosing that the dirt on the lens surface has been wiped off even when the operation timing of removing the dirt attached to the lens surface of the in-vehicle camera is unknown. Another object of the present invention is to provide an in-vehicle image pickup apparatus having a lens dirt wiping diagnostic apparatus.

In order to achieve the above object, the present invention detects the presence or absence of dirt on the lens or a dirt-attached area based on an in-vehicle camera that captures an image through a lens and captured images sequentially captured by the in-vehicle camera. A stain detection unit; a stain wiping diagnosis unit for diagnosing wiping of dirt on the lens; and a stain state management unit for managing dirt on the lens. The stain wiping diagnosis unit sequentially captures images with the in-vehicle camera. A stable state determination unit that determines whether the captured image is in a stable state based on a time change of the captured image, and a determination result by the stable state determination unit among the captured images sequentially captured by the in-vehicle camera is in a stable state A wiping area extraction unit that extracts a wiping area for dirt based on a temporal change of a captured image at a certain time, and the dirt state management part is configured to detect the presence or absence of dirt detected by the dirt detection unit or the adhesion of dirt. Storing the range, the wiping area of dirt the wiping region extraction unit has extracted, already reflected in the presence or dirt adhering area of the dirt that stores, performs dirt management of said lens, characterized in that.

According to the present invention, it is possible to reduce the possibility of erroneous determination when diagnosing that the dirt on the lens surface has been wiped off even when the operation timing of removing the dirt attached to the lens surface of the in-vehicle camera is unknown. It is possible to provide an in-vehicle imaging device having a lens dirt wiping diagnostic device that can be used.

Issues, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is a block diagram showing a schematic configuration of an in-vehicle imaging device according to Embodiment 1 of the present invention. The figure which shows an example of the captured image of a vehicle-mounted camera. The figure which shows an example of the captured image of a vehicle-mounted camera. The figure which shows an example of the difference image of the picked-up image of a vehicle-mounted camera. The figure which shows an example of the captured image of a vehicle-mounted camera. The figure which shows an example of the difference image of the picked-up image of a vehicle-mounted camera. The figure which shows an example of the flowchart of a process of the stable state determination part. The figure which shows an example of the flowchart of a process of the wiping area extraction part. FIG. 5 is a block diagram illustrating a schematic configuration of an in-vehicle imaging device according to Embodiment 2 of the present invention. The figure which shows an example of the flowchart of a process of the wiping area extraction part. FIG. 6 is a block diagram illustrating a schematic configuration of an in-vehicle imaging device according to Embodiment 3 of the present invention. The figure which shows an example of the captured image of a vehicle-mounted camera. The figure which shows an example of the flowchart of a process of the cleaning state determination part. FIG. 6 is a block diagram illustrating a schematic configuration of an in-vehicle imaging device according to Embodiment 4 of the present invention.

Hereinafter, an in-vehicle imaging device according to the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a schematic configuration of an in-vehicle imaging device according to Embodiment 1 of the present invention.

(Configuration of in-vehicle imaging device 100)
As shown in FIG. 1, the in-vehicle imaging device 100 of the present embodiment includes an in-vehicle camera 1, a dirt wiping diagnosis unit 2, a dirt detection unit 3, and a dirt state management unit 4. The dirt wiping diagnosis unit 2 includes a stable state determination unit 21 and a wiping region extraction unit 22. Further, the dirt wiping diagnosis unit 2, the dirt detection unit 3, and the dirt state management unit 4 are also referred to as a lens dirt wiping diagnosis device (the same applies to the other embodiments below).

The in-vehicle camera 1 is installed in a vehicle (not shown), and takes an image in front of the in-vehicle camera 1 via a lens (including a lens protection glass). The in-vehicle camera 1 sequentially outputs an image (captured image) obtained by imaging the front of the in-vehicle camera 1. For example, when the frame rate is 20 fps, the in-vehicle camera 1 captures an image and outputs the captured image every 50 milliseconds.

The stable state determination unit 21 sequentially acquires captured images output from the in-vehicle camera 1, extracts a change (time change of the captured image) from the captured image acquired immediately before each time, and the magnitude of the change is predetermined. If it is less than the value, it is determined that the captured image is temporally stable (stable state). Conversely, if it exceeds a predetermined value, the captured image is not temporally stable (unstable state). judge.

Note that “change in time of captured image” refers to a difference in comparison between image data of a captured image at a certain time (any type of image data) and image data of a captured image after a certain period of time has elapsed. (Change with time), for example, change of each pixel of the captured image (change with time), and “size of time change of captured image” is, for example, the size of the area of the changed region In some cases, for example, there is a change amount in each pixel of the image data. As an example of “time change of captured image”, “time change of feature amount of captured image” may be used.

In addition, “extraction of time-varying captured image” refers to a region in which a difference is generated as a result of comparison (that is, a region in which a change has occurred over time) among all regions of the captured image (entire image) In some cases, extraction may be performed, and in some cases, a captured image in which a difference is generated as a result of comparison is extracted from a large number of captured images that are sequentially captured.

Hereinafter, even when referring to image data of a captured image, it may be simply referred to as a captured image, and even when referring to image data of a difference image, it may be simply referred to as a difference image.

The stable state determination unit 21 finally outputs the determination result as a stable state signal. It is stable when a moving body (for example, a moving body such as a person who cleans the surface of the lens of the in-vehicle camera 1 or another vehicle) is not shown in the captured image, or when the moving amount of the moving body in the captured image is small. It becomes a state.

The wiping area extraction unit 22 refers to the stable state signal output from the stable state determination unit 21, sequentially acquires captured images output from the in-vehicle camera 1 only in the stable state, and acquires in the previous stable state each time. A change from the captured image is extracted. As a result, a change in the captured image (change in time of the captured image) in a state where the moving body is not reflected in the captured image or the moving amount of the moving body in the captured image is small (stable state) is extracted. Then, an area having a large change in the captured image is output to the subsequent dirt state management unit 4 as an area (wiping area) from which dirt attached to the lens surface of the in-vehicle camera 1 is wiped off.

The dirt detection unit 3 detects and outputs an adhesion area (dirt detection area) of dirt attached to the lens surface of the in-vehicle camera 1 from captured images sequentially output from the in-vehicle camera 1. For this, a known method described in JP 2012-38048 A can be used. For example, there is a method of detecting a region with a small luminance change over a long period from a captured image captured by the in-vehicle camera 1 while the host vehicle is traveling, and setting this region as a stain detection region. Some methods detect only the presence or absence of dirt, not the area where dirt is attached. In this case, the dirt detection unit 3 outputs the strength of the detection reaction (stain detection intensity) corresponding to the dirt adhesion area and density instead of the dirt detection region.

The dirt state management unit 4 refers to the dirt detection area or the dirt detection intensity output from the dirt detection unit 3 while the dirt detection unit 3 is functioning, and adheres to the lens surface of the in-vehicle camera 1. The area is set as a dirty storage area, and the presence / absence of dirt is managed as a dirty state (the dirty storage area and the dirty state are stored in a nonvolatile storage unit (not shown)). Further, the dirt state management unit 4 refers to the wiping area output from the wiping area extraction unit 22 and reflects it in the managed dirt state and dirt storage area. For example, when the wiping area from the wiping area extraction unit 22 exceeds a predetermined area, the dirt state management unit 4 changes the stored dirt state from “dirty” to “no dirt”. For example, the dirt state management unit 4 erases the wiping area from the wiping area extraction unit 22 from the stored dirt storage area.

Further, the dirt state management unit 4 may store the presence or absence of dirt or the size of the area of the dirt storage area as a dirt state.

Furthermore, the dirt state management unit 4 notifies the user of the managed dirt state and dirt storage area (lens dirt management status) with a warning light or a speaker (not shown), or automatically controls the vehicle. It also functions to notify a vehicle control system (not shown) such as a driving system.

(Processing of stable state determination unit 21)
FIG. 2A is an example of a captured image captured by the in-vehicle camera 1 when water droplets adhere to the lens surface of the in-vehicle camera 1. Water drops are reflected in the areas 31 and 32 of the captured image, respectively, and the scenery beyond that is blocked. Further, FIG. 2B shows a captured image in the case where water drops reflected in the region 31 are removed.

FIG. 3 is a difference image obtained by taking the difference between the two captured images of FIGS. 2A and 2B. In the difference image of FIG. 3, only the region 31 from which water droplets have been removed is extracted.

However, when a moving moving body such as a pedestrian or a vehicle is reflected in the captured image of the in-vehicle camera 1, the region where the moving body is reflected is also extracted in the difference image. FIG. 4 is an example of a captured image when a moving pedestrian appears in front of the in-vehicle camera 1. A moving pedestrian is shown in the region 33 of the captured image. 5 shows a difference image between the captured image of FIG. 2A and the captured image of FIG. In the difference image in FIG. 5, not only pedestrians in the region 33 but also water drops in the region 31 are extracted. This is because the pedestrian in the area 33 overlaps the water drop in the area 31 and the brightness of the water drop in the area 31 is changed by the reflected light from the pedestrian.

Therefore, the stable state determination unit 21 detects a state in which the moving body is not reflected on the in-vehicle camera 1. In FIG. 6, an example of the flowchart of the process of the said stable state determination part 21 is shown.

First, captured images sequentially output from the in-vehicle camera 1 are acquired (step 2101), and a difference image (sequential difference image) between the acquired captured image and the captured image acquired immediately before that is generated (step 2102). . That is, the acquired image data of the captured image is compared with the image data of the captured image acquired immediately before, and image data in which only the different portions of the comparison results are left is generated as the image data of the difference image. Next, an area having a difference equal to or larger than a predetermined value in the sequential difference image is extracted as a large difference area (sequential difference area) (step 2103), and an area of the sequential difference area (sequential difference area) is derived (step 2104). ). Note that “large” in the large difference area extracted in step 2103 is large enough to exclude differences other than the difference caused by the movement of the moving object, such as the difference in error due to noise in the image data of the captured image. Therefore, it is desirable to adjust according to the actual machine environment. Further, “extracting a region having a large difference” is extracting a region candidate having a difference caused by the movement of the moving body from the entire region of the captured image.

Then, it is determined whether the successive difference area is equal to or smaller than a predetermined value (step 2105). If the successive difference area is less than or equal to a predetermined value, it is determined that the moving object is not reflected or the movement of the moving object is small (stable state) (step 2106), and the successive difference area exceeds the predetermined value. For example, it is determined that the moving body is reflected (unstable state) (step 2107). “Small” when the movement of the moving body is small means that the movement of the moving body can be regarded as a stable state even if the moving body is reflected, that is, it is not mistaken for wiping dirt. It is desirable to show that the movement is small, and adjust according to the actual machine environment. Finally, the determination result is output as a stable state signal (steps 2106 and 2107). This series of processing is repeated each time a captured image by the in-vehicle camera 1 is input.

(Extended example 1 of processing of the stable state determination unit 21)
By the way, when a stable state signal is generated for each captured image sequentially output from the in-vehicle camera 1, it is determined that the captured image is in a stable state only when the movement of the moving body reflected in the captured image is temporarily delayed. As a result, the steady state signal becomes unstable.

Therefore, it is preferable to stabilize the steady state signal. There are several methods, but any method may be used. For example, in step 2104 of the flowchart of FIG. 6, there is a method of accumulating the successive difference area over a certain period and deriving the accumulated area of the successive difference area as the successive difference area.

Alternatively, in step 2105 of the flowchart of FIG. 6, the average value or maximum value of the successive difference areas is calculated over a certain period, and if the calculated value is smaller than a predetermined value, the stable state signal is set to the stable state, and the calculation is performed. If the value exceeds the predetermined value and is large, a method of making the stable state signal in an unstable state may be used.

In addition, in step 2101 of the flowchart of FIG. 6, there is a method of thinning out and acquiring captured images sequentially output from the in-vehicle camera 1. By thinning out the captured image, it becomes easy to capture the movement of the moving body, and the stable state signal can be stabilized.

(Extended example 2 of processing of the stable state determination unit 21)
By the way, the stable state determination unit 21 extracts a feature amount such as an edge image or a HOG (Histogram of Oriented Gradients) feature amount from the captured image without using the captured image of the vehicle-mounted camera 1 as it is (here, the extraction result) Is called a feature amount image), and this feature amount image may be used instead of the captured image. By using an appropriate feature amount, the detection performance can be improved and the amount of data to be processed can be reduced. For example, when the HOG feature amount is used, it is possible to make it difficult to extract unnecessary image changes due to slight fluctuations of the moving body and ambient light. In addition, if an average value such as luminance or edge strength is extracted from the captured image and used, the amount of data to be processed can be greatly reduced.

(Process of wiping area extraction unit 22)
As described above, it is possible to detect the wiping of dirt adhering to the lens surface of the in-vehicle camera 1 from the time change of the in-vehicle camera 1, but even when a moving object is reflected in the in-vehicle camera 1, Change will occur.

Therefore, the wiping area extraction unit 22 extracts the time change of the captured image from only the captured image in a stable state in which the moving object is not reflected on the in-vehicle camera 1 or the movement of the moving object reflected is small. Detect the wiped area.

FIG. 7 shows an example of a flowchart of processing of the wiping area extraction unit 22.

First, the stable state signal output from the stable state determination unit 21 is referenced to determine whether the stable state signal indicates a stable state (step 2201). If the stable state signal is in a stable state, an image captured by the in-vehicle camera 1 is acquired (step 2202). Then, a difference image between the acquired captured image in the stable state and the captured image in the previous stable state acquired in the previous process is generated (step 2203). Then, an area having a large difference (difference area) is extracted from the difference image (step 2204), and the difference area is output as a dirt wiping area (step 2205). The large difference area extracted in step 2204 is an area where the difference is equal to or greater than a predetermined value. This predetermined value may be the same value as the predetermined value used in the description of step 2103, or a different value. However, it is desirable to adjust according to the actual machine environment. This series of processing is repeatedly executed every time a captured image is output from the in-vehicle camera 1.

(Expansion example 1 of the process of the wiping area extraction unit 22)
By the way, when the in-vehicle camera 1 is irradiated with, for example, a headlight of a surrounding vehicle, the brightness of dirt (water droplets) attached to the lens surface of the in-vehicle camera 1 changes, and the difference image generated in the step 2203 is changed. In this case, a dirt region with a changed brightness is extracted. Also, when the headlight irradiation stops and the brightness of the stain returns, the stain area is extracted again from the difference image generated in step 2203.

Here, the sign of the difference generated in step 2203 is different between the start and end of headlight irradiation. Therefore, the wiping area extraction unit 22 may add a process of adding the difference images generated in Step 2203 for a certain period and accumulating between Step 2203 and Step 2204. If the difference image is added for a sufficient period including the start and end of headlight irradiation, the change in the brightness of the stain due to the temporary irradiation of the headlight can be detected from the difference image generated in step 2203. It can be removed by offsetting. Here, the period during which the difference image is accumulated is a method of setting a certain period after the difference is extracted, a method of setting the illuminance of the in-vehicle camera 1 to be a predetermined value or more, or a headlight detection process (not shown). There is a method of setting a period for detecting a headlight from a captured image of the in-vehicle camera 1. Here, simple headlight detection as an example of headlight detection processing can be realized by detecting a high-luminance circular region from a captured image.

In addition, since dirt such as water drops becomes extremely bright when irradiated with headlights, areas that have changed extremely brightly should be areas where the brightness has changed due to irradiation with headlights, and should be excluded from areas where dirt is wiped off. You can also.

(Expansion example 2 of the process of the wiping area extraction unit 22)
By the way, the stain | pollution | contamination adhering to the lens surface of the vehicle-mounted camera 1 may be removed in steps little by little. For example, there is a case where the lens is not cleaned manually and a part of the dirt remains, and the cleaning is repeated many times, or a part of the water drops run off by its own weight. In these cases, only a part of the dirt that is wiped off each time is extracted from the difference image extracted in step 2203.

Therefore, a process for accumulating the difference image for a certain period may be added between the step 2203 and the step 2204. Alternatively, in the step 2205, the difference area may be accumulated for a certain period to be the wiping area. As a result, it is possible to suppress the removal of the wiping area for dirt.

Here, the predetermined period for accumulating the difference image or the difference area may be a predetermined time, but until the dirt state managed by the dirt state management unit 4 becomes “no dirt” or dirt. It is preferable that the detection unit 3 functions. As a result, it is possible to avoid spillage of the wiping area for dirt. In addition, it is preferable that the wiping area is sequentially output so that the dirt state management unit 4 can manage the dirt wiping state even during the certain period in which the difference image or the difference area is accumulated.

(Extended example 3 of processing of the wiping area extraction unit 22)
By the way, the wiping area extraction unit 22 also extracts a feature amount such as an edge image or a HOG feature amount from the captured image without using the captured image of the in-vehicle camera 1 as it is (here, the extraction result is referred to as a feature amount image). This feature amount image may be used instead of the captured image. By using an appropriate feature amount, the detection performance can be improved and the amount of data to be processed can be reduced. For example, when the HOG feature amount is used, it is possible to make it difficult to extract unnecessary image changes due to slight fluctuations of the moving body and ambient light. Further, it is possible to significantly reduce the amount of data to be processed by extracting and using an average value such as luminance and edge strength from the captured image, but in this case, the wiping area cannot be detected, so the wiping area Instead, the wiping accuracy information is output.

(Processing of the dirt state management unit 4)
The dirt state management unit 4 manages the attached area as a dirt storage area by setting the presence or absence of dirt attached to the lens surface of the in-vehicle camera 1 as a dirty state.

Therefore, the dirt state management unit 4 refers to the dirt detection area or the dirt detection intensity output from the dirt detection unit 3 and stores the dirt state and the dirt storage area.

However, the dirt detection unit 3 may not function depending on the situation, for example, when the host vehicle is stopped. Even if the dirt on the lens surface of the in-vehicle camera 1 is wiped off in this state, the dirt wiping is not reflected in the dirt state or the dirt storage area stored in the dirt state management part 4 only by the dirt detector 3.

Therefore, the dirt state management unit 4 refers to the wiping area output from the wiping area extraction unit 22 and erases the wiping area from the dirt storage area stored in the dirt state management unit 4. Further, when the wiping area exceeds a predetermined area, the stored dirt state is changed from “dirty” to “no dirt”.

The dirty state management unit 4 may be configured to manage only the dirty state, may be configured to manage only the dirty storage area, or may be configured to manage both the dirty state and the dirty storage area. It may be. In the case where the dirty state management unit 4 manages both the dirty state and the dirty storage area, the stored dirty state is changed from “dirty” to “dirty” when the dirty storage area becomes less than a predetermined area. It may be changed to “None”.

(Extended example 1 of processing of the dirt state management unit 4)
Note that the wiping area output from the wiping area extracting unit 22 is obtained by extracting an area that has changed with time in the captured image of the in-vehicle camera 1, and may include an area in which dirt newly appears.

Therefore, when an area that is not in the dirt storage area stored by the dirt state management unit 4 is newly output from the wiping area extraction unit 22 as a wiping area, the wiping area is newly added to the dirt storage area as a dirt adhesion area. May be stored. Similarly, when the dirt state stored in the dirt state management unit 4 is “no dirt” and the wiping area exceeds a predetermined area, the dirt state is changed from “no dirt” to “dirty”. You may change to

(Extended example 2 of processing of the dirt state management unit 4)
By the way, the stain | pollution | contamination adhering to the lens surface of the vehicle-mounted camera 1 may be removed in steps little by little. For example, there is a case where the lens is not cleaned manually and a part of the dirt remains, and the cleaning is repeated many times, or a part of the water drops run off by its own weight. In these cases, as the wiping area output from the wiping area extraction unit 22, only a part of the dirt that is wiped each time is output.

Therefore, the dirt state management unit 4 accumulates the wiping areas output by the wiping area extraction unit 22, and when the accumulated wiping area exceeds a predetermined area, the stored dirt state is changed from “dirty” to “ It is preferable to change from “no dirt” or “no dirt” to “dirty”.

On the other hand, when the dirt state management unit 4 manages the dirt storage area, the wiping area output from the wiping area extraction unit 22 may be sequentially reflected in the dirt storage area.

By using the present embodiment described above, in the in-vehicle imaging device, after excluding the scene where a person or other moving body cleaning the lens surface of the in-vehicle camera 1 is reflected in the captured image of the in-vehicle camera 1, It is possible to provide a lens dirt wiping diagnostic device (dirt wiping diagnostic unit 2, dirt detection unit 3, and dirt state management unit 4) that extracts temporal changes in the captured image and detects wiping of lens dirt.

That is, in a state where the cleaner is reflected close to the in-vehicle camera 1, the stable state signal by the stable state determination unit 21 becomes “unstable state” and thus is excluded from the processing in the wiping region extraction unit 22, As for the wiping of dirt, there is a “stable state” when it is dirty, and there is a “stable state” even after the dirt is wiped off. Therefore, these “stable states” are targets of processing in the wiping area extraction unit 22. Thus, it becomes possible to detect the removal of dirt.

Further, there is no restriction that the captured image at the time of stopping must be used as a reference image, and it can be diagnosed at any time, avoiding the separation of the imaging time of the two images that take the difference, and the possibility of misjudgment. Can be reduced.

Further, even in a situation where the dirt detection unit 3 that detects dirt attached to the lens surface of the in-vehicle camera 1 does not function (for example, when the host vehicle is stopped), the wiping of the dirt attached to the lens of the in-vehicle camera 1 is detected, The autonomous control system (for example, automatic driving system) of the vehicle that has been stopped due to the dirt on the lens of the in-vehicle camera 1 can be resumed.

FIG. 8 is a block diagram showing a schematic configuration of the in-vehicle imaging device according to the second embodiment of the present invention. In the in-vehicle image pickup apparatus 200 shown in FIG. 8, the same reference numerals are given to components having the same functions as those of the in-vehicle image pickup apparatus 100 according to the first embodiment shown in FIG.

(Configuration of in-vehicle imaging device 200)
As shown in FIG. 8, the in-vehicle imaging device 200 of the present embodiment includes an in-vehicle camera 1, a dirt wiping diagnosis unit 2, a dirt detection unit 3, and a dirt state management unit 4. The dirt wiping diagnosis unit 2 includes a stable state determination unit 21 and a wiping region extraction unit 22. The dirt wiping diagnostic unit 2, the dirt detection unit 3, and the dirt state management unit 4 are also referred to as a lens dirt wiping diagnostic device.

The wiping area extraction unit 22 according to the present embodiment includes the in-vehicle camera stored in the dirt state management unit 4 in addition to the captured image output from the in-vehicle camera 1 and the stable state signal output from the stable state determination unit 21. Reference is made to the dirt adhesion area (dirt storage area) of the lens surface 1.

(Process of wiping area extraction unit 22)
FIG. 9 is an example of a flowchart of processing of the wiping area extraction unit 22 in the second embodiment.

First, the dirt storage area stored in the dirt state management unit 4 is acquired (step 2210). Thereafter, if the stable state signal output from the stable state determination unit 21 indicates the stable state, the captured image of the in-vehicle camera 1 is acquired, and the captured image and the image of the stable state immediately before acquired in the previous process are acquired. A difference image from the image is generated (steps 2201 to 2203). Then, the difference image is accumulated (step 2211), and an area having a large difference (difference area) is extracted from the accumulated difference image (step 2212). Next, the dirt storage area acquired in step 2210 is compared with the accumulated difference area (step 2213), and if most of the dirt storage area becomes the accumulated difference area, it is accumulated. The difference area is output as an area (wiping area) from which dirt is wiped off (step 2205). On the other hand, if the stable state signal indicates an unstable state in step 2201 or if it is not the difference area in which most of the dirt storage area is accumulated in step 2213, a new captured image in the stable state is displayed. In order to acquire, after stopping for a certain period of time (step 2214), the processing is restarted from step 2201.

Thus, even when the dirt is wiped off stepwise over a long period of time, the difference images can continue to be accumulated in step 2211 until most of the dirt is wiped off.

Here, it is preferable to store the direction (sign) of the difference in the accumulated difference area. As a result, as in the above (Extended example 1 of processing of the wiping area extraction unit 22), image changes that are not caused by wiping off dirt such as temporary irradiation of headlights of surrounding vehicles can be excluded from the difference image or the wiping area. .

In addition, the process of the wiping area | region extraction part 22 of a present Example is not limited to the flowchart shown in FIG. For example, the same function can be obtained by omitting step 2211 and extracting a region having a large difference from the difference image in step 2212 and accumulating the extracted region and using the accumulated region as a difference region. .

FIG. 10 is a block diagram illustrating a schematic configuration of the in-vehicle imaging device according to the third embodiment of the present invention. In the in-vehicle image pickup apparatus 300 shown in FIG. 10, the same reference numerals are given to components having the same functions as those of the in-vehicle image pickup apparatus 100 according to the first embodiment shown in FIG.

(Configuration of in-vehicle imaging device 300)
As shown in FIG. 10, the in-vehicle imaging device 300 of the present embodiment includes an in-vehicle camera 1, a dirt wiping diagnosis unit 2, a dirt detection unit 3, and a dirt state management unit 4. Further, the dirt wiping diagnosis unit 2 includes a stable state determination unit 21, a wiping region extraction unit 22, and a cleaning state determination unit 23. The dirt wiping diagnostic unit 2, the dirt detection unit 3, and the dirt state management unit 4 are also referred to as a lens dirt wiping diagnostic device.

(Processing of cleaning state determination unit 23)
When the lens of the in-vehicle camera 1 is manually cleaned, a cloth or a finger is pressed against the lens of the in-vehicle camera 1. Since light from the outside is blocked in the area where the cloth or finger is pressed, the illuminance of the in-vehicle camera 1 is reduced or the focus is shifted. Moreover, in the captured image of the vehicle-mounted camera 1, the area where the cloth or finger is pressed becomes dark.

FIG. 11 is an example of a captured image captured by the in-vehicle camera 1 when a cloth is pressed against the lens of the in-vehicle camera 1. Referring to FIG. 11, a region 34, which is a region pressed against the cloth, is displayed darkly and black.

Therefore, the cleaning state determination unit 23 refers to the illuminance of the in-vehicle camera 1 and determines that the lens of the in-vehicle camera 1 has been cleaned when the illuminance has greatly decreased by a predetermined value or more, and cleans the determination result. Output as a signal.

(Another example 1 of the process of the cleaning state determination unit 23)
Further, when the in-vehicle camera 1 has an autofocus function, the cleaning state determination unit 23 refers to the focal length of the in-vehicle camera 1 (not shown), and when the change amount of the focal length is equal to or greater than a predetermined value, the in-vehicle camera 1 The lens may be determined to be cleaned, and the result of the determination may be output as a cleaning signal.

(Another example 2 of the process of the cleaning state determination unit 23)
In addition, the cleaning state determination unit 23 calculates the average luminance from the captured image of the in-vehicle camera 1, and determines that the lens of the in-vehicle camera 1 is cleaned when the average luminance is greatly reduced by a predetermined value or more. The result of the determination may be output as a cleaning signal.

(Another example 3 of the process of the cleaning state determination unit 23)
Moreover, the cleaning state determination part 23 extracts the area | region (dark area) below the predetermined value from the picked-up image of the vehicle-mounted camera 1, and when the area of the said dark area becomes more than predetermined value, the said vehicle-mounted camera 1 The lens may be determined to be cleaned, and the result of the determination may be output as a cleaning signal.

(Extended example of another example 3 of the process of the cleaning state determination unit 23)
By the way, as shown in the captured image of FIG. 11, when the area 34 that is the area where the cloth is pressed does not reach the entire screen, a large luminance change occurs at the end of the area 34.

Therefore, when a dark region that does not reach the entire screen (the entire captured image) is detected, the cleaning state determination unit 23 determines whether there is a large luminance change at the end of the dark region, and the end of the dark region. If there is a luminance change greater than or equal to a predetermined value, it is determined that the dark area is caused by cleaning the lens, and if there is no luminance change greater than or equal to the predetermined value at the end of the dark area, the dark area is It is determined that it is not caused by cleaning. Thereby, the erroneous detection of cleaning determination can be suppressed.

(Another example 4 of the process of the cleaning state determination unit 23)
In addition, the dark region has not only low brightness but also has no edge. Therefore, the cleaning state determination unit 23 extracts a region (non-edge region) whose edge strength is equal to or less than a predetermined value from the captured image of the in-vehicle camera 1, and when the area of the non-edge region becomes equal to or greater than the predetermined value, It may be determined that the lens of the in-vehicle camera 1 has been cleaned, and the result of the determination may be output as a cleaning signal.

(Extended example of processing of cleaning state determination unit 23)
By the way, when the lens of the vehicle-mounted camera 1 is cleaned in several times, the amount of decrease in illuminance and focal length of the vehicle-mounted camera 1, the amount of decrease in the average luminance of the captured image, and the dark region are one cleaning. And the area of the non-edge region are not sufficiently large, and there is a concern that it is not determined that the lens of the in-vehicle camera 1 is cleaned.

Therefore, the cleaning state determination unit 23 may determine that the lens of the in-vehicle camera 1 has been cleaned when the decrease in illuminance, the change in focal length, the decrease in average luminance of the captured image, and the like continues for a longer period of time. .

Further, when determining the cleaning state from the area of the dark region and the area of the non-edge region, first, the dark region or the non-edge region is detected from the captured image of the in-vehicle camera 1 as a region where dirt has been cleaned. . Then, the dark region or the non-edge region is sequentially accumulated, and when the accumulated area of the dark region or the non-edge region becomes a predetermined value or more, it is determined that the lens of the in-vehicle camera 1 is cleaned. The result of the determination is preferably output as a cleaning signal.

Thus, even when the lens of the in-vehicle camera 1 is cleaned several times, it can be detected that the lens has been cleaned.

(Process of wiping area extraction unit 22)
While the lens of the in-vehicle camera 1 is being cleaned (cleaning period), the cloth or the finger pressed against the lens of the in-vehicle camera 1 is displayed large in the captured image of the in-vehicle camera. Cannot observe dirt attached to the lens surface of the in-vehicle camera 1 from the captured image of the in-vehicle camera 1. Further, by comparing the captured images immediately before and immediately after the cleaning period, it is possible to detect a dirt area wiped off by cleaning the lens of the in-vehicle camera 1.

FIG. 12 is an example of a flowchart of processing of the wiping area extraction unit 22 in the present embodiment.

First, it is determined whether the stable state signal output from the stable state determination unit 21 indicates a stable state (step 2201) and whether the cleaning signal output from the cleaning state determination unit 23 indicates that cleaning is being performed. Perform (step 2221). And when it is in a stable state and not being cleaned (not being cleaned), a captured image of the in-vehicle camera 1 is acquired (step 2202). Thereafter, it is determined whether or not the cleaning signal indicates that cleaning is being performed in the immediately preceding process (step 2222). If the cleaning signal indicates that cleaning is being performed, the captured image acquired in step 2202 is a captured image in a stable state immediately after cleaning. Therefore, a difference image between the captured image acquired in step 2202 and the non-cleaning captured image acquired in the previous process is generated (step 2223). The difference image is the difference between the captured images immediately before and after the cleaning period. Thereafter, an area with a large difference (difference area) is extracted from the difference image (step 2204), and the difference area is output as a wiped area (wiping area) (step 2205). If the steady state signal indicates an unstable state in step 2201 or if the cleaning signal indicates that cleaning is being performed in step 2221, the cleaning signal does not indicate that cleaning is being performed in the immediately preceding process in step 2222. In this case, after the processing is stopped for a certain time (step 2224), the processing is restarted from step 2201.

Note that the flowchart of the process of the wiping area extraction unit 22 may take other forms. For example, steps 2223, 2204, and 2205 are performed only when the order of steps 2221 and 2202 is reversed to acquire all the captured images in the stable state and the cleaning signal is switched from cleaning to non-cleaning. Even if executed, the same function can be obtained.

(Expansion example 1 of the process of the wiping area extraction unit 22)
Moreover, when the lens of the vehicle-mounted camera 1 is cleaned several times, it is assumed that the cleaning period continues several times.

Therefore, in step 2204, the wiping area extraction unit 22 may accumulate the difference image for a certain period, and may extract a large difference area (difference area) in the accumulated difference image. Alternatively, in step 2205, after the difference area is accumulated for a certain period, the accumulated difference area may be output as the wiping area.

Here, the period during which the difference image or the difference area is accumulated may be a predetermined period. However, until the dirt state managed by the dirt state management unit 4 becomes “no dirt” or dirt detection is performed. It is preferable that the part 3 is functioned. As a result, it is possible to avoid spillage of the wiping area for dirt. In addition, it is preferable that the wiping area | region extraction part 22 outputs a wiping area | region sequentially so that the wiping state of dirt can be managed also during the accumulation period of the said difference image thru | or the said difference area | region.

(Expansion example 2 of the process of the wiping area extraction unit 22)
Further, the wiping area extraction unit 22 extracts a feature amount such as an edge image or a HOG feature amount from the captured image without using the captured image of the in-vehicle camera 1 as it is (here, the extraction result is referred to as a feature amount image). ), This feature amount image may be used instead of the captured image. By using an appropriate feature amount, the detection performance can be improved and the amount of data to be processed can be reduced. For example, when the HOG feature amount is used, it is possible to make it difficult to extract unnecessary image changes due to slight fluctuations of the moving body and ambient light. Further, it is possible to significantly reduce the amount of data to be processed by extracting and using average values such as luminance and edge intensity from the captured image, but in this case, the wiping area for the dirt cannot be detected, and thus the wiping is not performed. Instead of the area, the wiping accuracy information is output.

In the first to third embodiments of the present invention, if information similar to the vehicle speed of the host vehicle and passengers getting on and off is detected, the period for performing the dirt wiping detection by the dirt wiping diagnosis unit 2 can be limited. In this embodiment, the vehicle speed of the own vehicle and information similar to the passenger getting on and off of the own vehicle are acquired and used.

Information similar to getting on and off of the passenger of the own vehicle is information on opening / closing of the door and / or window of the own vehicle or a change in the vehicle weight. In this embodiment, a sensor for detecting and acquiring such information is provided. Since the vehicle height also changes depending on the vehicle weight, a change in the vehicle height may be detected instead of the change in the vehicle weight.

In the lens dirt wiping diagnostic apparatus of this embodiment, when the door is opened and closed with the vehicle speed sufficiently lowered, or when the vehicle weight decreases, the occupant removes the lens of the in-vehicle camera 1 from the own vehicle. It is determined that the vehicle has got off, and this is used as a trigger for starting the process of the dirt wiping diagnosis unit 2.

Moreover, when the vehicle-mounted camera 1 is a side camera installed on the side surface of the host vehicle, the passenger can clean the lens of the vehicle-mounted camera 1 from inside the vehicle while the window near the vehicle-mounted camera 1 is open. Regardless of the vehicle speed, the stain wiping diagnosis unit 2 may perform the process on the captured image of the in-vehicle camera 1.

In addition, if the door opens and closes again after a predetermined time has elapsed since it is determined that the occupant got off the vehicle, or if the vehicle becomes heavy, it is determined that the occupant has boarded the vehicle, The processing of the dirt wiping diagnosis unit 2 may be stopped. Moreover, when the vehicle-mounted camera 1 is a side camera, when the window near the vehicle-mounted camera 1 is closed, the process of the dirt wiping diagnosis unit 2 for the vehicle-mounted camera 1 may be stopped.

In the first to third embodiments of the present invention, when the surroundings of the host vehicle are dark and dirt cannot be visually recognized in the captured image of the in-vehicle camera 1, the information similar to the passenger getting on and off of the host vehicle described in the fourth embodiment is used. The wiping of the lens of the in-vehicle camera 1 may be determined.

However, this alone cannot determine whether the dirt has actually been wiped off. In view of this, it is preferable to provide a mechanism that allows the user to determine the presence or absence of dirt on the lens surface of the in-vehicle camera 1 when the surroundings are dark and dirt cannot be visually recognized in the captured image of the in-vehicle camera 1. For example, when the illuminance of the in-vehicle camera 1 is extremely low, the captured image of the in-vehicle camera 1 is displayed on a monitor in the vehicle or the like, and the user is checked for the presence or absence of dirt and the confirmation result is input to the monitor. A display prompting input from (for example, a touch panel) is performed, and the user is checked for the presence or absence of dirt. When the user confirms that there is no dirt and the user makes an input, the dirt state management unit 4 receives the input and initializes the stored dirt state and / or dirt storage area.

By the way, when the host vehicle is stopped and the lens of the in-vehicle camera 1 is cleaned, the user may turn off the engine of the stopped vehicle. In this case, the lens dirt wiping diagnostic apparatus can be operated by a vehicle-mounted battery (not shown), but the lens dirt wiping diagnostic apparatus is also immediately stopped because there is a concern that the battery is out of charge. In this case, the lens dirt wiping diagnostic apparatus according to the first to fifth embodiments cannot detect the wiping of dirt attached to the lens of the in-vehicle camera 1.

(Process of wiping area extraction unit 22)
In the present embodiment, the wiping area extraction unit 22 of the lens dirt wiping diagnostic apparatus before the engine is stopped or immediately after the engine is stopped until the lens dirt wiping diagnostic apparatus is stopped, the captured image in a stable state is stored in a nonvolatile storage unit. (Not shown). The storage unit (eg, a non-volatile memory) can hold the memory even when power supply is interrupted. Then, when the engine is restarted, the wiping area extraction unit 22 extracts a difference image between the captured image stored in the storage unit and the latest captured image in the stable state acquired after the engine is restarted, and wipes off the dirt. Detect areas. In this case, the cleaning state determination unit 23 is not necessary.

(Expansion example 1 of the process of the wiping area extraction unit 22)
However, when the engine is stopped, there is a concern that it takes a long time to restart the engine. Therefore, it is preferable to extract an edge image or HOG feature value that is resistant to temporal changes without using the captured image of the in-vehicle camera 1 as it is (here, the extraction result is referred to as a feature value image), and use this feature value image. .

(Expansion example 2 of the process of the wiping area extraction unit 22)
If the engine stop time becomes extremely long, not only the brightness of the surroundings changes, but also there is a concern that the arrangement of movable surrounding objects (vehicles, etc.) may change.

Therefore, when the engine stop time is extremely long, it is preferable to provide a mechanism for allowing the user to determine whether the lens surface of the in-vehicle camera 1 is dirty. For example, the engine stop time is stored in a nonvolatile storage unit (not shown), and when the engine is restarted, the engine stop time is calculated as the difference between the engine stop time stored in the storage unit and the current time, and the engine When the stop time is greater than or equal to a predetermined value, the captured image of the in-vehicle camera 1 is displayed on a monitor in the vehicle or the like, the user is checked for the presence or absence of dirt, and the confirmation result is input from the monitor input means (for example, touch panel) To prompt the user to check for contamination. When the user confirms that there is no dirt and the user makes an input, the dirt state management unit 4 receives the input and initializes the stored dirt state and / or dirt storage area.

According to the present embodiment, it is possible to cope with the case where the stop time becomes long, the imaging time of the two images taking the difference is separated, and the surrounding brightness and the arrangement of the object are changed, and the possibility of erroneous determination can be reduced.

By the way, as described in the fourth embodiment, even when the passenger is traveling, the passenger in the passenger seat opens the door window and wipes the lens of the side camera on the passenger seat while the vehicle is traveling. In some cases, the lens of the in-vehicle camera 1 is manually cleaned. However, if the vehicle is not traveling on a monotonous roadway, the stable state determination unit 21 described in the third embodiment continues to output a stable state signal indicating an unstable state, and the dirt wiping diagnosis unit 2 does not function, and the dirt is wiped off. Cannot be detected.

Therefore, when the vehicle speed of the host vehicle is equal to or higher than the predetermined value, it is determined only by the cleaning state determination unit 23 that the lens of the in-vehicle camera 1 has been cleaned.

FIG. 13 is a block diagram showing a schematic configuration of an in-vehicle imaging device according to Embodiment 7 of the present invention. In the in-vehicle imaging device 400 shown in FIG. 13, the same reference numerals are given to components having the same functions as those in the in-vehicle imaging device 300 of the third embodiment shown in FIG.

(Configuration of in-vehicle imaging device 400)
As shown in FIG. 13, the in-vehicle image pickup apparatus 400 of the present embodiment includes an in-vehicle camera 1, a dirt wiping diagnosis unit 2, a dirt detection unit 3, and a dirt state management unit 4, and further includes a vehicle speed sensor. 5 is provided. Here, the vehicle speed sensor 5 detects and outputs the vehicle speed of the host vehicle. Further, the dirt wiping diagnosis unit 2 includes a cleaning state determination unit 23 in addition to the stable state determination unit 21 and the wiping region extraction unit 22 as in the third embodiment. However, in this embodiment, the stable state determination unit 21 and the wiping area extraction unit 22 can be omitted. Further, the dirt wiping diagnostic unit 2, the dirt detection unit 3, and the dirt state management unit 4 may be a lens dirt wiping diagnostic device, or may include a vehicle speed sensor 5 as a lens dirt wiping diagnostic device.

(Processing of the dirt wiping diagnosis unit 2)
The dirt wiping diagnostic unit 2 acquires the vehicle speed of the host vehicle output from the vehicle speed sensor 5, and when the vehicle speed exceeds a predetermined value, stops the processing of the stable state determination unit 21 and the wiping region extraction unit 22, The cleaning signal output from the cleaning state determination unit 23 is output to the dirt state management unit 4.

(Processing of the dirt state management unit 4)
When the cleaning signal output from the cleaning state determination unit 23 changes from “not cleaning” to “cleaning” and further from “cleaning” to “not cleaning”, the in-vehicle camera Assuming that the lens No. 1 has been cleaned, the stored dirt state is changed from “dirty” to “no dirt”. Also, the dirty storage area is deleted (initialized).

(Extended example of processing of the dirt state management unit 4)
However, the cleaning signal output from the cleaning state determination unit 23 merely indicates the presence or absence of cleaning, and it cannot be determined from the cleaning signal whether dirt has actually been wiped off. Therefore, it is preferable not to carry out this embodiment unnecessarily.

Therefore, for example, window opening / closing information is acquired from the vehicle, and only when the window is sufficiently open, the vehicle-mounted camera 1 mounted in the vicinity of the window becomes dirty from the cleaning signal output by the cleaning state determination unit 23. Determine the wiping. Thereby, this embodiment can be prevented from being unnecessarily performed.

In addition, limiting the implementation conditions, such as providing a load sensor in the seat of the host vehicle to detect the presence of an occupant and carrying out the present embodiment only for the in-vehicle camera 1 in which a passenger is present in a nearby seat Is preferred.

When the surrounding object is reflected on the in-vehicle camera 1, it can be determined that the area where the object is displayed is not dirty. Further, in the captured image of the in-vehicle camera 1, the outline of the dirt often appears unclear, and it can be determined that there is no dirt in the area where the clear outline appears.

Therefore, an area where an object can be detected by an object recognition unit (not shown) such as pedestrian detection for detecting a pedestrian reflected in a captured image of the in-vehicle camera 1 or a vehicle detection for detecting a vehicle reflected in the captured image, or the in-vehicle camera 1 It may be determined that there is no dirt in an area where a clear contour is detected in the captured image, and the dirt state management unit 4 may reflect this determination on the dirt state and dirt storage area to be stored.

According to the present embodiment, the dirt wiping area can be detected even if the past dirt adhesion area and dirt state are unknown.

<Appendix 1>
The present invention described above is configured as follows.
(1) A vehicle-mounted camera that captures an image through a lens, a stain detection unit that detects the presence or absence of a stain on the lens or a stain-attached region based on a time change of a captured image sequentially captured by the vehicle-mounted camera, A dirt wiping diagnostic unit for diagnosing lens dirt wiping, and a dirt state management unit for managing the lens dirt, wherein the dirt wiping diagnostic unit is based on captured images sequentially captured by the in-vehicle camera. A change in time of a captured image when a determination result by the stable state determination unit among the captured images sequentially captured by the in-vehicle camera is in a stable state; A wiping area extraction unit for extracting a wiping area for dirt based on
The dirt state management unit stores the presence / absence of dirt detected by the dirt detection unit or the dirt adhesion area, and the dirt wiping area extracted by the wiping area extraction unit is stored. Since it is an on-vehicle imaging device that manages the contamination of the lens by reflecting on the presence or absence or contamination area, the timing of performing the work to remove the contamination attached to the lens surface of the in-vehicle camera is unknown Even if it exists, when diagnosing that the dirt on the lens surface has been wiped off, it is possible to provide an in-vehicle imaging device having a lens dirt wiping diagnostic device that can reduce the possibility of erroneous determination.

Further, the present invention is configured as follows.
(2) In the in-vehicle imaging device, the stable state determination unit sequentially acquires captured images output from the in-vehicle camera, and the acquired image is acquired immediately before the acquired captured image and the acquired captured image each time. A difference image with the image is generated, a region with a large difference is extracted based on the generated difference image, and if the area of the region with the large difference is equal to or less than a predetermined value, the time change of the acquired captured image is small. Since the vehicle-mounted image pickup apparatus is characterized in that it is determined that the area is large in the difference and the area of the large difference exceeds a predetermined value, it is determined that the time change of the acquired captured image is not large and stable. The stable state can be appropriately determined by determining the stable state when the area of the large difference area is equal to or smaller than the predetermined value.

Further, the present invention is configured as follows.
(3) In the in-vehicle imaging device, the stable state determination unit sequentially acquires captured images output from the in-vehicle camera, and sequentially generates feature amount images obtained by extracting feature amounts from the sequentially acquired captured images. Each time, a difference image between the generated feature amount image and the feature amount image generated immediately before the generated feature amount image is generated, a region having a large difference is extracted based on the generated difference image, and the extraction is performed. If the area of the large difference area is less than or equal to a predetermined value, it is determined that the acquired captured image has a small temporal change and is in a stable state, and if the area of the large difference area exceeds the predetermined value, the acquired Since the in-vehicle imaging device is characterized in that it is determined that the change in time of the captured image is large and is not in a stable state, the detection performance can be improved or the amount of data to be processed can be reduced by using the feature image.

Further, the present invention is configured as follows.
(4) In the above-described in-vehicle imaging device, the wiping area extraction unit sequentially acquires captured images in which the determination result by the stable state determination unit is in a stable state, and the acquired captured image in the stable state, An in-vehicle imaging device characterized by generating a difference image with the acquired captured image in a stable state, extracting a region having a large difference from the generated difference image, and setting the extracted region as a dirt wiping region Therefore, the possibility of erroneous determination can be reduced by using a captured image in a stable state.

Further, the present invention is configured as follows.
(5) In the above-described in-vehicle imaging device, the wiping area extraction unit sequentially acquires captured images in which the determination result by the stable state determination unit is in a stable state, and the acquired captured image in the stable state, Generate a difference image with the acquired captured image in a stable state, accumulate the generated difference image for a certain period, extract a region with a large difference from the accumulated difference image, and wipe the dirt of the extracted region Since it is an in-vehicle imaging device characterized by the area, it is possible to deal with the case where the brightness of the dirt changes due to temporary irradiation of headlights of other vehicles by accumulating difference images. is there.

Further, the present invention is configured as follows.
(6) In the above-described in-vehicle imaging device, the wiping area extraction unit sequentially acquires captured images in which the determination result by the stable state determination unit is in a stable state, and the acquired captured image in the stable state, Generate a difference image from the acquired captured image in a stable state, acquire the presence or absence of dirt or a dirt adhesion region stored in the dirt state management unit, and the acquired dirt disappears or the acquired dirt adhesion The generated difference image is accumulated until the area of the area becomes a predetermined value or less, a region having a large difference is extracted from the accumulated difference image, and the extracted region is used as a dirt wiping region. Since the vehicle-mounted imaging device is configured to accumulate the difference images, it is possible to suppress the removal of the dirt wiping area.

Further, the present invention is configured as follows.
(7) In the on-vehicle imaging device, the wiping area extraction unit sequentially acquires captured images in which the determination result by the stable state determination unit is in a stable state, and features from the sequentially acquired captured images in the stable state The feature amount image extracted is sequentially generated, a difference image between the generated feature amount image and the previously acquired feature amount image is generated, and a region having a large difference is extracted based on the generated difference image. In addition, since the vehicle-mounted imaging device is characterized in that the extracted area is a dirt wiping area, the detection performance can be improved and the amount of data to be processed can be reduced by using the feature amount image. .

Further, the present invention is configured as follows.
(8) In the on-vehicle imaging device, the dirt state management unit acquires the presence / absence of dirt or a dirt adhesion area detected by the dirt detection unit, stores the acquired dirt adhesion area as a dirt storage area, and The acquired presence / absence of dirt or the size of the area of the stored dirt storage area is stored as a dirt state, the dirt wiping area detected by the wiping area extraction unit is acquired, and the acquired from the stored dirt adhesion area Since the wiping area is deleted and the dirt state is updated based on the updated dirt adhesion area or the area of the deleted wiping area, the vehicle-mounted image pickup apparatus is provided. It can be carried out.

Further, the present invention is configured as follows.
(9) In the above-described on-vehicle imaging device, the dirt state management unit acquires the presence or absence of dirt or a dirt adhesion region detected by the dirt detection unit, stores the acquired dirt adhesion region as a dirt storage region, and The acquired presence / absence of dirt or the size of the area of the stored dirt storage area is stored as a dirt state, the dirt wiping area detected by the wiping area extraction unit is acquired and accumulated for a certain period, and the accumulated wiping area If the area of the image sensor is greater than or equal to a predetermined value, the stored in-vehicle image pickup apparatus is characterized by eliminating the dirt. It is possible to deal with cases where it is removed step by step.

Further, the present invention is configured as follows.
(10) In the above-described in-vehicle imaging device, the dirt state management unit notifies the user of the dirt management status of the lens based on the stored presence or absence of dirt or a dirt adhesion region.
Since the vehicle-mounted imaging device is characterized by this, the user can know the management status of lens dirt, and can take appropriate measures.

Further, the present invention is configured as follows.
(11) In the on-vehicle imaging device, the dirt state management unit notifies the system for autonomously controlling the vehicle of the dirt management status of the lens based on the stored presence or absence of dirt or a dirt adhesion region. Since the vehicle-mounted imaging device is characterized by this, in a system that performs autonomous control such as automatic driving, it is possible to know the management status of dirt on the lens and perform appropriate control.

Further, the present invention is configured as follows.
(12) A vehicle-mounted camera that captures an image through a lens, a stain detection unit that detects the presence or absence of a stain on the lens or a stain-attached region based on captured images sequentially captured by the vehicle-mounted camera, and a stain on the lens A dirt wiping diagnosis unit for diagnosing wiping of the lens, and a dirt state management unit for managing dirt on the lens, and the dirt wiping diagnosis unit is in a stable state based on captured images sequentially captured by the in-vehicle camera. The lens is cleaned based on the determination of the stable state determination unit that determines whether or not there is a cleaning state determination unit that determines that the lens has been cleaned, and the determination of the stable state determination unit and the cleaning state determination unit. Wiping to extract a picked-up image having a small temporal change before and after the period during which the lens is cleaned, and extracting a dirt wiping region from the difference between the two picked-up images before and after the period during which the lens is cleaned The dirt state management unit stores the presence or absence of dirt detected by the dirt detection unit or a dirt adhesion region, and already stores the dirt wiping region extracted by the wiping region extraction unit. It is reflected in the presence or absence of dirt or the area where dirt is attached, and it is an in-vehicle image pickup device that manages the dirt on the lens, so it reflects the cleaning condition of the lens and adheres to the lens surface of the in-vehicle camera. Provided is an in-vehicle imaging apparatus having a lens dirt wiping diagnostic device that can reduce the possibility of erroneous determination when diagnosing that dirt on a lens surface has been wiped off even when the operation timing of removing dirt is unknown. can do.

Further, the present invention is configured as follows.
(13) In the on-vehicle imaging device, the cleaning state determination unit extracts a region having a luminance equal to or lower than a predetermined value from the captured image of the on-vehicle camera as a dark region, and the area of the extracted dark region is equal to or larger than a predetermined value. In this case, since the in-vehicle imaging apparatus is characterized in that it is determined that the lens is cleaned, the cleaning state can be determined by extracting a dark area with low luminance.

Further, the present invention is configured as follows.
(14) In the on-vehicle imaging device, the cleaning state determination unit determines whether there is a large luminance change at an end of the extracted dark region when the extracted dark region does not reach the entire captured image. Further, only a dark region having a large luminance change at the end is further extracted, and when the area of the dark region having a large luminance change at the extracted end is equal to or larger than a predetermined value, it is determined that the lens is cleaned. Since the vehicle-mounted imaging device is characterized by this, the cleaning state can be more accurately determined by determining the luminance change at the end of the dark region.

Further, the present invention is configured as follows.
(15) In the on-vehicle imaging device, the cleaning state determination unit extracts a region having a luminance equal to or lower than a predetermined value from the captured image of the in-vehicle camera as a dark region, accumulates the extracted dark region, and accumulates the extracted dark region. When the area of the dark region becomes equal to or larger than a predetermined value, the on-vehicle imaging device is characterized in that it is determined that the lens is cleaned. Even when the lens is cleaned, it can be detected that the lens has been cleaned.

Further, the present invention is configured as follows.
(16) In the on-vehicle imaging device, the dirt wiping diagnosis unit refers to information similar to the speed of the host vehicle and / or boarding / exiting information of the host vehicle, and only when an occupant of the host vehicle can clean the lens. Since the vehicle-mounted imaging device is characterized by operating, the processing load can be reduced by stopping the operation when the lens cannot be cleaned.

Further, the present invention is configured as follows.
(17) In the on-vehicle imaging device, the dirt detection unit, the dirt wiping diagnostic unit, and the dirt state management unit constitute a lens dirt wiping diagnostic device, and the lens dirt wiping diagnostic device is configured to stop an engine. Immediately after the lens dirt wiping diagnostic device is stopped, the captured image of the in-vehicle camera is stored at the time of stop, and when the engine is restarted, the captured image stored at the time of stop is stored in the past before cleaning the lens. Since the vehicle-mounted image pickup device is used as a picked-up image of the lens, it is possible to appropriately manage the dirt state of the lens even if the lens stain wiping diagnostic device is stopped when the engine is stopped.

Further, the present invention is configured as follows.
(18) The vehicle-mounted imaging device includes an object recognition unit that detects an object from a captured image of the vehicle-mounted camera, and the dirt state management unit uses a region where the object recognition unit detects an object as a dirt wiping region. Since the vehicle-mounted imaging device is characterized in that the extracted wiping area is deleted from the extracted and stored dirt adhesion area, it is considered that there is no dirt on the lens in the area where the object is detected By reflecting the detection result of the object recognition unit, more appropriate management of the dirt state of the lens can be performed.

Further, the present invention is configured as follows.
(19) The vehicle-mounted imaging device is provided, and the autonomous control system of the vehicle is permitted to the vehicle autonomous control system when the area of the dirt adhesion region stored in the dirt state management unit is reduced to a predetermined value or less.
Since the vehicle control system is characterized by this, autonomous control of the vehicle can be performed without being affected by dirt on the lens.

In addition, this invention is not limited to the above-mentioned Example, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Any combination of the elements of the individual embodiments described above is included in the present invention.

DESCRIPTION OF SYMBOLS 1 ... Car-mounted camera, 2 ... Dirt wiping diagnosis part, 3 ... Dirt detection part, 4 ... Dirt state management part, 5 ... Vehicle speed sensor, 21 ... Stable state determination part, 22 ... Wiping area extraction part, 23 ... Cleaning state determination part , 31, 32... Region (water drop display region), 33... Region (pedestrian display region), 34.

Claims (15)

1. An in-vehicle camera that captures an image through a lens;
   Based on captured images sequentially captured by the in-vehicle camera, a dirt detection unit that detects the presence or absence of dirt on the lens or a dirt adhesion region;
   A dirt wiping diagnostic unit for diagnosing wiping of dirt on the lens;
   A dirt state management unit for managing dirt on the lens;
   With
   The dirt wiping diagnostic unit is
   A stable state determination unit that determines whether or not a stable state based on a temporal change of captured images sequentially captured by the in-vehicle camera;
   A wiping region extraction unit that extracts a wiping region of dirt based on a temporal change of a captured image when a determination result by the stable state determination unit is in a stable state among captured images sequentially captured by the in-vehicle camera;
   Have
   The dirt state management unit stores the presence / absence of dirt detected by the dirt detection unit or the dirt adhesion area, and the dirt wiping area extracted by the wiping area extraction unit is stored as the presence / absence of dirt or dirt adhesion. Reflect in the area and manage the dirt on the lens,
   An in-vehicle imaging device characterized by the above.
2. The in-vehicle imaging device according to claim 1,
   The stable state determination unit sequentially acquires captured images output from the in-vehicle camera, and generates a difference image between the acquired captured image and the captured image acquired immediately before the acquired captured image each time, A region having a large difference is extracted based on the generated difference image, and if the area of the region having the large difference is equal to or less than a predetermined value, it is determined that the acquired captured image has a small temporal change and is in a stable state, and the difference If the area of the large area exceeds a predetermined value, it is determined that the time change of the acquired captured image is large and is not stable.
   An in-vehicle imaging device characterized by the above.
3. The in-vehicle imaging device according to claim 1,
   The stable state determination unit sequentially acquires captured images output from the in-vehicle camera, sequentially generates feature amount images obtained by extracting feature amounts from the sequentially acquired captured images, and each time the generated feature amount image and A difference image with the feature amount image generated immediately before the generated feature amount image is generated, a region with a large difference is extracted based on the generated difference image, and an area of the extracted region with a large difference is a predetermined value If it is below, it is determined that the time change of the acquired captured image is small and stable, and if the area of the region with the large difference exceeds a predetermined value, the time change of the acquired captured image is large and not stable. To determine,
   An in-vehicle imaging device characterized by the above.
4. The in-vehicle imaging device according to claim 1,
   The wiping area extraction unit sequentially acquires captured images in which the determination result by the stable state determination unit is in a stable state, and the acquired captured image in the stable state and a captured image in the stable state acquired previously Generating a difference image, extracting a large difference area from the generated difference image, the extracted area as a dirt wiping area,
   An in-vehicle imaging device characterized by the above.
5. The in-vehicle imaging device according to claim 1,
   The wiping area extraction unit sequentially acquires captured images in which the determination result by the stable state determination unit is in a stable state, and the acquired captured image in the stable state and a captured image in the stable state acquired previously Generating a difference image, accumulating the generated difference image for a certain period, extracting a region having a large difference from the accumulated difference image, and setting the extracted region as a dirt wiping region;
   An in-vehicle imaging device characterized by the above.
6. The in-vehicle imaging device according to claim 1,
   The wiping area extraction unit sequentially acquires captured images in which the determination result by the stable state determination unit is in a stable state, and the acquired captured image in the stable state and a captured image in the stable state acquired previously A difference image is generated, the presence or absence of dirt or a dirt adhesion area stored in the dirt state management unit is acquired, and the acquired dirt disappears or the area of the acquired dirt adhesion area becomes a predetermined value or less. , The generated difference image is accumulated, a large difference area is extracted from the accumulated difference image, and the extracted area is set as a dirt wiping area.
   An in-vehicle imaging device characterized by the above.
7. The in-vehicle imaging device according to claim 1,
   The wiping area extraction unit sequentially acquires captured images in which the determination result by the stable state determination unit is in a stable state, and sequentially generates feature amount images obtained by extracting feature amounts from the sequentially acquired captured images in the stable state. Generating a difference image between the generated feature amount image and the previously acquired and generated feature amount image, extracting a region having a large difference based on the generated difference image, and wiping the extracted region with dirt Territory,
   An in-vehicle imaging device characterized by the above.
8. The in-vehicle imaging device according to claim 1,
   The dirt state management unit acquires the presence / absence of dirt or a dirt adhesion area detected by the dirt detection unit, stores the obtained dirt adhesion area as a dirt storage area, and stores the obtained dirt presence / absence or the stored dirt. The size of the area of the storage area is stored as a dirty state, the wiping area of the dirt detected by the wiping area extraction unit is acquired, the acquired wiping area is deleted from the stored dirt adhesion area, and the updated dirt Update the dirt state based on the area of the attached area or the deleted wiping area,
   An in-vehicle imaging device characterized by the above.
9. The in-vehicle imaging device according to claim 1,
   The dirt state management unit acquires the presence / absence of dirt or a dirt adhesion area detected by the dirt detection unit, stores the obtained dirt adhesion area as a dirt storage area, and stores the obtained dirt presence / absence or the stored dirt. The size of the area of the storage area is stored as a dirt state, the dirt wiping area detected by the wiping area extraction unit is acquired and accumulated for a certain period, and the area of the accumulated wiping area is stored when the area is equal to or greater than a predetermined value To make the dirty state without dirt,
   An in-vehicle imaging device characterized by the above.
10. The in-vehicle imaging device according to claim 1,
    The dirt state management unit notifies the user of the dirt management status of the lens based on the stored presence or absence of dirt or a dirt adhesion area.
    An in-vehicle imaging device characterized by the above.
11. The in-vehicle imaging device according to claim 1,
    The dirt state management unit notifies the system of autonomously controlling the vehicle of the dirt management status of the lens based on the presence or absence of dirt stored or a dirt adhesion region.
    An in-vehicle imaging device characterized by the above.
12. An in-vehicle camera that captures an image through a lens;
    Based on captured images sequentially captured by the in-vehicle camera, a dirt detection unit that detects the presence or absence of dirt on the lens or a dirt adhesion region;
    A dirt wiping diagnostic unit for diagnosing wiping of dirt on the lens;
    A dirt state management unit for managing dirt on the lens;
    With
    The dirt wiping diagnostic unit is
    A stable state determination unit that determines whether or not a stable state based on captured images sequentially captured by the in-vehicle camera;
    A cleaning state determination unit that determines that the lens has been cleaned;
    Based on the determination of the stable state determination unit and the cleaning state determination unit, a captured image with a small time change before and after the period in which the lens is cleaned is extracted, and before and after the period in which the lens is cleaned. A wiping area extracting unit that extracts a wiping area for dirt from a difference between two captured images;
    Have
    The dirt state management unit stores the presence / absence of dirt detected by the dirt detection unit or the dirt adhesion area, and the dirt wiping area extracted by the wiping area extraction unit is stored as the presence / absence of dirt or dirt adhesion. Reflect in the area and manage the dirt on the lens,
    An in-vehicle imaging device characterized by the above.
13. The in-vehicle imaging device according to claim 12,
    The cleaning state determination unit extracts a region having a luminance of a predetermined value or less as a dark region from a captured image of the in-vehicle camera, and the lens is cleaned when the area of the extracted dark region becomes a predetermined value or more. It is determined that
    An in-vehicle imaging device characterized by the above.
14. The in-vehicle imaging device according to claim 13,
    The cleaning state determination unit determines whether there is a large luminance change at the end of the extracted dark region when the extracted dark region does not reach the entire captured image, and the dark region having a large luminance change at the end Only when the area of a dark region having a large luminance change at the extracted edge is equal to or greater than a predetermined value, it is determined that the lens is cleaned.
    An in-vehicle imaging device characterized by the above.
15. The in-vehicle imaging device according to claim 12,
    The cleaning state determination unit extracts a region having a luminance equal to or lower than a predetermined value from the captured image of the in-vehicle camera as a dark region, accumulates the extracted dark region, and the accumulated dark region has an area of a predetermined value or more. It is determined that the lens has been cleaned.
    An in-vehicle imaging device characterized by the above.

Images