DE102020124044A1 - DETECTION AND CORRECTION OF AN OPTICAL SURFACE DEGRADATION - Google Patents

Abstract

The disclosure provides detection and correction of optical surface degradation. Systems, devices and methods for detecting and correcting optical surface deterioration are provided herein. An exemplary method includes tracking a first object within images of an area of interest obtained over a period of time by an image sensor of a vehicle during vehicle movement; Tracking a light source within the area of interest; Determining a change in the relative reflection intensity of the first object using the images based on a light source angle formed between the light source and an optical plane of an optical surface of the vehicle, the change in the relative reflection intensity indicating stray light of the optical surface; and activating a corrective action in response to the stray light.

Description

TECHNICAL FIELD

The present disclosure relates to systems and methods that detect and, in some cases, correct the effects of stray light or lens reflection created by the existing degraded condition of optical surfaces in vehicles.

GENERAL STATE OF THE ART

Scattered light is defined as light that has been reflected or scattered from an imaging beam that eventually finds its way to an image plane. Scattered light cannot form an image, but can be distributed quite evenly over the image plane. Thus, it gives the appearance of looking through a fog (such as lens reflection) and the effect of reducing image contrast. Stray light can reduce the effectiveness of vehicle-mounted camera imaging systems and perception algorithms. Such stray light can be caused by obstructions on the external optical surfaces of an automobile optics, such as windshield surfaces or camera lens cover surfaces. In automotive applications, such obstacles can occur in operating environments where dust and other materials can cover the optical surface and cause stray light. In addition, other types of sensors that use active lighting, such as time-of-flight cameras (TOF cameras) or light detection and ranging (LIDAR), may experience a decrease in optical performance due to increased noise floor, resulting in a decreased maximum Area detection or an increased error in estimating the distance leads. This decrease in optical performance can also affect the ability to resolve dark and / or distant objects in systems such as advanced driver assistance systems (ADAS) or autonomous vehicles (AV).

In addition, there are a number of common driving environments or use cases that lead to increased challenges with scattered light, where the effect of the scattered light is more pronounced due to the angle of the light field. Some of these include driving into a tunnel or driving at night with street lights on while the windshield is dirty or scratched. Such scenarios can lead to difficulties in detecting objects that appear darker than the intensity of the scattered light.

SHORT REPRESENTATION

Systems, devices and methods for detecting and correcting optical surface deterioration are provided herein. An exemplary method includes tracking a first object within images of an area of interest obtained over a period of time by an image sensor of a vehicle during vehicle movement; Tracking a light source within the area of interest; Determining a change in the relative reflection intensity of the first object using the images based on a light source angle formed between the light source and an optical plane of an optical surface of the vehicle, the change in the relative reflection intensity indicating stray light of the optical surface; and activating a corrective action in response to the stray light.

Figure list

• 1 stellt eine veranschaulichende Architektur dar, in der Techniken und Strukturen zum Bereitstellen der in dieser Schrift offenbarten Systeme, Vorrichtungen und Verfahren umgesetzt sein können.
• Die 2A-2C veranschaulichen gemeinsam Bilder, die gemeinsam Streulicht oder Linsenreflexion und Objektdetektion darstellen.
• 3 stellt schematisch Hindernisse oder Ablagerungen dar, die Varianzen von Streulicht oder Linsenreflexion auf einer optischen Oberfläche erzeugen.
• Die 4A-4C stellen gemeinsam eine Streulicht von Linsenreflexionsdetektion gemäß der vorliegenden Offenbarung in einer nächtlichen Umgebung dar.
• 5 ist ein Ablaufdiagramm eines beispielhaften Verfahrens der vorliegenden Offenbarung, das die Detektion und Korrektur von Streulicht oder Linsenreflexion beinhaltet.

The detailed description is set forth with reference to the accompanying drawings. The use of the same reference symbols can indicate similar or identical elements. Different embodiments may utilize different elements and / or components than those illustrated in the drawings, and some elements and / or components may not be present in different embodiments. The elements and / or components in the figures are not necessarily drawn to scale. In this disclosure, singular and plural terminology may be used interchangeably depending on the context.

• 1 FIG. 10 depicts an illustrative architecture in which techniques and structures for providing the systems, devices, and methods disclosed in this document may be implemented.
• The 2A-2C jointly illustrate images that jointly represent scattered light or lens reflection and object detection.
• 3 shows schematically obstacles or deposits that create variances of scattered light or lens reflection on an optical surface.
• The 4A-4C collectively depict scattered light from lens reflection detection in accordance with the present disclosure in a nocturnal environment.
• 5 Figure 4 is a flow diagram of an exemplary method of the present disclosure that includes detecting and correcting for stray light or lens reflection.

DETAILED DESCRIPTION

overview

The systems, devices, and methods disclosed herein are configured to detect stray light or lens reflection due to degradation of an optical surface of a vehicle. Of course, the embodiments in this document are not limited to the detection of scattered light and the present disclosure can include the detection and correction of scattered light and / or lens reflection. In general, when reference is made to scattered light, it is assumed that lens reflection can also be taken into account, unless expressly stated otherwise. In some embodiments, the systems, devices, and methods of this document may be configured to detect stray light or lens reflection and, in response, to activate corrective action to correct deterioration in the optical surface.

In general, the present disclosure uses knowledge of the functional / physical relationships between an optical surface and an angle of incidence of the light source (also referred to as light source angle) in order to improve the detection and measurement of obstacles on the optical surface. The term obstacle should be understood to include temporary or permanent deterioration of an optical surface. Deterioration can be due to surface defects such as scratches or pitting, but it can also be due to foreign matter such as water, dust, dirt, mold, insects, and other foreign matter. Of course, the systems and methods disclosed herein can also be used to quantify an extent of obstructions present on an optical surface that are generally undetectable by human observation. Obstacles can also be created by deterioration processes such as deterioration by ultraviolet light and the like. Stray light can affect images obtained by a camera or other light sensor in a vehicle, or can obscure the driver's view in general.

For example, stray light could be caused by foreign matter on a windshield. A camera positioned behind the windshield can be adversely affected by the stray light caused by foreign matter on the windshield. In another example, foreign material, such as dust, may be on the lens of a camera.

In other words, in some embodiments, the present disclosure provides for the detection and quantification of obstructions on an optical surface (e.g., dust on a windshield) that result in stray light during operation. The detection of stray light can be used as a trigger or condition to actuate the cleaning systems or to modify ADAS / AV sensing and planning algorithms and system level responses such as disabling ADAS features.

In some embodiments, systems, devices, and methods of this document may utilize knowledge of an area of interest while the vehicle is moving, such as the vehicle entering / exiting a tunnel or forward movement near street lights, in order to quantify to help a size of scattered light. The systems, devices and methods of this document can infer that the size of the scattered light is sensitive to the relative alignment of the light source and the optical surface (defined by an optical plane). Of course, the light source can be from a single source or can be from many sources including, but not limited to, albedo, sky dome, sun, infrastructure lighting, reflected lighting, and so on.

Additionally, a vehicle can detect objects based on contrast (e.g., dark and light objects) in a range of time of interest as the relative orientation of the light source changes follow. In some embodiments, a variety of sensor inputs are used to synthesize contrast determinations. For example, cameras, infrared sensors, and other similar contrast detection sensors can be used.

When the relative orientation of the light source changes, the vehicle can monitor the original and current relative reflection intensities of the objects to detect the level of stray light and identify conditions such as a nominal or dirty windshield. When stray light is detected, one or more vehicle systems can be actuated to reduce the effect or root cause of the stray light.

The systems, devices, and methods in this document can determine scattered light based on an image as well as in multiple images obtained over a period of time to improve detection and measurement. In some embodiments, the present disclosure may include tracking objects in images obtained over time. The objects can be tracked and the contrast of the objects can be observed over time (e.g. brightness versus darkness of objects). In addition, a light source such as a street lamp or the sun can be tracked and a light source angle relative to an optical plane of an optical surface can be determined. Scattered light can be determined from a change in the relative reflection intensity of objects based on the light source angle formed between the light source and the optical plane of an optical surface of the vehicle. In some embodiments, if stray light is detected, the systems, devices, and methods herein can correct the stray light by actuating a cleaning assembly, such as dispensing wiper fluid and actuating wipers, when the optical surface includes a front or rear windshield of the vehicle . Of course, the scattered light detected in some embodiments relates to the location of the scattered light from the viewing angle of a camera. For example, a camera has a fixed and long focal length, which means that obstacles are typically blurred.

In another example, a parameter, attribute, or action of an ADAS / AV vehicle system can be modified to correct for the stray light. For example, a speed of the vehicle could be reduced when stray light is identified. The effectiveness of corrective actions, such as windshield cleaning, can be verified in some embodiments. In other embodiments, root causes of stray light can be determined using images from a plurality of image sensors. In one or more embodiments, a neural network can be used to identify or infer the root causes of stray light.

Illustrative embodiments

Reference is now made to the drawings, in which 1 an illustrative architecture 100 illustrates in which the techniques and structures of the present disclosure may be implemented. The illustrative architecture 100 can be a vehicle 102 , an area of interest 104 , one or more objects, such as the object 106 and 108 , a light source 110 , a service provider 112 and a network 115 include. In general, the first object could 106 include a road surface and could be the second object 108 include a bumper of a vehicle. Other objects can likewise be identified and tracked as disclosed herein.

Components of the architecture 100 such as the vehicle 102 and the service provider 112 , can use a network 115 communicate with each other. The network 115 may include any type or combination of several different types of networks, such as cable networks, the Internet, wireless networks, and other private and / or public networks. In some cases the network may 115 Include cellular, Wi-Fi, or Wi-Fi Direct.

In general, the vehicle can 102 include any vehicle that has an optical surface 114 , a sensor assembly 116 , a controller 118 and a communication interface 120 (an optional feature for some embodiments). In various embodiments, the optical surface includes 114 a front or rear windshield of the vehicle 102 . For the sake of brevity and clarity, examples provided herein can be applied to the optical surface 114 than the front windshield of the vehicle 102 To refer to. The optical surface can be other surfaces within the vehicle 102 as discussed more fully herein.

Stray light 122 can be generated when a beam of light 123 coming from the light source 110 is emitted with foreign objects or obstacles 126 on the optical surface 114 interacts and is scattered by them. Any foreign object that is on the optical surface 114 present can cause stray light. Examples of foreign objects / material can include dust, cracks, pits, or other similar artifacts. The extent, opacity, and / or reflectivity of the obstacles 126 can create varying amounts of scattered light.

In some embodiments, the sensor assembly comprises 116 a first image sensor 128 and a light sensor 130 . In optional embodiments, the sensor assembly 116 a second image sensor 131 include. Additional details regarding the use of the second image sensor 131 are provided in more detail below. A picture 127 that through the first image sensor 128 was recorded, contains the objects 106 and 108 , being stray light 122 due to foreign objects or obstacles 126 on the optical surface 114 is available.

In some embodiments, the first image sensor could 128 include any of a camera, time of flight (TOF) camera, plenoptic camera, LIDAR, or other similar system that can be adversely affected by stray light. The scattered light can be a lens 132 of the first image sensor 128 affect when the camera takes pictures of the area of interest 104 received during vehicle operation. Thus, in some cases, the lens can 132 be understood as an example of an optical surface for the purposes of the present disclosure. Other examples could include a filter, beam splitter, or other equivalent structure.

In other embodiments, the first image sensor 128 due to its position behind the optical surface 114 from stray light 122 to be influenced. That is, while in some embodiments, light strays onto an obstacle or artifacts on the lens 132 of the first image sensor 128 can be attributed to stray light on the optical surface 114 occur. In some embodiments, stray light can strike an obstacle / artifact on both the lens 132 of the first image sensor 128 as well as on the optical surface 114 of the vehicle 102 be due. In addition, the systems, devices and methods in this document enable determinations as to whether the stray light is on an obstacle / artifact on the lens 132 of the first image sensor 128 and / or on the optical surface 114 of the vehicle 102 using correction and validation processes, which are described in more detail in this document.

A distinction can also be made between lens reflection and scattered light. Lens reflection occurs when light hits an optical component, such as a composite lens structure or sensor chip package, and is scattered within the optical components until it is ultimately absorbed by the sensor. This can cause image artifacts known as scattered light, but they are structured due to the symmetry of the scattering of the optical component and are expressed as an angle function of the camera light source angle rather than the lens angle. In some applications it can be the same angle. Assuming that the lens reflection is fixed apart from the angular dependence, an effect can be modeled for a given optical system to produce the function. The function is then reversed to remove lens reflection to isolate only the flare effect. In some cases, the lens reflection may not be fixed. The resulting glare has a different geometric relationship to the light source and the resulting glare shows more structure than a layer of dust that causes stray light.

In one embodiment, the first image sensor receives 128 Photos or other similar images, such as the photos from the 2A-2C . There is image degradation due to stray light in these photos. For example is in a first picture 200 ( 2A) when the vehicle is under a bridge 202 drives in the area of interest, little to no stray light is present. An object 204 such as a vehicle can be identified and tracked. A contrast or a darkness of the object 204 can from the first picture 200 to be determined. In some embodiments, the contrast may be determined relative to an initial or comparative determination that is in the controller 118 is preprogrammed (see 1 ). That is, determinations of the contrast can be made relative to comparison values in order to identify light versus dark objects in an image of an area of interest. In other embodiments, contrast discriminations are made using a plurality of images obtained over time, with a first of the images being used as a comparison value with respect to contrast. Exemplary contrast detection probability methods that can be used are in Geese, Marc, Ulrich Seger, and Alfredo Paolillo. "Detection Probabilities: Performance Prediction for Sensors of Autonomous Vehicles." Electronic Imaging 2018.17 (2018): 1-14, included.

As the vehicle pulls out from under the bridge, the contrast or darkness of the object changes 204 as in the second picture 206 out 2 B and a third picture 208 out 2C illustrated. There is also an area with stray light 210 in a second picture 206 present and enlarges, as in the third picture 208 illustrated. The stray light 210 increases from 2 B to 2C due to changes in light source angle, as discussed above. An intensity or size of the scattered light 210 can be relative to changes in the relative reflectivity of the object 204 can be determined over time, such as between images 2 B and 2C . When the relative brightness of the object 204 increases, it can be deduced that the scattered light 210 has increased. This can be confirmed by identifying a relative change in the angle of the light source.

Referring again to FIG 1 can control 118 to be configured by the sensor assembly 116 process received images, calculate light source angles, and assess the presence of stray light and other related features such as lens reflection. The control 118 can have a processor 134 and a memory 136 include. The memory 136 stores instructions issued by the processor 134 to perform aspects of flare detection and / or correction as disclosed herein. When referring to operations performed by the controller 118 it is understood that this is the execution of instructions by the processor 134 includes.

The control 118 can be configured to take images from the first image sensor 128 to receive and process the images to identify objects therein. The objects can be judged relative to one another based on the contrast. That is, the controls 118 can be configured to detect and track objects based on their relative darkness or lightness. Exemplary objects can include cars, signage, road surface markings, and so on. In other words, can be in the area of interest 104 Objects with different relative intensities, such as a road surface or a black car, are tracked to identify changes in their relative intensities over time.

The control 118 (or a specific configuration of the light sensor 130 ) can also specify a location / position of the light source 110 , such as the sun, relative to the optical surface 114 of the vehicle 102 determine. The controller can set a light source _{angle L A} based on the determined or assessed position / location of the light source 110 in terms of the optical surface 114 determine. In particular, the light source _{angle L A is defined} as the angle between the light beam 123 and an optical plane O _P which is coplanar to the optical surface 114 lies. In some embodiments, such as when the light source 110 is the sun, a position of the sun relative to the optical surface can be determined based on a known time of day, a direction of travel of the vehicle and a location of the vehicle, a road gradient, a vehicle inclination, a position of the sun can be determined.

In various embodiments, an installation angle of the optical surface 114 , which is determined relative to a ground plane, be known from manufacturer information. With the known angle of the optical surface 114 and the position / location of the light source 110 can control 118 determine the light source angle L _A. In some cases, the optical surface can have a curvature such that the relative angle to the light source can vary along the optical surface. In other words, one or more light sources can be detected and tracked, for example using camera-mediated detection of street lights and / or a device (such as the controller 118 ), which is configured to perform an indirect determination of light sources. For example, the controller can 118 Use localization (e.g. real-time positioning) and high-resolution maps to get a relative position of the light source 110 to the optical surface 114 to identify. Some embodiments may use light field data or weather data obtained from a resource.

The control 118 can be configured to detect the presence of stray light on the optical surface 114 to calculate or derive over two or more images as a function of the relative intensities of objects (eg lightness / darkness) compared to the light source _{angle L A as a function of time.} In accordance with some embodiments, light scatter determinations can be made using only a single image. In general, the controller can 118 be configured to determine the relative intensities of light and dark objects in images versus the light source angle L _{A over} time. In embodiments relating to the use of a single image, the controller may 118 Use previous data from a scene, such as the reflectivity of a street sign or dark sky near a light source. The control 118 can be a dynamic range of the camera, the light source geometry (e.g. image border), the light source intensity and / or use other parameters to estimate that the light source at the current angle of inclination is scattered light and lens reflection (e.g. increase in luminosity / brightness depending on the radius of the scattered light + dependence on lens reflection, which is dependent on the radius and angle (due to the camera symmetry) around the light source) would produce a fixed amount. When the amount of flare / lens reflection, e.g. B. the radius of the intensity / brightness increase is larger, this is an indication of an optical deterioration. The control 118 can also be configured to distinguish between the scattered light and lens reflection modes based on the structure (e.g. fast Fourier transform (FFT) of the lens reflection region after conversion to a polar coordinate system).

In various embodiments, the controller can 118 configured to track a first object within an area of interest contained in images of the area of interest captured over time by an image sensor (e.g., a first image sensor 128 ) of a vehicle were obtained while the vehicle was moving. The control 118 can track a light source within the area of interest. The control 118 may further be configured to determine a change in the relative reflection intensity of the first object based on a light source angle formed between the light source and an optical plane of an optical surface of the vehicle. Of course, the change in the relative reflection intensity indicates scattered light from the optical surface. In some embodiments, the controller may 118 also distinguish between objects that end up in a shadow, such as entering the vehicle entering the tunnel or under an overpass. The lighting on the object can change, so the controls 118 can be configured to distinguish between constant lighting conditions and varying conditions.

Table 1 includes columns with values determined from individual images obtained over time. A contrast of a road surface (e.g. first object 106 ) and a contrast of a car bumper (e.g. second object 108 ) are rated across multiple images. The relative reflectivity, referred to as the contrast change, is also provided. This change in contrast can indicate the presence of stray light. Thus, if no relative change in contrast is determined, it can be deduced that there is no scattered light. While using the 8-bit image intensity values in Table 1, the controller can 118 also calculate the luminescence of the object knowing the camera / ISP properties and settings (e.g. exposure). Table 1 Road 220 217 218 228 211 Car bumper relative 52 108 116 138 140 change in contrast 0% 26% 29% 38% 42% % Difference 124% 67% 61% 49% 40%

In Table 1 it is derived that, due to the relative increase in contrast, scattered light is present on the optical surface. Further, since the contrast changes for both objects, it can be confirmed that there is scattered light on the optical surface. It is found that the change in the relative contrast between the objects decreases, which is indicative of stray light. In particular, neither the dark car bumper becomes more reflective nor has the photon flow of the bumper changed, but rather light scattering due to dust on the windshield increasingly contributes to the measured image intensity or luminance in the scene.

In addition, computational methods other than tracking objects and their corresponding image intensities relative to the other parameters mentioned can be used to determine stray light. Other examples may include converting the image to an intensity histogram and providing the other parameters to a neural network. Alternatively, input, such as images, intensity, light source orientation, and other parameters previously described, may be provided to a neural network, which may include recurring convolutional layers, attention mechanisms, etc. Such a neural network can output a map of the size of the scattered light over the image. The network may also provide a map of the amount of lens reflection in the image, a corrected image with no flare, or other outputs (e.g., semantic segmentation map). In another example, the lens reflection parameters can be output by the neural network in order to detect deterioration in the lens or sensor optics, e.g. B. failure of anti-reflective thin films.

In another embodiment, the controller 118 be configured to use a histogram of the intensity and / or the intensity gradients relative to the relative light source angle to determine the effects of stray light.

In general, the controller can 118 be configured to track changes in the relative reflection intensity of a single object or a plurality of objects present in the area of interest 104 is / are detected. The one from the controller 118 The scattered light determination carried out could be an average of the relative reflection intensities over all objects or only some of the objects. Again, the changes in the relative reflection intensity are made in the course of time relative to the light source angle L _A and how the light source _{angle L A} also changes in the course of time.

In some embodiments, the controller may 118 configured to perform pixel-level analysis on images captured by the first image sensor 128 can be obtained (see 3 as an an example). Scattered light, which is an amount of diffuse lighting due to obstructions 126 on the optical surface 114 depends on the dust / obstacle level and the light source angle L _A assuming that all other variables are kept constant. Other variables or factors can include exposure, gain, object orientation in space relative to the lighting source, lighting level, dynamic range, multiple exposure settings, effects of LED flicker, and so on. If these variables are present, some corrections may be needed. These corrections can include converting the brightness value into luminosity or estimated object reflectivity in the scene. For example, an object such as a dark bumper may change the relative angle between the light source, the surfaces of the object and the camera due to vehicle movement, e.g. B. Turn, change. Based on the image intensity value or the calculated luminosity, the object would suddenly appear brighter. In reality, the increased brightness can be fully explained due to Lambert's law of cosines, in which a radiation intensity observed from a diffuse surface is directly proportional to the cosine of the angle between the viewing angle and the surface angle, as well as changes in the photon flux (assuming a dominant point light source such as the sun ). Therefore, the reflectivity has remained constant and the increase in perceived object brightness can be fully explained due to the aforementioned physical effects, rather than additional light scattering due to a sudden increase in dust on the optical surface.

Of course, the local position of objects as recorded by the camera within the single image can be a factor for non-uniform obstacles 126 on the optical surface 114 be. For example, another vehicle may move in a perpendicular direction relative to the viewing camera mounted on the vehicle. The optical surface may have dust on one side of the optical path, e.g. B. the right half of the windshield in the region of the optical path of the camera. This localized layer of dust may not be directly detectable using machine vision techniques (this could be due to many reasons, such as missing features, effectively translucent, and fuzzy). The object's image intensity, calculated luminosity, and estimated reflectivity may change as the object moves, so the position moves from the left side of the image where the optical surface is clean and the right side of the image where the the optical surface is locally dusty, shifts. In the case of a dark vehicle, the pixels in the image belonging to the vehicle would increase in intensity and luminosity beyond the individual images, assuming that other factors are kept constant (e.g. position of the relative light source angle to the optical surface the camera). This effect would be due to the larger number of photons which are detected due to the light scattering from the dust layer on the optical surface relative to the number of photons reflected from the darkly colored vehicle surface. This local position can be controlled by the controller 118 can be used to identify average and local obstacles, such as per pixel, levels due to spatially varying scattered light. For example, in 3 (in combination with 1 ) an area with scattered light 300 on an optical surface 302 available. This stray light 300 can generally be identified using the methods described herein (e.g. as a function of relative intensities of objects (e.g. lightness / darkness) versus the light source angle L _{A over} time). In order to determine more precise details regarding the obstacle or other causes of the scattered light, the area with scattered light 300 on the image can also be evaluated at the pixel level. That is, the controls 118 can be an area 304 identify where the obstacle (s) is / are denser compared to areas 306 in which the obstacle is less dense. As noted above, this difference results in spatially varying scattered light. The areas of greater density or thickness of the obstacle may produce scattered light that is larger than areas where the obstacle is less dense (e.g. more light is scattered on the optical surface). It should also be noted that all optical surfaces, even if they are free of obstructions and none Deterioration, such as B. scratches, produce some amount of light scattering in relation to the relative light source angle. Additional obstacles and deterioration of the optical surface increase this light scattering. In short, the controller can 118 be configured to determine variances of the scattered light on the optical surface and to identify positions of obstacles on the optical surface based on the variances. The control 118 may determine variances based on changes in any of an offset, size, or functional form of change in the relative reflection intensities of the objects.

In some embodiments, a curvature of the optical surface can be taken into account when making stray light determinations. That is, since the scattered light is _{dependent on the light source angle L A} , differences in the light source angle L _A due to a curvature in the optical surface may cause a change in the scattered light that is not due to variations in the extent of the obstacle. The control 118 can compensate for these variations in light source angle L _A in some embodiments.

It goes without saying that scattered light levels can have a characteristic curvature, wherein the obstacle levels can influence an offset, a size and / or a functional form of the change in the relative object intensities. This curve can be measured in a laboratory or in a road test with known obstacle levels (e.g. dust) on the optical surface. Thus, a curve or a relative change in performance can be controlled by the control 118 normalized to a known obstacle level (s). In addition, changes in the lens reflex can be correlated with laboratory tests. For example, a lens can be measured in its nominal condition and after an anti-reflective coating has been removed to measure the increase or change in lens reflection relative to the relative orientation of the light source.

According to some embodiments, the scattered light can be caused by the presence of fog in the area of interest 104 to be influenced. Thus, the vehicle can 102 in some embodiments, a humidity sensor 138 which acquires data that can be analyzed for the presence of fog. A temperature sensor could also be used in some embodiments to indicate the likelihood of the presence of fog. In other embodiments, fog can be in the area of interest 104 be determined by the controller 118 weather-related data from a weather service 144 receives. The control 118 can over the network 115 through their communication interface 120 or access the weather service via a V2X connection (vehicle-to-everything).

As soon as stray light has been detected, the control can 118 be configured to activate a corrective action in response to the stray light. Corrective action could be activating a cleaning assembly 140 of the vehicle 102 include. The cleaning assembly 140 could for example be a windshield wiper system of the vehicle 102 to include the delivery of wiper fluid and activation of the wipers. This corrective action can reduce the stray light by removing at least some of the obstacles 126 on the optical surface 114 Will get removed. Of course, any remaining obstacles are likely to cause permanent deterioration of the optical surface 114 specify.

In another example, the corrective action may include the controller 118 an attribute or a parameter of an extended driver assistance system 142 or an autonomous vehicle operation. For example, the extended driver assistance system 142 a warning or message to a driver of the vehicle 102 output that stray light is detected and that the driver should activate his windshield wipers manually in order to clean the optical surface. In another example, the extended driver assistance system 142 prevent the driver from driving in the direction of the scattered light. In another example, the extended driver assistance system 142 a speed of the vehicle 102 automatically reduce when stray light is detected. The vehicle can be stopped based on an amount of the scattered light in some scenarios. Each of these options can also be carried out by a function or control of an autonomous vehicle or. In addition, the controller could 118 instead of activating a feature, deactivate an ADAS feature and provide the driver with a message or warning about such a change. For example, the controller can 118 cause the ADAS to slow down or stop the vehicle.

The control 118 may use the effect of the functional form of flare or lens reflection on an image to restore some of the original image quality for use in a display (rear Rear view camera) or for use in ADAS (removal of lens reflection ghosting effects to reduce false positives in machine vision).

In some embodiments, the processes for determining stray light can be used to verify a cleaning procedure. For example, the controller can 118 as soon as the controller 118 has determined that stray light is present and has caused a cleaning process to occur, reevaluate the presence of stray light on the optical surface after the cleaning process to determine whether the size of the stray light has changed. That is, a size of the scattered light at a first point in time by the controller 118 is determined. The size of the scattered light is determined by the controller at a second or later point in time 118 certainly. As noted above, any remaining stray light can be caused by permanent damage to the optical surface.

In general terms, the controller can validate the effectiveness of contaminant / obstacle removal by noting a change in the scattered light after contaminant / obstruction removal is removed from the optical surface 114 certainly. When the size of the flare light decreases from the first point in time to the second point in time, it is determined that the cleaning procedure has affected the flare light. The extent of this effect is represented by the percentage change in the size of the scattered light, which decreases from the first point in time to the second point in time. If no change in the size of the scattered light has occurred, the controller can do this 118 it can be deduced that a fault has occurred in the cleaning assembly (e.g. inoperative windshield wipers or missing wiper fluid).

No change in the size of the scattered light can also indicate that the scattered light hits obstacles on the lens 132 of the first image sensor 128 is due. A warning or message can be presented to the driver through a voice command system or other man-machine interface that the lens 132 of the first image sensor 128 should be checked and cleaned. Subsequent determinations of changes in the size of the scattered light can be made after each iterative step to determine whether a root cause for the scattered light has been detected and corrected. In cases where no root cause is identified or the stray light cannot be eliminated, the controller can 118 deduce that the obstacles that create the scattered light are structural in nature and are related to the optical surface. For example, the optical surface can be scratched or pitted. Warning messages can also be designed to inform the driver that the optical surface may be damaged.

According to some embodiments, scattered light can be an indication of deterioration or failure of a coating. For example, it is expected that after the cleaning action with a water-based spray, the moving water droplets can remove dirt on a lens or windshield. If the coating fails, the droplets can form a thin film of water that is not as effective at removing contaminants. This is another scenario in which the stray light is not removed by cleaning processes.

In an exemplary use case, systems, devices, and methods of the present disclosure can be used for multiple camera types and configurations, such as cameras mounted on an AV tiara (e.g., housing on top of a vehicle), front-facing cameras that are mounted behind a windshield, rear or right and left facing cameras mounted on exterior body locations, or even indoor / driver surveillance cameras. For example, a camera used to monitor the driver's vigilance can detect stray light from the rear glass surface of the hatchback of the car / truck / SUV. If it is detected, the scattered light indicator can be used to automatically activate the cleaning of the rear glass.

As noted above, the controller 118 programmed with data obtained using control or laboratory tests. This data can be used as a comparison value. For example, a laboratory test can be used to correlate the severity of the scattered light with the level of dirt obstacle. This can also result in negative non-scattered light effects such as blurring, a decrease in the signal-to-noise ratio (SNR), a decrease in the contrast detection probability (CDP) and / or a decrease in the modulation transfer function (modulation transfer function - MTF) can be correlated relative to the imaging capability of the camera.

As noted above, the vehicle can 102 the first image sensor 128 and a second image sensor 131 include. The output of these sensors can be cross-checked by the controller 118 be used. That is, the main cause of the scattered light such as a layer of dust can be located on the lens or the windshield, can be located. In AV applications there are often several cameras that perform object detection and sensor fusion. In addition, other sensor modalities that detect light can be used in addition to or instead of images / photos. Of course, the disclosed methods can be adapted for use using other sensor modes (LIDAR, TOF cameras) in which ambient lighting can be imaged or included in an equation for measuring the range. In addition, dust can help reduce the range of the sensors or deflect the lidar beam (active lighting), which causes the incorrect angular position to be measured in space, which leads to depth estimation errors. The presence of stray light would reduce the effectiveness of these sensors. Methods disclosed herein can be used to detect failure modes such as dust on a LIDAR housing using the environmental imaging mode of some LIDAR / TOF cameras.

It would be expected that the scattered light would reduce the signal-to-noise ratio or the probability of contrast detection of that region of the field of view of the camera, thereby reducing the probability of the detection, the correct classification of the detection or the misclassification. Objects that were or were not detected in the camera's field of view (relative to other overlapping image sensors 128 / 132 ), which currently has a layer of dust detected by the scattered light method, can be abandoned (or eliminated in some embodiments) and other sensor detection of objection can be emphasized compared to the nominal sensor drive during sensor fusion.

That is, the second image sensor 131 can be used to track a first object within the area of interest. The control 118 can be configured by the second image sensor 131 Ignore provided images when the first object is in by the second image sensor 131 generated images is not detected, but the first object is detected in the images of the region of interest, which by the first image sensor 128 can be obtained. If there are multiple image sensors, additional confidence can be gained through a multiplicity of images that are provided by multiple image sensors. If objects are only present in certain images, the control can 118 infer that the position of some of an image sensor is such that the object is not present in the images generated by the image sensor or that there is an error condition related to the image sensor.

The features of the present disclosure can be applied to specific scenarios, such as when the image sensor (s) include LIDAR. For example, if stray light increases due to dust on a lidar package in combination with the position of the sun (e.g., light source angle), a noise floor of the lidar sensor may increase. This would reduce the signal-to-noise ratio (SNR) and the detection ability at greater distances. The control 118 may be configured to determine a light source position according to any method disclosed herein (other sensors or knowledge of vehicle orientation, time of day, position of the sun, and so on). A determination of the noise floor can be carried out for each position of the LIDAR housing in relation to the light source angle of the light source. Such application of the present disclosure can detect contamination on the lidar housing (e.g., dust) and enable cleaning systems to be activated.

Scattered light detection can also be used when driving at night. Now the 4A-4C Referred to, which together illustrate an environment in which stray light can be detected when driving at night. In 4A drives a vehicle 400 a street at night 402 along, being a dark region of the sky 404 is available. A street lamp 406 (or other similar light source) is present and illuminates a section of the road 402 . In general, the street lamp produces 406 due to lens reflections and obstructions on a vehicle windshield 400 are present, a lens reflection region 408 .

When the vehicle 400 the street lamp 406 in 4B approaches, becomes the lens reflection region 408 the street lamp 406 due to a change in an angle of incidence (e.g., the light source angle) between the street lamp 406 and the windshield of the vehicle 400 greater. The enlargement of the lens reflection region 408 can also be partially due to obstacles on the windshield of the vehicle 400 be due. When the vehicle 400 the street lamp 406 in 4C approaches even further, becomes the lens reflective region 408 even larger and their shape becomes more irregular. This change in Size / shape of the lens reflection region 408 is based on further changes in the angle of incidence (e.g. the light source angle) between the street lamp 406 and the windshield of the vehicle 400 . In some embodiments, a gradient of the lens reflection region can be 408 per pixel and relative to a distance from the street light 406 and the angle of incidence (e.g. the light source angle) can be measured.

Some embodiments disclose that the processes for detecting and correcting stray light are performed at the control plane. Other embodiments may include the service provider 112 carries out the processes for the detection and / or correction of scattered light. In some embodiments, the methods disclosed herein for the detection of stray light by the service provider 112 carried out while the corrective actions at the vehicle level by the controller 118 implemented. In general, the controller 118 Get pictures and send them to the service provider 112 hand off. The service provider 112 can process the images to detect stray light. If stray light is detected, the service provider can 112 a message / signal to the controller 118 transmitted to activate a corrective action.

5 Figure 3 is a flow diagram of an exemplary method of the present disclosure. The process generally includes one step 502 tracking a first object within images of a region of interest obtained over a period of time by an image sensor of a vehicle during vehicle movement. Of course, several objects can be tracked over the same period of time using one or more image sensors.

Next, the method includes a step 504 tracking a light source within the area of interest. Tracking of the light source (such as the sun or a street lamp) can be accomplished using the images or position data of the vehicle. For example, a position of the sun can be determined if a time of day, a direction of travel of the vehicle and a location of the vehicle are known.

According to some embodiments, the method can include a step 506 determining a change in relative reflection intensity of the first object based on a light source angle formed between the light source and an optical plane of an optical surface of the vehicle. Of course, this process can include determining changes in the light source angle formed between the light source and the optical plane over time. In fact, the change in the relative reflection intensity indicates scattered light from the optical surface. In response to determining that stray light is present, the method may include a step 508 of activating a corrective action in response to the stray light. As noted above, corrective action may include activating a cleaning assembly of the vehicle to reduce stray light by removing contaminants from the optical surface.

In the above disclosure, reference was made to the accompanying drawings, which form a part hereof and illustrate specific implementations in which the present disclosure may be practiced. It goes without saying that other implementations can also be used and structural changes can be made without departing from the scope of the present disclosure. References in the specification to “an embodiment,” “an exemplary embodiment,” “an exemplary embodiment,” etc. indicate that the described embodiment may include, but are not necessarily limited to, one particular feature, structure, or characteristic Embodiment must include that particular feature, structure or property. In addition, such formulations do not necessarily refer to the same embodiment. Further, when a particular feature, structure, or property is described in connection with an embodiment, those skilled in the art will recognize such a feature, structure, or property in connection with other embodiments, whether or not so is expressly described or not.

Implementations of the systems, devices, devices, and methods disclosed in this document may include or utilize a special purpose or general purpose computer that includes computer hardware, such as one or more processors and system memory, as discussed in this document. Implementations within the scope of the present disclosure may also include physical and other computer-readable media for transporting or storing computer-executable instructions and / or data structures. Such computer readable media can be any available media that can be accessed by a general purpose or special purpose computer system. Computer-readable media that store computer-executable instructions are computer storage media (devices). In the case of computer-readable media, the computer-executable instructions transport, they are transmission media. Thus, by way of example and not by way of limitation, implementations of the present disclosure may include at least two distinctly different types of computer readable media: computer storage media (devices) and transmission media.

Computer storage media (devices) include RAM, ROM, EEPROM, CD-ROM, solid state drives (SSDs) (e.g., based on RAM), flash memory, phase-change memory (PCM), others Memory types, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code means in the form of computer-executable instructions or data structures, and that can be accessed by a general purpose or special purpose computer.

An implementation of the devices, systems and methods disclosed in this document can communicate via a computer network. A “network” is defined as one or more data connections that enable the transport of electronic data between computer systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computer over a network or other communications link (either hardwired, wireless, or any combination of hardwired or wireless), the computer sees the connection as a transmission medium for the purpose. Transmission media can include a network and / or data links that can be used to carry desired program code means in the form of computer-executable instructions or data structures, and that can be accessed by a general purpose or special purpose computer. Combinations of the foregoing should also be included within the scope of computer readable media.

Computer-executable instructions include, for example, instructions and data which, when executed on a processor, cause a general purpose computer, special purpose computer, or special processing device to perform a particular function or group of functions. The computer-executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, or else source code. Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the features or acts described above. Rather, the features and acts described are disclosed as exemplary forms of implementing the claims.

Those skilled in the art will understand that the present disclosure can be implemented in network computing environments with many types of computer system configurations, including dashboard vehicle computers, personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, consumer electronics microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, cell phones, PDAs, tablets, pagers, routers, switches, various storage devices, and the like. The disclosure can also be implemented in environments with distributed systems in which both local computer systems and remote computer systems that are connected by a network (either by hardwired data connections, wireless data connections or by any combination of hardwired and wireless data connections), tasks carry out. In a distributed systems environment, program modules can reside in both local and remote storage devices.

Furthermore, the functions described in this document can optionally be carried out in one or more of hardware, software, firmware, digital components or analog components. For example, one or more application specific integrated circuits (ASICs) may be programmed to perform one or more of the systems and operations described in this document. Certain terms that are used throughout the description and claims relate to specific system components. For the person skilled in the art it is obvious that the components can be referred to by other names. In this document, no distinction should be made between components that differ in terms of their names, but not in terms of their function.

It should be noted that the sensor embodiments discussed above may include computer hardware, software, firmware, or any combination thereof to perform at least a portion of their functions. For example, a sensor may include computer code configured to run on one or more processors and may include hardware logic circuitry / include electrical circuitry controlled by computer code. These exemplary devices are provided in this document for purposes of illustration and are not intended to be limiting. Embodiments of the present disclosure may also be implemented in other types of devices as known to those skilled in the relevant art (s).

At least some embodiments of the present disclosure have been directed to computer program products that include such logic (e.g., in the form of software) stored on any computer usable medium.

Such software, when executed in one or more data processing devices, has the effect that a device is operated as described in this document.

While various embodiments of the present disclosure have been described above, it should be understood that these are presented as examples and not of limitation. Those skilled in the relevant art will recognize that various changes in form and detail can be made therein without departing from the spirit and scope of the present disclosure. Thus, the breadth and scope of the present disclosure should not be limited by any of the exemplary embodiments described above, but should be defined only in accordance with the following claims and their equivalents. The foregoing description has been presented for purposes of illustration and description. It does not claim to be complete and is not intended to limit the present disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. It should also be noted that any or all of the foregoing alternative implementations can be used in any desired combination to form additional hybrid implementations of the present disclosure. For example, any of the functions described with respect to a particular device or component can be performed by a different device or component. Furthermore, while specific device properties have been described, embodiments of the disclosure may relate to numerous other device properties. Furthermore, it should be understood that although embodiments have been described in language specific to structural features and / or methodological acts, the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as illustrative forms of implementing the embodiments. Formulations that express conditional relationships, such as “can”, “could”, “can” or “could”, among others, are generally intended to convey that certain embodiments could include certain features, elements and / or steps, whereas other implementations these cannot be included, unless something else is specifically stated or something else results from the context used in each case. Thus, such formulations expressing conditional relationships are not generally intended to imply that features, elements, and / or steps are required in any way for one or more embodiments.

In accordance with an embodiment of the present invention, the sensor assembly is further configured to detect fog within the area of interest.

According to one embodiment, the processor is configured to track a second object, the second object having a relative reflection intensity that is different from the relative reflection intensity of the first object.

According to one embodiment, the sensor assembly further comprises a light detection and ranging (LIDAR) device, further wherein the processor is further configured to calculate a noise floor for various positions of a housing of the LIDAR device relative to the light source angle.

According to one embodiment, the corrective action includes the processor further configured to activate a cleaning assembly of the vehicle to remove contaminants from the optical surface.

In one embodiment, the sensor assembly is further configured to track the first object within the area of interest using a first image sensor, further wherein the processor is configured to ignore images provided by a second image sensor when the first object is not is detected in the images, but the first object is detected in the images of the region of interest obtained by the first image sensor.

Claims (15)

Method comprising: Tracking a first object within images of a region of interest obtained over a period of time by an image sensor of a vehicle during vehicle movement; Tracking a light source within the area of interest; Determining a change in relative reflection intensity of the first object using the images based on a light source angle formed between the light source and an optical plane of an optical surface of the vehicle, the change in relative reflection intensity indicating stray light or lens reflection of the optical surface; and Activate a corrective action in response to the scattered light or lens reflection. Procedure according to Claim 1 , further comprising determining changes in the light source angle over the period of time. Procedure according to Claim 1 , further comprising detecting fog within the area of interest. Procedure according to Claim 1 , further comprising tracking a second object, the second object having a relative reflection intensity that is different from the relative reflection intensity of the first object. Procedure according to Claim 4 , further comprising: determining variances of the scattered light on the optical surface; and identifying positions of obstacles on the optical surface based on the variances. Procedure according to Claim 5 wherein the variances are determined based on changes in any of an offset, magnitude, or functional form of change in relative reflection intensities of the first object and the second object. Procedure according to Claim 1 wherein the corrective action comprises activating a cleaning assembly of the vehicle to reduce the stray light by removing contaminants from the optical surface. Procedure according to Claim 1 further comprising validating the effectiveness of contaminant removal by determining a change in scattered light after removing contaminants from the optical surface. Procedure according to Claim 1 wherein the corrective action comprises changing a parameter of an extended driver assistance system or an autonomous vehicle operation. Apparatus comprising: a processor; and a memory for storing instructions, the processor configured to carry out the instructions to: Tracking a first object within images of a region of interest obtained over a period of time by an image sensor of a vehicle during vehicle movement; Tracking a light source within the area of interest; Determining a change in the relative reflection intensity of the first object based on a light source angle formed between the light source and an optical plane of an optical surface of the vehicle, the change in the relative reflection intensity indicating stray light of the optical surface; and activating a corrective action in response to the stray light. Device according to Claim 10 , wherein the processor models lens reflection using the scattered light. Device according to Claim 10 wherein the processor is further configured to detect fog within the area of interest based on any of a humidity sensor of the vehicle or weather data received from a weather data service over a network. Device according to Claim 10 wherein the processor is further configured to: determine variances of the scattered light on the optical surface; and identifying positions of obstacles on the optical surface based on the variances. Device according to Claim 13 wherein the processor determines the variances based on changes in any of an offset, magnitude, or functional form of change in the relative reflection intensities of the first object and the second object. System comprising: a sensor assembly for: Tracking a first object within images of a region of interest obtained over a period of time by an image sensor of a vehicle during vehicle movement; Tracking a light source within the area of interest; and Determining changes in a light source angle over time; and a controller, having a processor and a memory, the processor configured to execute instructions stored in the memory for: Determining a change in the relative reflection intensity of the first object based on the light source angle formed between the light source and an optical plane of an optical surface of the vehicle, the change in the relative reflection intensity indicating stray light of the optical surface; and Activate a corrective action in response to the stray light.

Images