DE102018008442A1 - Method for weather and / or visibility detection - Google Patents

Abstract

The invention relates to a method for weather and / or visibility detection. According to the invention, raw data from at least one lidar sensor of a vehicle is evaluated by machine learning to determine visibility in a vehicle environment, where evaluated results are fused to results from other types of vehicle environment sensing sensors, wherein Each environment detection sensor determines and / or classifies a performance that is taken into account in a sensor fusion.

Description

The invention relates to a method for weather and / or visibility detection.

From the prior art, as in the US 8,983,705 B2 described a method for detecting weather conditions, including fog, known by vehicle sensors. The method includes receiving laser data collected by scanning an environment of the vehicle and associating laser data points of one or more objects in the environment with a computing device. The method also includes comparing laser data points that are not associated with the one or more objects in the environment with stored laser data points that are representative of a pattern due to fog, and based on the comparison, identifying an indication that a Weather condition surrounding the vehicle contains fog.

The invention is based on the object to provide a comparison with the prior art improved method for weather and / or visibility detection.

The object is achieved by a method for weather and / or visibility detection with the features of claim 1.

Advantageous embodiments of the invention are the subject of the dependent claims.

In a method according to the invention for weather and / or visibility detection, in particular by means of lidar, raw data of at least one lidar sensor of a vehicle is evaluated by means of machine learning in order to determine visibility conditions, for example rain, fog or clear vision, in a vehicle environment.

Evaluated results are fused to results from other types of vehicle's environmental detection sensors, for example radar, with performance, e.g. H. a performance, determined and / or classified, which is taken into account in a sensor fusion.

Environmental detection sensors are disturbed to different degrees by external influences, for example weather conditions. In the case of a fusion of sensor data, an actual impairment of the individual subcomponents, in particular of the individual environment detection sensors, is a very important information.

By means of the method according to the invention can be closed by learning-based method based on features of the sensor raw data on the state of the environment, for example, whether rain, fog or clear view is present.

Each individual environment detection sensor recognizes its current performance status, d. H. its current performance, even, this can be used specifically in the sensor fusion. Thus, for example, detections of the lidar sensor from local fog, for example, by exhaust gases from a vehicle in front, may be filtered based on the radar data.

Embodiments of the invention will be explained in more detail below with reference to a drawing.

• 1 schematisch eine Analyse der Umwelteinflüsse auf Lidar-Punktwolken.

Showing:

• 1 schematically an analysis of the environmental influences on lidar point clouds.

Environmental awareness is a major challenge in autonomous driving and the major constraint on system availability and performance. In order to improve both the degree of automation and the availability of the system, different environmental conditions must be observed and recognized by the system and it must be responded to. In order to develop a robust detection and sensor fusion, it is of utmost importance to know the potential degradation of different sensor types and to reduce their influence. In order to develop a truly autonomous vehicle, system constraints need to be recognized without external intervention, and it is necessary to respond accordingly to control environmental impairments. According to the definition of SAE automation levels, Level 3 and 4 systems require the detection of system constraints to pass to a driver (Level 3) or to the system in to bring a safe state (level 4). Full automation (level 5) requires mastery of all environmental conditions.

Therefore, the study of environmental influences on sensor performance is an important research area for autonomous driving. Sensors are confronted with ever-changing and always unique weather conditions. In particular, since sensors are constantly influenced by the weather during the entire vehicle operation.

For lidar sensors, the toughest environmental conditions are bright sunshine, heavy fog, rain, dirt and drizzle. In this work the influence of rain and dense fog on available lidar sensors is investigated.

It is important to recognize and quantify the impact of weather on lidar performance in order to develop robust recognition and fusion algorithms. First, it is important for the fusion and behavioral planning of the autonomous vehicle to reliably classify the current sensor performance to optimally weight the modalities of the sensor. Therefore, the evaluation of the characteristics of different environmental conditions is important for a specific sensor system. The main contribution of this work is the detailed analysis of the influence of different weather conditions on the point cloud of different lidar sensors. The recorded data set provides controlled fog density in the meteorological field of view from 10m to - 90m.

In addition to determining the deterioration of the sensor performance itself, it is also important to reliably detect and classify environmental conditions with the sensor itself. In this thesis, a new approach to the detection of weather conditions is developed only by evaluating the output data of a lidar sensor. Therefore, the sensor itself determines its current performance and provides important information for each type of sensor fusion algorithm. Therefore, the fusion algorithm can use this information to combine two different types of sensors to filter out specific disadvantages of a sensor.

The approach was validated on a data set in a controlled environment with different fog visibility, different types of fog and a kind of stabilized rain at 55 mm / h. The proposed algorithm for environmental classification was also verified with a dataset taken in different scenarios on a test site and on roads in real traffic situations in clear weather conditions

Experimental construction

Defined and reproducible weather conditions are important in order to obtain a meaningful record of detection sensors in different weather conditions. In Europe, to the knowledge of the author, the CEREMA climatic chamber in Clermont-Ferrand (France) is the only climatic chamber capable of producing homogeneous advection and radiative fog with a controlled and constant visibility range. The chamber also has a rain simulator with a stabilized rainfall rate of 55mm / h.

On the basis of the data set in the climatic chamber, a detailed evaluation of the sensor output data was carried out in clear, foggy and rainy conditions. In addition, a classification method was developed to obtain information about the weather conditions from the point cloud of the laser scanner. Four different configurations were included with static and dynamic scenarios to reduce the time correlation of the data set. The dynamic scenario configuration includes pedestrians, cyclists, and cars. As static objects reflective road boundary posts, lane markers, raised mark buttons on the roadway, dolls to represent a child and a woman, a car and a doll for displaying a man with a reflective vest were provided.

The static setup may include retroreflective and diffuse reflective objects with 5%, 50%, and 90% reflectivity. A first dynamic setup may represent typical scenes of real traffic situations (e.g., pedestrian or cyclist crossing, approaching or departing vehicles, etc.) with different variants and combinations. For example, a departing vehicle may be provided in its own lane and multiple static objects, as is typical of real traffic situations. A second dynamic setup may include diffuse reflective targets that leave the field of view of the sensor. Accordingly, the impact of changing weather conditions and changing scenarios can be analyzed separately.

The data set recorded in the climate chamber contains about 274,000 images of the VLP16 sensor and 105,000 images of the Scala sensor. The road record included five shots without any rain (four in the day and one at night), three with occasional rain and four with almost constant rain (three in the day and one at night) and was recorded with the VLP16 sensor (270000 images ).

method

First, the raw data of the laser scanner was evaluated. The distortion of the point cloud by fog and rain is the basis for the feature extraction and for training a classification algorithm. For this purpose, the point cloud at time k has been represented as matrix P∈ℝ ² , where each row i = (1, ..., n) contains a point with a number j = (1, ..., m) of attributes , $$P{\left(k\right)}^{n\times m}=\left(\begin{array}{llll}{p}_{11}\left(k\right)\hfill & p_{12}\left(k\right)\hfill & ...\hfill & p_{1m\left(k\right)}\hfill \\ p_{21}\left(k\right)\hfill & p_{22}\left(k\right)\hfill & ...\hfill & p_{2m}\left(k\right)\hfill \\ ⋮\hfill & ⋮\hfill & \hfill & ⋮\hfill \\ p_n\left(k\right)\hfill & p_{n2}\left(k\right)\hfill & ...\hfill & p_{nm}\left(k\right)\hfill \end{array}\right),$$

wobei jede Spalte ein Attribut repräsentiert. Um eine klare Notation zu erhalten, ist der Spaltenindex einer Punktwolke nicht explizit notiert, sondern mit einer Variable je Spalte markiert. $$p_i=\left(x_1,y_i,z_i,r_i,{\theta }_i,{\phi }_i,c_i,I_i,cp{w}_i\right).$$

The number of attributes of a point is constant over all frames and sensor specific. To ensure a universal structure, the union is represented by all sensor specific signals in $$pij=\left(\mathrm{<figure-callout\; id="1"\; label="p\; i"\; filenames="DE102018008442A1\_0009.png"\; state="\{\{state\}\}">p\; </figure-callout>}i\mathrm{1,...,}pij\mathrm{,\; ...,}pim\right)$$

where each column represents an attribute. To get a clear notation, the column index of a point cloud is not explicitly noted, but marked with a variable per column. $$p_i=\left(x_1.y_i.z_i.r_i.{\theta }_i.{\phi }_i.c_i.I_i.cp{w}_i\right),$$

The corresponding notation is (x, y, z) for the Cartesian and (r, θ, φ) for spherical coordinates, e for the echo number, I for the intensity and epw for the echo pulse width.

The number of valid points i = (1, ..., n) varies from frame to frame and depends on the environment of the laser scanner. The influence of fog and rain on the attributes of the point cloud was evaluated by the following calculated signals.

A distinction by the echo number t∈ℕ, which represents the first, second or third echo signal of a transmitted light pulse, is reasonable, since the number of echoes per scanning direction is related to the number of particles and thus possible scattering points. Therefore, the set M _{t is} defined as M _t : = {e _i | e _i = t} with the maximum number of echoes t ∈ {1, 2, 3}. For the number of points for a specific echo, the signal Nt (k) is derived: $$N_t\left(k\right)=\left|e_i\left(k\right)\right|\forall e_i\in M_t,$$

Consequently, the number of all points N (k) in a frame is the sum of all N _t (k):
The mean and variance of an attribute p _j is calculated for each frame by: $$\overline{p_j}\left(k\right)=\frac{1}{n}{\displaystyle \underset{i=1}{\overset{n}{\Sigma }}{p}_{ij}\left(k\right)}\text{var}\left(p_j\left(k\right)\right)=\frac{1}{n}{\displaystyle \underset{i=1}{\overset{n}{\Sigma }}{\left(p_{ij}-\overline{p_{ij}}\left(k\right)\right)}^2,}$$

For example, the average distance of all points corresponding to a specific echo is given by: $$\overline{r_t}\left(k\right)=\overline{r_i}\left(k\right)\forall {\text{c}}_i\in M_t,$$

The assignment of the resulting feature vector f = (f ₁ ,..., F ₁₆ ) ^T for an environmental classification based on point cloud data is shown in Table I. In the feature set of the VLP16 sensor, the echo pulse width epw is replaced by the intensity I. Table I f ₁ N ₁ f ₂ N ₂ f ₃ N ₃ f ₄ r ₁ f ₅ r ₂ f ₆ r ₃ f ₇ eig (cov (x)) f ₈ eig (cov (y)) f ₉ eig (cov (z)) f ₁₀ var (EPW) f ₁₁ var (e) f ₁₂ mean (r) f ₁₃ mean (θ) f ₁₄ mean (φ) f ₁₅ mean (EPW) f ₁₆ mean (e)

In addition, a density measurement of the point cloud in mean and variance of the spherical coordinates (r, θ _y , φ _i ) is represented. Accordingly, the variables r and θ are derived for specific range or angle sections. The spatial distribution of the points is represented by the eigenvalues of the covariance matrices of x, y and z. Since the greatest influence of ambient conditions in the near range is expected, the point cloud was spatially filtered in a preprocessing step.

Based on these derivatives, the feature vector f was passed to describe a frame of the laser scanner. The influence of the features was analyzed by neighborhood component analysis to take into account only significant features. Nevertheless, features that are significant only because of the local environment need to be identified. For example, the absolute number of points was not used as a feature for the weather classification because it is highly dependent on the scenario (empty road versus congestion in the city center). For a robust description of the environment and to avoid over-fitting to local features, several scenarios have been taken into account when analyzing significant properties.

The assignment of the resulting feature vector f = (f ₁ , ..., f ₂₀ ) with 20 features is shown in Table II.

Table II. Feature vector for environment classification based on point cloud data. The influence of the features was demonstrated by an analysis of neighboring components. For ease of representation, only the echo pulse width epw is listed, which could be replaced by the intensity I depending on the sensor type. f ₁ N ₁ f ₅ r ₂ f ₉ eig (cov (z)) f ₁₃ mean (θ) f ₁₇ mean (r) f ₂ N ₂ f ₆ r ₃ f ₁₀ var (EPW) f ₁₄ mean (φ) f ₁₈ mean (θ) f ₃ N ₃ f ₇ eig (cov (x)) f ₁₁ var (e) f ₁₅ mean (EPW) f ₁₉ var (r) f ₄ r ₁ f ₈ eig (cov (y)) f ₁₂ mean (r) f ₁₆ mean (e) f ₂₀ var (θ)

experimental results

The derived signals of the previous section are first discussed for different weather conditions. Subsequently, a classification method is presented to derive the weather conditions from the point cloud of the laser scanner. For this the three classes 'clear', 'fog' and 'rain' are defined. The ground truth measurements for the meteorological visibility V in m and the rainfall rate R in mm / h were provided by the climate chamber.

Weather influence on point clouds

Comparing the point cloud in clear weather conditions with rain at 55 mm / h shows that the range of the sensor is limited, as the number of points at the end of the climate chamber decreases. An interesting point is the perception of objects and the corresponding response echo. For example, a car may be detected at a distance of about 18 m by the sensor in all scenarios. While in fog and rain the second and third echoes N ₂ , N _{3 provide} the bulk of the detection was at Clear conditions only the first echo N _{1 received} from the car. Accordingly, the presence of second echoes N ₂ on objects may be an indication of fog or rain.

In dense fog with a visibility of 50 m to 60 m, a large number of first echoes N _{1 was} caused by fog reflections with a very short distance. The environmental perception and the range of the sensor are limited. Concerning the second echo N ₂ , only a few reflections were detected directly in front of the sensor due to fog. The majority of the reflections were caused by objects. In dense fog (visibility 20 m to 40 m), the environmental perception is very limited. The first echo N ₁ was almost always caused by the fog. Nevertheless, highly reflective targets such as the taillights of a car are still visible in the second or third echo N ₂ , N ₃ .

Taking into account all the fog and rain measurements compared to the measurements in clear weather conditions, the number for the second and third echoes N ₂ , N _{3 increases} . In addition, due to the fact that there are no reflections on objects with the first echo N ₁ but only with the second or third echo N ₂ , N ₃ , it becomes clear that a multi-echo-sensing sensor is more robust in fog and rain is.

1 schematically shows an analysis of the environmental influences on lidar point clouds. Intensity I is from a VLP16 lidar sensor. All other signals come from a Scala sensor.

Taking into account the number of points for each echo N _t (k), as in 1 It can be expected that the number of second and third echoes N ₂ , N ₃ in fog and rain will increase due to multiple reflections. In 1 the different weather conditions are directly visible in the signal Nt (k). There is a significant increase for the second echo N ₂ in foggy and rainy conditions. It is also interesting that there is no marked difference in the number of third echoes N ₃ for fog and rain.

The total mean distance of the point cloud depends heavily on the scenario. Nevertheless, considering the point cloud in 1 the mean distance r _t , which was spatially filtered and calculated separately for each echo N ₁ , N ₂ , N ₃ , to be a good measure of the estimation of the presence of fog or rain. The signal described is in 1 shown and shows a strong drop for r ₁ in fog. In addition, the average distance increases r _2,3 in rain and fog compared to clear conditions.

In summary, the ability to receive multiple reflections is very important for robust environmental perception, especially in harsh weather conditions.

A resulting lower intensity I of the reflected signal occurs with a simultaneous increase in the echo pulse width epw due to multiple reflections on uniformly distributed mist particles. In 1 is the echo pulse width epw of the Scala lidar sensor as an average epw represented for all points in a frame. The corresponding intensity I of the Velodyne VLP16 is in 1 shown. For both representations, the point cloud was recorded for each weather condition over a similar period of time.

1 confirmed that epw increases in foggy conditions and is approximately inversely proportional to the fog density. In addition, reflections on raindrops show a smaller one epw because the drops of water in the rain are larger and less dense compared to fog. Therefore, the echo pulse width epw is clearly influenced by the weather and can be used as a signal to obtain information about the local environmental conditions.

Concerning the intensity I of the Velodyne sensor, there is only a small drop of intensity I in dense fog. Nevertheless, there is no clear change in the intensity I between the different visibility ranges in fog or rain at 55 mm / h.

Weather detection using lidar sensors

Table III. Proof truth matrix for trained kNN classifiers. Configuration 1-3 was used for training, configuration 4 for detection. The number of samples is given in the first line and the percentage in the second line. The overall accuracy is 96.4%.

For the development of a weather detection algorithm, a Support Vector Machine (SVM) and a k Next Neighbor Classifier were used. The predictive feature vector is given in Table II. The answer of the classifier is "clear", "fog" or "rain". The various visibility areas of the fog have not been considered for the classifier response.

The classifiers were trained on the data set in the climate chamber for fog, rain and clear weather. In addition, real road scenarios were used in clear conditions for training to avoid over-fitting to the specific conditions in the chamber. In addition, various configurations were used in the chamber to reduce the time correlation of the data set. Therefore, only configuration 1-3 was used for training while configuration 4 was used for detection. The detection results are presented as a truth matrix in Table III with extremely satisfactory results.

In addition, recordings on a country road, a highway, and a test site in clear and rainy conditions were used for the detection and analysis of whether the trained classifier can be applied to real road data based on climate chamber data. The resulting overall accuracy of detection with real scenarios is above 73.4% and therefore shows a significant increase, indicating that the feature space has local dependencies. Nevertheless, it is important to keep in mind that the weather conditions in road shots are not constant, not reproducible and not identical to the conditions in the climatic chamber. This is one of the reasons why the fog class could not be detected using road data.

Table IV shows the total classification test results for the climate chamber and road data. The number of samples used for the test is given in each column per class. The classification measures are the true positive rate (TPR), the false positive rate (FPR) and the accuracy (ACC). The classifier with the highest performance in terms of accuracy (ACC) is printed in bold.

The various setups in the chamber were used to reduce the time correlation of the data set. Therefore, Setups A and B were used for training while Setup C was used for testing. The verification results are presented in Table IV with an overall accuracy of 99.49% for the VLP16 sensor and thus extremely satisfactory results. The classification result for the Scala sensor is 89.53% and thus considerably lower than the results for the VLP16 sensor. This may be due to the significantly smaller vertical field of view and thus the number of dots per image.

Since the weather conditions of the real conditions and the climatic chamber are different, they were considered separately. The breakdown for training and verification of road data was done in a similar way. Therefore, four shots were reserved for clear conditions, three with occasional rain and two with continuous rain for testing (one shot in clear conditions, two with rain at night and during the day). The subdivision of the dataset was chosen so that each dataset contains samples of each traffic situation and weather condition and at the same time a subdivision of 80% to 20% was specified between training and test. In this way, time-series effects could be avoided. The overall accuracy achieved for the VLP16 sensor is 97.08% with the kNN classifier. Nevertheless, the false positive rate (FPR) of 17.62% is not negligible.

LIST OF REFERENCE NUMBERS

epw

Echo pulse width

I

intensity

N ₁

first echo

N ₂

second echo

N ₃

third echo

r ₁

Distance for first echo

r ₂

Distance for second echo

r ₃

Distance for third echo

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 8983705 B2 [0002]

Claims (5)

A method of weather and / or visibility detection, characterized in that raw data from at least one lidar sensor of a vehicle is evaluated by machine learning to determine visibility in a vehicle environment, where evaluated results are fused to results from other types of vehicle environment sensing sensors, each for each Environmental detection sensor, a performance is determined and / or classified, which is taken into account in a sensor fusion. Method according to Claim 1 characterized in that to determine the performance of the lidar sensor, images of the lidar sensor in clear, foggy and rainy weather conditions are made, evaluating raw data of the lidar sensor comprising a number of frames, wherein a point cloud of the photograph is represented as a two-dimensional matrix, each row of the lidar Matrix comprises a point having a number of attributes comprising Cartesian coordinates, spherical coordinates, an echo number, an intensity (I) and an echo pulse width (epw), the echo number being divided into first, second and third echoes (N ₁ , N ₂ , N ₃ ), wherein an average and a variance of the attribute is calculated for each frame, wherein the point cloud is spatially filtered in a preprocessing step, wherein a neighborhood component analysis is performed to account for only significant features. Method according to Claim 2 , characterized in that the images of the Lidarsensors are performed in various configurations of the male environment with static and dynamic scenarios. Method according to one of Claims 2 or 3 , characterized in that a mean distance (r ₁ , r ₂ , r ₃ ) of the point cloud for the first, second and third echo (N ₁ , N ₂ , N ₃ ) is calculated separately and for the estimation of the presence of fog or rain is used. Method according to one of Claims 2 to 4 , characterized in that an average value of the echo pulse width (epw) is used to distinguish between fog and rain.

Images