DE102004047914A1 - Method of estimating road condition, by detecting image of road or previously defined portion of road using camera system - Google Patents

Abstract

The method involves detecting the image of the road, or at least a previously defined portion to the road using a camera system. The direct and/or indirect light reflection on the road is measured using an infrared sensor. Signals from at least two different sensors are analyzed using a series of fuzzy rules.

Description

in the Following is a preferred embodiment of the three independent claims described with the present invention, in all these independent aspects be combined. From this description, the dependent claims may result.

one The most important factor when braking in the automotive sector is the maximum Friction coefficient. The prediction of its value requires knowledge of the various cooperating elements, i. Wheel, Vehicle, Roughness and road condition. Caused by all these influencing parameters the road condition generally the most important changes at the friction potential. Therefore, it is by determining whether the road surface is wet, icy or snowy, possible, the coefficient of friction classify the roadway to warn the driver and the adaptability the control systems in a vehicle, such as e.g. ACC, ABS or ESP, to the actual Increase friction situation. The friction information can over future GPS systems [1] can also be forwarded to other drivers.

in the general hangs the coefficient of friction due to typical wheel slip relationships described by the Paceijka or Burckhardt formula [2] can be.

In 1 the influence of the inter-medium is shown. It affects the entire curve, but especially the maximum friction coefficient.

Of the Friction value identify themselves with different methods in three groups can be divided: indirect friction measurement with the help of tire sensors, non-linear approximation methods in connection with vehicle models and systems with multiple sensors.

The The first category includes indirect friction measurement through processing the tire deformation in the contact area. For this must have special Sensors are integrated in the tires. Through these sensors is the tire is "intelligent" and can be used as a transmission medium as well as a sensor for automotive systems be used ([3], [4], [5]).

The second category refers to the detection of friction with Using an observation model containing different dynamic signals, such as. the wheel speed and the longitudinal or lateral acceleration considers. The vehicle models will be in conjunction with a model for the Slip rotation in dependence used by the current friction situation. Due to several changes Vehicle and tire condition become adaptive Kalman filters used to change the system to predict [6].

The third category needed a correct description of the different involved in the friction Elements. Therefore, it is appropriate system components to integrate into the entire system to the road surface, the Tire to describe the inter-medium and the vehicle. The coordination the sensor data is done by empirical or semi-empirical formulas, the data-based approximation methods were adapted ([7], [8]).

These Systems have the advantage that the coefficient of friction before the Vehicle and can be determined under different driving conditions. This method can also include free rolling, so that the Achievement of the critical limit of the degree of friction for the prediction the friction value no longer is mandatory. Therefore, the use of signal processing and the sensor fusion inevitable.

The The present description further illustrates the method, which is based on several sensors based. The Prometheus Project [7], in which various sensors integrated into a system to detect the friction imparted a general idea of it. The main differences In the present system are in the choice of sensors and the Software for the fusion of sensor data. With the help of an optimized fuzzy expert system is it is possible build a redundant system where the different sensors with little loss can be turned off at the power.

In section 2 the system components are described. The data processing presents various system features. They lead to the extraction of useful information for determining the condition of the road. In section 3 the combination of the signals is represented as a fuzzy network, which is designed as an expert system and in section 4 is optimized to measure data with nonlinear algorithms. Finally, some test results are presented and explained.

2. System description and data processing

For the desired implementation was a test vehicle with the necessary sensors and the prototype of a fast, electronic High-end control unit for data acquisition and a quick integration of the software code as control software.

The Data preparation and a part of the data processing are in form implemented by hardware to get more processing power available for processing and shortcut blocks to have.

Under The sensors considered are none that have all the required information about the Road condition can give. The measuring systems give more complementary or redundant information. The supplemental information will be on the one hand used to superimpose the entire parameter range. On the other hand, the redundant information serves to improve accuracy to improve the prediction.

The Sensor systems are described in the following subsections. The individual sensor signals were processed to be the preferred significant ones Extract information. You can be innovative yourself.

2.1 Temperature sensor

Of the Temperature sensor deals with the road surface and measures it Temperature. It is a passive measuring system that measures the temperature by processing the IR radiation reflected from the roadway outputs. The reflected energy is converted into a temperature value, after the emission factor of the road surface and the temperature of the Sensor box considered were. For the next Evaluation steps, the temperature range is more important than the exact Temperature value. The sensor is used to increase the accuracy of the Determination of ice and snow increase.

2.2 Rain sensor

It a standard IR optical rain sensor is used to determine if it rains. The information from the sensor can only be used in one direction because the information required is the wetness of the Road surface. If the sensor is not activated (dry Windscreen), the road condition determination algorithm works, without considering this sensor value. However, if it is activated, you go from a wet lane off (higher weighting).

2.3 Camera system

The Camera method is intended to detect snow, water and gutters in the lane.

she can be based on visual cues: while a dry lane can be seen on the relatively homogeneously structured surface, have wet road surfaces a striped appearance. To distinguish the different road conditions, we used a texture-based segmentation. To An image enhancement and restoration will help with the image filtered according to the method described in [9] was developed. Thereafter, a segmentation with the help of Sobel edge detector.

First, an area is cut out of the full screen. The image section represents the area of the roadway between the right and the left lane. Therefore, the length of the cutouts mainly depends on the width of the lane and the camera position. It is in a range between 150 and 250 pixels. The height of the section was set to 40 pixels in order to reduce the image size and thus also the computational effort required to process the relevant image regions (ROI = region of interest). Please refer 2 ,

Around to guarantee appropriate segmentation and edge detection The different areas in a picture should be on their edge so many Differences as possible exhibit. Conversely, there is a great homogeneity of the segmented areas important. For example, noise hinders due to convolutions a meaningful Segmentation and leads too disturbing Results.

Before an algorithm for the measurement of texture or an edge sensor in the relevant image area (ROI) is needed preparatory steps performed to meet the above criteria.

The contrast can be improved with the help of an increase in contrast [10]. Input values below the threshold m are compressed into a narrow range of dark zones in the output image; similarly, values greater than m are compressed into a narrow range of bright zones in the result. In this way, the gray scale values are stretched from the interval [min max] to the interval [0 255]. The result is a picture with sharper contrast, and the border between the areas is sharper, see 4a b.

To reduce the noise in the image that was partially enhanced by the previous step, the slices can be enhanced using a 5x5 median filter mask. The median filter is non-linear and calculates the median of the ordered sequence of pixel values under the filter mask. The use of linear filters has been avoided because the use of such filters results in blurred edges. After the corrective measures, the image section can be filtered with a kernel developed on the basis of two mnemonic characters with a length of 5 proposed in [9]:

In this equation corresponds to 1 L of the plane vector. The middle pixel has a higher weight here. E is the edge vector. He sets the coefficients of the second Derivation.

The Applying the convolution kernel to the input image results in a Grayscale image. The grayscale describes the texture features of each Pixels, in our case the participation of each pixel in one Edge. According to [9], every pixel in the resulting grayscale image becomes through a texture energy measure (TEM) replaced at the pixel. This is done by carrying out a non-linear windowing operation around each pixel. For the here desired Strip detection as a characteristic feature of a wet road surface the TEM is irrelevant. More important than the absolute values of pixel sharing The feature studied is the topological distribution of Pixels that belong to such a feature.

In contrast to a wet lane, which has clear light streaks, a dry lane shows scattered bright areas ( 3 ).

at Using the Sobel edge detector, the strip edges are detected. Therefore, the previously obtained grayscale image is converted into a black and white image, to obtain maximum contrast because of the Sobel edge detector is a gradient based detector.

If the road surface being examined is wet, the edge detection results in a series of long vertical edges, see 4d e. In the case of a dry surface, the best case is a black image (no edges) or a series of short edges.

In a final step, called "smoothing," all edges that fall below a certain threshold are discarded 4e f. The threshold is defined by the size of the relevant area (ROI). After this step, no edges are left in the image when the road is dry. 4 contains an overview of the steps described above. The last processed image allows an accurate determination of whether the road surface is wet.

The The suitability of the method described in this section has been increased tested as 1600 images on different road surfaces with different lighting conditions were recorded. In about 80% of the cases the road condition was recognized correctly. Nevertheless, the method is fundamentally inappropriate, if the ROI a shadow, the windshield wiper or roadway grooves. Therefore, an additional method is developed with the broad areas with relatively constant and low gray levels be recognized, with their absolute and relative values in the Compared to the rest of the picture are rated, shadows, wipers and road grooves are recognized or the image in case of ambiguity for not declared usable becomes.

2.4 The IR sensor

The IR sensor measures the direct and indirect reflection of the light on the road ( 5 ). The difference between these values defines the brilliance of the road surface, the sum defines the brightness. The sensor consists of a combination of transmitter and receiver and a second separate receiver unit.

Three different sensor parameters can be varied: Distance h ( 5 ) between sensor surface and roadway, distance d between transmitter and receiver and the angle of inclination α between the receiver and the vertical axis. Due to the optical properties of icy and snowy roads, it is possible to obtain different sensor values for these surfaces after correct positioning.

6 illustrates the results of measurements at different speeds on ice, snow and asphalt surfaces. The dark color of the asphalt leads to a low direct reflection value. When driving on snow, a middle part of the original light is detected by both receivers (maximum possible result: 255).

Of course you can Help the sensor made different classes of road surfaces become. These classes can be defined by using only indirect or direct reflection becomes. Nevertheless, leads the use of both values for a higher robustness. It is therefore possible, reduce the number of required examples to the mean to create.

2.5 microphone

As mentioned in [11], Is it possible, for the Detecting road condition using a microphone. The attempt the measurement of the tire / road noise in the rain from close proximity sets to mount the microphone in a protected position near the contact surface. The following measurements were taken at the front edge of the rear wheel carried out. It is appropriate to free-rolling rides at constant speed to take into account. It was due to the expected high frequency "hissing noise" of the water necessary to use microphones with a large dynamic range too use. additionally this led to the Using a 16-bit AD card for data acquisition to the Need for signal filtering and amplification in order to be able to the entire 16-bit resolution even at frequencies above 10 kHz to use.

Drying tests that started with very wet surfaces indicated a direct relationship between the water film height and the noise level at high frequencies. In 7 A drying test is carried out on a standard concrete and asphalt pavement on the Contintental test site.

The Signals are used to define the water level. It was Octave 9 chosen, for the wet to describe the roadway. This is the highest variance and the right one Assignment of the drying degrees for This frequency interval is due to most of the collected data.

In order to extend these observations to the general driving situation, which includes different speeds, control and braking situations, and noise generating road surfaces, the actual measured noise levels must be converted to a reference value ( 8th ).

In addition, the noise can also be used to detect snow on the road. In 9 the actual value of the tire / road noise is shown on different surfaces at 36 m / s for a previously selected frequency interval.

This Effect is a characteristic of tire construction and dampening Effect of snow. The detected in a frequency range around 2 kHz noise are lower in snow than other surfaces. This is between Ice and snow, which have similar characteristics, if other sensor values are taken into account.

Of the noise in this frequency interval depends also from the vehicle speed and the type of tire. Nevertheless Is it possible, a limitation of the characteristic sound of snow for one Reference Noise to be defined, which corrects for speed has been.

3. Fusion the data from several sensors

With Help of integrated sensor signals and processed sensor data the road condition can be detected. Provide as a particularly preferred embodiment the three subsystems produce a fuzzy result for the actual wetness, the Snow or the ice on the road. The dry state is complementary to these three states. Each subsystem includes two result classes: the detected class (Wet / snow / ice) and the unknown class. The acquisition belongs to the unknown class the negation of the already mentioned classes, but also the recording an invalid one Class e.g. due to insufficient Information in the input layer.

On Reason of piecewise redundant sensor information, a validity concept was developed. This concept allows the shutdown of some sensors and ensures a friction detection without additional Programming. The validity test is based on the control of the value range of the sensor data and the plausibility some scenarios. A central point for reusability the software in case of invalid sensor data is the definition of invalid Layer in which the sensor signal receives values that are invalid classes at the issue lead.

In this way, the system provides three class membership values, each accompanied by a binary validity value. In order to achieve the aforementioned coefficient of friction prediction, a complementary prediction block is absolutely necessary ( 10 ), where the friction values must be assigned to the different classes of road condition.

The already mentioned subsystems are described using the example of moisture detection ( 11 ) explained.

The elected shortcut based on a fuzzy system. The rules become after the observation the sensor data created at different road conditions. It uses a Sugeno network where the output is a linear combination of inputs.

The membership functions for the input classes are based on splines. They follow a Z or an S-shape. The parameters of these functions are defined as variables to allow optimization of the fuzzy expert network.

The Rules receive a different weighting so that e.g. the Detecting low temperatures to reject the rules for wet detection leads.

wobei K_i den Ausgabeklassen, w_i den Gewichtungen der Ausgabeklassen und μ_i den Ausgaben der Regeln entspricht.The defuzzification strategy used is the "centroid" method, where result R is calculated as Equation 2:

where K _i corresponds to the output classes, w _{i to} the weights of the output classes, and μ _{i to} the outputs of the rules.

R = K1 + μ·(K2 – K1), if N = 1 wobei N der Anzahl der aktiven Regeln entspricht. K1, if N = 0 The centroid method offers a 100% prediction if only a single rule is activated, even if the active rule has a low membership score of less than 20%. A switch had to be defined to achieve a fuzzy output regardless of the number of active rules. Equation 3 identifies the switch for a system with two output classes K ₁ ("unknown") and K ₂ ("wet") to explain the above idea.

R = K 1 + μ · (K 2 - K 1 ) if N = 1 where N equals the number of active rules. K 1 if N = 0

This section has defined a fuzzy system with about 159 parameters. The initial set of parameters was determined based on the observations. The setting of class bounds and rule weights has been achieved by non-linear least squares optimization. Several measurements were made on wet, dry, icy and snowy surfaces. The results of the optimized system are very good. 12 illustrates the affiliation values for ice, snow, and wet classes in a sequence of road conditions (ice, snow, dry asphalt).

Similar tests with the optimized mesh were carried out on a so-called "chess" surface where concrete and ice alternated. 13 shows that the tendency is recognized correctly.

Nevertheless, the transitions are as clear as the middle parts of each pattern. The vertical black lines in 13 represent the theoretical limits of the different patterns. The reasons for this are the low values of some sensor signals (IR: 10 Hz, temperature: 1 kHz) and the loud noise in the transition area. This loud noise leads to the false detection of wetness in the restriction of the patterns.

In This section will be a particularly preferred multi-sensor approach for the Detection of the road condition described. The system is based on an optical sensor for the brightness and brilliance of the roadway, an acoustic sensor, an optical rain sensor, a road temperature sensor and a camera system. The processed signals are generated using a fuzzy expert network with self-defined rules and software-optimized parameters assigned. As a result receives one the affiliation values for the classes wet, Ice and snow and the corresponding validity values. In most cases will the road condition is detected successfully. Nevertheless, it should be clear that every measurement method, especially that of the camera system or the Infrared reflection sensor independent mutually usable and innovative.

Mentioned references

• [1] R. Gnadler, H. Marwitz, "Neues System for determining the traction potential in driving mode ", in ATZ 05/2004, Germany, 2004, pages 458-467.
• [2] B. Heimann, H. Abdellatif, H. Blume, N. Bouzid, M. Lorenz and A. Trabelsi, "Interaction Road vehicle from the perspective of mechatronics ", in VDI reports 1791, VDI tire chassis lane, Hanover, Germany, 2003, pages 11-32.
• [3] Volk, H., Kluge, S., "The intelligent tire ", in proc. of the International Rubber Conference 2003, Nuremberg, Pages 177-180
• [4] Cyllik, A .; Strothjohann, Th .; Scholl, G., "The intelligent Tires - Application Options Tire Profile Sensor ", in VDI Report No. 1632, Düsseldorf, 2001, pages 115-124
• [5] Quandt, E, Hoffmann, K.-H., Moske. M., "tire friction remote sensor", Annual report 2002, Caesar, Bonn.
• [6] Canudas-de-Wit, C., Horowitz, R., "observers for Tire / road contact friction using only wheel angular velocity information ", Proceeding of the IEEE Conference on Decision and Control, December, 1999.
• [7] Bachmann, T., "Noncontact Power circuit detection " PROMETHEUS Phase III, Final Report, fzd Report No. 199/96, subject area Vehicle Technology, Technische Hochschule Darmstadt, 1996.
• [8] Gnadler, R., Unrau, H.-J., "Method and Device for Determining the Adhesion and Adhesion Limit for Vehicle Tires", DaimlerChrysler AG, patent WO00 / 32456A1, 2000.
• [9] K. Laws, "Textured Image Segmentation ", Ph. D. Dissertation, University of Southern California, January 1980th
• [10] Rafael C. Gonzales, Richard E. Woods, Steven L. Eddins, "Digital image processing using MATLAB, "Pearson Prentice Hall, 2004.
• [11] A. Trabelsi, B. Heimann, L. Dreilich, H.-J. Harting, "The Microphone as a roughness and wetness sensor of the road in automotive applications ", in Internoise 2004, Prague, Czech Republic, 2004, pages 11-32.

Claims (3)

Method for estimating the condition of the road by detecting an image of the roadway or at least one predetermined part of the roadway with a camera system Road condition assessment method by measuring direct and / or indirect light reflection the roadway with an infrared sensor Road condition assessment method by analyzing the signals from at least two different sensors with a set of fuzzy rules.