WO2020001935A1

WO2020001935A1 - Method and driving support system for operating a vehicle or for assisting a driver of said vehicle using lane information

Info

Publication number: WO2020001935A1
Application number: PCT/EP2019/064483
Authority: WO
Inventors: Ashwin-Dayal GEORGE; Riadh KALTHOUMI
Original assignee: Valeo Schalter Und Sensoren Gmbh
Priority date: 2018-06-26
Filing date: 2019-06-04
Publication date: 2020-01-02
Also published as: DE102018115317A1

Abstract

The invention relates to a method for operating a vehicle (28) by means or by assistance of a driving support system (36) which uses lane information resulting from the trajectory of recognized lane markings (20) of a road (12) on which the vehicle (28) is currently driving, wherein for a section (26) of the road (12) in which at least one of the lane markings (20) is missing or not recognizable, a trajectory of a virtual lane marking (40) complementing the lane marking (20) is determined, wherein lane information based on the virtual lane marking (40) is taken into account by the driving support system (36). The trajectory of the virtual lane marking (40) is determined from the trajectory of at least one recognized lane marking (20). The invention further relates to a corresponding computer program product and a corresponding driving support system (36).

Description

Method and driving support system for operating a vehicle or for assisting a driver of said vehicle using lane information

The invention relates to a method for operating a vehicle by means or by assistance of a driving support system which uses lane information resulting from the trajectory of recognized lane markings of a road on which the vehicle is currently driving, wherein for a section of the road in which at least one of the lane markings is missing or not recognizable, a trajectory of a virtual lane marking is determined, wherein the virtual lane marking is complementing the recognized part of the corresponding lane marking and is taken into account by the driving support system.

The invention further relates to a corresponding computer program product and a driving support system for operating a vehicle or for assisting a driver of said vehicle, where the driving support system uses lane information resulting from the trajectory of recognized lane markings of a road on which the vehicle is currently driving.

Most existing driving assistance and automation systems, from level 0 warning-only- systems up to level 5 fully automated ones (SAE J3016 autonomy levels), use various sensors to detect and keep track of the environment surrounding the vehicle. The quality and availability of this perception of the environment are one of the key elements that determine the performance of such systems. Various ways exist to make the perceived environment as precise as possible such as fusing the outputs of multiple sensors.

However, what these systems lack is the ability to handle incomplete data. This is especially relevant for lane markings that could be incomplete or missing in some situations. The human driver is capable of noticing this incomplete data and

compensating for it. Driving support systems, on the other hand, lack the intelligence to do so in most situations.

Document DE 10 2009 046 699 A1 describes a method for operating a vehicle by assistance of a driver assistance system where the system uses lane information resulting from recognized lane markings of a road on which the vehicle is currently driving, wherein for a section of the road in which at least one of the lane markings is missing or not recognizable, a trajectory of a virtual lane marking is determined, wherein the virtual lane marking is complementing the recognized part of the corresponding lane marking and is taken into account by the driving support system when outputting driving support information. The trajectory of the virtual lane marking is determined by means of detected roadside objects.

In some cases there are no detectable roadside objects or not enough roadside objects to determine a reasonable virtual lane marking.

The documents US 9 830 517 B2 and US 9 120 486 B1 describe further methods / systems for operating a vehicle by means / by assistance of a driving support system which uses lane information resulting from the trajectory of recognized lane markings of a road on which the vehicle is currently driving.

It is an object of the invention to provide an improved method for operating a vehicle by means / assistance of a driving support system using lane information, a corresponding improved computer program product, and a corresponding improved driving support system, which are able to handle incomplete or missing lane markings in a reliable manner.

This object is achieved by a method, a computer program product, as well as a corresponding driving support system having the features according to the respective independent claims. Advantageous implementations of the invention are the subject matter of the dependent claims, of the description and of the figures.

According to several aspects of the inventive method for operating a vehicle by means or by assistance of a driving support system which uses lane information resulting from the trajectory of recognized lane markings of a road on which the vehicle is currently driving, for a section of the road in which at least one of the lane markings is missing or not recognizable, a trajectory of a virtual lane marking complementing the lane marking is determined, wherein lane information based on the virtual lane marking is taken into account by the driving support system. The trajectory of said virtual lane marking is determined from the trajectory of at least one recognized lane marking. The trajectory of the virtual lane marking can, for example, be a kind of extrapolation at the end of the recognized lane marking or a kind of interpolation in a gap between the two ends of recognized lane marking sections facing each other, which lane marking ends border the section without lane markings.

According to a preferred embodiment of the invention, the trajectory of the virtual lane marking is determined by means of a history of the at least one previously recognized lane marking.

According to another preferred embodiment of the invention, the driving support system is either (a) a driver assistance system for assisting a driver of the vehicle or (b) a system for autonomous driving or semi-autonomous driving the vehicle. The driver assistance system is a system for assisting the driver of the vehicle when

driving/operating the vehicle. In contrast the system for autonomous driving or semi- autonomous driving the vehicle drives/operates the vehicle fully automated or automated to a great extent.

An autonomous vehicle (also known as a driverless car, self-driving car and robotic car) is a vehicle that is capable of sensing its environment and navigating without human input. A classification system based on six different levels (ranging from fully manual to fully automated systems) was published in 2014 by SAE International, an automotive standardization body, as J3016, Taxonomy and Definitions for Terms Related to On- Road Motor Vehicle Automated Driving Systems. This classification system is based on the amount of driver intervention and attentiveness required, rather than the vehicle capabilities, although these are very loosely related.

According to yet another preferred embodiment of the invention, the driving support system takes the virtual lane marking into account only temporarily. In other words, lane information based on the virtual lane marking is taken into account for a specified distance or a specified period of time only.

The presence of incomplete data in the environment or in the data is only a transient state. This means that the corresponding post-processing block is a monitoring module most of the time and will only have to intervene for a short duration and only when needed. According to another preferred embodiment of the invention, the virtual lane marking is taken into account by the driving support system when creating driving support infor mation like e.g. a warning signal for the driver and/or when performing the driving pro cess or intervening during the process of driving the vehicle.

Preferably the recognition of the lane markings is performed by use of at least one sen sor of a sensor type from the following group of sensor types: camera, lidar (light detec tion and ranging), and radar (radio detection and ranging). In other words, the sensor is an imaging sensor. These kinds of sensors are typical sensors installed in the vehicle body of vehicles for monitoring a surrounding area of said vehicles.

According to yet another preferred embodiment of the invention, the determination of the trajectory of a virtual lane marking is in consequence of the recognition of an unusual lane width change or road course of the road on which the vehicle is currently driving. In other words: The system is looking for specific patterns that differentiate such atypical scenarios from a scenario of lane widening. The detection of these patterns is ensured by the following measures:

(i) keeping a history of the evolution of the lane width over time;

(ii) monitoring this evolution - especially both at bumper level and at look-ahead in order to detect abnormal increases of the lane width;

(iii) monitoring the offsets and shapes of the lane markings over distance and over time to identify diverging/missing/jumping markings; and

(iv) identifying and considering lane marking types of the recognized lane markings. Some lane marking types point out to a scenario with missing or not recognizable lane markings others to a scenario with a lane width change.

Once a target pattern has been detected, the system intervenes by adding the virtual lane marking, which is an approximation of the missing lane marking. Stop conditions using the width and the offsets ensure that normal operating conditions are restored, once the sensor data / the lane information resulting from the recognized lane marking are back to normal.

The computer program product according to the invention comprises computer- executable program code portions having program code instructions configured to execute the aforementioned method, e.g. when loaded into a processor of a computer based driving support system.

According to several aspects of the inventive driving support system for operating a vehicle or for assisting a driver of said vehicle, wherein the system uses lane information resulting from the trajectory of recognized lane markings of a road on which the vehicle is currently driving, wherein the driving support system is adapted

to complement at least one lane marking by a virtual lane marking in a section of the road in which at least one lane marking is missing or not recognizable; and

to take lane information into account, which is based on the trajectory of the virtual lane marking. The trajectory of the virtual lane marking is a trajectory determined from the trajectory of at least one recognized lane marking.

Preferably the driving support system is arranged for executing the aforementioned method.

According to a preferred embodiment of the driving support system according to the in vention, the driving support system is a driver assistance system or a system for auton omous driving or semi-autonomous driving.

Further features of the invention are apparent from the claims, the figure and the description of the figure. All of the features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of the figure and/or shown in the figure alone are usable not only in the respectively specified combination, but also in other combinations or alone.

Now, the invention is explained in more detail based on a preferred embodiment as well as with reference to the attached drawings.

In the drawings:

Fig. 1 shows a top view of a road scene with a vehicle on a motorway with lanes and corresponding lane markings, wherein one lane marking is degraded or missing in a section of the road; Fig. 2 shows a top view of a road scene with a vehicle on a road with lane markings, wherein one lane marking has degraded in quality in a section of the road;

Fig. 3 shows the road scene of Fig. 2 together with a first erroneously detected lane marking complementing the lane marking degraded in quality;

Fig. 4 shows the road scene of Fig. 2 together with a second erroneously

detected lane marking complementing the lane marking degraded in quality; and

Fig. 5 shows the road scene of Fig. 2 together with a third erroneously detected lane marking complementing the lane marking degraded in quality.

Fig. 1 shows a top view of a scene 10 with a road 12 being a motorway (or highway) and a diverging road 14 of said road 12, namely a motorway exit. The road comprises two lanes 16, 18, which are laterally limited by lane markings 20. One type of lane marking 20 is a centre marking 22 between two lanes 16, 18. Other lane marking 20 are roadside markings 24 at the roadside of the road 12.

One of the lane markings 20, a roadside marking 24, is interrupted in a section 26 of the road 12 where the diverging road 14 separates from the road 12 . In other words: this is a section 26 of the road 12 in which this lane marking 20 is missing (or not

recognizable).

This lane marking 20 with the section 26 where the lane marking is missing is a roadside marking 24 of the first of the two lanes 16. A vehicle 28 is driving in a corresponding driving direction (arrow 30) on this first lane 16. The vehicle 28 shown in Fig. 1 is a motor vehicle, to be more specific: a passenger car.

The vehicle 10 comprises one or more sensors 32, wherein only one sensor 32 is shown in Fig. 1. The sensor 32 is an imaging sensor 32, namely a camera at the front of the vehicle 28 (front camera) with its viewing direction in the direction of travel (arrow 30). Other possible imaging sensors 32 for this purpose are based on ultrasonic, radar, lidar, etc. The corresponding detection area 34 of the sensor 32 is an area in front of the vehicle 28. Beside the sensor(s) 32 the vehicle 28 further comprises a driving support system 36. The driving support system 36 can either be a driver assistance system for assisting a driver of the vehicle 28 or a system for autonomous driving or semi-autonomous driving the vehicle 28. In the first case the driver assistance system assists the driver of the vehicle 28 to drive/operate the vehicle 28 and in the second case the system

drives/operates the vehicle 28 automatically / on its own.

Fig. 2 shows a top view of another scene 10 with a road 12, a widened section of the road 12 leading into a diverging road 14 of said road 12 and a vehicle 28 driving on a lane of said road 12. At the widened section with the junction 14 a lane marking 20 has degraded in quality, making it difficult for any perception sensor 32 to detect the lane marking 20.

In such scenarios, if the detection-range of the sensor(s) 32 is lower than the distance for which the lane marking 20 is missing or erroneously detected, the driving support system 36 will assume that the width of the road 12 is increasing. This can lead to false warnings and even erroneous steering for systems of level 1 or higher.

Figures 3 to 5 show the road scene 10 of Fig. 2 together with erroneous detections 38 that the sensor 32 could make.

Fig. 3 shows the road scene of Fig. 2 together with a first erroneously detected lane marking 38 complementing the lane marking degraded in quality. The first erroneously detected lane marking 38 is parallel to the recognized lane marking 20 on the opposite side of the corresponding lane.

Fig. 4 shows the road scene of Fig. 2 together with a second erroneously detected lane marking 38 complementing the lane marking degraded in quality. The second errone ously detected lane marking 38 is broadening the corresponding lane linearly.

Fig. 5 shows the road scene of Fig. 2 together with a third erroneously detected lane marking 38 complementing the lane marking degraded in quality. The third erroneously detected lane marking 38 is a virtual lane marking 38 broadening the corresponding lane non-linearly. The driving support system 36 uses lane information resulting from the trajectory of the lane markings 20 clearly recognized by the sensor(s) 32. For the section 26 of the road 12 in which the lane marking 20 is missing (or of poor quality and therefore not recognizable), a trajectory of a virtual lane marking 40 complementing the lane marking 20 is determined, wherein lane information based on the virtual lane marking 40 is taken into account by the driving support system 36, especially an Integrated Dynamics- control Module (IDM) of said system 36. The trajectory of the virtual lane marking 40 is determined from the trajectory of the recognized lane markings 20. This recognized lane marking 20 or at least one of these recognized lane markings 20 is the lane marking 20 which should be complemented by the virtual lane marking 40. The trajectory of the virtual lane marking 38 can, e.g. be a kind of extrapolation at the end of the recognized lane marking 20 or a kind of interpolation between the two ends of recognized lane marking sections of the lane marking 20 bordering the section without lane markings 26.

Due to the fact that the road course of the road 12 is linear in the depicted area, the trajectory of the virtual lane marking 40 shown in Figs. 3 - 5 is linear as well. In a curve/bend of the road 12 with a section of missing or non-recognizable lane mark 26, the virtual lane mark 40 is non-linear.

The method for complementing a lane marking 20 in a section 26 of the road 12 in which this lane marking 20 is missing or not recognizable, has the following main features:

Using data of the recognized lane markings 20 provided by the sensor 32, but considering that these data can be flawed or even accurately represent a degraded environment (i.e. missing lane marking).

Looking for specific patterns that differentiate such abnormal scenarios from normal situations of lane widening.

The detection of these patterns is insured by the following components:

- Keeping a history of the evolution of the lane width over time.

- Monitoring this evolution both at bumper level and at look-ahead to detect abnormal increases of the lane width.

- Monitoring the offsets and shapes of the lane markings 20 over distance and over time to identify diverging/missing/jumping markings 20. Once a target pattern has been detected, the system intervenes by adding the virtual lane marking 38, which is an approximation of the partly missing lane marking 20.

Stop conditions using the width and the offsets ensure that normal operating conditions are restored, once the sensor data / the lane information resulting from the recognized lane marking 20 are back to normal.

The virtual lanes are temporary and will no longer be generated as soon as the sensor/environment data does not show any of the erroneously detected lane markings 38.

List of Reference signs

10 road scene

12 road

14 branch

16 lane

18 lane

20 lane marking

22 centre marking

24 roadside marking

26 section of missing or non-recognizable lane mark

28 vehicle

30 arrow (driving direction)

32 sensor

34 detection area (sensor)

36 driving support system

38 erroneously detected lane marking

40 virtual lane marking

Claims

1. A method for operating a vehicle (28) by means or by assistance of a driving

support system (36) which uses lane information resulting from the trajectory of recognized lane markings (20) of a road (12) on which the vehicle (28) is currently driving, wherein for a section (26) of the road (12) in which at least one of the lane markings (20) is missing or not recognizable, a trajectory of a virtual lane marking (40) complementing the lane marking (20) is determined, wherein lane information based on the virtual lane marking (40) is taken into account by the driving support system,

characterized in that

the trajectory of the virtual lane marking (40) is determined from the trajectory of at least one recognized lane marking (20).

2. The method according to claim 1 ,

characterized in that the trajectory of the virtual lane marking (40) is determined by means of a history of at least one previously recognized lane marking (20).

3. The method according to claim 1 or 2,

characterized in that the driving support system (36) is

a driver assistance system for assisting a driver of the vehicle (28) or

a system for autonomous driving or semi-autonomous driving of the vehicle (28).

4. The method according to any one of claims 1 to 3,

characterized in that the driving support system (36) takes the virtual lane marking (40) into account only temporarily.

5. The method according to any one of claims 1 to 4,

characterized in that the virtual lane marking (40) is taken into account by the driving support system (36)

when outputting driving support information, especially warning signals, and/or when performing the driving process or intervening in the driving process of driving the vehicle (28).

6. The method according to any one of claims 1 to 5,

characterized in that the recognition of the lane markings (20) is performed by use of at least one sensor (32) of a sensor type from the following group of sensor types: camera; lidar; and radar.

7. The method according to any one of claims 1 to 6,

characterized in that the determination of the trajectory of a virtual lane marking (40) is in consequence of the recognition of an unusual lane width change or road course of the road (12) on which the vehicle (28) is currently driving.

8. The method according to any one of claims 1 to 7,

characterized by a search for specific patterns that differentiate a scenario with missing or not recognizable lane markings (20) from a scenario of a lane width change, wherein the search is performed by means of at least one of the following measures:

keeping a history of the evolution of the lane width over time;

monitoring this evolution of the lane width in order to detect atypical increases of the lane width;

monitoring the offsets and shapes of the lane markings (20) over distance and over time to identify missing or not recognizable lane markings (20); and identifying and considering lane marking types of the recognized lane markings (20).

9. A computer program product comprising computer-executable program code portions having program code instructions configured to execute the method according to one of claims 1 to 8.

10. A driving support system (36) for operating a vehicle (28) or for assisting a driver of said vehicle (28), wherein the driving support system (36) uses lane information resulting from the trajectory of recognized lane markings (20) of a road (12) on which the vehicle (28) is currently driving, wherein the driving support system (36) is adapted to complement at least one lane marking (20) by a virtual lane marking (40) in a section (26) of the road (12) in which the at least one lane marking (20) is missing or not recognizable; and

to take lane information into account, which is based on the trajectory of the virtual lane marking (40), characterized in that the trajectory of the virtual lane marking (40) is a trajectory determined from the trajectory of at least one recognized lane marking (20).

1 1 . The system according to claim 10, characterized in that the driving support system (36) is arranged for executing the method according to one of claims 1 to 6.

12. The system according to claim 10 or 1 1 ,

characterized in that the driving support system (36) is

a driver assistance system or

a system for autonomous driving or semi-autonomous driving.