DE102019128948A1 - Determination of a traffic light status - Google Patents

Abstract

The present invention relates to a method for determining a traffic light lighting state at a traffic light (42), the traffic light (42) indicating a lighting state of at least two different lighting states, for a vehicle (10), the vehicle (10) having several environmental sensors (14, 16, 18, 20, 22, 32) for monitoring an environment (40) of the vehicle (10), with the steps for receiving sensor information from the environment sensors (14, 16, 18, 20, 22, 32) with respect to the environment (40) of the vehicle (10) that covers the traffic light (42), identifying the traffic light (42) in the vicinity (40) of the vehicle (10) based on the sensor information for each environment sensor (14, 16, 18, 20, 22, 32), determining the relative positions of the identified traffic lights (42) relative to the vehicle (10) based on the sensor information for each environmental sensor (14, 16, 18, 20, 22, 32), determining an absolute position of the vehicle (10 ), B determining absolute positions of the identified traffic lights (42) based on the absolute position of the vehicle (10) and the relative positions of the identified traffic lights (42) relative to the vehicle (10), comparing the absolute positions of the identified traffic lights (42) with a map position of the Traffic light (42), determining a light state of the traffic light (42) based on the sensor information for each environmental sensor (14, 16, 18, 20, 22, 32) and executing a combined light state estimation of the traffic light (42) based on the determined light states of the traffic light (42) for each environmental sensor (14, 16, 18, 20, 22, 32). The present invention also relates to a driving support system (12) for a vehicle (10) which is set up to carry out the above method

Description

The present invention relates to a method for determining a traffic light lighting state at a traffic light, the traffic light indicating a lighting state of at least two different lighting states for a vehicle.

The present invention also relates to a driving support system for a vehicle, which has a processing unit which is connected to a plurality of environmental sensors in order to receive sensor information therefrom, the driving support system being configured to carry out the above method.

To make progress in semi-autonomous and autonomous driving, it is important to create a consistent environment around the vehicle. One of the most important elements of the environment is a traffic light that changes its lighting state dynamically. The light status of the traffic lights can typically switch between “red”, “yellow” or “green”, which indicates “stop”, “transitional status” or “drive / walk”. Due to its dynamic character, the lighting status must be determined upon arrival at the traffic light, e.g. by the ego vehicle. When determining the lighting status, incorrect measurements can occur. Alternatively, the vehicle receives the traffic light status via a communication device, so that incorrect detection of the light status can be avoided. In this case, the vehicle communicates with any other communication device that knows the lighting state to obtain the traffic light lighting state. This is commonly referred to as V2X communication. For example, the vehicle can set up a communication link to an infrastructure communication device, in particular a communication device connected to the traffic light, which receives the traffic light status and transmits the light status to the vehicle. In the event of failure of the other communication device, however, the ego vehicle cannot maintain the traffic light status and cannot continue driving, or the ego vehicle continues to drive at the traffic light without knowing the status of the traffic light, which is for the ego vehicle, its passengers and other road users is highly dangerous. In addition, such a failure of the other communication device affects all vehicles in the vicinity of the traffic light, so that all traffic will stop at the traffic light. A failure of the corresponding communication device of the vehicle only makes the respective vehicle “blind” with regard to the light status, so that the vehicle cannot continue its journey at the traffic lights. Therefore, a reliable method for detecting the lighting state in the vehicle is desired.

Apart from the typical and most common traffic lights with red, yellow and green lights to indicate the status of the traffic lights, which are most often located at intersections or junctions, other types of traffic lights are known in most countries according to road traffic regulations, e.g. traffic lights with only two lights, eg red and green, or with different displays, eg a recommended speed when approaching "normal" traffic lights or others. The different lighting states can be defined as required by activating one or more individual lights of the traffic lights. The same applies to changes in the color of individual "lights", i.e. different information is displayed at the same point in the traffic light.

The present invention is based on the object of specifying a method for determining a traffic light status at a traffic light and a driving support system for a vehicle which is set up to carry out the above method, which enables reliable and efficient determination of the traffic light status at a traffic light. It is also desirable that the determination of the traffic light status can be carried out independently of other facilities or infrastructure installations.

This problem is solved by the independent claims. Advantageous refinements are given in the subclaims.

In particular, the present invention specifies a method for determining a traffic signal light state at a traffic light, the traffic light indicating a light state of at least two different light states, for a vehicle, the vehicle having a plurality of environmental sensors for monitoring the surroundings of the vehicle, with the steps for Receiving sensor information from the environmental sensors regarding the surroundings of the vehicle that covers the traffic light, identifying the traffic light in the surroundings of the vehicle based on the sensor information for each environmental sensor, determining relative positions of the identified traffic lights relative to the vehicle based on the sensor information for each environmental sensor, Determining an absolute position of the vehicle, determining absolute positions of the identified traffic lights based on the absolute position of the vehicle and the relative positions of the identified traffic lights relative to the vehicle, comparison of the absolute positions of the identified traffic lights with a map position of the traffic lights and determination of a light status of the traffic light based on the sensor information for each environmental sensor, execution of a combined light status estimation of the traffic light based on the determined light status of the traffic light for each environmental sensor.

The present invention also specifies a driving support system for a vehicle, which has a processing unit which is connected to a plurality of environmental sensors in order to receive sensor information therefrom, the driving support system being set up to carry out the above method.

The basic idea of the invention is to carry out a fusion of detected light states on the basis of sensor information from a plurality of environmental sensors in order to reliably determine current light states of traffic lights. This enables efficient autonomous driving at traffic lights regardless of the externally provided light status of the traffic lights. The sensor fusion is based on independently determined light states of the respective traffic light-based sensor information from several environmental sensors. In this way, different environment sensors can be used simultaneously in order to create a consistent environment of the vehicle. Redundant information regarding the traffic light status helps to reliably determine the light status of the traffic light.

The traffic light can generally be any type of traffic light with a dynamic change between at least two different lighting states. The most commonly used traffic lights have red, yellow, and green lights to indicate the status of the traffic lights and are most commonly located at intersections or intersections. Traffic lights can, however, only have two lights, e.g. red and green, to indicate different lighting states. Furthermore, the traffic lights can have different displays as light states, e.g. a recommended speed when approaching "normal" traffic lights, or others.

The various traffic lights can be defined as desired by activating one or more individual lights of the traffic lights. The same applies to changes in the color of individual "lights", e.g. if different information is displayed at the same location of the traffic lights.

The plurality of environmental sensors for monitoring the surroundings of the vehicle provide sensor information that at least partially covers the surroundings of the vehicle. The environmental sensors can be dedicated to the driving support system or to universal environmental sensors which are not assigned to a driving support system or which are assigned to another driving support system. In the latter cases, the environmental sensors can be used by different driving support systems and provide sensor information for each of these driving support systems. The various environmental sensors can be provided as independent sensors on the vehicle. Alternatively, the environmental sensors can be part of a sensor system, e.g. a sensor cocoon, with the sensor information from the individual environmental sensors of the sensor system being processed independently of one another.

The receipt of sensor information from the environmental sensors with respect to the surroundings of the vehicle which is covered by the traffic light relates to the reception of the sensor information which is covered by the respective traffic light. Any suitable data format can be used for the sensor information.

The environment refers to an area around the ego vehicle. The environment generally refers to an unlimited area. In practice, the area is limited, for example, by a maximum sensor range of the environmental sensors or other parameters that define a maximum area of the environment that is covered by the environmental sensors. Different environmental sensors can have different coverage of the surroundings, e.g. different opening angles or different sensor ranges.

The identification of the traffic lights in the vicinity of the vehicle based on the sensor information for each environment sensor relates to the execution of a detection step for detecting the traffic lights. The detection can be based on image recognition or other techniques, depending on the type of environmental sensor.

The step of determining the relative positions of the identified traffic lights relative to the vehicle based on the sensor information for each environmental sensor provides the relative positions with respect to the respective environmental sensor. The relative positions of the identified traffic lights are preferably based on known positions of the respective environmental sensors on the vehicle or at least relative to one another, so that the relative positions of the traffic lights, how they are identified in the sensor information of the different environmental sensors, can be aligned and provided in a single coordinate system.

Determining an absolute position of the vehicle refers to determining a unique position on earth, e.g. based on degrees of latitude and longitude. In addition, a radial position can be determined.

The step of determining absolute positions of the identified traffic lights based on the absolute position of the vehicle and the relative positions of the identified traffic lights relative to the vehicle relates to mapping the determined relative positions to the absolute positions. Based on the absolute position of the vehicle, the absolute position of any object with a known relative position in relation to the vehicle or the environmental sensor can be determined.

The comparison of the absolute positions of the identified traffic lights with a map position of the traffic lights relates to an identification of the identified traffic lights in comparison to the map. In the event that an identified traffic light can be compared with a traffic light in the map, the identified traffic light is examined more closely and the sensor information that has been processed to identify the traffic light is further processed to determine the lighting status of the traffic light. Otherwise the sensor information of the respective environmental sensor is not processed further.

The map position relates to an absolute position of the respective traffic light. The map position is a position obtained from a map. The map position of the traffic light is evaluated based on the absolute position of the ego vehicle in order to determine the absolute position only for a traffic light in the vicinity of the ego vehicle.

The determination of a light state of the traffic light based on the sensor information for each environmental sensor relates to an individual determination of the light state of the traffic light for each environmental sensor. In the event that the traffic light is not identified on the basis of the sensor information from an environment sensor, e.g. if an object is located between the environment sensor and the traffic light, the respective environment sensor is not taken into account for further processing.

When carrying out the combined light state estimation of the traffic lights based on the determined light states of the traffic lights for each environment sensor, the light states that were determined based on the sensor information of the various environment sensors are processed together. This usually refers to the processing of all available light states.

According to a modified embodiment of the invention, the step of receiving sensor information from the environment sensors with regard to the environment of the vehicle which is covered by the traffic light includes receiving image information as sensor information. Image information refers to a matrix of information points which together form a two-dimensional image that represents at least part of the surroundings of the vehicle. The image information can be provided as monochromatic information containing only luminance information. The image information preferably comprises multicolor information, typically in RGB format or in a similar format, with luminance information for red, green and blue colors. The multiple environmental sensors are thus provided as camera devices, in particular as optical camera devices for receiving image information in the light range that is visible to humans.

According to a modified embodiment of the invention, the step of determining relative positions of the identified traffic lights relative to the vehicle based on the sensor information for each environmental sensor and / or the step of determining absolute positions of the identified traffic lights based on the absolute position of the vehicle and the relative positions of the identified traffic lights relative to the vehicle to determine an uncertainty area of the position of the identified traffic lights for each environmental sensor and comprises the step of comparing the absolute positions of the identified traffic lights with a map position of the traffic lights comparing the absolute positions of the identified traffic lights including the uncertainty area with the map position of the Traffic lights on. The uncertainty area refers to an area around the specific position of the traffic light, either the relative or the absolute position. The step of comparing the absolute positions of the identified traffic lights including the uncertainty area with the map position of the traffic light relates to determining whether the map position of the traffic light is enclosed by the respective uncertainty area, i.e. whether the uncertainty area of the identified traffic light overlaps the map position of the traffic light. If the uncertainty area is determined in relation to the absolute position of the traffic light, that is Uncertainty area already an absolute uncertainty area. Otherwise, the uncertainty range is a relative uncertainty range which is then used to determine the absolute uncertainty range based on the absolute position of the vehicle. The uncertainty range is based on an accuracy of the respective environmental sensor, which can be different for each environmental sensor type. For example, a camera usually provides reliable course information, but only inferior distance information. This results in a generally elliptical region of uncertainty that is applied to the identified traffic lights. The uncertainty area relates to a covariance of the positions of the traffic lights. The uncertainty range can be based on at least one parameter below a resolution of the environment sensor, a field of view (FOV) of the environment sensor, distortion coefficient of the environment sensor, uncertainty models with pinhole model and pixel uncertainty of the environment sensor and uncertainty models for environment sensors, in particular a set of optical cameras that form a cocoon around the ego - Vehicle form, can be determined together with a triangulation.

According to a modified embodiment of the invention, the method has additional steps for identifying multiple lanes in the vicinity of the vehicle based on the sensor information for each environmental sensor, identifying a current lane of the vehicle and filtering out traffic lights that are assigned to lanes from the current lane are different. Therefore, based on a current direction of travel of the vehicle, acquisitions of traffic lights that are assigned to lanes that are currently not used by the ego vehicle can be discarded, so that the processing of the sensor information is facilitated. The identification of the plurality of lanes in the vicinity of the vehicle is preferably carried out for the sensor information of each environment sensor individually based on the respective sensor information.

According to a modified embodiment of the invention, the step of determining an absolute position of the vehicle comprises determining an absolute position of the vehicle with high accuracy. The absolute position with high accuracy refers to an absolute position with an accuracy of better than a few tens of centimeters, ie better than one meter, preferably with an accuracy of better than a few tens of centimeters, ie better than twenty to thirty centimeters, and even further preferably with an accuracy of less than ten centimeters (centimeter accuracy). Such accuracy can be achieved, for example, using differential methods known, for example, as differential GPS, or using enhancement systems such as SBAS (Satellite Based Augmentation System) which apply correction data to improve accuracy in determining absolute positions.

According to a modified embodiment of the invention, the step of comparing the absolute positions of the identified traffic lights with a map position of the traffic lights includes comparing the absolute positions of the identified traffic lights with an HD map position of the traffic lights. HD map positions provide position information with a high accuracy of better than a few tens of centimeters, ie better than one meter, preferably with an accuracy of better than a few tens of centimeters, ie better than twenty to thirty centimeters, and even more preferably with an accuracy of less than ten centimeters, also known as centimeter accuracy.

The step of determining an absolute position of the vehicle is preferably carried out with a high degree of accuracy if the method also has the step of comparing the absolute positions of the identified traffic lights with an HD map position of the traffic lights. A high accuracy of the absolute position and a high accuracy of the map position of the traffic lights are preferably combined in order to reliably compare the traffic lights identified in the sensor information by the environmental sensors. Therefore, the absolute position with high accuracy, together with the high accuracy of the map position of the traffic lights, increases the probability of a correct comparison of the traffic lights with the traffic lights identified in the sensor information from the environmental sensors.

According to a modified embodiment of the invention, the step of determining a light state of the traffic light based on the sensor information for each environmental sensor comprises determining a probability for each light state of the traffic light for each environmental sensor and / or the step of performing a combined light state estimation of the traffic light based on the determined Luminous states of the traffic lights for each environmental sensor determining a probability for each luminous state of the traffic light. Based on the probabilities for each of the possible lighting states, the combined lighting state estimation can easily be processed in order to determine the current lighting state of the traffic light. The probabilities for each of the lighting states provide a simple means of fusing the sensor information from the Environmental sensors to carry out a combined light state estimation and to determine the light state of the traffic lights. The probabilities can be determined for each identified traffic light individually for further processing or together.

According to a modified embodiment of the invention, the step of determining a probability for each light state of the traffic light for each ambient sensor includes evaluating a time profile of the light status for each of the ambient sensors and determining a probability for each light status of the traffic light for each ambient sensor taking into account the time profile of the light state for each ambient sensor and / or the step of determining a probability for each light state of the traffic light includes evaluating a time profile of the light state and determining a probability for each light state of the traffic light taking into account the time profile of the light state. The light state is a dynamic state of every traffic light. The temporal course of the lighting state is normally subject to some rules, e.g. a duration of a lighting state, sequences of lighting states, or others that can create trust in order to support a determination of a current lighting state or to identify a determination as incorrect.

According to a modified embodiment of the invention, the step of evaluating a time profile of the lighting state includes defining an average duration for each lighting state and adapting the probability for each lighting state based on an elapsed time after a first detection of a specific lighting state and the average duration for the respective light status. This is based on the fact that the lighting states of traffic lights change according to a regular scheme, with each of the lighting states being active for a predetermined time, as is the case, for example, for typical traffic lights with red, yellow and green colors. This also applies if the traffic light is controlled adaptively based on the current traffic. Therefore, when the ego vehicle approaches, a change in the lighting state is triggered in order to allow the ego vehicle to pass so that such mean durations of the lighting state can be efficiently evaluated in order to determine the lighting state. In the case of a newly detected lighting state, this lighting state is typically assigned a high reliability or probability, whereas the other lighting states are assigned a low reliability or probability.

According to a modified embodiment of the invention, the step of determining a probability for each light state of the traffic light for each environmental sensor comprises determining sensor confidence information for the respective environmental sensor and adapting the probability for each light state of the traffic light based on the sensor confidence information of the respective environmental sensor. The sensor confidence information is typically provided as a specification of the respective environmental sensor. The sensor confidence information can be static information indicating, for example, that one environmental sensor is normally more reliable than another environmental sensor. This can lead, for example, to areas of uncertainty of different sizes. The sensor confidence information can, however, also be dynamic information that depends, for example, on lighting conditions, a certain lighting state, e.g. a color of the traffic lights, a distance to the traffic lights or others.

According to a modified embodiment of the invention, the step of determining a probability for each lighting state of the traffic light includes determining a traffic flow at the traffic light based on the sensor information received from the environment sensors relating to the surroundings of the vehicle and adapting the probability for each lighting state of the traffic light based the specific traffic flow. The traffic flow relates to a group of other road users in the vicinity of the vehicle who can show one or more movement patterns. For example, vehicles that move in a direction parallel to the ego vehicle typically display a “green” light state of the traffic light at a traffic light in front of the vehicle, which would enable the ego vehicle to continue. In addition, pedestrians who cross the street in front of the ego vehicle usually show a "red" light in front of the vehicle at a traffic light, which prevents the ego vehicle from moving on, as only one direction of travel at the traffic light at a time is permissible. Therefore, the ego vehicle has to stop at the traffic light when it detects cross traffic.

According to a modified embodiment of the invention, the step of determining a probability for each lighting state of the traffic light includes applying a penalty for lighting states based on impermissible transitions of lighting states. Transitions between the light states can support the detection of the current light state. In particular, incorrect recordings of the current lighting state can be identified and ignored. Therefore, in most countries, typical traffic lights with red, yellow and green colors to indicate their lighting status do not change the lighting status directly from green to red. In addition, these typical traffic lights in some countries do not change the lighting status directly from red to green, but show yellow in between while in other countries they do not show yellow in between. Therefore, when such a transition between a previous and a current lighting state is detected based on the sensor information, the penalty is applied to the current lighting state. The sanction is preferably applied to each lighting condition determined by each environmental sensor. Accordingly, the sanction is applied immediately if such a non-permitted transition is detected for an environmental sensor without influencing the determination of the lighting state based on other environmental sensors. However, it is also possible to apply the sanction to a combined determination of the probability for the various lighting states based on all environmental sensors.

According to a modified embodiment of the invention, the driving support system has the plurality of environmental sensors. Therefore, the environmental sensors are part of the driving support system. In this case too, however, the environmental sensors can be used jointly by several driving support systems of the vehicle.

According to a modified embodiment of the invention, the plurality of environmental sensors are implemented as camera devices which provide image data as sensor information. The camera devices provide image information as sensor information. The image information relates to a matrix of information points which together form a two-dimensional image which represents at least part of the surroundings of the vehicle. The camera devices can be provided as monochromatic camera devices which only provide luminance information. Preferably, the camera devices are provided as multicolor camera devices which typically provide images in an RGB or similar format that provides luminance information for red, green and blue colors.

These and other aspects of the invention will be apparent and illustrated in the embodiments described below. Individual features that are shown in the embodiments can form an aspect of the present invention on their own or in combination. Features of the various embodiments can be transferred from one embodiment to another embodiment.

• 1 eine schematische Draufsicht eines Fahrzeugs mit einem Fahrunterstützungssystem mit mehreren optischen Kameras als Umgebungssensoren gemäß einer ersten bevorzugten Ausführungsform;
• 2 eine schematische Draufsicht des Fahrzeugs mit dem Fahrunterstützungssystem von 1 zusammen mit einer Verkehrsampel und mit Unsicherheitsbereichen der mehreren Umgebungssensoren;
• 3 eine schematische Draufsicht des Fahrzeugs mit dem Fahrunterstützungssystem von 1 zusammen mit mehreren Fahrspuren und mehreren Verkehrsampeln der mehreren Fahrspuren;
• 4 eine schematische Draufsicht des Fahrzeugs mit dem Fahrunterstützungssystem von 1 zusammen mit mehreren Fahrspuren und Verkehrsampeln, die in einer HD-Karte angezeigt sind;
• 5 eine schematische Draufsicht des Fahrzeugs mit dem Fahrunterstützungssystem von 1 zusammen mit einer Verkehrsampel und Unsicherheitsbereichen der mehreren Umgebungssensoren und verschiedenen Verkehrsampelleuchtzuständen der verschiedenen identifizierten Verkehrsampeln basierend auf Sensorinformation der verschiedenen Umgebungssensoren;
• 6 ein schematisches Diagramm zum Darstellen des zeitlichen Verlaufs des Verkehrsampelleuchtzustands für verschiedene Leuchtzustände;
• 7 eine schematische Ansicht eines nicht zulässigen Verkehrsampelzustandsübergangs zwischen zwei verschiedenen Leuchtzuständen; und
• 8 ein Ablaufdiagramm eines Verfahrens zum Bestimmen eines Verkehrsampelleuchtzustands an einer Verkehrsampel, wie es durch das Fahrunterstützungssystem des Fahrzeugs von 1 der ersten Ausführungsform ausgeführt wird.

Show it:

• 1 a schematic top view of a vehicle with a driving support system with a plurality of optical cameras as environment sensors according to a first preferred embodiment;
• 2 FIG. 3 is a schematic top view of the vehicle with the driving assistance system of FIG 1 together with a traffic light and with areas of uncertainty of the multiple environmental sensors;
• 3 FIG. 3 is a schematic top view of the vehicle with the driving assistance system of FIG 1 together with several lanes and several traffic lights of the several lanes;
• 4th FIG. 3 is a schematic top view of the vehicle with the driving assistance system of FIG 1 along with multiple lanes and traffic lights displayed on an HD map;
• 5 FIG. 3 is a schematic top view of the vehicle with the driving assistance system of FIG 1 together with a traffic light and areas of uncertainty of the plurality of environmental sensors and different traffic light lighting states of the different identified traffic lights based on sensor information from the various environmental sensors;
• 6th a schematic diagram to show the course of the traffic light status over time for different light status;
• 7th a schematic view of an impermissible traffic light state transition between two different light states; and
• 8th FIG. 3 is a flowchart of a method for determining a traffic light lighting state at a traffic light, as it is done by the driving assistance system of the vehicle of FIG 1 of the first embodiment is carried out.

1 refers to a vehicle 10 with a driving assistance system 12th according to a first preferred embodiment. The driving support system 12th is for assisting a human driver of the vehicle 10 , typically referred to as a driver assistance system, or intended to support autonomous or semi-autonomous driving. In the former case it can Driving assistance system 12th Information using a user interface, for example a display and / or a loudspeaker of the vehicle 10 , output to a human driver. In the second case, the driving support system 12th Be part of an autonomous or at least semi-autonomous driving support system, for example, a longitudinal control and / or a lateral control of the vehicle 10 executes.

The driving support system 12th has a set of environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 on. The environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 exhibit a camera cocoon 14th with four individual cocoon cameras 16 , 18th , 20th , 22nd provided as optical cameras. In particular are the cocoon cameras 16 , 18th , 20th , 22nd provided as wide-angle cameras mounted on front, rear, right and left sides of the vehicle 10 are arranged. The cocoon cameras 16 , 18th , 20th , 22nd have a respective field of vision 24 , 26th , 28 , 30th that has a front field of view 24 , a rear field of view 26th , a right-hand field of view 28 and a left-hand field of view 30th each of which has an opening angle of more than 180 °.

The environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 also have a dual camera system 32 with two individual system cameras that are in 1 are not shown individually. In addition, the system cameras are provided as optical cameras, compared to the cocoon cameras 16 , 18th , 20th , 22nd a field of view 34 , 36 have a reduced angle. The field of view 34 , 36 however, the system cameras cover a greater distance than that of the cocoon cameras 16 , 18th , 20th , 22nd .

The driving support system 12th furthermore has a processing unit 38 on that with the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 is connected to receive sensor information from them. The environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 and the processing unit 38 are connected via a communication bus that is in 1 is not explicitly shown. Various bus types are known in the automotive industry, for example CAN bus or others, which can be used as a communication bus. The driving support system 12th enables shared use of the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 .

In an alternative embodiment, the environmental sensors are 14th , 16 , 18th , 20th , 22nd , 32 Part of the vehicle 10 . In particular, the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 Part of another driving support system of the vehicle 10 be, and the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 can through various driving support systems 12th can be used together. The environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 represent sensor information regarding an environment 40 of the vehicle 10 ready.

In this embodiment, the driving assistance system is 12th intended to support autonomous or semi-autonomous driving. Therefore, the driving support system 12th a control of the vehicle 10 execute or control the vehicle 10 to adjust.

The following is a method of determining a traffic light lighting state at a traffic light 42 described. The procedure is carried out using the driving assistance system 12th of the first embodiment carried out. 8th Figure 3 shows a flow chart illustrating the method. The procedure is described with additional reference to the 1 to 7th described.

The traffic light 42 can generally be any type of traffic light 42 with a dynamic change between at least two different lighting states. The most common traffic lights 42 have red, yellow and green lights to indicate the status of traffic lights and are most commonly located at intersections or intersections. The traffic light 42 however, it can also only have two lights, e.g. red and green, to indicate different light states. Furthermore, the traffic light 42 show different displays as light states, eg a recommended speed when approaching "normal" traffic lights 42 or others.

The different traffic light status can be defined arbitrarily by one or more individual lights of the traffic light 42 to be activated. The same applies to changes in the color of individual “lights”, ie at the same point in the traffic light 42 different information is displayed.

The method begins with step S100, which focuses on receiving the sensor information from the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 regarding the environment 40 of the vehicle 10 that refers to the traffic lights 42 covers. The sensor information from the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 covers the traffic lights 42 from. In this embodiment, the sensor information relates to image information obtained from the respective cameras 14th , 16 , 18th , 20th , 22nd , 32 is provided as environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 to be provided. The image information refers to a matrix of information points that together form a two-dimensional image that shows at least part of the environment 40 of the vehicle 10 represents. The image information comprises multicolor information, typically in RGB format or in a similar format, with luminance information for red, green and blue colors. Any suitable data format can be used for the sensor information.

The method then continues with step S110, which focuses on identifying the traffic light 42 in the neighborhood 40 of the vehicle 10 based on the sensor information for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 relates. Therefore, there becomes a detection step for the sensor information from each of the cameras 14th , 16 , 18th , 20th , 22nd , 32 individually executed. In this embodiment, the acquisition step is based on image recognition or other techniques, preferably using a neural network.

Step S120 relates to determining relative positions of the identified traffic lights 42 relative to the vehicle 10 based on the sensor information for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 . The relative positions of the identified traffic lights 42 relative to the vehicle 10 define the relative positions of the identified traffic lights 42 in relation to the respective environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 . By the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 at known points on the vehicle 10 or at least be provided relative to one another, the relative positions of the identified traffic lights 42 in a single coordinate system, particularly in relation to a position of the vehicle 10 , be aligned.

The relative positions of the area of the identified traffic light 42 are for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 along with an area of uncertainty 44 the position of the identified traffic light 42 for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 determined, such as in the 2 and 3 is shown. The area of uncertainty 44 refers to an area around the specific position of the traffic light 42 .

The area of uncertainty 44 is based on the accuracy of the respective environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 that for each type of environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 can be different, as in 2 can be seen. For example, the cameras deliver 14th , 16 , 18th , 20th , 22nd , 32 reliable course information, but poor distance information. This leads to elliptical areas of uncertainty 44 . The area of uncertainty 44 relates to a covariance of the positions of the traffic lights 42 . Therefore, the in 2 areas of uncertainty shown 44 different shapes and sizes. The shape and size of the uncertainty area 44 can based on at least one parameter under a resolution of the environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 , a field of view (FOV) of the environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 , Distortion coefficient of the environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 , Uncertainty models with pinhole model and pixel uncertainty of the environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 and uncertainty models for environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 , in particular a set of optical cameras 16 , 18th , 20th , 22nd , which are intended to be a camera cocoon 14th around the ego vehicle 10 around to be determined together with a triangulation.

According to step S130, there are multiple lanes 46 in the neighborhood 40 of the vehicle 10 based on the sensor information for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 identified. Identifying the multiple lanes 46 in the neighborhood 40 of the vehicle 10 is used for the sensor information of each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 individually executed based on its sensor information.

Identifying the multiple lanes 46 is based in this embodiment on image recognition or other techniques. The lanes 46 together form a road 48 on which the vehicle 10 currently driving. The lanes 46 can have parallel lanes 46 his or a junction of the road 48 show where the two lanes are 46 separate from each other. 3 shows a road 48 with two separate lanes 46 , being on both lanes 46 Traffic lights 42 be identified.

Step S140 relates to identifying a current lane 46 of the vehicle 10 . The current lane 46 can be based on position information, for example based on a global navigation satellite system (GNSS) and / or on by the vehicle 10 odometry information provided in relation to its movement.

Step S150 relates to filtering out traffic lights 42 that with lanes 46 are related to that of the current lane 46 of the vehicle 10 are different. Therefore, based on a current direction of travel of the vehicle 10 Detection of traffic lights 42 , the lanes 46 that are not currently assigned by the ego vehicle 10 are used, are discarded. In the in 3 The example shown are traffic lights 42 the left lane 46 filtered out.

Step S160 relates to determining an absolute position of the vehicle 10 . The absolute position of the vehicle 10 can e.g. be determined based on position information obtained from a global navigation satellite system (GNSS), e.g. GPS, is received. Determining an absolute position of the vehicle 10 refers to the determination of a unique position on earth, e.g. based on latitude and longitude. A radial position can also be determined. In this embodiment it is the absolute position of the vehicle 10 as a highly accurate absolute position of the vehicle 10 certainly. The highly accurate absolute position refers to an absolute position with an accuracy that is better than a few tens of centimeters, that is, better than a meter, preferably with an accuracy of less than a few tens of centimeters, that is, better than eight to thirty centimeters, and still more preferably with an accuracy of less than ten centimeters (centimeter accuracy). Such accuracy can be achieved, for example, using differential methods known, for example, as differential GPS, or using enhancement systems such as SBAS (Satellite Based Augmentation System) which apply correction data to improve the accuracy in determining the absolute position of the vehicle 10 to improve.

Step S170 relates to determining absolute positions of the identified traffic lights 42 based on the absolute position of the vehicle 10 and the relative positions of the identified traffic lights 42 relative to the vehicle 10 . There is a mapping of the determined relative positions of the identified traffic lights 42 executed to get the absolute positions. Based on the absolute position of the vehicle 10 can see the absolute position of each identified traffic light 42 with a known relative position with respect to the vehicle 10 can be determined by such a mapping. Determining the absolute positions of the identified traffic lights 42 in this embodiment also includes the uncertainty range 44 with the position of the traffic lights 42 based on the absolute position of the vehicle 10 and the relative position of the identified traffic light 42 is mapped.

Step S180 relates to aligning the absolute positions of the identified traffic lights 42 with a map position of the traffic lights 42 . The map position also relates to an absolute position of the respective traffic light 42 obtained from a card. The map position of the traffic light 42 is based on the absolute position of the ego vehicle 10 evaluated to the absolute position only for one traffic light 42 near the ego vehicle 10 to determine.

In this embodiment, the map position is an HD map position of the traffic light 42 . HD map positions provide position information with a high accuracy of better than a few tens of centimeters, ie better than one meter, preferably with an accuracy that is better than a few tens of centimeters, ie better than twenty to thirty centimeters, and even more preferably with an accuracy less than ten centimeters, also known as centimeter accuracy. Accordingly, the identified traffic lights 42 , as in 4th is shown, a small area of uncertainty 44 exhibit.

In this embodiment, the adjustment can be done simply using the uncertainty ranges 44 the identified traffic lights 42 are executed. It is only necessary to determine whether the map position is the traffic light 42 of the respective uncertainty area 44 is enclosed, ie whether the uncertainty area 44 the map position of the traffic lights 42 based on the absolute position determined as shown above.

The mapping relates to an identification of the identified traffic lights 42 in the map. In the event that an identified traffic light 42 with a traffic light 42 can be compared in the map, the traffic light 42 closer look, and the sensor information that has been processed to get the traffic light 42 is processed to determine the traffic light status, as discussed below with reference to step S190. Otherwise, the sensor information of the respective environment sensor 14th , 16 , 18th , 20th , 22nd , 32 not further processed.

Step S190 relates to determining a lighting state of the traffic light 42 based on the sensor information for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 , as in 5 is shown. In this way, the lighting status of the identified traffic lights is individually determined 42 for each of the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 . Therefore, the sensor information becomes each of the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 individually processed to provide a lighting status for each identified traffic light 42 to determine. Of course, when the traffic lights will 42 not based on the sensor information of an environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 is identified, e.g. if there is an object between the environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 and the traffic light 42 is located, or if the illustration of the identified traffic light 42 is not successful, the respective environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 not considered for further processing.

In this embodiment, a lighting state of the traffic light is determined 42 based on the sensor information for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 determining a probability for each light state of the traffic light 42 for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 on. Therefore, every possible light state of every identified traffic light 42 depending on the respective sensor information of the respective environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 assigned a probability.

In particular, the probabilities are based on a temporal course of the lighting state, a sensor confidence information for the respective environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 and a traffic flow at the traffic light 42 as discussed in detail below. Finally, a penalty is applied to light states based on impermissible transitions of light states.

In particular, the temporal course of the lighting state relates to a temporal course of the lighting state for each of the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 is determined.

Based on the temporal progression of the lighting status, for each environmental sensor 14th , 16 , 18th , 20th a probability for each possible lighting condition of the traffic light 42 based on the sensor information of the respective environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 certainly. The evaluation of the temporal course of the lighting state includes defining an average duration for each lighting state and adapting the probability for each lighting state based on an elapsed time after a first detection of a certain lighting state and the average duration for the respective lighting state. This is based on the fact that the lighting states change according to a regular scheme, with each of the lighting states being active for a predetermined time, as is the case, for example, for typical traffic lights with red, yellow and green colors. In the case of a newly detected lighting state, this lighting state is typically assigned a high reliability or probability, whereas the other lighting states are assigned a low reliability or probability. 6th shows how the reliability for the various lighting states changes over time.

The sensor confidence information for the respective environmental sensor is used to adapt the probability for each lighting state of the traffic light 42 for the respective environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 used. The sensor confidence information is typically used as a specification of the respective environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 provided. The sensor confidence information can be static information that indicates, for example, that an environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 is typically more reliable than any other environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 . The sensor confidence information can, however, also be dynamic information that depends, for example, on the lighting conditions, a certain lighting state, for example a color of the traffic light 42 , a distance from the traffic lights 42 or others.

The flow of traffic at the traffic lights 42 is based on that of the respective environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 received sensor information determined. The traffic flow relates to a group of other road users in the vicinity 40 of the vehicle 10 that can show one or more movement patterns. For example, show vehicles moving in a direction parallel to the ego vehicle 10 move, typically a "green" traffic light state, which is what the ego vehicle does 10 allows to continue along with the flow of traffic. In contrast, pedestrians point out the street 48 in front of the ego vehicle 10 cross, typically a "red" traffic lights, because at the traffic lights 42 only one direction of travel is permitted at the same time. Hence, the ego vehicle must 10 at the traffic light 42 stop when detecting such cross traffic. Based on the determined traffic flow, the probability for each lighting state of the traffic light is calculated 42 customized.

Finally, some transitions between the light states are not permitted according to road traffic regulations. The evaluation of transitions between different lighting states can therefore support the detection of the current lighting state. In most countries, typical traffic lights are changing 42 using red, yellow and green colors to indicate their lighting state does not directly change the lighting state from green to red, as in 7th is shown. In addition, these typical traffic lights change 42 in some countries the light status does not go directly from red to green, but show yellow in between, while in other countries they show no yellow in between. Therefore, if such an impermissible transition between a previous and a current lighting state is detected based on the sensor information, the penalty is applied to the current lighting state. The penalty is applied to every light condition detected by every environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 is determined. Therefore, if an illegal transition is detected, the penalty will be applied to the new lighting state.

Step S200 relates to the execution of a combined lighting state estimation of the traffic light 42 based on the specific light status of the traffic lights 42 for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 . Therefore, the lighting conditions as they are based on the sensor information from the various environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 are determined, processed together. This typically refers to all available light states as indicated by the environmental sensors 14th , 16 , 18th , 20th , 22nd , 32 to be provided. The estimate can be a combination of the individual probabilities of the various lighting states for each lighting state for each environmental sensor 14th , 16 , 18th , 20th , 22nd , 32 exhibit.

Based on the combined estimation of the status of the traffic lights 42 becomes a control of the vehicle 10 designed for autonomous driving.

List of reference symbols

10

vehicle

12th

Driving assistance system

14th

Camera cocoon, environmental sensor

16

Cocoon camera (front), environmental sensor

18th

Cocoon camera (rear), environmental sensor

20th

Cocoon camera (right), environmental sensor

22nd

Cocoon camera (left), environmental sensor

24

Field of view (front)

26th

Field of view (rear)

28

Field of view (right)

30th

Field of view (left)

32

Double camera system

34

Field of view

36

Field of view

38

Processing unit

40

Surroundings

42

Traffic lights

44

Uncertainty area

46

lane

48

road

Claims (15)

Method for determining a traffic light lighting state at a traffic light (42), the traffic light (42) indicating a lighting state of at least two different lighting states, for a vehicle (10), the vehicle (10) having several environmental sensors (14, 16, 18, 20 , 22, 32) for monitoring an environment (40) of the vehicle (10), with the steps: Receiving sensor information from the environment sensors (14, 16, 18, 20, 22, 32) relating to the environment of the vehicle (10) which the traffic light (42) covers; Identifying the traffic lights (42) in the vicinity (40) of the vehicle (10) based on the sensor information for each environment sensor (14, 16, 18, 20, 22, 32); Determining relative positions of the identified traffic lights (42) relative to the vehicle (10) based on the sensor information for each environmental sensor (14, 16, 18, 20, 22, 32); Determining an absolute position of the vehicle (10); Determining absolute positions of the identified traffic lights (42) based on the absolute position of the vehicle (10) and the relative positions of the identified traffic lights (42) relative to the vehicle (10); Comparing the absolute positions of the identified traffic lights (42) with a map position of the traffic lights (42); Determining a lighting state of the traffic light (42) based on the sensor information for each environmental sensor (14, 16, 18, 20, 22, 32); and Carrying out a combined light state estimation of the traffic lights (42) based on the determined light states of the traffic lights (42) for each environmental sensor (14, 16, 18, 20, 22, 32). Procedure according to Claim 1 , characterized in that the step of receiving sensor information from the environment sensors (14, 16, 18, 20, 22, 32) relating to the environment (40) of the vehicle (10) which the traffic light (24) covers, the receiving of Has image information as sensor information. Method according to one of the preceding Claims 1 or 2 , characterized in that the step of determining relative positions of the identified traffic lights (42) relative to the vehicle (10) based on the sensor information for each environmental sensor (14, 16, 18, 20, 22, 32) and / or the step of determining absolute positions of the identified traffic lights (42) based on the absolute position of the vehicle (10) and the relative positions of the identified traffic lights (42) relative to the vehicle (10) determining an uncertainty range (44) of the position of the identified traffic lights (42) for each environmental sensor (14, 16, 18, 20, 22, 32), and the step of comparing the absolute positions of the identified traffic lights (42) with a map position of the traffic lights (42) comparing the absolute positions of the identified traffic lights (42) including the uncertainty area (44) with the map position of the traffic light (42). Method according to one of the preceding claims, characterized in that the method has the additional steps: identifying multiple lanes (46) in the surroundings (40) of the vehicle (10) based on the sensor information for each environmental sensor (14, 16, 18, 20) , 22, 32); Identifying a current lane (46) of the vehicle (10); and filtering out traffic lights (42) that are related to lanes (46) that are different from the current lane (46) of the vehicle (10). Method according to one of the preceding claims, characterized in that the step of determining an absolute position of the vehicle (10) comprises determining an absolute position of the vehicle (10) with high accuracy. Method according to one of the preceding claims, characterized in that the step of comparing the absolute positions of the identified traffic lights (42) with a map position of the traffic lights (42) is comparing the absolute positions of the identified traffic lights (42) with an HD map position of the traffic lights (42). Method according to one of the preceding claims, characterized in that the step of determining a lighting state of the traffic light (42) based on the sensor information for each environmental sensor (14, 16, 18, 20, 22, 32) includes determining a probability for each lighting state of the Has traffic lights (42) for each environmental sensor (14, 16, 18, 20, 22, 32); and / or the step of carrying out a combined light state estimation of the traffic light (42) based on the determined light states of the traffic light (42) for each environmental sensor (14, 16, 18, 20, 22, 32) determining a probability for each light state of the traffic light (42). Procedure according to Claim 7 , characterized in that the step of determining a probability for each light state of the traffic light (42) for each environmental sensor (14, 16, 18, 20, 22, 32) is the evaluation of a time course of the light state for each of the environmental sensors (14, 16 , 18, 20, 22, 32) and the determination of a probability for each light state of the traffic light (42) for each environmental sensor (14, 16, 18, 20, 22, 32) taking into account the temporal course of the light state for each environmental sensor (14 , 16, 18, 20, 22, 32); and / or the step of determining a probability for each light state of the traffic light (42) comprises evaluating a time profile of the light state and determining a probability for each light state of the traffic light (42) taking into account the time profile of the light state. Procedure according to Claim 8 , characterized in that the step of evaluating a temporal course of the lighting state is defining an average duration for each lighting state and adapting the probability for each lighting state based on an elapsed time after a first detection of a certain lighting state and the average duration for the respective lighting state having. Method according to one of the Claims 7 to 9 , characterized in that the step of determining a probability for each lighting state of the traffic light (42) for each environmental sensor (14, 16, 18, 20, 22, 32) includes determining sensor confidence information for the respective environmental sensor (14, 16, 18, 20, 22, 32) and the adaptation of the probability for each lighting state of the traffic light (42) based on the sensor confidence information of the respective environmental sensor (14, 16, 18, 20, 22, 32). Method according to one of the Claims 7 to 10 , characterized in that the step of determining a probability for each lighting state of the traffic light (42) is the determination of a traffic flow at the traffic light (42) based on the received sensor information from the environmental sensors (14, 16, 18, 20, 22, 32) with respect to the surroundings (40) of the vehicle (10) and the adaptation of the probability for each lighting state of the traffic light (42) based on the determined traffic flow. Method according to one of the Claims 7 to 11 , characterized in that the step of determining a probability for each light state of the traffic light (42) comprises applying a penalty for light states based on impermissible transitions of light states. A driving support system (12) for a vehicle (10) having a processing unit (38) connected to a plurality of environmental sensors (14, 16, 18, 20, 22, 32) to receive sensor information therefrom, the driving support system (12) to do so is set up, the method according to one of the preceding Claims 1 to 12th to execute. Driving support system (12) according to Claim 13 , characterized in that the Driving assistance system (12) comprising a plurality of environmental sensors (14, 16, 18, 20, 22, 32). Driving support system (12) according to one of the preceding Claims 13 or 14th , characterized in that the plurality of environmental sensors (14, 16, 18, 20, 22, 32) are implemented as camera devices which provide image data as sensor information.

Images