DE102023002044B3 - Method for a model-based fusion of different sensor modalities - Google Patents

Abstract

The invention relates to a method for a model-based fusion of different sensor modalities, wherein the existence of objects in a vehicle environment and object-related quality measures are determined based on data recorded using the different sensor modalities. According to the invention, sensor-dependent trust values for the existence of an object and a probability of continued existence in the vehicle environment are determined as object-related quality measures by means of a centralized fusion system, the object-related quality measures being determined based on values estimated using dynamic models. Based on the quality measures, a fusion is carried out using the data from different sensor modalities of certain existence probabilities of objects.

Description

The invention relates to a method for a model-based fusion of different sensor modalities according to the preamble of claim 1.

From the DE 10 2008 002 576 A1 a method for checking the plausibility of objects located using sensors in a driver assistance system is known. Here, a plausibility measure is calculated based on location data about an object, which indicates the probability of existence of the object and the probability of the occurrence of a specific object property. Furthermore, a threshold function is applied to the plausibility measure in order to decide on the existence of the object and whether the object property applies. For a newly recognized object, a time of entry of the object into a location area of the sensor system is calculated based on stored data about the trajectory of the vehicle and the threshold function to be applied is selected depending on this time.

The invention is based on the object of specifying a novel method for a model-based fusion of different sensor modalities.

The object is achieved according to the invention by a method which has the features specified in claim 1.

Advantageous embodiments of the invention are the subject of the subclaims.

In a method for a model-based fusion of different sensor modalities, the existence of objects in a vehicle environment and object-related quality measures are determined based on data recorded using the different sensor modalities.

According to the invention, sensor-dependent trust values for the existence of an object and a probability of continued existence in the vehicle environment are determined as object-related quality measures using a centralized fusion system. The object-related quality measures are determined based on values estimated using dynamic models, with the quality measures being used to merge the data from different sensor modalities of certain existence probabilities of objects.

The present method enables determination of a quality measure in a centralized fusion system that takes tracked objects and untracked detections or features of different sensor modalities into account. It is possible to carry out an improved estimate of the probability of existence due to the model-based determination of the quality measures, in contrast to solutions known from the prior art, which determine the quality measures depending on fixed and sensor-dependent parameters. The determined existence probability can be used when merging the sensor modalities, for example, to improve important components, for example track management or data association, or to optimize the operation of modules downstream of the fusion, for example decision-making by a planning module for trajectory planning to optimize an automated, especially highly automated or autonomously operated vehicle. This makes it easier for the planning module or a downstream decision-making module to calculate the relevance of a merged object and to enable corresponding interventions and actions in the driving operation of the vehicle.

Exemplary embodiments of the invention are explained in more detail below with reference to drawings.

• 1 schematisch eine Draufsicht eines Fahrzeugs mit Radarsensoren und Erfassungsbereichen dieser,
• 2 schematisch eine Seitenansicht des Fahrzeugs gemäß 1 sowie der Radarsensoren und deren Erfassungsbereiche,
• 3 schematisch eine Frontansicht des Fahrzeugs gemäß 1 sowie der Radarsensoren und deren Erfassungsbereiche,
• 4 schematisch eine Draufsicht eines Fahrzeugs mit Kameras und Erfassungsbereichen dieser,
• 5 schematisch eine Seitenansicht des Fahrzeugs gemäß 4 sowie der Kameras und deren Erfassungsbereiche und
• 6 schematisch eine Frontansicht des Fahrzeugs gemäß 4 sowie der Kameras und deren Erfassungsbereiche.

Show:

• 1 schematically a top view of a vehicle with radar sensors and their detection areas,
• 2 schematically a side view of the vehicle according to 1 as well as the radar sensors and their detection areas,
• 3 schematically a front view of the vehicle 1 as well as the radar sensors and their detection areas,
• 4 schematically a top view of a vehicle with cameras and their detection areas,
• 5 schematically a side view of the vehicle according to 4 as well as the cameras and their detection areas and
• 6 schematically a front view of the vehicle 4 as well as the cameras and their detection areas.

Corresponding parts are provided with the same reference numbers in all figures.

In the 1 until 3 is a vehicle 1 with a plurality of radar sensors 2 to 6 and detection areas E2 to E6 of these shown in different views. The 4 until 6 show a vehicle 1, for example the vehicle 1 according to 1 until 3 , with a plurality of Cameras 7 to 14 and detection areas E7 to E14 in different views.

The radar sensor 3 is directed forward and is designed, for example, as a so-called mid-range radar sensor. The further radar sensors 2, 4 to 6 are arranged at outer corners of the vehicle 1 and are designed, for example, as so-called short-range radar sensors.

The camera 7 and the camera 8 are also oriented forward, with the camera 7 being a telephoto camera, for example. The camera 8 is, for example, a wide-angle camera. The cameras 9 to 12 are, for example, so-called surround view cameras and the cameras 13, 14 are side cameras, which are directed laterally to the rear and can therefore take on an exterior mirror function.

The radar sensors 2 to 6 and the cameras 7 to 14 are provided for detecting a vehicle environment, with vehicle functions, for example an automated, in particular a highly automated or autonomous driving operation of the vehicle, depending on data recorded by means of the radar sensors 2 to 6 and cameras 7 to 14 1 can be carried out.

The data is used to identify objects and untracked detections or features in the vehicle environment. To improve detection, the data recorded using the radar sensors 2 to 6 and cameras 7 to 14 and possibly other sensor modalities, for example lidar sensors, are merged.

This fusion is carried out dynamically using a centralized fusion system of the vehicle 1. The task of this dynamic fusion is to determine the positions, dimensions and movements of objects in the vehicle's surroundings, such as other road users. By fusing the different sensor modalities, a global hypothesis is created, for example by using so-called Bayesian tracking methods, for example a Kalman filter or particle filter.

In addition to known measurement inaccuracies, the sensors can deliver false-positive measurements, i.e. sensor measurements without an associated real object. On the other hand, different sensor modalities can provide false-negative measurements, i.e. no measurements of existing objects, due to their different characteristics. However, the case of false-positive and false-negative measurements is usually not taken into account by known tracking and filtering methods, so that the fusion requires additional estimators, such as in “Michael Aeberhard: Object-Level Fusion for Surround Environment Perception in Automated Driving Applications; Dissertation submitted with partial fulfillment of the requirements for the degree of “Doctor Engineer” at the Faculty of Electrical Engineering and Information Technology at the Technical University of Dortmund; Date of approval: May 31, 2017".

To fuse the sensor modalities, a method for fusing existence probabilities of objects detected by the sensors is used, the existence probability being determined according to the following explanations.

First, an estimator for the object state x _k determined by the corresponding sensor and a possible existence probability p ⁱ (∃x _k ) are used as input signals.

• - m_k ({∃}): Existenz-Belief-Masse
• - m_k ({∄}):Nicht-Existenz-Belief-Masse
• - m_k ({∃,∄}):nicht auflösbare Belief-Masse

The probability of existence of a merged object is then determined using the so-called evidence theory of Dempster and Shafer by adding the following masses to the object state:

• - m_k ({∃}): existence belief mass
• - m_k ({∄}):Non-existence belief mass
• - m_k ({∃,∄}): unresolvable belief mass

$$-m_k^i\left(\{\nexists \}\right)=p_p^i\left(x_k\right)p_{trust}^i\left(x_k\right)\left(1-p^i\left(\exists x_k\right)\right)$$

$$-m_i^k\left(\left\{\exists ,\nexists \right\}\right)=1-\left(m_k^i\left(\{\exists \}\right)+m_k^i\left(\{\nexists \}\right)\right)$$

mit:
pⁱ(∃x_k) = von der Sensormodalität abhängige Existenzwahrscheinlichkeit
$p_{trust}^i\left(x_k\right)$

= Sensor-Vertrauenswert
$p_p^i\left(x_k\right)$

= Wahrscheinlichkeit eines Fortbestehens der Existenz eines Objekts berechnet.Belief masses that depend on the respective sensor modality are then calculated according to $$-m_k^i\left(\{\exists \}\right)=p_p^i\left(x_k\right)p_{trust}^i\left(x_k\right)p^i\left(\exists x_k\right)$$

$$-m_k^i\left(\{\nexists \}\right)=p_p^i\left(x_k\right)p_{trust}^i\left(x_k\right)\left(1-p^i\left(\exists x_k\right)\right)$$

$$-m_i^k\left(\left\{\exists ,\nexists \right\}\right)=1-\left(m_k^i\left(\{\exists \}\right)+m_k^i\left(\{\nexists \}\right)\right)$$

with:
p ⁱ (∃x _k ) = probability of existence dependent on the sensor modality
$p_{trust}^i\left(x_k\right)$

= Sensor trust score
$p_p^i\left(x_k\right)$

= Probability of continued existence of an object calculated.

The merged masses are then updated according to the rules of the evidence theory of Dempster and Shafer.

bestimmt.The probability of existence is then according to $$-p\left(\exists x_k\right)=m_k\left(\{\exists \}\right)+\alpha m_k\left(\left\{\exists ,\nexists \right\}\right)$$

certainly.

Based on the vehicle 1 shown with the radar sensors 2 to 6 and cameras 7 to 14, a so-called reengineering of the corresponding sensors based on an object position in sensor data preprocessing takes place depending on the probability of existence determined in each case. For the radar sensors 2 to 6, this is done, for example, in such a way that when the object is in the detection range E3 of the radar sensor 3, the object is assigned to this radar sensor 3. Otherwise the object is assigned to the next radar sensor 2, 4 to 6. A similar approach is carried out for cameras 7 to 14, with the object being prioritized to the front telephoto camera (camera 7).

The sensor trust values and probabilities of the continued existence of the object used to determine the existence probability are calculated using dynamic models to describe the fusion of the existence probabilities.

eines in Abhängigkeit einer jeweiligen Sensormodalität gewählten Vertrauenswert-Parameters $c_{trust}^i,$

einer in Abhängigkeit einer jeweiligen Sensormodalität modellierten Detektionswahrscheinlichkeit p_D(x_k) und einer Übereinstimmung von Position p_{pos_conf}(x_k) und Geschwindigkeit p_{vei_conf}(x_k) mittels der Daten der verschiedenen Sensormodalitäten ermittelter Objekthypothesen für das Objekt ermittelt.The corresponding sensor trust value is a sensor-dependent trust value for the existence of an object, which is referred to below as just the trust value. This trust value depends only on the current state of the corresponding fusion object and the sensor measurement and is determined according to $$p_{trust}^i\left(x_k\right)=c_{trust}^i{p}_D\left(x_k\right)p_{pOs\_cOnf}\left(x_k\right)p_{vel\_cOnf}\left(x_k\right)$$

a trust value parameter selected depending on a respective sensor modality $c_{trust}^i,$

a detection probability p _D (x _k ) modeled depending on a respective sensor modality and a match between position p _{pos_conf} (x _k ) and speed p _{vei_conf} (x _k ) determined object hypotheses for the object using the data from the different sensor modalities.

That is, the confidence value of a particular tracking or detection depends on whether an artifact under consideration is either a tracked object or a processed feature or a raw detection from a particular sensor type. The confidence value is derived based on detection models for that particular object, observation characteristics and acceptance criteria of that object in the fusion system. The model can be further tuned based on the type of object by also taking semantic information into account.

mittels eines Alters p_LT des Objekts bzw. einer Spur des Objekts (exponentiell verteilt), einer Anzahl p_O von Detektionen des Objekts durch die verschiedenen Sensormodalitäten innerhalb eines vorgegebenen Zeitraums (BetalBernoulli Verteilung), in Abhängigkeit einer jeweiligen Sensormodalität ermittelter Fortbestehenswahrscheinlichkeiten p_S mittels der Daten der verschiedenen Sensormodalitäten ermittelter Objekthypothesen für das Objekt, einer Klassifizierung p_C des Objekts und einer Veränderung p_CR eines aus einer Differenz zwischen einer prädizierten Messung und einer tatsächlichen Messung des Objekts gebildeten Residuums, um langzeitbeobachtete Ziele zu berücksichtigen, durchgeführt.The model-based determination of the probability of the continued existence of an object is carried out over several time steps $$p_p^i\left(x_k\right)=p_S\left(x_k\right)\cdot p_O\left(x_k\right)\cdot p_{LT}\left(x_k\right)\cdot p_C\left(x_k\right)\cdot p_{CR}\left(x_{k-1}\right)$$

by means of an age p _LT of the object or a track of the object (exponentially distributed), a number p _O of detections of the object by the various sensor modalities within a predetermined period of time (BetalBernoulli distribution), survival probabilities p _S determined depending on a respective sensor modality by means of the Data from the various sensor modalities determined object hypotheses for the object, a classification p _C of the object and a change p _CR of a residual formed from a difference between a predicted measurement and an actual measurement of the object in order to take long-term observed targets into account.

Situational models are included to ensure that objects that are relevant to the safety aspects of automated driving are not impaired.

Claims (3)

Method for a model-based fusion of different sensor modalities, wherein the existence of objects in a vehicle environment and object-related quality measures are determined based on data recorded by means of the different sensor modalities, characterized in that - as object-related quality measures, sensor-dependent trust values for the existence of an object and a probability of continued existence the existence in the vehicle environment can be determined by means of a centralized fusion system, - the object-related quality measures are determined based on values estimated using dynamic models and - based on the quality measures, a fusion is carried out using the data from different sensor modalities of certain existence probabilities of objects. Procedure according to Claim 1 , characterized in that the model-based determination of the trust values for the existence of an object depending on a current object state by means of - a trust value parameter selected depending on a respective sensor modality, - a detection probability modeled depending on a respective sensor modality and - a match between position and speed Object hypotheses determined for the object are carried out using the data from the various sensor modalities. Procedure according to Claim 1 or 2 , characterized in that the model-based determination of the probability of the continued existence of an object over several time steps by means of - an age of the object, - a number of detections of the object by the different sensor modalities within a predetermined period of time - continued existence probabilities determined depending on a respective sensor modality the data from the various sensor modalities determined object hypotheses for the object, - a classification of the object and - a change in a residual formed from a difference between a predicted measurement and an actual measurement of the object.

Images