EP4371019A1

EP4371019A1 - Method and control unit for using uncertainty of object detection in a vehicle function

Info

Publication number: EP4371019A1
Application number: EP21745952.8A
Authority: EP
Inventors: Jan HAUG; Stefano Gasperini; Mohammad-Ali Nikouei Mahani; Alvaro Marcos-Ramiro
Original assignee: Bayerische Motoren Werke AG
Current assignee: Bayerische Motoren Werke AG
Priority date: 2021-07-14
Filing date: 2021-07-14
Publication date: 2024-05-22
Also published as: WO2023284953A1

Abstract

A control unit (101) for operating a vehicle function of a vehicle (100) is described. The control unit (101) is configured to determine a plurality of different characteristics of an object (204) which is located in an environment of the vehicle (100), based on a 2D representation of sensor data (205) using a trained neural network (200). The control unit (101) is further configured to determine a plurality of uncertainty values for the corresponding plurality of characteristics based on output data (215, 225, 235) of the neural network (200). In addition, the control unit (101) is configured to operate the vehicle function in dependence of the plurality of uncertainty values for the corresponding plurality of characteristics of the object (204).

Description

Method and Control Unit for Using Uncertainty of Object Detection in a Vehicle Function

The present document is directed at a driver assistance function and/or an autonomous driving function of a vehicle. In particular, the present document is directed at determining an uncertainty measure indicating the uncertainty of object detection performed in the context of a driver assistance and/or an autonomous driving function of a vehicle.

A driver assistance function and/or an autonomous driving function of a vehicle typically make use of means of perception of the environment of the vehicle. In this context, meta-information about the reliability, in particular the certainty, of information extracted from a perceived scene within the environment, such as the space occupied by another vehicle, may be valuable for increasing the reliability of the driver assistance and/or autonomous driving functionality.

Object detection algorithms, however, often perform poorly when it comes to quantifying the reliability of the information that is provided by the object detection algorithm. As a matter of fact, an object detection algorithm may not be aware of its own uncertainty.

The present document is directed at the technical problem of providing an efficient and reliable scheme for determining the uncertainty of an object detection algorithm, in particular for increasing the reliability of a vehicle function. The technical problem is solved by the independent claims. Preferred examples are specified in the dependent claims.

According to an aspect, a control unit for operating a vehicle function of a vehicle (e.g., a truck or a car) is described. The vehicle function may comprise a driver assistance function and/or a function for autonomous driving of the vehicle.

The control unit is configured to determine a plurality of different characteristics of an object which is located in an environment of the vehicle, based on a 2D representation of sensor data (e.g., an input image and/or a 2D representation of lidar and/or radar sensor data) using a trained neural network. The 2D representation of sensor data (notably the input image) may have been captured using an environment sensor (in particular a camera) of the vehicle. In particular, the 2D representation of sensor data may be an RGB image, an RGB-D image, and/or a projection of a lidar point cloud into a 2D image. Hence, the 2D representation of sensor data may comprise multiple matrices of input data (e.g., a matrix for the R channel, a matrix for the G channel, and a matrix for the G channel of an RGB image). Each matrix may comprise W x H pixels. The plurality of characteristics of an object may comprise: the presence of the object (also referred to herein as the “objectness”), the location of the center of the object, the object class (e.g., truck, car, motorcycle, bicycle, pedestrian, etc.), the dimensions (e.g., the width and/or the height) of the object, etc. It should be noted that in case of 3D object detection, the dimensions may further include the length (or the depth) of the object.

The neural network may comprise a core (neural) network configured to process the 2D representation of sensor data to provide intermediate data. Furthermore, the neural network may comprise multiple output networks which are configured to process the intermediate data to provide multiple output tensors, respectively. An output tensor may comprise one or more matrix maps (also referred to herein as heatmaps), wherein each matrix map may comprise w x h pixels for corresponding W x H pixels of the 2D representation of sensor data. Hence, the lower-case letters may refer to the dimensions within an output tensor. The upper case letters may refer to the dimensions within the 2D representation of sensor data. The number of pixels of a matrix map may be smaller than the number of pixels of the 2D representation of sensor data by a pre-determined downsampling factor. The downsampling factor may be 2 or more, or 4 or more, or 9 or more, etc. The 2D representation of sensor data may comprise 100 x 100 or more, or 500 x 500 or more pixels, however, the method may be used for all possible images sizes.

The pixel values of the pixels of a matrix map may be indicative of a characteristic of a detected object. The control unit may be configured to determine the plurality of characteristics of the object and/or the corresponding plurality of uncertainty values based on the one or more output tensors of the neural network.

The core network may comprise an encoder network followed by a decoder network. Alternatively, or in addition, the core network may comprise a CenterNet neural network. An example core network is described in ZHOU, Xingyi; WANG, Dequan; KRAHENBLJHL, Philipp. Objects as points. arXiv preprint arXiv: 1904.07850, 2019, the content of which is incorporated herein in its entirety by reference. The neural network, in particular the core network and the plurality of output networks, is typically trained beforehand using training data (and using state of the art training algorithms). It should be noted that the core network may comprise an alternative architecture to the CenterNet architecture. In particular, the core network may comprise an anchor-less or keypoint estimation object detector neural network.

An object detection scheme (such as the object detection scheme provided by the core network) typically provides a plurality of outputs or characteristics. In particular, each 2D (or 3D) box for a detected objection may provide the following characteristics: class, objectness, location along x, location along y, width, height, etc. In view of this, there is an interest in identifying the uncertainty associated with each characteristic (or aspect) individually, in order to be able to better take into account each characteristic in a downstream task such as trajectory planning.

In view of this, the control unit is further configured to determine a plurality of uncertainty values for the corresponding plurality of characteristics based on output data of the neural network, in particular based on the output tensors of the different output networks. Hence, for each of the different characteristics an uncertainty value may be determined. The uncertainty value for a characteristic may indicate how uncertain the value of the determined characteristic of the object is. The uncertainty values may span between 0 and 1, wherein 0 may indicate no uncertainty (i.e., full certainty) and wherein 1 may indicate full uncertainty (i.e., no certainty).

The uncertainty values for the different characteristics may be determined using the same neural network as is used for determining the (values of the) characteristics of the object, thereby allowing the uncertainty values to be determined in a precise and efficient manner.

Furthermore, the control unit may be configured to operate the vehicle function in dependence of the plurality of uncertainty values for the corresponding plurality of characteristics of the object. By taking into the account the uncertainty of multiple characteristics of a detected object when operating a vehicle function, the precision and the reliability of the vehicle function may be improved.

The control unit may be configured to determine a combined uncertainty value based on the plurality of uncertainty values, in particular based on a weighted sum (notably a weighted average) of the plurality of uncertainty values. The vehicle function of the vehicle may then be operated in a particularly efficient manner in dependence of the combined uncertainty value.

The control unit may be configured to apply a plurality of weights to the corresponding plurality of uncertainty values to determine the weighted sum of the plurality of uncertainty values. The plurality of weights may be dependent on the sensitivity of the precision of the neural network for determining the corresponding plurality of different characteristics of the object. The sensitivity of the precision of the neural network may be determined by varying each characteristic independently for determining the impact of each of the different characteristics. Based on this, corresponding weights for the different characteristics of the object may be determined.

In particular, the weights for the uncertainty value of a particular characteristic may be relatively high, if the sensitivity of the precision of the neural network for determining the particular characteristic is relatively high (and vice versa). As a result of this, the combined uncertainty value may be determined in a particularly precise manner, thereby further increasing the precision and the reliability of the vehicle function. The neural network may comprise a first output network which is configured to provide a first output tensor. The first output tensor may comprise a plurality of class (matrix) maps for a corresponding plurality of object classes. Example object classes are: a truck, a car, a motorcycle, a bicycle, a pedestrian, etc.

A pixel value of a particular pixel of a class map for a particular object class may be indicative of the certainty and/or the probability that the center of an object which belongs to the particular object class is located at the particular pixel. As such, the first output tensor may be used for determining the objectness of an object, the location of an object, and/or the object class of an object.

The first output network may comprise a first sub-network which is configured to provide a feature matrix of feature vectors based on the 2D representation of sensor data. A feature vector may have a dimension of 100 or more. Furthermore, the first output network may comprise a plurality of second sub-networks for the corresponding plurality of object classes. The feature matrix may be a matrix with individual matrix points (corresponding to individual pixels). Each matrix point may comprise a feature vector. The feature vector may comprise a set of features (e.g., each feature may correspond to a different dimension of the feature vector).

Hence, the overall neural network may comprise a core network, wherein the core network may comprise an input layer (of neurons) and one or more intermediate layers (of neurons). Each layer of the core network typically comprises a certain number of neurons, wherein each neuron performs mathematical operations such as a multiplication, an addition of an offset value, and/or a sigmoid function, etc. The number of neurons per layer may be relatively high, e.g., 100 or more, or 1000 or more. Furthermore, the neuronal network may comprise multiple output networks

(which are also referred to herein as heads). An output network may comprise one or more layers of neurons. An output network may be configured to take the output of the core network as an input and to perform further processing to provide an output tensor. An output network may be sub-divided into multiple sub-networks (which are arranged in series), wherein a sub-network may comprise one or more layers of neurons.

The second sub-network for an object class may be configured to, for each feature vector of the feature matrix (which are provided by the first sub-network), transform the feature vector into a corresponding hyperspace vector within a hyperspace, using a class-specific (linear) transformation. The hyperspace vector may have a higher dimension than the feature vector. Furthermore, the second sub-network for an object class may be configured to, for each feature vector of the feature matrix, determine a distance value of the hyperspace vector from a class-specific centroid within the hyperspace. In addition, the second sub-network for an object class may be configured to, for each feature vector of the feature matrix, map the distance value to a certainty and/or probability value using a pre determined certainty and/or probability distribution. For this purpose, the class- specific centroid within the hyperspace may be used. The certainty and/or probability value may be indicative of the objectness certainty. The second sub network, notably the class-specific transformation and the class-specific centroid, is typically trained beforehand based on training data.

An example first output network, notably an example second sub-network, is described in VAN AMERSFOORT, Joost, et al. Simple and scalable epistemic uncertainty estimation using a single deep deterministic neural network. International Conference on Machine Learning (ICML), 2020, the content of which is incorporated herein in its entirety. In particular, the above document describes a scheme for creating a hyperspace (and a corresponding transformation) for determining the certainty and/or probability value. The control unit may be configured to determine the presence of the object within a sub-region of pixels of the 2D representation of sensor data (i.e., the objectness) based on the pixel values of the pixels within a corresponding sub-region of the first output tensor. In particular, a sub-region of pixels within the first output tensor having pixel values which indicate a relatively high certainty of the presence of an object may be identified for determining the presence of an object. In other words, due to the incorporated uncertainty determined via the hyperspace as described above, the pixel values represent (for each class) the certainty of having an object at that location (objectness). Therefore, the first output tensor may be used to determine the presence of objects at each location for each class, alongside with an estimation of the objectness uncertainty.

Alternatively, or in addition, the control unit may be configured to determine an uncertainty value for the presence of the object within the sub-region of pixels of the 2D representation of sensor data based on the maximum pixel value of the pixels within the corresponding sub-region of the first output tensor. In particular, the maximum pixel value may be indicative of the uncertainty of the presence of the object. The uncertainty value may decrease with increasing maximum pixel value (and vice versa). By doing this, the uncertainty value for the objectness may be determined in an efficient and precise manner.

The control unit may be configured to determine an object class of the object within the sub-region of pixels of the 2D representation of sensor data by comparing the pixel values of the pixels within the corresponding sub-regions for the plurality of different class maps of the first output tensor. In particular, the class map among the plurality of different class maps may be identified, which exhibits one or more pixel values (within the sub-region) indicating the relatively highest certainty and/or probability of the presence of the object. As a result of this, the object class can be determined in a reliably manner. Alternatively, or in addition, the control unit may be configured to determine an uncertainty value for the object class of the object within the sub-region of pixels of the 2D representation of sensor data by determining a variance of the pixel values of the pixels within the corresponding sub-region across the plurality of different class maps of the first output tensor. In particular, it may be determined how different the pixel values are within the different class maps. If the pixel values are relatively similar within the different class maps, then the uncertainty value it typically relatively high (and vice versa). By doing this, the uncertainty value for the object class may be determined in an efficient and precise manner.

The control unit may be configured to determine the location of the center the object within the sub-region of pixels of the 2D representation of sensor data by determining the maximum pixel value of the pixels within the corresponding sub- region of the first output tensor. The pixel location of the pixel having the maximum pixel value may be used as the location of the center of the object.

Alternatively, or in addition, the control unit may be configured to determine an uncertainty value for the location of the center of the object within the sub-region of pixels of the 2D representation of sensor data by determining the variance of the pixel values of the pixels within the corresponding sub-region of the first output tensor. A relatively high variance may be indicative of a relatively low uncertainty value (and vice versa). By doing this, the uncertainty value for the object location may be determined in an efficient and precise manner.

Alternatively, or in addition, the control unit may be configured to determine an uncertainty value for the location of the center of the object within the sub-region of pixels of the 2D representation of sensor data by determining the deviation of the actual and/or the predicted distribution of the pixel values of the pixels within the corresponding sub-region of the first output tensor from a reference distribution. The reference distribution may be dependent on the object class of the object (and in particular on the predicted or typical dimension of an object of this object class). The actual and/or the predicted distribution of the pixel values of the pixels within the corresponding sub-region of the first output tensor may be determined to represent the (average) pixel values of the pixels as a function of the distance from the (predicted) location of the center of the object.

The uncertainty value may increase as the deviation of the actual distribution from the reference distribution increases, in particular, as the actual distribution becomes more flat (or vice versa). The actual and/or the reference distribution may indicate the pixel value as a function of the distance of the respective pixel from the center pixel (which corresponds to the pixel having the maximum pixel value). By doing this, the uncertainty value for the object location may be determined in an efficient and precise manner.

The control unit may be configured to determine an uncertainty value for the location of the object center independently along each axis (x horizontal, y vertical). This may be done by comparing the actual distribution and the reference distribution at different pixel locations for each axis. This may lead to a horizontal uncertainty which differs from the vertical uncertainty, there providing valuable information regarding the uncertainty of the location of the center of the object. In other words, the control unit may be configured to determine a deviation of the actual distribution of the pixel values of the pixels within the corresponding sub- region of the first output tensor from the reference distribution for the height direction (e.g., the vertical y axis) and for the width direction (e.g., the horizontal y axis) separately, to determine an uncertainty value for the location of the center of the object within the sub-region of pixels of the 2D representation of sensor data for the height direction and for the width direction separately. As a result of this, the precision of the uncertainty information may be further increased.

The neural network may comprise a second output network (also referred to herein as a head) which is configured to provide a second output tensor. The second output tensor may comprise one or more dimension maps for one or more dimensions, in particular for the width and/or the height (of the bounding box), of the object.

The control unit may be configured to determine a dimension of the object (notably of the bounding box of the object) based on the pixel value of the center pixel of the one or more dimension maps (the center pixel may be the pixel corresponding to the location of the center of the object). Furthermore, the control unit may be configured to determine an uncertainty value for the dimension of the object based on pixel values of the center pixel and one or more neighboring pixels of the one or more dimension maps, in particular based on a similarity of the pixel value of the center pixel and the pixel values of the one or more neighboring pixels. A relatively high similarity is typically indicative of a relatively low uncertainty (and vice versa). By doing this, the uncertainty value for the dimensions of an object may be determined in an efficient and precise manner.

The neural network may comprise a third output network which is configured to provide a third output tensor. The third output tensor may comprise one or more offset maps for one or more offsets, in particular offsets with regards to the horizontal and/or the vertical direction, of the location of the center of the object. The third output tenser may be used to compensate the fact that the number of pixels of the output tensors may be downsampled with regards to the number of pixels of the 2D representation of sensor data. The control unit may be configured to determine one or more characteristics of the object and/or one or more corresponding uncertainty values based on the third output tensor, thereby further increasing the precision of the uncertainty values.

The control unit may be configured to perform 3D object detection. In this case, the neural network may comprise a fourth output network which is configured to provide information regarding the distance of each pixel from the camera, which is used to determine the object location along the depth axis. The distance information may be provided within a fourth output tensor. Additionally, an uncertainty for this output characteristic (or aspect) may be extracted similarly to the uncertainty regarding the object dimensions.

In other words, the neural network may comprise a fourth output network configured to provide a fourth output tensor, wherein the fourth output tensor comprises a distance map comprising pixel values indicating a distance of a center of the object from the environment sensor having captured the 2D representation of sensor data. The control unit may be configured to determine a distance of the center of the object from the environment sensor based on the pixel values of a center pixel of the distance map. Furthermore, the control unit may be configured to determine an uncertainty value for the distance of the center of the object from the environment sensor by comparing the pixel value of the center pixel value with pixel values of the neighbors of the center pixel.

According to a further aspect, a (street) vehicle (e.g., a car, a truck, a bus, a motorcycle, etc.) is described, which comprises the control unit described in the present document.

According to a further aspect, a method for operating a vehicle function of a vehicle is described. The method comprises determining a plurality of different characteristics of an object which is located in an environment of the vehicle, based on a 2D representation of sensor data (e.g., an input image) using a trained neural network. Furthermore, the method comprises determining a plurality of uncertainty values for the corresponding plurality of characteristics based on output data of the neural network. In addition, the method may comprise operating the vehicle function in dependence of the plurality of uncertainty values for the corresponding plurality of characteristics of the object. According to a further aspect, a software program is described. The software program may be adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on the processor.

According to another aspect, a storage medium is described. The storage medium may comprise a software program adapted for execution on a processor and for performing the method steps outlined in the present document when carried out on the processor.

According to a further aspect, a computer program product is described. The computer program may comprise executable instructions for performing the method steps outlined in the present document when executed on a computer.

It should be noted that the methods and systems including its preferred embodiments as outlined in the present patent application may be used stand alone or in combination with the other methods and systems disclosed in this document. Furthermore, all aspects of the methods and systems outlined in the present patent application may be arbitrarily combined. In particular, the features of the claims may be combined with one another in an arbitrary manner.

The invention is explained below in an exemplary manner with reference to the accompanying drawings, wherein

Fig. 1 shows example components of a vehicle;

Fig. 2 shows an example neural network for 2D (two-dimensional) object detection;

Fig. 3 a shows an example transformation of a feature vector into a certainty and/or probability value;

Fig. 3b illustrates an example of different sub-regions of certainty and/or probability values;

Fig. 3 c illustrates an example of corresponding sub-regions of certainty and/or probability values for different object classes; Fig. 4 shows an example of different dimension values for different objects; and Fig. 5 shows a flow chart of an example method for using uncertainty values for operating a vehicle function.

As indicated above, the present document is directed at determining the uncertainty of an object detection algorithm in an efficient and reliable manner. Hence, in the present document, a method for estimating a comprehensive uncertainty for object detection is described. The determined uncertainty measure provides information on how much a detection of an object can be trusted with respect to one or more different detection aspects or characteristics of the object. This uncertainty measure may be used within a downstream task, such as trajectory planner, thereby increasing e.g., the safety of a vehicle.

An object detection algorithm may be configured to provide a plurality of different outputs or characteristics regarding an object. Example outputs are: object presence, location of the object, size of the object, etc. The method described in the present document may estimate the uncertainty for each one of the different outputs of the objection detection algorithm. The different uncertainties for the different outputs may then be used (standalone and/or in combination) for a vehicle function (such as path planning). By providing and/or by taking into account the uncertainty of the different outputs, a comprehensive uncertainty measure may be provided.

The method may be based on a 2D (two-dimensional) or 3D (three-dimensional) object detection algorithm. Fig. 1 shows an example vehicle 100 which comprises an environment sensor 102 (such as a camera, a radar sensor, a lidar sensor, etc.) which is configured to provide sensor data regarding the environment of the vehicle 100. A calculation unit 101 of the vehicle 100 may be configured to perform an object detection algorithm for detecting one or more different objects within the environment of the vehicle 100 based on the sensor data. The one or more detected objects may be taken into account within a driver assistance function and/or an autonomous driving function of the vehicle 100, e.g., for operating one or more actuators 103 of the vehicle 100.

Fig. 2 shows an example scheme 200 for performing object detection based on input sensor data 205, in particular based on an input image captured by a camera of the vehicle 100. The object detection scheme 200 may comprise a trained neural network (notably a core network) 201 (e.g., with an encoder network 202 and a decoder network 203), which is configured to provide information (also referred to herein as intermediate data 210) regarding one or more different objects 204 which are present within the sensor data 205. The object detection scheme 200 may comprise a plurality of different output networks 211, 212, 221, 231 (also referred to herein as “heads) for a corresponding plurality of characteristics (or outputs) regarding an object 204. Example characteristics of an object 204 are:

• the location of the object 204;

• the existence (or non-existence) of an object 204 at a particular location (also referred to herein as the objectness);

• the class of the object 204 (such as car, truck, pedestrian, bike);

• the dimensions (such as the height and/or the width) of the object 204; and/or

• a (sub-pixel) offset of the center of the object 204.

In the example of Fig. 2, the (first) output network 211, 212 is directed at determining the location, the objectness and/or the class of an object 204 (as part of a first output tensor 215), the (second) output network 221 is directed at determining the dimensions of an object 204 (as part of a second output tensor 225), and the (third) output network 231 is directed at determining the offset of the center of an object 204 (as part of a third output tensor 235).

As an object detector 201 the so-called CenterNet architecture, which is an anchor-free one-stage object detection algorithm, may be used. The CenterNet architecture is described in ZHOU, Xingyi; WANG, Dequan; KRAHENBUHL, Philipp. Objects as points. arXiv preprint arXiv: 1904.07850, 2019, which is incorporated herein in its entirety by reference.

CenterNet is compatible with a variety of backbones (network architectures), whose output features are up-sampled with a decoder 203 to generate a downscaled map 210 of features of an input image 205. For the desired output predictions, multiple "heads" 211, 212, 221, 231 may be used, wherein a head typically comprises a set of layers that extract respective predictive values (for different outputs or characteristics) from a feature-pixel and its neighbors. For 2D object detection, a classification head 211, 212 may be used, which outputs a class heatmap, trained to detect the objects centers, that are responsible for the objectness (existence of an object) at each location, as well as the object's class. Furthermore, a dimensions head 221 may be used, which outputs the dimensions heatmap estimating the respective object's bounding box size, and/or an offset head 331 outputting the offset heatmap estimating the (sub-pixel) offset of the object center. Further output aspects may be added as additional heads or output- channels (e.g., the object distance for 3D object detection). Each head 211, 212, 221, 231 predicts one or more aspects of the output for each feature-map pixel. A final bounding box output may be generated by filtering and combining the outputs of the different heads 211, 212, 221, 231.

The first output network 211, 212 may be configured to provide a class heatmap 215 for N different classes (with N>1). The heatmap (also referred to herein as a tensor) may comprise w x h pixels for each of the N different classes. In other words, the class heatmap may comprise N different class-specific heatmaps for the N different classes. A class-specific heatmap for a particular class (e.g., for a car or for a bike) may indicate for each pixel of the class-specific heatmap the probability and/or certainty that the center of an object 204 that belongs to the particular class is located at the respective pixel. The probability may e.g., be indicated by a greyscale value (e.g., a value between 0 and 1), wherein a dark pixel (e.g., a value of 1) may indicate a relatively high probability and a light pixel (e.g., a value of 0) may indicate a relatively low probability.

Hence, the class heatmap 215 may indicate for one or more different object- classes, a probability and/or certainty distribution 214 of the center of an object 204.

The class heatmap 215 may be determined using a first output network 211, 212 which comprises a first sub-network 211 configured to provide a set of features (i.e., a feature vector) for each pixel of the to-be-determined class heatmap 215. The set of features (i.e., the feature vector) may comprise K features for each pixel, with K being an integer which is e.g., greater than 100. Hence, the first sub network 211 may provide a feature map with w x h pixels, which each comprises a feature vector with K features (or layers) for each pixel.

Furthermore, a second sub-network 212 may be used to determine the class heatmap 215 based on the feature map. The second sub-network 212 may be configured to transform each feature vector into a hyperspace vector 301, wherein the hyperspace vector 301 may have a higher dimension than the corresponding feature vector (see Fig. 3a). By way of example, the dimension Q of the hyperspace vector 301 may be higher than the dimension K of the feature vector. The transformation between a feature vector and the corresponding hyperspace vector 301 may be performed using a linear transformation. For each one of the N different classes a different transformation may be provided, such that N different hyperspace vectors 301 may be determined based on a corresponding feature vector.

A hyperspace vector 301 may be viewed as a point 302 within the Q-dimensional hyperspace 303 (which is illustrated in Fig. 3a as a three-dimensional space). A typically and/or ideal representative of a particular class within the hyperspace 303 may correspond to the centroid 304 within the Q-dimensional space 303. Each of the N different classes may exhibit a different centroid 304.

Hence, the different feature vectors of the feature map may be transformed into a Q-dimensional hyperspace 303 using a class-specific transformation for a particular class, thereby providing a map of hyperspace vectors 301 for the particular class. Furthermore, a Q-dimensional centroid 304 may be provided for the particular class. For each of the hyperspace vectors 301, a distance value 305 (e.g., the Euclidian distance) between the respective hyperspace vector 301 and the centroid 304 may be determined, thereby providing a w x h map of distance values 305. A pre-determined mapping function 307, e.g., a Gaussian function, may be used to map a distance value 305 to a corresponding certainty or probability value 306, thereby providing a w x h map of certainty or probability values 306, which corresponds to the class heatmap for a particular class. In a similar manner, the class heatmaps for the N different classes may be determined.

Hence, a class heatmap 215 with N different sub-heatmaps for the N different classes may be provided, wherein a pixel of a sub-heatmap for a particular class indicates the certainty or probability value 306 (e.g., between 0 and 1) for the center of an object 214 of the particular class being located at the position of the pixel. The certainty or probability value 306 may be taken directly as an indicator of the certainty (or uncertainty) of the objectness (i.e., of the presence of the center of an object 214 at the position of the pixel).

The class heatmap 215, in particular each sub-heatmap, may be viewed as a two- dimensional function with local maxima, wherein a local maximum may indicate the location of an object 214. Fig. 3b illustrates two different distributions of certainty or probability values 306 in the vicinity (i.e., in a sub-region) of a local maximum or peak 311. In the example on the left side, the values 306 in the neighborhood of the peak 311 drop relatively fast with increasing distance from the peak 311, thereby indicating a relatively high certainty of the location estimate. On the other hand, in the example on the right side, the values 306 in the neighborhood of the peak 311 stay similar to the value 306 of the peak 311, thereby indicating a relatively low certainty of the location estimate.

A reference distribution of the certainty or probability values 306 within the neighborhood (i.e., the sub-region) of a peak 311 that indicates the location of an object 214 may be provided. The reference distribution may indicate the ground truth. Different reference distributions may be provided for the N different classes. A deviation of the actual distribution and the reference distribution may be determined, to determine an estimate of the location uncertainty. In particular, the location uncertainty may increase as the deviation of the actual distribution from the reference distribution increases (notably, if the actual distribution becomes more and more flat).

The reference distribution may also be referred to as a pseudo-reference distribution. During inference time of the neural network (for performing object detection), a pseudo-reference distribution may be created by centering a pre determined distribution (for a particular object class) at the predicted center of the object. The pre-determined distribution may depend on the predicted dimensions of the object. Hence, the reference distribution may be generated based on the predicted center of the object and/or based on the predicted dimensions of the object (which are provided by the neural network). This reference distribution may be compared to the actual and/or predicted distribution, and the deviation may be used as a measure of the location uncertainty.

For determining an estimate of the class uncertainty, the different distributions of the values 306 around a peak 311 for the N different classes may be compared, as illustrated in Fig. 3c. In particular, Fig. 3c shows three different distributions 321, 322, 323 of pixel values of pixels within a sub-region for three different classes.

In the example on the upper side of Fig. 3c, the first distribution 321 for the first class has significantly higher values 306 than the other distributions 322, 323 for the other classes, thereby indicating that the certainty for the object 204 belonging to the first class is relatively high. On the other hand, in the example on the lower side of Fig. 3c, the three different distributions 321, 322, 323 have similar values 306, thereby indicating that the certainty for the object 204 belonging to one of the classes is relatively low.

The second output network 221 may provide a heatmap 225 which is indicative of the dimensions (e.g., the width and/or the height) of an object 204. The heatmap 225 may comprise w x h pixels for each dimension. In particular, the dimension heatmap 225 may comprise a width heatmap with w x h pixels for the width and a height heatmap with w x h pixels for the height. A pixel value may indicate the dimension (i.e., the width or the height) of an object 204 at the location of the pixel.

As a result of this, the dimension heatmap 225 may indicate the dimensions of an object 204 at w x h different pixel locations (assuming that the center of the object 204 is located at the respective pixel location).

Fig. 4 indicates example dimensions which are indicated by the dimension heatmap 225. In particular, Fig. 4 indicates the dimensions 401 for a pixel location which corresponds to the center of a detected object 204 (as indicated by the class heatmap 215). Furthermore, Fig. 4 indicates the dimensions 402 for pixel locations within the direct neighborhood of the center of the detected object 204.

The upper part of Fig. 4 indicates a situation, where the dimensions of the center location 401 and the dimensions of the neighborhood locations 402 are relatively similar. This is an indication for a relatively high certainty of the dimensions of the detected object 204. On the other hand, the lower part of Fig. 4 indicates a situation, where the dimensions of the center location 401 and the dimensions of the neighborhood locations 402 strongly differ from one another. This is an indication for a relatively low certainty of the dimensions of the detected object 204.

Hence, for each dimension, the variance of dimension values of the center location 401 and the neighborhood locations 402 may be determined. A relatively high variance is indicative of a relatively high uncertainty, and a relatively low variance is indicative of a relatively low uncertainty.

As indicated above, the object size (i.e., the bounding box dimensions), may be provided by a dimensions heatmap 225. For 2D object detection, such a heatmap 225 provides two box dimensions (width and height) for each pixel. For a particular object 204, the box dimensions may be extracted from the center pixel 401 corresponding to the peak of the particular object 204 within the corresponding class heatmap 215.

The dimensions uncertainty may be determined based on the neighboring values of the neighboring pixels 402 of the center pixel 401 in the dimensions heatmap 225. The higher the agreement on the bounding box size among the adjacent pixels 401, 402, the lower the uncertainty. This indicator may be weighted by considering the objectness score associated with each bounding box prediction (higher score => more impact => higher weight), as well as the distance of its object center location to the main prediction (less distance => more impact => higher weight). The dimensions uncertainty may be separated spatially (width and height for 2D detections) and a covariance matrix may be calculated.

In the case of 3D object detection further outputs or characteristics of an object 204 may be considered, such as the rotation of the object 204, a third dimension of the bounding box of the object 204 and/or the distance of the object 204 from the environment sensor 102. The uncertainty for these additional outputs may be determined in a similar manner. Hence, different individual uncertainty estimates may be provided for different characteristics of a detected object 204. Example characteristics are: the objectness (i.e., the actual presence of the object 204 at a given location), the location of the object 204, the class of the object 204 and/or the dimensions (e.g., the width and/or the height) of the object 204. The different uncertainty estimates may be used individually within a vehicle function, such as autonomous driving. Furthermore, a comprehensive uncertainty estimate may be determined, e.g., as a weighted average of the different individual uncertainty estimates.

For providing a comprehensive uncertainty estimate which captures the total uncertainty of object detection, the different uncertainty estimates may be combined into a single scalar value. The different uncertainty estimates may be weighted depending on their respective impact on the detection quality.

A sensitivity analysis of the detector's mean average precision (mAP) may be performed to provide a common performance metric for object detection, with respect to each individual dimension. Noise may be added on each input dimension separately (one at a time) and the algorithms mAP performance may be evaluated. This results in a local derivative of the performance w.r.t. each aspect and/or characteristic of the detection output. The higher this gradient is, the higher is the impact of such aspect or characteristic on the total output quality. With a set of weights depending on these gradients, a weighted average of the various uncertainties over each aspect of the output may be calculated, leading to a combined and comprehensive uncertainty value. On the other hand, individual measures of the uncertainty for the various different aspects (or characteristics) of the bounding box for an object 204 may be provided. Hence, uncertainty values may be provided individually (for each aspect) or combined.

Fig. 5 shows a flow chart of an example (e.g., Computer-implemented) method 500 for operating a vehicle function of a vehicle 100. The method 500 may be performed by a control unit 101 of the vehicle 100. The method 500 comprises determining 501 a plurality of different characteristics of an object 204 which is located in an environment of the vehicle 100, based on an input image 205 using a trained neural network 200. The input image 205 may have been captured by an environment sensor 102 of the vehicle 100. The different characteristics of the object 204 may comprise: the existence (i.e., the objectness) of the object 204, the location of the center of the object 204, the object class (e.g., vehicle, bicycle, pedestrian, etc.) of the object 204, the dimensions of the bounding box (i.e., the spatial extent) of the object 204, etc. The neural network 200 has typically been trained beforehand using training data.

In addition, the method 500 comprises determining 502 a plurality of uncertainty values for the corresponding plurality of characteristics based on output data 215, 225, 235 of the neural network 200. The neural network 200 may comprise different output networks 211, 212, 221, 231 for different characteristics of the object 204. The uncertainty values for the different characteristics may be determined based on the output data (in particular, based on the output tensors) 215, 225, 235 of the different output networks 211, 212, 221, 231. Hence, the same output networks 211, 212, 221, 231 may be used for determining the characteristics (i.e., values of the characteristics) and uncertainty values for the characteristics.

The method 500 may further comprise operating 503 the vehicle function in dependence of the plurality of uncertainty values for the corresponding plurality of characteristics of the object 204. By taking into account the uncertainty values for the different characteristics of a detected object 204, the precision and/or the reliability of a vehicle function may be increased.

It should be noted that the description and drawings merely illustrate the principles of the proposed methods and systems. Those skilled in the art will be able to implement various arrangements that, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope. Furthermore, all examples and embodiment outlined in the present document are principally intended expressly to be only for explanatory purposes to help the reader in understanding the principles of the proposed methods and systems. Furthermore, all statements herein providing principles, aspects, and embodiments of the invention, as well as specific examples thereof, are intended to encompass equivalents thereof.

Claims

1) A control unit (101) for operating a vehicle function of a vehicle (100); wherein the control unit (101) is configured to - determine a plurality of different characteristics of an object (204) which is located in an environment of the vehicle (100), based on a two-dimensional, referred to as 2D, representation of sensor data (205) using a trained neural network (200);

- determine a plurality of uncertainty values for the corresponding plurality of characteristics based on output data (215, 225, 235) of the neural network (200); and

- operate the vehicle function in dependence of the plurality of uncertainty values for the corresponding plurality of characteristics of the object (204).

2) The control unit (101) of claim 1, wherein

- the neural network (200) comprises a core network (201) configured to process the 2D representation of sensor data (205) to provide intermediate data (210); - multiple output networks (211, 212, 221, 231) configured to process the intermediate data (210) to provide multiple output tensors (215, 225, 235), respectively; and

- the control unit (101) is configured to determine the plurality of characteristics of the object (204) and/or the corresponding plurality of uncertainty values based on the multiple output tensors (215, 225,

235).

3) The control unit (101) of claim 2, wherein

- the core network (201) comprises an encoder network (202) followed by a decoder network (203); and/or

- the core network (201) comprises a CenterNet neural network; and/or - an anchor-less or keypoint estimation object detector neural network.

4) The control unit (101) of any previous claims, wherein

- the neural network (200) comprises a first output network (211, 212) configured to provide a first output tensor (215);

- the first output tensor (215) comprises a plurality of class maps for a corresponding plurality of object classes;

- a class map for an object class comprises w x h pixels for corresponding W x H pixels of the 2D representation of sensor data (205);

- in particular, a number of pixels of a class map is smaller than a number of pixels of the 2D representation of sensor data (205) by a pre-determined downsampling factor;

- a pixel value of a particular pixel of a class map for a particular object class is indicative of a certainty and/or probability (307) that a center of an object (204) which belongs to the particular object class is located at the particular pixel.

5) The control unit (101) of claim 4, wherein the control unit (101) is configured to

- determine the presence of the object (204) within a sub-region of pixels of the 2D representation of sensor data (205) based on pixel values of the pixels within a corresponding sub-region of the first output tensor (215); and

- determine an uncertainty value for the presence of the object (204) within the sub-region of pixels of the 2D representation of sensor data (205) based on a maximum pixel value of the pixels within the corresponding sub-region of the first output tensor (215).

6) The control unit (101) of any of claims 4 to 5, wherein the control unit (101) is configured to - determine an object class of the object (204) within a sub-region of pixels of the 2D representation of sensor data (205) by comparing pixel values of the pixels within a corresponding sub-region for the plurality of different class maps of the first output tensor (215); and

- determine an uncertainty value for the object class of the object (204) within the sub-region of pixels of the 2D representation of sensor data (205) by determining a variance of the pixel values of the pixels within the corresponding sub-region across the plurality of different class maps of the first output tensor (215).

7) The control unit (101) of any of claims 4 to 6, wherein the control unit (101) is configured to

- determine a location of a center of the object (204) within a sub-region of pixels of the 2D representation of sensor data (205) by determining a maximum pixel value of the pixels within a corresponding sub- region of the first output tensor (215); and

- determine an uncertainty value for the location of the center of the object (204) within the sub-region of pixels of the 2D representation of sensor data (205) by determining a variance of the pixel values of the pixels within the corresponding sub-region of the first output tensor (215); and/or

- determine an uncertainty value for the location of the center of the object (204) within the sub-region of pixels of the 2D representation of sensor data (205) by determining a deviation of an actual distribution of the pixel values of the pixels within the corresponding sub-region of the first output tensor (215) from a reference distribution.

8) The control unit (101) of claim 7, wherein the control unit (101) is configured to determine a deviation of the actual distribution of the pixel values of the pixels within the corresponding sub-region of the first output tensor (215) from the reference distribution for a height direction and for a width direction separately, to determine an uncertainty value for the location of the center of the object (204) within the sub-region of pixels of the 2D representation of sensor data (205) for the height direction and for the width direction separately.

9) The control unit (101) of any of claims 4 to 8, wherein

- the first output network (211, 212) comprises a first sub-network (211) configured to provide a feature matrix of feature vectors based on the 2D representation of sensor data (205);

- the first output network (211, 212) comprises a plurality of second sub networks (212) for the corresponding plurality of object classes; and

- the second sub-network (212) for an object class is configured to, for each feature vector of the feature matrix,

- transform the feature vector into a corresponding hyperspace vector (301) within a hyperspace (303), using a class-specific transformation;

- determine a distance value (305) of the hyperspace vector (301) from a class-specific centroid (304) within the hyperspace (303); and

- map the distance value (305) to a certainty and/or probability value (306) using a pre-determined certainty and/or probability distribution (307).

10) The control unit (101) of any previous claims, wherein

- the neural network (200) comprises a second output network (221) configured to provide a second output tensor (225);

- the second output tensor (225) comprises one or more dimension maps for one or more dimensions, in particular for a width and/or a height, of the object (204); and

- the control unit (101) is configured to - determine a dimension of the object (204) based on a pixel value of a center pixel (401) of the one or more dimension maps; and/or

- determine an uncertainty value for the dimension of the object (204) based on pixel values of the center pixel and one or more neighboring pixels (402) of the one or more dimension maps, in particular based on a similarity of the pixel value of the center pixel and the pixel values of the one or more neighboring pixels (402).

11) The control unit (101) of any previous claims, wherein

- the neural network (200) comprises a third output network (221) configured to provide a third output tensor (235);

- the third output tensor (235) comprises one or more offset maps for one or more offsets, in particular offsets with regards to a horizontal and/or a vertical direction, of a location of a center of the object (204); and

- the control unit (101) is configured to determine one or more characteristics of the object (204) and/or one or more corresponding uncertainty values based on the third output tensor (235).

12) The control unit (101) of any previous claims, wherein

- the neural network (200) comprises a fourth output network configured to provide a fourth output tensor;

- the fourth output tensor comprises a distance map comprising pixel values indicating a distance of a center of the object (204) from an environment sensor (102) having captured the 2D representation of sensor data;

- the control unit (101) is configured to determine a distance of the center of the object (204) from the environment sensor (102) based on the pixel values of a center pixel of the distance map; and - the control unit (101) is configured to determine an uncertainty value for the distance of the center of the object (204) from the environment sensor (102) by comparing the pixel value of the center pixel value with pixel values of the neighbors of the center pixel.

13) The control unit (101) of any previous claims, wherein the control unit (101) is configured to

- determine a combined uncertainty value based on the plurality of uncertainty values, in particular based on a weighted sum of the plurality of uncertainty values; and

- operate the vehicle function of the vehicle (100) in dependence of the combined uncertainty value.

14) The control unit (101) of claim 13, wherein

- the control unit (101) is configured to apply a plurality of weights to the corresponding plurality of uncertainty values to determine the weighted sum of the plurality of uncertainty values; and

- the plurality of weights is dependent on a sensitivity of a precision of the neural network (200) for determining the corresponding plurality of different characteristics of the object (204).

15) The control unit (101) of any previous claims, wherein the control unit (101) is configured to operate a driver assistance function and/or a function for autonomous driving of the vehicle (100) in dependence of the plurality of uncertainty values for the corresponding plurality of characteristics of the object (204).

16) A method (500) for operating a vehicle function of a vehicle (100); wherein the method (500) comprises,

- determining (501) a plurality of different characteristics of an object (204) which is located in an environment of the vehicle (100), based on a 2D representation of sensor data (205) using a trained neural network (200);

- determining (502) a plurality of uncertainty values for the corresponding plurality of characteristics based on output data (215, 225, 235) of the neural network (200); and

- operating (503) the vehicle function in dependence of the plurality of uncertainty values for the corresponding plurality of characteristics of the object (204).