DE102023004171A1 - Method for automatically assigning labels and/or objects to sensor data from a vehicle surroundings sensor system - Google Patents

Abstract

The invention relates to a method for automatically assigning a label and/or an object (O, O1, O2) to sensor data (S, S', S.R, S'.R, S.L, S'.L) of a vehicle environment sensor system. At different times (t, t', t1 to t5), sensor data (S, S', S.R, S'.R, S.L, S'.L) are recorded from at least one sensor of the vehicle surroundings sensor system. From this, the position of object candidates is determined in relation to a vehicle-independent world coordinate system (W). A graph (G) is formed from their entirety. Each object candidate is assigned an object (O, O1, O2) for a time (t, t', t1 to t5) as well as the sensor data (S, S', S.R, S') assigned to this object (O, O1, O2). .R, S.L, S'.L) in a position related to the world coordinate system (W). A node of the graph (G) is assigned to the object (O, O1, O2) and the sensor data (S, S', S.R, S'.R, S.L, S'.L). The graph (G) is partitioned into trajectories (T1, T2) in such a way that a total cost value is minimized, which is formed from the sum of observation cost values and movement cost values. A trajectory (T1, T2) is assigned an object (O, O1, O2) and/or a label.

Description

The invention relates to a computer-implemented method for automatically assigning a label and/or an object to sensor data from a vehicle surroundings sensor system.

The detection of objects in the vehicle environment using perception systems is increasingly based on machine learning methods, in particular on deep learning methods, and requires large amounts of annotated data in which objects are identified based on sensor data, for example images from vehicle surroundings cameras, radar or lidar data identified and/or classified.

Manual annotation procedures are complex and inflexible because they have to be repeated when the vehicle's surroundings sensors change. They are also subject to subjective influences.

Automatic annotation methods used to recognize and/or classify objects are referred to as autolabeling. A label that is automatically assigned to an object can designate the class (type) of the detected object, for example “person”, “pedestrian”, “cyclist”, “foreign vehicle”, “traffic control device”. A label can also or alternatively denote the identity of an object.

Simple autolabeling methods assign labels based on sensor data collected at a particular time from sensors of one modality (e.g. radar sensor data) or multiple modalities (e.g. lidar and radar sensor data). These methods only have limited accuracy and require good calibration of the vehicle surroundings sensors. They are only suitable for multimodal sensor data if the modalities are compatible and calibrated. The verification of a label by comparing sensor data from different modalities is only possible to a limited extent or not at all.

Advanced fusion-based autolabeling methods assign labels based on sensor data collected at different (consecutive) points in time. Objects are mapped to a vehicle-independent world coordinate system and their movement (according to the sequence of capture times) is tracked in this world coordinate system. Such methods are computationally complex and cannot use acausal (future) sensor data to assign a label, or can only use them to a limited extent.

The document EP 3635625 A1 describes a method, system, medium and implementation of a global model update center. At least one model for detecting objects in the surroundings of each autonomous vehicle in a fleet is created in the model update center. A plurality of annotated data items are received from the fleet of autonomous driving vehicles, each of the annotated data items being recognized based on the at least one model of sensor data that characterizes the environment of the autonomous driving vehicles. The annotated data elements are automatically generated by the autonomous vehicles during operation. Based on the received annotated data items, at least some of the models are updated and corresponding model update information is generated. This generated model update information is then distributed to the fleet of autonomous vehicles.

The invention is based on the object of specifying an improved method for automatically assigning a label and/or an object to sensor data from a vehicle surroundings sensor system, that is: an improved autolabeling method.

The object is achieved according to the invention by a method with the features of claim 1.

Advantageous embodiments of the invention are the subject of the subclaims.

In a method for automatically assigning a label and/or an object to sensor data of a vehicle surroundings sensor system, sensor data from at least one sensor of the vehicle surroundings sensor system are recorded at different times. Object candidates are determined from the sensor data and their position is projected onto a vehicle-independent, time-invariable world coordinate system. A graph with nodes and edges is formed from the entirety of the object candidates.

Each object candidate is assigned an object for a specific point in time as well as the sensor data assigned to this object in a position assigned to this point in time in relation to the world coordinate system. A graph node is assigned to the object and the sensor data.

The graph, which includes the entirety of the nodes determined in this way for all points in time, is then partitioned into trajectories in such a way that a Total cost value is minimized. The total cost value is formed from the sum of observation cost values and movement cost values, with a movement cost value being determined from the distance between the positions of an object determined at successive points in time and an observation cost value from the distance between the positions of the object Sensor data and the position of the object assigned to this sensor data at the same time is determined.

An object and/or a label is then assigned to each trajectory.

The proposed method enables the rapid and independent generation of annotations from sensor data that are recorded by at least one vehicle, but preferably by numerous vehicles in a vehicle fleet. With such automatically annotated sensor data, perception systems for object detection in the vehicle environment can be improved and expanded.

Labels obtained with the method, by being assigned to a trajectory, also enable use in subsequent components of data evaluation, for example movement forecasting and in sensor fusion.

Furthermore, the method enables a subsequent calibration of a vehicle surroundings sensor system in such a way that those error terms that are caused by the sensor data that vary depending on the modality and/or time are optimized (minimized).

In one embodiment of the method, observation cost values and/or movement cost values are determined based on the Euclidean distance between corner points of a cuboid, which describes the positions of the sensor data and/or the positions of an object at successive times in the world coordinate system. Such cost values are particularly easy to determine.

In one embodiment of the method, sensor data is recorded as radar sensor data and/or as lidar sensor data and/or as recordings from a vehicle surroundings camera. Using such sensor data, objects in the vehicle environment can be recognized particularly well and reliably.

In one embodiment of the method, the formation and partitioning of the graph is implemented in a vehicle-independent computing architecture, preferably in a cloud computing architecture or in a server computing architecture. This means that sensor data collected from various, preferably very many or all, vehicles in a vehicle fleet can be automatically annotated and particularly large training data sets can be formed for perception systems. This improves the accuracy and robustness of perception systems trained in this way. In addition, the maintainability of such an implementation of the method is improved compared to a vehicle-based implementation. Furthermore, a cloud or server computing architecture offers the possibility of load-dependent scalability of computing power.

In one embodiment of the method, the graph is partitioned using a numerical optimization method. Implementations of such methods, for example the Ceres library (https://github.com/ceres-solver/ceres-solver), are particularly easy and available in high software quality.

In one embodiment of the method, sensor data are selected for manual evaluation based on the automatically assigned objects and/or labels. For example, the selection can be made based on an error term that is determined from the observation cost values and the movement cost values along the trajectory that was assigned to an object/label. This makes it possible to limit manual evaluation to potentially critical or error-prone sensor data.

In one embodiment of the method, the automatically assigned objects and/or labels are compared with annotation from another method. This enables cross-validation of various methods, which contributes to an overall improved annotation quality.

In one embodiment of the method, scenes are selected based on the automatically assigned objects and/or labels, each of which includes a sequence of sensor data recorded at different times. Such scenes can be used, for example, to recalibrate sensors.

In one embodiment of the method, extreme situations (also referred to as “corner cases”) are identified based on the automatically assigned objects and/or labels. Using sensor data and associated objects and/or labels selected in this way, the robustness of both the proposed autolabeling method and further method steps based on it can be evaluated and improved.

Exemplary embodiments of the invention are explained in more detail below with reference to drawings.

• 1 schematisch eine Verarbeitungspipeline für das Training von Perzeptionssystemen anhand von Roh- und Metadaten eines oder mehrerer Fahrzeuge,
• 2 schematisch die Abbildung von anhand von Sensordaten zu aufeinanderfolgenden Zeitpunkten erkannten Objekten in einem Weltkoordinatensystem,
• 3A, 3B schematisch die Optimierung der Abbildung von Objektkandidaten im Weltkoordinatensystem zu aufeinanderfolgenden Zeitpunkten, sowie
• 4 einen aus Objektkandidaten gebildeten Graphen mit zwei, jeweils einem Objekt zugeordneten Trajektorien.

Show:

• 1 schematically a processing pipeline for training perception systems using raw and metadata from one or more vehicles,
• 2 schematically depicting objects recognized based on sensor data at successive times in a world coordinate system,
• 3A , 3B schematically the optimization of the mapping of object candidates in the world coordinate system at successive times, as well as
• 4 a graph formed from object candidates with two trajectories, each assigned to an object.

Corresponding parts are provided with the same reference numbers in all figures.

1 shows purely schematically a processing pipeline P for processing raw data and / or metadata that is recorded by vehicles in a vehicle fleet (not shown in detail). The processing pipeline P includes first to seventh processing steps P.1 to P.7.

In the first processing step P.1, raw data and/or metadata are recorded. Raw data can be recorded using vehicle surroundings sensors, for example with one or more cameras, radar sensors or lidar sensors. Metadata can describe the capturing vehicle or its components (e.g. sensors) in terms of type, equipment, version or configuration. In addition, the location and/or movement parameters of the vehicle as well as the time can be recorded when raw data is recorded.

In the second processing step P.2, object candidates in the vehicle environment are determined based on the raw data and/or metadata. The second processing step P.2 is preferably carried out by an on-board system of the vehicle.

In the third processing step P.3, labels are assigned to at least some of the identified object candidates in an automated process. A label denotes the class of an object candidate (for example: person, vehicle, traffic control device). Optionally, a label can also denote the identity of an object candidate. An automated method of assigning labels is also known as autolabeling and is explained in more detail below.

The preliminary object classifications captured using autolabeling are stored in a first (automatically generated) training data repository R.A and compared with manually captured or validated object classifications that are stored in a second (manually generated) training data repository R.M.

In the fourth processing step P.4, a classifier is trained using the annotated (i.e. labeled) data from the automatically generated training data repository R.A. Classifiers can, for example, be designed as neural networks and trained using deep learning methods or other machine learning methods.

In the fifth processing step P.5, a classifier trained in this way is optimized. For example, a neural network can be optimized in terms of its topology.

In the sixth processing step P.6, the performance of the trained and optimized classifier is checked, for example using test data that is independent of the training data from the training data repositories R.A, R.M.

The fourth to sixth processing steps P.4 to P.6 are preferably carried out using a computing architecture that is independent of the detecting vehicle, for example in a cloud computing architecture.

After successfully checking the performance, the trained and optimized classifier is rolled out to vehicles in a vehicle fleet in the seventh processing step P.7. These vehicles can subsequently recognize and classify objects in their vehicle environment by evaluating raw data and/or metadata by the classifier.

Within the processing pipeline P, the third processing step P.3 is particularly important because machine learning methods rely on extensive classified (labeled) training data. Manual assignment of classifications to object candidates is therefore uneconomical, time-consuming and inflexible. In particular, training data must be updated in the event of changes in the vehicle environment sensors and/or in the event of changes in the classes (labels) assigned to the objects. Furthermore, it may make sense to vary sensory-captured training data in order to improve the generalization ability of a trained classifier. This increases the effort required for manual classification, potentially several times over.

2 shows purely schematically first sensor data S, which are detected by a vehicle surroundings sensor system at a first time t, and second sensor data S', which are detected by the same vehicle surroundings sensor system at a subsequent time t'. Purely by way of example, the sensor data S, S' can each be a camera image that was recorded with a vehicle surroundings camera at time t or t'.

Based on the sensor data S, S', an object O is recognized (that is, assigned to an object candidate) and labeled (that is, provided with a label). In the in 2 In the embodiment shown, the position of the object O is determined in relation to a time-invariant world coordinate system W. Such a position can be defined, for example, as a rectangle circumscribing the object O or a circumscribing cuboid (also referred to as a bounding box), but other circumscribing geometries are also possible.

In contrast, the sensor data S, S' are recorded based on a vehicle coordinate system, not shown, which typically changes in its position and orientation relative to the world coordinate system W between the first and the second time t, t'.

The position (for example the coordinates of a bounding box) of an object 0 in the world coordinate system W is determined in a measuring step M. By reversing the transformation from the vehicle coordinate system into the world coordinate system W, the position of the recognized and classified object O relative to the vehicle coordinate system is determined in a projection step J , and therefore also determined relative to the vehicle surroundings sensors.

An advantage of the proposed method is that in order to assign a label to an object O, sensor data S, S' collected multiple times at predetermined or at least approximately known time intervals are available. Typically, the location of an object O (with respect to the world coordinate system W), which is determined using only the first sensor data S, will differ at least slightly from the location of the same object 0, which is determined using only the second sensor data S'. In addition, it is possible that an object O is not or not completely correctly recognized based on only the first or only the second sensor data S, S'.

According to the invention, the representations of an object O (that is: an assigned label and/or its position and/or extent in the world coordinate system W) are compared in an optimization step Z based on the sensor data S, S' recorded at a time t, t'.

By evaluating sensor data S, S' recorded at different times t, t' (describing the same object O), there is a more precise and reliable association between the sensor data S, S' and the representation of the object O, and thus an improved quality of the training data the automatically generated training data repository R.A possible. In addition, sensors of the vehicle surroundings sensor system can be calibrated and corrected for their inertia or latency, in particular by detecting and incorporating the movement of the vehicle between times t, t'. The proposed method can therefore be used particularly flexibly.

3A and 3B illustrate the optimization step Z. First and second sensor data S, S', each comprising first and second lidar sensor data SL, S'.L and first and second radar sensor data SR, S'.R, respectively, is at a first time t ( shown in 3A) or at a second time t' (shown in 3B) assigned an object O. The object O has errors e in its position with respect to the world coordinate system W, which are illustrated here as Euclidean distances between points or edges of the object O and corresponding structures of the sensor data S, S '.

By way of example, the radar sensor data S.R, S'.R are shown corresponding to a corner point of the object O detected as a circumscribing rectangle. The lidar sensor data S.L, S'.L are shown by way of example as corresponding to the edges of the object O. By tracking the object O over several times t, t', the errors e can be reduced.

4 shows a graph G, which includes a first and a second object O1, O2 at different times t1 to t5. The movement of the objects O1, O2 in the world coordinate system W is illustrated by movement edges GB. Connected movement edges GB form a trajectory T1, T2. The first trajectory T1 is therefore assigned to the first object O1, the second trajectory T2 to the second object 02.

Each of the objects 01, 02 is detected at each of the times t1 to t5 by several sensors of the vehicle surroundings sensor system. Examples are in 4 Lidar sensor data SL and radar sensor data SR are shown, which detect the first and second objects O1, O2 for each time t1 to t5. It is possible for an object O1, O2 to certain times t3, t4 is only detected by one sensor.

Typically, representations of an object 01, 02 determined based on different sensor data SL, SR will deviate from one another and from the actual position of the object 01, 02, as already based on 3A and 3B and explained based on the errors shown there.

In the representation of graph G, the deviations of the sensor data S.L, S.R from the actual object O1, O2 are illustrated by distances between the assigned nodes. For each of the times t1 to t5, an object O1, O2 is/are assigned a first node of radar sensor data S.R and/or a second node of lidar sensor data S.L, which is each located at a certain distance from the node corresponding to the Object O1, O2 is assigned. The distance between a node that is assigned to the sensor data S.R, S.L and a node that is assigned to the object O1, O2, which was recognized based on these sensor data S.R, S.L, is assigned a cost value, which is referred to below as the observation cost value .

In an analogous manner, the distance between a node that is assigned to an object O1, O2 at a certain time t i and a node that is assigned to the same object 01, 02 at the subsequent time t i+1 (i=1, 2, 3, 4) is assigned (in other words: a movement edge G.B), a cost value is assigned, which is hereinafter referred to as a movement cost value.

To put it simply, the observation cost values result from inaccuracies in the measurement detection and evaluation of objects 01, 02 by the vehicle surroundings sensor system, while the movement cost values result from the movement of these objects 01, 02 relative to the vehicle and/or relative to the world coordinate system W .

The task of finding the trajectories T1, T2 (that is: a sequence of nodes of the graph G, each assigned to the same object O1, O2 and connected via movement edges G.B) can therefore be formulated as a partitioning of the graph G, in which a total cost value (resulting from the movement cost values and the observation cost values) is minimized. Such problems can be solved using optimization methods known from the prior art, for example using the Ceres solver (https://github.com/ceres-solver/ceres-solver) for solving numerical optimization problems.

The proposed method therefore represents a method of offline sensor fusion, in which object candidates (which from the second processing step P.2 of the processing pipeline P according to 1 determined by the surrounding sensor system of a vehicle) are provided as input data and in which objects 01, 02 are recognized and assigned to these labels without the need for manual interaction. “Offline sensor fusion” refers to processing that is independent of the vehicle itself (which provides the sensor data SR, SL), for example in a cloud computing architecture or on a vehicle-independent local computer. The object candidates can be provided, for example, as bounding boxes (circulating rectangles or cuboids), as segmentation maps, as object skeletons or in a similar form.

Since the method takes into account a complete sequence of object candidates (detected at successive times t1 to t5) when determining the trajectories T1, T2, not only current but also past and/or future sensor data flow into the assignment of labels to objects O1, O2 S.R, S.L a. In addition, sensor data S.R, S.L from various sensors and/or sensor modalities, for example radar, lidar and camera data, can flow into the object candidates.

The object candidates are mapped into the common world coordinate system W and, based on the times t1 to t5 at which the sensor data SR, SL were recorded, are compared with each other in the manner based on 4 explained graph G associated.

The graph G is partitioned into trajectories T1, T2, each of which is assigned to an object 01, 02. The method is based on the knowledge that certain properties of objects 01, 02 (for example their size) are unchangeable over time. This ensures that a label is consistently assigned to an object 01, 02.

The movement of object 01, 02 can be modeled using a simple movement model. Using the movement model, the position of an object 01, 02 can also be determined for times that lie between times t1 to t5 at which sensor data S.R, S.L were recorded.

The position of an object 01, 02 (and thus also the position of the assigned label) can be projected using this movement model for future times and times t1 to t5 based on the position of the same object 01, 02 in past times t1 to t5. By overlaying the different (past and current) From the positions determined at times t1 to t5, trajectories T1, T2 can be determined and assigned to the objects O1, O2 (and thus the assigned labels).

The sensor data S.R, S.L (from which the object candidates are determined) and the objects 01, 02 associated along a trajectory T1, T2 have errors e, which are caused both by inaccuracies in the measurement and evaluation by the vehicle surroundings sensors, as well as by the Movement of the objects 01, 02 themselves are caused. The errors e can be described, for example, as Euclidean distances between the corner points of a three-dimensional cuboid. A total error term is determined from the totality of errors e and a partitioning of the graph G into trajectories T1, T2 is determined such that the overall error term becomes minimal. To solve this optimization problem, numerical optimization methods are known and available, which typically work iteratively (that is, in a sequence of optimization substeps).

As a result, the proposed method provides improved, in particular refined, assignments of objects 01, 02 and labels to (typically multimodal) sensor data S.R, S.L. In addition, the method provides residual error information about the errors e and/or the overall error term formed therefrom, which remains as a solution to the optimization problem after the trajectories T1, T2 have been determined. Using this residual error information, for example, the quality of the determined assignments of objects O1, O2 and labels can be evaluated.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• EP 3635625 A1 [0007]

Claims (9)

Computer-implemented method for automatically assigning a label and/or an object (O, O1, O2) to sensor data (S, S', SR, S'.R, SL, S'.L) of a vehicle environment sensor system, characterized in that - at different times (t, t', t1 to t5) sensor data (S, S', SR, S'.R, SL, S'.L) are recorded by at least one sensor of the vehicle environment sensor system, - from this, object candidates in their position is determined in relation to a vehicle-independent world coordinate system (W) and a graph (G) is formed from the entirety of the object candidates, whereby each object candidate is assigned one object (O, O1.) for a time (t, t', t1 to t5). , O2) as well as the sensor data (S, S', SR, S'.R, SL, S'.L) assigned to this object (O, O1, O2) are assigned in a position relative to the world coordinate system (W), o the object (O, O1, O2) and the sensor data (S, S', SR, S'.R, SL, S'.L) are each assigned a node of the graph (G), - the graph (G) in this way is partitioned into trajectories (T1, T2) so that a total cost value is minimized, whereby - the total cost value is formed from the sum of observation cost values and movement cost values, where o is a movement cost value in each case from the distance to positions of an object (O, O1, O2) determined at successive times (t, t', t1 to t5) and o an observation cost value is determined from the distance between the position of the sensor data (S, S', SR, S'.R , SL, S'.L) and the position of the object assigned to these sensor data (S, S', SR, S'.R, SL, S'.L) at the same time (t, t', t1 to t5) ( O, O1, O2) is determined and - an object (0, 01, 02) and/or a label is assigned to a trajectory (T1, T2). Computer-implemented method Claim 1 , characterized in that observation cost values and/or movement cost values are determined based on the Euclidean distance between corner points of a cuboid, which contains the positions of the sensor data (S, S', SR, S'.R, SL, S'.L) and/or describes the positions of an object (O, O1, O2) at successive times (t, t', t1 to t5) in the world coordinate system (W). Computer-implemented method according to one of the preceding claims, characterized in that sensor data (S, S', SR, S'.R, SL, S'.L) as radar sensor data (SR, S'.R) and/or as lidar -Sensor data (SL, S'.L) and/or recorded as recordings by a vehicle surroundings camera. Computer-implemented method according to one of the preceding claims, characterized in that the formation and partitioning of the graph (G) is implemented in a vehicle-independent computing architecture, preferably in a cloud computing architecture or in a server computing architecture. Computer-implemented method according to one of the preceding claims, characterized in that the graph (G) is partitioned into trajectories (T1, T2) using a numerical optimization method. Computer-implemented method according to one of the preceding claims, characterized in that based on the automatically assigned objects (O, 01, 02) and/or labels, sensor data (S, S', SR, S'.R, SL, S'.L) for a manual annotation can be selected. Computer-implemented method according to one of the preceding claims, characterized in that the automatically assigned objects (O, O1, O2) and/or labels are compared with annotations of another method. Computer-implemented method according to one of the preceding claims, characterized in that based on the automatically assigned objects (O, 01, 02) and/or labels from sensor data (S, S', SR, S'.R, SL, S'.L) formed scenes can be selected. Computer-implemented method according to one of the preceding claims, characterized in that extreme situations are identified based on the automatically assigned objects (O, O1, O2) and/or labels.

Images