DE102021126079A1 - Method and processor system for generating training data for training a machine learning model for object detection - Google Patents

Abstract

The invention relates to a method for generating training data (34) for training a model (74) of machine learning for object detection, for which a training data set (35) containing the measuring points (39) of a 3D measurement of a traffic object ( 15, 16) is determined. The invention provides that a vector field (38), which indicates a respective displacement vector for a plurality of different spatial points (50) of a predetermined spatial volume (51), is generated by means of a predetermined generation method (70) and the measuring points (39) of the training data set ( 35) are shifted according to the vector field (38) and thereby shifted measuring points (39) are generated, which represent a deformed traffic object (42), and the shifted measuring points (39) are combined to form a new training data set (40), which is part of the Training data (34) is provided.

Description

The invention relates to a method for generating training data for training a machine learning (ML) model for object detection. Such a model can be based on an artificial neural network. The object detection can relate to the automated recognition of road users in the vicinity of a motor vehicle that operates the model. Said training data are generated artificially, initially starting from a training data set that contains measurement points of a (real or simulated) 3D measurement of a traffic object to be recognized. The invention also includes a processor system set up to carry out the method.

For autonomous driving, the perception of the environment is of great importance for both safety and comfort. Object recognition is one of the pillars of environmental perception as it includes the recognition of road users such as traffic vehicles (e.g. cars, bicycles) and vulnerable road users (e.g. pedestrians). Disturbances in object recognition can have serious consequences for autonomously driving cars, as they are no longer able to correctly recognize other road users and thus orientate themselves in traffic. It is important to find the causes of these disturbances and to make object detection robust against them.

Unusual shapes in vehicles have proven to be a source of such disturbances, i.e. shapes that are not intended by the manufacturer, e.g. superstructures attached by the user. Machine learning methods (more specifically, e.g. deep learning or neural networks) can be used to design physically realistic deformations of cars that disrupt the object detectors. Since laser scanners (also known as LI DAR) are usually the preferred sensors in autonomous vehicles, the robustness of the object detection of deformed road users is particularly interesting for this type of sensor.

Perturbations to machine learning models that intentionally fool the neural network are called "adversarial examples" or "attacks." The object detectors that are preferably considered for attacks are PointPillars[1], SECOND[2] and Part-A2[3] as they are popular prior art models. There are two main directions of attacks on LIDAR object detectors in the autonomous driving (AD) context: spoofing and physically feasible attacks.

Spoofing: This class of attack uses a device to inject fake lidar points (3D measurement points) directly into the sensor. The device consists of a photodiode that receives the laser pulses from the LIDAR and activates a delay component that triggers the attacker's laser to simulate true echo pulses. This is used in the work of [4] and [5] to generate an opposing pattern in the point cloud of the LIDAR points, which is recognized as an object (and e.g. brings the autonomously driving car to a standstill).

• [1] Lang, Alex H., et al. „Pointpillars: Schnelle Kodierer für die Objekterkennung aus Punktwolken.“ Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
• [2] Yan Yan, Yuxing Mao, and Bo Li. SECOND: Sparsely Em- 920 bedded Convolutional Detection. Sensors, 18(10):3337, Oct. 921 2018.
• [3] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From Points to Parts: 3D Object Detec- tion from Point Cloud with Part-aware and Partaggregation Network. arXiv: 1907.03670 [cs], Mar. 2020.
• [4] Cao, Yulong, et al. „Adversarial sensor attack on LIDAR-based perception in autonomous driving.“ Proceedings of the 2019 ACM SIGSAC conference on computer and communications security.
• [5] Sun, Jiachen, et al. „Towards robust LIDAR-based perception in autonomous driving: General black-box adversarial sensor attack and countermeasures.“ 29. {USENIX} Security Symposium ({USENIX} Security 20). 2020.
• [6] Cao, Yulong, et al. „Adversarial objects against LIDAR-based autonomous driving systems.“ arXiv preprint arXiv:1907.05418(2019). [6] Tu, James, et al. „Physically realizable adversarial examples for LIDAR object detection.“ Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

Physically viable attacks: These types of attacks aim to be carried out in the real world. In the work by [6], boxes with an edge length of 50 cm and 75 cm are specifically created in such a way that they are not recognized by an object detector. These boxes can theoretically be 3D printed and placed in the real world. However, this type of attack requires the explicit construction of an enemy object and its placement on the vehicle's roof. Generating sufficiently varied training data on this basis requires a great deal of effort.

• [1] Lang, Alex H., et al. "Pointpillars: Fast Encoders for Object Recognition from Point Clouds." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019
• [2] Yan Yan, Yuxing Mao, and Bo Li. SECOND: Sparsely Em- 920 bedded Convolutional Detection. Sensors, 18(10):3337, Oct. 921 2018.
• [3] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part Aggregation Network. arXiv: 1907.03670 [cs], Mar 2020.
• [4] Cao, Yulong, et al. "Adversarial sensor attack on LIDAR-based perception in autonomous driving." Proceedings of the 2019 ACM SIGSAC conference on computer and communications security.
• [5] Sun, Jiachen, et al. "Towards robust LIDAR-based perception in autonomous driving: General black-box adversarial sensor attack and countermeasures." 29th {USENIX} Security Symposium ({USENIX} Security 20). 2020
• [6] Cao, Yulong, et al. "Adversarial objects against LIDAR-based autonomous driving systems." arXiv preprint arXiv:1907.05418(2019). [6] Tu, James, et al. "Physically realizable adversarial examples for LIDAR object detection." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020

From the U.S. 10,262,236 B2 It is known that artificial images can be generated in order to train an artificial neural network, in that image contents of second output images can be superimposed. Here, however, it is difficult to create realistic image impressions.

The object of the invention is to train a model of machine learning, for example an artificial neural network, so that object detection carried out with it is also robust against unusual deformations, in order to be able to reliably detect road users in road traffic using the model trained in this way.

The object is solved by the subject matter of the independent patent claims. Advantageous embodiments are described by the dependent patent claims, the following description and the figures.

As a solution, the invention includes a method for generating training data for training a machine learning model for object detection. For this purpose, a training data record is generated or determined by a processor system, each of which contains measurement points of a 3D measurement (e.g. LIDAR points) of a traffic object. The training data set thus describes a point cloud made up of measurement points. The training data set can be obtained from a real or a simulated measurement. The measurement points can represent 3D coordinates of surface points at which a reflection of an electromagnetic radiation, for example a laser light or a radar signal, was detected or measured. A single training data set represents an image or snapshot of the surroundings of a sensor that generates the measurement points. It can be a LIDAR sensor, for example, which is used to determine the training data set. The measuring points can describe a traffic object or also a number of traffic objects and/or an area surrounding the sensor, for example a surface of a roadway and/or a road boundary and/or a building. The training data record can be generated, for example, during a measurement drive using a measurement vehicle that transports the sensor for generating the measurement points for the measurement data record. The invention is described below on the basis of a single training data set of this type, but of course the method can also be applied to multiple training data sets in order to obtain a larger amount of training data.

Starting from such a training data set, a further, new training data set can now be artificially generated or calculated using the method. This new training data set makes the machine learning model more robust when it comes to object detection, because by training the new training data set, for example, a motor vehicle with a dent or a dented rear can still be correctly recognized or detected. The following is provided for this purpose according to the invention.

A vector field is generated by means of a predetermined generation method, ie a predetermined program routine, which indicates a respective displacement vector for a plurality of different spatial points of a predetermined spatial volume. The spatial volume can, for example, be a cuboid with specified dimensions. Exemplary edge lengths of the cuboid can be in a range from 1m to 12m (for trucks).

The measurement points of the specified training data set (e.g. from a real test drive) are shifted according to the vector field. As a result, shifted measurement points are generated that represent a deformed traffic object. The shifted measurement points are combined into the said new training data set, which is provided as part of the training data. If several new training data sets are generated, these together form the artificially generated training data. The spatial points each represent a base point of a displacement vector. Within the volume of space, there can be a different displacement for the measurement points, depending on where in the volume of space or cuboid the respective measurement point is arranged, so that the displacement vector that is valid or defined there is on the measurement point is applied.

Said machine learning model can be trained by means of the new training data set or generally by means of the training data resulting therefrom, which comprise at least one such new training data set. A model and training method known from the prior art can be used for this purpose. The model trained in this way can then be operated in a motor vehicle, for example, in order to use the model to carry out object detection, for which at least one sensor of the motor vehicle can provide sensor data as input data for the model. By means of the object detection, for example, a driver assistance function, or a brake assistant, and/or an autonomous driving function in the motor vehicle can be operated or supported by objects detected by the model being signaled to the driver assistance function and/or the autonomous driving function.

The advantage of the invention is that, based on a training data set, such as can be determined using a real measurement and/or simulation, another new training data set can be determined using a calculated deformation that is defined by a vector field. It is therefore not necessary to find or provide corresponding traffic objects that are actually deformed, for example deformed motor vehicles, in which, for example, an outer panel is deformed or dented in order to generate corresponding training data sets using real measurements using a sensor. Instead, a training data record of an undamaged motor vehicle can be converted into a new training data record of a deformed motor vehicle using the vector field. An original design of a road user can be simulated by a vector displacement of the measuring points, which simulates a buckling, for example. If the resulting new training data set is trained or taken into account in a training of a machine learning model, then this model is accordingly able to also correctly apply the object detection to a traffic object that has such a deformation, for example a buckling.

The training data set, whose measuring points are shifted using the vector field, can also contain so-called label data in addition to the measuring points, i.e. data which provide information or details about the traffic object or traffic objects described by the measuring points. For example, the measuring points can represent or describe an environment of a motor vehicle in which at least one other road user, for example another motor vehicle, is located, e.g is shown. The label data can correspondingly indicate which traffic object is represented and with which relative spatial alignment with respect to the sensor and/or at which distance and/or through how many measurement points it is imaged. In addition to the type of traffic object (vehicle, pedestrian, cyclist, roadway surface, edge development), a spatial orientation or orientation of the respective traffic object can also be indicated by the label data. The label data can, for example, specify a so-called bounding box for each traffic object. The label data can be generated in a manner known per se, e.g. manually or automatically.

The invention also includes embodiments that result in additional advantages.

A development includes that a direction of a connecting line of the respective measuring point to a position of a sensor that generated the measuring point is determined and the displacement of the respective measuring point only takes place along the connecting line. In other words, a shift to a measurement point is performed only along a viewing direction of the sensor that generated the measurement points of the training data set, ie toward or away from the position of the sensor. This results in the advantage that a pattern of a scan or a grid that the sensor used in the area surrounding the motor vehicle when generating the measurement points is undistorted in the new training data record or the pattern is retained. There are only differences in distance or depth per measurement point compared to the non-displaced measurement point. The measuring points can, for example, indicate coordinates with respect to a coordinate origin, which can correspond to the position of the measuring sensor. The connecting line between the measuring point and the sensor can be calculated by means of geometric calculations that are known per se. For a given measurement point, a displacement vector of the vector field can be applied by first projecting the displacement vector onto the connecting line and then applying the projected displacement vector to the measurement point, which then gives the displaced measurement point, whose displacement is only along the displacement line. The projection of a vector onto a line is known per se from the prior art and can be based, for example, on the dot product.

A further development includes that, for the displacement of the measurement points, the spatial volume is scaled to a size of the traffic object described by the measurement points. The traffic object can be identified or described in the training data record, for example, by a so-called bounding box, as can be defined, for example, in the said label data. The spatial volume can also be defined as a cuboid or box in the manner described. By matching the bounding box of the traffic object and the bounding box of the volume of space, the vector field is fitted to the traffic object. This results in the advantage that the vector field nevertheless results in a uniform or the same deformation for traffic objects of different sizes. Thus, for example, a distance between the traffic object and the sensor that generated the training data set has no influence on the resulting deformation of the measurement points.

A development includes that a spatial alignment of the spatial volume with respect to the traffic object is adapted for the displacement of the measurement points. As already stated, an orientation or orientation of the traffic object can be indicated in the training data set by the described label data. By adapting the spatial volume or the cuboid with the displacement vectors in the orientation or relative rotation to the traffic object in the training data set, displacement vectors that are provided, for example, for a deformation of a rear end of a motor vehicle or generally a predetermined area of the traffic object, can also be achieved be applied correctly in the training data set to this area of the road user, ie, for example, to the rear.

A development includes that when the measuring points are displaced, a maximum displacement path is limited to a predetermined maximum displacement value. The maximum displacement value can be in a range from 5 cm to 40 cm, for example. Limiting the displacement value has the advantage that a physically realistic deformation of the traffic object results, for example a dent in a motor vehicle that is not so large that the motor vehicle would actually have to be unroadworthy or a breakthrough would have to occur. Limiting the displacement distance can be done by limiting or truncating the magnitude of the applied displacement vector to the maximum displacement value.

A development includes that when the measuring points are shifted, a difference between the respective shifting distances of adjacent measuring points is limited to a predetermined maximum difference value. The maximum difference value can range from 0.5 cm to 10 cm. Limiting a difference in the displacement ranges of adjacent measurement points results in the advantage that a physically realistic deformation is simulated, as can be expected or is realistic in the case of a metal panel of a motor vehicle. In English, this limitation is referred to as maintaining "smoothness".

A further development comprises that the spatial points of the displacement vectors in the volume of space are arranged at a distance from one another in a grid and that the displacement vector to be used for the respective measuring point is the one with the spatially closest spatial point or a displacement vector interpolated from several neighboring displacement vectors. The grid in which the spatial points or base points of the displacement vectors are arranged can, for example, provide a distance between adjacent spatial points in a range of 1 cm to 40 cm. If a measuring point does not fall exactly on such a point in space, but lies between several points in space, the spatially closest point in space or base point can be determined and the associated displacement vector can be used. As an alternative to this, an interpolation of the displacement vectors of the next spatial points or base points surrounding the measuring point along the grid lines can be undertaken. For example, the interpolation can provide a weighted superimposition or sum of the selected displacement vectors. The weighting can be proportional to the distance of the associated spatial point or base point from the measuring point. The sum of the weighted displacement vectors can then be divided by the number of displacement vectors used, resulting in an appropriate displacement vector. Spaced points in space result in the advantage that the vector field can be described with a small amount of data. In addition, an iterative method for calculating a vector field can be performed efficiently. Such a generation method is described below.

A further development includes that the generation method for generating the vector field includes that the vector field is initialized with predetermined initial displacement vectors and then once or iteratively using an already pre-trained predecessor model of machine learning, which uses the training data set and/or at least one further training data set for an object detection is trained, then a recognition result is generated for the training data record and/or for the at least one further training data record, which indicates whether and/or which traffic object was recognized. A vector change is then determined for at least one of the displacement vectors, which causes a deterioration in the recognition result. This vector change is then applied to the respective displacement vector. An exemplary initial displacement vector is a zero vector, ie a displacement with the absolute value or a displacement distance of zero. Thus, the new training data set generated in the first iteration is identical to the training data set originally used (the measurement points remain unchanged). If a precursor model of machine learning is then applied, i.e. a model of machine learning that is trained to detect undeformed traffic objects, i.e. undamaged motor vehicles, for example, a recognition result is obtained that can indicate, for example, which traffic object was recognized and/or which spatial orientation of this has. Then in the respective iteration by changing the displacement vectors or at least one displacement vector, a new training data record is generated and the new recognition result obtained when an object is detected again is checked to determine whether a recognition performance has deteriorated. For example, the known recognition confidence and/or a classification probability for a single object detection and/or a recognition statistic (proportion of correct detections) for a large number of object detections of different traffic objects can be evaluated. Here, a worsening of the recognition result is the goal or desired in order to determine an effective vector change. In this case, the vector change of the at least one displacement vector can be limited in amount per iteration, for example, ie a limited delta or a limited change step can be provided. The vector change of the at least one displacement vector can be chosen randomly, for example, i.e. it can include a random process when selecting at least one displacement vector or several displacement vectors in the vector field and/or when determining or determining the amount and/or the direction of the vector change.

However, starting from the initial vector field, the change or vector change of the at least one displacement vector is preferably carried out systematically in order to achieve a maximum possible deterioration of the recognition result per iteration in the case of a predefined limited vector change, for example in the case of a predefined maximum change in magnitude and/or change in direction of a respective displacement vector.

A further development in this respect includes using an artificial neural network as the model for machine learning and determining the vector change using a gradient method. Such a gradient method can, for example, be determined or formed on the basis of the gradient descent method known per se. In the present case, however, one is interested in a worsening of the recognition result, so that one can also speak of a gradient ascent method. In connection with an artificial neural network, the fact can be exploited that, starting from a training data set, the application of the vector field and the subsequent input of the shifted measurement points into an input layer or input layer of the artificial neural network as well as the processing in the intermediate layers of the network and the generation of the Recognition result at the output layer can be described overall as a calculation process or can be summarized as an analytical calculation rule for which a gradient can be calculated. This is known per se in connection with back-propagation training for artificial neural networks. This is now used in the present case in order to project or transfer the resulting gradient onto the vector field in order to change the vector change described there as a function of the propagated or projected gradient. This ensures that the deterioration or reduction in recognition performance due to the application of the vector field is maximized per iteration. This makes the process particularly efficient. The intermediate layers of the artificial neural network itself can remain unchanged, i.e. the network itself does not "learn".

A further development includes that at least one further vector field is generated and each vector field is assigned to another predetermined possible measurement situation of the traffic object and to shift the measurement points an actual measurement situation of the traffic object depicted in the training data set is determined and the assigned vector field is used to shift the measurement points . The measurement situation can be described by an indication of the relative spatial alignment and/or an indication of the distance of the traffic object from the sensor and/or an indication of the number of measuring points of a traffic object determined. In other words, a separate vector field can be defined for a deformation of a traffic object from different viewing directions and/or distances and/or measurement point clouds. Thus, for example, a vector field can be defined for a deformation of a motor vehicle from a rear view, a deformation of the motor vehicle from a side view and a deformation of the motor vehicle from a front view, just to name examples. For example, a vector field can be defined for a change in the viewing angle in the horizontal direction by 20 degrees to 50 degrees. Correspondingly, training data sets can be divided into groups, each group being able to result from the viewing angle, ie for example the group of training data sets of motor vehicles in frontal view. A vector field is then available for each of these groups. In order to recognize the group or alignment contained in a training data record, the described label data can be used. The same procedure can be used for a determined distance and/or the number or quantity of measurement points. The distance can be determined, for example, using a radar sensor and/or LIDAR sensor and/or a TOF sensor (TOF—Time of Flight).

A further development includes that the training data set includes measuring points of a LIDAR sensor and/or a radar sensor and/or a stereo image of a stereo camera or measuring points of a TOF sensor (TOF—time of flight). The specified sensors result in point clouds from measurement points as the sensor result, so that the method can be applied to these sensors in an advantageous manner.

As a solution, the invention comprises a processor system with a processor circuit that is set up to carry out a method according to any one of the preceding claims. The processor circuitry can be based on one or more microprocessors. The method can be implemented, for example, as a program code or as program instructions, which can be stored in a data memory of the processor system and which can be set up to carry out the embodiment of the method according to the invention when executed by the at least one microprocessor.

Further features of the invention result from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description and the features and combinations of features mentioned below in the description of the figures and/or shown alone in the figures can be used not only in the combination specified in each case, but also in other combinations or on their own.

• 1 eine schematische Darstellung einer Ausführungsform des erfindungsgemäßen Prozessorsystems;
• 2 ein Diagramm zur Veranschaulichung einer Anwendung eines Vektorfelds auf einen Trainingsdatensatz;
• 3 eine schematische Darstellung zur Veranschaulichung eines Erzeugungsverfahrens eines Vektorfelds.

The invention will now be explained in more detail using a preferred exemplary embodiment and with reference to the drawings. Show it:

• 1 a schematic representation of an embodiment of the processor system according to the invention;
• 2 a diagram for illustrating an application of a vector field to a training data set;
• 3 a schematic representation to illustrate a generation method of a vector field.

1 shows a motor vehicle 10, which can be, for example, a passenger car or a passenger bus or a truck. The motor vehicle 10 can move in a traffic or drive in a traffic. A driver assistance function 12 and/or an autonomous driving function 13 can be active in motor vehicle 10 via a control unit 11, by means of which at least one actuator 14 can be controlled in motor vehicle 10, by means of which longitudinal guidance (accelerating and/or braking) and / or a lateral guidance (steering) in the motor vehicle 10 can be carried out automatically, that is to say without any action on the part of the driver. For example, the motor vehicle 10 can be guided along a planned driving route and/or it can react to at least one traffic obstacle in an automated manner by means of a steering intervention and/or braking. The driver assistance function 12 and/or the autonomous driving function 13 can react to at least one other traffic object 15, 16 that can be detected or recognized in an environment or an environment 17 of the motor vehicle 10. For this object detection, motor vehicle 10 can have at least one sensor 18, which can be a lidar sensor or a radar sensor, for example. Sensor data 19 of the at least one sensor 18 can be transmitted to a control unit 20, which is control unit 11 or, as in 1 shown to another control unit, may act. The respective sensor data 19 can include 3D measurement points that indicate where in the space of the environment 17 a reflection on a surface of a traffic object 15, 16 was detected or has taken place. The sensor data 19 thus contain measurement points that each indicate 3D coordinates of reflection events of a reflected laser light or radar signal of the at least one sensor 18 in the area 17 .

A machine learning model 21, for example an artificial neural network or a decision tree model or a support vector machine (SVM), can be operated in the control unit 20, by means of which the object detection or object recognition is carried out on the basis of the sensor data 19 can be. On the basis of the object detection, a recognition result 22 can be generated which indicates which traffic object 15, 16 was currently recognized in the area 17 on the basis of the sensor data 19. The recognition result 22 can, for example, include corresponding recognition data that can describe, for example, a coordinate of the respective traffic object 15, 16 relative to a coordinate system of the motor vehicle 10 and/or the at least one sensor 18 and/or an absolute world coordinate system (geo coordinates). The recognition result 22 can be provided to the driver assistance function 12 and/or the autonomous driving function 13, so that the respective traffic object 15, 16 can be taken into account there by adapting a driving trajectory 23 when driving the motor vehicle 10 for collision-free vehicle guidance.

In order to carry out the object detection using the model 21 on the basis of the sensor data 19 To be able to, the model 21 can be generated by a processor system 30 based on a learning method 31 for machine learning. The fully trained model 21 can then have been made available, for example, when the control unit 20 was manufactured and/or when the motor vehicle 10 was manufactured and/or subsequently via a communication link 32 in the motor vehicle 10 . The communication connection 32 can be transmitted, for example, on an Internet connection and/or a radio connection (e.g. WiFi/WLAN and/or mobile radio and/or wired).

In order to be able to generate the model 21 using the learning method 31, training data 34 with training data records 35 can be determined or provided in a step S10. The training data records 35 can each include sensor data with measuring points from at least one sensor such as the sensor 18, ie for example lidar sensor data and/or radar sensor data, as well as the label data already described. Label data can indicate what kind of traffic object is described by the respective sensor data in the training data set 35 . The label data can be generated in a manner known per se, for example by an operator or an operator, or automatically to form sensor data. The training data 34 can be made available to the learning method 31 in a manner known per se. An example of a machine learning method 31 is supervised learning, unsupervised learning.

In the processor system 30 it is additionally provided that additional training data 36 are provided, which can be generated from the training data 34 . At least one new training data set 40 can be generated for each training data set 35 of a measurement from the training data 34 . In other words, no further test drive and/or measurement with a sensor is necessary to generate the further training data 36 . It is therefore artificially generated or augmented training data 36 based on the training data records 35.

For this purpose, a shifting method 37 can be operated in the processor system 30 by means of at least one microprocessor, which in a step S12 changes or distorts the individual training data sets 35 by means of a vector field 38 in each case by measuring points 39 from the respective training data set 35 being spatially shifted, i.e. changing their coordinates be, this being dependent in the vector field 38 on which spatial coordinates the original measurement point 39 describes or indicates in each case. The vector field 38 can be previously selected in a step S11 from several prepared vector fields, which is still related to 3 and the generation method 70 illustrated there.

The vector field 38 can, as in 1 shown, for example applied additively to the individual measurement points 39 . This results in a step S13 in each case in a new training data record 40 with shifted measuring points 41, that is to say the respective traffic object 15 is represented in the new training data record 40 as a deformed traffic object 42. In the case of a motor vehicle as the traffic object 15, a deformed traffic object 42 in the form of a dented and/or deformed motor vehicle can be represented by the new training data record 40. In the case of a person as the traffic object 15, for example, the person's physical stature can be distorted or altered, for example stretched out in width. The new training data set 40 can also be made available to the learning method 31 as part of the further training data 36 as further training material or as further training data. Thus, the recognition of a traffic object 15, 16 is also trained in the model 21 if this traffic object is deformed in each case, ie a motor vehicle has been changed, for example by bumps, or has been changed in its original form.

A traffic object 15, 16 in the surroundings 17 can thus be detected in the motor vehicle 10 by means of the model 21 even if this deviates from its original production-related design, for example because a motor vehicle has been deformed by an accident, for example has a dent in the outer panel.

2 12 illustrates which recognition result 22 can result if a machine learning model has not yet been trained on the deformation introduced by means of the respective vector field 38 . The vector fields 38 contained in 2 are shown each show base points or spatial points 50, at each of which a vector is defined. The vectors themselves are not shown here for the sake of clarity. As shown, the spatial points 50 in a given spatial volume 51 can each define vectors in a grid. The spatial volume 51 in which the vector field 38 is defined can be defined, for example, as a cuboid or as a so-called bounding box.

2 shows on the left side a training data set 35 with measurement points 39 which, as traffic objects 15, can describe two motor vehicles, a driving surface 52 and a development 53 as an example of a measurement result in the sensor data. Also described for the training data set 35, that is, for the measurement points 39 Label data 54 indicates that the traffic objects 15 are in each case a motor vehicle and a spatial orientation 55 can also be described by the label data.

It can now be specified on the basis of the object type of the respective traffic object 15, i.e. here the object type motor vehicle, for example a car, truck, passenger bus, cyclist, pedestrian can also be specified as the object type, and the relative orientation 55 with regard to the detecting sensor, i.e. for example the sensor 18 , one of several provided vector fields 38 can be selected, which is provided for this object type and/or this orientation or orientation and/or the distance of the traffic object from the sensor and/or the number of available measuring points. Furthermore, a scaling 56 can be applied to the spatial volume in order to adapt the spatial volume 51 to the size of the respective traffic object 15 present in the training data set 35 . This has the advantage that spatial points 50 are always applied in the same area of a traffic object of the object type used, i.e. spatial points for rear lights in motor vehicles that are at different distances are nevertheless also applied to the measuring points 39 of the rear lights by adapting the spatial volume 51 .

The vectors of the vector field 38 can then be applied additively to the measuring points 39, for example in the respective bounding box of the traffic object (known, for example, from the label data 54). It can also be provided that the respective displacement vector is not applied in full at the point in space 50, but only that portion that is extended in the viewing direction towards or away from the position of the sensor, so that a measurement grid is created by applying the vector field 38 in the new training data set 40 in relation to the original training data set 35 does not change or remains unchanged. The right-hand side shows how the recognition result 22 when applying the model 21 to the new training data set 40 shows that one of the motor vehicles is recognized with an incorrect orientation 60 and a second of the motor vehicles is not recognized 61, i.e. no bounding box for the unrecognized traffic object is output in the recognition result 22.

3 illustrates a generation method 70 by means of which the described vector fields 38 can be generated in each case. It is shown how a vector field 38 changed per iteration 71 can be formed iteratively in an iteration loop of the iteration 71 or also once starting from an initial vector field 72 . This can be done in connection with 2 described, the vector field 38 in its current state can be overlaid with a training data set 35, for example by vector addition 73, and the new training data set 40 that is created as a result can be provided to a machine learning model 74 as input for object detection, resulting in a recognition result 22 in the manner described results. The recognition result 22 is also visualized on the right. Starting from the recognition result 22, a vector change 76 can be determined using a gradient method 75, which can be applied to at least some of the displacement vectors V.

In 3 is shown again on the left side in the initial vector field 72 how the spatial volume 51 can be subdivided by individual spatial points 50 or spatial points 50 can be arranged or defined in the spatial volume 51 . Each point in space 50 can be a base point for a displacement vector V, as shown in 3 for the vector field 38 are represented symbolically as a line of a length which corresponds to the vector absolute value. An orientation of the line shows the direction of the displacement vector V.

The gradient method 75 can be used to determine that vector change 76 which would result in the greatest possible deterioration or deviation for the next recognition result in the next iteration 71 . Since the vector change 76 can be limited here, for example with regard to the amount and/or the direction, the vector field 38 can be derived or defined with the lowest possible number of iterations 71, i.e. particularly efficiently, by a plurality of iterations 71 starting from the initial vector field 72 . It can be provided here that a group 77 is used as training data records 35, which show traffic objects from a predetermined viewing direction, for example, for determining a vector field 38 in the iterations 71, different training data records 35 of traffic objects are shown that are of the same object type, for example motor vehicles , are and show the respective traffic object from the same perspective, for example the rear of the vehicle or the front of the vehicle. This results in specialized or type-specific and perspective-specific vector fields 38. These can then, as in 2 shown when the vector fields 38 are provided, from which a selection can then be made depending on the label data 54 .

Bumps on motor vehicles can be observed on the road and modern object detectors can be affected by such deformation genes are successfully "attacked" or disrupted by them. An object detector can now be trained to be resistant to this "attack" even if no measured training data set on which to train this model contains dented automobiles. By explicitly generating counter-examples with dented motor vehicles, the training data for the models can be extended. The result is object detectors that are more reliable when encountering similar deformations in the real world. Without this approach, finding and collecting thousands of dented cars in the real world would not be feasible due to cost and time constraints.

In order to make an attack physically realizable and realistic, vector fields are learned which, for example, deform 3D LIDAR measurement point clouds so that they are no longer or no longer correctly recognized with a model that is trained to detect undamaged road users (e.g. B. because a wrong orientation is detected). This approach has several key advantages: The deformation of the point cloud can be made physically realistic and produce a valid LIDAR scan. These vector fields are similar to real-world use cases, such as B. dented cars after accidents or people in unusual clothing. This is a unique contribution compared to previous work and compared to the prior art, the attack can be made much more subtle. The vector fields can be used to augment the training data to make the object detection models more robust to dented vehicles in the real world.

The example thus shows physically realizable "attacks" on LIDAR-based 3D object detectors by deforming point clouds from measurement points using scene-consistent vector fields.

reference list

10

motor vehicle

11

control unit

12

driver assistance function

13

driving function

14

actuator

15

traffic object

16

traffic object

17

Vicinity

18

sensor

19

sensor data

20

control unit

21

Model

22

recognition result

23

driving trajectory

30

processor system

31

learning method

32

communication link

34

training data

35

training data

36

training data

37

move procedure

38

vector field

39

measuring point

40

training data set

41

measuring points

42

traffic object

50

space point

51

room volume

52

driving surface

53

peripheral development

54

label data

55

orientation

56

scaling

60

orientation

61

non-detection

70

production process

71

iteration

72

Initial vector field

73

vector addition

74

Model

75

gradient method

76

Vector Alteration

77

group

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US 10262236 B2 [0007]

Claims (10)

Method for generating training data (34) for training a machine learning model (74) for object detection, a training data record (35) containing the measuring points (39) of a 3D measurement of a traffic object (15, 16 ) contains, is determined, characterized in that a vector field (38), which indicates a respective displacement vector (V) for a plurality of different spatial points (50) of a predetermined spatial volume (51), is generated by means of a predetermined generation method (70) and the Measuring points (39) of the training data set (35) are shifted according to the vector field (38) and thereby shifted measuring points (39) are generated, which represent a deformed traffic object (42), and the shifted measuring points (39) to a new training data set (40) are summarized, which is provided as part of the training data (34). procedure after claim 1 , wherein a direction of a connecting line of the respective measuring point (39) to a position of a sensor (18) which has generated the measuring point (39) is determined and the respective measuring point (39) is shifted only along the connecting line. Method according to one of the preceding claims, wherein for the displacement of the measuring points (39) • a scaling (56) of the spatial volume (51) to a size of the traffic object (15, 16) described by the measuring points (39) and/or • a spatial alignment of the spatial volume (51) with respect to the traffic object (15, 16) is adapted. Method according to one of the preceding claims, wherein when moving the measuring points (39) • a maximum displacement path to a predetermined maximum displacement value and/or • a difference between the respective shifting distances of adjacent measuring points (39) is limited to a predetermined maximum difference value. Method according to one of the preceding claims, wherein the spatial points (50) of the displacement vectors in the spatial volume (51) are arranged at a distance from one another in a grid and for the respective measuring point (39) as the displacement vector to be used that displacement vector (V) of the spatially closest arranged space point (50) or from a plurality of adjacent displacement vectors (V) interpolated displacement vector is used. Method according to one of the preceding claims, wherein the generation method (70) for generating the vector field (38) comprises that the vector field (38) is initialized with predetermined initial displacement vectors (72) and is generated once or iteratively by means of an already pre-trained predecessor model (74) of the machine learning, which is trained for object detection using the training data set (35) and/or at least one further training data set (35), generates a recognition result (22) for the training data set (35) and/or for the at least one further training data set (35). which indicates whether and/or which traffic object (15, 16) was detected, and a vector change (76) for at least one of the displacement vectors of the vector field (38), which causes a deterioration in the detection result (22), is determined and applied to the respective displacement vector (V). procedure after claim 6 , wherein an artificial neural network is used as the model (74) of the machine learning and the vector change (76) is determined by means of a gradient method (75). Method according to one of the preceding claims, wherein at least one further vector field (38) is generated and each vector field (38) is assigned to another predetermined possible measurement situation of the traffic object (15, 16) and to shift the measurement points (39) an actual measurement situation of the in the traffic object (15, 16) depicted in the training data set (35) is determined and the vector field (38) assigned to this measurement situation is used to shift the measurement points (39), the measurement situation providing information on the relative spatial orientation and/or a distance of the traffic object to the sensor and/or a number of the determined measuring points of a traffic object. Method according to one of the preceding claims, wherein the training data set (35) measuring points (39) of a LIDAR sensor (18) and / or a radar sensor (18) and / or a stereo image of a stereo camera or measuring points (39) of a time -of-flight sensors (18) included. Processor system (30) with a processor circuit which is set up to carry out a method according to one of the preceding claims.

Images