EP4107654A1

EP4107654A1 - Method and device for generating combined scenarios

Info

Publication number: EP4107654A1
Application number: EP21703723.3A
Authority: EP
Inventors: Özgür Nurettin Püskül; Jörn BOYSEN; Jan WEIDAUER
Original assignee: Ibeo Automotive Systems GmbH
Current assignee: Microvision Inc
Priority date: 2020-02-20
Filing date: 2021-02-11
Publication date: 2022-12-28
Also published as: IL295643A; CN115516527A; KR20220139984A; US20220405536A1; WO2021165129A1; CA3169989A1

Abstract

The invention relates to a method (100) for generating (101) combined scenarios for testing an object detection unit (17). The method (100) has the step of providing (103) first sensor data (11) of a first scenario and second sensor data (12) of a second scenario, said first sensor data (11) and second sensor data (12) each being a respective point cloud which comprises a plurality of points. The method (100) has a step of classifying (107) each point (11a) of the first sensor data (11) and each point (12a) of the second sensor data (12) as relevant or not relevant, and the method also has a step of combining (114) the first sensor data (11) and the second sensor data (12) in order to obtain third sensor data (28) of a combined scenario, wherein only relevant points (26) of the first sensor data (11) and relevant points (26) of the second sensor data (12) are combined in order to form the third sensor data (28) of the combined scenario.

Description

Method and device for creating composite scenarios

Technical area

The invention relates to a method and a device for creating composite scenarios for testing an object recognition unit.

State of the art

In the automotive industry it is known to test as many scenarios as possible in the real environment in order to use an object recognition unit that is part of a driver assistance system or a device for driverless navigation of a vehicle. It is also known from the prior art to artificially bring about such scenarios by means of an environment simulation system in order to enable reproducibility and repeatability.

DE 20 2018 105 162 U1 describes, for example, such an environment simulation system for a test bench for testing technical systems or machines.

However, the prior art does not go into how the recorded data can lead to a full evaluation of an object recognition unit.

Presentation of the invention: task, solution, advantages

The present invention is based on the object of providing a method for creating composite scenarios for testing an object recognition unit which, compared to the prior art, makes it possible to combine more complicated scenarios from various scenarios in order to be able to fully test an object recognition unit . Furthermore, it should be ensured that both the individual scenarios and the composite scenario can be evaluated. The above-mentioned object is achieved by the method according to the invention for creating composite scenarios for testing an object recognition unit. The object recognition unit is primarily part of a driver assistance system and / or a device for driverless navigation of a vehicle.

The method comprises the provision of first sensor data of a first scenario and of second sensor data of a second scenario, the first sensor data and the second sensor data each comprising at least one point cloud comprising a plurality of points. In other words, at least one first point cloud from a first scenario and at least one second point cloud from a second scenario are provided.

The method comprises the merging of the first sensor data and the second sensor data to obtain third sensor data of a composite scenario, the method comprising a classification of the respective points of the first sensor data and the respective points of the second sensor data into relevant and not relevant, with only relevant ones Points of the first sensor data and relevant points of the second sensor data are combined to form sensor data of the composite scenario. In other words, each point of the first sensor data and each point of the second sensor data is classified as either relevant or not relevant. The term “composed” is to be understood in particular in such a way that the scenario is formed from individual scenarios, in other words it is brought together.

The term “scenario” is in particular a scene that was recorded by a sensor. This scene primarily comprises at least one object. The scene can be dynamic or static. In particular, the scenario comprises at least one point in time of a movement sequence of a dynamic object Furthermore, the sensor data can represent a time span of a movement sequence. Then the sensor data consist of temporally adjacent points in time, in other words snapshots, of the movement sequence. Each snapshot would then be provided with a time stamp.

The first scenario and the second scenario are in particular “simple scenarios”. In particular, the respective sensor data comprise only a few Objects, for example only one object. A composite scenario would then comprise several objects. The composite scenario is, in particular, a complex scenario, where it is more difficult to represent it in a real environment than in the case of a “simple scenario”.

In particular, the method comprises the acquisition of the sensor data of the first scenario and the acquisition of the sensor data of the second scenario. The sensor data are recorded in particular by means of a unit for recording sensor data. The first sensor data and the second sensor data are in particular recorded in a test in which at least one object, for example a robot, is preferably moving.

Furthermore, sensor data could also originate from a real environment and have been "retracted" by means of a unit for capturing sensor data. In other words, this means that a vehicle was equipped with a unit for capturing sensor data and a corresponding scenario was “captured”, in other words, has. Reference data for the determination of a ground truth are preferably also recorded by a reference sensor appropriately arranged on the vehicle. The ground truth and the sensor data can be checked by another camera. Objects in the sensor data can be labeled using the ground truth. . Furthermore, points of a point cloud can be assigned to an object. Correspondingly, sensor data can also be “extracted” from the recorded data. For example, only the points of the point cloud that were assigned to an object can be extracted and then sensor data can be displayed with regard to this object only.

The ground truth includes in particular the following parameters: the number of objects in the scenario and / or the object class, the remission of the object, the relative speed, the direction of movement, the distance to a unit for capturing the sensor data, the environmental conditions, for example Rain or fog and / or the position, preferably in a detection area of the unit for detecting the scenarios (for example, top right). This information or parameters are determined in particular from the reference data and preferably checked manually. The first scenario and the second scenario preferably differ, in particular in at least one of the following parameters: the number of objects that can be seen in the scenario, the object class, the remission of the object, the relative speed, the direction of movement , the distance to a unit for capturing the sensor data, the environmental conditions, for example rain or fog, and / or the position, preferably in a capturing area of the unit for capturing the scenarios (for example top right). The first sensor data and the second sensor data are preferably recorded at different locations, for example within a test stand, and / or at different times. The first sensor data of the first scenario and the second sensor data of the second scenario are not different views of the same scene. On the contrary, the first sensor data and the second sensor data are preferably based on different scenes. This means that the composite scenario did not take place in reality for recording the first sensor data and the second sensor data, but is composed of the various scenes.

As a result of the merging of the first sensor data and the second sensor data, the third sensor data of a composite scenario are obtained, which likewise represent at least one point cloud. In other words, the third sensor data are the common sensor data of the first sensor data and the second sensor data, the classification serving to combine the sensor data in such a way that a meaningful result is achieved.

The advantage of the method is that sensor data of simple, easily reproducible scenarios, i.e. in other words sensor data of complex scenarios, can be combined with the aid of sensor data, which in this complexity are not represented in a test stand, or at least only with great effort would let. Even in reality, these complex scenarios cannot be reproduced with sufficient repeatability. A higher coverage of scenarios is thus realized without having to "run in" these scenarios on public roads, which could possibly lead to risky situations or having to be laboriously reproduced in test stands.

As part of the classification, the first sensor data and the second sensor data are shown in particular in a common voxel map. This is especially Achieved in particular with the help of a voxelizer, which enters the points of the point cloud of the first sensor data and the points of the point cloud of the second sensor data in a common voxel map. In other words, by means of the voxelizer, the points of the point clouds are entered into a three-dimensional space that is segmented by the definition of the voxels. The point clouds are not only defined selectively in space, but can each be assigned to a corresponding voxel. This serves to establish a spatial relationship between the points of the first sensor data and the points of the second sensor data.

Free voxels can be identified as free spaces. In particular, a received measurement pulse can be assigned to each point of a point cloud. A voxel can be assigned to the point and thus also to the measurement pulse, which, among other things, depicts the position at which the measurement pulse was reflected. This received measuring pulse has an energy so that an energy can be assigned to each point of the point cloud. Furthermore, an energy of the points located therein can be assigned to each voxel. In particular, the energy of the points that are assigned to a common voxel is added. If the energy is above a previously defined threshold value, the voxel is identified as occupied and therefore not free. If the energy is below the threshold value, the corresponding voxel is identified as free.

A free voxel can thus be a voxel without points with an energy above the threshold value. With the help of the common representation within a voxel map, it is possible to identify free spaces, which is not possible with a normal point cloud representation. Furthermore, free voxels can be identified which lie behind an occupied voxel in the radial direction, since measurement pulses resulting from these voxels could not be detected. Points can therefore already be classified as irrelevant at this point, each of which lies in a voxel that is arranged radially behind an occupied voxel.

In addition, the respective sensor data can be displayed in individual voxel maps, which can then be combined to form a common voxel map. Free spaces can already be identified with this individual representation. These free spaces can then be validated in the representation in a common voxel map. Furthermore, the method can include a respective identification of objects on the basis of the first sensor data and the second sensor data, the position of the detected objects of the first sensor data and the second sensor data being compared in order to detect possible overlaps and / or obscurations of the detected objects. The term "concealment" means in particular an area concealed by an object. In other words, the objects recognized are brought into a spatial relationship to one another projects overlap and / or cover one another.

Objects are preferably identified as at least a part of the sensor data that has a coherent spatial shape or geometry. Corresponding parts of the sensor data with a coherent geometry can thus be recognized for identification. Alternatively, the ground truth can also be used. The ground truth can include information about the number of objects, respective object classes and / or positions. This knowledge from the ground truth can be used in object recognition, for example by making a comparison between the shape and / or the position of a recognized object and the corresponding information from the ground truth.

One possibility of comparing the positions of the recognized objects would be to assign voxels of the common voxel map to the recognized objects. Voxels that are assigned to an object with regard to both sensor data are thus based on an overlap. Voxels which are assigned to an object of a point cloud, but which are behind a voxel of an object of the other point cloud, are covered by the object of this other point cloud.

The method may include identifying noise points. Noise points do not come from objects, but result from noise. Noise points are identified in particular by comparing neighboring points of a point cloud and / or corresponding neighboring voxels. In particular, a basic level of the received energy can be determined by comparing neighborhoods, with points whose energy does not differ significantly from this energy level being identified as noise points. In detail, every measurement pulse received has an energy, so that an energy is assigned to every point of the point cloud. can be net. Furthermore, an energy of the points located therein can be assigned to each voxel. In particular, the energy of the points that are assigned to a common voxel is added. In addition, the energy of the immediate neighbors of a voxel can be averaged (in order to determine the "basic level") and compared with the energy of the voxel Points located in the voxel can also be identified as a noise point. Furthermore, a threshold value can also be established which takes into account more than just the directly adjacent voxels.

Furthermore, the classification can take place on the basis of identified objects, identified free spaces and / or identified noise points. In particular, relevant points and non-relevant points are marked with the help of a point cloud marker within the scope of the classification.

The points that are not relevant include, for example, identified noise points. In particular, points are selected and classified as relevant, on the basis of which a combination of different scenarios makes sense in order to obtain third sensor data of a realistic composite scenario.

In the context of the classification, hidden points of a point cloud can be classified as irrelevant, the points within the voxel map being covered by at least one point of the same and / or the other point cloud.

For example, the points that are covered by other points of the same or another point cloud can already be classified as irrelevant on the basis of the free space detection. The radially first occupied voxel in each case defines that all points that lie in voxels located radially behind are classified as irrelevant.

Furthermore, within the scope of the classification, points of a point cloud can be classified as not relevant if the points are covered by an object in the other point cloud. For example, an object that originates from one point cloud could lie radially in front of points on the other point cloud. These hidden points can for example be assigned to an object of these other points. However, these points are covered by the object of the other points in such a way that taking them into account does not make sense, since the third sensor data of this composite scenario, if recorded in this way, would not contain the points due to the cover.

Furthermore, within the scope of the classification, points of a point cloud can be classified as not relevant if the points lie within an object of the other point cloud. An analogous logic applies here. Corresponding points could not have been detected in the composite scenario due to the presence of the object of the other point cloud, so that merging these points into a common point cloud does not make sense.

Within the scope of the classification, points of a point cloud can be classified as irrelevant if the points originate from a first object that overlaps with a second object of the other point cloud. In the event of an overlap, an object that was identified using the first sensor data overlaps with an object that was identified using the second sensor data. Due to the overlap, it cannot make sense to merge all the corresponding points of the overlapping objects. It can thus be decided to only adopt the points of one of these objects in the third sensor data of the composite scenario. Then it makes sense to classify all points of the other object that intersect with the second object as irrelevant.

Within the scope of the classification, points in a point cloud can also be classified as not relevant if the points occur in both sensor data. For example, an object can have been detected in exactly the same place in both sensor data. Then it makes sense to classify the points that occur twice in only one of the point clouds as not relevant.

Hidden points can therefore be understood as irrelevant points. These can be covered within the voxel map by at least one point of the same and / or the other point cloud, by other points of the same or another point cloud and / or by an object of the other point cloud. Even they can be covered because they lie within an object of the other point cloud and / or come from a first object that overlaps with a second object of the other point cloud.

Furthermore, points that are not relevant can be understood as redundant points, since they occur in the first sensor data and the second sensor data, and / or as noise points.

The merging of the first sensor data and the second sensor data includes in particular an entry of the points classified as relevant in the first sensor data and the second sensor data in a common coordinate system. Furthermore, the merging can include saving in a common data record. A merged point cloud is thus achieved as a result, which represents a composite scenario of the two scenarios. A composite scenario is thus created.

The method can also include the above steps for further scenarios, the merging including the first sensor data, the second sensor data and further sensor data.

If the first sensor data and the second sensor data each map a time span that is used to map the motion sequence of an object, the above-mentioned steps are carried out for different points in time, in other words recordings, of the respective motion sequence of the sensor data. In such a case, the first sensor data and the second sensor data comprise a plurality of point clouds at different points in time of the time span shown in each case. These form the corresponding snapshots. A point cloud of the first sensor data is thus always merged with a point cloud of the second sensor data, from which a movement sequence of the objects contained in the composite th scenario is reproduced. The third sensor data of the composite scenario thus also map a period of time and can thus include several point clouds obtained.

In principle, in the case of sensor data with a different time span, these can be put together differently, depending on when the merging takes place begins. In other words, the sensor data can be shifted relative to one another on a time scale. The time relation between the snapshots of the same sensor data is always maintained. In other words, the sensor data of a period of time are neither “stretched” nor “compressed” in time, but are combined with other sensor data in the time sequence and the time interval in which they were recorded. The merging is easily possible since the first sensor data and the second sensor data are composed of several point clouds at different points in time of the respective time span shown, so that a different time combination simply means that other point clouds are merged in terms of time.

The time spans that represent the first scenario and the second scenario do not have to be of the same length. If, for example, the first sensor data and the second sensor data have different time periods, the longer time period can simply be taken. When merging the corresponding snapshots of the sensor data, from the end of the shorter period of time only the "last snapshot" of the sensor data of the shorter time periods can be taken Time span is mapped, no longer moved. Or the snapshots of the sensor data of the longer time span are simply taken, since there are no more snapshots of the sensor data of the shorter time span, so that there is actually no longer any real merging. The same can apply to the initial point in time of the composite scenario. In other words, one scenario could start earlier and another "added" later.

For example, the first sensor data have a length of 10 seconds and the second sensor data a length of 4 seconds. You could now stagger the time spans to one another as you like. For example, the first sensor data could be "started" immediately and the second sensor data could not begin until the third second. This would mean that only the corresponding snapshots of the first three seconds of the first sensor data are available before the third second no merging takes place. Between the third and the seventh second, snapshots of both sensor data are available in terms of time, so that they are merged. Recordings from the fourth to seventh seconds of the first sensor data combined with the corresponding snapshots from the first to fourth seconds of the second sensor data. After the seventh second there are two options. You could let the first sensor data continue to run, ie take the snapshots of the last three seconds of the first sensor data, without merging, since there are no longer any corresponding snapshots of the second sensor data, or you can take the "last snapshot" of the fourth second of the second sensor data and perform this for the rest of the period of time with the corresponding snapshots of the first sensor data. Overall, the third sensor data consists of the snapshots of the first sensor data in the first three seconds, and the merging of the snapshots of both between the third and seventh seconds Sensor data, and after the seventh second either from the merging of the "last snapshot" of the second sensor data and the corresponding snapshots of the first sensor data or only from the snapshots of the first sensor data.

Furthermore, first sensor data, which depict a movement sequence, and second sensor data, which depict a static object, and thus, strictly speaking, "statistical" sensor data, can be merged by combining different times of the first sensor data with the "static" second sensor data will.

In particular, the method can include providing a first ground truth for the first scenario and a second ground truth for the second scenario. In particular, the first ground truth and the second ground truth can be determined by means of reference data, it being possible for the method to include the acquisition of the first scenario and the second scenario by at least one reference sensor for acquiring these reference data. The reference data is in particular a point cloud, so that the reference sensor is a lidar sensor. The reference data can also be image data or radar data with a camera or a radar sensor as the reference sensor.

The first and the second ground truth serve in particular to validate the first scenario and the second scenario. The ground truth shows what should be seen in the respective sensor data. In particular, the method comprises a Validation of the first sensor data of the first scenario and the second sensor data of the second scenario.

In particular, the method comprises a merging of the first ground truth and the second ground truth to form a ground truth of the composite scenario. The summary includes, in particular, a summary of the above-mentioned parameters determined. The ground truth of the composite scenario is also used to validate the third sensor data of the composite scenario. A very simple example is as follows: The ground truth of the first sensor data can be “car, coming from the left” and the ground truth of the second sensor data “cyclist, coming from the right”, so the ground truth of the third sensor data would be “both car , coming from the left, as well as cyclists coming from the right ".

Furthermore, the method can include the testing of an object recognition unit, where the testing includes the provision of the third sensor data of the composite scenario and a comparison of the objects recognized by the object recognition unit with the ground truth of the composite scenario. In detail, the object recognition unit is designed to determine at least one, in particular all of the above-mentioned parameters, preferably at least a number of the recognized objects and a respective object class. This output of the object recognition unit can be compared with the ground truth. It can be compared, for example, whether the object recognition unit has recognized the same number of objects and / or in each case the correct object class. The parameters determined by means of the object recognition can be compared with the corresponding parameters of the ground truth.

The term “testing” can in particular be understood to mean training the object recognition unit. In particular, the object recognition unit can include a recognition module that includes a neural network with weights As described above, an output of the object recognition unit is compared with equivalent parameters of the ground truth. Furthermore, “testing” can be understood to mean a check of already adjusted weights. In this case, the weights are no longer changed, but instead a learning progress of the network is examined on the basis of the already adjusted weights from the previous training phase.

The method can also include the creation of a library for creating sensor data of composite scenarios, in which the sensor data and optionally the corresponding ground truths of various scenarios are stored. "Filing" is to be understood in particular as storing. The stored sensor data were in particular recorded beforehand. The composite scenarios are created in particular as described above. The stored scenarios are classified or categorized by specifying the following parameters, for example the object class, which can be seen in the scenario, the remission of the object, the relative speed, the direction of movement, the distance to a unit for capturing the sensor data, the environmental conditions, e.g. rain or fog, and / or the position, preferably in a detection area the unit for capturing the scenarios (e.g. top right).

The method can include the use of the library in order to create sensor data for defined composite scenarios. Furthermore, the library can be used to randomly merge sensor data in order to create sensor data for composite scenarios. This can be done automatically or manually. For example, all sensible permutations can be used to create compositions of the sensor data of all previously stored scenarios. For example, all sensible combinations of sensor data from two and / or three of the stored scenarios can be created. This creates a large number of different composite scenarios. In particular, it would not make sense to combine a scenario with itself, whereby the corresponding combinations would have to be ruled out. The sensor data of composite scenarios, which are merged with the help of the library, can be used to automatically test an object recognition unit. In particular, the corresponding ground truths of the combined sensor data are also combined and used for testing. It is advantageous that, compared to the prior art, not all possible scenarios have to be "run in". The term "run in" describes that a vehicle is equipped with a corresponding unit for detection and a reference sensor, with very long travel times being necessary in this case to be able to really encounter and capture all eventualities in a real condition. Using the library means that at least a large part of this effort can be omitted, as many scenarios from stored scenarios can be combined.

In a further aspect, the invention relates to a device for creating composite scenarios for testing an object recognition unit, the device comprising a unit for merging the first sensor data and the second sensor data to obtain third sensor data of a composite scenario. Furthermore, the device comprises a classification unit for classifying the respective points of the first sensor data and the respective points of the second sensor data into relevant and not relevant. The device is preferably designed to carry out the method described above.

The device can furthermore comprise a unit for acquiring the first and second sensor data, in which case it is preferably a lidar sensor. The point clouds of the first sensor data and the second sensor data are accordingly lidar point clouds. The unit is designed in particular to send measurement pulses by means of a transmission unit comprising a plurality of transmission elements and to receive reflected measurement pulses by means of a reception unit comprising a plurality of reception elements. Each transmission unit can be assigned to a receiving element and a three-dimensional section of the measurement area, i.e. a voxel. The point that results from the reception of a measurement pulse can thus be clearly assigned to a voxel.

The classification unit can in particular comprise a voxelizer for the first and the second sensor data, which represents the corresponding sensor data in a voxel map. The device can also comprise a free space detection unit which detects free space in the common voxel map.

Furthermore, the classification unit preferably comprises an object recognition unit which recognizes objects on the basis of the respective sensor data. Furthermore, the pre direction include an object merging unit, in other words a unit for merging the recognized objects, so that overlaps and / or obscurations of the objects can be recognized.

In particular, the classification unit further comprises a noise point identification unit which is designed to identify noise points directly.

Furthermore, the device can comprise a point cloud marker which, on the basis of the identified objects, the identified free space and the identified noise points, marks points of the various sensor data as relevant and not relevant. Furthermore, the device can comprise a unit for merging the first and the second sensor data, which can store the points of the first sensor data and the second sensor data classified as relevant in a common coordinate system.

The invention further comprises a computer program product which comprises a computer-readable storage medium on which a program is stored that enables a computer, after it has been loaded into the memory of the computer, a method as described above for classifying objects and / or for To carry out distance measurement, if necessary in conjunction with a device described above.

In addition, the invention relates to a computer-readable storage medium on which a program is stored that enables a computer, after it has been loaded into the memory of the computer, a method as described above for classifying objects and / or for distance measurement, possibly in combination with a device described above.

Brief description of the drawings

They show schematically:

FIG. 1 shows a process scheme of a process according to the invention; FIG. 2 first sensor data; FIG. 3 second sensor data;

FIG. 4 shows a common voxel map with the first sensor data from FIG. 2 and the second sensor data from FIG. 3;

FIG. 5 shows the common voxel map of FIG. 4 after identification of noise points;

FIG. 6 shows the common voxel map of FIG. 5 according to the classification of hidden points as not relevant;

FIG. 7 shows a voxel map with the third sensor data of the composite scenario;

FIG. 8 shows a device according to the invention; and

FIG. 9 shows a schematic sequence of a method according to the invention incorporated into the device according to FIG. 8.

Preferred embodiments of the invention

FIG. 1 shows a process diagram of a process 100 according to the invention, which comprises the following steps:

The method 100 is used to create 101 composite scenarios in order to test 117 an object recognition unit 17. In particular, the method 100 can comprise the creation 101 of composite scenarios.

The method 100 comprises the provision 103 of first sensor data 11 of a first scenario and of second sensor data 12 of a second scenario. The method 100 may include the prior acquisition 102 of the first sensor data 11 and the second sensor data 12.

Furthermore, the method 100 can include providing 105 a first ground truth for the first scenario and a second ground truth for the second scenario, wherein the method 100 can furthermore preferably include the acquisition 104 of the two ground truths. The first sensor data 11 of the first scenario and the second sensor data 12 of the second scenario can be validated 106 on the basis of the provided ground truths.

The method 100 comprises a classification 107 of the respective points 11a of the first sensor data 11 and of the respective points 12a of the second sensor data 12 as relevant or not relevant. For the purpose of classification, the method 100 can include the representation 108 of the first sensor data 11 and the second sensor data 12 in a common voxel map 23, with free voxels being able to be identified 109 as free spaces.

The method 100 can further include the identification 110 of objects on the basis of the first sensor data 11 and the second sensor data 12. The positions of the detected objects can be compared 111 and thus possible overlaps and / or obscurations of the objects can be detected 112. The method 100 can furthermore include the identification 113 of noise points 24.

The method 100 further comprises the merging 114 of the first sensor data 11 and the second sensor data 12 in order to obtain third sensor data 28 of the composite scenario. In this way a composite scenario is created 101.

The merging 114 can include the entry 115 of the points classified as relevant in a common coordinate system.

Furthermore, the method 100 can comprise the merging 116 of the first ground truth and the second ground truth to form a ground truth of the composite scenario.

The method can also include the testing 117 of an object recognition unit 17, the testing 117 including the provision 118 of the third sensor data 28 of the composite scenario and a comparison 119 of the objects recognized by the object recognition unit 17 with the ground truth of the composite scenario . FIG. 2 shows first sensor data 11. The first sensor data represent a point cloud, the individual points 11a of the first sensor data being clearly visible.

FIG. 3 shows second sensor data 12, which also consists of individual points 12a as a point cloud.

A common voxel map 23 is shown in FIG. 4, in which the first sensor data 11 from FIG. 2 and the second sensor data 12 from FIG. 3 are shown. The common voxel map 23 with voxels 23a comprises 16 columns and 16 rows. A voxel 23a is clearly defined by specifying the row / column.

The first sensor data 11 and the second sensor data 12, as shown in FIGS. 2 and 3, have no spatial relationship to one another. Only by assigning the points to voxels and then bringing them together in a common voxel map 23 is a spatial relationship established.

For reasons of simplified representation, the first sensor data 11, the second sensor data 12 and the common voxel map 23 as well as the third sensor data 28 generated therefrom (see FIG. 7) are shown in two dimensions, although these are of course typically three-dimensional.

The points in the common voxel map 23 in the voxels 23a at the positions 5/8, 6/4, 10/7, 10/8, 10/9, 11/6, 11/10, 12/11 are points 11a of the first point cloud, while the points 12a of the second sensor data 12 are entered in the voxels 23a at the positions 6/11, 6/12, 6/13, 7/10, 8/9, 9/2, 10/15.

The voxel map of FIG. 4 is shown in FIG. 5 after a few points have already been identified as noise points 24. This applies to points 9/2, 6/4, 5/8 and 10/15. These noise points 24 are classified as irrelevant points 27.

FIG. 5 also shows how points 11a of the first sensor data and points 12a of the second sensor data generate covered areas 29. Namely, all voxels 23a are hidden behind the points 11a at the positions 11/6, 10/7 and 10/8. With the The term “behind” is intended to be arranged radially behind it, which in the voxel map 23 means that the corresponding voxel 23a have a higher line number.

Similarly, the points 12a of the second sensor data 12 in the voxels 23a at the positions 8/9, 7/10, 6/11, 6/12 and 6/13 generate a corresponding, covered area 29. In the covered area 29 there are three Points 11a of the first sensor data

11 and are classified as hidden points 25. These are points 11a in voxels 23a at positions 10/9, 11/10, 12/11.

FIG. 6 shows the voxel map of FIG. 5 after the hidden points 25 have been classified as not relevant. Furthermore, the remaining points 11a of the first sensor data 11 and the remaining points 12a of the second sensor data 12 were classified as relevant points 26.

FIG. 7 shows a voxel map with the third sensor data 28, which results from merging the points 26 of the first sensor data 11 classified as relevant and the points 26 of the second sensor data 12 classified as relevant. The third sensor data 28 likewise represent a point cloud with corresponding points 28a, specifically with the points 28a in the voxels 23a at the positions 6/11, 6/12, 6/13, 7/10, 8/9, 10/7 , 10/8 and 11/6.

FIG. 8 shows a device 10 according to the invention, which can include a unit 13 for acquiring the first sensor data 11 and the second sensor data 12. The device 10 further comprises a classification unit 15 and a unit 14 for merging the first sensor data 11 and the second sensor data 12.

The classification unit 15 can comprise a voxelizer 16, a free space identification unit 19, an object recognition unit 17, an object merging unit 18 and a noise point identification unit 20 as well as a point cloud marker 21. The voxelizer 16 is used to collect the first sensor data 11 and the second sensor data

12 in a common voxel map 23, the free space identification unit 19 classifying free voxels as free spaces. The object recognition unit 17 serves to recognize objects and the object merging unit 18 to place them in spatial relation to one another in order to recognize possible occlusions and / or overlaps. The noise point identification unit 20 serves to identify noise points 24 to identify. The point cloud marker 21 is used to classify all points of the first sensor data 11 and the second sensor data 12 as relevant or not relevant. Only relevant points 26 are then merged by the unit 14 for merging into third sensor data 28.

FIG. 9 shows a schematic sequence of the method 100 incorporated into the device 10 according to FIG. The device 10 does not have a plurality of illustrated units, such as, for example, voxelizers or object recognition units, but rather it is only shown schematically in FIG. 9 which paths are followed by the sensor data within a device 10.

First sensor data 11 and second sensor data 12 are provided as input, which are fed to the classification unit 15, namely first to the voxelizer 16 and then to the free space identification unit 19, in order to display the first sensor data 11 and the second sensor data 12 in a common voxel map 23 and identify free voxels as free spaces.

Furthermore, the first sensor data 11 and the second sensor data 12 are fed to the object recognition unit 17 and then to the object merging unit 18 in order to compare the positions of the recognized objects with one another and to identify possible overlaps and / or occlusions.

In addition, the first sensor data and the second sensor data are fed to the noise point identification unit 20, which can identify noise points 24.

The point cloud marker 21 can then mark the noise points and identified hidden points or points classified otherwise as not relevant as not relevant, while the remaining points are marked as relevant. The points 26 classified as relevant are fed to the unit 14 for merging the first sensor data 11 and the second sensor data 12, which merges them into third sensor data 28. List of reference symbols

100 Method according to the invention

101 Creating Compound Scenarios

102 Acquisition of first sensor data of a first scenario and of second sensor data of a second scenario

103 Provision of first sensor data of a first scenario and of second sensor data of a second scenario

104 Capturing a first ground truth for the first scenario and a second ground truth for the second scenario

105 Providing a first ground truth for the first scenario and a second ground truth for the second scenario

106 Validation of the first sensor data of the first scenario and the second sensor data of the second scenario

107 Classification of the respective points of the first sensor data and the respective points of the second sensor data as relevant or not relevant

108 Representation of the first sensor data and the second sensor data in a common voxel map

109 Identification of free voxels as free spaces

110 Identification of objects on the basis of the first sensor data and the second sensor data

111 Comparison of the positions of the detected objects of the first sensor data and the second sensor data

112 Detection of possible overlaps and / or occlusions

113 Identification of Noise Points

114 Combination of the first sensor data and the second sensor data to obtain third sensor data of a composite scenario

115 Entry of the points classified as relevant in a common coordinate system

116 Merging of the first ground truth and the second ground truth into a ground truth of the composite scenario

117 Testing an Object Recognition Unit

118 Provision of the third sensor data of the composite scenario 9 Comparison of the objects recognized by the object recognition unit with the ground truth of the composite scenario Device according to the invention First sensor data a Point of the first sensor data Second sensor data a Point of the second sensor data Unit for acquiring the first sensor data and the second sensor data Unit for merging the first sensor data and the second Sensor data classification unit Voxelizer object recognition unit Object merging unit Free space identification unit Noise point identification unit Point cloud marker Common voxel map a voxel Noise point Concealed point Relevant point Non-relevant point Third sensor data a Point of the third sensor data Concealed area

Claims

Expectations

1. A method (100) for creating (101) composite scenarios for testing an object recognition unit (17), the method (100) providing (103) first sensor data (11) of a first scenario and second sensor data (12) of a second Scenario includes, wherein the first sensor data (11) and the second sensor data (12) each have at least one point cloud comprising a plurality of points, characterized in that the method (100) a classification (107) of the respective points (11a) of the first sensor data (11) and the respective points (12a) of the second sensor data (12) in relevant or not relevant, the method (100) combining (114) the first sensor data (11) and the second sensor data (12) for obtaining third sensor data (28) of a composite scenario, and wherein only relevant points (26) of the first sensor data (11) and relevant points (26) of the second sensor data (12) to the third sensor data (28) of the composite scenario can be merged.

2. The method (100) according to claim 1, characterized in that the first sensor data (11) and the second sensor data (12) are displayed in a common voxel map (23), free voxels (23a) being identified as free spaces .

3. The method (100) according to any one of claims 1 or 2, characterized in that the method (100) comprises a respective identification (110) of objects on the basis of the first sensor data (11) and the second sensor data (12), wherein the Positions of the detected objects of the first sensor data (11) and the second sensor data (12) are compared (111) in order to identify possible overlaps and / or occlusions.

4. The method (100) according to any one of the preceding claims, characterized in that the method comprises the identification (113) of noise points (24).

5. The method (100) according to any one of claims 2 to 4, characterized in that the classification takes place on the basis of identified objects, identified free spaces and / or identified noise points (24).

6. The method (100) according to any one of claims 2 to 5, characterized in that hidden points (25) of a point cloud are classified as irrelevant within the scope of the classification, hidden points (25) within the common voxel map (23a) be covered by at least one point of the same and / or the other point cloud.

7. The method (100) according to any one of the preceding claims, characterized in that within the scope of the classification, points of a point cloud are classified as irrelevant if the points are covered by an object of the other point cloud and / or if the points are within an object which lie at their point cloud and / or if the points come from a first object that intersects with a second object in the other point cloud and / or if the points occur in the first sensor data and the second sensor data.

8. The method (100) according to any one of the preceding claims, characterized in that the merging (114) of the first sensor data (11) and the second sensor data (12) an entry (115) of the points classified as relevant (26) in one includes common coordinate system.

9. The method (100) according to any one of the preceding claims, characterized in that the method (100) comprises providing (105) a first ground truth for the first scenario and a second ground truth for the second scenario.

10. The method (100) according to claim 9, characterized in that the method (100) comprises a combination (116) of the first ground truth and the second ground truth to form a ground truth of the composite scenario.

11. The method (100) according to any one of claims 9 or 10, characterized in that the method comprises testing (117) an object recognition unit (17), the testing (117) providing (118) the third sensor data (28) of the composite scenarios and a comparison (119) of the objects recognized by the object recognition unit (17) with the ground truth of the composite scenario.

12. The method (100) according to any one of the preceding claims, characterized in that the method (100) comprises creating a library for creating sensor data of composite scenarios, with sensor data from various scenarios and optionally the corresponding ground truths of the scenarios in the library be filed.

13. Device (10) for creating composite scenarios for testing an object recognition unit (17), characterized in that the device (100) has a unit (14) for merging the first sensor data (11) and the second sensor data (12) to obtain of third sensor data (28) of a composite scenario, the device (10) having a classification unit (15) for classifying (107) the respective points (11a) of the first sensor data (11) and the respective gen points (12a) of the second sensor data (12) in relevant and not relevant.

14. Computer program product that comprises a computer-readable storage medium on which a program is stored that allows a computer light after it has been loaded into the memory of the computer, a method (100) according to one of claims 1 to 12, possibly in cooperation with a device (10) according to claim 13 to perform.

15. Computer-readable storage medium on which a program is stored which enables a computer after it has been loaded into the memory of the Compu age, a method according to one of claims 1 to 12, optionally in conjunction with a device (10) according to Claim 13 to perform.