EP3850536A1

EP3850536A1 - Analysis of dynamic spatial scenarios

Info

Publication number: EP3850536A1
Application number: EP19773727.3A
Authority: EP
Inventors: Halil BEGLEROVIC; Jonas RÜBSAM; Steffen METZNER
Original assignee: AVL List GmbH
Current assignee: AVL List GmbH
Priority date: 2018-09-14
Filing date: 2019-09-13
Publication date: 2021-07-21
Also published as: AT521647A1; CN112673379A; JP7511544B2; JP2022500762A; AT521647B1; US20220237889A1; KR20210060535A; WO2020051618A1

Abstract

The invention relates to a method and a system (100) for preparing data on dynamic spatial scenarios (10), to a computer-supported method, to a system (200) for training artificial neural networks (1), to a computer-supported method, and to a system (300) for analyzing sensor data. A display (20) is generated of a time curve of an angular sector covered by another object (12, 12a, 12b, 12c) from the perspective of an ego object (11a). The time curve is ascertained from sensor data, and the sensor data is suitable for characterizing a dynamic spatial scenario (10) with respect to the ego object (11a) and at least one other object (12, 12a, 12b, 12c).

Description

ANALYSIS OF DYNAMIC SPATIAL SCENARIOS

The present invention relates to a method and a system for data preparation of dynamic spatial scenarios for processing by an artificial neural network, a computer-aided method and a system for training an artificial neural network as well as a computer-aided method and a system. system for analyzing sensor data.

Modern vehicles are increasingly equipped with advanced driver assistance systems (ADAS), which support the driver in certain driving situations. Support ranges from the mere display of possibly relevant information (e.g. issuing a warning by a lane change assistant) to semi-autonomous interventions (e.g. regulation of the torque applied to the wheel axles by an anti-lock braking system) to fully autonomous interventions in the control of the vehicle (e.g. adaptive cruise control using adaptive cruise control (ACC).

The basis for such driver assistance systems is usually formed by sensor data, for example provided signals from ultrasound sensors, radar sensors or cameras, which can be used to determine the current driving situation and in response to this the function of the respective driver assistance system can be carried out. In particular in the case of driver assistance systems which (autonomously) intervene in the control of the vehicle, the current driving situation must be able to be classified very reliably on the basis of the sensor data.

In general, certain rules or criteria assigned to a driving situation are set up, and if they are fulfilled, it can be concluded that a known driving situation exists. Fulfilling the rule or criteria acts, for example, as a trigger for an action by the driver assistance system. For example, a traffic scenario in which a neighboring vehicle in front of the ego vehicle and equipped with the driver assistance system can enter the same lane. shear, can be recognized by the fact that a sensor-detected transverse distance decreases perpendicular to the direction of travel to the neighboring vehicle and finally, at least essentially, assumes the value 0 when the neighboring vehicle is located directly in front of the ego vehicle.

The programming of algorithms that are suitable for (reliably) recognizing such driving situations or scenarios is generally extremely complex and, in particular, possibly even impossible in the case of a large number of driving situations to be taken into account. Therefore, machine learning is increasingly being used to automatically find characteristics in sensor data that indicate a driving situation. The sensor data used is usually, e.g. Manually, classified sensor data with respect to different driving situations (English labeled sensor data), based on which, for example, an artificial neural network can extract the relevant criteria for the respective driving situation. The classified sensor data must usually be in a specified form, i.e. be prepared accordingly so that it can be meaningfully processed by the artificial neural network.

One approach to such preparation is represented by so-called grid maps, which show the corresponding driving situations from a bird's eye view. The dynamics of an underlying driving situation are reproduced by, for example, all the positions of the respective positions taken over time Traffic participants are shown in the corresponding grid map.

Such grid cards are for example in Grüner et. al. , “Spatiotemporal Representation of Driving Scenarios and Classification using Neural Networks”, 2017 IEEE Intelligent Vehicle Symposium, pp. 1782-1788.

In Cara and de Gelder, “Classification for safety-critical car-cyclist scenarios using machine learning”, 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pp. 1955-2000, various algorithms for the classification of vehicle cyclists are described Scenarios compared. In the case of supervised learning, a set of excellent objects (labeled objects) is used to train so-called classifiers. After training the classifiers are able to mark a new unknown scenario by assigning it to an estimated class.

It is an object of the present invention to further improve the processing of sensor data which are suitable for characterizing a dynamic spatial scenario in relation to an ego object and at least one other object. It is a particular object of the invention to improve the processing of such sensor data for processing by an artificial neural network, in particular to prevent or at least reduce a loss of information during processing.

This object is achieved by a method and a system for data processing, a method and a system for training an artificial neural network and a method and a system for analyzing sensor data in accordance with the independent claims.

A first aspect of the invention relates to a method for data processing of dynamic spatial scenarios, in particular traffic scenarios, for processing by an artificial neural network. In this case, a representation of a temporal course of an angle sector, which is covered by another object, in particular another vehicle, from the perspective of an ego object, in particular an ego vehicle, is generated. For this purpose, the course over time is determined from sensor data, the sensor data being suitable for characterizing a dynamic spatial scenario in relation to the ego object and at least one other object.

A spatial scenario in the sense of the invention is in particular formed from a chronological sequence of spatial, in particular static, scenes. The spatial scenes give, for example, the spatial arrangement of the at least one other object relative to the ego object, e.g. the constellations of road users. A spatial scenario can in particular contain a driving situation in which a driver assistance system at least partially controls the vehicle called the ego vehicle and equipped with the driver assistance system, e.g. carries out at least one vehicle function of the ego vehicle autonomously.

Dynamic in the sense of the invention means, in particular, temporal sequence or temporal course. Traffic scenarios are dynamic, for example, because the individual traffic turn participants move relative to each other over time, ie change their positions relative to each other.

A representation in the sense of the invention is in particular a graphic representation, e.g. a graph or chart. The representation preferably contains a, in particular two- or three-dimensional, image. Alternatively or additionally, the representation can also contain a mathematical representation, e.g. an assignment rule or a function. The representation can, for example, show the course over time of an angle sector, which is covered by the other object from the perspective of the ego object, in a map-like illustration.

Sensor data in the sense of the invention are, in particular, data generated by real or simulated sensors, in particular environmental sensors, in particular in the form of signals, i.e. real or simulated sensor data. The sensors are preferably set up to record an environment of the ego vehicle and to generate corresponding sensor data so that the sensor data characterize the environment of the ego vehicle. The sensor data are preferably sensor fusion data which are obtained from a combination of signals provided by the sensors and, if appropriate, have already been prepared, at least to a certain extent.

An ego object in the sense of the invention is in particular the object from the perspective of which the dynamic spatial scenario is viewed. An ego object can, for example, be a (ego) vehicle, from the point of view of which a traffic scenario is recorded and possibly analyzed and / or evaluated, for example in order to be able to adequately control a driver assistance system or to be able to react to the recorded traffic scenario. For this reason, sensor data that are processed, for example, for further processing are preferably generated by environmental sensors of the ego vehicle, the environmental sensors, such as ultrasound sensors, radar sensors, cameras and / or the like, being set up to detect the environment of the ego vehicle.

An angular sector in the sense of the invention is in particular that area which is taken up by another object from the perspective of the ego object. From the perspective of an ego vehicle, the angle sector corresponds, for example, to the area that is covered by another vehicle. Preferably the The angle sector is defined by the cross-sectional area of the object or its contour. For example, the angle sector can be defined by a maximum extension of the cross-sectional area or contour along a predetermined direction, in particular in the horizontal direction.

The invention is based in particular on the approach of encoding information relating to a dynamic spatial scenario, for example a traffic situation, determined from sensor data in a representation. The representation can therefore also be understood as an abstraction of the scenario and serves in particular to summarize different types of information, for example positions, translations, number of objects and / or the like.

The representation is preferably generated on the basis of the temporal development of an angular sector in which the field of vision of an ego object is covered by another object. With the help of the angle sector, in particular constellations of objects in spatial scenes can be described relative to each other. The representation is preferably a graphic, in particular two- or three-dimensional, image, which - in an abstract manner - shows both the position and the movement of the objects relative to one another. For example, the movement of an object in the illustration can be represented by a shape, in particular a geometric shape.

A characteristic form can preferably be found on the basis of its analysis of such an image. The shape is particularly characteristic of the dynamic spatial scenario in relation to the ego object and at least one other object. In other words, the characteristic shape is a pattern that e.g. can be found by means of an artificial neural network by analyzing such a representation, possibly also several such representations that were generated on the basis of the sensor data.

Conversely, if the characteristic shapes or patterns for different dynamic spatial scenarios are known and are available, for example, as templates, the corresponding dynamic spatial scenario can be determined on the basis of an evaluation of a representation. The representation of a dynamic spatial scenario based on the temporal development of the angle sector has the advantage that its information content does not depend, or at least only slightly, on variations in the scenario. Different variants of the scenario, for example a particularly fast or particularly slow driving maneuver, produce different images in the display, but with essentially the same or at least a similar shape.

For example, plotting the angular sector, which is covered by another vehicle with respect to the field of vision of a ego vehicle and thus represents the position of the other vehicle relative to the ego vehicle, generates a pattern over time, the shape of which is for that of driving maneuvers performed in another vehicle is characteristic. If the other vehicle moves away from the ego vehicle, the covered angular sector becomes somewhat smaller, while it increases correspondingly when approached. If the other vehicle is moving e.g. with respect to the direction of travel to the side of the ego vehicle, the covered angle sector shifts. The resulting pattern, which, for example, has a characteristic curvature, can then be reliably assigned to a driving maneuver, even if the image generally varies, for example aggressively or defensively, depending on the execution of the driving maneuver, e.g. is compressed or stretched.

The representation of the temporal development of the angular sector thus enables various types of information to be summarized in a compact manner. Since a time lapse is shown here, overlaps, e.g. of graphic elements, and an associated loss of information avoided or at least reduced. At the same time, for example, storage space can be saved and the evaluation of sensor data can be accelerated. At the same time, the display enables reliable identification of dynamic spatial scenarios.

Compared to grid maps, the representation of the temporal development of the angle sector has the advantage, for example, that the same information can be represented by a much smaller amount of data. In a grid map, individual pixels that are not occupied by another vehicle (ie “empty” pixels), for example, do not contribute to the information content that can be used by an artificial neural network, for example. In particular, the amount of data required to create an artificial neural network train can be reduced by a factor of 27 by using the representation of the temporal development of the angular sector.

Overall, the present invention makes it possible to further improve the processing of sensor data. In particular, the processing of sensor data for processing by an artificial neural network can be improved, for example by preventing or at least reducing information loss during processing.

Preferred embodiments of the invention and their further developments are described below, each of which, as far as this is not expressly excluded, can be combined with one another as desired and with the aspects of the invention described below.

In a preferred embodiment, the representation is output to an artificial neural network or made available for processing by the artificial neural network. For example, the representation can be stored as a data packet, in particular as a digital image, on a storage medium or transmitted directly to the artificial neural network via an interface. As a result, large amounts of sensor data can also be used to train the artificial neural network, in particular to determine patterns in the representation that are characteristic of a given scenario, such as the shape of a figure in the representation.

A second aspect of the invention relates to a computer-based method for training an artificial neural network on the basis of sensor data which are suitable for characterizing a known dynamic spatial scenario in relation to an ego object and at least one other object. A representation of a time profile of an angle sector, which is covered from the perspective of an ego object, in particular an ego vehicle, by another object, in particular another vehicle, is generated on the basis of the sensor data. The generated representation together with information about the spatial scenario is fed to the artificial neural network. As a result, the artificial neural network can be trained particularly quickly and reliably to recognize patterns that are characteristic of different scenarios. The information about the spatial scenario preferably contains an indication of a classification of the scenario, with the aid of which the scenario can be identified, preferably uniquely. The information can already be contained in the sensor data, for example if it is simulated sensor data that was generated by a simulator when simulating a specific scenario. Alternatively, the information can also be supplied to the artificial neural network as a separate data stream, for example if the sensor data have already been analyzed and classified in relation to at least one scenario. This classification can in particular have been carried out manually, for example by analyzing an image stream corresponding to the sensor data.

A third aspect of the invention relates to a computer-aided method for analyzing sensor data which are suitable for characterizing a dynamic spatial scenario in relation to an ego object and at least one other object. A representation of a time profile of an angle sector, which is covered by another object, in particular another vehicle, from the perspective of an ego object, in particular an ego vehicle, is generated on the basis of the sensor data. The generated representation is compared with at least one predefined template of a known dynamic spatial scenario. This means that a known scenario can be detected particularly quickly and reliably on the basis of a sensor data stream, such as is provided by environmental sensors of a vehicle.

The predefined template is preferably a generic representation of the temporal development of the angle sector, which e.g. contains all essential characteristic patterns of a scenario. For example, the predefined template can be a representation that contains at least one figure, the shape of which is characteristic of a driving maneuver of a vehicle that is carried out in the context of a traffic scenario.

The predefined template is preferably determined on the basis of a number of representations which are characteristic of the known dynamic spatial scenario. In particular, the multiple representations that are characteristic of the known scenario can be averaged. These representations can be generated beforehand, for example, from sensor data that have been classified in relation to the scenario have been. As a result, a high degree of reliability can be achieved when comparing the generated representation with the predefined template.

In a preferred embodiment, the dynamic spatial scenario is classified on the basis of the comparison, for example by assigning the determined representation to the at least one predefined template. This enables the scenario to be reliably identified and, if necessary, reactions of a driver assistance system to be triggered.

When comparing the generated representation with the predefined template, a measure of similarity is preferably determined, on the basis of which the scenario can be classified, for example, and the generated representation can be assigned to a scenario. It is e.g. it is conceivable to assign the generated representation to the scenario on the predefined templates of which the generated representation can best be represented by means of an elastic matching or nonlinear template matching method, i.e. for which a similarity measure obtained by the method becomes maximum. This enables the scenario to be identified particularly reliably.

In a further preferred embodiment, a template is defined for a new dynamic spatial scenario if the generated representation cannot be assigned to the at least one predefined template of a known dynamic spatial scenario. This enables new dynamic scenarios to be identified, in particular classified. In particular, new dynamic scenarios can thus be identified essentially in real time.

For example, a template can be defined for a new dynamic spatial scenario if a similarity measure determined when comparing the generated representation does not reach a predetermined threshold value. As a result, a catalog of predefined templates of known dynamic spatial scenarios can be generated in a simple manner, in particular in real time.

In a further preferred embodiment, the generated representation of the further gives a, in particular transverse, distance of the ego object to the other object and / or a, in particular transverse, speed of the ego object. the distance and / or the speed is also or are determined from sensor data. The corresponding distance information or speed information is preferably encoded in the representation in such a way that it can be read independently of at least the spatial arrangement of the other object relative to the ego object. By considering the distance and / or the speed, a dynamic spatial scenario can be identified particularly reliably. In particular, the consideration of the distance and / or the speed also enables a particularly fine distinction between different dynamic spatial scenarios.

The speed of the ego object can in particular be a transverse speed, i.e. a velocity component essentially perpendicular to the dominant (longitudinal) direction of movement, e.g. a direction of travel, the ego object. The speed of the ego object reproduced in the representation produced is preferably formed exclusively from the transverse speed.

For example, when driving straight ahead, the transverse speed of the ego vehicle is zero, while when changing lanes it increases to a maximum value and then drops back to zero.

Based on the speed shown in the illustration, e.g. a distinction is made between a lane change carried out by the ego vehicle and a lane change carried out by another vehicle. Since these two driving maneuvers do not differ with respect to the relative movement of the ego vehicle and the other vehicle to one another, i.e. the representation in both cases, for example, shows the same figure, only when the transversal speed of the ego vehicle is taken into account does it enable a reliable statement to be made as to which vehicle is changing lanes.

The distance of the ego object to the other object can in particular have a transversal distance, ie a distance component essentially perpendicular to the dominant (longitudinal) direction of movement, for example a direction of travel, of the ego object. The distance of the ego object to the other object reproduced in the representation produced is preferably formed exclusively from the transverse distance. If the ego vehicle is overtaken by another vehicle in an adjacent lane, for example, the absolute distance between the ego vehicle and the other vehicle changes. However, the transverse distance remains constant as long as neither the ego vehicle nor any other vehicle leaves its respective lane. The change in the transverse distance therefore allows (further) conclusions to be drawn about the maneuver carried out, for example in the context of a scenario.

In a further preferred embodiment, the time profile of an angle sector is represented by a line, the width of which indicates a value of the respective angle sector. As an alternative or in addition, a distance of the ego object to the other object and / or a speed of the ego object is or become by a stored value or a coloring at the respective position of the line, which corresponds to the time of the presence of the distance and / or Corresponds to the speed shown.

The stored value or the coloring of the line can alternatively or additionally be determined by a, in particular generic, function, into which, for example, input variables the, in particular transverse, distance between the ego object and the other object and / or the, in particular transverse, speed of the ego object. The function can be used, for example, to add or multiply the distance and the speed, possibly taking a weighting into account. In particular, the speed can be weighted with the distance or the distance with the speed. The information obtained in this way, represented in the representation or by the stored value, enables a particularly reliable and detailed distinction to be made between driving maneuvers and the corresponding dynamic spatial scenarios.

The function is preferably used to manipulate the perception of an artificial neural network. In other words, the function can be selected so that the resulting representations influence the pattern recognized by the artificial neural network. As a result, different dynamic spatial scenarios can be distinguished particularly reliably.

Alternatively or additionally, parameters other than the distance between the ego object and the other object and / or the speed of the ego object can also be considered Input variables of the function can be selected. The parameters can be selected, for example, depending on the, in particular known, dynamic spatial scenario. Parameters are preferably selected as input variables that at least partially characterize the dynamic spatial scenario.

A fourth aspect of the invention relates to a system for data processing of dynamic spatial scenarios, in particular traffic scenarios, for processing by an artificial neural network. The system has a determination module which is set up to use sensor data to plot a time profile of an angle sector, which is covered by another object, in particular another vehicle, from the perspective of an ego object, in particular an ego vehicle to investigate. The sensor data are suitable for characterizing a dynamic spatial scenario in relation to the ego object and at least one other object. The system also has a generation module which is set up to generate a representation of the determined time profile.

A fifth aspect of the invention relates to a system for training an artificial neural network based on sensor data which are suitable for characterizing a known dynamic spatial scenario in relation to an ego object and at least one other object. The system has a generation module which is set up to display a time profile of an angle sector which is covered by another object, in particular another vehicle, from the perspective of an ego object, in particular an ego vehicle. to generate based on the sensor data. The system also has an interface that is set up to supply the generated representation together with information about the spatial scenario to the artificial neural network.

A sixth aspect of the invention relates to a system for analyzing sensor data which are suitable for characterizing a dynamic spatial scenario in relation to an ego object and at least one other object. The system has a generation module which is set up to display a time profile of an angular sector which is covered by another object, in particular another vehicle, from the perspective of an ego object, in particular an ego vehicle. to generate based on the sensor data. The system also has a comparison module which is set up to display the generated data. Compare position with at least one predefined template of a known dynamic spatial scenario.

The features and advantages described in relation to the first aspect of the invention and its advantageous embodiment also apply, at least where it makes technical sense, to the second, third, fourth, fifth and sixth aspect of the invention and its advantageous embodiment, and vice versa.

Further features, advantages and possible uses of the invention result from the following description in connection with the figures, in which the same reference numerals are used throughout for the same or corresponding elements of the invention. At least partially, they show schematically:

1 each shows a preferred exemplary embodiment of a system according to the invention for data processing and of a system according to the invention for training an artificial neural network;

2 shows a preferred exemplary embodiment of a system according to the invention for analyzing sensor data;

3 shows an example for determining an angle sector in a traffic scenario;

4 shows a first example to explain the relationship between a representation according to the invention and a corresponding dynamic spatial scenario;

5 shows a second example to explain the relationship between a representation according to the invention and a corresponding dynamic spatial scenario;

6 each shows a preferred exemplary embodiment of a method according to the invention for data processing and of a method according to the invention for training an artificial neural network; and 7 shows a preferred exemplary embodiment of a method for data evaluation according to the invention.

1 shows a preferred exemplary embodiment of a system 100 according to the invention for data processing in dynamic spatial scenarios and a system 200 according to the invention for training an artificial neural network 1. The system 100 for data processing has a determination module 2 and a generation module 3 , wherein the determination module 2 is set up to determine a temporal profile of an angle sector, which is covered by another object from the perspective of an ego object, from sensor data S, and the generation module 3 is set up to generate a representation of the determined course over time. The system 200 for training the artificial neural network 1 has the generation module 3 and an interface 4, the interface 4 being set up to supply the generated representation together with information about a dynamic spatial scenario to the artificial neural network 1.

The sensor data S are generated, for example, by environmental sensors of a ego vehicle when recording a traffic scenario and characterize, for example, the number of neighboring vehicles, the relative arrangement, in particular the relative distances, of the other vehicles from the ego vehicle, the speed of the ego vehicle, and / or similar. From these sensor data S, the determination module 2 can preferably determine how wide at least one angle sector hidden by the other vehicles is in the field of vision of the ego vehicle and in what position this angle sector, for example relative to the direction of travel of the ego vehicle, is arranged.

Since the environment sensors preferably record the surroundings of the ego vehicle continuously, at least with a high frequency, the resulting sensor data S can also be used to determine the time profile of the angle sector, in particular a change in its width and / or its position from the perspective of the ego vehicle. The time course can be used by the generation module 3 to generate a graphical representation that depicts the traffic scenario in an abstract manner. In other words, the generation module 3 is set up to encode the information contained in the sensor data S regarding the traffic scenario in the representation, in particular in compressed form. The sensor data S are preferably sensor data classified in relation to dynamic spatial scenarios, ie the sensor data S are assigned, for example, to one of several classes of traffic scenarios, such as overtaking maneuvers or lane change maneuvers. This classification can be done manually, for example, by viewing an image data stream. The representation generated on the basis of the determined time profile of the angle sector can therefore be transmitted from the interface 4 to the artificial neural network 1 with information about the dynamic spatial scenario.

The artificial neural network 1 is preferably set up to recognize at least one pattern in each case in all representations which are assigned to the same traffic scenario class. On the basis of such a recognized pattern, a template can be defined that characterizes a known traffic scenario. The templates defined in this way can be stored in a database 5 for further use, for example for evaluating sensor data generated in the normal operation of a vehicle.

FIG. 2 shows a preferred embodiment of a system 300 for analyzing sensor data S which are suitable for characterizing a dynamic spatial scenario in relation to an ego object and at least one other object. The system 300 has a generation module 3, which is set up to generate, based on the sensor data S, a representation of a temporal profile of an angle sector, which is covered by another object from the perspective of the ego object. The system 300 also has a comparison module 6, which is set up to compare the generated representation with at least one predefined template of a known dynamic spatial scenario. For this purpose, the comparison module 6 preferably has access to a database 5, in which at least one predefined template is stored.

The result of the comparison is output by the comparison module 6 in a preferred manner and can be used, for example, to control a driver assistance system with which an ego vehicle is equipped. If, for example, at least a predetermined degree of agreement is ascertained between the generated representation and the at least one predefined template, for example by analyzing a same generated measure of similarity, it can be concluded that the known scenario is present, and the comparison module 6 can output approximately one scenario class as the output signal. Alternatively or additionally, the start and / or the end of an identified driving maneuver can also be output, in particular signaled.

FIG. 3 shows an example for the determination of an angle sector F in a dynamic spatial scenario 10, in which an ego object 11 moves along a direction of movement, here along the x-axis of a coordinate system, and from other objects 12 is surrounded. In the present example, the spatial scenario 10 is a traffic scenario with an ego vehicle 1 1a, which is moving in the direction of travel in a middle lane and is surrounded by other vehicles 12a, 12b in adjacent lanes.

FIG. 3A shows the traffic scenario from a bird's eye view, a temporal sequence of spatial scenes being summarized in the illustration shown. At different times, the other vehicles 12a, 12b are in different positions relative to the ego vehicle 1 1a, which are indicated by different fillings of the rectangles representing the vehicles 12a, 12b. A denser filling corresponds to a position further back in time. As can be clearly seen in this illustration, a first vehicle 12a performs a lane change from an outer lane to the lane used by the ego vehicle 1 1a and in doing so, it turns in front of the ego vehicle 1 1a. A second vehicle 12b, which is initially at approximately the same level as the ego vehicle 1 1 a with respect to the direction of movement, falls backwards over time. Due to the fact that the change in position of the second vehicle 12b is relatively small, the positions assumed by the second vehicle 12b at different times overlap in this illustration.

In order to generate an alternative representation of the traffic scenario, an angle sector can be determined for each of the two vehicles 12a, 12b, which indicates the area in the field of vision of the ego vehicle 11a that is covered by the respective vehicle 12a, 12b. This is shown in FIG. 3B as an example for the first vehicle 12a. The contour 13 of the first vehicle 12a that results in relation to the perspective of the ego vehicle 1 1 a is indicated as a solid line and spans the angle sector F. A position cp of the angle sector F, and thus also of the first vehicle 12a View of the ego vehicle 1 1 a can be specified relative to a predetermined direction, for example the direction of movement. If the position of the vehicle 12a shifts relative to the ego vehicle 11a, both the width of the angle sector F and its position cp can change.

This is shown in Figure 3C. The bar diagram depicted there, in which the inverse distance d, indicated as a black arrow in FIG. 3A, between the ego vehicle 1 1a and the other vehicles 12a, 12b is plotted against the position cp of the angle sector F, forms an abstract representation of the one shown in FIG. 3A traffic scenarios shown. The three bars on the right correspond to the angular range F covered by the first vehicle 12a, while the three bars on the left correspond to the angular range F covered by the second vehicle 12b. As in FIG. 3A, the different times are indicated by the corresponding filling of the bars.

Accordingly, the angular range F covered by the first vehicle 12a moves in the direction of a 0 ° position when it is reeved in front of the ego vehicle 11a, the 0 ° position of the angle sector F corresponding to a position in front of the ego vehicle 11a . As the distance d between the first vehicle 12a and the ego vehicle 11a decreases as shown in FIG. 3A, the width of the angular range F covered by the first vehicle 12a also increases. In addition to the width of the angular range F, the distance d between the first vehicle 12a and the ego vehicle 11a is also encoded in the height of the bars, which increases with the passage of time.

By contrast, the angular range F covered by the second vehicle 12b moves away from the 0 ° position to the extent that the second vehicle 12b falls behind the ego vehicle 1 1a. Since the distance between the second vehicle 12b and the ego vehicle 1 1a increases, the height of the bars and their width also decrease.

FIG. 4 shows a first example to explain the relationship between a preferred exemplary embodiment of a representation 20 according to the invention and a corresponding dynamic spatial scenario 10. To this end, FIG. 4A shows the representation 20 of time profiles of angle sectors FQ, F ^ FO, that of others Objects from the perspective of an ego object, such as an ego vehicle 1 1a, are covered become. In the illustration 20, the time t is plotted against the position cp of the angle sectors Oa, Ob, Oc, the 0 ° position corresponding to a position in front of the ego object.

The temporal courses of the angle sectors Oa, Ob, Oc are represented as lines, the width of which corresponds to the distance of the respective object from the ego object. A first object, for example a first vehicle 12a, is accordingly at the beginning at t = 0 some distance in front of the ego object. By contrast, a second object, for example a second vehicle 12b, is initially closer to the ego object, but in a position to the side of the ego object. At about t = 40, the second object changes its position and moves between the ego object and the first object. From this point in time, the time profile of the angle sector Ob, which is covered by the second object, overlaps the profile of the angle sector Oa, which is covered by the first object.

At about t = 70, another angle sector Oc is covered by a third object, such as a third vehicle 12c. The third object is located to the side of the ego object, on a side of the ego object that is opposite to the side on which the second object was located at t = 0. The position cp of the angle sector Oc covered by the third object subsequently shifts in the direction of the 0 ° position. This temporal course can be caused, for example, by a movement of the third object parallel to the direction of movement of the ego object, the distance between the third object and the ego object increasing.

Such temporal profiles of the angle sectors Oa, Ob, Oc can, for example, be characteristic of the spatial dynamic scenario 10 shown in FIG. 4B, here a traffic scenario. The development over time is indicated in FIG. 4B by trajectories of the vehicles on the carriageway marked with dots.

Initially, the ego vehicle 1 1a is at the position (x = 0, y = 0), and the first vehicle 12a is approximately at the position (x = 60, y = 0). Accordingly, the first vehicle 12a drives the ego vehicle 11a in a middle lane approximately at a distance Dc = 60. At the beginning, the second vehicle 12b is approximately at the position (x = 45, y = -4), it accordingly being in a neighboring lane, ie laterally offset with respect to the ego vehicle 11a, between the ego vehicle 1 1a and the first vehicle 12a drives. In the further course of time, the second vehicle 12b reevers between the ego vehicle 11a and the first vehicle 12a in the middle lane. The cutting-in movement of the second vehicle 12b begins approximately at the position (x = 100, y = -4) and ends approximately at the position (x = 150, y = 0). After the further vehicle 12b has been cut in, the ego vehicle 1 1a, the first vehicle 12a and the second vehicle 12b continue to move together in the middle lane. Because the second vehicle 12b now obscures the view of the first vehicle 12a from the perspective of the ego vehicle 1 1a, only one line is visible in the corresponding representation 20 in FIG. 4A in the 0 ° position.

The third vehicle 12c is initially located behind the ego vehicle 1 1a, so that it is initially not detected, for example by environmental sensors of the ego vehicle 1 1a. However, the third vehicle 12c moves at a higher speed than the ego vehicle 1 1 a, so that it overtakes the ego vehicle 1 1 a in the further course on a further neighboring lane at y = 4. Only from the time of overtaking does the third vehicle 12c become visible to the first-person vehicle 11a, so that the time profile of the angular range <t> c covered by the third vehicle 12c in the corresponding illustration 20 in FIG. 4A only begins from this time .

The representation 20 in FIG. 4A created by the time profiles of the angle sectors has a pattern which is characteristic of the spatial dynamic scenario 10 described. If many such representations are generated on the basis of sensor data that were repeatedly collected when such a scenario was recorded, an artificial neural network can learn this pattern or can be trained to recognize this pattern. During regular operation of a vehicle, the temporal course of angle sectors can then be represented in real time from the sensor data generated and analyzed by the trained artificial neural network, in particular compared with the learned pattern. This preferably determines whether or at what point in time a known traffic scenario is pending.

4C shows the result of such a comparison, for example between the illustration 20 shown in FIG. 4A and a corresponding template, an output signal a being plotted against time t. This jumps around time t = 30 Output signal a to the value 6, which signals the presence of the known traffic scenario, in this case the reeving of a second vehicle 12b. Around time t = 85, the output signal a falls back to zero, which signals the end of the single maneuver.

In addition to the start and duration of the driving maneuver, the value of the output signal a can also be used to output the classification of the traffic scenario. In another example, not shown, the comparison of the generated representation with different templates could show the greatest agreement with a template that is assigned to a swinging-out maneuver, and the output signal a could assume a different value accordingly.

FIG. 5 shows a second example to explain the relationship between a preferred exemplary embodiment of an illustration 20 according to the invention and a corresponding dynamic spatial scenario 10. To this end, FIG. 5A shows a scene from the traffic situation corresponding to the dynamic spatial scenario 10, in this case one by the arrow indicated, ego vehicle 1 1a shears into a lane in which another first vehicle 12a is already driving. In addition to the first vehicle 12a, a further, second vehicle 12b is traveling in a further lane.

The development over time of the dynamic spatial scenario 10 is shown in FIG. 5B by trajectories of the vehicles 11a, 12a, 12b, identified by points. At the beginning, the ego vehicle 1 1 a is approximately at the position (x = 0, y = 2.5), and the first vehicle 12a drives the ego vehicle 1 1 a approximately at the position (x = 35, y = 0) ahead in a middle lane, ie approximately at a distance Dc = 35. At the beginning, the second vehicle 12b is approximately at the position (x = 25, y = -2.5).

In the further course of time, the ego vehicle 1 1 a begins approximately at the position (x = 60, y = 2) by reeving into the middle lane and ends the driving maneuver approximately at the position (x = 100, y = 0) ). The other two vehicles 12a, 12b continue to drive straight ahead at a somewhat higher speed, so that the ego vehicle 11a slowly falls behind further. 5C shows the temporal profiles of the angle sectors Oa, Ob, which are covered by the first vehicle 12a and the second vehicle 12b from the perspective of the ego vehicle 1 1a, in a representation 20. The time t is plotted against a position f of the angle sectors Oa, Ob. A 0 ° position shown in FIG. 5c corresponds to a position in front of the ego vehicle 1 1 a.

In the course of time, the angle sectors Oa, Ob shift in the direction of the 0 ° position, since the first and second vehicles 12a, 12b, as described above, move away from the ego vehicle 11a due to their higher speed. At the time the ego vehicle 1 1a is cut in, here at time t = 30, the time profiles of the angle sectors Oa curve, whether in the direction of the 0 ° position. The angle sector Oa covered by the first vehicle 12a subsequently runs along the 0 ° position, since, as shown in FIGS. 5A and 5B, the first vehicle 12a drives the ego vehicle 11a in the same lane.

On the basis of the resulting pattern, only statements about the relative movement of the ego vehicle 1 1a with respect to the first and second vehicles 12a, 12b can be made, i.e. it is evident from the shape of the temporal course of the angle sectors Oa, whether at first it is not clear whether the ego vehicle 1 1 a is changing to the middle lane or whether the first and the second vehicle 12 a, 12 b, in particular essentially at the same time, each carry out a lane change, the first vehicle 12a swiveling onto the lane of the first-person vehicle 1 a and the second vehicle 12b onto the lane previously used by the first vehicle 12a.

In order to be able to distinguish these two cases from one another, the illustration 20 shows, in addition to the angle sectors Oa, Ob additionally a value which is characteristic of the, in particular transverse, distance between the ego vehicle 1 1 a and the other vehicle 12 a, 12 b and / or for the, in particular transverse, speed of the ego vehicle 11 a at time t. The transverse distance or the transverse speed refer here to a transverse component of the distance or the speed, i.e. of the y components in the illustration shown in FIG. 5B.

The value which is characteristic of the distance and / or the speed is shown in the illustration 20 as a coloration of the time profile indicated by the tint of the angular sectors shown. A dark tint corresponds, for example, to a high transverse speed of the ego vehicle 1 1 a. It can thus be seen from the illustration 20 in FIG. 5C that the ego vehicle begins a transverse movement, ie a movement perpendicular to the lane course, at about the time t = 20, the ego vehicle 1 a at approximately t = 30 reaches the highest transverse speed and the transverse movement ends at approximately t = 40.

Since the ego vehicle 11a does not change its (transversal) speed when a lane change is carried out by one of the other vehicles 12a, 12b, in the present case the ego vehicle 11a can be swiveled into the lane used by the first vehicle 12a getting closed.

6 each shows a preferred exemplary embodiment of a method V1 according to the invention for data processing and a method V2 according to the invention for training an artificial neural network.

In a method step S1, sensor data are generated, for example by sensory detection of the environment of an ego object, and classified, i.e. assigned to different dynamic spatial scenarios. The classification can be carried out manually, for example, by evaluating an image data stream. Alternatively, the sensor data can also be classified automatically, in particular if it is sensor data that was generated by a simulator when simulating various dynamic spatial scenarios.

In a further method step S2, a temporal course of an angle sector, which is covered by another object from the perspective of the ego object, can be determined on the basis of the sensor data. For example, the contour, in particular the cross-sectional area, of the other object can be determined and its width in or the proportion of the field of vision of the ego object can be determined. In addition, a geometric center of gravity of the contour or the cross-sectional area can be determined and its position in the field of view of the ego object, in particular relative to the direction of movement of the ego object, can be determined.

In a further method step S3, a, in particular graphical, representation is generated from the time profile of the angle sector, which represents the maps over time. In such a representation, the time profile of the angle sector can, for example, form a pattern, for example a figure. For each point in time when sensor data is generated or at least one angle sector covered by another object is determined, the width of the angle sector, in particular its share in the field of view of the ego object, and its position in the field of view of the ego object are preferably determined , in particular relative to the direction of movement of the ego object.

In method step S3, a speed of the ego object, in particular a transverse speed, and / or a distance, in particular a transverse distance, of the ego object from the other object is also taken into account in generating the representation. In particular, a value can be determined and stored, for example using a function into which parameters such as the speed and / or the distance characterizing the dynamic spatial scenario are included, or the representation can be colored accordingly. The generated representation thus preferably provides information regarding the width and position of the angle sector covered by the other object in the field of view of the ego object and the speed of the ego object and / or the distance to the other object.

In a further method step S4, the generated representation is fed, for example by means of an interface, to an artificial neural network, which is thereby trained in particular to recognize patterns in the generated representation. For this purpose, information about the dynamic spatial scenarios according to which the sensor data were classified in method step S1 is preferably supplied to the artificial neural network, so that the artificial neural network generates the representation or the recognized pattern can each correlate with one of the dynamic spatial scenarios.

7 shows a preferred exemplary embodiment of a method V3 according to the invention for analyzing sensor data which are suitable for characterizing a dynamic spatial scenario in relation to an ego object and at least one other object. The sensor data are preferably sensor data generated by environmental sensors of a ego vehicle. In method step S3, a representation of a time profile of an angle sector is generated from the sensor data. The angle sector corresponds to the area in the field of view of the ego object that is covered by the other object. Such a representation can be, for example, an illustration in which the temporal course of the angle sector forms a pattern, for example a figure.

The time profile of the angle sector can also be determined in a separate, previous method step (not shown) based on the sensor data.

In a further method step S5, the generated representation is compared with at least one predefined template of a known dynamic spatial scenario. In this way, it can be determined in which dynamic spatial scenario the ego object is currently located and, if necessary, a driver assistance system can be controlled accordingly.

Optionally, in a further method step S6, the generated representation can be saved as a further predefined template, for example in a database, if there is no or at least insufficient correspondence of the generated representation with the at least one predefined template that a known dynamic is assigned to spatial scenarios, can be determined. In this way, a catalog can be generated with predefined templates, which are suitable for identifying dynamic spatial scenarios, in particular essentially in real time.

Reference list

1 artificial neural network

2 generation module

3 determination module

4 interface

5 database

6 comparison module

10 spatial dynamic scenario

1 1 ego object

1 1 a ego vehicle

12 other vehicle

12a first vehicle

12b second vehicle

12c third vehicle

13 contour

20 representation

100 data processing system

200 System for training an artificial neural network

300 system for analyzing sensor data

F angular sector

F Position of the angular sector

S sensor data y direction of movement

d distance

a output signal

V1 Data preparation procedure

V2 Method for training an artificial neural network

V3 Method for analyzing sensor data S1 - S6 procedural steps

Claims

1. Method (V1) for data processing of dynamic spatial scenarios (10), in particular traffic scenarios, for processing by an artificial neural network (1), with a representation (20) of a time profile of an angle sector (F), which is covered from the perspective of an ego object (1 1), in particular an ego vehicle (1 1 a), by another object (12), in particular another vehicle (12a, 12b, 12c) is (S3), the time profile being determined from sensor data (S) (S2) and the sensor data (S) being suitable for creating a dynamic spatial scenario (10) in relation to the ego object (11) and to characterize at least one other object (12).

2. The method (V1) according to claim 1, wherein the representation is output to an artificial neural network (1) or provided for processing by the artificial neural network (1).

3. Computer-aided method (V2) for training an artificial neural network (1) on the basis of sensor data (S) which are suitable for a known dynamic spatial scenario (10) in relation to an ego object (1 1 ) and to characterize at least one other object (12), a representation (20) of a temporal course of an angular sector (F), which is from the perspective of an ego object (11), in particular an ego vehicle (1 1 a), is covered by another object (12), in particular another vehicle (12a, 12b, 12c), is generated (S3) on the basis of the sensor data (S), and the generated representation (20) together is supplied to the artificial neural network (1) with information about the spatial scenario (10) (S4).

4. Computer-aided method (V3) for analyzing sensor data (S) which are suitable for creating a dynamic spatial scenario (10) in relation to an ego object (11) and at least one other object (12) to be characterized, wherein a representation (20) of a time profile of an angular sector (F), which from the perspective of an ego object (1 1), in particular an ego vehicle (1 1 a), through another object (12), especially another driving tool (12a, 12b, 12c) is covered, is generated on the basis of the sensor data (S) (S3), and the generated representation (20) with at least one predefined template of a known dynamic spatial scenario (10) is compared (S5).

5. The method (V3) according to claim 4, wherein the dynamic spatial scenario (10) is classified on the basis of the comparison.

6. The method (V3) according to claim 4, wherein a template for a new dynamic spatial scenario (10) is defined if the generated representation (10) does not match the at least one predefined template of a known dynamic spatial scenario (10). can be assigned (S6).

7. The method (V3) according to one of claims 1 to 6, wherein the generated representation (10) furthermore a distance (d) of the ego object (1 1) to the other object (12) and / or a speed of the ego object (11) reproduces, the distance (d) and / or the speed likewise being or being determined from sensor data (S).

8. The method (V3) according to one of claims 1 to 7, wherein the temporal course of an angle sector (F) is represented by a line, the width of which indicates a value of the respective angle sector (F), and / or wherein a distance (d) the ego object (1 1) to the other object (12) and / or a speed of the ego object (1 1) by a stored value or a coloring at the respective point on the line which corresponds to the time of the presence of the distance (d) and / or the speed, is or will be represented.

9. System (100) for data processing of dynamic spatial scenarios (10), in particular traffic scenarios, for processing by an artificial neural network (1), comprising:

a determination module (2), which is set up to plot a temporal profile of an angle sector (F), which from the perspective of an ego object (1 1), in particular an ego vehicle (1 1 a), through another object ( 12), in particular another vehicle (12a, 12b, 12c), is covered from sensor data ten (S), the sensor data (S) being suitable for characterizing a dynamic spatial scenario (10) in relation to the ego object (1 1) and at least one other object (12); and

- A generation module (3) which is set up to generate a representation (20) of the determined time profile.

10. System (200) for training an artificial neural network (1) on the basis of sensor data (S) which are suitable for a known dynamic spatial scenario (10) in relation to an ego object (1 1) and to characterize at least one other object (12), comprising:

- A generation module (3), which is set up to display (20) a time profile of an angle sector (F), which from the perspective of an ego object (1 1), in particular an ego vehicle (1 1a) generate another object (12), in particular another vehicle (12a, 12b, 12c), on the basis of the sensor data (S);

an interface (4) which is set up to supply the generated representation (20) together with information about the spatial scenario (10) to the artificial neural network (1).

1 1. System (300) for analyzing sensor data (S), which are suitable to characterize a dynamic spatial scenario (10) in relation to an ego object (1 1) and at least one other object (12) :

a generation module (3), which is set up to display (20) a time profile of an angle sector (F), which is from the perspective of an ego object (1 1), in particular an ego vehicle (1 1 a), is covered by another object (12), in particular another vehicle (12a, 12b, 12c), on the basis of the sensor data (S); and

a comparison module (6) which is set up to compare the generated representation (20) with at least one predefined template of a known dynamic spatial scenario (10).