DE102022128538A1 - Method, system and computer program for developing, securing, training and/or operating a vehicle system - Google Patents

Abstract

The invention relates to a method, a system and a computer program for developing, securing, training and/or operating a vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3'). The method comprises:- determining (S1) one or more first data sections (DA1, DA1', DA1'') which characterize one or more real situations of one or more real vehicles (RF1, RF2, RF3);- training (S2) and/or operating (S3) at least one neural network (DNN1) depending on the first data sections (DA1, DA1', DA1'');- generating (S4) second data sections (DA2, DA2', DA2'') which characterize one or more at least partially virtual situations, depending on the output layer of the neural network (DNN1);- providing (S5) the second data sections (DA2, DA2', DA2'') or the data determined depending on the second data sections (DA2, DA2', DA2'') for developing, securing, training (S6) and/or operating (S7) the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3').

Description

The invention relates to a method, system and a corresponding computer program for developing, securing, training and/or operating a vehicle system. This is understood in particular to mean the (further) development, securing and/or operating of a development stage of a vehicle system, such as a hardware development stage and/or software development stage, for a vehicle or a vehicle. The development, securing, training and/or operation of a vehicle system can also include operating a corresponding development environment and a corresponding development operation, test operation or security operation for vehicles based on the development environment and/or development stages. Furthermore, the scope of the invention also includes a vehicle system - in particular one comprising artificial intelligence - or a development stage of a vehicle, in particular one that can be driven at least partially automatically, which was or is developed, secured, trained and/or operated according to the features of the invention.

When developing, and in particular when validating, modern vehicles, a very large number of different situations (e.g. driving situations), states of several systems and functionalities of the vehicles, as well as conditions to which vehicles are exposed during their later operation, must be taken into account. In particular, testing and validation of the algorithms or software must be repeated at each so-called integration level.

As a result, the effort, costs and time required for the corresponding development and validation increase significantly or in a non-linear manner as the complexity of the systems and functionalities of modern vehicles continues to increase. This applies in particular to the development, validation and market launch of vehicles that can be driven or parked at least partially automatically.

In particular, the development of automated driving, maneuvering and/or parking requires the consideration of a large number of situations that occur or are possible in reality, in particular driving situations. As the level of automation of at least partially automated driving or parking vehicles increases, the number of relevant driving situations, e.g. arrangements and constellations of road users that must be handled appropriately by the vehicles, increases greatly (also in a highly non-linear manner).

According to the current state of the art and the legal framework for the so-called homologation or market launch of the vehicles, this results in an enormous distance being covered, which would have to be covered in tests and to ensure that such vehicles were mostly error-free or with appropriate performance and freedom from errors. This must be repeated at several integration stages and (to a specified extent) after completion of each development stage.

Therefore, the object of the present invention is to demonstrate an improved method, system and computer program for developing, securing, training and/or operating a vehicle system. In particular, the object of the invention relates to an at least partially automated vehicle (also understood to mean parking and/or maneuvering). Furthermore, the object of the invention can include the use (also understood to mean operating) of a stationary development environment (e.g. at least partially detached from a real or physical vehicle) and/or a backend, in particular a data center. This is also to be understood in particular as the use of the proposed method, system or computer program within the framework of a corresponding development operation, in particular for training or securing a vehicle system using such a development environment.

The object of the invention is achieved by the features of each of the independent patent claims. Advantageous embodiments are described in the dependent claims. It is pointed out that additional features of a patent claim dependent on an independent patent claim can form a separate invention without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim, which invention is independent of the combination of all features of the independent patent claim and can be made the subject of an independent claim, a divisional application or a subsequent application. This applies equally to technical teachings described in the description, which can form an invention independent of the features of the independent patent claims.

A first aspect of the invention relates to a method for developing, securing, training and/or operating a development stage of a vehicle system. The method comprises determining one or more first data sections that represent one or more real situations of one or more real vehicles, (in particular thereafter) training and/or operating at least one neural network depending on the first data sections, and generating second data sections which characterize one or more at least partially virtual situations, depending on the output layer of the neural network. The second data sections or the data determined depending on the second data sections can then be provided for developing, securing, training and/or operating the vehicle system.

The method may further comprise developing, securing, training and/or operating the vehicle system depending on the second data sections provided or the data determined depending on the second data sections.

For example, the one or more first data sections can represent a respective real situation, in particular a driving situation in the environment of the (respective real) vehicle, e.g. in relation to one or more objects in the environment of the real vehicle. The one or more real situations can include or be situations relating to the respective vehicle in the environment of the respective vehicle (e.g. within a radius of 1 - 50 meters).

The one or more real objects can be represented in the one or more first data sections as respective features, position, arrangement, state, speed, acceleration, motion vector, intention, status, a role of one or more objects, e.g. in relation to the real vehicle and/or in relation to another real object.

For example, one or more first data sections can (explicitly or implicitly) identify a role of one or more objects within a driving situation, such as the role of an object as a vehicle in front or behind the real vehicle or another object, a vehicle overtaking the real vehicle, or a vehicle that has the right of way over the real vehicle or must or should give way to the real vehicle.

The real objects can preferably be mobile, movable and/or changeable objects, in particular road users. Alternatively or additionally, immobile and/or permanently occurring or unchanged objects in the real environment can be features of a road section, e.g. road markings, lane markings, zebra streets, traffic signs (e.g. certain traffic signs or traffic signs of a certain class or type), traffic lights, traffic light phases or changeable traffic signs. The first data sections can also represent the state of the objects, such as the phase of the traffic light, a (variable) traffic display, railway crossings, etc.

Alternatively, preferably additionally, the one or more first data sections may represent one or more spatial and/or temporal relationships between the real vehicle and the one or more real objects, and/or between one or more of the real objects to each other (within the situation).

For example, several first data sections can be generated that relate to the same situation, in particular the same driving situation. Several first data sections can each identify different aspects of the situation that correspond, are associated or causally related to one another. These can be understood as a set of first data sections.

The first data sections particularly preferably represent a temporally and/or spatially extended process (e.g. in the time interval of 1 - 60 seconds or 1 - 100 meters). In particular, the first data sections characterize the dynamics of the corresponding real driving situation or the real objects. In particular, the first data sections (e.g. a set of the first data sections) represent a driving maneuver (e.g. lane change maneuver, merging maneuver, evasive maneuver, overtaking maneuver, entry maneuver, exit maneuver, a parking or maneuvering maneuver), and/or driving on a road arrangement with a certain condition, e.g. roundabout, entrance, exit, intersection, etc.

The generation of one or more second data sections that characterize one or more virtual situations can include or be the generation (e.g. simulation) of the virtual situation. The generation of one or more second data sections depending on the output layer of the neural network can be understood as the simulation of the one or more virtual situations by means of the neural network and/or as controlling a simulation of the virtual situations (which can be done e.g. by other means) depending on the output layer of the neural network. In the latter case, the second data sections can characterize or include e.g. simulation parameters and/or situation formation rules.

The at least partially virtual situations can be situations that are completely virtual or based to a certain extent on real or concrete circumstances or include real objects (e.g. in the form of data recorded in real time). For example, an at least partially virtual situation (also as data sections that characterize the situation) can be composed of real parts (e.g. objects that exist in reality, movement and/or changes to the objects) and virtual parts (e.g. virtual or virtualized objects, movement and/or changes to virtual or virtualized objects). The neural network can be trained or can be and/or operated to compose the at least partially virtual situation from real parts and virtual (or virtualized) parts or to modify the data sections to form a real part with data about the virtual part. For the sake of simplicity, the at least partially virtual situations or data about such situations are referred to below as virtual situations.

The second data sections can be provided in a processed or post-processed form. The data dependent on the second sections can include or be reaction data of a vehicle system to the second data sections.

The term “neural network” refers in particular to a deep neural network. Alternatively or additionally, the term also refers to another form of artificial intelligence (in particular comprising a neural network or a comparable structure). In particular, the training and/or operation of the at least one neural network is to be understood as an analogous application of other types of machine learning. For example, the at least one neural network, the training and/or operation of it can include or be an analogous application of one or more of the following or a combination thereof: supervised learning, unsupervised learning, reinforcement learning, multitask learning, classification and regression trees, support vector machines, logical and relational learning, probabilistic graphical models, rule learning, instance based learning, latent representation, bio-inspired approaches.

The term “training” refers in particular to machine learning, non-rule-based learning and/or deep learning or the execution of machine learning, non-rule-based learning and/or deep learning. Training can include or be supervised or unsupervised training. Training the neural network is in particular (also) to be understood as setting up the neural network. Training (or setting up) of at least one neural network can include or be end-to-end learning and/or reinforcement learning. Several, in particular a predominant number, or (essentially) all of the intermediate steps necessary for pursuing one or more goals can be mapped or integrated in the (at least partially trained) neural network.

• - hinsichtlich eines oder mehrerer - insbesondere wählbarer - Merkmalen, real sein könnten; und/oder
• - in der Realität, insbesondere mit einer bestimmten Wahrscheinlichkeit und/oder nach einer (bestimmter bzw. statistischer) Fahrstrecke und/oder Betriebsdauer eines realen Fahrzeugs, zu erwarten sind bzw. auftreten könnten; und/oder
• - zweite Datenabschnitte zu virtuellen Situationen mit einer bestimmte, insbesondere möglichst breiten Verteilung von Parametern (z.B. Parametern der Objekte, Anordnungen, Bewegungen, Veränderungen und/oder Verhalten von Objekten) erzeugen.

The training can be carried out for one or more predetermined and/or method-determinable goals. The goal of the training (in particular also to be understood as the goal of machine learning) can include the generation of second data sections on virtual situations, where the virtual situations:

• - could be real with regard to one or more - in particular selectable - characteristics; and/or
• - are to be expected or could occur in reality, in particular with a certain probability and/or after a (specific or statistical) driving distance and/or operating time of a real vehicle; and/or
• - generate second data sections on virtual situations with a specific, in particular as broad as possible distribution of parameters (e.g. parameters of the objects, arrangements, movements, changes and/or behavior of objects).

The training of the at least one neural network can be carried out, in particular in several phases, over a long period of time or over several executions or operating cycles of the method. The training of the at least one neural network can be carried out at least partially, in particular only or predominantly automatically, during the operation of a neural network that has already been at least partially trained or in a separate process. The method can comprise a change, in particular a long-term, slow update covering a large number of operating cycles, of the mathematical relationship or of the at least one neural network depending on a large number of the first data sections.

The term “operating” can be understood as using the at least one neural network, in particular in active operation (eg for generating the second data sections). The operation of the at least one neuro The operation of the neural network may alternatively or additionally comprise or be the operation of the corresponding use of the second data sections by means of the neural network. The at least one neural network can be controlled in its operation (at its input layer) and/or control a functionality in which the second data sections are used. The neural network can be controlled (possibly independently of how it was trained) depending on the first data sections and/or on the third data sections described below.

In particular, the operation is carried out with a neural network that has been trained to at least a certain degree. For example, the operation is carried out with the data (e.g. learning data, a checkpoint) of a neural network that has been trained (e.g. on the basis of past time intervals or data sections from the past). Depending on the variant of the method, the method therefore includes or is the training of the neural network (in which case it is to be understood as "the neural network is being trained") and/or the operation of the neural network that has already been at least partially trained (in which case it is to be understood as "the neural network is trained").

The input layer of the neural network can be controlled (when operating) depending on the first data chunks and/or depending on the third data chunks (described below). In other words, the neural network can be operated (also regardless of how it was trained) depending on first data chunks and/or third data chunks (e.g. the third data chunks described later in the document).

The generation of second data sections which characterize one or more at least partially virtual situations occurs depending on the output layer (e.g. depending on one or more outputs) of the neural network. In particular, it is also to be understood that the neural network is trained to generate the second data sections described in this document or the data underlying them and to output them by means of its output layer.

For example, the at least one neural network can be trained and/or operated to reduce a deviation (e.g. a measure of the deviation) from a target or target value (e.g. related to second data sections, data dependent on the second data sections or reaction data). For example, the at least one neural network can be trained and/or operated to reduce a (e.g. possible, probable, predicted and/or statistical) deviation from a (e.g. predetermined) target or target value.

• - ein Ermitteln von Trainingsdaten, insbesondere auf Basis der ersten Datenabschnitte und/oder der zuvor erzeugten zweiten Datenabschnitte und/oder weiterer Daten; und/oder
• - ein Ermitteln der Backpropagation; und/oder
• - eine Auswahl oder ein Anpassen eines oder mehrerer Parameter des Trainiervorgangs.

Training (especially to be understood as the process of training) can include or be:

• - determining training data, in particular on the basis of the first data sections and/or the previously generated second data sections and/or further data; and/or
• - determining the backpropagation; and/or
• - a selection or adjustment of one or more parameters of the training process.

One or more aspects of training described above may be determined or adjusted depending on the initial data segments.

Alternatively or additionally, one or more aspects of the training described above may be determined or adapted depending on the second data sections (e.g. depending on the second data sections generated as an intermediate result) and/or on the data determined depending on the second data sections.

The training of the at least one neural network (first or second described below) can be carried out at least partially, in particular only or predominantly automatically, during the operation of a neural network that has already been at least partially trained or in a separate process. For example, the invention includes a corresponding method for training the neural network and/or a method in which the trained neural network is operated. The operation includes, for example, active use, wherein the neural network is used according to a feature of the invention. The method can include one or both of these methods, which can be carried out, for example, simultaneously, one after the other or alternately.

Preferably, a significant part of the processing of the data described in the present document and/or the achievement of a significant effect of the method can be carried out by means of the at least one neural network. This does not exclude the use or co-use of other means, e.g. computer-implemented formulas, rule-based algorithms, etc. and/or human involvement, e.g. as an intervention in the processing and/or influencing of one or more steps of the method. For example, at least part of one or more method steps can be carried out using other means. These can e.g. for preprocessing and/or postprocessing of the data for training and/or operating the neural network.

For example, the one or more first data sections can represent a respective real situation, in particular a driving situation in the environment of the (respective real) vehicle, in particular with respect to one or more objects in the environment of the real vehicle.

The real objects can preferably be moving, movable and/or changeable objects, in particular road users.

It should be noted that the first data sections do not have to include a complete description of the situation in the surroundings of the real vehicle (e.g. in the form of a photorealistic image). Rather, selected or selectable aspects of the situation can advantageously be represented, e.g. in a correspondingly defined data format. For example, several first data sections can be generated that relate to the same situation, in particular the same driving situation. These can be understood as a set of the first data sections. The first data sections particularly preferably represent a temporally and/or spatially extended process. In particular, the first data sections (e.g. a set of the first data sections) represent a driving maneuver (e.g. lane change maneuver, merging maneuver, evasive maneuver, overtaking maneuver, entry maneuver, exit maneuver), a parking or maneuvering maneuver, and/or driving on a roadway arrangement with a certain condition, e.g. roundabout, entrance, exit, intersection, etc.

The second data sections can be generated depending on the output layer of the neural network. This is to be understood in particular to mean that the second data sections (with or without additional data processing) are generated depending on the neural network (e.g. depending on the operation of the neural network that has been trained completely or at least to a certain extent, in particular one or more outputs of the neural network).

Operating the neural network is or includes in particular controlling the neural network, in particular the input layer (one or more inputs) of the neural network. Furthermore, changing the operating mode or the operating settings of the neural network can also be counted as operating.

In particular, the second data sections identify the properties, configuration and/or change of one or more virtual situations. In particular, one or more virtual situations (e.g. any parameters and/or aspects of the situations) can be generated or changed depending on the neural network. For example, the second data sections can identify (specify) a rule, e.g. a simulation parameter and/or situation creation rule, according to which the (concrete) virtual situations, virtual objects, their respective movement, change and/or behavior are generated (e.g. simulated).

Particularly preferably, the second data sections (also) identify the virtual objects involved in the virtual situation and/or a virtual (ego) vehicle (e.g. involved in the situation or affected by the situation). The second data sections can identify their respective arrangement, movement, change and/or behavior, in particular in relation to one another. This can relate to specific objects in an explicit or implicit form or in relation to the (e.g. statistical) relationships between respective parameters (e.g. within the context of the situation).

The at least one virtual object can be a virtual moving object and/or an object with a virtual behavior. The one or more second data sections can represent the movement and/or behavior of the one or more virtual objects in relation to the virtual vehicle and/or in relation to the real environment of the real vehicle. The movement and/or behavior of the virtual object can be a virtual movement and/or a virtual behavior (of the virtual object) in the context of this document.

The movement of the virtual object can be represented and/or carried out virtually within the partially real and partially virtual environment or driving situation generated in the method. For example, the movement of the virtual object is represented in (e.g. global and/or local) coordinates or in relation to coordinates in which real objects are also represented. Preferably, the movement and/or behavior of the virtual object in relation to a virtual vehicle is represented in an at least partially virtual environment and/or in relation to the real environment of the real vehicle.

Alternatively or additionally, the second data sections may comprise all or any selection of the features relating to the virtual situation that the first data sections in Reference to one or more real situations.

In the method, second data sections for virtual situations can be generated in such a way that the (at least most) virtual situations differ from the concrete real situations identified in particular in the first data sections. These can include some of the real situations. However, they can (advantageously) have features that are to be expected in reality (e.g. with a certain probability, after a certain operating period and/or distance). This can be very advantageous for different applications for developing, securing, training and/or operating a vehicle system. Various variants of the invention can further increase the amount, type and/or diversity of the virtual situations and the trends depicted by them.

In an example that can be combined with all other features described in this document, a target relating to the second data sections or the corresponding virtual situations and/or a desired difference to the one or more (possibly specific) real situations can be specified.

For example, the neural network can be trained and/or operated (e.g. controlled during operation) to generate the second data sections or the virtual situations according to a specific, in particular predeterminable, goal and/or difference to one or more real situations (e.g. to a statistical value that relates to real situations). In this case, a specific goal and/or a specific difference to one or more (selectable) real situations can be aimed for with regard to the resulting second data sections or the generated virtual situation.

For example, the method can include the parameters of the virtual situations and/or virtual objects (to be generated) having a specific (e.g. predeterminable or adjustable) difference to the parameters of the one or more real situations or real objects. Such a difference can relate to concrete parameters (e.g. concerning the situation, certain virtual objects, the virtual ego vehicle, movement, change and/or the behavior of these) and/or to the respective statistical relationships and/or trends (e.g. concerning situations, virtual objects, the virtual vehicle, movement, change and/or the behavior of these). The goal and/or the difference can be quantitative and/or qualitative or can be quantitatively and/or qualitatively predeterminable.

The goal can be specified, for example, as a quantitative and/or qualitative target parameter characterizing the characteristics of second data sections or virtual situations to be generated (e.g. during training and/or operation). For example, a certain number of virtual objects of certain types (e.g. virtual pedestrians, virtual vehicles, virtual animals) should occur in the virtual environment (possibly in contrast to the real situations) or possibly appear on the route of the virtual vehicle. For example, the goal can be a specific, in particular as broad as possible distribution of parameters (e.g. parameters of the objects, arrangements, movements, changes and/or behavior of objects) within a virtual situation to be generated and/or between several situations to be generated.

The difference, for example, can be specified as a quantitative and/or qualitative difference (e.g. factor, difference, mathematical formula and/or a statistical value) of the characteristics of the second data sections or virtual situation to be generated compared to the one or more real situations.

For example, the neural network can be trained and/or operated on the basis of initial data sections from a large number of real situations with only a certain small proportion of critical situations (e.g. emergency braking, drastic evasive maneuvers, near misses, accidents) in order to generate virtual situations with a much larger proportion of critical virtual situations. Data on such situations are particularly well suited to achieving the goals of developing, safeguarding, training and/or operating a vehicle system. Alternatively or additionally, the number and/or speed of the crossing virtual vehicles can be increased by 10, 20, 50, 200% and/or instead of crossing vehicles (which occurred in one or more of the real situations), virtual objects of a different type (e.g. virtual giraffes, kangaroos or elephants) can appear.

The target and/or the difference can include a statistical measure (e.g. statistical frequency and/or statistical distribution) and/or a probability measure (e.g. frequency of occurrence and/or probability distribution) or can be specified in the form of such measures. The generation of the second data sections or the corresponding virtual situations can be influenced according to the specification.

For example, using the neural network trained with the first data sections (in particular, this also means data from one or more real situations), second data sections can be generated that relate to a difference in the arrangement of the objects relative to one another or to the virtual ego vehicle (angle and/or distance between them, a (relative) speed, etc.). In this case, virtual situations corresponding to certain specifications or goals (e.g. with regard to statistical measures) can be generated.

These virtual situations can (nevertheless or despite the desired or specified goal or difference) sufficiently simulate situations that are relevant to reality or as close to reality as possible. For example, second data sections can identify critical virtual situations (also to be understood as situations that can be critical or dangerous in reality).

One advantage of the method can be that data on new virtual situations can be generated very effectively. The amount of second data sections generated (or the corresponding virtual situations) can be significantly (e.g. by one or more orders of magnitude) larger than the amount of first data sections used (or the corresponding real situations). For example, second data sections on virtual situations can be generated quickly, effectively, in a cost-saving and/or energy-saving manner, which in reality would have required billions of kilometers of driving or several decades. This can be particularly advantageous for the development, validation, training and/or operation of a vehicle system, in particular one relating to at least partially automated driving and/or parking.

A further advantageous embodiment of the invention comprises using the second data sections to train the vehicle system, in particular a (further, different) neural network of the vehicle system. For example, a (further, second) neural network for operating the vehicle system that carries out at least partially automated driving and/or parking can be trained using one or more of the virtual situations (generated or predetermined in this way). It should be noted that this can in particular be a different (second) neural network that does not match or differs from the (first) neural network with which the second data sections are generated. The invention therefore includes the proposal to set up an artificial intelligence, in particular to train it, in order to use it to train another artificial intelligence (the one intended for use in the real vehicle).

The at least one virtual object can be a virtually movable object and/or an object with a virtual behavior. The movement and/or behavior of the virtual object can be a virtual movement and/or a virtual behavior (of the virtual object) within the scope of this document.

Preferably, the movement and/or the behavior can have parameters that are determined and/or changed at least partially according to a certain dependency (e.g. according to a certain mathematical relationship, e.g. logic, formula, function, pattern) and/or at least partially according to a random principle. For example, the virtual object can execute a movement and/or a maneuver according to a certain dependency, e.g. according to a pattern, and/or with parameters that adjust over time according to a specification, criterion, chance, a mathematical relationship.

A maneuver can be or include, for example, a driving maneuver, a parking maneuver or a shunting maneuver. A maneuver can be characterized, for example, by a certain pattern (e.g. a data pattern that represents certain characteristics of the maneuver, proportions between its parameters, etc.). Several virtual objects that perform a certain maneuver do not necessarily have to have the same parameters. Rather, the parameters that characterize its movement, behavior, etc. can be determined according to a pattern. Rather, the parameters can be determined or adjusted depending on parameters of the environment and/or a change in the parameters of the environment. In other words, the virtual object - almost like an independently acting virtual being - can be placed in the environment of the virtual vehicle and, if necessary, enabled for further movement or activity (so to speak "released into the wild"). In this case, the at least one virtual object can be enabled or activated to perform an activity that is at least partially controlled (from outside the object) and/or at least partially self-controlled (quasi self-determined) (within the object), e.g. a movement, action, interaction.

A pattern and/or one or more parameters of the movement and/or behavior of the virtual object can be varied for different virtual objects and/or over time for one virtual object. The variation can be at least partially according to a predetermined dependency (e.g. according to a function) and/or at least partly by chance.

In a simplified case, a virtual object can (also) be created by selecting, retrieving and/or playing (e.g. at least partially pre-stored, possibly pre-recorded) data representing the virtual object (e.g. the corresponding data sections from a database). These can, for example, represent one or more features of the nature of the virtual object, its movement and/or its behavior.

The data sections representing the one or more virtual objects can include images (e.g. camera data), thermal images, image sequences, point clouds, so-called radar objects and/or LIDAR objects or can be modeled on them that a real vehicle would also capture when encountering a corresponding real object, or can be of a similar nature. In particular, the data sections representing virtual objects can have approximately the same or corresponding features, format and quality as images (e.g. camera data), thermal images, image sequences, point clouds, so-called radar objects and/or LIDAR objects that a real vehicle captures (e.g. with its sensors). The data sections can simulate or model the data of a real sensor of the vehicle when a real object similar to or corresponding to the virtual object occurs.

A virtual object can be modeled on a specific real object, e.g. a road user of a certain class, such as a pedestrian, cyclist, motorcyclist, scooter rider, child, animal (e.g. cat, dog, horse, sheep, cow, ostrich, kangaroo, wolpertinger, etc.), with regard to a property or can simulate the property, in particular dynamically.

A virtual object can have a specific or associated behavior and/or a certain ability to react to other (real and/or virtual) objects.

For example, the virtual object can be designed to move, behave, act and/or react in a certain way. All movements, actions and/or reactions of the object can be predetermined (or programmed). A property and/or a certain behavior of the object (e.g. in relation to its environment, other real objects and/or the vehicle) can be specified. This can be done without (having to) specify or foresee specific movements or actions.

The movements, actions and/or reactions of the object can be determined at least partially depending on a parameter of the environment, a parameter of a real object in its environment, and/or at least partially depending on a random variable.

For example, a virtual object can move, act and/or react in a way that an average (e.g. typical) object corresponding to a certain class and/or an object of an average shape (e.g. for its class) with an average (e.g. typical or statistically normal) behavior or behavior that deviates from an average behavior (e.g. by a deviation value or tolerance value) would do.

For example, a virtual object that corresponds to a virtual animal, e.g. a cat, can behave (e.g. as a corresponding simulation or the data sections) in a manner similar to a virtual animal, e.g. a cat, and react to or interact with the real environment, including other road users. The virtual cat can, for example, be "let out for a walk" and in particular enter the (virtual) environment of the vehicle or play a role in the driving simulation. A virtual animal can also, for example, slip and/or suffer a (virtual) death, e.g. if a road friction coefficient as a parameter of the environment indicates that the road or a sidewalk is currently slippery.

Alternatively or additionally, a virtual object can become valid (i.e. appear or be activated in the form of corresponding data) when a situation, e.g. a situation characterized by a certain pattern of parameters, such as a driving situation, is recognized. The method can thus generate or make usable conditions, situations, states and the corresponding reaction data that are particularly relevant to practice and/or are representative of practice and/or desired special cases to a certain, desired (even high) degree. These can be largely independent of a test of the vehicle system (otherwise typically planned or simulated by humans). The influence of possible human errors on the final product, e.g. on the version of the vehicle system intended for delivery, can therefore be significantly reduced.

• - seine aktuelle Eigenschaft und/oder sein aktuelles Verhalten;
• - seine Position, seine Abmessungen, und/oder seine Bewegung;
• - seine Wahrnehmung (z.B. für einen von dem Objekt einsehbarer Raumbereich und/oder für das, was das Objekt aus seiner Perspektive wahrnehmen kann);
• - eine Aktion und/oder Einwirkung des virtuellen Objekts gegenüber dem Fahrzeug bzw. auf das Fahrzeug und/oder gegenüber einem weiteren realen Objekt bzw. auf das weitere reale Objekt;
• - eine Aktion des Fahrzeugs und/oder eines weiteren realen Objekts mit Bezug auf das virtuelle Objekt;
• - eine Einwirkung eines weiteren realen Objekts und/oder des Fahrzeugs auf das virtuelle Objekt.

The one or more parameters of the one or more virtual objects can, for example, be representative of one or more entries from the following list:

• - its current characteristics and/or behaviour;
• - its position, its dimensions, and/or its movement;
• - its perception (e.g. for a spatial area visible to the object and/or for what the object can perceive from its perspective);
• - an action and/or influence of the virtual object towards the vehicle or on the vehicle and/or towards another real object or on the other real object;
• - an action of the vehicle and/or another real object with respect to the virtual object;
• - an influence of another real object and/or the vehicle on the virtual object.

The one or more virtual objects (in themselves) can approximately correspond to the measured or recorded real objects. These can be moved or changed depending on the output layer of the neural network (within the scope of the virtual situation) and/or change their behavior. These can form part of the virtual situation.

Preferably, the neural network can be trained and/or operated to generate new virtual objects (e.g., objects that differ from real objects). These can be objects (analogous to the virtual situations) that are (nevertheless) sufficiently relevant to reality or realistic objects that, for example, only rarely occur in road traffic or have never been detected by sensors, but could nevertheless occur (e.g., very rarely). These can alternatively or additionally form a (further) part of the virtual situation.

For example, the neural network can be trained and/or operated to generate virtual objects (e.g. new virtual road users), in particular depending on a predefined target and/or difference to real objects. Parameters of the shape, movement and/or behavior of the virtual objects (depending on the output layer of the neural network or in the form of the second data sections) can be generated.

For example, the neural network can be trained depending on a large number of real objects (represented by their parameters in the first data sections), e.g. parameters of the shape, movement and/or behavior of the real objects (e.g. captured by real vehicles). A goal of training and/or (later) operating the neural network can be specified. For example, the goal can include or be the creation of rarely occurring objects, strangely moving objects and/or objects with rarely occurring behavior. Alternatively or additionally, a desired degree of difference from typical, statistical objects, for example with regard to one or more selected parameters, can be specified.

Preferably, the training and/or operation of the neural network is carried out with the aim of generating new virtual objects that can correspond to the features that occur in reality and/or at least do not contradict them. The one or more features can be abstracted from a concrete object or from a real object or an object that can be observed or observed in reality. For example, objects that are much larger or smaller than average, fast or slow and/or objects with behavior that deviates significantly from the average (which are rare in reality but can still occur) can be generated. The development, validation, training and/or operation of the vehicle system can also take such objects or corresponding special cases into account.

Such virtual objects (also independently of other features contained in this document) can also be used according to the invention for other purposes and in a different context. Therefore, virtual objects or features described in connection with virtual objects can be considered as independent aspects of the invention.

In one example of the method, the (first) data sections are determined (e.g. recorded and/or processed) in a real driving operation of one (preferably several) real vehicles, which are representative of a real situation in the environment of the real vehicle, e.g. real objects, their arrangement, movement, a corresponding driving situation, etc. Preferably, these are stored (e.g. after an optional, in particular short, intermediate storage, e.g. in a storage structure set up for this purpose, e.g. in a ring buffer in the vehicle) and transmitted to the backend (also to be understood as a stationary computing unit, a server, a data center, a stationary system).

The first data sections can preferably be selected or recorded depending on a trigger condition (also to be understood as: depending on the application of the trigger condition) while the (respective) vehicle is in operation. For example, initial data sections are determined which represent a change or development of a real situation in the environment of the vehicle during a time interval of e.g. 1, 5, 8, 12, 30 or 60 seconds.

For example, the respective real environments of the respective (real) vehicle, e.g. comprising one or more real objects, arrangement formed by the objects and/or driving situation, can be determined (also: recorded) on one or more road sections and/or in an operating state of a system of the vehicle if a road section, a driving situation and/or an operating state of a system of the vehicle fulfill a trigger condition, in particular a specific, predetermined and/or dynamically (e.g. according to a certain dependency) changing trigger condition, in approximately the same time interval, e.g. shortly before, currently or in the near future of a few seconds. The trigger condition can, e.g. (also) refer to a parameter of the environment, object, road section and/or the operating state of the vehicle. A trigger condition can comprise several conditions relating to different information (e.g. parameters, occurrences, states, etc.).

For example, the at least one trigger condition relates to the detection of a driving situation in the presence of a road layout (e.g. arrangement, angle, radius, inclination, etc. of roads, lanes, entrances, exits, roundabouts) and/or traffic rules (e.g. traffic signs, traffic lights, traffic light phases, displays, traffic information) corresponding to certain parameters (e.g. parameter value ranges) and/or corresponding to certain patterns (e.g. sufficiently similar or dissimilar to one or more specific patterns).

The at least one trigger condition (and the corresponding first data sections) may also include geo-positions or an environment or roadway sections delimited by certain geo-positions that correspond to one or more specific features, parameters and/or patterns (e.g. urban environment, country road, expressway, etc., e.g. in a specific country or province, an area of a certain type).

The one or more of the first data sections can include or be a selected driving situation when a certain road layout is present, e.g. an overtaking maneuver in an S-curve, railroad crossing, a certain arrangement of real objects in a roundabout and/or with certain traffic light circuits. Preferably, the first data sections can also include (explicitly and/or implicitly) information about compliance with one or more traffic regulations by the real vehicle and/or one or more real objects. In other words, by means of the trigger condition and/or processing of the first data sections in the backend, one or more data sections (from an overall very high mileage or operating time of a large number of vehicles) can be determined predominantly or essentially only first data sections for which the reaction data is desired or required (e.g. still missing for security).

For example, a trigger condition may apply (and the first data sections may be captured, recorded and/or read out from a ring buffer) if a driving situation characterized by a parameter (e.g. parameter value range) or by a pattern (e.g. by sufficient similarity to a certain pattern) is recognized or predicted.

Alternatively or additionally, a trigger condition (and the first data sections are captured, recorded and/or read out from a ring buffer) can apply if a feature, parameter and/or pattern relating to an operating state or data of a (further) system of the vehicle, and/or boundary conditions in the environment of the real vehicle (e.g. weather, time of day, road condition, etc.) apply.

Alternatively or additionally, the trigger condition can also be applied after the first data sections have been determined. This can be applied to the data collected (e.g. from a large number of vehicles) (in retrospect, for example in the backend). The trigger condition can be used to select the first data sections from larger data, e.g. data representing a longer driving distance. The first data sections can be determined, in particular selected, according to one or more of the principles or criteria described.

The one or more first data sections can be determined or processed with or, preferably, without reference to specific vehicles, their respective absolute coordinates or times.

Preferably, the training and/or operation of the at least one (first) neural network and/or the generation of second data sections is carried out at least partially, at least predominantly or entirely in a backend. This is particularly advantageous because corresponding resources (e.g. computing power, memory, data access, etc.) can be provided in the backend.

This can involve operating the neural network (which may be complex and computationally intensive) or generating second data sections. In particular, it can be a backend that is set up for quasi-simultaneous (e.g. simultaneous, parallelized or parallel) generation or further processing of a large number of second data sections.

In particular, the backend is equipped with resources that exceed (e.g. corresponding or next similar type of resources) a vehicle system that was common at about the same time or of the same development status by 10 ² , 10 ^{3 ,} 10 ^{4 ,} 10 ^{5 ,} 10 ^{6 ,} 10 ⁸ . The backend also belongs to the invention or can be considered as an aspect of the invention or a part of the system according to the invention.

In the context of this document, the term "backend" can be understood to mean a server, cloud, data center (each with a centralized or decentralized architecture). Furthermore, the backend can include or be a development environment.

For example, the (first) neural network can be used to generate second data sections (also to be understood as sets of the second data sections) that represent quantitatively and/or qualitatively different environments of the vehicle or different driving situations. In the context of this document, the term "environment" of the vehicle is to be understood in particular independently of a real position, e.g. real geo-position or the corresponding location. Also, not all features, objects or conditions of the environment have to be taken into account in the process or, for example, represented by the first data sections. Rather, a selective and/or, in particular dynamic, changing selection of real objects, parameters and/or data about the environment can be processed.

Data sections that represent the one or more virtual objects (e.g. as the corresponding real object that corresponds to the virtual object and is simulated by it) can be or include sensor data and/or a data model, in particular based on a geometric model (e.g. in 2D or 3D), a movement model, a behavior model, or 3D+Time of the virtual object. This step is preferably provided in the backend. On the one hand, this means that many more computing resources are available for this or further steps than would be possible in a vehicle (which may make the method possible in the first place), and on the other hand, different variants of the second data sections can be generated (also simultaneously or quasi-simultaneously).

Furthermore, reaction data of the vehicle system, in particular of a development status, replica and/or an emulation of the vehicle system, can be generated by using the second data sections as an input for the vehicle system. The vehicle system, a replica operated in a vehicle or stationary or the development status can be or are operated on the basis of the second data sections.

Furthermore, the reaction data of the development stage can include data of the vehicle or of an emulation of the vehicle or of a part of the vehicle (e.g. with which the first development stage is operated) that have been changed under the influence of the development stage. For example, these can be changes to states or parameters of the vehicle or of an emulation of the vehicle or of a part of the vehicle that are conditioned or caused by the reaction of the (first) development stage to the second data sections.

In particular, a first (or possibly a second) development stage of the vehicle system can be operated as a replica or emulation of the real vehicle system in the backend.

The reaction data can identify an output of the vehicle system and/or a parameter, state or a change in state of the corresponding system of the vehicle. For example, the reaction data can identify one or more internal parameters, states, changes in state (e.g. of a state machine, control, regulation, formula, etc.) (e.g. visible at the output or immediately effective in regular or subsequent operation of the corresponding vehicle system).

• - zu einer Betätigung eines Aktors des Fahrzeugs und/oder einer Ausgabe einer Fahrerinformation dienen; und/oder
• - eine (insbesondere geplante) Trajektorie (alternativ oder zusätzlich zu verstehen als eine Bahnplanung) des Fahrzeugs, insbesondere eine aufgrund des virtuellen Objekts veränderte Trajektorie bzw. Bahnplanung, beschreiben;
• - zum Ansteuern eines weiteren Systems des Fahrzeugs dienen, z.B. Steuerungssignale; und/oder
• - an das Bordnetz des Fahrzeugs und/oder nach außerhalb des Fahrzeugs versendet werden; und/oder
• - einen Ressourcenverbrauch des Fahrzeugsystems (z.B. betreffend Rechenleistung, Speicher, Laufzeit, etc.) beschreiben; und/oder
• - einen Zustand, Zustandsübergang, Parameter und/oder Speicherinhalt beschreiben.

Alternatively or additionally, the reaction data may correspond to or identify data that are used in a real vehicle system (which is represented, for example, by the development status, replica and/or emulation or is developed, secured, tested or trained using the development status):

• - serve to operate an actuator of the vehicle and/or to output driver information; and/or
• - a (particularly planned) trajectory (alternatively or additionally to be understood as a path planning) of the vehicle, in particular describe a trajectory or path planning that has been changed due to the virtual object;
• - serve to control another system of the vehicle, e.g. control signals; and/or
• - sent to the vehicle's electrical system and/or outside the vehicle; and/or
• - describe the resource consumption of the vehicle system (e.g. regarding computing power, memory, runtime, etc.); and/or
• - describe a state, state transition, parameter and/or memory content.

Alternatively or additionally, the reaction data (also to be understood as data based on the reaction data) can be determined, which characterize a measure of a consequence of a reaction of the vehicle system to the second data sections. In this case, data can be determined and taken into account that characterize a consequence of an event, in particular a changed trajectory of the vehicle, a new or changed collision, which is determined depending on the one or more virtual objects (e.g. occurs virtually and would therefore also occur in reality). For example, reaction data of a vehicle (e.g. automated driving) can be determined, which characterize a consequence that would occur if a child were to run onto the road in a certain driving situation. The invention makes it possible to operate one or more development stages with a large number of variants of such a partially real and partially virtual situation (generated by means of a virtual child and without a risk to the public) in the environment represented by the second data sections (for testing purposes and/or to train a neural network).

The reaction data can describe a new reaction of the vehicle system or a difference in the reaction (e.g. in the form of a difference, a factor, a measure of a deviation or a function of the deviation), e.g. in comparison to a reaction that would apply without the presence of the one or more virtual objects or without the modification of the first data sections. For example, this can relate to a trajectory or path planning of the vehicle that has changed due to the virtual object or alternatively or additionally a difference to the reaction that applies without modifying the data sections. The latter reaction can correspond to the reaction of the real system of the real vehicle (which can be determined in addition to the first data sections) or be a reaction of the vehicle system, e.g. of a development stage that is operated with first data sections.

For example, data can be determined on the basis of the reaction data that characterize a comparison of the reaction data of the development status to (unmodified) first data sections and the reaction data of the development status to second data sections and/or several variants of the second data sections, which were generated, for example, by different virtual objects or different modifications.

Furthermore, on the basis of the reaction data, data can be determined which are characteristic of a sensitivity of the reaction data to a change in a parameter of the one or more virtual objects.

A variety of, in particular different, development stages can be used. For example, a variety of development stages can be used. These can, for example, include a variety of electronic modules or interact with them.

The reaction data can indicate a conclusion about a reaction (very valuable for developing, securing, training and/or operating a vehicle system) of a real vehicle (with a real system that has the characteristics of the development stage) to a real environment, condition or situation that is supplemented, modified or enriched with the virtual object (or with a virtually supplemented or modified part of the environment or the driving situation).

Therefore, the method may include, in particular in an automated manner, determining a conclusion about the behavior and/or performance of a real system of the vehicle in a real environment, under real conditions, in a real situation, in particular a driving situation, which corresponds to the modified environment of the vehicle (for example according to the second data sections).

Furthermore, the method can comprise various processing and/or uses of the reaction data, in particular the conclusions based on the reaction data for developing, securing, training and/or operating a development status, e.g. in real vehicles.

For example, one or more parameters of a (corresponding to the first or second development stage) vehicle system are updated depending on the reaction data.

Each parameter, each feature and each step of the method described in this document can be selected depending on the degree of automation of the (possibly real or simulated) driving operation of the (real or virtual) vehicle. For example, the determination of the first data sections and/or the generation of the reaction data takes place depending on the second data sections and/or depending on the (real or virtual) degree of automation of the respective vehicle.

The term “vehicle system” can be understood as a system suitable or intended for use in vehicles (e.g. with a certain functionality), or a sufficiently similar system (with regard to certain features), an alternative, further developed system and/or a system to be trained, or a system used as a representative of such a vehicle system.

The term "development status" is to be understood as a development status of a system of the vehicle, e.g. comprising a version of software and/or hardware of a system of the vehicle or of the vehicle. In particular, the development status can include so-called middleware and/or one or more neural networks or a checkpoint of a neural network. Furthermore, a corresponding status of the vehicle system, a status at the time of delivery of the vehicle or an update or upgrade or a further development status can also be understood as the development status within the scope of this document. For example, the term development status can also be understood as a developed, secured or trained development status for a system of the vehicle (e.g. final, intended for delivery or update to vehicles).

The development can in particular include or be further development. For example, depending on the second data sections, the reaction data and/or analysis data of a vehicle system are generated in order to develop the vehicle system (e.g. in the form of its next version, variant and/or generation). The method can include an analogous use of so-called data-driven development (e.g. as further, additional or subsequent steps). Securing the vehicle system can include testing the vehicle system (e.g. carrying out various tests known to the person skilled in the art) depending on the second data sections generated or depending on data determined from the second data sections.

In the context of this document, the term "characterize" is to be understood in particular (also) as "represent" or "characterize". For example, one or more of the first data sections and/or second data sections can represent absolute or relative coordinates, movements and/or changes of the (real or virtual) vehicle and/or of (real or virtual) objects from the respective environment (more or less explicitly). Depending on the variant of the invention, statistical values can also be used.

In the context of this document, the term “driving” can alternatively or additionally be understood to mean (e.g. automated) parking or maneuvering.

In the context of this document, “real” is understood to mean physically, particularly mechanically, effective conditions, e.g. objects, object features, etc.

In the context of this document, “virtual” is understood to mean circumstances based on data or represented or expressed by data, which in particular do not entail any corresponding direct physical or mechanically effective circumstances or events, e.g. the actuation of an actuator.

In this document, the term "reaction" is to be understood as an alternative or additional term for an "interaction". For example, reaction data (alternatively or additionally to a reaction) can identify or represent an interaction, in particular an interaction of a development stage with one or more virtual objects.

The term "one or more first data sections or one or more second data sections" is to be understood in particular as a set of the (first or second) data sections that relate to approximately the same surroundings of the vehicle, approximately the same driving situation, approximately the same time interval and/or approximately the same objects or arrangements of objects, driving on a section of road such as road users. For example, a set of the first data sections and/or the second data sections relates to a maneuver of the vehicle and/or a road user in the surroundings, and/or to an approximately corresponding time interval (in practice, for example, 5, 20, 40, or 60 seconds).

For example, one or more first data sections are determined, each of which represents at least a part of a real environment or real driving situation of a real vehicle, in particular depicts it or characterizes it qualitatively and/or quantitatively. The part of the real environment or driving situation can be expedient, adaptive, controllable or selectable according to a predetermined condition.

The at least one first data section can comprise data based on one or more sensors of the respective vehicle. For example, such a data section can be determined based on a recording (also to be understood as a recording of corresponding signal traces) of the data based on one or more sensors of the vehicle.

• - zumindest einen Raumteil, der insbesondere nach bestimmten (z.B. dynamischen) Kriterien gewählt wird, wie z.B. in Bezug auf das jeweilige reale Fahrzeug (z.B. aus Blind-Spot-Bereichen);
• - zumindest ein reales Objekt in der Umgebung des realen Fahrzeugs bzw. einen Teil der Umgebung des realen Fahrzeugs mit diesen Objekten;
• - eine bestimmte Situation, z.B. Fahrsituation in der sich das reale Fahrzeug befindet;
• - eine oder mehrere Anordnungen, Aktionen, Interaktionen zwischen einem Objekt in der Umgebung und dem realen Fahrzeug und/oder den Objekten in der Umgebung; und/oder
• - Daten, insbesondere Eingangsdatendaten, kennzeichnend den (mit der Umgebung, Fahrsituation, etc. korrespondierenden) Zustand und/oder Ausgangsdaten eines Systems des realen Fahrzeugs.

The first data sections can represent, for example:

• - at least one part of the space, which is selected in particular according to certain (e.g. dynamic) criteria, such as in relation to the respective real vehicle (e.g. from blind spot areas);
• - at least one real object in the environment of the real vehicle or a part of the environment of the real vehicle with these objects;
• - a specific situation, e.g. driving situation in which the real vehicle is located;
• - one or more arrangements, actions, interactions between an object in the environment and the real vehicle and/or the objects in the environment; and/or
• - Data, in particular input data, characterizing the state (corresponding to the environment, driving situation, etc.) and/or output data of a system of the real vehicle.

Preferably, the at least one first data section represents a dynamic process and/or a temporally and/or spatially extended process. The process can relate to almost all, or certain, individual (e.g. selectable) real objects within the environment of the vehicle. One or more of the first data sections can represent a driving situation, in particular a maneuver (driving maneuver, parking maneuver), e.g. relating to the real vehicle and/or one or more road users in the environment. For example, real objects can be represented depending on their role within a driving situation, for example as a vehicle in front, a vehicle behind, a vehicle overtaking the real vehicle, a road user who has the right of way over the real vehicle or who must give way.

For example, one or more of the first data sections can represent a driving situation occurring in the environment of the real vehicle, in particular a maneuver of the vehicle, in particular in relation to an object in the environment, and/or a maneuver of a road user in the environment of the vehicle, in particular in relation to the vehicle. These can be specific maneuvers, maneuvers of certain types and/or maneuvers corresponding to certain criteria and/or maneuvers in which the real vehicle plays a certain role. The first data sections can only, predominantly or essentially only refer to or describe relevant objects, e.g. those involved in a maneuver.

A maneuver can be, for example, a lane change maneuver, a merging maneuver, an evasive maneuver, a merging maneuver, an overtaking maneuver, a parking maneuver, a parking exit maneuver, a shunting maneuver or the like.

Alternatively or additionally, the first data sections can represent a specific process carried out by the real vehicle. This can be a dynamic process and/or a process that is extended in time and/or space. For example, the first data sections represent a specific arrangement, process and/or interaction of the real vehicle with an object. In another example, such an interaction can include a mutual disturbance, influence, impact or, exceptionally, a physical interaction, e.g. an obstruction, friction or collision.

The first data sections and/or second data sections can (also only) identify parameters, parameter value ranges, classes and/or patterns of the movements, changes or behavior of the vehicle or the objects (e.g. in the form of corresponding characters or codes). Such data can be comparatively compact in size and can be processed efficiently. These can preferably, but do not have to, include a complete, graphic and/or photorealistic representation of a part of the (real or virtual) environment.

For example, a part of the real environment, the real driving situation, one or more real objects, their actions, parameters, behavior can be recognized, e.g. by means of corresponding patterns (preferably recognized in the method) and/or represented, in particular marked, in the data sections (e.g. by representative numbers, codes, patterns). For example, the first data sections can be represented, e.g. by a combination of codes that describe the environment of the vehicle and/or a driving situation in the surroundings of the vehicle. Alternatively or additionally, the first data sections or a set of the first data sections can identify different data, e.g. map sections, sensor data or chassis data, which are relevant for operating the real system of the vehicle, for example.

For example, the first data sections can each represent one or more objects or arrangements (i.e. static or dynamic constellations) of objects or a specific relation of an object to the real vehicle. For example, a data section represents a specific road user or a road user of a specific type, e.g. a motorcycle, car or truck in the vicinity of the vehicle. This can be represented in the data sections as an identifier of a class of the object, a dimension or shape, a relative position, an angle, an orientation, an absolute or relative speed, an acceleration, a recognized intention and/or a specific action or an action of a specific type in the respective data section.

For example, the first data sections can each represent a particular and/or selectable part, e.g. a spatial part, angle and/or distance range from the real vehicle. For example, a data section represents a road user who carries out a maneuver relating to the real vehicle (e.g. overtaking it) or a parameter, in particular a change in a parameter during the maneuver.

For example, the first data sections can each represent certain arrangements, actions or interactions between an object in the environment and the real vehicle and/or the objects in the environment, in particular with regard to their dynamics, e.g. through a change or a course of the respective parameters.

For example, the at least one first data section represents a part of the environment that is located behind the real vehicle, to the side of the real vehicle, in a blind spot of the real vehicle, or on a planned trajectory of the real vehicle.

The one or more second data sections can have one or more features of the first data sections described in this document in terms of their nature. Preferably, the second data sections can have essentially or partially, e.g. selectively with regard to the representation, identification of the objects, arrangements of objects, geometric and/or temporal relationships, the same or corresponding features, structure, data structure, data format, conventions, logic, and/or codes or coding of certain information. Particularly preferably, the second data sections and the first data sections can be compatible with each other (one-sided or two-sided), e.g. with regard to their respective structure, data structure, data format, conventions, codes or coding of certain information. This results in the advantage, for example, that a virtual object represented in the second data sections or the influence of this object on the driving situation can elicit a realistic reaction of the respective development status and/or the reaction data of the development status to the first data sections, to the second data sections, e.g. different variants of the second data sections, are easy or advantageous to compare (with each other).

In particular, the at least one first data section can represent a real environment of a real vehicle that occurs comparatively rarely or very rarely in real vehicle operation. For example, the environment of the vehicle can include arrangements of objects, driving situations and/or events that occur or are to be expected in an identical or similar variant (e.g. statistically or on average) less often than once in 10 ³ , 10 ⁴ , 10 ⁶ , 10 ⁸ , 10 ¹⁰ kilometers.

In one example of the method, the determination of the first data sections can relate at least primarily to a real environment or driving situation for which it has been determined or recognized that it occurs particularly rarely and/or differs from one or more frequently, in particular typically occurring environments, e.g. driving situations.

For example, the first data sections, in particular respective, selected recordings of the data of a first data interface, an on-board network communication, can comprise or be a (e.g. temporal and/or spatial) course of an internal parameter in the vehicle, in particular in or on the real vehicle system and/or in another system of the vehicle, e.g. in an actuator, sensor, a computing unit of the vehicle, which in particular are in a specific connection with the vehicle system, e.g. control it or are controlled by it.

In particular, the at least one trigger condition can comprise a criterion relating to a (e.g. statistical) frequency relating to an occurrence of certain parameters of the real vehicles and/or the real environments. For example, the at least one trigger condition can comprise a comparison of the data of an environment, in particular a driving situation in the environment of a real vehicle, with certain, in particular frequent and/or typically occurring, real environments or driving situations. In particular, depending on the result of the comparison, predominantly rare and/or atypical real environments or driving situations can be selected at least primarily and the first data sections that represent them can be recorded and/or further processed.

In a further example of the method, the trigger conditions can identify a deviation from one or more predetermined conditions, e.g. value ranges, combinations of value ranges and/or data patterns. For example, the first data sections can be determined as data sections that represent an environment of the vehicle that deviates from a norm or from a known and/or typical or statistically criterion-fulfilling arrangement of objects, driving situation, consequences of a driving situation, etc.

In a further example, the at least one trigger condition comprises a criterion relating to a performance, e.g. a measure of performance, a so-called KPI (= Key Performance Indicator) of the real vehicle or a functionality of the real vehicle.

In a simplified case, the first data sections can be data determined based on a radar, a camera, a LIDARS and/or already processed or fused data (such as part of the data of a so-called environment model) and/or (in each case) circumstances, e.g. objects, that have been converted (e.g. converted, converted) into a model and/or vector-based form. For example, the first data sections can initially be stored in a ring buffer of the vehicle and determined when at least one trigger condition is met (retroactively in time, so to speak).

For example, the content, in particular the number of certain parameters, e.g. certain signals, and/or the duration of a recording for a data section can be selected depending on a respective trigger condition. The data sections are in particular data sections that are not required for the operation of the vehicle (with which they are recorded) or are not used functionally there.

The invention further includes the idea of generating the corresponding data using one or more virtual objects, in particular the modification of the first data sections, instead of waiting for the occurrence of a specific (e.g. extremely rare) situation of a vehicle (also to be understood as: what is happening in the environment of the vehicle) or for a driving situation that meets specific (e.g. rarely fulfilled) criteria, which is suitable for the development, validation and/or training of a vehicle system or which is at least partially missing. This allows data and findings to be obtained which would otherwise have required driving several billion kilometers with a correspondingly large test fleet.

The process can be carried out at least partially using (real) vehicles that are already in customer operation and have been set up accordingly (e.g. the so-called customer vehicles). This has the advantage that the mileage of a large number of vehicles can be practically added together or made usable as a whole. The determination or selection of the first data sections can be carried out "idle" or in a "shadow mode", so to speak, without having a functional effect or disturbing the respective vehicle customer or third parties.

In particular, the one first data section or several of the first data sections can be determined by means of a vehicle system or a vehicle functionality which is at least partially decoupled from an execution of a functionality effective for the vehicle or for an occupant of the vehicle at least in one time interval or is operated in parallel to the execution of (its actual) functionality in the respective real vehicles.

For example, the method can be carried out using an at least partially decoupled mode (set up for this purpose) of the corresponding sensors and/or functions. The data obtained from the new environment (this also means a new artificially constructed, designed, synthesized driving situation as described here) can be used, for example, to prove a positive risk balance and/or to train neural networks.

In an advantageous embodiment, the method comprises the step of providing (e.g. generating and/or transmitting) at least one trigger condition for or to the real vehicle and the step of determining the at least two data sections depending on the at least one trigger condition.

The trigger condition is, for example, sent to the vehicle or to the vehicle fleet and/or determined in the vehicle (e.g. according to a predetermined condition with or without a parameter sent to the vehicle for the trigger condition). The trigger condition can be used to define and/or control the criteria, scope or format according to which a first data section is to be determined. Alternatively or additionally, the trigger condition defines and/or controls the composition of the first data section, e.g. under which conditions certain parts of the environment, objects, parts of the driving situations should be represented in one or more first data sections.

This trigger condition can identify a parameter, in particular a pattern, of an occurrence of one or more specific objects in the environment of the vehicle, a specific situation, e.g. driving situation in which the vehicle is located, one or more arrangements, actions or interactions between an object in the environment and the vehicle and/or the objects in the environment, and/or data, in particular input data, status and/or output data of a system of the vehicle. It can be an intelligent trigger condition that, for example, links one or more of the above parameters, in particular patterns, with one another or varies them depending on further information. In particular, the trigger condition identifies a rule according to which a first data section (e.g. from a specific vehicle or from a vehicle selected according to a specific condition) is determined.

If one of the trigger conditions is met, the first data sections can be transmitted to the backend, with or without a processing step within the vehicle, in particular in a compressed and/or coded form, through the rule described in the trigger condition.

This can also be understood, for example, to mean that a computer program, e.g. a script, comprising the instructions of a trigger condition, is transmitted to the vehicle and/or executed in the vehicle. The script can comprise at least two trigger conditions. Such a script can trigger a sensor or data processing of a customer function in the real vehicle and/or be set up to read in data from one or more interfaces of the vehicle.

For example, the second data sections can identify parameters and/or data for controlling a simulation or a situation formation rule. The virtual situation or the virtual objects, the virtual (ego) vehicle, other conditions in the virtual environment and/or their movement, change and/or behavior can be generated depending on the simulation rule or situation formation rule identified by the one or more second data sections. For example, the generation of the virtual situation is controlled (indirectly) depending on the output layer of the neural network. The situation formation rule can include or be a rule that is predetermined and/or can be determined or adapted in the process, in particular dynamically.

For example, the situation formation rule can identify an application of certain (predetermined and/or determinable in the process) criteria, e.g. for selecting the real and/or virtual objects, the parts of the environments, the behavior of the objects or the situation patterns. Preferably, the predetermined and/or determinable in the process situation formation rule can specify the formation of the data that represent a new situation having a predetermined feature, in particular parameters.

In particular, the situation creation rule can include at least a first criterion relating to the frequency (also to be understood as a measure of frequency or a measure of "rarity") of the driving situation to be generated according to the situation creation rule. For example, the said situation creation rule can determine and take into account a statistical frequency relating to the occurrence of certain driving situations.

Preferably, the second data sections (and/or consequently the reaction data the reactions thereto) represent arrangements of or driving situations involving the one or more virtual objects and one or more real objects, wherein the relationship between the number, type of virtual objects and/or certain parameters of the virtual objects can be selected, set, controlled, in particular dynamically controlled, in different ways.

The reaction data (which may include in particular data based on them) may be used to ensure, in particular to demonstrate, certain properties, in particular regarding risk, consequence and/or performance of the corresponding vehicles comprising the vehicle system (in reality or during driving). The data can be provided and/or used for homologation of vehicles and/or as a condition, in particular a prerequisite for carrying out certain functions or performance characteristics of the vehicles, operating permit for vehicles, in particular at least partially automated vehicles.

In particular, a conclusion can be drawn about the real system of the vehicle on the basis of the reaction data, e.g. in the event that the development status or a real vehicle system corresponding to one or more features of the development status were operated in a real vehicle. In particular, a conclusion can be drawn about how a real vehicle system would react to a certain movement and/or a certain behavior of a real object (with corresponding parameters), in particular how it would react if the development status had been used in the real vehicle.

In particular, a conclusion can be drawn about the reaction with respect to one or more first real objects in the case where other, second real objects were present, in particular in the case of a certain movement and/or a certain behavior of one or more such second real objects (which are represented in the method by means of the virtual objects) that did not actually occur in the real environment of the vehicle in a real driving operation.

For example, the reaction data are data that cause a reaction of a real vehicle to a real object corresponding to the virtual object. They can be reaction data that are caused depending on the virtual object, in particular according to one or more parameters of the virtual object and/or according to one or more parameters of the real environment that are represented by (or simulated by) the second data sections. Alternatively or additionally, the reaction data can be data that is related to the one or more parameters of the one or more virtual objects, in particular standardized.

In particular, the reaction data represent a reaction of the real system of the vehicle or of the development stage to the one or more virtual objects in an arrangement with one or more real objects and/or to one or more real objects in an arrangement with one or more virtual objects. For example, the reaction data can characterize a sensitivity of the reaction of the development stage with respect to one or more specific parameters of the one or more virtual objects, in particular in an arrangement with one or more real objects and/or to one or more real objects, in particular in an arrangement with one or more virtual objects.

Sensitivity (e.g. measure, function or gradient of sensitivity) can be understood as a change, in particular a gradient or a function of change, of the reaction of the developmental stage to a change in an object or a parameter of an object.

For example, a virtual object can be a replica of a road user, e.g. a pedestrian. When processing and/or using the reaction data, it can be determined (e.g. with the aid of a computer) whether the development status reacts correctly, incorrectly, appropriately or inappropriately in relation to certain parameters of the virtual object or changes to certain parameters of the object.

For example, the reaction data can identify a decision determined by the development stage, e.g. an action decision, maneuver decision. This can be a decision influenced by a virtual object. For example, the reaction data can identify a decision determined (made or taken) using means of the development stage to avoid the virtual object (virtually, so to speak), to brake on the virtual object, to carry out, change and/or abort a certain maneuver because of the object, or to carry out a collision with the object (e.g. with as little damage as possible).

The reaction data can, for example, identify parameters of a corresponding path planning (also to be understood as trajectory planning) or an evasive curve, a braking force or information, such as a display or announcement of the vehicle for the user of the vehicle, etc.

The reaction data may identify one or more operating parameters of the vehicle system (e.g. a sensor, a control unit, etc.) that are determined on the basis of the modified situation. Alternatively or additionally, the reaction data may (e.g. essentially only) indicate a difference between the reaction of the vehicle system to the the multiple virtual objects indicate the modified situation in the environment of the vehicle compared to the real situation of the vehicle.

For example, based on the reaction data, a measure of the performance of the development status and/or a measure of the sensitivity of the development status to a change in a parameter of the virtual object is determined.

For example, processing is carried out, in particular a comparison of reaction data and/or a comparison of the data determined on the basis of the reaction data. This can be data (or data sets) or a level of performance achieved in each case. The result of the comparison can be counted among the data determined depending on the second data sections. This can be made available for developing, securing, operating and/or training the vehicle system.

For example, a plurality of sets of the reaction data of a development stage, in particular several different development stages, of the vehicle system are generated for one or more sets of the second data sections. This can in particular take place at least partially simultaneously and/or by means of a plurality of processor cores and/or processors (preferably in the backend). At least partially simultaneously can mean, for example, quasi-simultaneously or within a short time interval.

A large number of different virtual situations can be generated, particularly qualitatively (in principle) or quantitatively (with regard to their quantitative parameters). Preferably, a large number of sets of reaction data are processed, which were (each) generated on the basis of several different second data sections with one or more development stages. The second data sections can thus each be generated on the basis of a set of the first data section or several sets of the data sections.

For example, in the backend, a plurality (e.g. 10 ² , 10 ^{4 ,} 10 ⁵ , 10 6 ...) of sets of the second data sections are initially generated from a set of the first data sections (which were recorded based on a first and/or second trigger condition and in a real environment of the real ^vehicle ). This computing task can be distributed across a plurality of processor cores and/or processors, which preferably process the respective computing tasks essentially in parallel. This results in a particular advantage, since at least most of the different reactions of the development stage to differently modified real environments (or driving situations) require approximately similar processing time. A corresponding stationary system comprising a plurality of development stages operated in parallel can be operated particularly efficiently.

Preferably, a plurality of identical or different development stages are used to generate and/or compare the reaction data (also to be understood as data based on the reaction data). Preferably, one development stage or several development stages can be emulated multiple times, e.g. on a plurality of processors or processor cores. For example, a plurality of development stages can be used in the sense of (more or less separate and/or simultaneously or parallel operable) electronic modules which have the features (particularly relevant for determining the reaction data) of a real system of the vehicle.

Alternatively or additionally, development stages or prototypes of the vehicle system can have hardware, e.g. as a so-called A-model, B-model, C-model, D-model or near-production control unit or a series control unit and/or software belonging to the respective development stage can be operated using the prototypes. The development stages can be operated (e.g. in a so-called closed-loop operation) with at least partially real and/or at least partially virtual hardware and/or with real or virtual actuators.

Particularly preferably, different sets of the second data sections (which were generated, for example, based on different specifications) are distributed across a plurality of development stages (e.g. belonging to the backend). For example, one (preferably a plurality) of the development stages can be operated with different, in particular quantitatively and/or qualitatively different, sets of the second data sections.

Preferably, a plurality of the second data sections (also to be understood as: sets of the second data sections), which represent an environment or driving situation of the vehicle that has been modified quantitatively and/or qualitatively differently with one or more of the virtual objects, can be divided into a plurality of copies of a development status of the system of the vehicle. In other words, a plurality of the second data sections, which represent a plurality of the respective variants of an environment of the Vehicle and/or variants of a driving situation, are used for (preferably substantially simultaneous) operation of a plurality of the copies of the development status. The second data sections can be used as input to a plurality of the emulated development statuses (eg substantially parallel).

With the method according to the invention, the development, validation, training and/or operation of a vehicle system can be accelerated by a factor of 10 ² , 10 ^{4 ,} 10 ⁵ or even 10 ⁶ , or can be carried out more effectively and/or economically.

Processing (also understood as: offsetting) of the different sets of second reaction data with each other and/or with reaction data based on real data can be carried out. Preferably, the processing can include the execution of one or more statistical operations. The result of the processing can be statistical information, in particular applicable to the plurality of reaction data.

Preferably, the processing includes determining a performance, e.g. a measure of performance, e.g. characterizing one or more performance characteristics, e.g. one or more KPIs (= key performance indicators), for the respective reaction data. From the determined performance, a conclusion can be drawn about a performance characteristic (KPI) of the development status or a vehicle system comprising this or based on it. The determined performance can be characteristic of the case that the development status or a vehicle system comprising the development status would operate in a real vehicle in a comparable environment and/or driving situation.

• - zum Testen und/oder Absichern eines Entwicklungsstands des Fahrzeugsystems, z.B. indem die entsprechenden Reaktionsdaten ausgewertet werden; und/oder
• - zum Ermitteln zumindest eines Performanceindikators, z.B. eines KPI des Entwicklungsstands, wobei insbesondere ein Rückschluss auf einen Performanceindikator eines realen Systems des Fahrzeugs (z.B. eines korrespondierenden Systems, z.B. mit gleichen oder ähnlichen Merkmalen) möglich ist; und/oder
• - zum Nachweis einer positiven Risikobilanz des Systems des Fahrzeugs, insbesondere für einen Rückschluss auf die erwartete Performance in Bezug auf reale Objekte; und/oder
• - zum Trainieren eines (weiteren, zweiten) neuronalen Netzes, wobei die resultierenden Lerndaten (z.B. Gewichtsdaten, Bias Terms, Checkpoints) insbesondere zum Betreiben eines realen Fahrzeugs einsetzbar sind; und/oder
• - zum Freischalten und/oder Sperren von einem oder mehreren Leistungsmerkmalen zumindest eines Fahrzeugs, insbesondere mit dem gleichen oder ähnlichen System wie der Entwicklungsstand. Dies kann auch nach der Auslieferung des Fahrzeugs, insbesondere für ausgewählte Fahrzeuge (z.B. ferngesteuert, per Remote) erfolgen; und/oder
• - zum Freischalten und/oder Sperren von (bestimmten) realen Reaktionen des Systems des Fahrzeugs für künftige (reale) Situationen, die der zumindest einen (zumindest teilweise) virtuellen Situation hinreichend ähnlich sind; und/oder
• - zum Verändern einer digitalen Karte und/oder von Daten zur Interpretation einer digitalen Karte, insbesondere zur Freigabe oder Sperrung bestimmter Fahrbahnabschnitte für die Ausführung bestimmter Leistungsmerkmale, insbesondere bestimmter Reaktionen auf bestimmte reale Situationen, die der zumindest einen virtuellen Situation hinreichend ähnlich sind.

The data determined by the second data sections, including in particular data determined dependent on (e.g. on the basis of) the second data sections (e.g. reaction data, comparison data, performance data, etc.), can be provided and/or used:

• - to test and/or secure a development stage of the vehicle system, e.g. by evaluating the corresponding reaction data; and/or
• - to determine at least one performance indicator, e.g. a KPI of the development status, whereby in particular a conclusion can be drawn about a performance indicator of a real system of the vehicle (e.g. a corresponding system, e.g. with the same or similar features); and/or
• - to demonstrate a positive risk balance of the vehicle system, in particular to draw conclusions about the expected performance in relation to real objects; and/or
• - for training a (further, second) neural network, whereby the resulting learning data (e.g. weight data, bias terms, checkpoints) can be used in particular for operating a real vehicle; and/or
• - to enable and/or disable one or more performance features of at least one vehicle, in particular with the same or similar system as the development status. This can also be done after delivery of the vehicle, in particular for selected vehicles (e.g. remotely controlled, via remote); and/or
• - to enable and/or disable (certain) real reactions of the vehicle system for future (real) situations that are sufficiently similar to at least one (at least partially) virtual situation; and/or
• - to modify a digital map and/or data for interpreting a digital map, in particular to enable or disable certain road sections for the execution of certain performance features, in particular certain reactions to certain real situations that are sufficiently similar to the at least one virtual situation.

These applications can also be counted as part of the development, security, training or operation of the vehicle system.

With the invention, this can also be done with respect to driving situations that have not occurred in the real environment of the vehicle in real driving conditions and, in particular, are very rarely to be expected and/or are associated with a high risk or damage. For example, the neural network can be trained and/or operated (e.g. by specifying the target or difference to real situations) to generate second data sections for at least partially virtual situations that are to be expected in reality with a statistical probability of less than 10 ^-6 , 10 ^{-8 ,} 10 ^-10 , 10 ^-12 or 10 ^-14 per kilometer of driving.

The invention also includes the data based on the reaction data, e.g. a database with a large number of data or data sets based on various, in particular a large number of different reaction data. This can be a development database or a security database.

Furthermore, the invention includes the above-mentioned or further systems or functionalities in vehicles or systems or functionalities for operating vehicles, which are developed, secured, trained, executed or made executable on the basis of the at least one virtual environment.

A further advantageous embodiment of the invention is the use, in particular the operation of a partially virtual environment of a vehicle generated according to the features of the invention for testing and/or safeguarding a development status of a vehicle system. The vehicle system can be a system for at least partially automated driving, maneuvering or parking for use in a vehicle. In this way, data can be obtained (very quickly, inexpensively and safely) which is also characteristic of the behavior of the vehicle system in comparable real environments of the vehicle.

For example, the vehicle system (and in particular the first and/or second development stage) is a driver assistance system or system for carrying out or supporting at least partially automated longitudinal guidance, lateral guidance and/or maneuvering of the vehicle, such as a system for at least partially automated driving and/or parking (also to be understood as a part of the vehicle system or a system comprising the vehicle system). Alternatively or additionally, the method can be used for various (other) systems and/or functionalities of the vehicle (if necessary in an appropriate or expediently applied embodiment).

The invention provides the possibility of testing and securing the development status of a (any or specific) system and/or a functionality of a vehicle or for installation or use in a vehicle (so to speak in a "dry run" or with reduced costs and risks) and/or of training the artificial intelligence belonging to the development status or system, in particular a (further) neural network.

In particular, virtual environments can be used to obtain very rarely occurring situations and/or boundary conditions as well as the reaction data of the development stage (e.g. for proof of performance, for the approval and/or market launch of the vehicle). A correspondingly high number (e.g. required for proof) of (e.g. desired) driving situations and/or reaction data (e.g. corresponding to the desired driving situations) can be generated very quickly, in particular by processes that mostly run simultaneously.

For example, errors or performance weaknesses can be found or the performance, safety or reliability of at least one system of the vehicle can be proven. For example, one or more regression tests can be carried out (just as quickly and effectively) after a change, e.g. error correction of the development status. This can significantly shorten the development time or cycle.

The actual distance to be covered, e.g. as part of the development, testing, validation and/or training of vehicles, can be significantly reduced. For example, time, effort and costs can be saved and/or improved validation of at least one system of the vehicle (to be developed, validated or trained) can be achieved.

Through the use of the neural network according to the invention, the resulting (at least most) second data sections can identify virtual objects or virtual situations that differ from the individual real situations used during training. These do not have to correspond to a product of a rule-based process or a combination of different situations. These can be independent of the desired, entered (programmed) rules or algorithms observed by people. This makes it possible to achieve a much more objective result, particularly with regard to developing, securing, training and/or operating the vehicle system. The result is therefore suitable for effectively and promptly finding the weak points of the vehicle system, for better prediction or proof of the performance of the vehicle system to be expected or implemented in vehicles already delivered and/or for the homologation or market launch of the vehicles.

According to a further embodiment, the first data sections are characteristic of: features of the real objects involved in the real situation; and/or of an arrangement and/or of movement parameters of the real objects involved in the respective real situation; and/or of a driving situation in the real environment of the one or more respective real vehicles; and/or of features of the movement of the one or more respective real vehicles; and/or of boundary conditions in one or more real environments of the one or more respective real vehicles, in particular other road surface parameters and/or weather parameters; and/or one or more traffic signs and/or traffic rules applicable in one or more real-world environments.

The movement of the real vehicle can be the movement of the real vehicle in relation to the real environment and/or to one or more real objects.

The characteristics can be respective parameters or value ranges, classes and/or patterns or can be expressed as such. For example, a characteristic of the (real) situation can be a class and/or pattern. For example, one or more characteristics of the real objects involved in the real situation can comprise or be a class (so-called object class). This can be, for example, a car, truck, van, two-wheeler, scooter, pedestrian, child or animal.

The boundary conditions in one or more real environments can be time-varying or position-dependent.

The at least one weather parameter can be indicative of the temperature, wind, precipitation, etc., in the respective environment. The at least one road parameter can be indicative of the friction coefficient and/or the road surface in the respective environment.

As already described, the second data sections generated depending on the neural network can each be (selectively, also) characteristic of one or more such features.

According to a further embodiment, the determination of the one or more first data sections takes place depending on the process of training the neural network and/or depending on the output layer of the neural network trained at least to a certain extent and/or depending on the second data sections or the data determined depending on the second data sections.

In particular, this is to be understood in such a way that the one or more aspects of determining the one or more first data sections are influenced, in particular controlled, depending on the process of training the neural network and/or depending on the output layer of the neural network that has been trained at least to a certain extent.

For example, a neural network can (co-)determine or control the determination of one or more trigger conditions, a selection of the first data sections, further processing of the first data sections or the generation of the data of the virtual situation. This can be done during ongoing operation (e.g. in productive operation) and/or for a new learning cycle of the neural network.

According to a further embodiment, determining the one or more first data sections is or includes: selecting one or more real vehicles and/or determining one or more trigger conditions that can be used in real vehicles and/or sending one or more trigger conditions to one or more real vehicles; and/or determining one or more first data sections based on the data of the vehicle.

For example, at least one trigger condition for determining the first data sections (e.g. the rule for detecting a real environment of a real vehicle) is changed. This can be done depending on the output layer of the neural network (depending on the result of the operation) and/or depending on the (already determined or processed) second data sections and/or depending on the operation of the neural network.

• - abhängig von dem Vorgang des Trainierens des (ersten) neuronalen Netzes; und/oder
• - abhängig von der Ausgangsschicht des zumindest bis zu einem gewissen Grad trainierten (ersten) neuronalen Netzes; und/oder
• - abhängig von den zweiten Datenabschnitten oder der abhängig von den zweiten Datenabschnitten ermittelter Daten.

For example, the selection of one or more real vehicles that can or should be used in particular to determine the first data sections and/or the determination of one or more trigger conditions that can be used in real vehicles, in particular in selected vehicles, and/or the sending of one or more trigger conditions to one or more real vehicles, in particular to the selected vehicles and/or the determination of one or more first data sections based on the data of the one or more vehicles:

• - depending on the process of training the (first) neural network; and/or
• - depending on the output layer of the (first) neural network, which has been trained at least to a certain extent; and/or
• - depending on the second data sections or the data determined depending on the second data sections.

The neural network may already be at least partially trained or may be in the process of being trained.

Preferably, the neural network can decide or co-decide during training and/or operation which new environments, in particular objects, combinations or configurations relations of objects, behavior of objects, situations are necessary or advantageous. Depending on the neural network, this can be done in the mode of recursive training. Accordingly, the determination of the first data sections can also be influenced accordingly.

One or more steps of the method described in the present document, in particular the determination of the first data sections, can be influenced (at least partially and/or temporarily) by the second data sections and/or data dependent on the second data sections (e.g. generated on the basis of the second data sections). In other words, one or more closed circles can be formed (at least partially and/or temporarily).

In another combinable example, the neural network can specify how and according to which criteria the addition, replacement and/or integration of virtual objects into a virtual situation, in particular a virtual driving situation, should take place.

According to a further embodiment, the determination of the one or more first data sections takes place depending on an activation, a change in an operating mode, and/or a control of a sensor and/or a processing of the sensor data and/or a real functionality, in particular relating to automated driving and/or parking; of the respective real vehicle.

Particularly preferably, the activation, a change of an operating mode, and/or the control of at least one sensor and/or a processing of the sensor data and/or the real system can be carried out without being necessary and/or useful for the operation, in particular driving operation, of the specific real vehicle. The activation, change of an operating mode, and/or the control of at least one sensor and/or a processing of the sensor data and/or the real system can take place without a need, a functional benefit and/or a customer benefit in the real vehicle, e.g. also without any benefit for the specific real vehicle or its operation.

For example, an activation or a change of the parameters, the operating mode of a sensor, the processing of the sensor data and/or a real system of the vehicle, in particular comprising a functional logic, can take place. For example, depending on the trigger condition, the modeling of the environment of the vehicle and/or a parameter or logic of a customer function or a system function can be changed.

In other words, depending on whether the trigger condition is met, a change can be brought about in the vehicle, in particular stimulated, in order to thereby determine (e.g. desired, particularly suitable) first data sections.

It can, for example, be a change, in particular stimulation, of a real system that corresponds to the developmental stage, e.g. for which the developmental stage serves.

The trigger condition and/or the first data sections can serve predominantly, essentially only or exclusively to carry out the procedure, e.g. to determine, evaluate and/or use the reaction data of a development status (different from the concrete real system in the real vehicle) in the backend.

For example, initial data sections can be determined that the specific real vehicle would otherwise not need (on its own), e.g. for functional reasons. For example, the trigger condition is sent to the vehicle with which the first data sections are determined (regardless of a functional need and/or use in the vehicle).

For example, the trigger condition initiates a process in the vehicle or in the vehicle's environment that serves to determine the first data sections that do not represent a functional necessity and/or benefit for driving in the environment for the specific vehicle or are independent of it.

According to a further embodiment, the second data sections are characteristic of features and/or an arrangement of the virtual objects involved in the virtual situation and/or of movement parameters of the virtual objects involved in the virtual situation and/or of behavior parameters of the virtual objects involved in the virtual situation and/or of simulation parameters and/or situation formation rules for generating the one or more virtual situations involving one or more virtual objects.

The arrangement and/or movement parameters of the virtual objects can refer to the coordinate system of the virtual environment and/or to the virtual vehicle or can be expressed in relation to the coordinate system of the virtual environment and/or to the virtual vehicle.

According to a further embodiment, the neural network is trained and/or operated to modify data sections that characterize a part of a real environment and/or one or more real objects with one or more virtual parts and/or one or more virtual objects. The second data sections can characterize a resulting, at least partially virtual environment.

The generation of the second data sections can include or be an insertion, in particular integration, of the virtual object(s) in an environment represented by the one or more third data sections, e.g. in an (existing arrangement represented by the third data sections) of one or more real objects. In the context of the present document, this is also referred to as modifying the first data sections with one or more virtual parts and/or one or more virtual objects.

The third data sections used for modification can (selectively) correspond to the already described features of the third data sections (used for training and/or operating the neural network). For example, the same first data sections that were already used for training the neural network and/or other, in particular new, current or almost real-time third data sections can be modified.

For example, second data sections are generated from the one or more third data sections and one or more virtual objects. This can be a (possibly complex) processing. This can also take different effects into account.

By generating second data sections from the one or more third data sections and one or more virtual objects, in particular by inserting the one or more virtual objects and/or by modifying the (third) data sections representative of a real situation in the environment of the vehicle with the one or more virtual objects, modified (second) data sections can be generated that represent a partially real and partially virtual environment of the vehicle.

Alternatively or additionally, the generation of second data sections from the one or more (e.g. currently valid) third data sections and one or more virtual objects can take place at least partially in a real vehicle (e.g. in the respective real vehicle). For example, the one or more real vehicles can be set up to generate the one or more second data sections from the one or more third data sections and one or more virtual objects (the data sections representing the virtual objects). Furthermore, the one or more real vehicles can be set up to provide the second data sections (in this case also to be understood as preprocessed data for generating the second data sections). The generation and/or provision of the second data sections can (also) take place depending on the trigger condition.

Preferably, the determination of the third data sections and/or the second data sections comprises a selection (in particular selective, e.g. executable according to certain criteria), an extraction and/or an exclusion of data, e.g. concerning individual real or virtual objects and/or processes.

For example, based on a set of the third data sections and one or more virtual objects (in particular each with quantitatively and/or qualitatively different parameters), several quantitatively and/or qualitatively different sets of the second data sections can be generated. These can represent quantitatively and/or qualitatively different arrangements of real and virtual objects, in particular in relation to the vehicle, in particular quantitatively and/or qualitatively different driving situations (in the partially real, partially virtual environment of the vehicle).

Preferably, the second (modified) data sections can be generated as data sections that correspond to input data for a system of the vehicle, e.g. sensor data, parameter patterns, etc., which include the influence of the virtual object on the (otherwise recorded) arrangements of real objects or environment.

For example, data sections that represent the virtual object, such as images, thermal images, point clouds, so-called radar objects, LIDAR objects or corresponding image sequences (at least approximately, approximately, model-like, model-based) can be generated that the real vehicle, in particular a real sensor or a sensor-driven system of the vehicle, would generate if a real object corresponding to the virtual object appeared in the real environment.

Modifying the third data section or sections can be done, for example, by means of a mathematical model, a simulation, e.g. by means of a simulation of a sensor of a vehicle or the data of the sensor. Output data of the sensor or the sensor system or input data of the vehicle system can at least partially represent the virtual object (or its influence on the third data sections) and/or at least partially represent the real environment of the vehicle.

During the modification, second data sections may be generated which represent a changed real environment that would exist in the event of the presence of a real object corresponding to the virtual object (preferably including its movement and/or behavior).

The modification can be carried out in such a way that a dynamic change or the behavior of the resulting (at least partially virtual) environment and/or the driving situation in the environment is determined, in particular reproduced and/or simulated.

• - auf das ein oder die mehreren virtuellen Objekte (z.B. ausgehend von einem oder mehreren realen oder virtuellen Objekten bzw. Anordnung realer Objekte) und/oder
• - auf ein oder mehrere reale Objekte (z.B. ausgehend von dem einen oder den mehreren virtuellen Objekten bzw. Anordnung realer Objekte) wirken würden.

The modification can be carried out taking into account physical conditions, relationships and/or effects that affect the virtual object and/or the real objects. For example, physical conditions and/or relationships can be taken into account that, in the case of the existence of a real object that corresponds to the virtual object,

• - to which one or more virtual objects (e.g. based on one or more real or virtual objects or arrangement of real objects) and/or
• - would act on one or more real objects (e.g. starting from the one or more virtual objects or arrangement of real objects).

Put simply, physical conditions, relationships, and/or effects can be taken into account that would apply (e.g. to a selected part of the real environment) if the virtual object were a (corresponding) real object.

In another example, the method includes modifying the third data portions (to generate the second data portions) according to a situation formation rule.

For example, a driving situation can be generated from some of the real objects represented by the third data sections and one or more virtual objects according to a situation creation rule. Real and virtual objects can be added, replaced, integrated and/or merged. The second data sections that can then be generated can, for example, include driving situations that have not occurred in any of the real vehicles in reality or that would only be expected with a very low probability.

For example, the third data sections are modified (to generate the second data sections) in such a way that driving situations represented by the second data sections correspond to a specific criterion. It can also be just a specific pattern for which one or more variants of the second data sections (also to be understood as: the partially virtual environment represented by the second data sections) are generated, each with several variants of a driving situation (or the development of the driving situation).

For example, the predetermined and/or method-determinable rule can specify the formation of the data of a new situation, which in particular comprises certain objects in the environment, a certain situation, in particular a driving situation, one or more arrangements, actions or interactions between an object in the environment and the vehicle and/or the objects in the environment, and/or data, in particular input data, status and/or output data of a system of the vehicle.

Particularly preferably, the predetermined and/or method-determinable rule identifies a specific pattern according to which the new environment, in particular a new situation, can be generated. For example, data can be determined on a new situation (created in the method, perhaps never occurred in the physical world) in which, for example, two motorcycles are in the blind spot of the vehicle and overtake the vehicle from two sides at the same time or drive too close to each other.

In a further example, the invention comprises the step of modifying the third data sections according to a situation formation rule by adding, replacing, integrating, and/or merging data of real objects (also to be understood as: data of parts of real objects) with one or more virtual objects (also to be understood as: with parts of virtual objects).

For example, a proportion of virtual objects in the second data sections may be less than 10, 20, 30, 50% or greater than 50, 70, 80, 90% of the total number of objects. Preferably, the proportion of virtual objects or types of objects used within an at least partially virtual situation can be adjustable. Depending on the use of the second data sections, different proportions of virtual objects or real objects within the at least partially virtual situation (or in the second data sections) can be advantageous.

Preferably, the second data sections predominantly represent real objects, e.g. an environment consisting of real objects with an arrangement or driving situation of real objects (according to the third data sections), with a comparatively small proportion of virtual objects being integrated into the environment. For example, an environment or driving situation changed by one or more virtual objects can be generated predominantly on the basis of reality. Such a driving situation already carries its reference or proximity to reality within itself.

The reaction data can be generated (even far away from the respective real vehicles), which (thus) are based largely on reality, whereby the influence (also: change or contribution) caused by one or more virtual objects can be recorded and/or tracked. The reaction data can also (thus) have a high degree of relevance to reality and be sufficiently representative of it. For example, the data based on the reaction data can represent and/or predict the behavior of the real vehicle with a vehicle system based on the respective development status with a high degree of reliability and/or accuracy.

For example, the reaction data or the respective performance measures are compared with one or more thresholds.

Particularly preferably, the at least one neural network can be trained and/or operated to generate different (at least partially) virtual situations, in particular to set, maintain or change the relationship between the number and/or certain parameters of the virtual objects and certain parameters of the real objects (which are represented in the one or more second data sections) according to a goal, in particular a target value.

For example, the quantity, type and/or diversity of virtual situations, the trends depicted by them or the data based on them (e.g. reaction data and data derived from them) and the uses of these can be further increased.

• In einem weiteren Beispiel erfolgt das Erzeugen der zweiten Datenabschnitte durch (insbesondere auch zu verstehen als „mittels“ oder „umfassend“) Einfügen, insbesondere Integrieren, eines oder mehrerer virtuellen Objekten zu den dritten Datenabschnitten (bzw. in die dritten Datenabschnitte), und/oder ein Modifizieren der dritten Datenabschnitte abhängig von dem einen oder mehreren virtuellen Objekten.

The environments (partly real and partly virtual) represented by the second data sections or information based on the one or more such data sections (or environments) can be provided and/or used for:

• In a further example, the second data sections are generated by (in particular also to be understood as “by means of” or “comprising”) inserting, in particular integrating, one or more virtual objects into the third data sections (or into the third data sections), and/or modifying the third data sections depending on the one or more virtual objects.

In particular, the generation of the second data sections by inserting one or more virtual objects into the third data sections is to be understood (or occurs) in such a way that the second data sections comprise or represent both real objects (represented by the third data sections) and the one or more virtual objects. The modification of the third data sections can comprise or be an addition, replacement, integration and/or merging of specific parts or parts selected according to specific criteria from one or more third data sections and from one or more data sections representing a virtual object.

Alternatively or additionally, the third data sections are modified depending on the one or more virtual objects, in particular depending on the parameters of the one or more virtual objects. The modification preferably comprises a change to the one or more objects represented in the third data sections or the third data sections, with modified real objects (or objects previously based on real objects) being represented in the second data sections.

An arrangement (e.g. as one or more arrangements valid at different times or a dynamic scene) of the one or more real objects and one or more virtual objects can also be taken into account.

The generation of the second data sections by inserting one or more virtual objects into the third data sections or the modification of the third data sections can be carried out in such a way that the second data sections are a result of the detection of the environment of the vehicle. stuff with the real and the virtual objects.

For example, the second data sections can represent at least partial (e.g. spatially, due to the arrangement of the objects or the detection perspective) concealment of a real object by a virtual object (or vice versa or in any combination). For example, the second data sections can represent a reflection and/or attenuation of at least one signal, in particular a radar signal reflected (e.g. from a real object or virtual object), laser beams (laser pulses from a LIDAR) or an ultrasound signal.

The second data sections can represent a (static or preferably dynamic) arrangement of the one or more real objects and one or more virtual objects with one or more effects to be expected from the arrangement, in particular in reality (if the virtual objects were real). For example, the second data sections can identify the one or more virtual objects in an arrangement with one or more real objects and/or one or more real objects in an arrangement with one or more virtual objects.

According to a further embodiment, the data determined depending on the second data sections are or include the reaction data of a (the) vehicle system to the second data sections and/or analysis data relating to the second data sections and/or relating to the reaction data; and/or control data and/or parameterization data for operating the vehicle system.

For example, control data or parameterization data can include or be the (improved, optimized) control signals or parameters for operating the vehicle system. These can be transmitted to the vehicles (e.g. remotely) and contribute to the operation of the vehicles.

According to a further embodiment, the one or more real situations relate to a real driving situation of the real vehicle and/or the one or more virtual situations relate to a virtual driving situation of a virtual vehicle.

In the context of this document, the term “driving situation” can be understood, for example, as a specific situation (real, fully or partially virtual) characterized by an arrangement, action or interaction of road users or by certain driving parameters of road users.

In particular, the meaning of the term “situation” or “driving situation” differs from the frequently used colloquial meaning of the term “traffic situation”, which rather corresponds to summary, general categories such as “free traffic”, “heavy traffic”, “slow-moving traffic”, “traffic jam”, “end of traffic jam”, etc.

In particular, a driving situation is (rather) characterized by a certain pattern (also to be understood as a data pattern), e.g. a pattern characterizing an arrangement and/or speed of the objects and/or a pattern of the parameters of the driving situation. The driving situation can also be characterized by a spatial pattern of the so-called free spaces in the surroundings of the vehicle. For example, a trigger condition can refer to such features or corresponding parameters of the driving situation.

• - eine (bestimmte) räumliche Verteilung der Verkehrsteilnehmer und/oder der Bewegungsparameter der Verkehrsteilnehmer, insbesondere ein Anordnungsmuster der Verkehrsteilnehmer in der Umgebung des (realen) Fahrzeugs,
• - eine (bestimmte) räumliche Verteilung unbeweglicher Objekte in der Umgebung des Fahrzeugs,
• - eine relative Position und/oder Bewegungsparameter zu bestimmten Arten von Spurmarkierungen, Verkehrszeichen, Ampeln (nicht notwendigerweise zu bestimmten Ampeln, etc.),
• - eine Information über die Vorfahrt des Fahrzeugs, insbesondere gegenüber bestimmten Verkehrsteilnehmern und/oder Verkehrsteilnehmern, die tatsächlich oder zumindest potentiell aus bestimmten Richtungen, z.B. einer querenden Straße von rechts oder von links kommen oder kommen können,
• - eine Information zu einer, z.B. einen Grenzwert überschreitenden, Handlung eines Verkehrsteilnehmers in der Umgebung des Fahrzeugs, wie z.B. ein Hupen, Lichthupen, Drängeln, Überholen des Fahrzeugs, ein Überholversuch und dergleichen.

Preferably, the at least one driving situation can be characterized by one or more of the following features:

• - a (certain) spatial distribution of road users and/or the movement parameters of road users, in particular an arrangement pattern of road users in the environment of the (real) vehicle,
• - a (certain) spatial distribution of immobile objects in the vicinity of the vehicle,
• - a relative position and/or movement parameters to certain types of lane markings, traffic signs, traffic lights (not necessarily to specific traffic lights, etc.),
• - information about the right of way of the vehicle, in particular in relation to certain road users and/or road users who are actually or at least potentially coming or may come from certain directions, e.g. a crossing road from the right or from the left,
• - information about an action by a road user in the vicinity of the vehicle, e.g. one which exceeds a limit value, such as honking, flashing headlights, pushing, overtaking the vehicle, attempting to overtake and the like.

Furthermore, the driving situation can be characterized by one or more parameters related to relevant traffic rules, traffic signs, right of way, traffic lights and/or traffic light phases.

Preferably, the at least one driving situation can be a driving situation that exceeds certain limit values or a driving situation that is characterized by parameters that exceed certain limit values. For example, the at least one driving situation can be a driving situation with an undesirable or dangerous approach to an object or a road user, an acceleration value that exceeds a limit value, an undesirable arrangement with other road users, etc.

The driving situation can be a special situation (e.g. one that occurs relatively rarely) or a dangerous driving situation, e.g. a driving situation for which an increased risk is determined or assumed. Such driving situations can be recognized, in particular predicted, using appropriately defined trigger conditions (also parameters or criteria of the trigger conditions). Data sections can then be determined for generating a virtual environment of a vehicle, in particular a virtual environment with a certain parameter or a virtual environment of a certain type or a virtual environment that corresponds to a certain pattern of real vehicles.

According to a further embodiment, the vehicle system is a driver assistance system and/or a system for carrying out or supporting at least partially automated longitudinal guidance, lateral guidance and/or maneuver execution of a vehicle, in particular a system for at least partially automated driving, maneuvering and/or parking.

For example, the vehicle system of the real vehicle (also to be understood as: the vehicle system to which the first development status refers, and/or the vehicle system to which the second development status refers, and/or a vehicle system of the real vehicle which is used in determining the first data section(s)) is a driver assistance system and/or a system for carrying out or supporting at least partially automated longitudinal guidance, lateral guidance and/or maneuvers (e.g. driving maneuvers, parking maneuvers or shunting maneuvers) of a vehicle. In particular, it can be a system for at least partially automated driving and/or parking.

According to a further embodiment, a comparison is carried out between the reaction data of a (the) vehicle system determined on the basis of several (preferably a plurality) second data sections (in particular sets of the second data sections) and the second data sections and/or analysis data relating to the second data sections and/or relating to the reaction data and/or control data and/or parameterization data for operating the vehicle system. The result of the comparison can be provided and/or used for developing, securing, training and/or operating the vehicle system. In this sense, the result of the comparison can (also) be counted among the data determined depending on the second data sections.

According to a further embodiment, training a vehicle system comprises training a second neural network of the vehicle system depending on the second data sections, wherein the vehicle system is trained: to control a virtual vehicle within the virtual environment, in particular to drive and/or park at least partially automatically; and/or to strive for a specific goal when operating the virtual vehicle in the virtual environment.

Training a vehicle system can include or be training a neural network of the vehicle system. The result of training the (second) neural network can be provided and/or used to update, retrofit and/or operate a real vehicle, in particular a vehicle similar to the real vehicle or a vehicle that has been further developed compared to the real vehicle.

The result of the training includes the data of the (second) neural network), e.g. a part of the checkpoint of the trained neural network. A checkpoint can in particular be or include the weights (also to be understood as weight information) and/or bias terms of the (up to a certain stage or sufficiently) trained neural network. At least one component of the trained neural network (also to be understood as: one or more so-called checkpoints) can be at least partially adopted into a next stage of development, in particular into a real system of a vehicle.

For example, it can be determined how the vehicle system would react to a road user (with a certain characteristic, with a certain movement and/or a certain behavior) if such a user were to appear within the real environment of the real vehicle.

For example, data can be determined, processed or used based on the reaction data of a development stage (or of the real system, e.g. a driver assistance system and/or a safety system) on second data sections that represent a partially real and partially virtual environment of the vehicle in which, e.g. a child, cyclist or pedestrian suddenly appears and/or moves or behaves in a certain way.

• - abhängig von dem Vorgang des Trainierens des zweiten neuronalen Netzes (des zum Einsatz für den Betrieb der Fahrzeugen genutzten neuronalen Netzes, insbesondere des neuronalen Netzes des Fahrzeugsystems); und/oder
• - abhängig von der Ausgangsschicht des zumindest bis zu einem gewissen Grad trainierten zweiten neuronalen Netzes.

Preferably, one or more real vehicles are selected, which can or should be used in particular to determine the first data sections, and/or
determining one or more trigger conditions usable in real vehicles, in particular in selected vehicles, and/or sending one or more trigger conditions to one or more real vehicles, in particular to the selected vehicles and/or determining one or more first data sections based on the data of the one or more vehicles (also):

• - depending on the process of training the second neural network (the neural network used to operate the vehicles, in particular the neural network of the vehicle system); and/or
• - depending on the output layer of the second neural network, which has been trained at least to a certain extent.

The second neural network may already be at least partially trained or may be in the process of being trained.

Preferably, the second neural network can decide or co-decide during training and/or operation which new environments, in particular objects, combinations or constellations of objects, behavior of objects and/or situations are necessary or advantageous. This can be done in recursive training mode depending on the second neural network. Accordingly, the determination of the first data sections can also be influenced accordingly.

In other words, the determination of the first data sections or the training data for training the (first) neural network can be carried out depending on the training process and/or the operation of the second neural network. The method can be carried out particularly efficiently.

According to a further embodiment, the determination of the one or more first data sections takes place depending on the reaction data of the vehicle system.

In particular, depending on the reaction data or depending on the data determined on the basis of the reaction data, the selection of one or more real vehicles that can or should be used in particular to determine the first data sections and/or the determination of one or more trigger conditions that can be used in real vehicles, in particular in selected vehicles, and/or the sending of one or more trigger conditions to one or more real vehicles, in particular to the selected vehicles and/or the determination of one or more first data sections based on the data of the one or more vehicles can take place.

The term ‘dependent’ should be understood in particular to mean ‘also dependent’, so that several of the dependencies proposed in this document may each apply alternatively and/or additionally and/or alternately.

Dependent on the reaction data is also to be understood in particular as dependent on the data determined on the basis of the reaction data. For example, the determination of the one or more first data sections can be dependent on the achieved performance or freedom from errors of the vehicle system (in an operation dependent on the second data sections).

The invention can achieve various advantages in terms of costs, effort in the development and market launch of modern vehicles.
For example, the functionality, performance and/or safety of the development status and the resulting system of the vehicle based on the development status can be increased.

By implementing the procedure, the development, testing, validation and training of vehicles can be carried out more quickly and with less cost and risk.

A very large amount (and/or number of variants) of data can be obtained, e.g. for training a neural network. This can also include data on driving situations that would otherwise usually only be ascertainable through extremely dangerous driving situations.

Vehicle systems or vehicles can therefore also be designed for or with situations and/or properties, movement and/or behavior of objects that occur very rarely in reality. and/or undesirable, are developed and, if necessary, trained.

• - Reduktion denkbarer Risiken für Dritte bzw. für die Gesellschaft;
• - Reduktion der Kosten bzw. des Verbrauchs von Ressourcen, insbesondere Fahrzeugen;
• - eine Absicherung und/oder ein Trainieren der Fahrzeuge mit geringerer Fahrleistung;
• - Energieeinsparung, insbesondere betreffend oben genannte Fahrleistungen.

Furthermore, one or more of the following benefits may arise:

• - Reduction of possible risks for third parties or for society;
• - Reduction of costs and consumption of resources, particularly vehicles;
• - securing and/or training vehicles with lower mileage;
• - Energy saving, particularly with regard to the above-mentioned driving performance.

The process can be used to shorten development cycles, particularly in the development of automated driving.

Furthermore, the method may also include further measures, whereby the measures encode information or assign it to anonymous identifiers and/or process it with anonymous identifiers in such a way that personal or personally identifiable data cannot be reconstructed or can only be reconstructed with great difficulty during or after use of the data.

Furthermore, numerous other benefits can be achieved based on the features described in this document.

According to a second aspect, the invention comprises a system for developing, securing, training and/or operating a vehicle system. The system is designed to determine one or more first data sections that characterize one or more real situations of one or more real vehicles, and to train and/or operate at least one neural network depending on the first data sections and: to generate second data sections that characterize one or more at least partially virtual situations, depending on the output layer of the neural network, and is further designed to provide the second data sections or data determined depending on the second data sections for developing, securing, training and/or operating the vehicle system.

The system can comprise one or more real vehicles, in particular vehicles from a customer and/or test fleet, and a backend (e.g. a data center). These can be set up for a corresponding operative connection. For example, the backend, in particular in an operative connection with one or more real vehicles, can be set up to carry out at least part of the method, in particular to generate the second data sections, the reaction data, and/or to use the reaction data (according to one or more features of the method). Furthermore, the system can be designed to update the at least one neural network, in particular regularly, according to maintenance intervals or versions of a new product, e.g. with new data (learning data, checkpoints, etc.).

According to a third aspect, the invention comprises a neural network. The neural network is designed to develop, secure, train and/or operate a vehicle system. The neural network is designed to generate one or more second data sections that characterize one or more at least partially virtual situations, wherein the neural network is trained and/or operable depending on one or more first data sections that characterize one or more real situations of one or more real vehicles.

The at least one neural network can be understood as (e.g. portable) data or features, in particular as topology, learning data, checkpoint, weight information (also to be understood in particular as connection weights), bias terms of the neural network. The data of the at least one neural network are to be understood in particular as data that has been trained to a certain degree or for specific cases up to a certain degree. These can already be loaded or deployed onto one or more respective computing units. “Trained” is also to be understood in particular as “trainable” and/or as trained to a certain degree.

For example, the at least one neural network (e.g. to be used during operation) can comprise features of or data based on the at least one (e.g. at least partially trained) neural network. For example, the data is generated on a computing unit when training the at least one neural network in order to then make it operable on one or more other computing units, in particular in (e.g. in the backend).

The inputs and outputs that are used as inputs or outputs in the respective application or in the corresponding process step are referred to as the input layer and the output layer.

The neural network may be configured according to one or more of the methods described in this document. Features can be trained or updated and/or can be updated. The at least one neural network is preferably a neural network that is set up according to the features described in the present document and is at least partially optimized or trained for these. The neural network can be set up to carry out one or more steps of the method or to carry out one or more functionalities according to one or more features described in the present document when it is operated in a data processing unit.

For example, the topology of the at least one neural network and/or interconnection or a direct or indirect control of at least two neural networks can be set up, in particular optimized, to carry out one or more steps of the method. The at least one neural network is preferably a DNN (= Deep Neural Network). It can preferably be a recurrent neural network. In particular, the at least one neural network has one or more direct feedbacks and/or one or more indirect feedbacks.

According to a further aspect of the invention, the invention comprises the training data for training the at least one first neural network and/or training data of the second neural network, wherein the training data are determined according to one or more features described in the present document.

According to a further aspect, the invention comprises at least one computer program, in particular a computer program product comprising a computer program. The one or more computer programs are designed to carry out the method described in this document (to be understood as: a respective part of the method) when executed on at least one computing unit.

The computer program product can be designed as an update of a previous computer program, which, for example, comprises parts of the computer program or the corresponding program code as part of a functional extension, for example as part of a so-called "remote software update". The computer program product comprises in particular a medium that can be read by the data processing device and on which the program code is stored, or at least an encrypted file. According to a further aspect, a program product is described that comprises an authorized right of access to stored data of the computer program product.

According to a further aspect, the invention comprises a (real) vehicle. The (real) vehicle can be set up to receive and execute a trigger condition, in particular the trigger condition according to one or more features described in this document, and/or to transmit data sections determined depending on the trigger condition to a backend. For example, one or more real vehicles can be set up to carry out at least part of the method, in particular in an (information technology) operative connection with the backend. For example, one or more real vehicles can be set up to apply the at least one trigger condition determined in the backend (or transmitted by it), to determine the first data sections, and/or to generate the second data sections. Alternatively or additionally, the (real) vehicle can comprise part of the development status (e.g. software, data from a trained neural network, etc.) or a system based on the development status.

The term "vehicle" in this document is to be understood in particular as a motor vehicle, in particular a car, a truck, a two-wheeler, e.g. a motorcycle, or a bicycle. For example, one or more real vehicles can be motor vehicles that can be set up in particular for a driving operation typical for motor vehicles of the type mentioned above (e.g. customer operation, e.g. on public roads).

Alternatively or additionally, the real vehicle may be a special-purpose vehicle, a remote-controlled vehicle, a vehicle other than a motor vehicle, a watercraft, an aircraft, an agricultural vehicle, a machine for carrying out earthmoving or for extracting (e.g. fossil) raw materials, or a construction machine.

According to a further aspect, the invention relates to a development status of a vehicle system, in particular a vehicle system for at least partially automated driving, which is developed, secured, trained and/or operated according to the features of the invention. In particular, this includes a status of a trained (second) neural network, e.g. in the form of data from a checkpoint and/or bias term of the development status. Such a development status can, for example, be transmitted (rolled out) to vehicles as an update or upgrade (also retrospectively) or used in the vehicles.

• 1 einen beispielhaften Ablauf des Verfahrens,
• 2 eine Darstellung zur Veranschaulichung realer Situationen, zumindest teilweise virtuelle Situationen sowie mögliche Zusammenhänge zwischen diesen,
• 3 eine Darstellung zur Veranschaulichung des Ermittelns von ersten Datenabschnitten,
• 4 eine Darstellung zur Veranschaulichung des Erzeugens von zweiten Datenabschnitten einer teilweise virtuellen Situation,
• 5 eine Darstellung zur Veranschaulichung ausgewählter Zusammenhänge im Verfahren, und
• 6 eine Darstellung zur Veranschaulichung des Erzeugens von Reaktionsdaten, von Analysedaten eines Fahrzeugsystems, sowie Steuerungsdaten bzw. Parametrierungsdaten zum Betreiben des Fahrzeugsystems.

The invention is described below without restricting its generality using combinable embodiments. It is pointed out that all features and functional principles described in the present example description can be combined with each other and/or with the features and functional principles described above.

• 1 an example of the procedure,
• 2 a representation to illustrate real situations, at least partially virtual situations and possible connections between them,
• 3 a diagram to illustrate the determination of first data sections,
• 4 a representation to illustrate the generation of second data sections of a partially virtual situation,
• 5 a representation to illustrate selected relationships in the procedure, and
• 6 a representation to illustrate the generation of reaction data, analysis data of a vehicle system, as well as control data or parameterization data for operating the vehicle system.

The 1 shows an exemplary flow chart of the method according to the invention using exemplary steps of the method. The method can be carried out cyclically, with several consecutive operating cycles.

The solid arrows indicate a suggested sequence of steps. The arrows can also mean that the step to which the arrow points is carried out depending on the result of the step where the arrow starts. The dash-dotted arrows each mean an influence on an operating cycle, in particular a control of the execution of a step to which the arrow points, depending on the result of one or more steps (from a previous operating cycle) from which the arrow or arrows start.

In particular, the method may comprise only part of the process described and/or further steps.

• Zu Schritt S1(a): Es wird eine Trigger-Bedingung TB1, TB2, TB3 (siehe nachfolgende Figuren) ermittelt. Diese kann an ein oder mehrere reale Fahrzeuge RF1, RF2, RF3 (siehe Figuren) übermittelt und/oder innerhalb der jeweiligen realen Fahrzeug RF1, RF2, RF3 (z.B. nach einer vorausbestimmten Logik bzw. Anweisungen) ermittelt werden.

Regarding step S1: Step S1 may include or be one or more of the following sub-steps or processes S1 (a), S1 (b), S1 (c), S1 (d):

• For step S1(a): A trigger condition TB1, TB2, TB3 (see following figures) is determined. This can be transmitted to one or more real vehicles RF1, RF2, RF3 (see figures) and/or determined within the respective real vehicle RF1, RF2, RF3 (e.g. according to a predetermined logic or instructions).

The trigger condition TB1, TB2, TB3 can be transmitted from a backend BE (see the following figures), in particular the backend described in this document (this also includes a computer connected to the backend). The trigger condition TB1, TB2, TB3 is preferably sent to a large number of vehicles RF1, RF2, RF3, in particular to part of a vehicle fleet (e.g. a customer fleet), or received by a large number of vehicles RF1, RF2, RF3.

• - eine Situation, z.B. Fahrsituation, in der sich das Fahrzeug RF1, RF2, RF3 befindet; und/oder
• - eine Aktion des Nutzers, insbesondere des Fahrers des Fahrzeugs RF1, RF2, RF3; und/oder
• - einen Betriebszustand eines oder mehrerer Systeme des Fahrzeugs RF1, RF2, RF3.

The one or more trigger conditions TB1, TB2, TB3 can, for example, identify a parameter, in particular a pattern, for which the first data sections DA1, DA1', DA1''' (each assigned to the individual conditions) (see following figures) are to be determined. The one or more trigger conditions TB1, TB2, TB3 can in particular be characteristic of:

• - a situation, e.g. driving situation, in which the vehicle RF1, RF2, RF3 is located; and/or
• - an action of the user, in particular the driver of the vehicle RF1, RF2, RF3; and/or
• - an operating state of one or more systems of the vehicle RF1, RF2, RF3.

It can be an intelligent trigger condition TB1, TB2, TB3, which, for example, links one or more of the above parameters, in particular patterns. Depending on the trigger condition TB1, TB2, TB3 (if the corresponding condition applies), any other data, e.g. concerning the environment of the respective vehicle RF1, RF2, RF3, can be determined (if necessary transmitted to the backend BE) and taken into account in the process.

Alternatively or additionally, the first data sections DA1, DA1', DA1" can be determined using so-called test fleets (e.g. of the vehicle manufacturer, supplier or service provider).

For step S1(b): One or more first data sections DA1, DA1', DA1" are determined S1 during the driving operation of one or more real vehicles RF1, RF2, RF3, whereby the first data sections DA1, DA1', DA1" are characteristic of a real situation in the respective real environment RU1, RU2, RU3 (see following Figures) of the respective vehicle RF1, RF2, RF3. This can take place at certain road sections, e.g. at positions in the real environment RU1, RU2, RU3 corresponding to specific features and/or depending on a predetermined condition, in particular with regard to the driving situation.

For example, the data sections can be selected depending on (also by means of) a trigger condition TB1 TB2, TB3 (described above) and recorded (depending on whether the respective conditions apply). For example, if a trigger condition TB1 TB2, TB3 applies (many, specific or selectable according to the trigger condition TB1 TB2, TB3), first data sections DA1, DA1', DA1', which characterize the situation and/or one or more actions of the user of the vehicle RF1, RF2, RF3 or an operating state of one or more systems of the vehicle (e.g. in a compressed form), are sent to the backend BE (see the following figures), e.g. to a data center.

The first data sections DA1, DA1', DA1" can include or be recordings of an on-board network communication, in particular a (e.g. temporal and/or spatial) course of an internal parameter of a computing unit of the vehicle RF1, RF2, RF3. For example, the data corresponding to the first data sections DA1, DA1', DA1 ' can (initially) be stored in a ring buffer (e.g. within the vehicle RF1, RF2, RF3) and (only) when the trigger condition TB1, TB2, TB3 is met (retroactively in time, so to speak) determined and/or transmitted to the backend BE.

The first data sections DA1, DA1', DA1'' can represent a part of the situation in a part of the environment RU1, RU2, RU3 of the vehicle RF1, RF2, RF3, in particular at least approximately mathematically depict it.

According to an example that can be combined with all the features described in this document, operating data are determined that are characteristic of an operation of the real vehicle RF1, RF2, RF3 and correspond to the real situation in the real environment RU1, RU2, RU3 of the respective vehicle RF1, RF2, RF3 (e.g. temporally and/or causally).

The operating data can be taken into account in the process, in particular when operating the vehicle system (also: the replica or the development stage) when processing and/or evaluating the reaction data RD, RDA, RDI, KPI ... (see following figures).

The operating data can (also) be determined as data sections that characterize a change, e.g. a change in another variable, such as time, position or travel distance or the operating parameters. For example, the operating data can characterize a course (e.g. the course within a certain time interval and/or a travel distance) of one or more parameters of a device of the vehicle RF1, RF2, RF3, in particular a device that exchanges data (e.g. control signals) with the vehicle system during real vehicle operation.

The data sections of the operational data may refer to a time interval comprising the last seconds before the occurrence of the real situation, during the real situation and/or after the real situation.

• - einen Teil des Systems, für welches der Entwicklungsstand bestimmt ist, bilden; und/oder
• - zu dem System gehören, für welches der Entwicklungsstand bestimmt ist; und/oder
• - ein Schnittstellensystem zu dem System bilden, für welches der Entwicklungsstand bestimmt ist.

The data sections of the operating data can refer to a vehicle system of the same type as the vehicle system FS1, FS2, FS3 (for which the development status is determined or the second neural network DNN2 is trained), and/or to an interface system of the vehicle system FS1, FS2, FS3. For example, operating data that are characteristic of an operation of the real vehicle RF1, RF2, RF3 can refer to a system of the real vehicle RF1, RF2, RF3, which:

• - form part of the system for which the development stage is intended; and/or
• - belong to the system for which the development level is intended; and/or
• - form an interface system to the system for which the development stage is intended.

• - Daten auf Basis eines Sensors oder Sensorsystems; und/oder
• - Daten eines weiteren Systems bzw. einer Datenquelle, die als Eingangsdaten zum Betreiben des Fahrzeugsystem FS1, FS2, FS3 des berücksichtigt werden; und/oder
• - Daten kennzeichnend Aktionen des Nutzers des Fahrzeugs RF1, RF2, RF3; und/oder
• - Daten kennzeichnend einen Zustand des Fahrwerks des Fahrzeugs RF1, RF2, RF3; und/oder
• - Steuerungsdaten zum Steuern eines Teils des Fahrwerks des Fahrzeugs RF1, RF2, RF3.

The operating data, in particular data sections of the operating data, may include or be one or more entries from the following list:

• - Data based on a sensor or sensor system; and/or
• - Data from another system or data source that are taken into account as input data for operating the vehicle system FS1, FS2, FS3; and/or
• - Data characterising actions of the user of the vehicle RF1, RF2, RF3; and/or
• - Data characterising a condition of the chassis of the vehicle RF1, RF2, RF3; and/or
• - Control data for controlling part of the chassis of the vehicle RF1, RF2, RF3.

The operating data can be (respective) status data of a corresponding device of the respective vehicle RF1, RF2, RF3, which were valid in connection with the respective real situation (in the respective real environment RU1, RU2, RU3.

The sensor-based data may be at least partially raw, preprocessed and/or interpreted data from a sensor, e.g. a radar, a camera and/or a LIDAR, etc.

The operating data can be assigned to the corresponding first data sections DA1, DA1', DA1", second data sections DA2, DA2', DA2" and/or reaction data RD, RDA, RDI, KPI and/or taken into account (e.g. used) when processing or using these (e.g. in one or more steps of the method). For example, these can be (co-)used when training S2 the first neural network DNN1 (see the following figures), when operating S3 the first neural network DNN1 and/or when training S6 the second neural network DNN2 (see the following figures).

For step S1(c): The first data sections DA1, DA1', DA1" (this also includes the raw data for determining the first data sections DA1, DA1', DA1") are transmitted to the backend BE. For example, only data sections DA1, DA1', DA1" that correspond to or are selected according to certain criteria can be transmitted.

Regarding step S2: A training S2 of at least one neural network DNN1 is carried out depending on the first data sections DA1, DA1', DA1".

• - Ermitteln von Trainingsdaten TD1, insbesondere auf Basis der ersten Datenabschnitte DA1, DA1', DA1" und/oder der in einem der vorangegangenen Betriebszyklen erzeugten zweiten Datenabschnitte DA2, DA2', DA2" oder der abhängig von den zweiten Datenabschnitten DA2, DA2', DA2" ermittelten Daten RD, AD, SD, ... und/oder weiterer korrespondierender Daten (z.B. der Betriebsdaten des Fahrzeugs RF1. RF2, RF3); und/oder
• - Ermitteln der Backpropagation zur Ausführung des Trainierens S2; und/oder
• - Auswahl oder Anpassen eines oder mehrerer Parameters des Trainiervorgangs S2.

Training S2 may include:

• - Determining training data TD1, in particular on the basis of the first data sections DA1, DA1', DA1" and/or the second data sections DA2, DA2', DA2" generated in one of the preceding operating cycles or the data RD, AD, SD, ... determined depending on the second data sections DA2, DA2', DA2" and/or other corresponding data (e.g. the operating data of the vehicle RF1, RF2, RF3); and/or
• - Determining the backpropagation to execute the training S2; and/or
• - Selecting or adjusting one or more parameters of the training process S2.

One or more aspects of the training S2 described above can be determined or adapted depending on the first data sections DA1, DA1', DA1" (also: data based on the first data sections DA1, DA1', DA1").

Regarding step S3: Alternatively or additionally, an operation S3 of a neural network DNN1 (e.g. trained as described or at least partially in a different way) can be carried out depending on the first data sections DA1, DA1', DA1". The operation can be understood as a control of the neural network DNN1, in particular at its input layer, and/or as a selection of one or more operating modes or one or more settings of the neural network DNN1 during operation.

Preferably, the control S3 of the neural network DNN1 takes place depending on the first data sections DA1, DA1', DA1". For this purpose, data determined in a processing step DV can also be counted depending on the first data sections DA1, DA1', DA1".

• - eine oder mehrere statistische Größen einer oder mehrerer realer Situationen, insbesondere eine statistische Verteilung bestimmter Parameter in einer oder mehreren realen Situationen; und/oder
• - ein oder mehrere Wahrscheinlichkeitsmaße auf Basis einer oder mehrerer realer Situationen, insbesondere eine Wahrscheinlichkeitsverteilung bestimmter Parameter in einer oder mehreren realen Situationen; und/oder
• - aggregierte Daten aus mehreren realen Situationen, insbesondere von mehreren realen Situationen, die jeweils zu einer bestimmten Art gehören (z.B. bestimmte Fahrmanöver umfassen oder bei bestimmten Fahrmanöver auftreten).

Preferably, the first data sections DA1, DA1', DA1" used to operate the neural network DNN1 are identified by:

• - one or more statistical quantities of one or more real-world situations, in particular a statistical distribution of certain parameters in one or more real-world situations; and/or
• - one or more probability measures based on one or more real situations, in particular a probability distribution of certain parameters in one or more real situations; and/or
• - aggregated data from several real-world situations, in particular from several real-world situations, each of which belongs to a certain type (e.g. involving certain driving manoeuvres or occurring during certain driving manoeuvres).

This may also be data determined depending on the first data sections DA1, DA1', DA1'', in particular aggregated data based on several first data sections DA1, DA1', DA1'' from several different situations (driving situations).

By operating S3 (e.g. setting, influencing, controlling, regulating) the at least one neural network DNN1 depending on the first data sections DA1, DA1', DA1'', in particular those prepared in this way, a generation S4 of second data sections DA2, DA2', DA2'' can be effected, which particularly well represent reality-relevant and/or represent realistic virtual situations. Therefore (in step S3, S4 and the subsequent steps) a result can be achieved that is representative of possible variants of the (future) reality and is therefore very advantageous for using the second data sections DA2, DA2', DA2'' for various purposes.

Step S4 (S4a / S4b): When operating S3 the neural network DNN1, the second data sections DA2, DA2', DA2'' are generated S4. The second data sections DA2, DA2', DA2'' are characteristic of one or more at least partially virtual situations. Second data sections DA2, DA2', DA2'' can be generated that reproduce input data for the vehicle system (e.g. development status of the vehicle system, replica or emulation), in particular in the form of sensor data, parameter patterns, on-board network signals, etc., according to selectable criteria (e.g. more or less exactly or up to certain permissible deviations).

In one variant, the at least partially virtual situations are purely, essentially or predominantly virtual situations. These or the corresponding second data sections DA2, DA2', DA2'' can be generated in variant S4a of step S4.

In a further variant S4b, the second data sections DA2, DA2', DA2'' (at the latest in the result) can represent both real objects RO1, RO2, RO3 (to be understood: at least a selection of the real objects), as well as the one or more virtual objects VO1, VO2, VO3. The corresponding at least partially virtual situation can comprise a real part (possibly taking place almost in real time) and a virtual part (concerning the environment TVU itself and/or objects).

In particular, the generation S4b of the second data sections DA2, DA2', DA2'' can comprise a modification of third data sections DA3, DA3', DA3'' or can be carried out by a modification of third data sections DA3, DA3', DA3''.

The modification can be carried out by inserting, in particular by integrating, one or more virtual objects VO1, VO2, VO3 into or into the third data sections DA3, DA3', DA3'' or into the situation characterized by the third data sections DA3, DA3', DA3''. The modification is carried out depending on the neural network DNN1 (in particular: by the neural network DNN1).

For example, the third data sections DA3, DA3', DA3'' are modified depending on one or more virtual objects VO1, VO2, VO3 (e.g. on the data that represents virtual objects) or by means of the one or more virtual objects VO1, VO2, VO3. These can be one or more virtual objects VO1, VO2, VO3 that are (virtually) located outside the vehicle RF1, RF2, RF3. These can be added to the real situation (or the third data sections DA3, DA3', DA3'' that represent the real situation) and/or integrated into it (in an appropriate manner) or offset against it.

Alternatively or additionally, the real environment can be modified depending on other (e.g. immobile) virtual parts (e.g. road markings, traffic signs, road edges, buildings, trees, etc. as data representing virtual conditions) or by means of other virtual parts.

Alternatively or additionally, a real object RO1, RO2, RO3 or a number of real objects RO1, RO2, RO3 (also to be understood as data sections DA3, DA3', DA3'' corresponding to the objects) that existed in the original real situation can be at least partially removed and/or replaced by one or more virtual objects VO1, VO2, VO3.

In particular, the neural network DNN1 is trained S2 and/or operated S3 to carry out the modification of the third data sections DA3, DA3', DA3'' according to a target G (which can be specified in the method) and/or difference D, e.g. in relation to the characteristics of one or more real situations and/or to previously generated data sections DA2, DA2', DA2''.

Preferably, the modification comprises changing the one or more objects RO1, RO2, RO3 represented in the third data sections DA3, DA3', DA3'' or a corresponding change in the third data sections DA3, DA3', DA3''. In the second data sections DA2, DA2', DA2'', both changed real objects (or objects previously based on real objects RO1, RO2, RO3) and the one or more virtual objects VO1, VO2, VO3 can be represented.

The second data sections DA2, DA2', DA2'' of the virtual object VO1, VO2, VO3 (e.g. as the corresponding real object that corresponds to the virtual object VO1, VO2, VO3 and is simulated by it) can be sensor data and/or a data model (e.g. a 2D or 3D model) of the virtual object VO1, VO2, VO3. This is done in the backend BE (with the available computing resources, which are significantly larger than the resources of the real vehicle). The second data sections DA2, DA2', DA2'' can simulate sensor data or corresponding signals of a real vehicle RF1, RF2, RF3, which would have been generated by a real sensor upon the occurrence of a real object RO1, RO2, RO3 (corresponding to the virtual object VO1, VO2, VO3).

The at least one neural network DNN1 can be trained S2 and/or operated S3 to bring together the real parts of the real environment or the real objects RO1, RO2, RO3 on the one hand and the virtual (e.g. virtually generated or inserted) parts or the virtual objects VO1, VO2, VO3 on the other hand, in particular depending on a or the predeterminable target G and/or difference D. This process is also referred to in the present document as modifying S4b. In this case, a real situation (e.g. represented by the third data sections DA3, DA3', DA3'') can be influenced or modified (also to be understood as: influenced in a targeted manner or in a desired direction, in particular changed) with one or more virtual objects VO1, VO2, VO3 (e.g. which simulate certain road users or road users behaving in a certain way).

For example, in the context of the modification mentioned, virtual objects VO1, VO2, VO3 are added to the third data sections DA3, DA3', DA3'' (depending on the output layer of the neural network DNN1 and/or according to a predetermined dependency and/or at least partially randomly). In this context, a virtual object VO1, VO2, VO3 stands for an object that did not exist in the real situation (at least not with the parameters of the virtual object or not at all). In particular, one or more virtual objects VO1, VO2, VO3 or virtual situations with parameters that did not occur in any real situation (in this way) are created.

For example, certain real objects RO1, RO2, RO3 can be replaced by virtual objects VO1, VO2, VO3. The resulting virtual object VO1, VO2, VO3 can have a shape that differs from the real object RO1, RO2, RO3, in particular one that can be selected, determined or generated in the process. It can move in the same way as the real object RO1, RO2, RO3 within the context of the at least partially virtual situation. Alternatively or additionally, a real object RO1, RO2, RO3 can have a movement parameter and/or a corresponding change and/or behavior within the context of the at least partially virtual situation that differs from the corresponding reality, in particular one that can be selected, determined or generated in the process.

The modification S4b can also be applied with respect to the third data sections DA3, DA3', DA3'' determined in real time (in other words, to the reality currently taking place).

The result can be generated (e.g. almost in real time) by an at least partially virtual situation in the environment TRU of the vehicle represented by the second data sections DA2, DA2', DA2''. The remaining part of the environment can be purely real or partially virtual TVU environments.

For example, in the third data sections DA3, DA3', DA3'', which correspond to a real environment RU1, RU2, RU3 or real situation, data sections characterizing a real (normally behaving) pedestrian can be at least partially removed and/or at least partially replaced by data of a virtual pedestrian or a pedestrian with a changed behavior, another road user such as a skateboarder, an animal or a child.

Optionally, a parameter of a (real) road user RO1, RO2, RO3 can be changed. For example, a behavior parameter of the road user RO1, RO2, RO3 and/or a parameter relating to their reaction and/or ability to react to a real object RO1, RO2, RO3 can be changed. The correspondingly modified data would correspond to a changed situation in the environment of the vehicle (a parallel reality, so to speak) in which the situation develops differently (e.g. according to several different variants V1, V2, V3).

For example, the one or more virtual objects VO1, VO2, VO3 generated in the method can be designable, changeable, configurable and/or interactive objects VO1, VO2, VO3.

The at least one neural network DNN1 can be trained S2 and/or operated S3 to determine or adapt one or more parameters of virtual objects VO1, VO2, VO3 or to control the movement, a change and/or the behavior of these. In particular, this can be done in such a way that one or more goals G and/or differences D (in particular related to the resulting at least partially virtual situation or the second data sections DA2, DA2', DA2'') are aimed for, in particular maintained.

The parameters of the virtual object VO1, VO2, VO3 can be parameters related to the virtual object VO1, VO2, VO3, in particular to the actions of the virtual object VO1, VO2, VO3.

• - seine aktuelle Eigenschaft und/oder sein aktuelles Verhalten;
• - seine Position, seine Abmessungen, und/oder seine Bewegung;
• - seine Wahrnehmung (z.B. ein von dem Objekt einsehbarer Raumbereich und/oder das, was das Objekt aus seiner Perspektive wahrnehmen kann);
• - eine Aktion und/oder eine Einwirkung des virtuellen Objekts VO1, VO2, VO3 auf das (reale bzw. virtuelle) Fahrzeug RF, VF und/oder auf ein weiteres (realen bzw. virtuellen) Objekt RO, VO;
• - eine Einwirkung des (realen bzw. virtuellen) Fahrzeugs RF, VF und/oder eines weiteren (realen bzw. virtuellen) Objekts RF, VO auf das virtuelle Objekt VO1, VO2, VO3.

The parameters of the virtual object VO1, VO2, VO3 can be representative of one or more elements from the following list:

• - its current characteristics and/or behaviour;
• - its position, its dimensions, and/or its movement;
• - its perception (e.g. an area of space visible to the object and/or what the object can perceive from its perspective);
• - an action and/or an influence of the virtual object VO1, VO2, VO3 on the (real or virtual) vehicle RF, VF and/or on another (real or virtual) object RO, VO;
• - an impact of the (real or virtual) vehicle RF, VF and/or another (real or virtual) object RF, VO on the virtual object VO1, VO2, VO3.

The virtual objects VO1, VO2, VO3 can be of a certain type, e.g. they can replicate an object of a certain type or class. For example, they can correspond to a road user of a certain class or replicate them, e.g. in terms of their behavior and/or their effect on the vehicle system. A virtual object VO1, VO2, VO3 can also be characterized by an at least partially randomly determined movement, behavior and/or an at least partially random effect on the vehicle or a vehicle system.

Alternatively or additionally, a virtual object VO1, VO2, VO3 can be generated as impact data that characterizes one or more dimensions of their impact on a vehicle system and/or as corresponding data from a sensor that detects the environment of the vehicle or corresponding data from a vehicle function. For example, a virtual object VO1, VO2, VO3 can be represented by data that characterizes an image of the (corresponding) object VO1, VO2, VO3 in the data of the vehicle system, e.g. in the raw, preprocessed or interpreted sensor data.

For example, based on the third data sections DA3, DA3', DA3', which correspond to a real situation, reaction data RD, RDA, RDI, KPI of the (original and/or alternative) vehicle system FS1, FS2, FS3 can be determined that would result in the event of a (specific) change or a variant V1, V2, V3 of the situation. For example, it can be determined what could have happened if a road user, e.g. a pedestrian, had reacted a few seconds later or if a cyclist had suddenly appeared. In this way, a large number of variants V1, V2, V3 can be generated, for example to find and analyze borderline cases.

A large number of different sets of the second data sections DA2, DA2', DA2'' can be generated with different modifications. For example, on the basis of the data sections DA3, DA3', DA3'' corresponding to a real situation, several hundred, thousands or millions of modified data that correspond to modified situations, in particular driving situations, can be generated. It is therefore particularly advantageous to carry out this step in a backend BE (e.g. on several processors or processor cores that are preferably operated in parallel).

In the method, several sets of the second data sections DA2, DA2', DA2'' can be generated, which are characteristic of a change in the real environment RU1, RU2, RU3 of the vehicle (also to be understood as the process or result of the change) by the one or more virtual objects VO1, VO2, VO3. This is therefore a change that would apply if, instead of the virtual object VO1, VO2, VO3, a corresponding real object RO1, RO2, RO3 were present in the real environment RU1, RU2, RU3.

The (virtual or virtual situation-based) reaction data RD, RDA, RDI, KPI and/or analysis data AD and/or control data SD or parameterization data PD of at least one vehicle system (e.g. only stationary) can be determined. In particular, the (virtual) reaction data RD, RDA, RDI represent a reaction of a vehicle system emulated (e.g. in a stationary computer) to the (relevant) part of the at least partially virtual situation.

The use of the resulting data for the (further) development, validation, training S6 and operation S7 of a vehicle system FS1, FS2, FS3 for at least partially automated driving, parking or a functionality of the chassis of the vehicle RF1, RF2, RF3 is particularly advantageous because it enables the very high demand for data from practice-relevant and/or sufficiently different driving situations to be satisfied much better.

In the method, the (already mentioned) one or more goals G and/or one or more differences D that are to be aimed for and in particular maintained can be taken into account. The goal G and/or the difference D preferably relate to statistical measures and/or probability measures that are to be aimed for and in particular maintained. This can (already) take place when determining, sending and/or executing the trigger conditions TB1, TB2, TB3, when training S2 the neural network DNN1, when operating S3 the neural network DNN1, and/or in a (further) processing step (which can in particular be added to the steps S3, S4, or S4).

• - Parameter virtueller Objekte VO1, VO2, VO3 (z.B. kennzeichnend die Gestalt, Bewegung und/oder Verhalten des einen oder mehreren virtuellen Objekte VO1, VO2, VO3); und/oder
• - Anzahl, Dichte, Bewegungsparameter von Objekten VO1, VO2, VO3, RO1, RO2, RO3 bestimmten Arten oder Klassen innerhalb einer (zu erzeugenden) zumindest teilweise virtuellen Situation; und/oder
• - Häufigkeitsmaß und/oder Wahrscheinlichkeitsmaß für bestimmte Vorkommnisse, Anordnungen von Objekten VO1, VO2, VO3, RO1, RO2, RO3, Wertebereiche der Parameter betreffend Gestalt, Bewegung und/oder Verhalten des einen oder der mehreren virtuellen Objekte VO1, VO2, VO3 (z.B. in Bezug auf das virtuelle Fahrzeug, andere virtuelle Objekte und/oder ggf. reale Objekte RO1, RO2, RO3.

The one or more objectives G and/or differences D may relate to:

• - Parameters of virtual objects VO1, VO2, VO3 (e.g. characterising the shape, movement and/or behaviour of the one or more virtual objects VO1, VO2, VO3); and/or
• - number, density, movement parameters of objects VO1, VO2, VO3, RO1, RO2, RO3 of certain types or classes within an (to be generated) at least partially virtual situation; and/or
• - Frequency measure and/or probability measure for certain occurrences, arrangements of objects VO1, VO2, VO3, RO1, RO2, RO3, value ranges of the parameters relating to the shape, movement and/or behaviour of the one or more virtual objects VO1, VO2, VO3 (e.g. in relation to the virtual vehicle, other virtual objects and/or possibly real objects RO1, RO2, RO3.

• - Reaktionsdaten RD1, RDA1, RDI1, ... RD2, RDA2, RDI2, RD3, RDA3, RDI3... KPI1, KPI2, KPI3,... die für die Reaktion eines Fahrzeugsystems FS1, FS2, FS3, insbesondere mittels eines jeweiligen Entwicklungsstands FS1, FS2, FS3) auf die zweiten Datenabschnitte DA2, DA2', DA2'' kennzeichnend sind; und/oder
• - Analysedaten AD1, AD2, AD3 betreffend die zweiten Datenabschnitte DA2, DA2', DA2'' und/oder betreffend die Reaktionsdaten RD1, RDA1, RDI1, ... RD2, RDA2, RDI2, RD3, RDA3, RDI3... KPI1, KPI2, KPI3; und/oder
• - Steuerungsdaten SD1, SD2, SD3 bzw. Parametrierungsdaten PD1, PD2, PD3 zum Betreiben S7 des Fahrzeugsystems FS1, FS2, FS3, und/oder eines oder mehrerer Fahrzeuge RF1, RF2, RF3, für die das Fahrzeugsystem FS1, FS2, FS3 entwickelt wird oder ist, insbesondere zum Steuern des Betriebs eines Teils der Fahrzeugflotte RF1, RF2, RF3;

The method (e.g. step S4) preferably comprises generating or determining:

• - reaction data RD1, RDA1, RDI1, ... RD2, RDA2, RDI2, RD3, RDA3, RDI3... KPI1, KPI2, KPI3,... which are characteristic of the reaction of a vehicle system FS1, FS2, FS3, in particular by means of a respective development status FS1, FS2, FS3) to the second data sections DA2, DA2', DA2''; and/or
• - analysis data AD1, AD2, AD3 relating to the second data sections DA2, DA2', DA2'' and/or relating to the reaction data RD1, RDA1, RDI1, ... RD2, RDA2, RDI2, RD3, RDA3, RDI3... KPI1, KPI2, KPI3; and/or
• - Control data SD1, SD2, SD3 or parameterization data PD1, PD2, PD3 for operating S7 the vehicle system FS1, FS2, FS3, and/or one or more vehicles RF1, RF2, RF3 for which the vehicle system FS1, FS2, FS3 is or is being developed, in particular for controlling the operation of part of the vehicle fleet RF1, RF2, RF3;

These data can be considered as the data determined depending on the second data sections DA2, DA2', DA2''. These data can be provided (with or without a further processing step, in particular data aggregation) for developing, securing, training S6 and/or operating S7 a vehicle system FS1, FS2, FS3, FS1', FS2', FS3', in particular for controlling the operation S7 or an update of part of the vehicle fleet.

For example, control data SD1, SD2, SD3 or parameterization data PD1, PD2, PD3 can be used to operate the vehicle system FS1, FS2, FS3 and/or one or more vehicles RF1, RF2, RF3 for updating, parameterizing and/or controlling the one or more vehicle systems FS1, FS2, FS3, in particular vehicles RF1, RF2, RF3 that have already been delivered (e.g. customer vehicles or vehicle fleet). For example, these can be used (e.g. by means of wireless provision to the vehicles RF1, RF2, RF3) to operate the vehicles RF1, RF2, RF3, in particular to achieve a (specific) improvement in the operation of the vehicle system.

To step S5:

In step S5, the second data sections DA2, DA2', DA2'' or data dependent on the second data sections DA2, DA2', DA2'', in particular reaction data RD1, RDA1, RDI1, ... RD2, RDA2, RDI2, RD3, RDA3, RDI3... KPI1, KPI2, KPI3, analysis data AD1, AD2, AD3, and/or control data SD1, SD2, SD3 or parameterization data PD1, PD2, PD3 are provided for developing, securing, training (S6) and/or operating (S7) the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3').

The analysis data AD1, AD2, AD3 can include or be the result of the application of various types of (preferably fully or predominantly automated) data analysis to the second data sections DA2, DA2', DA2'', to the reaction data RD, RDI, RDA, KPI and/or other corresponding data.

• - zum Test und/oder zur Absicherung des Fahrzeugsystems FS1, FS2, FS3 bzw. eines Entwicklungsstands ES1, ES2, ES3 des Fahrzeugsystems FS1, FS2, FS3, insbesondere des realen Fahrzeugs RF1, RF2, RF3;
• - zum Ermitteln zumindest eines Performanceindikators, z.B. KPI, des Fahrzeugsystems FS1, FS2, FS3;
• - zum Nachweis einer positiven Risikobilanz des Fahrzeugsystems FS1, FS2, FS3, insbesondere für einen Rückschluss auf die erwartete Performance KPI1, KPI2, KPI3 in Bezug auf reale Objekte;
• - zum (Weiter-)Entwickeln des Fahrzeugsystems FS1, FS2, FS3, insbesondere gemäß einer oder mehrerer Methoden der datengetriebenen Entwicklung.

The data provided will be used for one or more entries from the following list:

• - for testing and/or securing the vehicle system FS1, FS2, FS3 or a development stage ES1, ES2, ES3 of the vehicle system FS1, FS2, FS3, in particular the real vehicle RF1, RF2, RF3;
• - to determine at least one performance indicator, e.g. KPI, of the vehicle system FS1, FS2, FS3;
• - to demonstrate a positive risk balance of the vehicle system FS1, FS2, FS3, in particular to draw conclusions about the expected performance KPI1, KPI2, KPI3 in relation to real objects;
• - for the (further) development of the vehicle system FS1, FS2, FS3, in particular according to one or more methods of data-driven development.

For example, the vehicle system FS1, FS2, FS3 (in the target variant of the vehicle system) is a driver assistance system or a system for executing or supporting at least partially automated longitudinal guidance, lateral guidance and/or maneuver execution of the vehicle, such as a system for at least partially automated driving and/or parking.

Regarding step S6: In step S6, training of a vehicle system FS1, FS2, FS3, ... FS1', FS2', FS3', in particular training of a second neural network DNN2 of the vehicle system FS1, FS2, FS3, ... FS1', FS2', FS3' is preferably carried out depending on the second data sections DA2, DA2', DA2'' or on the data determined depending on the second data sections DA2, DA2', DA2'' (e.g. reaction data RD1, RDA1, RDI1, ... RD2, RDA2, RDI2, RD3, RDA3, RDI3... KPI1, KPI2, KPI3, analysis data AD1, AD2, AD3, and/or control data SD1, SD2, SD3 or parameterization data PD1, PD2, PD3).

• - ein virtuelles Fahrzeug VF1 innerhalb der (zumindest teilweise) virtuellen Umgebung VU1, TVU anzusteuern, insbesondere um zumindest teilweise automatisiert zu fahren und/oder zu parken; und/oder
• - ein bestimmtes Ziel (z.B. Performanceziel) beim Betreiben des virtuellen Fahrzeugs VF1 innerhalb der (zumindest teilweise) virtuellen Umgebung VU1, TVU anzustreben, insbesondere zu erreichen.

In particular, the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3') is trained:

• - to control a virtual vehicle VF1 within the (at least partially) virtual environment VU1, TVU, in particular to drive and/or park at least partially automatically; and/or
• - to strive for, and in particular to achieve, a specific goal (e.g. performance goal) when operating the virtual vehicle VF1 within the (at least partially) virtual environment VU1, TVU.

One or more features and/or principles already used for the (first) neural network DNN1 can be applied (analogously).

• - abhängig von dem Vorgang des Trainierens S6 des zweiten neuronalen Netzes DNN2 (des zum Einsatz für den Betrieb der Fahrzeugen genutzten neuronalen Netzes DNN2, insbesondere des neuronalen Netzes des Fahrzeugsystems DNN2); und/oder
• - abhängig von der Ausgangsschicht des zumindest bis zu einem gewissen Grad trainierten zweiten neuronalen Netzes DNN2; und/oder
• - abhängig vom Betreiben S7 des zweiten neuronalen Netzes DNN2.

Preferably, additional feedback data is generated:

• - depending on the process of training S6 of the second neural network DNN2 (the neural network DNN2 used for the operation of the vehicles, in particular the neural network of the vehicle system DNN2); and/or
• - depending on the output layer of the second neural network DNN2, which has been trained at least to a certain extent; and/or
• - depends on the operation S7 of the second neural network DNN2.

• - Auswahl eines oder mehrerer realer Fahrzeuge RF1, RF2, RF3, die insbesondere zum Ermitteln der ersten Datenabschnitte DA1, DA1', DA1'' genutzt werden können oder sollen; und/oder
• - Ermitteln einer oder mehrerer in realen Fahrzeugen RF1, RF2, RF3, insbesondere in ausgewählten Fahrzeugen RF1, RF2, RF3, nutzbaren Trigger-Bedingungen TB1, TB2, TB3; und/oder
• - Versenden einer oder mehrerer Trigger-Bedingungen TB1, TB2, TB3 an ein oder mehrere reale Fahrzeuge RF1, RF2, RF3, insbesondere an die ausgewählten Fahrzeuge RF1, RF2, RF3; und/oder
• - Ermitteln eines oder mehrerer erster Datenabschnitte DA1, DA1', DA1'' auf Basis der Daten des einen oder der mehreren Fahrzeug RF1, RF2, RF3; und/oder
• - Trainieren (Schritt S2) und/oder Betreiben (Schritt S3) des (ersten) neuronalen Netzes DNN1; und/oder
• - Ermitteln von den von den zweiten Datenabschnitten DA2, Da2', DA2'' abhängiger Daten.

Depending on the feedback data, one or more of the following conditions may be determined or adjusted (in one of the subsequent operating cycles of the process):

• - Selection of one or more real vehicles RF1, RF2, RF3, which can or should be used in particular to determine the first data sections DA1, DA1', DA1''; and/or
• - Determining one or more trigger conditions TB1, TB2, TB3 that can be used in real vehicles RF1, RF2, RF3, in particular in selected vehicles RF1, RF2, RF3; and/or
• - Sending one or more trigger conditions TB1, TB2, TB3 to one or more real vehicles RF1, RF2, RF3, in particular to the selected vehicles RF1, RF2, RF3; and/or
• - Determining one or more first data sections DA1, DA1', DA1'' based on the data of the one or more vehicles RF1, RF2, RF3; and/or
• - Training (step S2) and/or operating (step S3) the (first) neural network DNN1; and/or
• - Determining the data dependent on the second data sections DA2, Da2', DA2''.

In other words, the second neural network DNN2 can (also) decide or co-decide during training (step S6) and/or operation (step S7) how the first neural network DNN1 is trained S2 and/or operated S3 (e.g. in a subsequent operating cycle). This allows the method to be carried out particularly efficiently.

Regarding step S7: In step S7, the operation of a vehicle system FS1, FS2, FS3, FS1', FS2', FS3' is preferably carried out depending on the second data sections DA2, DA2', DA2'' or on the data determined depending on the second data sections DA2, DA2', DA2'' (e.g. reaction data RD1, RDA1, RDI1, ... RD2, RDA2, RDI2, RD3, RDA3, RDI3... KPI1, KPI2, KPI3, analysis data AD1, AD2, AD3, and/or control data SD1, SD2, SD3 or parameterization data PD1, PD2, PD3). This can include operating S7 several real vehicles RF1, RF2, RF3, in particular a so-called vehicle fleet with the vehicle system FS1, FS2, FS3, FS1', FS2', FS3'.

• - Freischalten und/oder Sperren von Leistungsmerkmalen zumindest eines Fahrzeugs, insbesondere von Fahrzeugen mit dem gleichen oder einem ähnlichen Fahrzeugsystem FS1, FS2, FS3, FS1', FS2', FS3'; Dies kann ggf. selektiv und/oder dynamisch (im Betrieb der Fahrzeugflotte) erfolgen;
• - Freischalten und/oder Sperren von (bestimmten) realen Reaktionen des Fahrzeugsystems FS1, FS2, FS3, FS1', FS2', FS3' auf reale, insbesondere mit dem zumindest einem virtuellen Objekt VO1, VO2, VO3 korrespondierenden Objekte RO1, RO2, RO3; Dies kann ggf. selektiv und/oder dynamisch (im Betrieb der Fahrzeugflotte) erfolgen;
• - Verändern einer digitalen Karte und/oder von Daten zur Interpretation einer digitalen Karte, insbesondere zur Freigabe und/oder Sperrung bestimmter Fahrbahnabschnitte für die Ausführung bestimmter Leistungsmerkmale, insbesondere bestimmter Reaktionen auf bestimmte reale Objekte RO1, RO2, RO3 bzw. auf bestimmte Parameter realer Objekte RO1, RO2, RO3; Dies kann ggf. selektiv und/oder dynamisch (im Betrieb der Fahrzeugflotte) erfolgen;

The operation S7 of the vehicle system FS1, FS2, FS3, FS1', FS2', FS3' or the vehicles RF1, RF2, RF3 with the vehicle systems FS1, FS2, FS3, FS1', FS2', FS3' can include or be:

• - Activating and/or blocking performance features of at least one vehicle, in particular vehicles with the same or a similar vehicle system FS1, FS2, FS3, FS1', FS2', FS3'; this can be done selectively and/or dynamically (during operation of the vehicle fleet);
• - Activating and/or blocking (certain) real reactions of the vehicle system FS1, FS2, FS3, FS1', FS2', FS3' to real objects RO1, RO2, RO3, in particular those corresponding to at least one virtual object VO1, VO2, VO3; this can be done selectively and/or dynamically (during operation of the vehicle fleet);
• - Modifying a digital map and/or data for interpreting a digital map, in particular for enabling and/or blocking certain road sections for the execution of certain performance features, in particular certain reactions to certain real objects RO1, RO2, RO3 or to certain parameters of real objects RO1, RO2, RO3; this can be done selectively and/or dynamically (during operation of the vehicle fleet);

2 shows some examples of real situations, at least partly virtual situations and some connections between them.

Three real situations from three different real environments RU1, RU2, RU3 are shown as examples. These were recorded at different time intervals and/or at different locations (on the same or a similar road).

The corresponding first data sections DA1, DA1', DA1'' are determined using the external sensors of the respective real vehicle RF1, RF2, RF3. Alternatively or additionally, the first data sections DA1, DA1', DA1'' relating to the one or more real vehicles RF1, RF2, RF3 can be determined using sensors outside the vehicle RF1, RF2, RF3, e.g. an infrastructure sensor (e.g. a sensor installed in the environment RU1, RU2, RU3, a surveillance camera or sensors of other vehicles in the environment RU1, RU2, RU3).

In this example, the first data sections DA1, DA1', DA1'' are used both for training (step S2) and for operating (step S3) the at least one neural network DNN1.

When training the neural network DNN1, the first data sections DA1, DA1', DA1'' are preferably used as a representation of real objects RO1, RO2, RO3, their arrangement, movement, behavior, action or interaction (or data determined depending on the first data sections DA1, DA1', DA1''). These first data sections DA1, DA1', DA1'' can include or be vector-based models of the respective real situation.

When operating the neural network DNN1, processed, in particular aggregated (already described) data based on the first data sections DA1, DA1', DA1'' are used. The first data sections DA1, DA1', DA1'' from several real vehicles RF1, RF2, RF3 are (pre)processed, in particular aggregated, at different locations and time intervals. This can be done within the data processing DV shown in the figure.

When operating the neural network DNN1, several purely virtual situations (essentially virtual situations) are generated in the virtual environment VU. A variant VU(V1) of these situations is shown as a representative. A large number of situations can be generated that differ (e.g. in small steps).

Alternatively or additionally, several partially virtual (partly real) situations are created in a partially virtual environment TVU. A variant TVU(V1) is shown as a representative example.

The second data sections DA2, DA2', DA2'' output at the output of the neural network DNN1 represent the respective (generated) at least partially virtual situations.

The output can be different (e.g. in different sizes, formats or post-processing steps, etc.) for different uses of the data sections DA2, DA2', DA2'' or depending on the second data sections DA2, DA2', DA2''.

3 shows the determination of first data sections DA1, DA1', DA1'' as well as some other parts of the method according to the invention.

The determination is carried out by means of the (possibly predictive) transmission of trigger conditions TB1, TB2, TB3 to a number of vehicles RF1, RF2, RF3 from a vehicle fleet. In particular, the trigger conditions TB1, TB2, TB3 are adjusted continuously, regularly or at a new cycle of the process. The one or more trigger conditions TB1, TB2, TB3, which include the regulation for recording a real environment or real situation of the respective real vehicle RF1, RF2, RF3, are determined. The determination of the trigger conditions TB1, TB2, TB3 can include or be their adjustment or updating.

• - Auswahl eines oder mehrerer realer Fahrzeuge RF1, RF2, RF3, die insbesondere zum Ermitteln der ersten Datenabschnitte DA1, DA1', DA1'' genutzt werden können oder sollen; und/oder
• - Ermitteln einer oder mehrerer in realen Fahrzeugen RF1, RF2, RF3, insbesondere in den ausgewählten Fahrzeugen RF1, RF2, RF3, nutzbaren Trigger-Bedingungen TB1, TB2, TB3; und/oder
• - Versenden einer oder mehrerer Trigger-Bedingungen TB1, TB2, TB3 an ein oder mehrere reale Fahrzeuge RF1, RF2, RF3, insbesondere an die ausgewählte Fahrzeuge RF1, RF2, RF3; und/oder
• - Ermitteln DV eines oder mehrerer erster Datenabschnitte DA1, DA1', DA1'' auf Basis der Daten des einen oder der mehreren Fahrzeug RF1, RF2, RF3.

For example, when determining the first data sections DA1, DA1'DA1'', one or more entries from the following list can be executed:

• - Selection of one or more real vehicles RF1, RF2, RF3, which can or should be used in particular to determine the first data sections DA1, DA1', DA1''; and/or
• - Determining one or more trigger conditions TB1, TB2, TB3 that can be used in real vehicles RF1, RF2, RF3, in particular in the selected vehicles RF1, RF2, RF3; and/or
• - Sending one or more trigger conditions TB1, TB2, TB3 to one or more real vehicles RF1, RF2, RF3, in particular to the selected vehicles RF1, RF2, RF3; and/or
• - Determining DV of one or more first data sections DA1, DA1', DA1'' based on the data of the one or more vehicles RF1, RF2, RF3.

• - abhängig von dem Vorgang des Trainierens S2 des neuronalen Netzes DNN1; und/oder
• - abhängig von der Ausgangsschicht des zumindest bis zu einem gewissen Grad trainierten neuronalen Netzes DNN1; und/oder
• - abhängig von den zweiten Datenabschnitten DA2, DA2', DA2'' und/oder
• - abhängig von den von den zweiten Datenabschnitten DA2, DA2', DA2'' ermittelte Daten, insbesondere Reaktionsdaten RD, RDA, RDI, KPI und/oder Analysedaten AD

ausgeführt werden.One or more entries from the list can be

• - depending on the process of training S2 of the neural network DNN1; and/or
• - depending on the output layer of the neural network DNN1, which has been trained at least to a certain extent; and/or
• - depending on the second data sections DA2, DA2', DA2'' and/or
• - depending on the data determined by the second data sections DA2, DA2', DA2'', in particular reaction data RD, RDA, RDI, KPI and/or analysis data AD

be executed.

In other words, one or more (at least partially or temporarily) closed circles or feedback loops can be formed. Optionally, observation, monitoring, control and/or adaptation by a person E (e.g. dispatcher, developer, etc.) as shown schematically in the figure in the P/C/A module can be provided (or prescribed due to guidelines for dealing with artificial intelligence).

• - das Verfahren beschleunigt werden; und/oder
• - dafür gesorgt werden, dass vorrangig Daten ermittelt werden, die tatsächlich verwendet werden bzw. verwendbar sind; und/oder
• - die erforderliche Übertragungsbandbreite bzw. die Speicherungskosten und/oder anfallende Übertragungskosten bzw. Speicherungskosten reduziert werden.

These dependencies can be extremely beneficial for the efficient functioning of the procedure. For example, they can

• - the procedure can be accelerated; and/or
• - ensure that priority is given to identifying data that is actually used or usable; and/or
• - the required transmission bandwidth or storage costs and/or the resulting transmission costs or storage costs are reduced.

4 shows an example of generating the second data sections DA2, DA2', DA2'' by modifying the third data sections DA3, DA3, DA3. In this case, a large number of sets of the second data sections DA2, DA2', DA2'' are generated. These can each be characteristic of a variant V1, V2, V3, ... of a partially virtual environment TVU(V1), TVU(V2) or of an at least partially virtual driving situation.

First, one or more third data sections DA3, DA3', DA3'' are determined, which represent a part of the real situation, in particular the driving situation in the environment of the real vehicle RF. This can be done in real time using a real (possibly operated in real time) vehicle RFRT in an environment RURT recorded in real time using means of the vehicle RFRT. The movement trajectory of the vehicle RF is referred to in this case as TRFRT. Furthermore, the third data sections DA3, DA3', DA3'' can (selectively) have the same features that are described in this document with regard to the first data sections DA1, DA1', DA1''.

Second data sections DA2, DA2', DA2'' can then be generated using the trained neural network DNN1 (in operation of the neural network DNN1). Second data sections DA2, DA2', DA2'' can be generated for several different variants V1, V2, ... of a partially virtual situation. In the 4 is the creation of two different variants of the part wise virtual situation in the partially virtual environments TVU(V1) and TVU(V2).

In this case, different modifications S4b can be carried out, e.g. modifying different parts of the original real environment RURT and/or modifying S4b by means of different virtual objects VO1, VO2, VO3 on the basis of different parameters (e.g. movement parameters, behavior, models), and/or by generating different driving situations by means of one or more virtual objects VO1, VO2, VO3.

When generating S4 such partially virtual situations, in particular almost in real time, the resulting second data sections DA2, DA2', DA2'' can (also) be used at least partially in real time and/or together with one or more systems or data of the real vehicle RF. For example, these can be used as an input variable for operating the vehicle system FS1, FS2, FS3 (directly in the vehicle). For example, the reaction data RD1, RD2, RD3, ... RDA1, RDA2, RDA3, ... RDI1, RDI2, RDI2, ... KPI1, KPI2, KPI3, analysis data AD1, AD2, AD3, and/or control data SD1, SD2, SD3, or parameterization data PD1, PD2, PD3 can be generated using predominantly real components of the vehicle and/or almost in real time. This can be very advantageous, for example, in the context of test operation of a test vehicle.

5 shows selected procedural relationships in an overview.

In operation S3, the neural network DNN1 is controlled by the first data sections DA1, DA1', DA1'' or by third data sections DA3, DA3', DA3' from several vehicles RF1, RF2, RF3 and/or by one or more (predeterminable) targets G and/or differences D that are to be aimed for, in particular to be achieved.

The one or more objectives G and/or differences D can refer to certain qualitative, quantitative and/or statistical characteristics and/or probability measures. These can be expressed in relation to the characteristics of one or more real situations (e.g. appendix of the corresponding first data sections DA1, DA1', DA1'') and/or to previously generated data sections DA2, DA', DA2''. For example, a desired frequency for the occurrence of objects or arrangements of objects (as an absolute measure or in comparison to real situations or (previous) at least partially virtual situations) can be specified.

The goals G and/or differences D are provided or input to the neural network DNN1 as control signals, operating data and/or settings.

The first data sections DA1, DA1', DA1'' are provided in the form of (pre-)processed data. These can be pre-processed, in particular aggregated, using the DV module.

Furthermore, the 5 three different variants of at least partially virtual situations are presented. The second data sections DA2(V1), ... DA2(V2), ... DA2(V3), ..., representing these at least partially virtual situations (at least with regard to selected features) are provided. These are used for an (in this example) at least partially temporally overlapping or parallel generation of the data dependent on the second data sections DA2(V1), ... DA2(V2), ... DA2(V3), .... In this case, respective reaction data RD(V1), RD(V2), RD(V3), ... analysis data AD(V1), AD(V2), AD(V3)..., and/or control data SD(V1), SD(V2), SD(V3), or parameterization data PD(V1), PD(V2), PD(V3) for controlling the vehicle system or corresponding vehicles RF1, RF2, RF3 can be generated (for each variant V1, V2, V3).

Preferably, the neural network DNN1 is trained and/or operated to generate second data sections DA2, DA2', DA2'' for at least partially virtual situations that represent borderline cases. For example, the second data sections DA2, DA2', DA2'' can at least partially characterize virtual driving situations with parameters in a value range close to a critical limit and/or transitions between situations of different types.

For example, a set of the second data sections DA2, DA2', DA2'' is generated for a driving situation that has a medium criticality and/or can only transition to another (e.g. significantly different and/or significantly more critical or significantly less critical) driving situation with a moderate probability. One or more further second data sets DA2, DA2', DA2'' are generated for one or more driving situations that have a gradually higher criticality and/or a gradually increased probability of transitioning to a driving situation of a different type.

Depending on such sets of the second data sections DA2, DA2', DA2'', the reaction data RD, RDI, RDA, KPI of the vehicle system FS1, FS2, FS3, FS1', FS2', FS3' can be generated. These can include a conclusion about which (limit) parameter values a correct or incorrect functionality of the vehicle system FS1, FS2, FS3, FS1', FS2', FS3' is executed, and/or the result of the functionality of the vehicle system FS1, FS2, FS3, FS1', FS2', FS3' leads to the resolution of the driving situation or to the transition to another driving situation.

The generated data can optionally be evaluated, processed, and/or aggregated in the EVAL module.

The data provided can be used to apply one or more principles of data-driven development (DDD). The second data sections DA2, DA2', DA2'' or the data determined based on the second data sections DA2, DA2', DA2''' can be used instead of real data (e.g. instead of the first data sections DA1, DA1', DA1'').

Alternatively or additionally, the training S6 of the second neural network DNN2 of the vehicle system takes place depending on the second data sections DA2, DA2', DA2'' or depending on the data determined by the second data sections DA2, DA2', DA2'''.

• - ein Ermitteln von Trainingsdaten TD2 zum Trainieren S6 des zweiten neuronalen Netzes DNN2; und/oder
• - ein Ermitteln der Backpropagation zur Ausführung des Trainierens S6 des zweiten neuronalen Netzes DNN2; und/oder
• - eine Auswahl oder ein Anpassen eines oder mehrerer Parameters des Trainiervorgangs zum Trainieren des zweiten neuronalen Netzes DNN2;

Depending on the second data sections DA2, DA2', DA2'' or the data determined depending on the second data sections DA2, DA2', DA2''',

• - determining training data TD2 for training S6 of the second neural network DNN2; and/or
• - determining the backpropagation for carrying out the training S6 of the second neural network DNN2; and/or
• - selecting or adjusting one or more parameters of the training process for training the second neural network DNN2;

6 shows schematically an example of the determination of the data determined depending on the second data sections DA2, DA2', DA2'', in particular of reaction data RD, RDA, RDI, KPI, analysis data AD, and/or control data SD or parameterization data PD.

The development stage ES1 of the vehicle system FS1 is operated stationary or detached from the real vehicle (e.g. exclusively) to determine the reaction data RD, RDA, RDI, KPI, analysis data AD, and/or control data SD or parameterization data PD.

The development status ES1 can be a system that corresponds to the real vehicle system of the vehicle RF1, an emulation and/or a further development of at least an essential part of a system of the real vehicle RF1. This development status can be the basis for the next vehicle system FS1 (for a subsequent variant, further development or next generation) for installation, as an update, upgrade or for use as the vehicle system (also to be understood as part of the vehicle system). In particular, the stationary development status ES1 can be a part of the vehicle RF1 (e.g. a "dry run" so to speak) operated, in particular an entire vehicle RF1.

The development status ES1 itself can include signal processing, functional logic, control, regulation and/or a (second) neural network DNN2. In a preferred embodiment, this corresponds to a specific system (planned for use in a real vehicle) or with corresponding functionalities, in particular with a functionality for at least partially automated driving or parking. The development status ES1, ES2, ES3 ... can include a neural network DNN2 to be trained.

The one or more development stages ES1, ES2, ES3, ... do not have to control any movements of a real vehicle RF1 or any actuation of real (physical) actuators or display elements. For example, the development stage FS1 can only cause an actuation of actuators that are also stationary and/or simulated by a data model. Furthermore, an automated evaluation and/or visualization of the actuation of the actuators and/or a corresponding operation of the vehicle can be carried out.

In this example, the development status ES1 is operated with the second data sections DA2, DA2', DA2''. The second data sections DA2, DA2', DA2'', which belong to a set and each represent a driving situation (e.g. a development or further development of a driving situation within a certain time interval), are provided as one or more input variables to the development status ES1.

The development stage ES1 is also operated by virtual data VRF1, which represent a part of a real vehicle or the real vehicle RF1. The data VRF1 can be based on the operating data recorded together with the first data sections DA1, DA1', DA1'', which are characteristic of the operation of the real vehicle, and/or on a (mathematical, computational) model (possibly also a simulation) of the part of the real vehicle VRF1 and made available to the development stage ES1.

Furthermore, one or more actuators of the vehicle (in particular the same or similar actuators as in a real vehicle RF1) can be provided, which are operated (e.g. controlled or regulated) by the development stage ES1. Preferably, at least partially virtual or virtualized actuators VFA1 are used. These can include a model of a real actuator, such as an actuator of the chassis of a real vehicle RF1. Furthermore, one or more real (physical) actuators can also be used (optionally).

The corresponding reaction data RD1 can include data that characterize a feedback of the one or more actuators VFA1 to the development status ES1 and/or a state or a change in state of the one or more actuators VFA1.

Furthermore, information VFI1, e.g. driver information or data for generating driver information, can be generated from the development status ES1. Such information VFI1 can also be counted as reaction data RD1. The information VFI1 can be an output of the development status ES1, such as driver information, which relates to the driving situation represented in the second data sections DA2.

For example, the information VFI can be indicative of what a real vehicle system FS1, FS2, FS3 in a real vehicle RF1, RF2, RF3 would output in a driving situation that approximately corresponds to the at least partially virtual situation represented by the second data sections DA2, DA2', DA2''. This information can also be used to develop, secure, train S6 and/or operate S7 a vehicle system FS1, FS2, FS3.

The corresponding reaction data RD1 may also include additional data that the development stage ES1 outputs to the (virtually represented) rest of the vehicle RF1, RF2, RF3.

One or more of the 6 The components shown can form modules that can be operated in a stationary manner (e.g. as part of a stationary system). The development status ES1, ES2, ES3, ... can each be executed on a computing module M1, processor or processor core P1. This can achieve a particularly effective execution of the method. Several such devices can be operated (at least partially overlapping in time).

The method can comprise parallel operation and/or comparison of the operation, in particular of the reaction data RD1, RDA1, RDI1 of several (each identical or different) development stages ES1, ES2, ES3, ... In this case, identical or different, at least partially virtual situations (e.g. in the form of real situations modified to varying degrees from one or more real environments) can be presented to them.

The reaction data RD, RDA, RDI, KPI can correspond (at least approximately) to the reaction data RD, RDA, RDI, KPI or the aspects to be analyzed of a real vehicle RF that would have been generated in a real situation that corresponds at least partially to the virtual situation. The reaction data RD can therefore correspond, for example, to the reaction data RD, RDA, RDI, KPI of a real vehicle RF1, RF2, RF3 that would be generated if the virtual objects VO1, VO2, VO3 occurred in the real situation. Likewise, the analysis data AD are also (sufficiently) characteristic of the real operation of the vehicle system FS1, FS2, FS3 in the real situation.

In the context of this document, reaction data RD, RDA, RDI, KPI are (also) any operating data of the vehicle system (e.g. incoming and outgoing data) that are created or influenced by the presence, movement, change and/or behavior of the virtual object VO1, VO2, VO3.

For example, the reaction data RD, RDA, RDI, KPI can indicate a decision to (virtually) avoid the virtual object VO1, VO2, VO3, to brake on the virtual object VO1, VO2, VO3, to perform a certain maneuver or to carry out a collision with the virtual object VO1, VO2, VO3 (e.g. with as little damage as possible).

The reaction data RD, RDA, RDI, KPI can, for example, identify parameters of a corresponding path planning (trajectory planning) or an evasive curve, a braking force, a display of the vehicle for the user of the vehicle, etc. The reaction data RD, RDA, RDI, KPI can identify one or more operating parameters of the vehicle system (e.g. the sensors, the control unit, etc.) that are determined on the basis of the modified driving situation or (e.g. essentially only) a difference between the reaction of the vehicle system FS1, FS2, FS3 indicate the situation in the (at least partially real or at least partially virtual) environment of the vehicle TVU modified with one or more virtual objects VO1, VO2, VO3 and the reaction of the vehicle system FS1, FS2, FS3 to the real situation.

For example, the reaction data RD, RDA, RDI, KPI indicate a reaction of the vehicle system FS1, FS2, FS3 to the virtual object VO1, VO2, VO3 or an interaction with the virtual object VO1, VO2, VO3, whereby at least one reaction of the object to the vehicle is simulated. The reaction data RD, RDA, RDI, KPI can (also) be the result of the effect of the virtual object on the vehicle.

Reaction data RD, RDA, RDI, KPI can be generated that a real vehicle RF1, RF2, RF3 would generate in the real environment RU1, RU2, RU3 with a vehicle system FS1', FS2', FS3' corresponding to the development status ES1, ES2, ES3. At least one reaction of the virtual object VO1, VO2, VO3 can in turn be taken into account when determining a parameter of the reaction of the vehicle (to the reaction of the object).

The reaction data (overarching reference symbol “RD”) can also include one or more performance indicators KPI1(V1), KPI2(V1), KPI2(V1), ... achieved in each case. For example, the performance indicators KPI1(V1), KPI2(V1), KPI2(V1) that were achieved in different variants of the partially virtual situation (of one or more variants of the vehicle system FS1, FS2, FS3) are executed. The result of the comparison of the performance indicators KPI1(V1), KPI2(V1), KPI2(V1), ... or data dependent on the performance indicators KPI1(V1), KPI2(V1), KPI2(V1), ... can be used to develop, secure, train and/or operate a vehicle system.

For example, a large number of reaction data RD(V1), RD(V2), RD(V3), ... analysis data AD(V1), AD(V2), AD(V3), ... and/or control data SD(V1), SD(V2), SD(V3), ... or parameterization data PD(V1), PD(V2), PD(V3) can be generated by means of a development status ES1, e.g. on different modules M1, processors or processor cores P1 and/or in the backend BE, in particular at least partially overlapping in time.

For example, a large number of ES1 development levels that are at least essentially or fundamentally the same and/or essentially only differ in terms of parameters can be used as the first development level ES1, ES2, ES3 or as the second development level ES1', ES2', ES3'. These can, for example, be predominantly duplicates, copies or similar emulations that are preferably executed on modules, processors or processor cores that are operated in parallel.

In a further example that can be combined with all features of the invention, several, in particular substantially different, development stages ES1, ES2, ES3 ... or ES1, ES2, ES3 can be used. For example, the development stage ES2 can be a substantially or fundamentally different development stage than the development stage ES1 due to different logic or parameters.

Preferably, the first reaction data RD1(V1), RD1(V2), RD1(V3), ... analysis data AD1(V1), AD1(V2), AD1(V3), ... and/or control data SD1(V1), SD1(V2), SD1(V3), ... or parameterization data PD(V1), PD(V2), PD(V3), ... are generated with a first development level ES1, and the second reaction data RD1(V1), RD1(V2), RD1 (V3), ... analysis data AD1 (V1), AD1 (V2), AD1 (V3), ... and/or control data SD1(V1), SD1(V2), SD1(V3), ... or parameterization data PD(V1), PD(V2), PD(V3), ... are generated with a second development level ES2. The second development level ES2 can differ qualitatively and/or quantitatively from the first development level ES1. However, this can (also) only include or be the next step in the development of the vehicle system.

Different development stages ES2, ES3, ... can be several variants of a development stage ES or of a development stage ES (which forms a genre, for example) formed (according to a certain dependency and/or at least partially randomly or quasi-randomly). In particular, the first development stage ES1 and the second development stage ES2 can differ by a systematically selected and/or (quasi) random change in a parameter of logic, function, etc. For example, on the basis of one, e.g. the most successful development stage ES1, ES2, ES3, further development stages ES1', ES2', ES3', ... can be developed or formed according to a certain dependency.

In other words, (e.g. thousands or millions) of reaction data (e.g. from sets of reaction data RD2(V1), RD2(V2), RD2(V3), ...) can be determined, which characterize a respective reaction of a development stage ES1, ES2, ES3, ES1', ES2', ES3' ... to at least one variant of the second data sections. These can be (further) processed (e.g. in the EVAL module already discussed), in particular subjected to a comparison and/or aggregation. For example, an achieved performance and/or a resulting consequence are determined and compared with corresponding comparative values.

The result of the processing, in particular the comparison, the aggregation of the data can be provided (as the data dependent on the second data sections DA2, DA2', DA2'') for developing, securing, training (S6) and/or operating (S7) the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3').

• Beispielsweise werden unterschiedliche virtuelle Situationen (z.B. durch das unterschiedliche Modifizieren) bzw. entsprechende zweite Datenabschnitte DA2, DA2', DA2'', ... ermittelt (erzeugt), die unterschiedliche Varianten V1, V2, V3, ... einer Fahrsituation in der Umgebung (z.B. des Fahrzeugs VF1) repräsentieren. Dabei werden Varianten V1, V2, V3, ... gebildet, die quantitative und/oder qualitative Unterschiede zueinander aufweisen.

Below are some further examples that can be combined with each other and with all the features described in this document, especially regarding the use of the second data sections and reaction data RD, RDA, RDI, KPI:

• For example, different virtual situations (e.g. by modifying them differently) or corresponding second data sections DA2, DA2', DA2'', ... are determined (generated), which represent different variants V1, V2, V3, ... of a driving situation in the environment (e.g. of the vehicle VF1). In this case, variants V1, V2, V3, ... are formed which have quantitative and/or qualitative differences from one another.

A quantitative difference can, for example, be or include a change in the driving situation with regard to its parameters. A qualitative difference can be a difference in which a fundamentally different result of the driving situation results or is to be expected. For example, the parameters of one or more virtual objects VO1, ... are changed (e.g. step by step from one variant to another variant).

For example, when processing the reaction data RD1(V1), RD1(V2), RD1(V3), ..., the variant V1, V2, V3, ... of a driving situation can be determined in which (e.g. almost or for the first time) a fundamentally different result, in particular a different consequence, results. For example, the parameters of the driving situation can be determined in which (at a development level ES1, ES2, ES3 ...) a certain (e.g. sufficient or insufficient) performance results or a fundamentally different result results. A qualitatively changed result can, for example, include a decision for a fundamentally different maneuver (e.g. swerving instead of braking or braking instead of swerving) and/or the occurrence of a different consequence, such as a collision.

For example, it can be determined at which parameters of a real driving situation in the case of a real vehicle a fundamentally changed result is to be expected (e.g. at which limit parameters the result of the driving situation changes, so to speak), a particularly strong or fundamentally changed decision (e.g. maneuver parameters) and/or consequence results and/or a tolerance limit of a certain development level is observed with regard to parameters of the driving situation.

In a further example, several sets of reaction data RD1(V1), RD1(V2), RD1(V3), ... are determined, which characterize the reactions of a development stage ES1 to a plurality of the variants V1, V2, V3 of modifying a set of the first data sections DA1, DA2, DA3.

The reaction data RD1(V1), RD1(V2), RD1(V3), ... based on several variants of the driving situation represented by the second data sections DA2, DA2', DA2'' are processed and/or compared with each other. For example, depending on the reaction data RD1(V1), RD1(V2), RD1(V3), ... performance indicators KPI1(V1), KPI1(V2), KPI1(V3), ... of the development status ES1 can be determined, which are achieved with different sets of the second data sections DA2(V1), DA2(V2), DA2(V2).

The performance indicators KPI1(V1), KPI1(V2), KPI1(V3), ... and/or the result of the comparison of the performance indicators KPI1(V1), KPI1(V2), KPI1(V3), ... can be used to develop, validate, train and/or operate a vehicle system.

In a further example, several sets of reaction data RD2(V1), RD2(V1), RD2(V3), ... are determined, which characterize the reactions of several development states ES1, ES2, ES3, ... to one or more variants V1, V2, V3, ... of modifying a set of the first data sections DA1, DA2, DA3.

The reaction data RD2(V1), RD2(V1), RD2(V3), ... can characterize the reactions of different development stages ES1, ES2, ES3, ... to the same driving situation and/or to different driving situations. This can be done with a separate or the same set of the second data sections DA2(V1), DA2(V2), DA2(V3). For example, depending on the reaction data RD2(V1), RD2(V1), RD2(V3), ... performance indicators KPI1(V1), KPI2(V1), KPI2(V1), ... of the development stages ES1, ES2, ES3, ... can be determined. These can be processed and/or compared with each other (e.g. according to a predetermined criterion or a predetermined formula).

The performance indicators KPI1(V1), KPI2(V1), KPI2(V1), ... and/or the result of the The performance indicators KPI1(V1), KPI2(V1), KPI2(V1), ... or data dependent on the performance indicators can be used to develop, validate, train and/or operate a vehicle system.

• - Reaktion auf eine teilweise reale und teilweise virtuell modifizierte Umgebung TVU des Fahrzeugs; dies kann z. B. die Umgebung des Fahrzeugs sein, die durch die zweite Datenabschnitte repräsentiert ist;
• - Reaktion auf ein oder auf mehrere virtuelle Objekte VO1, VO2, VO3, insbesondere eine Bewegung und/oder ein Verhalten eines virtuellen Objekts VO1, VO2, VO3; Dies können Datenabschnitte sein, die das eine oder die mehreren virtuellen Objekte VO1, VO2, VO3 repräsentieren bzw. durch diese modifiziert sind;
• - Reaktion auf ein oder mehrere reale Objekte RO1, RO2, RO3 beim Vorhandensein eines oder mehrerer virtuelle Objekte, insbesondere bei einer bestimmten Bewegung und/oder einem bestimmten Verhalten des einen oder der mehreren virtuellen Objekte VO1, VO2, VO3.

In another example, the reaction data RD2(V1), RD2(V1), RD2(V3), ... of the vehicle system can be indicative of a reaction of the vehicle system or the development status to one or more entries from the following list:

• - Reaction to a partially real and partially virtually modified environment TVU of the vehicle; this can be, for example, the environment of the vehicle represented by the second data section;
• - Reaction to one or more virtual objects VO1, VO2, VO3, in particular a movement and/or behavior of a virtual object VO1, VO2, VO3; These can be data sections that represent the one or more virtual objects VO1, VO2, VO3 or are modified by them;
• - Reaction to one or more real objects RO1, RO2, RO3 in the presence of one or more virtual objects, in particular in the event of a particular movement and/or behaviour of the one or more virtual objects VO1, VO2, VO3.

For example, the second data sections DA2, DA2', DA2'' can (also) represent or include a (temporary) concealment of a real object RO1, RO2, RO3 in the environment by a virtual object VO1, VO2, VO3. For example, the second data sections DA2, DA2', DA2'' can be generated in such a way that a driving situation (within the environment VU, TVU) is represented in which an object that is not present (in the real environment of the real vehicle) at least partially conceals part of the environment of the real vehicle RF1, RF2, RF3 and/or in which sensor data is generated that influence, e.g. interfere with, detection of the real object RO1, RO2, RO3.

For example, second data sections DA2, DA2', DA2'' are generated which at least approximately correspond to the sensor data representing at least one region of the environment (e.g. a solid angle and/or a real object) in which the real sensor data would be changed by a virtual object if the virtual object VO1, VO2, VO3 were real RO1, RO2, RO3.

For example, a partial absence and/or a change in the data of an area of the environment VU, TVU can at least model which data would have been caused by a real object RO1, RO2, RO3 if, instead of the virtual object VO1, VO2, VO3, a (corresponding) real object RO1, RO2, RO3, e.g. with approximately the same features or parameters, were present.

For example, a plurality of sets of reaction data RD are processed on the basis of a plurality of second data sections DA2, DA2', DA2'', which were generated in particular by modifying a set of first data sections DA1, DA1', DA1''. There may be a plurality of second data sections DA2, DA2', DA2'' (each different from one another).

For example, several sets of reaction data RD2(V1), RD2(V1), RD2(V3), ... are determined, which characterize the reactions of several development stages ES1, ES2, ES3, ... to several variants of modifying one or more sets of the first data sections DA1, DA2, DA3(V1), DA1, DA2, DA3(V2), DA1, DA2, DA3(V3).

The reaction data RD2(V1), RD2(V1), RD2(V3), ... can characterize the reactions of different development stages ES1, ES2, ES3, ... to the same driving situation and/or to different driving situations. This can be done with one set of the second data sections DA2(V1) or with several sets of the second data sections DA2(V1), DA2(V2), DA2(V3), ... For example, depending on the reaction data RD2(V1), RD2(V1), RD2(V3), ... performance indicators KPI1(V1), KPI2(V1), KPI2(V1), ... of the development stages ES1, ES2, ES3, ... can be determined. These can be processed and/or compared with each other (e.g. according to a predetermined criterion or a predetermined formula).

Such results of the evaluation of the reaction data RD, RDA, RDI, KPI can also be taken into account in the homologation of vehicles RF1, RF2, RF3 with the vehicle system FS1, FS2, FS3, in particular of vehicle systems FS1, FS2, FS3 for at least partially automated driving. For example, a homologation of the vehicles RF1, RF2, RF3 can be carried out at least partially based on the processing and/or comparison of the reaction data RD, RDA, RDI, KPI.

Furthermore, the data based on the reaction data RD, RDA, RDI, KPI can be used to train a (second) neural network DNN2. This variant of the invention is particularly This is advantageous because the method can (also in a targeted manner) generate a very large amount of usable (sufficiently realistic and/or sufficiently diversified) data for training such a neural network or a neural network DNN2 intended for such an application.

The invention can, for example, achieve a reduction in the effort and time required and/or thus enable efficient development, safeguarding, training and further development of driving functions.

Claims (15)

Method for developing, securing, training (S6) and/or operating (S7) a vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3'), comprising: - determining (S1) one or more first data sections (DA1, DA1', DA1'') which characterize one or more real situations of one or more real vehicles (RF1, RF2, RF3); - training (S2) and/or operating (S3) at least one neural network (DNN1) depending on the first data sections (DA1, DA1', DA1''); - generating (S4) second data sections (DA2, DA2', DA2'') which characterize one or more at least partially virtual situations, depending on the output layer of the neural network (DNN1); - Providing (S5) the second data sections (DA2, DA2', DA2'') or data determined depending on the second data sections (DA2, DA2', DA2'') for developing, securing, training (S6) and/or operating (S7) the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3'). Procedure according to Claim 1 ,wherein the first data sections (DA1, DA1', DA1'') are characteristic of: - characteristics of the real objects involved in the real situation (RO1, RO2, RO3); and/or - an arrangement and/or movement parameters of the real objects involved in the respective real situation (RO1, RO2, RO3); and/or - a driving situation in the real environment (RU1, RU2, RU3) of the one or more respective real vehicles (RF1, RF2, RF3); and/or - characteristics of the movement of the one or more respective real vehicles (RF1, RF2, RF3); and/or - boundary conditions in one or more real environments (RU1, RU2, RU3) of the one or more respective real vehicles (RF1, RF2, RF3), in particular roadway parameters and/or weather parameters; and/or - one or more traffic signs and/or traffic rules applicable in one or more real environments (RU1, RU2, RU3). Procedure according to Claim 1 or 2 , wherein the determination (S1) of the one or more first data sections (DA1, DA1', DA1'') - is carried out, in particular controlled, as a function of the process of training (S2) of the neural network (DNN1); and/or - as a function of the output layer of the neural network (DNN1) trained at least to a certain extent; and/or - as a function of the second data sections (DA2, DA2', DA2'') or of the data determined as a function of the second data sections (DA2, DA2', DA2''). Method according to one or more of the preceding claims, wherein the determination (S1) of the one or more first data sections (DA1, DA1', DA1'') is or comprises: - selection of one or more real vehicles (RF1, RF2, RF3); and/or - determination of one or more trigger conditions usable in real vehicles (RF1, RF2, RF3); and/or - sending one or more trigger conditions to one or more real vehicles (RF1, RF2, RF3); and/or - determination of one or more first data sections (DA1, DA2, DA3) based on the data of the vehicle (RF1, RF2, RF3). Method according to one or more of the preceding claims, wherein the determination (S1) of the one or more first data sections (DA1, DA1', DA1'') takes place depending on an activation, a change in an operating mode, and/or a control of: - a sensor; and/or - a processing of the sensor data; and/or - a real functionality, in particular relating to automated driving and/or parking; of the respective real vehicle (RF1, RF2, RF3). Method according to one or more of the preceding claims, wherein the second data sections (DA2, DA2', DA2'') are characteristic of: - features and/or an arrangement of the virtual objects (VO1, VO2, VO3) involved in the virtual situation; and/or - movement parameters of the virtual objects (VO1, VO2, VO3) involved in the virtual situation; and/or - behavior parameters of the virtual objects (VO1, VO2, VO3) involved in the virtual situation; and/or - simulation parameters and/or a situation formation rule for generating the one or more other virtual situations involving one or more virtual objects (VO1, VO2, VO3). Method according to one or more of the preceding claims, wherein the neural network (DNN1) is trained (S2) and/or operated (S3) to modify data sections which characterize a part of a real environment (RU1, RU2, RU3) and/or one or more real objects (RO1, RO2, RO3) with one or more virtual parts and/or one or more virtual objects (VO1, VO2, VO3), wherein the second data sections (DA2, DA2', DA2'') characterize a resulting at least partially virtual situation. Method according to one or more of the preceding claims, wherein the data determined depending on the second data sections (DA2, DA2', DA2'') comprise the - reaction data (RD, RDA, RDI, KPI) of a vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3') to the second data sections (DA2, DA2', DA2''); and/or - analysis data (AD1, AD2, AD3) relating to the second data sections (DA2, DA2', DA2'') and/or relating to the reaction data (RD, RDA, RDI, KPI); and/or - control data (SD1, SD2, SD3) and/or parameterization data (PD1, PD2, PD3) for operating (S7) the vehicle system (FS1, FS2, FS3). Method according to one or more of the preceding claims, wherein the one or more real situations relate to a real driving situation of the real vehicle (RF1, RF2, RF3); and/or the one or more virtual situations relate to a virtual driving situation of a virtual vehicle (VF1). Method according to one or more of the preceding claims, wherein the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3') is a driver assistance system and/or a system for carrying out or supporting at least partially automated longitudinal guidance, transverse guidance and/or maneuvering of a vehicle, in particular a system for at least partially automated driving, maneuvering and/or parking. Method according to one or more of the preceding claims, wherein a comparison of the reaction data (RD, RDA, RDI, KPI) of a vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3') determined as a function of several second data sections (DA2, DA2', DA2'') to the second data sections (DA2, DA2', DA2''); and/or - analysis data (AD1, AD2, AD3) relating to the second data sections (DA2, DA2', DA2'') and/or relating to the reaction data (RD, RDA, RDI, KPI); and/or - control data (SD1, SD2, SD3) and/or parameterization data (PD1, PD2, PD3) for operating (S7) the vehicle system (FS1, FS2, FS3); is carried out and the result of the comparison is used to develop, secure, train (S6) and/or operate (S7) the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3'). Method according to one or more of the preceding claims, wherein the training (S6) of a vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3') comprises training a second neural network (DNN2) of the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3') depending on the second data sections (DA2, DA2', DA2''), wherein the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3') is trained: - to control a virtual vehicle (VF1) within the at least partially virtual environment (VU, TVU), in particular to drive and/or park it at least partially automatically; and/or - to strive for a specific goal when operating the virtual vehicle (VF1) in the virtual environment (VU, TVU). Method according to one or more of the preceding claims, wherein the determination (S1) of the one or more first data sections (DA1, DA2, DA3) takes place depending on: - reaction data (RD, RDA, RDI, KPI) of a vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3') to the second data sections (DA2, DA2', DA2''); and/or - analysis data (AD1, AD2, AD3) relating to the second data sections (DA2, DA2', DA2')' and/or relating to the reaction data (RD, RDA, RDI, KPI); and/or - control data (SD1, SD2, SD3) and/or parameterization data (PD1, PD2, PD3) for operating (S7) the vehicle system (FS1, FS2, FS3). System for developing, securing, training (S6) and/or operating (S7) a vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3'), wherein the system is designed to determine (S1) one or more first data sections (DA1, DA1', DA1'') which characterize one or more real situations of one or more real vehicles (RF1, RF2, RF3), and to train (S2) and/or operate (S3) at least one neural network (DNN1) depending on the first data sections (DA1, DA1', DA1''); and is designed to generate (S4) second data sections (DA2, DA2', DA2'') which characterize one or more at least partially virtual situations, depending on the output layer of the neural network. Network (DNN1), and is further designed to provide (S5) the second data sections (DA2, DA2', DA2'') or data determined depending on the second data sections (DA2, DA2', DA2'') for developing, securing, training (S6) and/or operating (S7) the vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3'). Neural network (DNN1) for developing, securing, training (S6) and/or operating (S7) a vehicle system (FS1, FS2, FS3, ... FS1', FS2', FS3'), wherein the neural network (DNN1) is designed to generate (S4) one or more second data sections (DA2, DA2', DA2'') which characterize one or more at least partially virtual situations, and wherein the neural network (DNN1) is trained (S2) and/or operable (S3) depending on one or more first data sections (DA1, DA1', DA1'') which characterize one or more real situations of one or more real vehicles (RF1, RF2, RF3).

Images