DE102020130748A1 - Method, system and a computer program for generating a virtual environment of a vehicle - Google Patents

Abstract

One aspect of the invention relates to a method for generating a virtual environment of a vehicle. One step of the method includes determining at least two data sections (DA1, DA2), each of which represents at least part of a real environment (RU1, RU2) of a real vehicle (RF1, RF2). A further step of the method includes generating the virtual environment (VU12a, VU12b) from or on the basis of the at least two data sections (DA1, DA2).

Description

The invention relates to a method, system and a corresponding computer program for generating a virtual environment of a vehicle. In particular, the invention relates to the use of a virtual environment of a vehicle for developing, validating and/or training vehicles or vehicle systems. This can include operating the virtual environment and a corresponding development operation or validation operation for vehicles based on a virtual environment.

In the development, in particular the safeguarding of modern vehicles, a very large number of diverse situations (e.g. driving situations), states of several systems and functionalities of the vehicles, as well as conditions to which vehicles are exposed during their later operation, must be taken into account. Therefore, the effort, costs and time required for the corresponding development and validation with a further increasing complexity of the systems and functionalities of modern vehicles can increase extremely (also strongly non-linearly) and hinder progress in the automotive industry. This is particularly important for the development, validation and market launch of at least partially automated vehicles.

In the context of the document, the term "automated driving" can be understood as driving with automated longitudinal or lateral guidance or autonomous driving with automated longitudinal and lateral guidance. The term "automated driving" includes automated driving with any degree of automation. Exemplary degrees of automation are assisted, partially automated, highly automated or fully automated driving. These degrees of automation were defined by the Federal Highway Research Institute (BASt) (see BASt publication "Research compact", issue 11/2012). With assisted driving, the driver constantly performs longitudinal or lateral guidance, while the system takes over the other function within certain limits. With semi-automated driving (TAF), the system takes over longitudinal and lateral guidance for a certain period of time and/or in specific situations, whereby the driver has to constantly monitor the system, as with assisted driving. With highly automated driving (HAD), the system takes over longitudinal and lateral guidance for a certain period of time without the driver having to constantly monitor the system; however, the driver must be able to take control of the vehicle within a certain period of time. With fully automated driving (VAF), the system can automatically handle driving in all situations for a specific application; a driver is no longer required for this application. The four levels of automation mentioned above, as defined by BASt, correspond to SAE levels 1 to 4 of the SAE J3016 standard (SAE - Society of Automotive Engineering). For example, according to BASt, highly automated driving (HAF) corresponds to level 3 of the SAE J3016 standard. Furthermore, SAE J3016 also provides SAE Level 5 as the highest degree of automation, which is not included in the BASt definition. SAE Level 5 corresponds to driverless driving, in which the system can automatically handle all situations like a human driver throughout the journey; a driver is generally no longer required.

In particular, for the development of automated driving functions, a very large number of vehicle environments that occur or are possible in reality must be taken into account.

Especially with the increase in the desired degree of automation of at least partially automated vehicles, an increased number of relevant driving situations, e.g. arrangements, constellations of road users, which have to be handled appropriately by the vehicles, is extremely (also extremely non-linear). According to the state of the art and the legal framework for the homologation or market launch of the vehicles, this results in an enormous distance of several billion kilometers, which would have to be covered within the framework of testing and validation of such vehicles mostly error-free or with a respectively appropriate performance.

Therefore, the object of the present invention is to show a method, device and a corresponding computer program for generating a virtual environment of a vehicle. In particular, the object of the invention relates to the generation of a virtual environment that can be used for the development, testing or validation of vehicles. In particular, the object of the invention can include using (also to be understood as operating) a virtual environment (also to be understood as a corresponding development operation or validation operation based on the virtual environment).

The object of the invention is solved by the features of each of the independent patent claims. Advantageous embodiments are described in the dependent claims. It is pointed out that additional features of an independent patent claim dependent patent claim without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim can form a separate invention independent of the combination of all features of the independent patent claim, which is made the subject of an independent claim, a divisional application or a subsequent application can. This applies equally to the technical teachings described in the description, which can form an invention independent of the features of the independent patent claims.

A first aspect of the invention relates to a method for generating a virtual environment of a vehicle.

One step of the method includes determining at least two data sections, each of which characterizes, in particular represents, at least part of a real environment of a real vehicle. This is also to be understood in such a way that the at least two data sections are each representative of at least part of a real environment of a real vehicle.

The at least one data section may include data based on one or more sensors of the respective vehicle. For example, such a data section can be determined on the basis of a recording (also to be understood as a recording of a corresponding trace or snippet) of the data from one or more sensors of the vehicle.

• • zumindest einen Raumteil, der insbesondere oder nach bestimmten (z.B. dynamischen) Kriterien gewählt wird, z.B. in Bezug auf das jeweilige reale Fahrzeug (z.B. aus Blind-Spot-Bereichen),
• • zumindest ein Objekt in der Umgebung des realen Fahrzeugs bzw. einen Teil der Umgebung des realen Fahrzeugs mit diesen Objekten,
• • eine bestimmte Situation, z.B. Fahrsituation in der sich das reale Fahrzeug befindet,
• • eine oder mehrere Anordnungen, Aktionen, Interaktionen zwischen einem Objekt in der Umgebung und dem realen Fahrzeug und/oder den Objekten in der Umgebung, und/oder
• • Daten, insbesondere Eingangsdatendaten, Zustand und/oder Ausgangsdaten eines Systems des realen Fahrzeugs.

For example, the data sections may represent:

• • at least one part of the room that is selected in particular or according to specific (e.g. dynamic) criteria, e.g. in relation to the respective real vehicle (e.g. from blind spot areas),
• • at least one object in the area surrounding the real vehicle or part of the area surrounding the real vehicle with these objects,
• • a specific situation, e.g. driving situation in which the real vehicle is located,
• • one or more arrangements, actions, interactions between an object in the environment and the real vehicle and/or the objects in the environment, and/or
• • Data, in particular input data, state and/or output data of a system of the real vehicle.

Particularly preferably, the at least one or two (of the at least two) of the data sections represents part of the real environment of a vehicle, a dynamic process and/or a process that extends in time and/or space. The process can relate to individual, specific objects within the environment of the vehicle. Such a process can be a maneuver relating to one or more (real) objects. For example, it can be a maneuver performed in the vicinity of the respective real vehicle by one or more other road users.

A maneuver can be, for example, a lane change maneuver, merging maneuver, an evasive maneuver, a merging maneuver, an overtaking maneuver, a parking maneuver, a parking maneuver, a maneuvering maneuver or the like. In particular, it can be a maneuver of one or more objects, e.g. road users in the respective environment, relating to the (respective) real vehicle.

Alternatively or additionally, the at least one or two (of the at least two) of the data sections may represent a specific process performed by the (respective) real vehicle. This can be a dynamic process and/or a process that extends in terms of time and/or space. For example, the at least one or two (of the at least two) of the data sections represents an interaction of the respective vehicle with an object. In another example, such interaction may include interference, interference, interference, or, exceptionally, physical interaction such as interference, friction, or collision. In particular, the at least one or two (of the at least two) of the data sections represents a specific driving situation relating to the vehicle and/or an object from its surroundings.

In the context of this document, the term "represent" is to be understood as "identify". For example, one or more of the data portions may, but need not, represent (more or less explicitly) the absolute or relative coordinates, movements of the vehicle, and/or objects in its environment. These can be recognized, for example, by means of recognized patterns and/or represented in the data sections (e.g. by representative numbers, codes, patterns), in particular identified.

For example, said data sections can each represent one or more objects, arrangements (constellations) of objects or a specific relation of an object to the real vehicle. For example, represent A data section indicates a specific road user or a road user of a specific type, eg a motorcycle, car, truck in the vicinity of the vehicle. This may be presented in the data sections as an identifier of a class of the object, a dimension or shape, a relative position, angle, orientation, absolute or relative speed, acceleration, detected intent, and/or a specific action or an action of a specific type are represented in the respective data section.

For example, the data sections can each represent a specific and/or selectable part, e.g. spatial part, angle, and/or distance range from the real vehicle. For example, a data section represents a road user who is executing a maneuver relating to the real vehicle (e.g. overtaking it), a parameter, in particular a change in the parameters during the manoeuvre.

For example, the data sections can each represent specific arrangements, actions, interactions between an object in the environment and the real vehicle and/or the objects in the environment, in particular with regard to their dynamics, e.g. through a change or course of the respective parameters.

For example, the data section represents part of the environment behind the real vehicle, to the side of the real vehicle, in a blind spot of the real vehicle, on the planned trajectory of the real vehicle.

The at least two of the data sections, which each represent real surroundings (each) of a real vehicle, can originate from different time intervals, locations, situations, boundary conditions and/or vehicles. For example, there can be at least two driving situations in the same area (e.g. at the same or similar exit, driveway, roundabout, parking lot, etc.) with two different real vehicles at different time intervals.

In particular, the at least one data section represents a real environment of a real vehicle that occurs less frequently than once in 10, 100, 100, 10,000, 1,000,000 kilometers in real vehicle operation in an identical or similar variant (e.g. statistically or on average) or is to be expected. For example, the data sections, in particular relevant, selected recordings of the data of an interface, an on-board network communication, can include or be a (e.g. temporal and/or spatial) course of an internal parameter of a computing unit of the vehicle. In a simplified case, this can at least partially include or be data based on a radar, a camera, a LIDARS or already processed or merged data (such as data from a so-called environment model). For example, the data sections can initially be stored in a ring memory of the vehicle and determined when the at least one trigger condition applies (retrospectively, so to speak).

For example, the content, in particular the number of specific parameters, e.g. signals, and/or the duration of a recording for a data section can be or can be selected depending on a respective trigger condition. The data sections are in particular data sections that are not required for the operation of the vehicle (with which they are recorded) or are functionally used there. For example, depending on the at least one trigger condition, an activation of a sensor of the vehicle and/or a change in the operating mode of a sensor system can be initiated. This can be used predominantly, essentially only or exclusively to determine the data sections.

In one example of the method, the determination of the data sections can relate at least primarily to a real environment for which it has been determined or recognized that this occurs particularly rarely and/or differs from one or more frequently occurring, particularly typically occurring environments, e.g. driving situations .

In particular, the at least one trigger condition can include a criterion relating to a (e.g. statistical) frequency relating to the occurrence of specific parameters in the surroundings of real vehicles. For example, the at least one trigger condition can include a comparison of the data of an environment of a real vehicle with specific, particularly frequent and/or typically occurring real environments. In particular, depending on the result of the comparison, at least predominantly rare and/or atypical environments can be selected and the data sections representing them can be recorded and/or further processed.

A further step of the method depends on the generation of the virtual environment, in particular from or on the basis of the at least two data sections. The virtual environment can refer to a virtual vehicle (eg not driving at least at the time interval or to a vehicle that does not exist per se or is represented predominantly only by a position). In a simplified way In this case, the virtual surroundings of the vehicle can also be determined and/or used without the actual vehicle. The virtual environment can include or be data sections of the virtual environment that represent the virtual environment, in particular data sections that represent the virtual environment from the (virtual) detection perspective (so to speak, the visual perspective) of the (virtual) vehicle.

Alternatively or additionally, the virtual environment can include or be data for executing a simulation and/or a data generator. Depending on the virtual environment, such a simulation or data generator can in particular generate data sections, e.g. relating to sensor data, processed or fused sensor data and/or data of an environment model, which would have been determined for a real vehicle (if the virtual environment were a real environment).

Preferably, a vehicle to which the virtual environment relates (so to speak, the virtual EGO vehicle) itself can be generated as data representing it. For example, the (ego) vehicle can be mapped or operated within the virtual environment as corresponding data, in particular a data model.

The virtual environment can preferably be generated on the basis of two or more (preferably a large number) of different data sections.

As a result of the method, for example through the inventive processing, in particular combination or synthesis of the data from the (“real”) data sections, virtual environments, in particular representing driving situations of a vehicle, can thus be generated that are rarer in real or realistic operation of the vehicles than 100, 10000, 10 ⁸ , 10 ¹² kilometers are or would be expected.

Furthermore, at least one system of a vehicle (e.g. to be developed) can be operated with the virtual environment. The vehicle's system may be one or more controllers, software modules for a controller, a simulated or emulated device of the vehicle, or a (substantially entire) vehicle.

The invention makes it possible to operate the vehicle system (at least virtually, so to speak in “dry run”) using the virtual environment, in particular to test, secure and/or train it. In the process, virtual environments (e.g. relevant to proof of the performance, approval and/or market launch of the vehicle) can also be generated that include very rarely occurring situations and/or boundary conditions. For example, a detection of errors, performance weaknesses, or proof of performance security or reliability relating to at least one system of the vehicle through the use (in particular operation of the vehicle system) of the virtual environment (also to be understood: on the basis of the virtual environment dependent data).

As a result, a real route to be completed, e.g. as part of a development, test, validation and/or training of vehicles, can be significantly reduced. For example, time, effort, and costs can be saved and/or improved validation of the at least one system of the vehicle (to be developed, validated, or trained) can be achieved.

The creation of the virtual environment preferably includes a selection of extraction and/or exclusion of data, e.g. relating to individual objects and/or processes, from at least one of the at least two data sections. For example, the virtual environment can only represent a (chosen, selectable) part of the respective real environment.

Data representing the virtual environment can preferably be determined that (at least in total) were not recorded in reality, each based on a (specific or selected according to specific criteria) first part of a first real environment, in particular situation, of a vehicle AND based on a second part from a second real environment, in particular situation, of a vehicle.

Generating the virtual environment (or the data representing or corresponding to it) can include adding, replacing, integrating and/or merging specific parts or parts selected according to specific criteria from the at least two data sections, each from at least two different Environments, vehicles and/or time intervals originate. In particular, a part of a first real environment of a (first) vehicle and a part of a second real environment of a (second) vehicle are used.

The generation of the virtual environment can be a determination of the data from at least two data sections that were determined using at least two different trigger conditions.

• • ein neues Objekt, eine neue Kombination und/oder Anordnung von Objekten,
• • ein aus zumindest zwei Objekte kombiniertes Objekt, und/oder
• • eine neue oder (aus zumindest zwei realen Situationen) erzeugte Situation.

The at least one virtual environment can represent a (new, virtual, represented by data) situation, in particular a driving situation. The at least one virtual environment can include, for example:

• • a new object, combination and/or arrangement of objects,
• • an object combined from at least two objects, and/or
• • a new situation or a situation created (from at least two real situations).

For example, the data of the new environment includes the data from sensors of at least one of the at least two of the real vehicles that represent a (new) virtual environment. This can be chunks of data that would represent a virtual environment if it had been real.

The virtual environment or the data of the virtual environment can be represented in relation to a position or a (virtual) object of a (ego) vehicle (so to speak, from the visual position of the vehicle which would determine this).

For example, the virtual environment is essentially independent or detached from (absolute) position information. Rather, this data can characterize that a real vehicle (with certain features) would be determined if it were located in a corresponding real environment, in particular if it were an environment of a real vehicle.

The invention also includes the idea of waiting for the occurrence of a specific (e.g. extremely rare) environment of a vehicle with a driving situation that corresponds to specific criteria, which is suitable for the development, validation, and / or training of a vehicle system or which is still missing to create a corresponding virtual environment. This can also include determining the corresponding data necessary for generating the environment (in particular individually, using different vehicles, with respectively suitably selected trigger conditions). In doing so, data and knowledge can be gained that are necessary for the development, validation, training of a vehicle system and/or proof of the positive risk balance, but for which it would otherwise have been necessary to drive in billions of kilometers of the test fleet.

With the resulting virtual environment, both a vehicle system in the vehicle and/or a stationary system that emulates the vehicle system (e.g. a development status) can be operated at least partially and/or temporarily with the data of a virtual environment generated in this way and/or to train.

Furthermore, such a method can also be carried out using (real) vehicles already in customer operation (the customer vehicles). This results in the advantage that the mileage of very many (e.g. hundreds of thousands) vehicles (to a total of several billion kilometers) can be practically added up or can be made usable overall. The determination or selection of the corresponding data sections can be done “idle” so to speak, in a “shadow mode” without showing a functional effect or disturbing the respective vehicle customer or the third party.

In particular, the one or more of the data sections can be determined by means of a vehicle system or vehicle functionality that is at least partially decoupled from an execution of a functionality effective for the vehicle or for an occupant of the vehicle, at least in the time interval, or parallel to the execution of a (their actual) functionality is operated in the respective real vehicles.

For example, the method can be carried out using an at least partially decoupled mode (set up for this purpose) of the corresponding sensors and/or functions. The data obtained from the new environment (this is also a new, artificially constructed, designed, synthesized driving situation as described here) can be used, for example, to demonstrate a positive risk balance and/or to train neural networks.

Based on the data sections that come from different times, places, situations and/or vehicles (and represent part of a respective environment of the vehicle), data can be generated that contain a new, artificially generated (desired for various purposes but not really existing ) represent the surroundings of the vehicle.

For example, a desired environment or situation can be created as the corresponding data from data sections of the cameras, radars, LIDARe, ultrasonic sensors, thermal cameras, pre-processed data, so to speak, by calculation. It can be created from parts that correspond to camera recordings, radar sequences (“radar video”), etc., for example, in the same way as a video cut.

The resulting data on the new environment can be used in a stationary calculation unit, eg within a simulation, processed, in particular reprocessed. For example, a stationarily operated system of the vehicle (control device, for example at a test station with the same software and/or alternative software) can be operated with the data from the new environment. Alternatively, a computing unit that emulates (simplified: replicates) a system of the vehicle (corresponding to the vehicle or new) can be operated with the data of the new environment.

For example, the data of the new surroundings, in particular situations, can be determined using appropriately set up game consoles (or computing units with a comparable architecture). Alternatively or additionally, the game consoles can be used for further processing of the data, e.g. for determining the risk balance, identifying situations that can or cannot be controlled with certain algorithms. Furthermore, game consoles (or computing units with a comparable architecture) can be used to emulate a system of the vehicle. An extreme number of situations that never existed in this way can be played through virtually on the game consoles.

The corresponding output data of a (e.g. stationary) system of the vehicle can be obtained from the data of the new (artificially generated) surroundings, in particular situations, in particular determined. In particular, the resulting output data represents a reaction of an emulated (e.g. in a stationary computer) system of the vehicle to the data of the one or more new environments. The term “system of the vehicle” can essentially be understood to mean the same system as the system installed in (or the) real vehicle, an alternative, further developed system and/or a system to be trained.

These can (at least approximately) correspond to the output data, e.g. the reaction data of a real vehicle, as if the data corresponded to a real situation. For example, as if a certain combination of objects (which come from two different situations) occurred in one situation. In the context of this document, the output data (also) are any operating data of the vehicle system that arise or are influenced by the change in the data of one environment by the data of a new environment.

In an advantageous embodiment, the method includes the step of providing at least one trigger condition for the real vehicle, and the step of determining the at least two data sections as a function of the at least one trigger condition.

The trigger condition is, for example, sent to the vehicle and/or determined in the vehicle (e.g. according to a predetermined condition with or without a parameter sent to the vehicle relating to the trigger condition). The trigger condition is used to define and/or control which of the conditions identified in this condition is to be used to determine a data section. Alternatively or additionally, the composition of the data section is defined and/or controlled with the trigger condition.

This trigger condition can identify a parameter, in particular a pattern, from the occurrence of one or more specific objects in the area surrounding the vehicle, a specific situation, e.g. driving situation in which the vehicle is located, one or more arrangements, actions, interactions between an object in the Environment and the vehicle and / or the objects in the environment, and / or data, in particular input data data, state and / or output data of a system of the vehicle.

It can be an intelligent trigger condition that e.g. links one or more of the above parameters, in particular patterns. In particular, the trigger condition identifies a rule according to which a data section (e.g. from a specific vehicle or from a vehicle selected according to a specific condition) is determined.

If, for example, one of the trigger conditions applies, data sections that are predetermined or determined by the trigger condition and characterize a section of the area surrounding the vehicle and/or data from a system of the vehicle, in particular in a compressed and/or coded form, are sent to a remotely located computing unit (e.g. so-called backend), transmitted.

The at least two data sections are preferably determined using at least one trigger condition at different time intervals and/or vehicles.

For example, a motorcyclist is searched for using a trigger condition. At least two motorcyclists are found at different time intervals. Corresponding data sections are determined (individually). These can be processed into a modification of the current situation and/or the synthesis of a new situation.

In a further advantageous embodiment, the method includes the provision of at least two trigger conditions for the real vehicle and the determination of the at least two data sections as a function of different trigger conditions in each case.

This can, for example, also be understood to mean that a script is transmitted to the vehicle and/or executed in the vehicle. The script can include at least two trigger conditions.

For example, a first data section representing a first road user who is e.g. in a specific first spatial area (e.g. within specific angles and/or distances) from the vehicle has a first parameter, e.g. speed, acceleration, curve radius, and/or a specific action executes, records.

For example, a second data section representing a second road user who is e.g. in a specific second spatial area (e.g. within specific angles and/or distances) from the vehicle has a second parameter, e.g. speed, acceleration, curve radius, and/or a specific action executes, records. These result in a very large and/or advantageous combination possibility.

For example, based on the data sections, one or more virtual environments can be generated (integrated, as it were), which represent one or more new (e.g. virtually generated) situations. The at least one virtual environment can, for example, include driving situations that in reality did not occur in any of the real vehicles or could only be expected with a very low probability.

In a further advantageous embodiment, the invention includes the transmission of the data sections to a computing unit located at a distance from the real vehicle. In particular, a simulation based on the data sections can be or can be operated in the remote computing unit (backend, server, data center, etc.).

In a further advantageous embodiment, the method includes generating the virtual environment from the at least two data sections according to a situation formation rule from (also to be understood as dependent on or on the basis of) the at least two data sections.

The at least two parts of the at least two data sections can be added, replaced, integrated and/or combined according to a situation formation rule.

The situation formation rule can include or be a rule that is predetermined and/or dynamically determinable or adaptable in the method.

The situation formation rule preferably includes a rule for synthesizing the virtual environment based on the data of the at least two of the data sections.

For example, the situation formation regulation can identify an application of specific (predetermined and/or determinable in the process) criteria, e.g. for the selection of objects, parts of the surroundings, the behavior of objects, the situation pattern.

Preferably, the situation formation rule that is predetermined and/or that can be determined in the method can specify the formation of the data representing a new situation having a predetermined feature, in particular parameters.

In particular, the situation formation rule can include at least a first criterion in relation to the frequency (also to be understood as “rarity”) of one or two of the at least two of the real surroundings (real vehicles).

In particular, said situation formation rule can determine and take into account a statistical frequency relating to the occurrence of specific driving situations. In particular, the situation formation rule can include at least a first criterion in relation to the frequency (also to be understood as “rarity”) of one or two of the at least two of the real surroundings (real vehicles).

For example, to generate the virtual environment, at least primarily data sections can be selected which (at least tendentially or predominantly) rarely occur environments of a real vehicle, e.g. comparatively rarely occurring driving situations. In particular, the situation formation rule can include at least one criterion in relation to the virtual environment to be generated.

For example, at least primarily virtual environments can be generated which (at least tend to) represent (in particular simulate) environments of a real vehicle that rarely occur in real vehicle operation. Furthermore, several (e.g. basically similar or the same, specific criterion or sample corresponding) variants to a virtual environment. A virtual environment can in particular identify several variants of the virtual environment (eg basically similar or corresponding to a certain criterion or pattern), eg several variants of a process in the environment of the (virtual) vehicle that differ more or less from one another.

In other words, a situation, e.g. The data of a new virtual situation can then be “constructed” or “synthesized” based on the data from several different real situations. It can also be just a specific pattern for which one or more variants of a virtual environment, each with a plurality of variants of a situation, driving situation, are generated.

For example, the predetermined rule and/or the rule that can be determined in the method can specify the formation of the data for a new situation, which in particular includes specific objects in the area, a specific situation, in particular a driving situation, one or more arrangements, actions, interactions between an object in the area and the vehicle and/or the objects in the environment, and/or data, in particular input data data, status and/or output data of a system of the vehicle.

The predetermined regulation and/or the regulation that can be determined in the method particularly preferably characterizes a specific pattern according to which the new environment, in particular a new situation, can be generated.

For example, data can be determined on a new situation (formed in the method, perhaps never occurring in the physical world) in which, for example, two motorcycles are in the blind spot of the vehicle, overtake the vehicle from two sides at the same time, or drive too close to each other .

Alternatively or additionally, a first object (also to be understood as data sections corresponding to the corresponding objects) that existed in a first situation can be at least partially removed and/or replaced by a second object (also to be understood as data sections corresponding to the corresponding objects) can be replaced from a second situation or merged with it to create a new object.

The at least one new environment, in particular situation, is particularly preferably generated depending on the (already determined or processed) output data of a system on a real environment or a new environment and/or depending on the operation of the neural network.

For example, a neural network, in particular for itself and/or for a new learning cycle, can (co-)determine or control a definition of a trigger condition, a selection of the data sections, a combination of the data sections, generation of the data of the at least one new environment.

In a further embodiment, the invention includes the step of generating the virtual environment from the at least two data sections according to a situation formation rule from the at least two data sections by adding, replacing, integrating and/or merging objects, parts of objects, parts of space in relation to the real one Vehicle, actions, action parameters and/or behavior from the at least two data sections.

In particular, the invention includes modifying data about an object within the data of the new environment. The data relating to an object within the data of the new environment is particularly preferably generated from at least two data sections identifying at least two different objects from the at least two data sections.

For example, data sections representing different heads or faces from one or more second surroundings can be added to a pedestrian from a first environment (a first vehicle).

For example, a trailer can be added to a vehicle.

Output data of a specific system of the vehicle can then be determined (e.g. by means of what is known as reprocessing). These can be characteristic, for example, of the performance of a specific method or parameters of the method and/or risks that are to be expected when the specific method is actually used in real vehicles.

At least one neural network can also be trained on the basis of the data from the new surroundings.

In a further advantageous embodiment, the method includes changing the at least one trigger condition, at least one feature of the virtual environment and/or the situation formation rule.

At least one trigger condition for determining at least one of the data sections (e.g. the rule for detecting a real environment of a real vehicle), a feature of a new environment to be determined, and/or the situation formation rule is particularly preferred.

This can be changed depending on the (already determined or processed) output data and/or depending on the operation of the neural network.

For example, the neural network influences or controls the creation, properties, configuration, actions, reactions of new situations. For example, at least one first data section is added, replaced and/or integrated, in particular characterizing a first object, a first part of space, a first behavior parameter in and/or with a second data section, in particular characterizing a second object, a second part of space, a second behavior parameter depending on the (already determined or processed) output data; and/or depending on the operation of the neural network.

The neural network can already be at least partially trained or be in the process of training;

The neural network can preferably decide or participate in deciding which new environments, in particular objects, combinations or constellations of objects, behavior of objects, situations are necessary or advantageous. This can be done by the neural network in recursive training mode.

For example, the neural network can specify certain "data sections", e.g. from several vehicles, which correspond to certain criteria, patterns or trigger conditions. In other words, depending on an output of the neural network, it can be decided which data sections, e.g. of which objects or situations from real vehicles, are to be determined, which can then be appropriately combined, in particular for a specific application.

In a further combinable example, the neural network can specify how or according to which criteria the data sections are to be added, replaced and/or integrated.

A further new situation (e.g. a dynamic environment of the vehicle) (useful or required for further use, in particular for a specific use, e.g. for training a neural network) can be determined.

The advantage of the invention is that data on new situations can be obtained relatively quickly, which would only occur in several decades or billions of kilometers driven. A particular advantage results for the development, validation, training and/or operation of a system of the vehicle for at least partially automated driving.

A further advantageous embodiment of the invention is the use of a virtual environment of a vehicle generated according to one of the preceding claims for testing and/or for validating a development status of a system of a vehicle.

• • zum Test und/oder Absicherung eines Entwicklungsstands eines Systems eines Fahrzeugs.
• • zum Ermitteln zumindest eines Performanceindikators, z.B. KPI des Systems des Fahrzeugs.
• • zum Nachweis einer positiven Risikobilanz des Systems des Fahrzeugs, insbesondere für einen Rückschluss auf die erwartete Performance in Bezug auf reale Objekte.
• • zum Betreiben eines weiteren Fahrzeugs, z.B. einer Funktionalität eines weiteren Fahrzeugs.
• • zum Trainieren eines neuronalen Netzes, wobei die resultierenden Lerndaten (Gewichtsdaten, Bias Terms, Checkpoints) insbesondere zum Betreiben eines realen Fahrzeugs einsetzbar sind.
• • zum Freischalten und/oder Sperren von Leistungsmerkmalen zumindest eines Fahrzeugs, insbesondere mit dem gleichen oder ähnlichen Systemen des Fahrzeugs.
• • zum Freischalten und/oder Sperren von (bestimmten) realen Reaktionen des Systems des Fahrzeugs für künftige (reale) Situationen die der zumindest einer neuen Situation hinreichend ähnlich sind.
• • zum Verändern einer digitalen Karte und/oder der Daten zur Interpretation einer digitalen Karte, insbesondere zur Freigabe, Sperrung bestimmter Fahrbahnabschnitte für die Ausführung bestimmter Leistungsmerkmale, insbesondere bestimmter Reaktionen auf bestimmte reale Situationen die der zumindest einer neuen Situation hinreichend ähnlich sind.

The virtual environments or information based on the one or more virtual environments can be provided and/or used for:

• • to test and/or validate a development status of a vehicle system.
• • to determine at least one performance indicator, eg KPI of the system of the vehicle.
• • to prove a positive risk balance of the vehicle's system, in particular to draw conclusions about the expected performance in relation to real objects.
• • to operate another vehicle, eg a functionality of another vehicle.
• • for training a neural network, the resulting learning data (weight data, bias terms, checkpoints) being usable in particular for operating a real vehicle.
• • to enable and/or block performance features of at least one vehicle, in particular with the same or similar vehicle systems.
• • to enable and/or block (certain) real reactions of the vehicle's system for future (real) situations that are sufficiently similar to at least one new situation.
• • for changing a digital map and/or the data for interpreting a digital map, in particular for enabling or blocking certain sections of the road for the execution of certain performance features, in particular certain reactions to certain real situations which are sufficiently similar to at least one new situation.

The invention also includes the above-mentioned or other systems or functionalities in vehicles or systems or functionalities for operating vehicles that are developed, secured, trained, executed or made executable on the basis of the at least one virtual environment.

A further advantageous embodiment of the invention is the use, in particular the operation, of a virtual environment of a vehicle generated according to the features of the invention for testing and/or for safeguarding a development status of a system of a vehicle. The vehicle system can be a system for at least partially automated driving of a vehicle.

For example, a system of the vehicle (as a possible part of the system according to the invention) is operated depending on the one or more (preferably a large number) of virtual environments. In this way, data can be obtained (very quickly, inexpensively and without risk), which are also characteristic of the behavior of the vehicle's system in comparable real surroundings of the vehicle.

For example, the vehicle system is a driver assistance system or a system for executing or supporting at least partially automated longitudinal guidance, lateral guidance, maneuver execution of the vehicle, e.g. a system for at least partially automated driving and/or parking.

For example, on the basis of the data sections that correspond to a real situation, output data of a (the original and/or an alternative) system of the vehicle can be obtained which would result if the situation changed or varied.

A further advantageous embodiment of the invention is the use of the generated virtual environment of a motor vehicle for training an artificial neural network. For example, a neural network for operating at least partially automated driving can be trained using one or more virtual environments.

According to a further aspect, the invention includes one or more vehicles for generating a virtual environment. The (real) vehicle can be set up to receive a trigger condition, in particular the trigger condition according to one or more features described in this document, to execute it and/or to transmit data sections determined as a function of the trigger condition to a computing unit arranged at a distance. Preferably, at least the vehicle to be developed using the virtual environment is a vehicle that can be driven at least partially automatically. This vehicle can be a vehicle that is represented or represented in the method at least partially virtually or by one or more devices.

The term "vehicle" in the context of this document is to be understood in particular as a motor vehicle, in particular a car, truck, two-wheeler, e.g. motorcycle, bicycle. For example, at least one or at least two of the real vehicles can be motor vehicles, in particular motor vehicles in typical driving operation on public roads for customers. Alternatively or additionally, the real vehicles can be vehicles of different types or types. For example, at least one real vehicle can be a special vehicle, a remote-controlled vehicle, or a drone. This results in a number of advantages that are described in this document as well as other advantages that are easily understandable for a person skilled in the art. The (real) vehicles can be predominantly or entirely manually drivable or driven vehicles. The optionally provided virtual vehicle (which can be operated with the virtual environment) can be a vehicle that can be driven at least partially in an automated manner.

According to another aspect, the invention includes a system for creating a virtual environment. The system can include one or more (real) vehicles (preferably a large number of vehicles), a separate, in particular stationary, computing unit, e.g. a backend. The (real) vehicles can be vehicles (preferably a large number of vehicles located in a customer's business) with features of the vehicle described above. Furthermore, the system can comprise a stationary device or a device that can be operated while driving, in particular comprising a (real or simulated) control unit of the vehicle that can be operated at least partially separately from a vehicle. Furthermore, the system can be designed to process the data sections, in particular to generate the virtual environment of a vehicle.

The invention allows a significant advantage in terms of costs, effort in the development and market launch of modern vehicles can be achieved.

In addition, there can be a significant advantage in terms of the environmental compatibility of the development, validation or market launch of the vehicles and a corresponding advantage in terms of the image of the automobile manufacturer, since an extremely high, otherwise required mileage of test vehicles can be at least partially eliminated or replaced by means of the invention. Furthermore, the functionality, performance and/or safety of both the resulting vehicle (secured and/or trained developed by means of virtual environments) can be increased.

The invention is described below without loss of generality using an exemplary embodiment with the aid of the attached figure. It is pointed out that all the features described in the present example description can be combined with one another and/or with the features described above.

1. (a) eine erste reale Umgebung RU1 eines realen Fahrzeugs RF1.
2. (b) eine zweite reale Umgebung RU2 eines realen Fahrzeugs RF2.
3. (c) eine beispielhafte virtuelle Umgebung RV12a.
4. (d) eine weitere beispielhafte virtuelle Umgebung RV12b.

Parts (a), (b), (c) and (d) of the figure show:

1. (a) a first real environment RU1 of a real vehicle RF1.
2. (b) a second real environment RU2 of a real vehicle RF2.
3. (c) an example virtual environment RV12a.
4. (d) another exemplary virtual environment RV12b.

In one step of the method, at least two data sections DA1 and DA2 are determined. One of the data sections DA1 represents at least part of a real environment RU1 of a real motor vehicle RF1. One of the data sections DA2 represents at least part of another real environment RU2 of a (another) real motor vehicle RF2. A large number, e.g. more than 4, 8, 16 different data sections DA1, DA2, ... for different real environments RU1, RU2, ... are preferably determined.

The (real or virtual) environment can be understood to mean static and/or dynamic objects in a specific area (shown in dashed lines in the figure) of the respective real vehicle. For example, it is a radius of 10, 20, 50, 100 meters. In each case, they can be objects that are more or less directly relevant to the current management and/or safety of the respective vehicle. For example, the surroundings do not have to include all of the objects in the surroundings of the vehicle, but rather objects that are preferably selected according to specific criteria. The data sections can preferably relate only or predominantly to specific objects, e.g.

The at least one or two of said data sections DA1, DA2 can relate to specific driving situations or represent them. In particular, the at least one environment of the (real) vehicle RU1 or RU2 can include or be a driving situation.

The data section DA1 represents a movement of the respective objects O11, O22, O33 and/or a change in the driving situation (resulting, for example, from the objects O11, O22, O33 and in particular the first real vehicle RF1). The data section DA2 represents a movement of the respective objects O21, O22, O23 and/or a change in the driving situation (resulting, for example, from the objects O21, O22, O23) and in particular the second real vehicle RF2).

A time interval of 1, 5, 10, 13 seconds or the duration of a maneuver (including the respective dynamics) can be represented in a data section DA1 or DA2. The data sections are determined as a function of the respective trigger conditions, in particular recorded by means of a vehicle sensor system and in particular pre-processed. For example, specific objects, features of the environment can be represented in a specific (sparse) way, encoded or represented as representative parameters, data structures, vector graphics, and/or mathematical functions.

At least one or two trigger conditions are selected in particular in such a way that a virtual environment VU12a VU12b of the vehicle, which is characterized by a specific parameter, feature, pattern, type and/or driving situation, can be generated on the basis of the data sections. For example, the trigger conditions are selected in such a way that data sections DA1 or DA2 are suitable for generating a virtual environment VU11a or VU12b (in particular a specific virtual environment or a virtual environment of a specific type).

For example, a trigger condition can include a criterion that indicates that a rare, unusual, unexpected driving situation and/or a driving situation that does not correspond to a typical or known pattern is represented.

The at least two data sections are determined, for example, not simultaneously and therefore at different times. The first motor vehicle as object 012 and the second motor vehicle as object 021 were therefore not located at the same time in the direction of travel in front of the real motor vehicle RF1 or RF2 in any of the real surroundings of the vehicles RF1 or RF2.

Instead, the first motor vehicle 012 is detected at a first time in the vicinity of the real motor vehicle RF1 and the second motor vehicle 021 is detected at a second time in the vicinity of the real motor vehicles RF1 and RF2.

In a further step of the method, the virtual environment is generated from the at least two data sections by merging the data sections representing the first motor vehicle K1 and the second motor vehicle K2, preferably according to a predetermined situation formation rule.

For example, the virtual environment VU12a is generated dependently, in particular on the basis of the data sections DA1, DA2 each representing the first real environment RU1 and the second real environment RU2, e.g. according to a first situation formation rule. Alternatively or additionally, a further virtual environment VU12b is generated as a function, in particular on the basis of the data sections representing the first real environment RU1 and the second real environment RU2, in accordance with a second situation formation rule. Therefore, two or more different virtual environments can also be generated on the basis of the same data sections DA1, DA2. For example, a large number of virtual environments, in particular with different parameters, can be generated.

For example, a virtual environment VU12a is generated comprising a moving object 012 from the first real environment RU1, which has moved in relation to the first real vehicle RF1, and another moving object O23 from the second real environment RU2, which has moved in relation to the second real Vehicle RF2 has moved. The movement of the objects O12 or O23 is illustrated by an exemplary trajectory or a second position of the objects O12' or O23'.

As a result, a virtual environment VU12a representing a new driving situation can be generated. This can include at least two objects 012, 023 or their movement. It can be objects 012 or O23, which in reality have never come close or have never seen each other.

For example, a (further) virtual environment VU12b is generated (possibly from the same data sections DA1 or DA2). In this case, a parameter of an object O23 from a real environment RU2, in this example the type and/or size of the object O23, is changed. In this example, a virtual environment VU12b is generated in which data or a data model representing a (e.g. the same) movement of a car was generated from real movement data of a motorcycle O23.

For example, instead of a real object O23 of a specific first class (car, truck, tractor, motorcycle, etc.) and/or with a specific first parameter (size, shape, condition) and/or with specific first movement data from a data section DA1 or DA2 uses (also to be understood: processed) a data model of an object of a specific second class and/or with a specific second parameter and/or with specific second movement data in the virtual environment VR12b. Thus, an (almost unlimited) variety of (desired) virtual environments can be created.

For example, the virtual environment can represent the same or similar movement of an object with a first parameter or an object of a first type as has been carried out in a real environment by a real object with a second parameter or an object of a second type. In this example, too, data can be generated for a driving situation which, as in the present case, represents an extremely rare driving situation which, in reality, could only be generated or simulated with great effort.

For example, the method includes determining trigger conditions (also to be understood: parameters or criteria according to which the data sections are to be determined) for real vehicles, for recognizing certain types of driving situations and recording data sections the time intervals before, during or after the occurrence of the affect driving situations. This can be done in such a way that the data sections desired or required for generating the at least one virtual environment are (appropriately) determined, in particular generated. These can, so to speak, be "sought together appropriately" and/or "generated appropriately". A virtual environment representing a specific (e.g. selectable) driving situation or a driving situation of a specific (e.g. selectable) type is preferably generated.

In the context of the present document, the term “driving situation” can be understood, for example, as a specific arrangement, action, interaction, driving parameter of road users.

In particular, the meaning of the term "situation" or "driving situation" differs from a frequently used colloquial meaning of the term "traffic situation" which rather includes a summarizing, general categories "free traffic", "heavy traffic", "slow-moving traffic", "traffic jam ", "end of traffic jam" etc.

In particular, a driving situation is (rather) characterized by a specific pattern (also to be understood as a data pattern), e.g. a pattern characterizing the arrangement, speed of the objects and/or pattern of the parameters that characterize a driving situation as such. The driving situation can also be characterized by a spatial pattern of the so-called free spaces in the vicinity of the vehicle.

Furthermore, the driving situation can take into account one or more parameters of the traffic rules (relevant in connection with this), traffic signs, right of way, traffic lights, traffic light phases.

• - eine (bestimmte) räumliche Verteilung der Verkehrsteilnehmer und/oder der Bewegungsparameter der Verkehrsteilnehmer, insbesondere ein Anordnungsmuster der Verkehrsteilnehmer in der Umgebung des (realen) Fahrzeugs,
• - eine (bestimmte) räumliche Verteilung unbeweglicher Objekte in der Umgebung des Fahrzeugs,
• - eine relative Position und/oder Bewegungsparameter zu bestimmten Arten von Spurenmarkierungen, Verkehrszeichen, Ampeln (nicht notwendigerweise zu bestimmten Ampeln, etc.),
• - eine Information über die Vorfahrt des Fahrzeugs, insbesondere gegenüber bestimmten Verkehrsteilnehmern und/oder Verkehrsteilnehmern, die tatsächlich oder zumindest potentiell aus bestimmten Richtungen, z.B. einer querenden Straße rechts oder von links kommen oder kommen können,
• - eine Information zu einer, z.B. einen Grenzwert überschreitenden, Handlung eines Verkehrsteilnehmers in der Umgebung des Fahrzeugs, z.B. Hupen, Lichthupe, Drängeln, Überholen des Fahrzeugs, ein Überholversuch und Dergleichen.

The at least one driving situation can preferably be characterized by:

• - a (specific) spatial distribution of the road users and/or the movement parameters of the road users, in particular a pattern of the arrangement of the road users in the environment of the (real) vehicle,
• - a (certain) spatial distribution of immovable objects in the vicinity of the vehicle,
• - a relative position and/or movement parameters to certain types of lane markings, traffic signs, traffic lights (not necessarily to certain traffic lights, etc.),
• - information about the right of way of the vehicle, in particular over certain road users and/or road users who are actually or at least potentially coming from certain directions, e.g. a crossing road to the right or left,
• - Information about an action, for example exceeding a limit value, by a road user in the vicinity of the vehicle, for example honking the horn, flashing the headlights, jostling, overtaking the vehicle, an attempt to overtake and the like.

The at least one driving situation can preferably be a driving situation that exceeds specific limit values or a driving situation that is characterized by parameters that exceed specific limit values. For example, the at least one driving situation can be a driving situation with an unwanted or dangerous approach to an object or a road user, an acceleration value that exceeds a limit value, an undesired arrangement in relation to other road users, etc. The driving situation can be a special situation (e.g. one that occurs comparatively rarely) or a dangerous driving situation, e.g. a driving situation for which an increased risk is determined or assumed. Such driving situations can be recognized, in particular predicted, by means of defined trigger conditions (also parameters or criteria of the trigger conditions) and the data sections for generating a virtual environment of a vehicle, in particular a virtual environment with a specific parameter or virtual environment of a specific type or pattern determine from real vehicles.

Claims (10)

Method for generating a virtual environment (VU12a, VU12b) of a vehicle, the method comprising the following steps: • determining at least two data sections (DA1, DA2), each of which represents at least part of a real environment (RU1, RU2) of a real vehicle (RF1, RF2), and • Generating the virtual environment (VU12a, VU12b) depending on the at least two data sections (DA1, DA2). procedure after claim 1 , The method comprising the following steps: • providing at least one trigger condition to the one or more real vehicles (RF1, RF2), and • determining the at least two data sections (DA1, DA2) as a function of the at least one trigger condition. procedure after claim 2 , The method comprising the following steps: • providing at least two different trigger conditions to the real vehicle (RF1, RF2), and • determining the at least two data sections (DA1, DA2) depending on the different trigger conditions. Method according to one of the preceding claims, wherein the method comprises the following step: • Transmission of the data sections (DA1, DA2) to a computing unit located at a distance from the one or more real vehicles (RF1, RF2). Method according to one of the preceding claims, wherein the method comprises the following step: • Generating the virtual environment (VU12a, VU12b) depending on the at least two data sections (DA1, DA2) according to a situation formation rule from the at least two data sections (DA1, DA2). procedure after claim 5 , the method comprising the following step: • Generating the virtual environment (VR12a, VU12b) according to a situation formation rule from the at least two data sections (DA1, DA2) by adding, replacing, integrating and/or merging objects (011, O22, O33 , O21, O22, O23), sharing objects (011, O22, O33, O21, O22, O23), sharing space within a real environment (RU1, RU2) in relation to the real vehicle (RF1, RF2), actions, in particular Maneuvers, action parameters and/or behavior that are each represented in the at least two data sections (DA1, DA2). Use of a virtual environment (VR12a, VU12b) of a vehicle generated according to one of the preceding claims for testing and/or for validating a development status of a system of a vehicle. use after claim 7 , wherein the system of the vehicle is a system for the at least partially automated driver of the vehicle. Use one after one of Claims 1 until 6 generated virtual environment (VR12a, VU12b) of a vehicle for training an artificial neural network. System for generating a virtual environment (VR12a, VR12b) of a vehicle, the system being set up to determine at least two data sections (DA1, DA2), each of which contains at least part of a real environment (RU1, RU2) of a real vehicle (RF1, RF2) represent; is further set up: dependent on generating a virtual environment (VU12a, VU12b), in particular on the basis of the at least two data sections (DA1, DA2).

Images