DE102020120141A1 - Procedure for optimizing tests of control systems for automated vehicle dynamics systems using probabilistically predicted system responses - Google Patents

Abstract

Method for optimizing tests of control systems for automated vehicle dynamics systems, in which relevant parameters and relevant system responses are defined, quasi-random parameter combinations of the relevant parameters and system responses of the test system are generated, a probabilistic system model is created and trained depending on the generated parameter combinations and the system responses , the system responses of the model generated by the system model are evaluated, the interesting system responses and parameter combinations are selected and system responses of the test system are generated again depending on the interesting parameter combinations. This enables iteration for targeted training of the probabilistic system model in critical areas of the test system.

Description

The invention relates to a method for optimizing tests of control systems for automated vehicle dynamics systems, in particular for optimizing the selection of test parameters and carrying out tests more efficiently over large parameter spaces.

State of the art

As part of the development of complex driver assistance systems and driving dynamics systems, the number of parameters and test scenarios to be examined is constantly increasing. Even modern methods and technologies, such as computer-aided simulations, cannot provide the necessary resources for a comprehensive examination of systems with certainty. There are currently different approaches to testing the systems mentioned.

Before checking the test system, the parameters to be included are usually defined. These describe the circumstances, influencing factors and framework conditions for a test. System-internal and system-external parameters can be selected with which a test is carried out. The test system can be checked for compliance with the defined requirements by means of qualification, verification or validation.

Based on the parameters identified, a parameter space is created whose dimensions correspond to the number of parameters selected. Various parameter settings or parameter characteristics are tested within the parameter space. Different combinations of parameters are used, in which several or all parameters are varied and/or combined with different characteristics.

As a result of the tests, test data, which are also called system responses, are generated depending on the tested parameter combinations. The tests themselves can be done in different ways. Tests can be carried out on virtual test systems, e.g. B. simulations, and on real test systems, z. B. vehicles are carried out. In between there are a large number of tests in which parts of the test system are virtual and other parts are real. Based on the test data generated, knowledge about the test system to be tested is gained, which, combined with appropriate specialist knowledge, leads to optimization.

The meaningfulness of the test data and thus of the tests as well as the effort involved in generating meaningful test data depends on how the parameter combinations used for testing are generated or selected. Among other things, the number, the characteristics, the combinations and the distribution of the parameters in the parameter space play a role.

In full factorial design of experiments, every possible combination of parameters is tested. In the case of continuous parameters, fixed parameter characteristics must be selected as a subset of all possible settings. Depending on the number of parameter characteristics, the total number of parameter combinations to be tested increases. The parameter space is tested along a grid. The test data obtained can then be used directly for evaluations or knowledge gain. Depending on the number of parameters and the grid density, there may be combination numbers for which it is not possible to carry out all tests due to their quantity. For this reason, the parameter spaces are sometimes reduced or subsets of the full factorial parameter combinations are selected in a targeted manner. Their selection or restriction is arbitrary. Due to the complex interrelationships, critical areas cannot be identified with sufficient certainty and may therefore be overlooked.

In the statistical modeling of the parameter combinations, the parameter combinations are based on probability distributions, e.g. B. a normal distribution, and it is assumed that a certain parameter combination occurs most frequently. The remaining tests are then distributed around the parameter combination with the greatest probability of occurrence with decreasing density. The test data obtained is then used directly for the corresponding evaluations.

Simulations with feedback are also known. The result of a test is used to decide which parameter combination is to be simulated next. The aim is to uncover local and/or global maxima or minima, as well as system boundaries. This approach has some disadvantages. Especially with the full factorial simulation approach, the number of tests increases exponentially with the number of parameters. A successful simulation can only take place with the use of high resources and time. If the system responses are ignored, the simulation will be performed according to a pre-established schedule. It is therefore possible that insufficient measurement data is recorded in critical areas of the parameter space. In the case of the simulation with feedback, a dynamic test planning takes place, but should there be several minima or maxima, these are regularly not found by simple simulations, as these are usually designed to find an extreme value. Complicated simulations with feedback may be able to localize several extremes in parallel, but then they consume a lot of resources.

In order to limit this, interesting sub-areas of the parameter space can be described in which the density of the parameter combinations to be tested is increased compared to the rest of the parameter space. These subareas can be described using hypercubes. This independently excludes the value ranges for all parameters for which no interesting system behavior can occur or is expected. The partial areas can take on complex forms, especially for higher-dimensional parameter spaces, which in turn can be very complex to determine in detail. In addition, geometrically defined hypercubes circumscribe a sub-area of the parameter space that includes a large number of uncritical parameter combinations, since the accuracy of the geometric delimitation of the sub-areas has inherent limits.

the WO 2015 067 649 A1 concerns a virtual test optimization for driver assistance systems (DAS). A test scenario, defined by test parameters, is run through in real driving tests and/or virtually. To create a second test, the first test is modified to move the test parameter within a critical range. The disadvantage is that the parameter variation, which leads to critical driving behavior, was determined arbitrarily, at least a priori, or resulted randomly from a test. There is no systematic delimitation of critical areas in which the activation of an ADAS is provoked.

object of the invention

In contrast, it is the object of the invention to improve the generation, implementation and results of tests of control systems for automated vehicle dynamics systems. In particular, relevant test parameters should be found efficiently in parameter spaces of any complexity.

Presentation and advantages of the invention

The object is solved by a method for optimizing tests of control systems for automated vehicle dynamics systems in accordance with the measures of independent claim 1.

Automated vehicle dynamics systems are systems for regulating components for vehicle control, such as drive, deceleration or steering, which are not operated directly by the vehicle driver, but are activated and operated automatically or autonomously by dedicated control units. These vehicle dynamics systems can be designed as driver assistance systems for specific driving situations. Examples of such driver assistance systems are the anti-lock braking system (ABS), the electronic stability program (ESP), the adaptive cruise control (Automatic Cruise Control ACC), the emergency brake assistant (Automatic Emergency Breaking AEB) or the lane departure warning system (Lane Keep Assist LKA). These functions are regarded as automation level 1 or 2, which support vehicle operation or take on subtasks of vehicle guidance. Automated vehicle dynamics systems can also be assigned to automation level 3 (operational automation or conditional automation) and higher, where the vehicle temporarily or permanently takes on tasks independently. The latter includes the integration of several systems, each of which takes on subtasks and, in the overall system, guides the vehicle in a highly automated and even autonomous manner.

In the first step of the method according to the invention, relevant parameters and relevant system responses of the control system to be tested are defined. The relevant parameters form the parameter space, the dimensions of which depend on the number of parameters. The variable parameters and the system responses to be logged or specific observation variables, also known as key performance indicators (KPI), are primarily dependent on the design of the test system itself, i.e. the hardware and software design, its range of functions, i.e. the actual control objective of the driving dynamics system, and the environment in which the control system is to be used. However, they also depend on the test objectives, the degree of abstraction of the control system to be tested and its system limits. Not only is it defined which KPIs are relevant, but also the extent to which they are relevant. Defining the parameters and system responses is part of the original design of the experiment, along with other specifications. The KPIs can be classic accident-relevant key figures, such as time to impact, or comfort-related responses from the system, such as maximum lateral acceleration.

In the next step of the method according to the invention, a test system is provided as a representation of the control system to be tested. The control system to be tested can be a real one Be driving dynamics system, so a arranged in a vehicle control unit, which depending on the configuration determines the vehicle condition and the vehicle environment, interprets and causes reactions by the vehicle associated actuators. These reactions can be interventions in the longitudinal and/or lateral dynamics of the vehicle or the issuing of warnings in the vehicle or to the outside. The test of a real vehicle dynamics system can be a classic experiment, eg in the course of a driving test. In this case, the test system can represent the vehicle as a whole or a selected function (keep lane, keep distance) of the vehicle, including special environmental conditions (driving test in wet conditions). The definition of relevant parameters and system responses also defines the design of the test system, for example the inclusion of additional sensors or actuators of the vehicle, and thus the interface to the surrounding environment is defined. However, the test system can also be a simulation of a real vehicle dynamics system, which deterministically maps the control behavior of the vehicle dynamics system in a complex manner. The test system can also be what is known as a black box, the function of which is known but whose mode of operation is not known or has not yet been determined. This is particularly relevant in the early stages of development, in which the specific implementation of the function has not yet been determined. The test system as a representation of the control system to be tested can be the control system to be tested itself or a modeled representation of its function. The text system is provided accordingly with vehicle-related, mathematical and/or computer-related means.

In the next step of the method according to the invention, quasi-random start parameter combinations of the relevant parameters are generated, distributed over the entire parameter space. Each defined parameter can have different characteristics, for example as a Boolean variable that can only assume a few states, through discrete nominal or ordinal scalable variables, to continuous distributions. These include different types of probability distributions, for example the discrete uniform distribution or the normal distribution. A combination of both types for individual parameters can also be useful. The type of distribution is selected in connection with the test objective or the desired representativeness of reality. The discrete uniform distribution is suitable for investigations in the entire value range of the parameters. This allows, among other things, the performance of a function in the entire application area, such as e.g. B. the required environment or scenarios, analyze and evaluate. Different functional and system characteristics or limits of system performance, such as sensor tolerances or physical limitations, can also be evaluated in this way. Finally, realistic probability distributions based on real driving data and measurement recordings can also be determined for the parameters.

Quasi-randomly means that the parameter values are not only distributed across the parameter space by a truly random mechanism, such as a defined probability distribution. The number and density of the test points can be adjusted depending on the system complexity and test objective, if necessary using known DoE methods (Design of Experiments - statistical test planning). After the calculation of the quasi-random parameter combinations, these can be checked for their confidence or quality using different metrics and have a significantly reduced scope compared to full factorial experimental design. However, they still cover the entire parameter space. The number of start parameter combinations as well as their distribution and density in the entire parameter space depends on the test capacity of the test system. Test capacity is the relationship between the test duration of testing a combination of parameters and the resources available for testing, e.g. time specifications. The number of start combinations therefore depends on how many tests can be carried out realistically or practicably. Real vehicle tests, for example, are strongly determined in terms of time and resources. Even high-dimensional, deterministic simulations that require several seconds or minutes to simulate a parameter combination are subject to time restrictions as development cycles become shorter. As a result, the start parameter combinations cover the entire parameter space, but have a very low density.

In the next step, system responses of the test system are generated as a function of the quasi-randomly generated start parameter combinations. As described, the test system itself can represent any configuration of a control system for automated vehicle dynamics systems along the entire development process, from a modeled concept idea to software to be tested, an individual component to be tested, an assembly to be tested to a vehicle to be tested. The test system can be modeled using known concept, software, hardware or model-in-the-loop methods (CiL, SiL, HiL, MiL). With the help of the modeling of the test model or the experimental testing, responses of the test system to the input parameters, here the start parameter combination nations, are determined or generated. Existing (partial) models can be used as well as complex deterministic software codes including complex physical backgrounds. In the case of a real (overall) system to be tested, the system responses can also be generated by means of parameterized test drives on a test bench or in the real world. Finally, combinations of modeled and actually generated system responses are also conceivable, depending on the definition of the test system.

In the next step, a probabilistic system model for predicting the test system behavior is initiated and the probabilistic system model is trained using the quasi-random start parameter combinations and the system responses of the test system. The defined test systems, even if they are also modeled in the form of simulations, are so complex and time-consuming or computationally intensive that you can only work with them to a limited extent. So it may be necessary to create a model of the actual model that behaves similarly but is significantly more performant. These models (of models) are called meta-models. A meta-model will not represent a system exactly, and its predictions will always be erroneous. However, these can predict the results of further tests within a very short time and thus test a large number of parameter combinations in a practicable manner. The probabilistic system model is designed as a probabilistic meta-model that can assess the uncertainty (confidence or quality) of your predictions. Instead of discrete predictions (e.g. y = 0.3), these predictions deliver in the form of a probability density, for example as a normal distribution. Selecting and, if necessary, adapting a corresponding meta-model, the underlying algorithms and mathematical methods is referred to as initiating.

In order to take into account the many possible behaviors of the test system, concrete function classes can be used, e.g. second degree polynomials. However, this approach is very special and can therefore only be used in a meaningful way if the corresponding system can actually be described well within the narrow assumptions made. However, a larger number of functions can also be taken into account. The choice of the possible functions is already a challenge, since there are an infinite number of possible functions. Here, Gaussian processes (GP) represent a mathematically elegant and at the same time powerful way of choosing a function. A GP can be viewed as a probability distribution over functions. Just as specific values can be drawn from a normally distributed variable, specific functions can be drawn from a GP. The set of possible functions is defined by a mean function and a kernel function that represents a generalization of a covariance matrix.

A current further development of the Gaussian processes are the so-called deep Gaussian processes. Known as Deep Gaussian Processes (DGP), this model family is very similar to another model family that has gained enormously in importance in recent years: Deep Neural Networks (DNN). In comparison to single-layer or few-layer networks, the enormous performance of deep neural networks is mainly attributed to the fact that hierarchical learning can take place between input and output. With increasing abstraction, connections can be recognized that are difficult or impossible to find directly in the data. DNN can also be used to make probabilistic predictions. They are then referred to as Bayesian Neural Networks (BNN). The idea behind BNN is to consider all network parameters not as deterministic values but as probability distributions. Training is also difficult for BNN, since there are no analytical solutions for equations to determine the probability distribution of the parameters for certain input data.

Generally speaking, the probabilistic system model is built using machine learning methods based on training data. The initial training of the model based on the system responses of the test system to the starting parameter combinations and the starting parameter combinations (training data) can also be referred to as training.

The following steps of the method according to the invention represent an iterative method. For this purpose, in the next step, a multiplicity of quasi-random parameter combinations of the relevant parameters are generated. Like the start parameter combinations, these are generated distributed over the entire parameter space, but with a significantly greater density and number. Advantageously, a number of distributed clusters of critical parameter combinations can also be recorded in this way.

In the next step, system responses of the probabilistic system model are generated as a function of the large number of quasi-random parameter combinations as described above.

In the next step of the method according to the invention, the system responses are evaluated and interesting system responses are identified ten. When evaluating the system responses, their interestingness is determined. This can be read directly from the system responses or by comparing the system responses with predetermined threshold values or specified intervals. Which experiments or behavior are interesting can be defined depending on the application by defining intervals for system outputs (e.g. minimum distance < 10m) and then the parameter combinations are interesting whose system responses lie in these intervals.

An acquisition value, which estimates the information content of the training data, is preferably determined from the system responses. The functions used for this are referred to as acquisition functions. The predictions of the model are thus evaluated with the aid of an acquisition function, whereby the values of the acquisition function should be high at the points at which a high information gain, i.e. a high level of interest, can be expected.

A special acquisition function can be provided for each application. In order to obtain a meta-model that is consistently good for the entire parameter space, a large number of tests are first predicted and the acquisition value is chosen proportionally to the standard deviation. In order to investigate specific behavior in a targeted manner, an interval defined as interesting can be included in the calculation. The acquisition value is then equal to the probability that the system response y lies in the interval M. The acquisition function can also be designed in such a way that parameter combinations at the boundaries of the interval M are searched for in a targeted manner. Depending on the goal of the investigation, further acquisition functions are conceivable, e.g. the execution of all tests for which the probability of an interesting system behavior exceeds a threshold value. If interesting parameter ranges are interpreted as a probability distributed over the parameter space, a continuous relevance of all tests results. In order to map this through a (discrete) test plan, many possible tests can be generated, evaluated by the meta-model and an acquisition function and kept or discarded according to their relevance. This procedure, known as the rejection method, ultimately serves to adjust the test density and thus discretely map the continuous acquisition function. The evaluation of the system responses thus makes it possible to identify interesting system responses because the evaluation separates between interesting and uninteresting system responses. A major advantage of this approach is that the regions of interest do not have to be described and the transition between the sub-areas is continuous.

In the next step, the interesting parameter combinations for the relevant system responses are determined, ie the parameter combinations that cause or generate the relevant system responses of the probabilistic system model are determined. These input parameters can be read directly from the system model or calculated back.

In the next step, the system responses of the test system are generated depending on the interesting parameter combinations. This happens as with the generation of the system responses depending on the start parameters. Since only the interesting starting parameters are tested, the number of tests to be carried out is small. This makes testing the test system practical. Nevertheless, there is a higher density of parameter combinations to be tested in interesting areas of the parameter space, since the uninteresting, i.e. uncritical system responses, were discarded in the previous step. This enables better analysis of critical areas with reduced use of test resources.

In the last step of the iterative procedure started above, the probabilistic system model is trained using the interesting parameter combinations and the system responses of the test system depending on the interesting parameter combinations. By training the probabilistic system model using interesting training data, its prediction capability is increased in critical areas of the test system and thus the control system for automated vehicle dynamics systems.

Repeating the steps of generating a large number of quasi-random parameter combinations, generating system responses of the probabilistic system model to these parameter combinations, evaluating these system responses and identifying interesting system responses, determining the interesting parameter combinations for these interesting system responses and training the probabilistic system response using these interesting parameter combinations and the corresponding interesting system responses of the test system, creates an iteration loop. This is carried out for targeted training of the system model. Depending on the iteration loop, the relevance or interest, on the basis of which the system responses are evaluated, can differ. Different acquisition functions can also be used. Because only the interesting systems respond and their parameter combinations to further Prediction can be used, new parameter combinations can be generated, which fill the parameter space according to the relevance and not, as in full factorial testing, independently of the relevance.

The iteration loop is terminated by ending the repetition, preferably as a function of a specific model uncertainty. Since the uncertainty or confidence level or quality of the predictions of the probabilistic system model can be determined, a sufficient test density for relevant areas is assumed from a predetermined level of quality of the system model without having to define them parametrically, i.e. delimit them within the parameter space. The quality can be determined specifically for relevant parameter combinations or generally across the entire parameter space. The model uncertainty is preferably determined after each repetition or iteration step. Determining the confidence level of a trained model is fundamentally known. For example, cross-validation can be performed using known input and output combinations. For example, by means of targeted tests of the system model using non-critical parameter combinations of systems that have already been rejected, or reference parameters from standard tests, an over-adaptation of the system model can be detected or prevented.

Finally, the optimization of tests of control systems for automated vehicle dynamics systems and/or the control system for automated vehicle dynamics systems can be carried out using the last determined parameter combinations and/or the trained probabilistic system model and/or the system responses of the probabilistic system model and/or the test model.

The parameter combinations determined last show a high density in relevant areas of the parameter space and can therefore be used to evaluate the statements on the safety, i.e. the functionality in critical parameter situations, of the test system with a few critical tests. This applies regardless of whether it is a real or simulated test system.

In this way, optimized parameter combinations for testing control systems for automated vehicle dynamics systems are obtained. For highly automated driving functions in particular, it must be said with a high degree of certainty that the system can guarantee safe functioning in all possible situations. Relevant parameter combinations are identified whose interest for the system behavior is so great that the entire system can be tested on the basis of these fewer parameter combinations, which in turn enables a sufficiently reliable statement to be made about the functionality or the safety level of the overall system. In this way, partial areas of the parameter space in the system model are optimized without having to identify or even describe these areas geometrically or parametrically. The parameter combinations obtained in this way can then be used in (further) tests of the automated vehicle dynamics systems, e.g. in test drives or complex hardware-in-the-loop (HiL) tests but also in software or model-in-the-loop tests ( SiL, MiL). As the development stage of the driver assistance and driving dynamics systems progresses, more complex parameter combinations and thus more complex tests are also required due to the increased complexity of the parameters involved (sensor inputs, situation variables, functions involved).

The trained probabilistic system model itself can contribute to the optimization of tests. With modifications, it can be used as a basis for system models of similar test systems, so that fewer iteration steps have to be carried out later. It can also be included in more complex models for testing control systems in later development phases, which in turn results in less training effort.

In addition, knowledge about undiscovered system limits can be gained from the system responses of the test system or improvement potential of a simulation can be identified. The optimization of the tests not only results in the more efficient execution of the tests, the increase in the validity of the tests carried out on the test system and the certifiability of the test or control systems, but can also enable further analyzes and optimizations along the entire development process as well as development impulses by recognizing system limits at an early stage Set stages of development.

The method according to the invention also offers the advantage of making available optimized parameter combinations for each development stage. Advantageously, simulating higher level system responses could be performed at least in part by the previous level prediction model. In addition, as critically delimited sub-areas, critical weaknesses of the automated vehicle dynamics system can already be uncovered without subsequent tests, especially in the early development or functional design phases, whose early elimination saves significant resources for subsequent adjustments.

A significant advantage of the method according to the invention is the efficient mapping of large parameter spaces with a small number of parameter combinations to be tested. In this way, it is particularly advantageous to find a number of extremes, ie a number of critical areas, which can be analyzed individually or in parallel.

A further advantage is the simple generation of initial system responses of the test system to the parameter combinations due to the quasi-random distribution of the parameter combinations. Due to the low starting density, this enables fast, resource-saving generation of test data for the prediction model. In most cases, the test system already exists or can be assembled from model building blocks. As a result, a sufficiently precise starting combination of system inputs and outputs can be generated, which enables the prediction model to be trained for estimating system responses for any, unknown parameter combinations. Above all, the combination of training a prediction model with simply generated data and the subsequent iterative focus on critical or interesting areas causes a significant increase in efficiency in test case optimization with simultaneous applicability over the entire development chain from concept creation to series validation.

In an advantageous embodiment of the method according to the invention, the test system is modeled on the basis of a functional model, a system model and an environment model of the control system for automated vehicle dynamics systems. The functional model models the function of the automated driving dynamics system, i.e. its actual purpose. The system model models how the purpose is to be implemented, including the sensors, processors and actuators involved and the communication between these elements. The environment model models where this is to be implemented, i.e. in which scenarios. It is particularly advantageous that these modules should also be adapted independently of one another to the driving dynamics system to be tested. In this way, the same functions can be tested in different systems without creating a new function model. The individual building blocks can therefore be reused. The same applies to different environments or scenarios. Furthermore, the division into these three building blocks is very easy to model on the one hand and particularly realistic on the other. This approach is therefore particularly resource-friendly.

The intended function can be simplified, for example, with scene-based function development using scenarios, which means that part of the environment can already be defined. In addition, relevant system properties and environment variables must be defined. In addition to general requirements such as weather conditions, the environment also includes identified scenarios that the overall system should be able to handle. The scenarios are defined by road and infrastructure elements as well as objects with a specific starting situation and various actions over time. The three elements to be modeled can vary greatly in complexity depending on the objective. In system modeling, for example, a perfect system behavior can be represented like an ideal sensor, insofar as the influences of these three elements are to be explicitly excluded in the analysis. Or complex sensor characteristics are mapped if the focus is on their influence. Alternatively or additionally, the initial generation method can also be created and implemented using real vehicle data and/or by means of simple deterministic software codes while omitting physical backgrounds. The complexity of the modeling of the test system can depend on the respective development phase and usually increases in the course of the further development of the concept phase through various series development phases to series release and series monitoring, both with regard to the individual function and the entire vehicle. Advantageously, this allows a modeling of the control system to be tested that is appropriate to the respective development status to be created quickly and easily. Functional and system characteristics as well as scenario specifications can be described by parameters, such as sensor tolerances or speeds of other objects in the scenarios. The variation of the individual parameters and their combination lead to specific scenarios with fixed functional behavior and specific system performance.

One aspect of the present invention relates to a device for optimizing tests of control systems for automated vehicle dynamics systems, the device being designed as a computing unit in order to carry out all the steps of a method according to one of claims 1 to 4. It goes without saying that each step of the method according to the invention can be carried out on the same or on different processing units, such as computers, which are connected by communication. These processing units can be arranged locally or distributed globally. The processing unit can be part of a stationary or portable computer. Each processing unit has its own memory or one that is used jointly with other processing units, as well as at least one processor.

One aspect of the invention relates to a computer program for optimizing tests of control systems for automated vehicle dynamics systems, the computer program causing a computing unit to carry out all the steps of a method according to one of claims 1 to 4 when it is executed on the computing unit according to claim 5. The processing unit has its own memory or one that is used jointly with other processing units, as well as at least one processor. One of the specified methods is stored in the memory in the form of the computer program and the processor is provided for executing the method when the computer program is loaded from the memory into the processor.

The invention further relates to a computer-readable storage medium on which a computer program according to the present invention is stored. Computer-readable storage media, also referred to as machine-readable storage media, are known per se and can be designed as magnetic storage (floppy disks), optical storage (CD), flash storage (USB stick), read-only storage (ROM) or many more.

The invention also relates to a program code with processing instructions for creating a computer program executable on a computer according to the present invention, wherein the program code results in the computer program when the program code is converted into an executable computer program according to the processing instructions.

The invention also relates to a computer program product, the computer program product having a computer-readable storage medium according to the present invention and a computer program stored on the computer-readable storage medium according to the present invention with a program code according to the present invention, the program code for executing a method according to the present invention is suitable when the computer program is run on a computer.

example

Further features, application possibilities and advantages of the invention result from the following description of exemplary embodiments of the invention with reference to the schematically illustrated drawings. These are only for understanding the invention and do not have any limiting effect on the subject matter of the invention as set out in the patent claims.

• 1a-c eine Gegenüberstellung verschiedener Dichteverteilungen von Parameterkombinationen eines zweidimensionalen Parameterraums mittels geometrisch abgegrenzten Teilbereichen des Parameterraums und
• 2a-c eine Abfolge von Iterationen zur Erhöhung der Dichte der Parameterkombinationen in interessanten Bereichen des Parameterraums.

Here show:

• 1a-c a comparison of different density distributions of parameter combinations of a two-dimensional parameter space by means of geometrically delimited sub-areas of the parameter space and
• 2a-c a series of iterations to increase the density of parameter combinations in regions of interest in the parameter space.

1a shows an equidistant and based on a purely exemplary two-dimensional representation 1b a quasi-random distribution of parameter combinations over the entire parameter space. Both distributions shown do not reflect the true ratio of the respective numbers of parameter combinations, but only serve to clarify that it is 1b to not evenly or equidistantly distributed and to a compared to 1a significantly lower number, and therefore a significantly lower density, of parameter combinations. In 1a a (rough) boundary 11 is shown, which separates the critical parameter combinations 10 lying within this from those lying outside. Whether a parameter combination is critical results from the system response corresponding to this parameter combination. If this is critical, for example due to falling below a critical threshold value, such as the time to impact, or due to a negative system response, such as hitting an obstacle, then this results in a critical parameter combination. In comparison, the parameter combinations of the areas 12 lying outside the boundary 11 are uncritical because their corresponding system answers are uncritical or uninteresting.

The density of parameter combinations in 1b is initially not sufficient to make a sufficiently precise statement on the basis of the model prediction as to which specific parameters and in what form are the cause of the criticality of the system response. Accordingly, no meaningful remedial or improvement measures can be derived. One possibility is to limit the parameter space around the critical parameters and increase the density of the quasi-randomly generated parameter combinations 14 ( 1c ) in the delimited sub-area 15. The invention, on the other hand, makes it possible to generate a compression of parameter combinations in interesting or critical areas of the parameter space, which does not require the definition of the sub-area.

2a-c clarifies the inventive effect of training the system model using interesting parameter combinations. 2a shows a quasi-random distribution of seed parameter combinations 20 (circles). This is used to generate the system responses of the test system. The probabilistic system model is trained using the start parameter combinations and the associated system responses of the test system. This is then tested with a large number of other parameter combinations distributed over the entire parameter space, i.e. system responses are generated. After these have been evaluated, the in 2 B shown parameter combinations 21 (triangles) as new parameters to be tested in the test system.

After one or more more iteration loops, the parameter combinations are 22 (stars) in 2c added as highly relevant parameter combinations. Testing with these parameters enables a particularly reliable statement about the function of the test system or the control system in critical areas. In contrast to the delimitation using hypercubes, the areas with a high density of parameter combinations and expected critical system responses merge continuously into the adjacent non-critical parameter space. As a result, only the tests that are actually relevant are carried out and not the irrelevant ones that are also included in the hyperquader.

The interesting parameter combinations can be selected, for example, by means of the rejection method. The system responses of each tested parameter combination are retained or discarded. Based on existing knowledge in the form of observations of specific parameter combinations, a model is created that can predict the mean µ and standard deviation σ. The predictions of the model are evaluated with the help of the acquisition function, whereby the values of the acquisition function should be high at the points where a high information gain can be expected. The system is then examined at the point y where the acquisition function reaches its maximum. The next iteration begins with the new knowledge.

In order to investigate specific behavior in a targeted manner, the interval M _i defined as interesting is included in the calculation in the two other cases: $$a\left(y\sim N\left(\mu ;{\sigma }^2\right);M_i\right)=P\left(y\in M_i\right)$$

The acquisition value a is equal to the probability that y lies in the interval M _i . In this way, interesting experiments can be found in a targeted manner.

können gezielt Versuche an den Grenzen des Intervalls M_i gesucht werden. Der Akquisitionswert erreicht das Maximum von 1 für P(y ∈ M_i) = 0.5 und fällt zu P(y ∈ M_i) = 0 bzw. 1 linear ab. Je nach Ziel der Untersuchung sind weitere Akquisitionsfunktionen denkbar, z.B. die Durchführung aller Versuche, für die die Wahrscheinlichkeit eines interessanten Systemverhaltens einen Schwellwert überschreitet. Ein großer Vorteil dieser Vorgehensweise liegt darin, dass die interessanten Regionen nicht beschrieben werden müssen und der Übergang zwischen den Bereichen kontinuierlich ist.With the help of $$a\left(y\sim N\left(\mu ;{\sigma }^2\right);M_i\right)=1-2\left|P\left(y\in M_i\right)-0.5\right|$$

tests can be specifically searched for at the limits of the interval M _i . The acquisition value reaches the maximum of 1 for P(y ∈ M _i ) = 0.5 and falls linearly to P(y ∈ M _i ) = 0 or 1. Depending on the goal of the investigation, further acquisition functions are conceivable, eg the execution of all tests for which the probability of an interesting system behavior exceeds a threshold value. A major advantage of this approach is that the regions of interest do not have to be described and the transition between the areas is continuous.

Reference List

10

critical parameter combinations

11

rough limit of critical sub-area

12

non-critical area

13

non-critical area

14

additional parameter combinations

15

detailed demarcation of critical sub-area

20

Parameter combinations after first iteration

21

Parameter combinations after second iteration

22

Parameter combinations after third iteration

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• WO 2015067649 A1 [0011]

Claims (9)

Method for optimizing tests of control systems for automated vehicle dynamics systems, having the following steps: - Defining relevant parameters and relevant system responses of the control system to be tested, - Provision of a test system as a representation of the control system to be tested, - Generation of quasi-random seed parameter combinations of the relevant parameters, - Generation of system responses of the test system depending on the start parameter combination, - Initiation of a probabilistic system model to predict the test system behavior and training of the probabilistic system model using the quasi-random start parameter combinations and the system responses of the test system, - Generation of a large number of quasi-random parameter combinations of the relevant parameters, - Generation of system responses of the probabilistic system model depending on the large number of quasi-random parameter combinations, - Evaluate the system responses and identify interesting system responses, - determining the interesting parameter combinations for the interesting system responses, - generation of system responses of the test system depending on the interesting parameter combinations, - Training of the probabilistic system model using the interesting parameter combinations and the system responses of the test system depending on the interesting parameter combinations, - Repeating the steps of generating a variety of quasi-random parameter combinations, generating system responses of the probabilistic system model to these parameter combinations, evaluating these system responses and identifying interesting system responses, determining the interesting parameter combinations for these interesting system responses and training the probabilistic system response using these interesting parameter combinations and the corresponding interesting system responses of the test system, - Ending the repetition and optimization of tests of control systems for automated vehicle dynamics systems and/or the control systems for automated vehicle dynamics systems using the last determined parameter combinations and/or the trained probabilistic system model and/or the system responses of the probabilistic system model and/or the test model. procedure after claim 1 , wherein the probabilistic system model for predicting the control system behavior is modeled as a Gaussian process, deep Gaussian process or Bayesian neural network. Method according to one of the preceding claims, in which the system responses are evaluated using an acquisition function. Method according to one of the preceding claims, wherein the test system is modeled on the basis of a functional model, a system model and an environmental model of the control system for automated vehicle dynamics systems. Device for optimizing tests of control systems for automated vehicle dynamics systems, the device being designed as a computing unit to carry out all the steps of a method according to one of Claims 1 until 4 to execute. Computer program for optimizing tests of control systems for automated vehicle dynamics systems, the computer program causing a computing unit to perform all the steps of a method according to one of Claims 1 until 4 run when it is down to the processing unit claim 5 is performed. Computer-readable storage medium, characterized in that on the computer-readable storage medium, a computer program claim 6 is saved. Program code with processing instructions for creating a computer program that can be run on a computer, characterized in that the program code follows the computer program claim 6 results when the program code is converted into an executable computer program in accordance with the processing instructions. Computer program product, characterized in that the computer program product is a computer-readable storage medium claim 7 and a computer program stored on the computer-readable storage medium claim 6 with a program code claim 8 comprises, wherein the program code for executing a method according to one of Claims 1 until 4 is suitable when the computer program is run on a computer.

Images