EP3622451A1 - Product maturity determination in a technical system and in particular in an autonomously driving vehicle - Google Patents

Abstract

According to the subject matter of the invention, a method for determining a product maturity by means of tests is claimed, wherein a test comprises the execution of a test case by means of a test environment applied to a test object, and for at least one test there is no result, and the method comprises the steps of prescribing rules for calculating a probability of a test for which there is no result being successful or unsuccessful, wherein the rules use available or expected results of tests as input variables and return probabilities as output variables, and calculating the probability of a test for which there is no result being successful by means of at least some of the prescribed rules, and presenting the product maturity on the basis of the probabilities calculated in the previous step.

Description

 Product maturity determination of a technical system and in particular an autonomous driving

 vehicle

The invention relates to a method and a test system for

Product maturity determination of a technical system according to the preamble of patent claim 1.

Particularly in the field of semi-autonomous or autonomous vehicles, there is a demand for vehicles, parts of vehicles (e.g., electronic control units), and driving functions (for example, as algorithms of the

ECUs running software). Since most of these driving functions are safety-critical, it is necessary to have a sufficiently secure criterion for the release of the next

Development step to define and review. Especially after completion of the last development step, since then the series production of the vehicle begins. In the case of partially autonomous or autonomously driving vehicles, this is particularly difficult since the number of all possible scenarios to be tested is virtually infinite due to the never fully reproducible reality of the traffic situation. Thus, a meaningful criterion which can be checked in finite time and sufficiently secure for the release of the next development step must be found.

Methods for product maturity determination are known from the prior art, in which the product maturity of a technical system is determined by means of test coverage. Under test coverage is here in

In general, the ratio of tests performed to the total number of tests executable for the technical system, or the ratio of successfully performed tests to the total number of tests that can be performed on the technical system. In the European Patent Application Publication EP3082000A1, product maturity is determined by means of test coverage, and proposals are made for improving the test coverage. Against this background, the object of the invention is to provide a device which further develops the prior art.

The object is achieved by a method for determining a product maturity with the features of patent claim 1, as well as by a test system having the features of patent claim 15. advantageous

 Embodiments of the invention are the subject of dependent

Dependent claims.

According to the subject invention, a method for determining a product maturity by means of tests is claimed, wherein a test comprises executing a test case by means of a test environment applied to a test object, and there is no result for at least one test, and the method comprises the steps of setting rules for calculating a probability that a test for which there is no result will be successful or unsuccessful, the rules using as input variables present or expected results of tests and returning probabilities as output variables, calculating the probability that a test for which no result is present, will be successful, by means of at least part of the predetermined rules, as well as representation of the product maturity in dependence of the probabilities calculated in the previous step.

An advantage of the method according to the invention is that the product maturity of a technical system is evaluated not only on the basis of already performed tests, but also tests that have not yet been carried out are taken into account in the evaluation. In the event that not all tests are performed, this results in a more complete picture of product maturity, otherwise you will get more meaningful statements about product maturity at an earlier stage, as the tests that are still in the future will also be included in the evaluation. Furthermore, based on this forward-looking consideration, an improved selection of the tests still to be performed and their order can be made. Further, it is advantageous that by means of the method easier to find but meaningful representations of product maturity found and presented can be. Statements about product maturity can simply be made according to the existing criteria, such as: B. the present test cases, test environments or test objects are summarized or grouped. In one embodiment, a test object (SUT) is an at least partially autonomously driving vehicle, a part of an at least partially autonomously driving vehicle or a functionality of an at least partially autonomously driving vehicle, and a test case (TC) is a driving maneuver of the at least partially autonomously driving vehicle, a driving maneuver with the part of the at least partially autonomously driving vehicle or a driving maneuver in which the functionality of an at least partially autonomously driving vehicle is taken into account, and is a test environment (TE) one, in particular virtual, environment of the at least partially autonomously driving vehicle or one Vehicle comprising the part of the at least partially autonomously driving vehicle or the functionality of an at least partially autonomously driving vehicle. In another embodiment, there is a test case or a test environment or a test object in a first and a second version, wherein the second version represents a state of development of the test case or the test environment or the test object, the time of the development of the first version of the test case or Test environment or the test object.

In a further development of the embodiment described above, by means of the rules or in addition to the rules, a part of the present and expected results of the tests is not taken into account when calculating the probability that a test will be successful or unsuccessful, the part not too contingent results of one or more versions of at least one test case, at least one test environment and / or at least one test object, wherein the at least one test case, the at least one test environment and / or the at least one test object is included in the test. In a further embodiment, the predetermined rules represent a technical or statistical relationship between a first test case, a first test environment or a first test object in the first or second version and a second test case, a second test environment or a second test object in the first or second version.

In another embodiment, the result of a test may take at least the values "Test Success" (passed) and "Test Failed". In yet another embodiment, the predetermined rules are automatically created and / or verified by analyzing a database of at least a portion of the tests, including test cases, test environments, test objects, and results.

In a development of the previous embodiment, the analysis comprises a static evaluation of relationships of tests, in particular relationships of tests with positive results.

In one embodiment, prior to the probability calculation step, a first set of tests is determined using a statistical distribution, with results being or being generated for the first set of tests.

In a development of the previous embodiment, the determination of the tests of the first group of tests is based on one or more further static distributions of the test cases, the test environments and / or the test objects. In another embodiment of the previous embodiments, the static distribution and / or one or more of the further static distributions are random distributions.

In an alternative development of the previous embodiments, the calculation of the probability that a test for which no Result, will be successful, from the static distribution of the tests, the test cases, the test environments, and / or the test objects.

In a further embodiment, the presentation of the product maturity takes place in the form of a numerical, in particular percentage, test coverage or in the form of a, in particular color-coded graphic.

In an alternative embodiment, one or more criteria are specified and some of the tests for which in the second method step a probability that the test will be successful or unsuccessful have been calculated will be executed or suggested for execution, with the ones to be executed or executed proposed tests meet at least one of the given criteria.

In another embodiment, at least one of the predetermined criteria is above or below a threshold for the probability that the test will be successful or unsuccessful. In a further embodiment, a weighting is assigned to a test case (TC), a test environment (TE), a test object (SUT) or a combination of at least two elements from test case (TC), test environment (TE) and test object (SUT) and the weighting is taken into account in calculating the likelihood that a test for which no result will be successful or unsuccessful in the second step and / or in the presentation of the product maturity in the last step.

In an alternative embodiment, a threshold value of the product maturity is determined and, if the threshold for product maturity is exceeded, the test object (SUT) is released for a further development step.

In a development of the previous embodiment, the test object is assigned to a class of test objects (SUT), in particular by means of a level of the degree of automation, and the threshold value of the product maturity is defined as a function of the assigned class. The object is also achieved by a test system for testing a technical system, wherein the test system carries out one of the methods described above.

The invention will be explained in more detail with reference to the drawings. Here are similar parts with identical

Labels labeled. It shows:

Figure 1 shows schematically the composition of a test (1)

 Test environment (TE), test case (TC) and test object (SUT),

FIG. 2 schematically shows the relationships between test (1),

 Test environment (TE), test case (TC), test object (SUT) and result of the test (TR),

Figure 3 schematically shows the storage or storage of

 Test cases (TC) and test objects (SUT) and the

 Deriving different tests (1) from this,

FIG. 4 schematically shows the prediction of results (TR)

 unexecuted tests (1) by means of rules (5),

FIG. 5 schematically shows the prediction of results (TR)

 unexecuted tests (1) by means of rules (5),

Figure 6 is a tabular representation of results of tests

 (1) relative to different versions of a

 Test object (SUT),

Figure 7 is a tabular representation of predicted

Results of test (1), schematically the derivation of rules (5) from the in a data storage (3) stored or stored tests (1) and results of tests (TR). schematic representation of an at least partially autonomously driving vehicle. schematic representation of a classification into levels of different degrees of automation in at least partially autonomously driving vehicles.

In the figure of Figure 1, the composition of a test (1) is shown schematically. A test (1) comprises at least one test environment (TE) from a possible plurality of test environments (TE), one test case (TC) from a possible plurality of test cases (TC) and one test object ("system under test"). SUT) from a possible plurality of test objects (SUT).

The aim of the maturity determination of a technical system is to evaluate product characteristics such as performance, reliability or user-friendliness. From this it follows whether a next stage, a so-called milestone, has been reached in the development of the technical system. The final milestone in most cases is the release of the product for sale or delivery of the product. One area in which product maturity plays a major role, especially in terms of reliability and safety, is the automotive sector and, in this connection, the development of electronic control units (ECUs), so the test objects (SUT) can be control devices The choice of test cases (TC) depends, among other things, on the functionality to be tested and the objectives to be achieved for safety and reliability In the automotive industry, so-called "hardware-in-the-loop" tests (HIL tests) for the validation of real ECUs have been established.With appropriate test tools, test cases (TC) can be developed on the HIL test environments (usually special real-time computers) An example of such a test tool is the software product AutomationDesk from dSPACE When testing ECU software, simulation environments, so-called offline simulators, are also used, some of which can be executed on commercially available PCs. TE), HIL and offline simulator, it is also possible to use several identical or different ones for testing, so that possibly different test environments (TE) can be used for the test cases (TC).

2, schematic relationships between test (1), test environment (TE), test object (SUT) and result (TR) of a test (1) are shown schematically. In the execution of a test (1), data are collected which represent, at least partially, a result (TR) of the test (1) or from which a result (TR) of the test (1) can be derived. Typically, a test case (TC) is executed within or through a test environment (TE) for testing a test object (SUT). Upon or after this execution, results (TR) of the execution of the test (1), too Test results (TR) called, for example, by writing a file (English, "write") recorded. These results (TR) are assigned to the test (1) in order to be able to understand the conditions of the corresponding test execution in the later evaluation of the test results (TR). Typical results (TR) of tests (1) are "passed" and "failed". Since the use of the English terms "passed" and "failed" in the environment of testing and test administration are more common, they are also used in the following. For example, if the technical system to be tested is a control device for an automobile, the test object (SUT) is typically a prototype of this control device which is connected to an HIL simulator. The HIL simulator and the executable test software will then provide one Test environment (TE). The exact configuration of the HIL simulator is relevant for the traceability and reproducibility of the tests (1) and is defined as a test environment (TE). Changes to the hardware or software of the HIL simulator result in a new test environment (TE). After the execution of the test (1) comprising the test environment (TE), the test case (TC) and the test object (SUT), the results (TR) of the test (1) are stored and for the purpose of traceability with the test (1 ) connected. This data is preferably stored in a database and managed by a test management tool with connection to the database. An example of such a test management tool is the SYNECT software from dSPACE.

The depiction in FIG. 3 schematically shows the storage or storage of test cases (TC) and test objects (SUT) in a data storage system (3). The data storage can be a file system, a database or another, known from the prior art, electronic data storage. It is also possible that in the data storage only representatives of the actual test cases (TC) or test objects (SUT) or references to the actual test cases (TC) or test objects (SUT) are stored. This is e.g. useful or even necessary if the test objects are not electronically available data but physically present objects (eg electronic devices). From the stored test cases (TC) and test objects (SUT) different tests (1) can then be generated by distribution to one or more test environments (TE).

FIG. 4 schematically shows the determination according to the invention of expected test results (TR). To differentiate the expected test results (TR) from the present test results (TR), the former shows dashed rather than solid lines in the figures. Rules (5) are used to calculate the test results (TR) and assigned probabilities. These rules (5) can be specified by the user or created automatically by evaluating existing databases. An example of such a rule is that a test (1) comprising a test object (SUT) in a third version (V3) provides 99% of the same test result (TR) as a test (1) comprising the same test object (SUT) in a second version (V2) if test environment (TE) and test case (TC) are the same for both tests (1).

The determination according to the invention of expected test results (TR) is also shown schematically in FIG. 5. The above-described calculation of a test result (TR) can also be derived from a previously calculated instead of a present test result (TR) by means of a rule (5). This case is shown in Figure 5, wherein the tests shown (1) differ only in the version (VI, V2, V3) of the test object (SUT). The expected result of the test comprising version 2 (V2) of the test object (SUT) is calculated by means of a rule (5) from the present test result (TR) of the test (1) comprising version 1 (VI) of the test object (SUT) and the expected result of the test comprising version 3 (V3) of the test object (SUT) is calculated by means of a rule (5) from the expected test result (TR) of test (1) comprising version 2 (V2) of the test object (SUT). In the second calculation, the probability of the expected test result (TR) from the previous calculation is taken into account. An example of this is given if in the previous example to FIG. 4 the result (TR) of the test (1) comprising the test object (SUT) in version 2 (V2) is not present but in an analogous manner from an existing test result (TR ) of a test (1) comprising the test object (SUT) in version 1 (VI), with a probability of 99%. The calculation of the test result (TR) for test (1) comprising the test object (SUT) in version 3 (V3) by means of the same rule (5) also results in the test result (TR) for the test (1) comprising the Test object (SUT) in version 3 (V3) 99% equal to the test result (TR) for test (1), comprising the test object (SUT) in version 2 (V2). The overall result is then 98.01% (= (99/100) * (99/100) * 100) for the probability that the test result (TR) for the test (1) comprising the test object (SUT) in version 3 (FIG. V3) is the same as the test result (TR) for test (1), comprising the test object (SUT) in version 1 (VI). As an alternative to the different versions (VI, V2, V3), other variations of the test cases (TC), test environments (TE) or test objects (SUT), such as variants or different elements of a related group, are also possible. FIG. 6 shows tabular results (TR) of tests (1). There are different test environments (TE1, TE2, TE3), different test cases (TC1, TC2, TC3) and one test object (SUT) in different, consecutive versions (Versionl, Version2, Version3). The empty fields of the tables represent tests (1) for which there is no result (TR). Thus, FIG. 5 shows by way of example a database stored in a data storage system (3).

The product maturity of version 3 of the test object (SUT) is shown in tabular form and analogously to the tables in FIG. The individual fields of the table contain, according to the method of the invention, calculated expected results of the illustrated test (1), as well as the correspondingly calculated associated probabilities. It is readily apparent that the representation determined by means of the invention and shown in FIG. 7 gives a significantly better, because more complete, overview of the product maturity of the test object (SUT) in version 3 than the comparable representation in the lower third of FIG. For further improved readability, the fields of the table can also be displayed in color or in addition to the textual contents. Typically, green is used for tests (1) with the result passed and red for tests (1) with the result faiied. To represent the probabilities it is additionally possible to vary the hue or the color intensity of the green and red fields.

Assuming that the test object (SUT) is a controller prototype of an automotive control unit that is responsible for controlling the power window function, the test environments (TE1, TE2, TE3) could be HIL simulators with different software configurations , which different vehicle types represent. The test cases (TC1, TC2, TC3) could be, for example, the prevention of window closing (TC1), the emergency opening of the window in an accident (TC2), and the automatic closing of the window when the car is locked. The product maturity resulting according to the invention, as shown in FIG. 7, can now be used for different conclusions, depending on the criterion of evaluation and status in development. On the one hand, the lack of positive test results for the test case TC3 can lead to it being executed again or possibly after any improvement in the corresponding functionality. It could also be decided that the current development milestone (eg, safety-related tests with over 90% probability passed) has been achieved and the next stage of development is being addressed.

Likewise, from the product maturity determined according to the invention, as shown in FIG. 7, further information can be obtained or further compressed. For example, it can be concluded that less than four tests (1) are considered passed with a probability of over 90%.

FIG. 8 schematically shows an automatic derivation of the rules (5), represented by the filled-in arrow, from a database, stored in a data storage system (3), comprising tests (1) and test results (TR). As shown in FIG. 2, the test results (TR) are assigned to tests (1). The automatic derivation can be z. B. create one or more rules (5) from a common correlation. A possible example here is that a test (1) comprising a specific combination of test case (TC) and test object (SUT) is associated with a result (TR) percentage more frequently than a predefined threshold and from this the rule (5) is derived it will be that other tests (1) comprising the same combination of test case (TC) and test object (SUT) with a probability equal to the threshold will have the same result (TR) as the previously determined tests (1). At least partially or even fully autonomous vehicles have one or more sensors for collecting data, in particular data about the environment of the vehicle. Additionally, such vehicles typically have one or more interfaces to communicate with their environment. The figure Figure 9 illustrates this schematically. Typical sensors are radar, Lidar- or optical camera sensors, with which the environment is detected. The data exchange is usually realized via mobile radio standards (eg 4G or 5G). Furthermore, satellite-based systems (eg GPS) are frequently used for determining the position.

In the area of driver assistance systems or as their further development of highly automated or even autonomous driving, different levels or levels of the degree of automation are typically defined. These are shown in Figure 10. Level 0 stands for a system without assistance systems, in which all driving maneuvers, especially steering, acceleration and braking emanate only from the driver. Level 1 is referred to as assisted driving, as either steering or acceleration and braking are done automatically at times. Known systems are z. B. Automatic cruise control or Adaptive Cruise Control (ACC) system. At level 2, the system temporarily performs both steering, acceleration and braking tasks. This level of automation is called semi-automated driving. In levels 1 and 2, the driver must be able to intervene at any time so that he can take full control of the vehicle again at any time. Although the system temporarily takes over parts of the driving tasks, the driver must be able to follow the traffic at all times and intervene as part of his natural reaction time. High Automated Driving (HAF) is Level 3. In this case, the vehicle drives automatically and the driver merely represents a fallback position if the system can not cope with a traffic situation. The system will in such cases prompt the driver to intervene and to perform driving maneuvers appropriate to the traffic situation. This is the Driver given a finite but beyond the typical human reaction time period. Therefore, the driver can temporarily turn his attention away from the traffic. In levels 4 and 5, the automated system takes complete control and the driver no longer has to intervene. Whereby in Level 4 this may only apply in certain traffic situations (eg motorway driving), while at Level 5 the vehicle can cope with any traffic situation and thus practical driving without driver can be on the way.

These different levels of automation require different levels of safety when securing or testing the corresponding automated driving functions. According to their risk, the product release or even just the next step in the development process can be carried out with varying degrees of safety with regard to product maturity. A Level 3 system may not have a 99% probability error, as the driver is available as a fallback position, while Level 4 or 5 systems are 99.9% likely to have no errors. The achievement of these different threshold values can be determined or checked with the method according to the invention for product maturity determination. Since product maturity is determined by the existing test results (TR), the distribution of existing test results (TR) for the quality of the calculated product maturity is the decisive factor. For example, if all existing test results (TR) were generated with only one test environment (TE), the quality of test results (TR) on other test environments (TE) is probably not very meaningful. In order to obtain as meaningful a product maturity as possible, it may therefore be advantageous to generate the present test results (TR) on the basis of a predefinable distribution of tests (1) to be carried out previously. This distribution can z. B. be a random distribution. Knowledge of the way in which existing test results (TR) are distributed can then also be used to assess product maturity.

Claims

claims: 1. Method for determining product maturity by means of tests,  wherein a test comprises the execution of a test case (TC) by means of a test environment (TE) applied to a test object (SUT), and there is no result (TR) for at least one test, and  the method comprises the following steps:  a) specification of rules for calculating a probability that a test for which there is no result will be successful or unsuccessful,  where the rules use input or expected results of tests as inputs and return probabilities as output variables,  b) calculating the likelihood that a test for which there is no result will be successful or unsuccessful, by means of at least part of the given rules, and c) representing the product maturity depending on the probabilities calculated in the previous step. 2. Method for determining product maturity by means of tests,  wherein a test comprises the execution of a test case (TC) by means of a test environment (TE) applied to a test object (SUT) and there is no result (TR) for at least one test,  a test object (SUT) an at least partially autonomously driving vehicle, a part of an at least partially autonomous driving Vehicle or functionality of an at least partially autonomous vehicle, a test case (TC) is a driving maneuver of the at least partially autonomously driving vehicle, a driving maneuver with the part of the at least partially autonomously driving vehicle or a driving maneuver in which the functionality of an at least partially autonomously driving vehicle is taken into account, a test environment (TE) a, in particular also virtual, environment of the at least partially autonomously driving vehicle or a vehicle comprising the part of the at least partially autonomously driving vehicle or the functionality of an at least partially autonomously driving vehicle, and the method comprises the following steps:  d) specification of rules for calculating a probability that a test for which there is no result will be successful or unsuccessful,  where the rules use input or expected results of tests as inputs and return probabilities as output variables,  e) calculating the likelihood that a test for which there is no result will be successful or unsuccessful, by means of at least part of the given rules, and f) representing the maturity of the product as a function of the probabilities calculated in the previous step. A method according to claim 1 or 2, wherein there is a test case (TC) or a test environment (TE) or a test object (SUT) in a first and a second version, wherein the second version represents a development state of the test case (TC) or the test environment (TE) or the test object (SUT), which is temporally according to the state of development of the test case (TC) or the test environment (TE) or the test object (SUT). The method of claim 3, wherein Given the rules or in addition to the rules, part of the present and expected results of the test will not be taken into account in calculating the likelihood that a test will be successful or unsuccessful, with the portion of unacceptable results from one or more versions at least one test case (TC), at least one test environment (TE) and / or at least one test object (SUT) depends, wherein the at least one test case (TC) comprising at least one test environment (TE) and / or the at least one test object (SUT) from the test. 5. The method according to any one of the preceding claims, wherein  the predetermined rules a technical or statistical relationship between at least a first test case (TC), a first test environment (TE) or a first test object (SUT) in the first or second version and at least a second test case (TC), a second test environment (TE ) or a second test object (SUT) in the first or second version. 6. The method according to any one of the preceding claims, wherein  the result of a test can take at least the values "test successful" (passed) and "test failed". 7. The method according to any one of the preceding claims, wherein  the predetermined rules are automatically generated by analyzing a data set of at least part of the tests, including test cases (TC), test environments (TE), test objects (SUT) and results, and / or verified. 8. The method of claim 7, wherein  the analysis includes a static evaluation of correlations of tests, in particular correlations of tests with positive results. 9. The method according to any one of the preceding claims, wherein  before method step b) a first group of tests was determined by means of a statistical distribution and for the first group of Tests results are available or generated. 10. The method of claim 9, wherein the determination of the tests of the first group of tests is based on one or more further static distributions of the test cases (TC), the test environments (TE) and / or the test objects (SUT). 11. The method of claim 9 or 10, wherein  the static distribution and / or one or more of the further static distributions are random distribution. 12. The method according to any one of claims 9 to 11, wherein  the calculation of the probability that a test for which no Result, whether successful or unsuccessful, depends on the static distribution of the tests, the test cases (TC), the test environments (TE) and / or the test objects (SUT). 13. The method according to any one of the preceding claims, wherein  the representation of the product maturity takes place in the form of a numerical, in particular percentage, test coverage or in the form of a, in particular color-coded, graphic. 14. The method according to any one of the preceding claims, wherein  one or more criteria are specified and a part of the test for which in method step b) a probability that the test will be successful or unsuccessful has been calculated or proposed for execution, wherein the tests to be carried out or proposed for execution at least one of meet predetermined criteria. 15. The method of claim 14, wherein  at least one of the predetermined criteria, the exceeding or falling below a threshold for the probability that the Test will be successful or unsuccessful. 16. The method according to any one of the preceding claims, wherein A test case (TC), a test environment (TE), a test object (SUT) or a combination of at least two elements from test case (TC), test environment (TE) and test object (SUT) is assigned a weighting and the weighting when calculating the probability that a test for which there is no result will be successful or unsuccessful in step b) and / or the presentation of the product maturity in step c) is taken into account. 17. The method according to any one of the preceding claims, wherein  a threshold value of the product maturity is fixed and when exceeding the Product maturity threshold, the test object (SUT) will release for another development step. 18. The method of claim 17, wherein  the test object is assigned to a class of test objects (SUT), in particular by means of a level of the degree of automation, and the threshold value of the product maturity is determined as a function of the assigned class. 19. Test system for testing a technical system, in particular an electronic control unit or a part of an electronic control unit, wherein the test system performs one of the methods of claims 1 to 18.