CN114169125A - Method for testing a technical system using a simulation model and verification of a simulation model - Google Patents

Method for testing a technical system using a simulation model and verification of a simulation model Download PDF

Info

Publication number
CN114169125A
CN114169125A CN202111055770.2A CN202111055770A CN114169125A CN 114169125 A CN114169125 A CN 114169125A CN 202111055770 A CN202111055770 A CN 202111055770A CN 114169125 A CN114169125 A CN 114169125A
Authority
CN
China
Prior art keywords
simulation model
input parameters
technical system
testing
test
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111055770.2A
Other languages
Chinese (zh)
Inventor
A·格里姆
G·利佩
尹智洙
K·博恩
M·施密特
J·范凯勒
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Robert Bosch GmbH
Original Assignee
Robert Bosch GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Robert Bosch GmbH filed Critical Robert Bosch GmbH
Publication of CN114169125A publication Critical patent/CN114169125A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F30/00—Computer-aided design [CAD]
    • G06F30/20—Design optimisation, verification or simulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3696—Methods or tools to render software testable

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Geometry (AREA)
  • Quality & Reliability (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention relates to a method (100) for testing a technical system by means of a simulation model of the technical system, comprising the following steps: providing (110) a simulation model comprising a set of input parameters and a set of output variables; selecting (120) at least one output variable of the simulation model to be checked; deriving (130) at least one test requirement based on the output variable to be checked; selecting (140) a relevant input parameter from the set of input parameters by means of a sensitivity analysis of the input parameter from the set of input parameters based on the at least one test requirement; adapting (150) the at least one test requirement based on the relevant input parameters, and testing (160) the technical system by means of the simulation model based on the at least one test requirement.

Description

Method for testing a technical system using a simulation model and verification of a simulation model
Technical Field
The disclosure relates to a method for testing a technical system by means of a simulation model and to the use of the method for verifying a simulation model.
Background
A simulation model is understood to be a computer-implemented simulation of a technical system, in particular a computer-implemented simulation of a computer program, hardware or embedded system.
According to the present disclosure, the technical system is a computer program, hardware or embedded system.
It is desirable to efficiently perform testing of a technical system and/or verification of a simulation model.
Disclosure of Invention
The disclosure relates to a method for testing a technical system by means of a simulation model of the technical system. The method comprises the following steps:
providing the simulation model comprising a set of input parameters and a set of output variables;
selecting at least one output variable of the simulation model to be checked;
deriving at least one test requirement based on the output variable to be checked;
selecting a relevant input parameter from the set of input parameters by means of a sensitivity analysis of the input parameter from the set of input parameters based on the at least one test requirement;
adapting the at least one test requirement based on the relevant input parameters, an
Testing the technical system with the aid of the simulation model on the basis of the at least one test requirement.
According to the present disclosure, the technical system is a computer program, hardware or embedded system.
The dependence of the input parameters on the selected output variables to be examined is not always known a priori, in particular in the case of a large number of input parameters and/or in the case of a simulation model having a complex or unknown internal structure.
Thus, the test requirements derived based on the selected output variables may first include a large number of input parameters.
It is checked with a sensitivity analysis whether and how sensitively the at least one output variable reacts to a change of the input parameter. Based on this, an input parameter is selected from the set of input parameters as a relevant input parameter, the at least one output variable sensitively reacting to a change of the relevant input parameter.
By adapting the test requirements based on the relevant input parameters, the test requirements can be optimized.
Advantageously, the number of input parameters to be taken into account in the test requirements and when testing the system by means of the simulation model can be reduced by applying a sensitivity analysis. In particular, input variables whose changes do not lead to a change in the output variables are no longer taken into account in the test requirements and when testing the system with the aid of the simulation model. The test effort in the later stages during the automatic test generation is thereby also significantly reduced.
According to an advantageous embodiment of the method, the steps of selecting the relevant input parameters by means of a sensitivity analysis and adapting the at least one test requirement are carried out repeatedly. The step of selecting the relevant input parameters by means of a sensitivity analysis is then performed on the basis of the previously adapted test requirements. These steps may be performed repeatedly a number of times.
According to an advantageous embodiment, the method further comprises the step of pre-selecting an input parameter from the set of input parameters. The preselection may be performed manually, for example. Advantageously, irrelevant input parameters can thus be excluded already before the sensitivity analysis is applied.
According to one specific embodiment, the input parameters of the input parameter set are grouped based on their association with subsystems of the technical system and/or subsystems of a simulation model of the technical system. By grouping, subsystems that may cause deviation requirements or even violations may be more easily identified.
For example, if the simulation model is a black box simulation model, the user does not know the internal structure of the simulation model. For the user, grouping input parameters based on their association with possible subsystems of the technical simulation model may help to identify the cause of the error when testing the system (especially using search-based testing). In search-based testing, the input parameters of the simulation model are varied in order to find the minimum of formalized test requirements, i.e. violations of test requirements, on the simulation model. If only certain input parameters have a significant impact on the violation of the test requirements during the optimization (or search) traversal, the subsystems participating in the violation can be inferred therefrom.
According to a further advantageous embodiment, it is provided that the method further comprises the step of formalizing the at least one test requirement, in particular the signal comprised by the test requirement, using a signal time logic STL. STL is a specification language that allows specification of the temporal properties of signals with real values.
According to a further advantageous embodiment, it is provided that the method further comprises the step of generating the test case, in particular automatically. The test case generation is carried out in particular using the "search-based test" SbT method. In addition to generating test cases, the SbT method can also be used to search for critical test cases in a targeted manner. If the formalized requirements are violated, the test case is defined as a critical test case. By applying the sensitivity analysis and reducing the number of input parameters to be taken into account in the test requirements and when testing the system by means of a simulation model as described above, the test effort during the automatic test generation is significantly reduced. Furthermore, the optimization effort in the SbT method can be reduced.
According to a further advantageous embodiment, it is provided that the test case is generated, in particular automatically, on the basis of at least one formalized test requirement.
According to a further advantageous embodiment, provision is made for requirements to be quantified when testing the technical system with the aid of a simulation model. In this way, it can be determined to what extent the technical system meets or does not meet the test requirements.
Further embodiments relate to a device for testing a technical system by means of a simulation model of the technical system, wherein the device is designed to carry out the steps of the method according to an embodiment.
Further embodiments relate to the use of a method according to an embodiment and/or a device according to an embodiment for verifying a simulation model, in particular a black box simulation model. By means of the described method, simulation models with complex or unknown internal structures, so-called black-box simulation models, can be effectively verified.
Advantageously, the efficiency of verification is improved by the combination of automatic test case generation and formalization of test requirements.
Using the SbT method, it is possible to check, during the verification, how well the simulation model meets the requirements set for the intended application purpose.
Further features, application possibilities and advantages of the invention emerge from the following description of an embodiment of the invention which is illustrated in the figures of the drawings. All features described or shown here, individually or in any combination, form the subject matter of the invention, irrespective of how they are summarized in the patent claims or their citations, and irrespective of their wording or display in the description or the drawings.
Drawings
In the drawings:
fig. 1 shows a schematic representation of the steps of a method according to an embodiment in a flow chart.
Detailed Description
Fig. 1 shows a schematic representation of the steps of the method 100 in a flow chart. The method 100 is a method for testing a technical system by means of a simulation model of the technical system. Technical systems are in particular software, hardware or embedded systems. The technical system comprises for example a large number of technical-specific components and/or subsystems.
The method 100 comprises the steps of:
step 110, providing a simulation model comprising an input parameter set and an output variable set;
step 120, referring to and selecting at least one output variable to be checked of the simulation model;
step 130, deriving at least one test requirement based on the output variable to be checked;
a step 140 for selecting relevant input parameters from the set of input parameters by means of a sensitivity analysis of the input parameters from the set of input parameters based on the at least one test requirement;
a step 150 for adapting the at least one test requirement based on the relevant input parameter, an
Step 160 for testing the technical system by means of the simulation model based on the at least one test requirement.
It is checked with a sensitivity analysis whether and how sensitively the at least one output variable reacts to a change of the input parameter. Based on this, an input parameter is selected 140 from the set of input parameters as a relevant input parameter, the at least one output variable sensitively reacting to a change of the relevant input parameter.
Input parameters for which the output variables do not react sensitively to changes in the input parameters can be discarded from the test requirements.
By adapting 150 the test requirements based on the relevant input parameters, the test requirements can be optimized.
According to an advantageous embodiment of the method, the steps of selecting 140 relevant input parameters by means of a sensitivity analysis and adapting 150 the at least one test requirement are performed repeatedly. A step 140 of selecting relevant input parameters by means of a sensitivity analysis is then performed on the basis of the previously adapted test requirements. These steps may be performed repeatedly a number of times. This is schematically illustrated in fig. 1 by arrows at steps 140, 150.
According to an advantageous embodiment, the method further comprises an optional step 170 of pre-selecting an input parameter from the set of input parameters. The preselection 170 may be made manually, for example. Advantageously, irrelevant input parameters can thus be excluded already before the sensitivity analysis is applied.
According to one specific embodiment, it is provided that the input parameters of the input parameter set are grouped on the basis of their association with a subsystem of the technical system and/or a subsystem of a simulation model of the technical system.
According to a further advantageous embodiment, it is provided that the method further comprises a step 180 of formalizing the at least one test requirement, in particular the signal comprised by the test requirement, using a signal time logic STL. STL is a specification language that allows specification of the temporal properties of signals with real values.
According to a further advantageous embodiment, it is provided that the method further comprises a step 190 of generating the test case, in particular automatically. The test case generation 190 takes place in particular using the "search-based test" SbT method.
According to a further advantageous embodiment, provision is made for the test case to be generated 190, in particular automatically, on the basis of at least one formalized test requirement.
According to a further advantageous embodiment, provision is made for requirements to be quantified when testing 160 the technical system by means of a simulation model. In this way, it can be determined to what extent the technical system meets or does not meet the test requirements. For example, the method may be used for testing technical systems suitable for autonomous driving, in particular software and/or hardware and/or embedded systems for autonomously driven vehicles.
In testing systems for autonomous driving, scenarios are relevant in which a large number of changes in input parameters are unavoidable due to the numerous influences on the technical system to be tested. On the one hand, this relates to parameters of the vehicle, but on the other hand, parameters from the vehicle environment are also relevant. For example, in simulating city driving, other parameters such as driving/walking speed of other traffic participants, traffic light status, worksite location, etc. may also change in addition to the vehicle-related parameters. Thus, the testing requirement first includes a large number of input parameters. The sensitivity of the system to be tested to the input parameters is a priori unknown. Therefore, sensitivity analysis is performed in order to select the relevant parameters and the parameters identified as relevant in this process are absorbed into the adapted test requirements. The adapted test requirements are then formalized and, on the basis thereof, the test requirements are checked to what extent the system to be tested meets by means of automatic test case generation.
Other embodiments relate to the use of the method 100 to validate simulation models, particularly black box simulation models. The method is particularly suitable for simulation models whose internal structure is complex or unknown, so that a priori prediction of relevant input parameters is almost impossible. By means of the sensitivity analysis, relevant input parameters can be selected and the test requirements adapted on the basis thereof.

Claims (10)

1. Method (100) for testing a technical system by means of a simulation model of the technical system,
the method comprises the following steps:
providing (110) the simulation model comprising a set of input parameters and a set of output variables;
selecting (120) at least one output variable of the simulation model to be checked;
deriving (130) at least one test requirement based on the output variable to be checked;
selecting (140) a relevant input parameter from the set of input parameters by means of a sensitivity analysis of the input parameter from the set of input parameters based on the at least one test requirement;
adapting (150) the at least one test requirement based on the relevant input parameters, an
Testing (160) the technical system with the simulation model based on the at least one test requirement.
2. The method (100) according to claim 1, wherein the steps of selecting (140) relevant input parameters and adapting (150) the at least one test requirement by means of a sensitivity analysis are performed repeatedly.
3. The method (100) according to at least one of the preceding claims, wherein the method further comprises the step of pre-selecting an input parameter from a set of input parameters (170).
4. Method (100) according to at least one of the preceding claims, wherein the input parameters of the set of input parameters are grouped based on their association with subsystems of the technical system and/or subsystems of a simulation model of the technical system.
5. Method (100) according to at least one of the preceding claims, wherein the method further comprises the step of formalizing (180) the at least one test requirement, in particular the signal comprised by the test requirement, using a signal time logic, STL.
6. Method (100) according to at least one of the preceding claims, wherein the method further comprises the step of generating a test case (190), in particular automatically.
7. The method (100) as claimed in claim 6, wherein the test case is generated (190) in particular automatically on the basis of at least one formalized test requirement.
8. The method (100) according to at least one of the preceding claims, wherein a requirement violation is quantified when testing the technical system by means of the simulation model.
9. Device for testing a technical system by means of a simulation model of the technical system, wherein the device is configured to perform the steps of the method (100) according to at least one of claims 1 to 8.
10. Use of the method (100) according to at least one of claims 1 to 8 and/or the device according to claim 9 for verifying a simulation model, in particular a black-box simulation model.
CN202111055770.2A 2020-09-10 2021-09-09 Method for testing a technical system using a simulation model and verification of a simulation model Pending CN114169125A (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102020211361.0 2020-09-10
DE102020211361.0A DE102020211361A1 (en) 2020-09-10 2020-09-10 Procedure for testing a technical system using a simulation model and verification of the simulation model

Publications (1)

Publication Number Publication Date
CN114169125A true CN114169125A (en) 2022-03-11

Family

ID=80266652

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111055770.2A Pending CN114169125A (en) 2020-09-10 2021-09-09 Method for testing a technical system using a simulation model and verification of a simulation model

Country Status (2)

Country Link
CN (1) CN114169125A (en)
DE (1) DE102020211361A1 (en)

Also Published As

Publication number Publication date
DE102020211361A1 (en) 2022-03-10

Similar Documents

Publication Publication Date Title
US8612171B2 (en) Method of automatically formulating test cases for verifying at least part of a piece of software
US10025696B2 (en) System and method for equivalence class analysis-based automated requirements-based test case generation
CN112654933A (en) Computer-implemented simulation method and apparatus for testing control devices
King et al. A taxonomy and survey on validation approaches for automated driving systems
US20230236088A1 (en) Computer-aided method and device for predicting speeds for vehicles on the basis of probability
US8751094B2 (en) Method for validation of a graphically based executable control specification using model extraction
CN112182782B (en) Carrier rocket environment adaptability analysis method, system, terminal and medium
US20220245260A1 (en) Method for checking the security of a technical unit
CN114169125A (en) Method for testing a technical system using a simulation model and verification of a simulation model
US20220358024A1 (en) Computer-implemented method for scenario-based testing and / or homologation of at least partially autonomous driving functions to be tested by means of key performance indicators (kpi)
Raghupatruni et al. Credibility of software-in-the-loop environments for integrated vehicle function validation
KR20210023722A (en) Method for testing a system to a request
Zander-Nowicka et al. Automotive validation functions for on-line test evaluation of hybrid real-time systems
Hallerbach et al. Simulation-Enabled Methods for Development, Testing, and Validation of Cooperative and Automated Vehicles
Lauber et al. Virtual test method for complex and variant-rich automotive systems
Neubauer et al. Model-based development and simulative verification of logical vehicle functions using executable UN/ECE regulations
EP4191462A1 (en) Method and system for quantifying vehicle requirement
US20240104273A1 (en) Verification system, verification method, and verification program
CN113673088B (en) Automatic test system construction method and system
US20240193071A1 (en) Computer-implemented method and system for generating test data for computer-implemented automated driving functions
CN116911006A (en) Drive test accident analysis method and device, storage medium and computer equipment
US20230350699A1 (en) Schema driven user interface creation to develop autonomous driving applications
Wittel et al. Automatic Test Evaluation for Driving Scenarios Using Abstraction Level Constraints
CN116848542A (en) Risk handling method and related device
CN117289682A (en) Vehicle diagnosis test method and system and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination