KR101291817B1 - Test case generating system and method based on requirements model - Google Patents

Abstract

PURPOSE: A requirement model based test case automatic generation system and a method thereof are provided to generate a test case from a standardized requirement model which describes a requirement of an embedded software in a standardized way. CONSTITUTION: A test case generation tool (110) generates a test case by using a coverage target manager (120), a model state tree manager (130), an input generator (140), and a model simulator (150) according to a test case generation algorithm. The coverage target manager generates a coverage target from a model according to a coverage reference which a user selects and confirms whether the coverage target is achieved or not in a test case generation process. The model state tree manager makes and manages a state tree in order to visit every corner of a requirement model. The input generator generates a value to be applied according to an end port of a necessary system when finding the coverage target in a test case generation algorithm. The model simulator simulates a model in order to use a dynamic data analysis technique. A test case DB (160) stores the generated test case. [Reference numerals] (110) Test case generation tool (test case generation main algorithm); (120) Coverage target manager; (130) Model state tree manager; (140) Input generator; (150) Model simulator; (160) Test case DB

Description

System and method for automatic generation of requirement model-based test cases {TEST CASE GENERATING SYSTEM AND METHOD BASED ON REQUIREMENTS MODEL}

The present invention relates to an automatic test case generation technology of an embedded system, and more particularly, an automatic test for generating test cases in a standardized manner from a formal requirement model that describes the requirements of an embedded software formally. It relates to a case generation system and method.

In general, in the process of developing various electronic products (embedded systems) such as automobiles, heavy industry equipment, household appliances, etc., the testing step is an essential process to ensure the quality of the product. This test step is done in several stages, usually four stages. First, the first step is to perform unit tests during software and hardware development, and to develop hardware and software at the same time due to the characteristics of the electronic device, and to test the software itself and the hardware itself in each development department. The second stage is the electrical equipment unit test, in which software is installed on the hardware, and the electrical test is performed on the electrical unit. In this case, the input / output for the test uses a hardware input / output port. The third stage is the integrated test, in which automobiles and heavy equipment are developed and assembled separately, and a single finished product is produced. Therefore, these components are assembled and tested. Therefore, we test that the development electronics perform their normal functions when combined. The fourth step is the full test, which is usually performed during the production stage, to look at the very basic functions to look for quality problems due to assembly or component errors.

On the other hand, in order to perform the test, it is important to create a test case for each step purpose. In particular, of the four stages, the most suitable test stage with the technology is the second stage, and the third stage can also be utilized. The electronics unit test must ensure that the electronics are implemented as originally planned.

Conventionally known automatic test case generation methods include an automatic test case generation method and system registered under registration No. 10-0709964. The conventional automatic case generation system converts a standard SDL into readable data according to a conversion rule. A test case for generating a test module in a table combining notation format corresponding to the converted SDL according to the analysis module and the result of analysis and verification of the coverage module; It consists of a generation module to generate a test case used in the communication system.

Here, a test case refers to a sequence of inputs and their corresponding expected outputs. Therefore, test cases can be represented in a table form. For example, as shown in Table 1, the vertical axis (column) represents time, the horizontal axis (column) represents input variables and output variables, and the corresponding values are represented. The electronic unit test is usually performed in the form of a black box, so the input and output variables correspond to the input and output of the actual ECU. Internal variables used only for the requirements model cannot be used.

Time (msec) I ₁ I ₂ I ₃ O ₁ 0 0 0 0 0 100 One 0 0 1100 One One 0 2000 One One 0 One 3000 One One One One 3100 One One One 0

As a result, the test cases that need to be performed in electronics unit tests are the key to determining test success. Therefore, test cases must be able to be elaborated for the purpose in a standardized way. In particular, test cases used in electronic unit tests must satisfy the following items: First, in the unit test of the electronics, the test case must be generated based on the requirements because basically all the items described in the requirements must be checked to ensure that they are correctly implemented. Second, test cases need not only test cases made by formalized and standardized methods, but also test cases with the experience of engineers. Third, test cases made by standardized methods need to be quantified to increase the reliability of test results.

The present invention is proposed for a black box test that can be applied to the electronics unit or integrated test in the developed embedded system. The object of the present invention is to provide a general formal requirements model (especially Matlab Simulink / Stateflow and graphical system requirements). The present invention provides a system and method for automatically generating a requirement model based test case for automatically generating test cases based on coverage.

Another object of the present invention is to analyze and analyze the items that need to be tested by analyzing the internal logic of the requirements by using a coverage technique, without converting Simulink / Stateflow or graphical system requirements into other forms. The present invention provides a system and method for automatically generating a requirement model based test case for extracting an external input to be applied in order of time and extracting an output at this time.

Thus, the test case generated according to the present invention is used for testing in the form of a black box in order to confirm whether the developed embedded system is implemented according to the requirements envisioned at the beginning of development.

In order to achieve the above object, the system of the present invention includes a coverage target management unit for generating a coverage target from a requirements model according to a coverage criterion selected by a user and checking whether the coverage target is achieved in a test case generation process; Define the state by using internal state variables and internal state values of the requirement model, but define the state of the requirement model as one node by applying inputs from the initial state of the requirement model, and create the tree A model state tree management unit managed by; An input generation unit for generating a value to be applied for each end port of a requirement model required to find a coverage target in a test case generation algorithm; To input test cases for black box tests without changing the formal requirements model, use dynamic data analysis techniques to authorize test case inputs found in the requirements model, to verify that the actual input targets have been met by the inputs. A model simulator for examining the system state and obtaining an expected output value; Test case DB; And a test case generation tool configured to generate a test case generated by satisfying the coverage target and store the test case generated in the test case DB.

The coverage goal management unit analyzes requirements to generate coverage goals, classifies and manages coverage goals that have not been achieved and coverage goals that have not been achieved as test cases are generated, and displays statistics, and the model simulator. Obtains the state of the system model from the unsatisfied coverage targets, and the model state tree management unit evenly distributes the nodes when the test case generation tool attempts to select a starting node for expansion. Nodes are randomly selected using weights in order to extend to a uniform distribution and avoid exhaustion.

The input generation unit analyzes the correlation with the input to arrive at the desired internal state, and inputs the corresponding input, generates the input in sequence according to a specific diagram, uses a mathematical statistical distribution, and The value to be put in the input variable is determined by either weighting the value and generating the value according to the weight, or randomly inputting the value.

The model simulator examines an output value by simulating a model according to an input in the test case generation tool, and confirms that an internal variable value or state changes while simulating a model according to the input. To this end, a change value of an output corresponding to an input that changes with time is provided, or an internal variable or state value corresponding to an input that changes with time is provided. In order to move to a specific state of the system, a function of storing a state of a specific time point and rolling back to the stored time point is performed.

In order to achieve the above object, the method of the present invention includes a first step in which a coverage target manager generates a coverage target according to a user setting; A second step of the model state tree manager selecting a state node for tree extension; A third step of generating, by the input generator, an input to satisfy the new coverage target; A fourth step of applying an input to the model simulator and simulating it; A fifth step of confirming and recording whether a coverage goal is achieved during the simulation by the coverage goal manager using the model simulator; If the simulation ends and the newly achieved coverage target exists, the generated input is valid as a test case, and thus storing the test case including the input order and the expected output in the test case DB; And if all of the coverage targets are satisfied, and if there is a coverage target that is not yet satisfied by the coverage target management unit, it moves back to selecting a node for the tree extension to find the coverage target, and otherwise ends the test case generation. Characterized in that the seventh step.

In the second step of selecting a state node for the tree extension, the node is selected using a weight-based random selection scheme using weights. After the end of step 6, if a new coverage target is achieved, or if the output changes, if the state of the internal state diagram changes, or if the key variable value specified by the engineer changes, then a new node is created and Expand the tree from selected nodes.

The requirements model-based test case automatic generation system and method according to the present invention finds an external input sequence over time for black box testing. In order to generate test cases, the original formal requirements are directly analyzed without changing to other types of models, and the state of the system is tracked using dynamic analysis methods as well as static analysis.

In addition, the present invention can construct a tree to track the state of the system in dynamic analysis and track the state of the system, and use all formal requirements, but preferably test cases from Matlab Simulink / Stateflow and graphical system requirements. The test case generation technique can be processed by finding an input sequence to satisfy specific coverage of the formal requirements internal conditions.

1 is a flow chart showing the electrical equipment unit test procedure according to the present invention,
2 is an example for explaining generating a test case from a formal requirement model according to the present invention;
3 is a block diagram illustrating a system for automatically generating a requirement model based test case according to the present invention;
4 is a flowchart illustrating a test case generation procedure of the test case generation system according to the present invention;
5 is a tree example of a model state for management by the model state management unit according to the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features, and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings. The following examples are merely illustrated to illustrate the present invention and are not intended to limit the scope of the present invention.

1 is a diagram illustrating an electronic device unit test procedure according to an embodiment of the present invention, in which a test case generation tool receives a requirement model that is formally represented to generate a test case set for an electronic device test (S1 to S3), and a black box The test execution tool receives a set of test cases for the test of the electronic device, and performs a black box testing of the system under test (S4 and S5).

Referring to FIG. 1, the test case generation tool S2 generates a test case from a standardized requirement model in a standardized manner, and the generated test case is black box tested using the test execution tool S4. Proceed.

As described above, the standardized test case generation method capable of quantifying the test case based on the requirements according to the present invention uses a test case generation method using coverage criteria, and the coverage criteria widely used are as follows.

Statement coverage is a condition that a sentence must be examined or performed once, if the condition or action of the requirement is expressed in a sentence.

Condition coverage is called a set of test cases that satisfy condition coverage when the generated test case examines all conditions. Here, the condition refers to the logical minimum input variable used in the conditional sentence. For example, when the sentence "C ₁ C ₂ " is present, a total of two conditions C ₁ and C ₂ exist in the sentence. Therefore, the test case must examine every value of each condition. In other words, we need a test case to check when C ₁ is 'True' and 'False', when C ₂ is 'True' and 'False'.

Decision Coverage says that when the generated test case examined all the decisions, the set of test cases met Decision Coverage. Here, the decision is the result value in the conditional sentence. For example, if you have a "D = C ₁ C ₂ " statement, you should check that the D value is 'True' and 'False'.

Modified Condition and Decision Coverage (MC / DC) means that all test cases meet decision coverage and condition coverage, and in addition, the decision is made by changing one condition with a fixed value of another condition. The addition of a restriction to see changes. For example, when there is a sentence "D = C ₁ C ₂ ", the MC / DC must satisfy the Truth Table in Table 2 below.

No. C ₁ C ₂ D = C ₁ ∧C ₂ One T T T 2 T F F 3 F T F

When, as shown in the above table, when C ₁ is fixed, and to determine (Decision) value is changed in accordance with the change of the C _2, when C ₂ is fixed, the crystal (Decision) value is changed in accordance with the change of the C ₁ It can be seen that.

State coverage refers to the need to visit a state once in a state diagram. Transition Coverage means that you must visit every transition between states in a state diagram once.

Easily create test cases for blackbox testing according to different coverage criteria, from formal requirements written models, especially from formal requirements created with Matlab Simulink / Stateflow or graphical system requirements representation techniques. The reason for this is as follows. First, internal variables or data items used in the formal requirements model are not reflected in the development of the actual electronics. The same data item or variable as the actual electronics is the input / output. Second, coverage criteria are created from the logic inside the requirement model. And such logic consists of internal variables that are temporarily used by many models. Therefore, extracting a test coverage item does not make it possible to apply it as an input to a developed electronic device or to view data values. In addition, the target electrical equipment to be tested usually prefers to test by controlling only external input / output, and even if an interface for controlling internal variables is provided, all items cannot be applied.

2 is an example to illustrate generating a test case from a formal requirement model. In FIG. 2, the requirement is that n inputs are defined as I ₁ , I ₂ , .... I _{n -1} , I _n , with m outputs O ₁ , O ₂ , .... O _m-1 , O _m is defined. Depending on these inputs and outputs, the requirements logic can be represented using state diagrams or other means. At this time, internal variables "From" and "To" are used to express the internal state. Test coverage is then expressed using these internal variables. For example, for the transition condition "To == T && From == F", the MC / DC is satisfied if the following condition is satisfied.

To == T From == F To == T && From == F T T T T F F F T F

However, since the actual implementation of the electronics (ECU) cannot directly control the variable To and the variable From, the inputs I ₁ , .... I _n of the electronics are changed according to the time order so that "From" and It should produce a "To" value.

3 is a block diagram illustrating a system for automatically generating a requirement model-based test case according to the present invention.

Test case automatic generation system 100 according to the present invention is a test case generation tool 110, coverage target management unit 120, model state tree management unit 130, input generation unit 140, as shown in FIG. The model simulator 150 and the test case database 160 may be configured to automatically generate an input sequence in chronological order by applying coverage criteria from formal requirements, and generate an expected output value according to the corresponding input sequence.

Referring to FIG. 3, the test case generation tool 110 may include a coverage target manager 120, a model state tree manager 130, an input generator 140, and a model simulator 150 according to a test case generation algorithm as described below. To create a test case.

The coverage target manager 120 generates a coverage target from a model according to a coverage criterion selected by a user, and checks whether the coverage target is achieved in a test case generation process. The coverage target is referred to as a coverage target as an item that must be satisfied according to the coverage criteria. The test case generation tool 110 generates a test case, that is, an input application sequence over time, for satisfying this coverage goal.

The model state tree manager 130 creates and manages a state tree in order to visit every corner of the requirement model. In order to create a test case that satisfies high coverage, many conditions of the requirement must be identified. Therefore, every corner of the requirements model needs to be reviewed. To do this, first, state is defined using internal state variables and internal state values of the requirements model. As the model accepts input values from the initial state of the system (requirement model), the state of the system (requirement model) continues to be newly defined. This state is defined as a node, and these nodes are managed as a tree as shown in FIG.

In the test case generation algorithm, the input generator 140 generates a value to be applied for each terminal port of the system, which is required when trying to find a coverage target. The input generator 140 generates a value by statically analyzing the model, generates a random value at any time, generates a specific line, or generates a value according to a statistical distribution.

The model simulator 150 simulates a model to use a dynamic data analysis technique. Dynamic data analysis is used as well as static analysis to enter test cases for black box tests without changing the formal requirements model. The model simulator 150 is used to apply a test case input found to the model and to check whether the actual coverage target is achieved by the input, or to check the system state or to obtain an expected output value.

The test case DB 160 is a storage place for storing the generated test cases.

4 is a flowchart illustrating a test case generation procedure of the test case generation system according to the present invention.

Referring to FIG. 4, a first step S101 is a step of generating a coverage target according to a user setting. The coverage goal is in charge of the coverage goal manager 120, and generates a goal in the corresponding module.

The second step (S102) is a step of selecting a state node for tree expansion, and a new node is generated according to a change in time or an input value at a specific node. The node from which the tree starts to expand is selected using the model state tree manager 130.

The third step S103 is an input generation step, in which an input is generated to satisfy a new coverage target. The corresponding step is in charge of the input generator 140.

The fourth step S104 is a step of applying an input to the model simulator 150 and simulating the generated input.

The fifth step S105 is a step of searching for a newly achieved coverage target, and confirming and recording the coverage target management unit 120 using the model simulator 150 to determine whether the coverage target is achieved during the simulation.

The sixth step S106 is a test case registration step including an input order and an expected output. If the simulation is terminated and a newly achieved coverage target exists, the generated input is valid as a test case, so that the test case DB 160 Save the test case in

The seventh step S107 is a step of checking whether all of the coverage targets are satisfied. If there is a coverage target that is not yet satisfied in the coverage target management unit 120, the mobile terminal moves to the input generation part again to find the coverage target. End the test case generation.

Referring back to FIG. 3, the coverage target manager 120 is responsible for the following functions. Create a coverage goal by analyzing requirements models against user-specified coverage criteria. The embodiment of the present invention supports the coverage criteria shown in Table 4 below.

Statement Coverage Criteria
Condition Coverage Criteria
Decision Coverage Criteria
Modified Condition and Decision Coverage (MC / DC) Criteria
State Coverage Criteria
Transition Coverage Criteria
Boundary Coverage Criteria

In addition, the coverage target manager 120 manages the entire coverage target, classifies and manages the coverage targets that have not been achieved and the coverage targets that have not been achieved as test cases are generated, serves to show statistical data, and the model simulator 150. Gets the state of the system model from) and finds the newly achieved coverage target among the unsatisfactory coverage targets. The coverage goal is expressed as a conditional expression of internal data variables of the formal requirements. For example, in the case of state coverage, assuming that a variable called v_state representing a state exists, it is expressed in the form of v_state == "state_off".

However, in the actual implementation of the tool, the above format is expressed as a post-fix formula so that it can be quickly checked when checking the coverage target from the dynamic requirements model.

The model state manager 130 defines the model state by using internal state variables of the requirement model and state values of the state diagram. That is, the state of the model can be uniquely defined using Equation 1 using internal variables and state values of the state diagram.

Here, ValueOf (V _k ) means the value of the internal variable or state variable V _k .

And to manage these model states as a tree, we define a model state as a node of the tree. The tree node is expanded as shown in FIG. 5 by S _m (I), which is a set of time-dependent input variable values. The module generating the input variable value set S _m (I) is the input generator 140.

Referring to FIG. 5, the tree is expanded in accordance with the input value in the initialization state S ₀ . Accordingly, the model state manager 130 manages the state of the requirement model in a tree as shown in FIG. 5.

When the internal state changes, the node expands in the fifth step S105 of FIG. 4 by any input. However, if the node is expanded indefinitely, the state may explode. Therefore, the node is expanded according to the input only in the following table.

When new coverage goals are achieved
If the output has changed
When the state of the internal state diagram changes
When the value of the key variable specified by the engineer has changed

In addition, when the node is newly expanded, the coverage target management unit 120 is used to record the coverage achieved when the node is expanded. Using this information, the test case generation is finished, and the final test case is generated by concatenating S _m (I), which is an input set for visiting a node that has achieved new coverage. For example, in FIG. 5, when S ₆ achieves a coverage goal, an input sequence of the following Equation 2 is a test case for the coverage goal.

Finally, the model state tree manager 130 selects a node and delivers it to the main algorithm when the start node for the next extension is selected in the test case generation main algorithm. When you expand a tree, how you choose which nodes you want to expand is important. To this end, the present invention uses a weight-based random selection technique using weights so that nodes are evenly selected to expand to a uniform distribution and starvation does not occur. The weight-based random selection technique is as follows. That is, node selection is randomly selected but weighted. The default weight of the node is 10, but decrements by 1 when the node is selected. When the weight of the node becomes zero, the weight of the node is initialized to ten. At this time, the default value of weight can be tuned by the engineer so it does not restrict the relevant part.

The input generator 140 is a module that determines what value to put in the terminal input variable of the system. The module generates inputs in the following ways to achieve new coverage goals. First, in order to reach the desired internal state, the correlation with the input is analyzed and the corresponding input is input, and the input is generated in order according to a specific diagram. And the method of using mathematical statistical distribution (Poisson Distribution, Exponential Distribution), weighting the input value, generating the value according to the weight, and randomly inputting the method. have.

The model simulator 150 is responsible for simulating the behavior of the model according to the input. The model simulator 150 is used to examine an output value by simulating a model according to an input in a test case generator, and to determine whether an internal variable value or state changes according to the input. This is because checking the internal variable values or state changes can only confirm whether the test coverage targets have been met.

The model simulator 150 provides the following functions to the test case generation tool 110. In other words, the 'Snapshot' function can obtain the change of output corresponding to the input that changes with time, the internal variable or the state value corresponding to the input that changes with time, and save the state of the simulator. And, restore function is provided.

The test case DB 160 stores an input sequence that satisfies the test coverage target. The stored test case obtains and records not only the input sequence over time but also the corresponding expected output value from the model. It also stores the test coverage information achieved by the test case.

Although the present invention has been described with reference to some embodiments shown in the drawings, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

100: test case generation system 110: test case generation tool
120: coverage target management unit 130: model state tree management unit
140: input generator 150: model simulator
160: test case DB

Claims (8)

A coverage target manager configured to generate a coverage target from a requirement model according to a coverage criterion selected by the user, and confirm whether the coverage target is achieved in a test case generation process;
Define the state by using internal state variables and internal state values of the requirement model, but define the state of the requirement model as one node by applying input values from the initial state of the requirement model, and create the nodes A model state tree management unit managed by;
An input generation unit for generating a value to be applied for each end port of a requirement model required to find a coverage target in a test case generation algorithm;
To input test cases for black box testing without changing the formal requirements model, use dynamic data analysis techniques to authorize test case inputs found in the requirements model, to verify that the actual input targets have been met by the authorized inputs, or A model simulator for viewing system status or obtaining expected output values;
Test case DB; And
And a test case generation tool for generating a test case for satisfying the coverage target and storing the generated test case in the test case DB. The method of claim 1, wherein the coverage target management unit
As test cases are generated, it manages and manages the coverage goals that have not been achieved and the coverage goals that have not been achieved, displays statistics, and obtains the status of the system model from the model simulator. A requirement model based test case automatic generation system characterized by finding the coverage targets achieved.
The method of claim 1, wherein the model state tree management unit
When the test case generation tool wants to select a start node for expansion, the node is selected evenly by using a weight so that the nodes are evenly selected to expand to a uniform distribution and depletion does not occur. Model-based test case automatic generation system.
The method of claim 1, wherein the input generating unit
In order to reach the desired internal state, we analyze the correlation with the input and insert the corresponding input, generate the inputs according to a specific line in turn, use the mathematical statistical distribution, and weight the input values. And a method model-based test case automatic generation system comprising determining a value to be put in an input variable by one of a method of generating a value according to a weight and a method of randomly inputting a value. The method of claim 1, wherein the model simulator
For the purpose of reviewing the output value by simulating the model according to the input in the test case generation tool, or for the purpose of confirming that the internal variable value or state changes when the model is simulated according to the input, Requirement model-based test case automatic generation system which provides change value of corresponding output, internal variable or state value corresponding to input that changes over time, and save and roll back the state of simulator . A first step of the coverage target manager generating a coverage target according to a user setting;
A second step of the model state tree manager selecting a state node for tree extension;
A third step of generating, by the input generator, an input to satisfy the new coverage target;
A fourth step of applying an input to the model simulator and simulating it;
A fifth step of confirming and recording whether a coverage goal is achieved during the simulation by the coverage goal manager by using the model simulator, and expanding the node;
If the simulation ends and the newly achieved coverage target exists, the generated input is valid as a test case, and thus storing the test case including the input order and the expected output in the test case DB; And
Check if all coverage goals are met and if there is a coverage goal that is not yet satisfied by the coverage goal management unit, go back to the input generation part to find the coverage goal, otherwise the requirement consists of the seventh step of ending test case generation. How to automatically generate model-based test cases. 7. The method of claim 6, wherein the second step of selecting a node to start tree expansion for the tree expansion
A method for automatically generating a requirement model based test case, wherein nodes are selected using a weight based random selection technique using weights. The method of claim 6, wherein the fifth step of performing the tree expansion
Automated generation of requirement model-based test cases, characterized in that nodes are expanded only when a new coverage target is achieved, when the output changes, when the state of the internal state diagram changes, or when the key variable specified by the engineer changes. .

Images