WO2005059744A1

WO2005059744A1 - A method of automatically generating test scripts from a system specification model

Info

Publication number: WO2005059744A1
Application number: PCT/PL2003/000144
Authority: WO
Inventors: Marcin Lipinski; Lukasz Rykala
Original assignee: Motorola Inc
Priority date: 2003-12-18
Filing date: 2003-12-18
Publication date: 2005-06-30
Also published as: AU2003291794A1

Abstract

A method of automatically generating test scripts from a system specification model is presented. A system specification model can include variables and events that define values of these variables and also events that pass these values within the specified system. The testing of this model is enhanced by tracing the variables and their associated values as they are relayed around the specified system, and detecting the veracity of the values at strategic points in the system.

Description

A Method Of Automatically Generating Test Scripts From A System Specification Model

Technical Field

The invention relates to a method of automatically generating test scripts from a system specification model. In particular, it relates to a method of automatically generating test scripts from a system specification model that enable variable value equality checks.

Background

Methods are known for automatically generating test scripts from a system specification model, thus providing a set of tests to be carried out upon a system. Methods are also known for achieving substantially optimal test coverage of the system whilst reducing the amount of time required to carry out the test.

An example of such a method, together with a means of providing a test script produced by such a method and a system for executing test script produced may be found in EP 1 176 512 A2.

A system specification model for use in EP 1 176 512 A2 can include variables and events that define values of these variables and also events that pass these values within the specified system.

However, it is desirable to enhance the testing of this model by tracing the variables and their associated values as they are relayed around the specified system, and detecting the veracity of the values at strategic points in the system. This requires automatic identification of the appropriate strategic points in the system at which to store and compare variable values.

The purpose of the present invention is to address the above problem.

Summary of the Invention

In a first aspect, the present invention provides an enhanced method of automatically generating test scripts from a system specification model, characterised by the steps off- selecting at least a first group of receiving events; assigning, within a test script, an operation to store a variable value and group index to that receiving event of said group which is located closest to the root of said test script; and assigning, within said test script, an operation to compare said variable value and group index to each of the remaining members of said group, as claimed in claim 1.

Further features of the present invention are as defined in the dependent claims .

Embodiments of the present invention will now be described by way of example with reference to the accompanying drawings, in which:

Brief description of the drawings

FIG. 1 is a schematic of an example message sequence chart comprising three sub-systems P, Q and R. FIG. 2a is a directional graph generated from the example message sequence chart of FIG. 1.

FIG. 2b is a directional graph generated from the example message sequence chart of FIG. 1.

FIG. 3a is a further annotated version of Fig 2a illustrating set selection and vertex removal in accordance with an embodiment of the present invention.

FIG. 3b is a notional graph further illustrating set selection in accordance with an embodiment of the present invention

FIG. 4 is a flowchart of a label assignment process in accordance with an embodiment for the present invention.

FIG. 5 is a further annotated version of Fig 2a highlighting label assignment in accordance with an embodiment for the present invention.

Fig. 6a is a test script generated from the example message sequence chart of FIG. 1 in accordance' with an embodiment for the present invention.

Fig. 6b is a test script generated from the example message sequence chart of FIG. 1 in accordance with an embodiment for the present invention.

Detailed description

A method of automatically generating test scripts from a system specification model is disclosed. A system specification model is typically given as a message sequence chart (MSC) . An exemplary MSC 100 is given in FIG. 1. MSCs comprise events, such as the sending of a message, receiving of a message, or defining a system variable value.

The system represented in FIG. 1 consists of three subsystems P, Q and R. For the purposes of example it is assumed that the system includes a system under test (SUT) , here consisting of sub-systems P and Q.

Each arrow in the figure indicates a message, and comprises a sending event and a receiving event. In addition, there are six events defining values of the system variable V : defX, def_2, def_3, def_4, def_5, def_6.

In this example, each sending event passes a value of variable v' and each receiving event obtains a value of variable V .

Each event belongs to one of sub-systems P, Q, R. The events are ordered within each sub-system as they are presented on the sub-system lines from^' the top of the figure to the bottom (e.g. within the sub-system P: event send A must occur after event def_l) . The dashed lines indicate that some events are not ordered (e.g. events def_5 and def_6 could occur in any order) . The ^Λopt' labelled area indicates that event def_2 is optional i.e. it may or may not occur.

As is known in the art, a system specification is typically converted into a graph or set of graphs . Graphs are directed, in that each edge connects the predecessor vertex with the successor vertex. Each graph vertex represents an event and each graph edge represents a direct relationship between two events. A vertex can be a send, receive or define vertex. A pair of a send and receive vertices can be associated, i.e. they represent the sending and the receiving events of the same message. Each event can be represented in a plurality of graphs. FIGs 2a and 2b show two graphs obtained from the exemplar MSC of FIG 1. The graphs differ in the presence or absence of the optional define vertex def_2. The edges indicate direct relationships among events, represented by the connected vertices (e.g. event send A must occur after event def_l; events def_5 and def_6 could occur in any order) . Vertices belong to the same sub-systems as the events they represent. The dashed lines separate vertices belonging to particular sub-systems P, Q and R.

In an embodiment of the present invention, vertices that are not essential to checking values of the variable being considered are removed from the graphs.

When a vertex is removed from the graph, all the edges connected to it are also removed. New edges are added to the graph connecting the vertices that are predecessors of the removed vertex with the vertices that are successors of the removed vertex; only predecessors and successors that belong to the same sub-system as the removed vertex are thus connected.

Defining the ^owning sub-system' as the sub-system that defines the variable, then a vertex is only removed if it does not represent one of the following events: i. a defining of a variable value by the owning subsystem; ii. a sending of a variable value to a non-owning subsystem; and iii. a receiving of a variable value by a non-owning sub-system.

FIG 3a. shows, inter alia, the exemplar graph of FIG. 2a following the above removal process. For variable v, the owning sub-system is P. Vertices ^Λsend G' and recv G' have been removed, as they did not represent any event in accordance with the above embodiment: i. defining of a variable V value by P; ii. sending a value of ^λv^r to Q or R; or iii. receiving of a value of V by Q or R.

In an embodiment of the present invention, each vertex is assigned a vertex value. Each vertex value acts a label that represents an actual value or values of the considered variable (in the present example, ^Λv' ) , but is not an actual variable value itself.

To describe this step of assignment, it is useful to define the following sets of vertices for at least a first selected vertex: i. complete successors; ii. previous vertices, next vertices; iii. complete previous vertices, complete next vertices; iv. concurrent vertices; v. previous values vertices, concurrent values vertices; and vi. common value vertices i. The complete successors set is grown by adding the direct successors of the selected vertex. For each of these direct successors, the process is repeated, recursively populating the complete successors set of the selected vertex.

ii. The previous vertices set is a set of all direct predecessor vertices of the selected vertex, within the same sub-system as the selected vertex. The next vertices set is a set of all direct successor vertices of the selected vertex, within the same sub-system as the selected vertex.

iii. The complete previous vertices set is grown by adding the previous vertices set of the selected vertex, and for each vertex within that previous vertices set, adding its previous vertices set to the complete previous vertices set of the selected vertex. This process is repeated, recursively populating the complete previous vertices set. The complete next vertices set is grown by adding the next vertices set of the selected vertex, and for each vertex within that next vertices set, adding its next vertices set to the complete next vertices set of the selected vertex. This process is repeated, recursively populating the complete next vertices set. The difference between this and the complete successor set is the constraint to within a single sub-system.

iv. The concurrent vertices set is the set of all vertices in the same subsystem as the selected vertex, except for those vertices found in the complete previous vertices set of the selected vertex, the complete next vertices set of the selected vertex, and the selected vertex itself.

v. The previous values vertices set is generated by adding all the define or receive vertices within the complete previous vertices set of the selected vertex, and then for each of the vertices thus added, any vertices within its respective complete previous vertices set that are found in the previous values vertices set are removed from the previous values vertices set.

The concurrent values vertices set comprises all define or receive vertices of the concurrent vertices set of the selected vertex that are not members of the complete successors set of the selected vertex.

vi. The common values vertices set is defined as not empty only if the concurrent values vertices set of the selected vertex is empty. If so, then it is a set of all send vertices whose previous values vertices sets are equal to the previous values vertices set of the selected vertex and whose concurrent values vertices sets are empty.

For purposes of illustration, FIG 3b shows the previous vertices set, complete previous vertices set, concurrent vertices set, next vertices set, complete next vertices set^' and complete successors set for a selected vertex ^λX' in a notional graph.

FIG. 3a also illustrates several of the above sets for the exemplar graph: Thin-dotted areas comprise some vertices that are mutually their concurrent vertices.

Thick-dotted areas comprise some vertices that are mutually their common value vertices. Thick-dotted arrows go from some vertices to their concurrent values vertices .

Thick-dashed arrows go from some vertices to their previous values vertices.

For clarity, not all sets are shown in FIG. 3a.

To further illustrate the step of assignment, a selection of the above definitions are populated for the highlighted vertices of FIG. 3a as follows:

For vertex send B_l : the previous vertices are: send A; the complete previous vertices are: def_l, send A; the previous values vertices are: def 1.

For vertex send B_5 : the previous vertices are: recv B_l, recv B_2; the next vertices are: recv F; the complete previous vertices are: recv B_l, recv B_2; the complete next vertices are: recv F, recv C_l, recv C_2, send C_3, send C_4, recv C_5, recv C_6; the concurrent vertices are: recv B_3, recv B_4; the previous values vertices are: recv B_l, recv B_2; the concurrent values vertices are: recv B_3, recv B 4. For vertex send C_4 : the complete successors are: recv C_4, send C_6, recv C_6, recv D_l, recv D_2; the previous vertices are: recv F; the next vertices are: none; the complete previous vertices are: recv F, recv B_3, recv B_4, send B_5, recv B_l, recv B_2; the complete next vertices are: none; the concurrent vertices are: recv C_l, recv C_2, send C_3, recv C_5, recv C_6; the previous values vertices are: recv F; the concurrent values vertices are: recv C_l, recv C_2, recv C_5; the common value vertices are: none. Note that vertex recv C_6 is not among concurrent values vertices of vertex send C_4.

For vertex send C_5 : the previous vertices are: recv C_3; the complete previous vertices are: recv C_3, recv E, send F, recv B_5, recv A; the previous values vertices are: recv C_3; the concurrent values vertices are: recv C_4; the common value vertices are: none.

For vertex send C_6: the previous vertices are: recv C_4, send C_5; the complete previous vertices are: recv C_4, send C_5, recv C_3, recv E, send F, recv B_5, recv A; the previous values vertices are: recv C_4; the common value vertices are: none. For vertex send D_l: the concurrent vertices are: send D__2; the previous values vertices are: def_5, def_6; the concurrent values vertices are: none; the common value vertices are: send D_2.

Having established sets for each vertex, then in an embodiment of the present invention, assignment of a value for each vertex comprises the following steps: i. If the selected vertex is a define vertex then a unique value is assigned to the selected vertex, ii. If the selected vertex is a send vertex then the values of all previous values vertices and all concurrent values vertices of the selected vertex are examined. If all these values are equal then this common value is also assigned to the selected vertex. Otherwise, a unique value is assigned to the selected vertex and this unique value is also assigned to all vertices in the common value vertices set of the selected vertex. iii. If the selected vertex is a receive vertex then the value that is assigned to the send vertex associated with the selected vertex is also assigned to the selected vertex.

In the above conditions a unique value' is simply a value that is not equal to any other ^Λunique value' .

This assignment process is applied to each vertex in each graph. Each value is assigned to the vertex permanently. The one exception is a neutral value' which can be assigned to a send or receive vertex temporarily, during determination of a vertex's permanent value. When the value of a send vertex is to be determined, while previous values vertices and concurrent values vertices are examined, a neutral value assigned to a receive vertex is considered to be equal to any other value. When the value of a receive vertex is to be determined, the neutral value is assigned to this receive vertex until the value of the associated send vertex is determined and, consequently, the value of this receive vertex is determined. This alleviates the issue of selecting in which order to assign values to vertices.

An embodiment of the assignment process is summarised in the flow diagram of FIG. 4:

If the selected vertex does not have a value; Determine if it is a define vertex. If so, assign it unique value. Otherwise, determine if it is a receive vertex. .If so, assign it a neutral value, until the value of the corresponding send vertex is established and then use that value. Otherwise, determine if it is a send vertex. If so, determine the values of the vertex's previous values vertices and concurrent values vertices. If these values are the same, assign this common value. Otherwise, assign a unique value. Then, assign this value to any common value vertices. The resulting output for the graph of FIG 3a is illustrated in FIG 5. The purely exemplary values chosen take the form of ^Λ=1234' in the FIG. 5.

The process is detailed below:

For the define vertices def_l through def_6, unique values are assigned.

For the send vertices send A, send B_l, send B_2, send B_3, send B_4, send E, send C_l and send C_2, the previous values vertex is a define vertex and there are not any concurrent values verti ces. Thus these send vertices inherit the value of their respective previous define vertex.

For the receive vertices recv A, recv B_l, recv B_2, recv B_3, recv B_4, recv E, recv C_l and recv C_2, the values of the send vertices associated with them are assigned to them.

For send vertex send B_5: the previous values vertices are: recv B_l, recv B_2; the concurrent values vertices are: recv B_3, recv B_4; and the values assigned to vertices recv B_l, recv B_2, recv B_3, recv B_4 are equal. Thus this common value is also assigned to vertex send B_5. Consequently, receive vertex recv B_5 is also assigned this value.

For send vertex send F: the previous values vertex is vertex recv B_5 and there are not any concurrent values vertices . Thus the value of vertex recv B_5 is assigned to vertex send F. Consequently, receive vertex recv F is also assigned this value.

For send vertex send C_4 : the previous values vertex is: recv F; the concurrent values vertices are: recv C_l, recv C_2, recv C__5; and the values assigned to vertices recv F, recv C_l, recv C_2 are not equal (regardless of the value of vertex recv C_5) . Thus a unique value is assigned to vertex send C_4. Consequently receive vertex recv C_4 is also assigned this value.

For send vertex send C_3 : the previous values vertex is: recv F; the concurrent values vertices are: recv C_l, recv C_2, recv C_5; and the values assigned to vertices recv F, recv C_l, recv C_2 are not equal (regardless of the value of vertex recv C_5) . Thus a unique value is assigned to vertex send C_3.

Consequently receive vertex recv C_3 is also assigned this value. Note that values assigned to vertices send C_3 and send C_4 are not equal.

For send vertex send C_5: the previous values vertex is: recv C_3; the concurrent values vertex is: recv C 4; and the values assigned to vertices recv C_3, recv C_4 are not equal. Thus a unique value is assigned to vertex send C_5. Consequently receive vertex recv C_5 is also assigned this value.

For send vertex send C_6: the previous values vertex is vertex recv C_4 and there are not any concurrent values vertices. Thus the value of vertex recv C_4 is assigned to vertex send C_6. Receive vertex recv C_6 is consequently also assigned this value.

For send vertex send D_l : the previous values vertices are: def_5, def_6 and there are not any concurrent values vertices and by definition the values assigned to vertices def_5, def_6 are not equal. Thus a unique value is assigned to vertex send D_l. This value is also assigned to all common value vertices of the vertex send D_l, namely vertex send D_2. Receive vertices recv D_l and recv D_2 are consequently assigned the values of their respective send vertices.

The vertices send C_4, recv C_4, send C_5 and recv C_5 in Figure 4 highlight the choice of appropriate values to assign to a vertex, since the vertex recv C_5.is among concurrent values vertices of the vertex send C_4 and, at the same time, the vertex recv C_4 is among concurrent values vertices of the vertex send C 5. In an embodiment of the present invention, those receive vertices existing outside the system under test, but whose corresponding send vertices are within the SUT, are the focus of interest as they will receive variable values from the SUT.

In the example shown in FIG. 5, the receive events of subsystem R are: recv A, recv B_5, recv E, recv C_3, recv

C_4 , recv D_l and recv D_2.

These vertices are grouped according to the equality of their assigned values:

(recv A, recv B_5) = 1;

(recv E) 2; (recv C_3) = 1134;

(recv C_4), = 2134; and

(recv D_l, recv D_2)= 56.

Recall that FIG. 5 is based on the graph of FIG 2a. The vertex groups resulting from the graph of FIG 2b are:

(recv C_3), (recv C_4) and (recv D_l, recv D_2) as before, and (recv A, recv B_5, recv E) , the difference being due to the absence of the def_2 vertex in FIG^" 2b, so that recv E has the same value as recv B 5.

The groups generated for a plurality of graphs are then collated such vertex groupings common across graphs are preserved as groups whilst vertices differing between corresponding groups are separated out.

Thus in the example graphs of 2a and 2b, the collated groups are (recv A, recv B_5), (recv E) , (recv C_3) , (recv C 4) and (recv D 1, recv D 2) . recv A and recv B_5 are in common groups across both graphs whilst recv E occurs in different groups in each graph.

recv D_l and recv D_2 are similarly in common groups across both graphs.

The resulting suite of receive vertex groups describe corresponding groups of receive events that in operation would receive variable values from the system under test.

As is known in the art, a graph or set of graphs is typically converted into at least a first test script.

In an embodiment of the present invention, 'store' and 'compare' operations are assigned within these test scripts to the receive events grouped above:

A test script can be considered as a tree, and thus has a root (start) node and leaf (stop) nodes. The other nodes of a test script represent MSC events.

In an embodiment of the present invention, a unique index is assigned for each of the previously obtained groups of receiving events that includes more than one element.

In the present example, these are (recv A, recv B_5) and (recv D_l, recv D_2) .

For each such selected group, the nodes in the test script corresponding to the events in the group are identified, and the node closest to the root in the test script is assigned a 'store (variable, index)' operation; the remaining node or nodes are assigned a 'compare (variable, index) ' operation.

The above process is illustrated in FIGs 6a and 6b, which illustrate two test scripts generated for the exemplary MSC of FIG. 1. These scripts differ in the assumed order of events recv C_4 and send C_5 (recall from FIG. 1. that the order of these two events is not predetermined by the specification) .

The indexed groups of receiving events are (recv A, recv B_5)_2 and (recv D_l, recv D_2)_2.

For the test script tree of FIG 6a, to leaf nodes exist, meaning there are two paths to consider.

For group (recv A, recv B_5)_l, recv A is closest to the root in both paths, and is assigned a 'store' operation, recv B_5 is assigned a compare operation.

For group (recv D_l, recv D_2)_2, In the first path 610, the node for recv D_l is closest to the root and so is assigned a 'store' operation whilst recv D_2 is assigned a 'compare' operation. In the second path the order is reversed and so recv D_2 is assigned a 'store' operation and recv D_l a 'compare' .

This process is carried out for each test script tree comprising any of the grouped events, and for each variable under consideration. The final result is a set of test scripts with '.store' and 'compare' operations assigned to some of their nodes.

These test scripts can be used by a test script execution engine such as that as described in EP 1 176 512 A2. Each 'store' operation is parsed into a statement storing the variable value within a test environment, using the associated group index as a reference. Each 'compare' operation is parsed into a statement comparing the variable value with a value previously stored in the test environment that has a reference the same as the associated group index.

A compare operation returns a fail if the received variable values and previously stored value are different. In this way variable value equality checks are executed.

It will be clear to a person skilled in the art that where a receive event may receive more than one variable's values, a variable index may also be stored and referenced.

It will be understood that the method as described above provides at least one or more of the following advantages: i. A facility to check variable value equalities is provided, ii. This facility is provided whilst avoiding the obvious solution of generating all possible event sequences, with its attendant factorial growth in the number of equalities to check.

Claims

1. A method of automatically generating test scripts from a system specification model, characterised by the steps of_f- selecting at least a first group of receiving events; assigning, within a test script, an operation to store a variable value and group index to that receiving event of said group which is located closest to the root of said test script; and assigning, within said test script, an operation to compare said variable value and group index to each of the remaining members of said group.

2. A method according to claim 1, wherein operations are assigned for each of a plurality of variables.

3. A method according to any one of claims 1 and 2, wherein a test script comprising assigned operations is parsed by a test script execution engine.

4. A method according to claim 3, wherein a 'store' operation is parsed into a statement storing the received variable value within a test environment, using the associated received index as a reference; and a 'compare' operation is parsed into a statement comparing the received variable value with a value previously stored in the test environment that has a reference the same as the received index.

5. A method according to claim 4 wherein a compare operation returns a fail if the received variable values and previously stored value are different.

6. A method according to any one of the preceding claims, wherein the selection of at least a first group of receiving events comprises the step of_f- removing from a specification based graph any vertex not essential to checking values of a variable under consideration.

7. A method according to claim 6 wherein a vertex is removed if it does not represent one of the following events: i. a defining of a variable value by an owning subsystem; ii. a sending of a variable value to a non-owning subsystem; and iii. a receiving of a variable value by a non-owning sub-system. Where the owning sub-system is that sub-system which initially defines the variable.

8. A method according to any one of the preceding claims, wherein the selection of at least a first group of receiving events comprises the step of; assigning a vertex value to each vertex of a specification based graph.

9. A method according to claim 8 wherein the assignation of a vertex value to a define vertex comprises the step off- assigning a unique value to said vertex.

10. A method according to any one of claims 8 and 9 wherein the assignation of a vertex value to a send vertex comprises the steps of; i. examining the values of all previous values vertices and all concurrent values vertices of said vertex; ii. if all said examined values are equal then this common value is also assigned to said vertex; and iii. otherwise, assigning a unique value to said vertex, this unique value also being assigned to all vertices in the common value vertices set of said vertex, where previous values vertices, concurrent values vertices and common value vertices are defined herein.

11. A method according to any one of claims 8 to 10 wherein the assignation of a vertex value to a receive vertex comprises the step off- assigning to said vertex the value that is assigned to the send vertex associated with said vertex.

12. A method according to any one of claims 8 to 11 wherein the assignation of a vertex value to a send or receive vertex comprises the step of; temporarily assigning a neutral value, considered to be equal to any other value, to a receive vertex during determination of a send or receive vertex' s permanent value.

13. A method according to any one of the preceding claims, wherein the selection of at least a first group of receiving events comprises the step of; grouping according to the equality of their assigned values those receive vertices existing outside a system under test, but whose corresponding send vertices are within the system under test.

14. A method according to claim 13, wherein groups generated for a plurality of graphs based on a specification are collated such that vertex groupings common across graphs are preserved as groups whilst vertices differing between corresponding groups are separated out.

15. A method according to any one of the preceding claims, wherein after a system specification model is converted into a graph, a graph reduction is performed upon the graph.

16. A method of automatically generating test scripts from a system specification model according to claim 1 and substantially as hereinbefore described with reference to the accompanying drawings .