WO2002048728A1

WO2002048728A1 - Functional testing based on a state machine description of the unit under test

Info

Publication number: WO2002048728A1
Application number: PCT/NL2001/000254
Authority: WO
Inventors: Eilko Nijboer
Original assignee: Telefonaktiebolaget L.M. Ericsson
Priority date: 2000-12-15
Filing date: 2001-03-28
Publication date: 2002-06-20
Also published as: AU2001244866A1

Abstract

Method to carry out a test of an operation of a device (41) by a computer (42) which runs a program (21, 31) which processes a script (22), in which a statemachine (fig. 1) is defined comprising indicaitons of states (11-16) of the device which may occur upon supplying a command and upon receiving an event indicaiton by the computer with the commands and events being transitions between the states, the script is made to describe the statemachine, a test environment definition (24) is made which comprises specificaitons for communication between the computer and the device, the program comprises statements to detect the occurrence of state transitions and to decide about the current state of the device dependent on a latest deiced current state and to proceed processing of the script form the recent decided current state.

Description

Ti tle : FUNCTIONAL TESTING BASED ON A STATE MACHINE DESCRIPTION OF THE UNIT UNDER TEST

The invention relates to a method to carry out a test of an operation of a device, which device is capable to generate an indication of an event which occurs upon receiving a command from a computer which is connected to the device, in which the computer runs a program to process a script which contains a definition of the way to proceed its processing depending on commands supplied to the device and event indications received from the device. A method of this type is known from practice, in particular with the device being an electronic device.

With the prior art method the script comprises statements which are generally of the if-then-else type which are in particular used to compare response values received from the device under test in response to a command from the computer with specific predetermined values or a range of values contained in the script. If the outcome of the comparison is a negative one, that is the response does not meet an expected or wanted one, the program terminates processing of the script and therewith terminates testing of the device. The operator must be aware of the possibility that such termination of the test may occur, so that it will be necessary that he is monitoring the proceeding of the test to decide about which appropriate action he must take upon such termination of the test. Since such test is carried out for a device which is in general under development, or possibly under service or maintenance, such termination of the test is very likely to occur. Therefore in many cases the presence of the operator is required throughout the whole duration of the test. This may be a very long period of time since upon every such termination of the test the operator needs to consult manuals, data sheets or the like to decide which action he must take, that is which command he must have the computer to supply to the device upon the occurrence of such termination of the test. In addition, it will take much time to make log notes of the test containing sufficient details for use in making a decision upon such termination and/or modifying the script and/or modifying the device ■ to obtain a proper test of the device with a positive result of its operation. The script of the prior art method comprises all kinds of statements to properly communicate commands and event indications between the computer and the device under test in accordance with communication protocols and data formats, which in general will be different for different types of device under test. In particular this is true when said communication must take place through the same medium and according to the same rules when used by the device during normal operation of the device while communicating with a further device. Such communication can be considered as one possible operation to be tested of the device under test. Examples of different test environments are testing a Bluetooth device, a GSM device, a DECT device or an immobile digital telephone device. Describing a test by using, amongst others, if-then-else statements and many statements relating to the communication between the computer and the device is very complex and time consuming. This requires well educated operators, that is in the field of the device under test, in the field of communications and in the field of writing and modifying of scripts for testing purposes. For many fields of technology, such as in the field of communication, it will be difficult to obtain and to maintain sufficient knowledge and skill against reasonable costs. Even then, the writing and modifying of such scripts will be time consuming and expensive.

Another drawback of the prior art method is that because of the complexity of the script it will be difficult to avoid and to recognise an error therein, in particular if the code of the script describing the test can still be processed by the program without the occurrence of an error. In other words, the script may contain a description of a test of an operation which was not intended. If the processing of such test would lead to a positive outcome of the test the operator may wrongly decide that the device operates well as far as the intended operation is concerned.

It is an object of the invention to solve the above drawbacks. Therefore the invention provides a method according to claim 1. With the method according to the invention the program can be made generic for processing different statemachines and different test environment definitions. The only thing an operator or developer of a device must take care of is to describe the statemachine which may comprise relatively few lines, which each contains a transition between two states and those two states. Such script is easy to write and to maintain. This will safe time and costs and will avoid the occurrence of a description or definition of an unintended test operation of the device. The invention also relates to a computer program and a computer program script according to claims 8 and 11 respectively. The invention will be described in further detail below with reference to the accompanied drawings, in which: fig. 1 shows a diagram of a statemachine representing an example operation of a device under test which can be used with the method according to the invention; fig. 2 shows a diagram of an example of implementation of the method according to the invention; fig. 3 shows a diagram representing the making of a test engine program for use with the method according to the invention; and fig. 4 shows a diagram of a test environment in which the method according to the invention is applied.

The diagram shown in fig. 1 is an example of a statemachine comprising states 11, 12, 13, 14, 15, 16 of a device under test (41 in fig. 4) which may occur upon the supply of a command from a computer (42 in fig. 4) or of an event indication from the device to the computer. In fig. 1 commands are indicated by double-line arrows and event indications are indicated by single-line arrows. Fig. 1 shows the following commands: INQ(10, 5, 3) as transition from state 11 or INIT to state 12 or WAIT, transition TERM between state 13 TIMEOUT and state 16 END, transition RESET between state 14 ILL and state 11 INIT, and transition TERM between state 15 OK and state 16 END. Fig. 1 shows the following event indications: TIME (10) from state 12 WAIT to state 13 TIMEOUT, RESP(10, 18, 2) from state 12 WAIT to state 15 OK, RESP(,,) from state 12 WAIT to state 14 ILL and RESP(10,,) which loops from state 12 WAIT to state 12 WAIT. The states 11-16 and transitions shown in fig. 1 and mentioned above are specified in eight lines of table 1, one line for each different transition. Table 2 shows the same lines as of table 1 but written in more readable form.

TABLE 1

State Transition State

1 INIT INQ(10,5,3) ^■ WAIT

2 WAIT RESP(10, 18, 2) OK 3 WAIT RESP(10, , ) WAIT 4 WAIT RESP( ,,) ILL 5 WAIT TIME (10) ^• TIMEOUT

6 OK TERM ^■ END

7 TIMEOUT TERM ; END

8 ILL RESET ^• INIT

TABLE 2

State Transition State

1 initialise :send inquiry (10, 5, 3) ;wait for response

2 wait for response : response (10,18,2) ; correct responsive 3 wait for response ^•.response (10, , ) ; illegal responsive 4 wait for response : response (, , ) ;wait for response 5 wait for response : timeout (10) ;not responsive

6 correct responsive : terminate ;end of process

7 not responsive : terminate ;end of process

8 illegal responsive : reset process ; initialise

The part of the tables 1, 2 containing the states and state transitions is an example of a script which is suitable to be used with the method according to the invention. Such script (contained in the right corner cell of the tables) can be written in plain text format and can be stored as such as script file on a computer. From the above and tables 1 and 2 the following will be understood. The test of the device and, in accordance therewith, the script starts from state INIT for initialising the test, for example by sending an inquiry command INQ with parameter values (10, 5, 3) 5 to the device. According to line 1 of the tables the device will then enter a state WAIT during which the computer will wait for a response from the device. This first command INQ and the states INIT and WAIT between which it occurs is shown by line 1 of the tables. Lines 2, 3 and 4 of the tables relate to the receiving of an event

10 indication or response from the device. Depending on parameter values returned together with the response to the computer it will decide about the current state of the device. If the response is within limits the state of the device is decided to be ok (line 2) and if not,- the computer may decide that the device is on error

15 (line 3) or the computer may wait for further response (line 4) . If no^' response is received upon timeout of a specified time the occurrence of such timeout is still considered to be an event indication (line 5) upon which the computer will take action. When the computer decides that the device is in an OK state it will

20 terminate the test and it may supply a terminate command TERM to terminate some or all operation of the device (line 6) . The same applies on the occurrence of a timeout (line 7) . If the computer decides that the. device, entered an illegal state ILL it will reset the process' of testing the device by returning to the initialisation

25 state INIT (line 8) to restart the processing of the script from there on, that is from line 1- of the table.^'

The diagram of fig. 2 shows a script engine 21, which is in fact a computer program- which can process a script contained in a script file. Fig. 2 shows several script files 22, each of which may

30 contain a different script defining a different statemachine or operation of the device under test. Fig. 2 also shows a batch file 23 containing .a definition, in particular a script in plain text format, about a predefined order in which the script files 22 must be processed by the script engine 21. Especially for communication

35. between the computer and the device under test the script engine 21 uses a specification file 24 which is specific for a test environment. For example, there may be a specification file 24 for testing Bluetooth devices, a different specification file 24 for testing GSM devices, a different specification file 24 for testing DECT devices, a different specification file 24 for testing immobile telephone devices, and so on.

When processing the script files 22 the script engine 21 may generate script logs (log files) 25 comprising supplied commands and event indications in relation to time and decided current states of the device under test.

In addition, the script engine 21 may generate a batch file log reflecting the processing of batch file 23. The script engine 21 shown in fig. 2 can be an interpreter, which needs an additional program to interpret code of the files 22- 24 at the very moment of processing thereof to generate appropriate code which can be executed by the computer.

Instead of the script engine 21 shown in fig. 2 a test engine program 31 shown in fig. 3 can be used. This is a program which is obtained by writing code in some higher level computer language, by which one or several source files 32 containing such code can be obtained. Then the source files 32 are converted or compiled by a compiler 33 which is specific for said higher level programming language. The compiler 33 generates the test engine program 31 from the source code such that the test engine program can be run on the computer by using only standard software, such as an operating system, in addition. The advantage of test engine program 31 above script engine 21 is that it may run faster on the computer and it can be protected better against unwanted or unskilled modification thereof.

Fig. 4 illustrates an example of a test environment in which the method according to the invention is used. A device under test 41 is connected to a computer 42. As schematically shown the test engine program 31 of fig. 3 and the script files 22, the batch file(s) 23 and the specification file 24 are supplied to the computer 42. The computer 42 runs test engine program 31 which processes the files 22, 23 and 24. The test engine program 31 may generate log files, such as script logs 25 and batch file log 26 of fig. 2, and may save same on a storage disc or the like (not shown) . Communication between the device under test 41 and the computer 42 may take place over any suitable medium, which may depend on the operation of the device 41 which is to be tested. For example, if the device 41 is a Bluetooth device said communication can be over wires, which are temporarily or not, or by RF communication which is used during normal operation by the device, for example for testing hardware components of the device or the communication itself respectively.

From the above it will be understood that for testing a device in some testing environment the only thing one has to do is to write some representation of a statemachine, for example in the form of table 1. The writer of the script only needs to have specific knowledge of the operation of the device he wants to test. Depending on the outcome of the test he may easily modify that specific kind of operation of the device and/or the script taking the contents of the log files 25, 26 into account.

Since the processing of the script will come to an end only upon the occurrence of some specific transitions between states there is less need for the operator to constantly attend the progress of the test. This gives the opportunity to carry out several tests at the same time, even for the same device under test 41 with little chance to be overloaded or getting lost in searching in different kinds of manuals and data sheets for ways to proceed as was the case with the prior art.

It is observed that the method according to the invention is not limited in its use for testing electronic devices, such as the RF devices mentioned as examples herein before. Instead any device, construction, apparatus or machine comprising appropriate sensors and transducers for responding to commands from the computer and for transmitting event indications to the computer can be tested by the method and by using a program and script according to the invention.

Claims

C L A I M S

1. Method to carry out a test of an operation of a device (41), which device is capable to generate an indication of an event which occurs upon receiving a command from a computer (42) which is connected to the device, in which the computer runs a program to process a script which contains a definition of the way to proceed its processing depending on commands supplied to the device and event indications received from the device, characterized in that a statemachine (fig. 1) is defined comprising indications of states (11-16) of the device which may occur upon supplying a command and upon receiving an event indication by the computer with the commands and events being transitions between the states, the script (22) is made to describe the statemachine, a test environment definition (24) is made which comprises specifications for communication between the computer and the device, the program (21, 31) comprises statements to detect the occurrence of state transitions and to decide about the current state of the device dependent on a latest decided current state and to proceed processing of the script from the recent decided current state.

2. Method according to claim 1, characterized in that a log file (25) is made comprising supplied commands and event indications in relation to time and decided current states.

3. Method according to claim 1, characterized in that the script is processed until the decided current state is a terminated state (16).

4. Method according to claim 1, characterized in that one or more transitions are timeout events occurring upon not receiving an event indication from the device by the computer within a specified time.

5. Method according to claim 1, characterized in that the program is an interpreter.

6. Method according to claim 1, characterized in that the program is an executable program (31) of compiled source code (32) .

7. Method according to claim 1, characterized if that it comprises processing of a batch of several scripts v22) which describe different statemachines for different kinds of operation of the device (41) .

8. Computer program which, when run on a computer (42) and dependent on a script which is processed by the program, supplies commands to a device (41) which is connected to the computer for testing the device, with the device being capable to supply an event indication of an event which occur upon receiving a command from the computer, and with the script containing a definition of the way to proceed its processing depending on commands supplied to the device and event indications received from the device, characterized in that when the script describes a statemachine (fig. 1) comprising indications of states (11-16) of the device, the program (21, 31) comprises statements to decide about the current state of the device dependent on supplied commands and event indications and a previous decided current state and to proceed processing of the script from the recent decided current state.

9. Computer program according to claim 8, characterized in that it is an interpreter.

10. Computer program according to claim 8, characterized in that it comprises compiled source code (32) .

11. Computer program script which is suitable to be processed by a program run on a computer (42) which is connected to a device (41) for testing the device, with the device being capable to supply an event indication of an event which occurs upon receiving a command from the computer, and with the script containing a definition of the way to proceed with its processing depending on commands supplied to the device and event indications received from the device, characterized in that the script (22) comprises indications of possible states of the device and transitions between states (11- 16) , in which a transition is a command or an event indication.