GB2460407A

GB2460407A - Using coverage data to choose software regression tests

Info

Publication number: GB2460407A
Application number: GB0809574A
Authority: GB
Inventors: Coffey Michael; Aley James
Original assignee: Nokia Oyj; Symbian Software Ltd
Current assignee: Nokia Oyj
Priority date: 2008-05-27
Filing date: 2008-05-27
Publication date: 2009-12-02
Also published as: GB0809574D0

Abstract

A first version of a software application is tested using a plurality of tests. For each test, the parts of the software application exercised by the test are recorded. When a new version of the application is created, the recorded data is used to identify which of the tests exercise the parts of the application which have been modified. The identified tests are then used to test the new version. The functions or the lines of code of the application exercised by the test may be recorded. Alternatively, the conditional branches taken by the application may be recorded.

Description

Method and System for Software Regression Testing

Technical Field

The present invention relates to a method and system for regression testing of software, and in particular to such a system wherein testing is only performed of those software modules which have changed.

Background to the Invention and Prior Art

Software systems are endemic in modern life in the western world. Most modern consumer appliances include, to some degree, a controlling software program of some description. As well as controlling computers, software can also be found in audio visual equipment such as DVD players, CD players, MP3 players, televisions, set top boxes, and the like, as well as other appliances, such as fridges, cookers, and washing machines, and vehicles such as cars, aircraft, etc etc. The list of devices and machines which include software is almost endless.

A software system typically comprises numerous "modules" being blocks of code which perform a particular overall task. Each module is often formed of several standard or bespoke functions, being smaller blocks of code each of which performs a particular program function. Functions themselves comprise individual lines of code, containing programming language commands, variable names, boolean operators, and the like.

Once a software system has been developed, it is absolutely essential that the software system is rigorously tested. It is almost impossible that a software system will perform correctly as soon as it is created. Software development is typically an iterative process, involving testing and then making appropriate changes, based on the test results. In this respect, an actual programming language compiler performs a first test on a set of code, to determine whether in fact an executable can be compiled based thereon. However, this is a very basic test. It is often possible to write a software program which will compile, which does not perform as intended. Therefore, software testing of an initial product is of prime importance.

I

In order to test software systems, automated test systems are available, which take as their input one of a number of test conditions, known as a test harness. Several test harnesses are shown collectively as reference numeral 26 in Figure 1. The tester module 22, which would typically be a computer system of some sort, takes the software system 10 which has been developed, and runs the software system 10 inputting each test harness 26 into the software system in turn. By such means, the software system 10 can be tested.

In order to determine the efficacy of the tester module 22, and the test harnesses 26, it is known in the art that a code coverage analyser 24 can be provided, which keeps a track of which software modules, functions, and even which lines of software code are exercised by each test. An example of a code coverage analyser 24 known in the art is the "Bullseye Coverage" analyser, available from Bullseye Testing Technology, of Redmond, Washington, United States. The Bullseye Coverage analyser provides a graphical indication as to which modules, functions, and lines of code a particular test harness has exercised. Therefore, a software developer can be sure that, when a test harness is applied to a tester, it is testing the appropriate parts of the software which require testing.

Once a software system has first been developed and tested, it can then be released.

However, it is typical that further developments and refinements will be made to the software system during its life. It is usual to give a software system a version number i.e. version 1.0. When a modification is made to the software, the version number can be updated i.e. version 1.1. Where significant revisions occur, then the cardinal version number can be updated i.e. version 2.0.

As software is updated during its lifetime, it is necessary to then perform "regression testing". Regression testing is any type of software testing which seeks to uncover regression bugs. Regression bugs typically occur as an unintended consequence of program changes, and relate to the case where software functionality that previously worked as desired stops working or no longer works in the same way that was previously planned. As software is developed and updated, regression bugs can be quite common. Hence, once software has been updated, it is necessary to regression test the updated software, to detect any regression bugs which may occur, such that a fix can subsequently be performed.

It is known that regression testing can be performed using automated test tools.

Commonly, a test suite contains software tools that allow the testing environment to execute all regression test cases automatically. However, selecting which actual tests to run to exercise the parts of the software which have been updated has typically been performed using static analysis of the code, by the test team. That is, the code is statically analysed (i.e. the actual lines of code are simply examined, rather than being run), and an estimate is made as to which test harnesses when run by the test suite would exercise the updated code. Using such static analysis, to err on the side of caution, commonly more tests are run than are strictly required, in order to be certain that the updated part of the software is exercised. Testing therefore takes a long time, and time is wasted running irrelevant tests that do not exercise the change.

Summary of the Invention

In order to address the above problem embodiments of the invention provide that for each test which is made against the software system when the software system is first produced, the results of the test are collated into a database which maps each particular test against the functions (or other discrete and identifiable bit of code) which it exercises, with the results in the database being based upon the dynamic coverage data provided as the results of the test program. Once this data has been collated, when the software system is then updated in the future, it is possible to use the database to determine which tests exercise a particular function which may have been changed.

Therefore, regression testing can be performed more easily, as it becomes possible to know which specific test can be used to regression test just the particular piece of code which has been updated. The advantages of such an arrangement compared to the prior art are that the database which is collated is based upon dynamic coverage data obtained from actually running the tests against the code, and monitoring which functions are called, using the coverage analysis program.

In view of the above the present invention, from a first aspect, provides a method of determining one or more tests for use in regression testing of a software system, comprising the steps: testing a first version of the software system against a plurality of tests; and determining which part or parts of the software system is/are dynamically exercised by each test; the method being characterised by further comprising: storing information specifying which part of the software system is dynamically exercised by which test. This is a necessary precursor step to gather and store the information as to which tests exercise which part of the software system. This information can then be used subsequently for simplifying a regression test process when the software system is updated.

In a preferred embodiment, the determining step determines which functions of the software system are exercised by each test, and the storing step stores said information indexed by function. Thus, at the level of an individual function, it is possible to record which test exercises each individual function.

In another embodiment the determining step determines which lines of code or conditions of the software system are exercised by each test, and the storing step stores said information indexed by line or condition. This allows even finer granularity in the recording of information relating to which tests exercise which parts of the software system.

From a second aspect there is also provided a method of regression testing an updated version of a software system to test parts of the system which have been altered since a previous version of the system, the method comprising the steps: selecting, in dependence on stored information that relates parts of the software system to one or more tests which are known to dynamically exercise those parts, the tests belonging to a set of a plurality of available tests, one or more tests from the plurality of available tests, the selected tests being those tests which dynamically exercise the altered parts of the software system; and testing the software system using the selected test or tests; wherein regression testing may be performed using tests which are known to dynamically exercise the altered parts of the software system. With such an arrangement, regression testing can be performed in a very straightforward manner, and only those tests which actually exercise the change need be performed. As such, testing can be completed more quickly, and with greater reliability than has heretofore been the case.

In the preferred embodiment stored information is indexed by function, such that it relates which tests of the plurality of available tests exercise each function, and wherein the selecting step selects the one or more tests for use in testing in dependence on the functions which have been altered in the updated version of the software system. Hence, updates to individual functions can be tested.

In another embodiment the stored information is indexed by line of code or by condition, such that it relates which tests of the plurality of available tests exercise each line of code or condition, and wherein the selecting step selects the one or more tests for use in testing in dependence on the line(s) of code or condition(s) which has/have been altered in the updated version of the software system. This allows for finer granularity of testing, in that small changes to individual lines of code or conditions in a line of code can be individually tested, using the smallest number of tests.

From another aspect there is provided a system for determining one or more tests for use in regression testing of a software system, comprising: a test module that, in use, tests a first version of the software system against a plurality of tests; and a code coverage analyser that, in use, determines which part or parts of the software system is/are dynamically exercised by each test; the system being characterised by further comprising: an information store that stores information specifying which part of the software system is dynamically exercised by which test.

Within this aspect the same advantages, and same further features and advantages can be obtained as in the first aspect.

From a yet further aspect there is also provided a system for regression testing an updated version of a software system to test parts of the system which have been altered since a previous version of the system, the system comprising: a test selector that selects, in dependence on stored information that relates parts of the software system to one or more tests which are known to dynamically exercise those parts, the tests belonging to a set of a plurality of available tests, one or more tests from the plurality of available tests, the selected tests being those tests which dynamically exercise the altered parts of the software system; and a test module that, in use, tests the software system using the selected test or tests; wherein regression testing may be performed using tests which are known to dynamically exercise the altered parts of the software system.

Within this aspect the same advantages, and same further features and advantages can be obtained as in the second aspect.

From another aspect the invention also provides a computer program or suite of computer programs, arranged such that when executed by a computer system itlthey cause the computer system to perform the method of any of the first and second aspects.

Additionally, there is also provided a computer readable storage medium storing a computer program or at least one of the suite of computer programs according to the preceding aspect.

With the above, therefore, as explained the invention provides a method of software regression testing comprising: keeping a record of which tests of a plurality of tests operate on which parts of a software system when the system is operated in accordance with the plurality of tests; and, when the software is updated, regression testing the updated parts of the software using tests which were recorded to operate on the part of the software system which was updated; wherein regression testing is performed without unnecessary tests which do not operate on the updated part of the software system. This allows for improved regression testing having the advantages described previously.

Brief Description of the Drawings

Further features and advantages of the present invention will become apparent from the following description of embodiments thereof, presented by way of example only, and by reference to the accompanying drawings, wherein like reference numerals refer to like parts, and wherein: -Figure 1 is a block diagram of a test collation system according to a first embodiment of the present invention; Figure 2 is a flow diagram of the operation of the test collation system of Figure 1; Figure 3 is a block diagram of a regression test system according to the embodiment of the invention; and Figure 4 is a block diagram of the operation of the regression test system of Figure 3.

Description of the Embodiments

Embodiments of the invention will now be described with respect to the figures.

More particularly, Figure 1 shows a test collation system used in embodiments of the invention, for collating a database of test harnesses against individual parts of a software system under test. More particularly, a software system under test, comprising a plurality of software modules, each having a plurality of functions, each having themselves a plurality of lines of code, is input into a test system 22. The exact configuration of the test system 22 is not important, and it can be a manual or an automated test system, as is known in the art. Also provided is a plurality of test harnesses 26, being a set of test inputs which are used by the test system 22 to test the operation of the software system 10. Also provided is a code coverage analyser 24, which monitors the test system 22 in operation, to determine which parts of the software system have been exercised by a particular test harness, when run by the test system 22.

The code coverage analyser 24 could be, for example, the commercially available coverage analyser "Bullseye Coverage", as mentioned previously. Thus far, the elements of the test suite are conventional.

Further provided according to the first embodiment is a data storage medium 28, such as a hard disk, or the like, on which is stored a database 30, shown by way of example in the form of a table. The database 30 contains a list of the software functions in the software system 10 as an index to the table, and then lists for each software function the identifications of each test harness which are known, from the code coverage analyser 24, to exercise the particular software function when run by the test suite 20. More particularly, the information in the database 30 can be obtained from the code coverage analyser 24, which keeps track, when the test system 22 runs a particular test harness 26 against the software system 10, of which functions are exercised by the particular test harness. By storing such results for each test harness, then the database 30 can be compiled.

Figure 2 illustrates the operation of the test collation system of Figure 1, to compile the database 30.

More particularly, at step 2.2 the test module 22 begins to run each test harness against the software system 10 in turn. The code coverage analyser 24 is configured to monitor the results, to determine which code in the software system is exercised by each test harness.

At step 2.4 a test harness is applied to the software system, and at step 2.6 the code coverage analyser determines the code coverage of the software system which is exercised by the test harness. This information is then stored at step 2.8 in the database.

That is, in this embodiment the code coverage analyser 24 determines which functions have been dynamically exercised by a particular test harness, and then the identifications of these functions which have been dynamically exercised by a particular test are stored in the database 30, indexed by software function. At step 2.10 the next test harness is selected, and then processing proceeds to step 2.2. The procedure iterates from step 2.2 to step 2.10 for each test harness, until all test harnesses have been applied to the software system 10 by the tester module 22. At that point, the database 30 will be complete, and will contain, for each software function, list of test harnesses which are known to dynamically exercise that particular function.

Having obtained the database 30, this database is then stored until some time in the future, when the software system is updated. When the software system is updated, as explained previously, regression testing must be performed to ensure that the software continues to operate as originally intended, allowing for the update.

In order to ensure that no more testing than is absolutely necessary is performed, the data in the database 30 is used to select the particular tests to be run which are known to exercise the function which has been changed. For example, with reference to Figure 3, here the software system 10' is the updated software system, from which it will be seen that the function fn2() has been updated, to provide function fn2'O. This updated software system 10' is then input to tester module 22. The tester module 22 notes which function has changed in the software system 10', and accesses the database 30 so as to determine which test harnesses were known previously to exercise the function which has changed. The database 30 returns a list of test harnesses which are known to exercise the function which has changed, and the tester module 22 can then apply each test harness to the software system 10' in turn, logging the results as test results 32. In this way, the tester module 22 applies only those tests which, during the original testing, were noted to dynamically exercise the function which has been updated in the software system. Therefore, only those tests which dynamically exercise the updated function or module in the software system need be run. No other, unnecessary, tests need be performed.

Figure 4 is a flow diagram illustrating the operation of the system of Figure 3. Firstly, at step 4.2, the test module 22 receives the updated software system 10', and at step 4.4 it determines which functions have been changed. This can be performed automatically either by comparing the software system 10' with the original software system 10, or an operator input can be put into the tester module 22, to indicate to the tester module 22 which modules, functions, or the like have been updated.

Assuming that a function has been changed, at step 4.6 a processing ioop is started to process each changed function, such that it is tested. At step 4.8, for a particular function which has been changed which is to be tested, the tester module 22 looks up in the database 30 the tests which dynamically exercise the particular function which has been updated. Thus, for example, with reference to Figure 3, where function fn2() has been updated, it can be seen from the database 30 that test 1, test 2, test 3, etc. were known to dynamically exercise the function. Having determined this information at step 4.8, at step 4.10 the tester module 22 runs the tests which were determined to dynamically exercise the function against the software system 10', and logs the results as test results 32. Then, the next changed function is selected at step 4.12, and the processing ioop proceeds again, from step 4.6.

Once all of the changed functions have been tested, using the tests which are known to dynamically exercise the functions, but no other tests, from step 4.14 the tester module 22, or an operator, can attempt to identify regression bugs from the test results. This last step is outside the scope of the present invention.

In a particularly preferred embodiment any particular test would only be run once, to avoid duplication. In this case it is preferable to compile a list of all the tests that are to be run, and only once all required tests have been identified are the tests run. Thus step 4.10 in Figure 4 would not take place until step 4.12 had been completed for all changed functions.

With the embodiments described above in relation to Figure 4, firstly it can be determined which tests dynamically exercise a particular piece of software, and a record of a list of such tests can be kept. Subsequently, when the software is updated, the record can be accessed, to determine those tests which are known to dynamically exercise the part of the software which has changed, and then only those tests need be run against the software. Thus, regression testing can be performed against an updated software system in a more timely and cost effective manner, using only those tests which are known to dynamically exercise the software, and meaning that no unnecessary tests need be run. Because the tests are determined on dynamic coverage date, rather than a static analysis, one can be certain that the test definitely exercises the particular part of the software, and hence greater reliability and certainty can be obtained in the testing process. Overall, the embodiment of the invention allows for software updates to be tested more quickly and reliably than has heretofore been the case.

Various modifications may be made to the above described embodiment to provide further embodiments. For example, in the first embodiment described above the database 30 is collated using function identifiers as an index, and listing the test harnesses against each software function. However, it should be appreciated that the database 30 can be collated at a finer, or larger, granularity than this. For example, instead of listing the tests by function, the tests may be listed by software module.

Typically, a software module would contain several functions, and hence this would represent a larger granularity, and typically more tests would be listed against each module. Likewise, the database may be collated at a finer granularity, for example by each line of code, or each condition in a function. Particular lines of code in a function can be identified by their line number, and the database 30 may be collated by function and line number as an index, listing each test which exercises each individual line of code. With such an arrangement, for each line of code or condition in a function the list of tests will be a subset of the list of tests which dynamically exercise the function in which the line or condition resides. Therefore, where only a few lines of code are changed in a later version of the software system, then by providing a database with greater resolution as to identifying which tests exercise which lines of code, then even fewer tests than was necessary when recording by function are required. This is because, for example, whereas a particular software function may be exercised by, for example, ten test harnesses, an individual line of code in that function may only be exercised by, for example, a subset of those ten such as, for example, three. Thus, where minor changes to the software occur, for example to an individual line of code, then very quick testing can be performed using only those tests which are known to dynamically exercise that particular line of code.

Further modifications in the form of additions, deletions, or other changes will be apparent to the intended reader, being a person skilled in the art, to provide further embodiments, any and all of which are intended to fall within the appended claims.

Claims

Claims 1. A method of determining one or more tests for use in regression testing of a software system, comprising the steps: testing a first version of the software system against a plurality of tests; and determining which part or parts of the software system is/are dynamically exercised by each test; the method being characterised by further comprising: storing information specifying which part of the software system is dynamically exercised by which test.
2. A method according to claim 1, wherein the determining step determines which functions of the software system are exercised by each test, and the storing step stores said information indexed by function.
3. A method according to claim 1 or 2, wherein the determining step determines which lines of code or conditions of the software system are exercised by each test, and the storing step stores said information indexed by line or condition.
4. A method of regression testing an updated version of a software system to test parts of the system which have been altered since a previous version of the system, the method comprising the steps: selecting, in dependence on stored information that relates parts of the software system to one or more tests which are known to dynamically exercise those parts, the tests belonging to a set of a plurality of available tests, one or more tests from the plurality of available tests, the selected tests being those tests which dynamically exercise the altered parts of the software system; and testing the software system using the selected test or tests; wherein regression testing may be performed using tests which are known to dynamically exercise the altered parts of the software system.
5. A method according to claim 4, wherein the stored information is indexed by function, such that it relates which tests of the plurality of available tests exercise each function, and wherein the selecting step selects the one or more tests for use in testing in dependence on the functions which have been altered in the updated version of the software system.
6. A method according to claim 4 or 5, wherein the stored information is indexed by line of code or by condition, such that it relates which tests of the plurality of available tests exercise each line of code or condition, and wherein the selecting step selects the one or more tests for use in testing in dependence on the line(s) of code or condition(s) which has/have been altered in the updated version of the software system.
7. A system for determining one or more tests for use in regression testing of a software system, comprising: a test module that, in use, tests a first version of the software system against a plurality of tests; and a code coverage analyser that, in use, determines which part or parts of the software system is/are dynamically exercised by each test; the system being characterised by further comprising: an information store that stores information specifying which part of the software system is dynamically exercised by which test.
8. A system according to claim 7, wherein the code coverage analyser determines which functions of the software system are exercised by each test, and the information store stores said information indexed by function.
9. A system according to claim 6 or 7, wherein the code coverage analyser determines which lines of code or conditions of the software system are exercised by each test, and the information store stores said information indexed by line or condition.
10. A system for regression testing an updated version of a software system to test parts of the system which have been altered since a previous version of the system, the system comprising: a test selector that selects, in dependence on stored information that relates parts of the software system to one or more tests which are known to dynamically exercise those parts, the tests belonging to a set of a plurality of available tests, one or more tests from the plurality of available tests, the selected tests being those tests which dynamically exercise the altered parts of the software system; and a test module that, in use, tests the software system using the selected test or tests; wherein regression testing may be performed using tests which are known to dynamically exercise the altered parts of the software system.
11. A system according to claim 10, wherein the stored information is indexed by function, such that it relates which tests of the plurality of available tests exercise each function, and wherein the test selector selects the one or more tests for use in testing in dependence on the functions which have been altered in the updated version of the software system.
12. A system according to claim 10 or 11, wherein the stored information is indexed by line of code or by condition, such that it relates which tests of the plurality of available tests exercise each line of code or condition, and wherein the test selector selects the one or more tests for use in testing in dependence on the line(s) of code or condition(s) which has/have been altered in the updated version of the software system.
13. A computer program or suite of computer programs, arranged such that when executed by a computer system it/they cause the computer system to perform the method of any of claims ito 6.
14. A computer readable storage medium storing a computer program or at least one of the suite of computer programs according to claim 13.
15. A method of software regression testing comprising: keeping a record of which tests of a plurality of tests operate on which parts of a software system when the system is operated in accordance with the plurality of tests; and, when the software is updated, regression testing the updated parts of the software using tests which were recorded to operate on the part of the software system which was updated; wherein regression testing is performed without unnecessary tests which do not operate on the updated part of the software system.