Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/36—Preventing errors by testing or debugging software
  • G06F11/3668—Software testing
  • G06F11/3672—Test management
  • G06F11/3688—Test management for test execution, e.g. scheduling of test suites

Definitions

• the invention relates to a method for evaluating tests of a computer program by analyzing mutations comprising executing mutated programs by inserting mutations and identifying mutated programs providing, with a predetermined test, an identical result. to a predetermined result.
• Mutational analysis is a technique for evaluating tests that requires significant computing resources.
• errors or mutations are inserted into an initially unmutated program. Then we check if the tests detect the inserted mutations.
• a mutated program provided, with a predetermined test a result identical to a result provided by said non-mutated program, executed with said test, the mutation is identified as undetected.
• the result provided by the mutated program, with a predetermined test is different from the result provided by said non-mutated program, executed with said test, the mutation is identified as detected.
• the corresponding test is stopped and the next mutation is analyzed. Tests that detect mutations are retained for an improved test suite.
• the interest of the mutation analysis is to evaluate the quality of the test suite and possibly to optimize the test suite.
• the duration of the mutation analysis can be very important.
• the number of mutations is reduced, for example, by choosing a certain kind of mutation or by randomly choosing a number of mutations. In both cases, the quality of the test evaluation is deteriorated.
• Document DE19959157 describes a technique for improving a test suite comprising a first step for quickly distinguishing between subtle and non-subtle mutations and a second step for evaluating the tests for non-subtle mutations.
• the object of the invention is to remedy these drawbacks and, in particular, to minimize the duration of a method of evaluating tests of a computer program by analysis of mutations, in particular by activation of a programmable circuit comprising logical sets.
• the method comprises selecting, from among a plurality of tests, a test suite relevant to each mutation, determining at least one ranking by applying at least one simplicity criterion to the selected tests, - executing the programs mutated by order of ranking, each mutated program being executed with the tests of the associated test suite.
• the method comprises the programming of a programmable circuit by said program comprising the mutations and the activation of the programmable circuit, the programmable circuit comprising logical sets associated respectively with the mutations, the inputs of tests on which a control system applies signals representative of test data and test outputs transmitting, to the control system, signals representative of test results, at least one activation input on which the control system selectively applies activation or deactivation signals of the logic units, the programmable circuit being programmed in such a way that its operation corresponds to said non-mutated program when said logic units are deactivated and to one of the mutated programs when the corresponding logic unit is activated , the method comprising the test of non-mutated circuit and mutated circuits, a mutated program being identified when the corresponding mutated circuit provides, with a predetermined test, a result identical to the result provided by the non-mutated circuit, executed with said test.
• the invention also aims at a system for performing a method of evaluating tests of a computer program by mutation analysis, the method comprising the execution of mutant programs by insertion of mutations and identification of the mutated programs providing, with a predetermined test, a result identical to a predetermined result, the system comprising means for selecting, from among a plurality of tests, a series of tests relevant for each mutation the determination at least one classification by applying at least one simplicity criterion to the selected tests, the execution of the mutated programs in order of ranking, each mutated program being executed with the tests of the associated test suite.
• the invention also aims at a computer-readable medium comprising instructions for controlling a computer for executing a method of evaluating tests of a computer program by analysis of mutations involving the execution of mutant programs by insertion. of mutations and the identification of the mutated programs providing, with a predetermined test, a result identical to a predetermined result, the medium comprising control instructions for performing the selection, among a plurality of tests, of a series of tests relevant to each mutation, the determination of at least one ranking by application of at least one criterion of simplicity to the selected tests, the execution of the mutated programs in order of ranking, each program, mutated being executed with the tests of the test suite associated.
• Figures 1 to 4 show different steps of a method of evaluating tests of a program by analysis of mutations by activation of a programmable circuit.
• FIG. 5 illustrates a particular embodiment of an evaluation method according to the invention.
• FIG. 6 illustrates a particular embodiment of a step of determining a ranking of tests of a method according to the invention.
• Figure 7 illustrates a particular embodiment of two steps of determining additional rankings.
• FIG. 8 illustrates a particular embodiment of a step for selecting a reduced classification of mutations of a method according to the invention.
• FIG. 9 illustrates a particular embodiment of a step of eliminating mutations of a method according to the invention.
• the analysis of mutations may include, as shown in Figure 1, the programming of a programmable circuit 1 by a program P with the mutations Mj (M1, M2, M3, etc.). Below, the mutations are referenced by the index j.
• the program P models, by example, an integrated circuit.
• the programmable circuit 1 is programmed via a control system 2.
• the mutation analysis which is then carried out comprises the activation of the programmable circuit 1.
• the programmable circuit 1 comprises logical sets Gj (G.1, G2, G3, etc.) associated respectively with the mutations Mj.
• the programmable circuit 1 comprises at least one activation input on which the control system 2 selectively applies activation signals Aj or deactivation D logical sets Gj.
• the control system 2 applies, on test inputs, Etk signals (Et1, Et2, Et3, etc.) representative of test data Tk (T1, T2, T3, etc.). Below, the tests are referenced by the index k.
• Signals STkD (FIG. 2) and STkj (FIG. 3) representative of test results Tk are transmitted by test outputs to the control system 2.
• the programmable circuit 1 is programmed so that its operation corresponds to the program P that is not mutated when the logic units Gj are deactivated by a deactivation signal D.
• the outputs of the programmable circuit 1 thus transmit STkD signals representative of the results of Tk tests applied to the non-mutated circuit.
• the operation of the programmable circuit 1 corresponds to the program Pj mutated by the mutation Mj when the corresponding logical set Gj is activated by an activation signal Aj.
• the outputs of the programmable circuit 1 thus transmit STkj signals representative of the results of tests Tk applied to the mutated circuit corresponding to the mutated program Pj. 5 001790
• the control system 2 comprises a comparator for comparing the signals STkD and STkj.
• the comparator provides a signal IDkj making it possible to identify the test Tk and the corresponding program Pj.
• a mutated program Pj is then identified when the corresponding mutated circuit provides, with a predetermined test Tk, a result STkj identical to the result STkD provided by the non-mutated circuit, executed with said test Tk.
• the mutated programs Pj are generated, in a step F1, from the mutations Mj and the non-mutated program P, by inserting the mutations Mj in the program P. All the mutations Mj are preferably inserted in the same program and can be selectively activated or deactivated, in a known manner, by activation parameters respectively corresponding to mutations Mj.
• results RTkD and RTkj are stored and compared later.
• the results RTkD and RTkj can be determined simultaneously and compared directly, without requiring storage. Moreover, as soon as a mutation is detected, with a predetermined test, the execution of the program Pj mutated with said test is stopped and the next mutation is analyzed.
• a step F2 the non-mutated program P is executed with the tests Tk of the plurality of tests Tk and provides the results RTkD which are stored in the step F3.
• a selection step F4 a test suite Uj relevant is selected from among a plurality of tests Tk for each mutation Mj.
• the selection F4 of the tests relevant for a predetermined mutation Mj may, for example, be performed from the code coverage CC of a portion Lj of the non-mutated program P to be modified by the predetermined mutation Mj.
• the method may comprise, in a step F5, the determination of the CC code coverage of the program P not mutated for each test Tk of the plurality of tests.
• the code coverage CC which is for example determined by simulation, indicates the number of times that a portion Lj 1 typically a line of a program P is executed, with a given test Tk.
• Table 1 illustrates the CC code coverage of three lines 45 to 47 of a program P, for example a program of the VHDL type, comprising a loop of the IF / THEN type, in which an instruction (line 46) is subject to one condition (line 45). When the condition is fulfilled, the instruction is executed. When the condition is not fulfilled, the instruction is not executed.
• the CC code coverage indicates that the instruction in line 46 is executed three times, while lines 45 and 47 are executed five times. In the example, the condition is then as well filled, in particular three times, than non-filled, in particular twice.
• Table 2 shows an example of a mutation Mj inserted in line 45 of the program P illustrated in Table 1, so as to generate the mutated program Pj.
• the mutation Mj is then likely to modify the RTkj result of Tk test and Tk test is, thus, relevant for mutation Mj.
• the test Tk is then selected for the sequence Uj of tests relevant for the given mutation Mj, in the step F4.
• Table 3 illustrates another example of CC code coverage of the three lines 45 to 47 of the previous program performed with another test Tk '.
• the code coverage CC indicates that the instruction of line 46 is executed four times with the test Tk 'and that the lines 45 and 47 are also executed four times with the test Tk'. In the example, the condition of line 45 is always filled.
• the Mj mutation illustrated in Table 2 is then not likely to modify the RTk'j result of the Tk 'test and the Tk' test is thus not relevant for the Mj mutation.
• the test Tk ' is then not included in the sequence Uj of tests relevant for the given mutation Mj. In the case where all the tests are irrelevant, the mutation is not used in the rest of the process.
• the method comprises determining at least one classification by applying at least one criterion of simplicity to the selected tests.
• the determination of at least one classification comprises the determination F6 of a classification Cm of the mutations Mj by applying a first criterion of simplicity to Uj suites of previously selected tests.
• the classification Cm of mutations Mj is established, after determining the number Nj of tests Tk of each test suite Uj, in ascending order of the number Nj of tests Tk of each test suite Uj.
• the classification Cm mutations Mj can also be established in ascending order of the calculation resources required by each sequence Uj tests or in increasing order of the estimated duration of the execution of each suite Uj tests.
• the duration of the execution of the program P not mutated with all the tests Tk (step F2) of the associated test suite Uj is determined and the classification Cm of the mutations Mj is established in ascending order of the duration of the execution of the unmutated program P executed with all the tests Tk of the test suite Uj.
• the classification Cm of the mutations can be established according to several first criteria, which are applied simultaneously, possibly according to a hierarchy of first criteria. For example, the mutations Mj are first classified according to the number Nj of tests Tk of each test suite Uj, and then, when several numbers Nj of tests Tk are equal, a first additional criterion is applied.
• the rank Cm mutations Mj may comprise a sequence of subclassifications of mutations.
• Mutations can, in known manner, be grouped by different types. Types of mutations are, for example, mutations by omitting a numerical value or mutations by inserting an incorrect logical factor.
• the classification Cm mutations Mj can, thus, be modified or adjusted according to the types of mutation. Within a subclassification of mutations, for example, mutations can be classified according to mutation types.
• the classification Cm of the mutations Mj can also be adjusted according to parts Lj of the program P that are not mutated and intended to be modified by the mutations Mj, for example by grouping, at the beginning of the classification Cm, all the mutations Mj intended to be inserted into a same predetermined line (Lj) and grouping, following in the classification Cm, all the mutations Mj 'intended to be inserted in another predetermined line (Lj').
• the classification Cm mutations Mj may comprise a sequence of mutations subclasses each corresponding to a predetermined portion of the program P unmutated.
• the different subclassifications of the mutations are obtained by applying the first criterion of simplicity as described above.
• the Cm ranking mutations Mj can also be modified or adjusted depending on the probability of mutations to be detected. Thus, when certain types of mutations prove to be detected more difficultly, that is to say less often, they can be placed more at the beginning of the classification Cm mutations.
• a simplicity criterion corresponds, a priori, to a probability criterion of detecting mutations and thus automatically creates a classification of the mutations in order of increasing probability of the mutations to be detected.
• the classification of mutations can also be determined taking into account the subtlety of mutations.
• One way of measuring the subtlety of a mutation inserted in a single line is the number of times that said line is activated by the tests. The longer this line is activated, the less the corresponding mutation is considered subtle.
• the subtle mutations are preferably placed at the beginning of the classification of the mutations.
• the mutated programs Pj from step F1 are then executed, respectively in steps F7, in order of the rank Cm mutations Mj.
• Each program Pj mutated is executed with the tests Tk of the test suite Uj associated selected in step F4.
• the mutations Mj associated with the Uj suites of tests having the greatest simplicity are tested first.
• a maximum of different mutations Mj is tested before a final interruption of the process.
• the method makes it possible to obtain a greater percentage of undetected mutations than a method selectively selecting mutations, for example.
• the method makes it possible more effectively to determine the ineffective Tk tests to eliminate them, eventually.
• the time t is equal to or greater than a predetermined stopping time ts, as represented at the YES output of the step F8, the execution of the mutated programs F7 is interrupted. in a step F9.
• the time t is incremented, in a step F10, and compared again with the time ts stop in the step F8.
• a step F12 are compared the results RTkD and RTkj respectively associated with the execution of the program P not mutated, with the predetermined test Tk, and the program Pj mutated, with the predetermined test Tk.
• the mutation index j and the test index k are stored in a step F13.
• the mutated program Pj providing, with a predetermined test Tk, a result RTkj identical to a result RTkD provided by said non-mutated program P, executed with said test Tk.
• Such identification is equivalent to the non-detection of the corresponding Mj mutation.
• a mutation Mj is considered detected when the corresponding mutated program Pj provides, with a predetermined test Tk, a result RTkj different from the result RTkD provided by said program P not mutated, executed with said test Tk.
• the mutated program Pj is executed with the following test Tk, in the step F7, until the program Pj mutated is executed with all the tests of the associated test suite Uj. Then, the next mutated program Pj, according to the rank Cm, is executed, with the tests of the associated test suite Uj.
• the method comprises the programming of the programmable circuit 1 by the program P and by the mutations Mj, so as to create logical sets Gj associated with the mutations Mj.
• the programmable circuit 1 is then activated according to the method according to the invention, which makes it possible to obtain a test evaluation result in a time less than the time required for a method of activating a programmable circuit 1 according to the invention. prior art, while maximizing the number of logical sets Gj activated and tested.
• the determination of at least one classification comprises, after selection F4 of a sequence Uj of tests relevant for each mutation Mj, the determination F14 of a classification Ctj of the tests of a test suite Uj by applying a second criterion of simplicity to the Tk tests of said Uj series of tests.
• Each mutated program Pj is then executed in order of the ranking Ctj of the tests of the associated sequence Uj.
• the duration of the execution of the program P not mutated with each test Tk (step F2) of the test suite Uj is determined and the ranking of the tests Tk of a test sequence Uj is established in ascending order of the duration of the execution of the program P not mutated with each test Tk of the test suite Uj.
• the Ctj ranking of tests of a test suite Uj can also be established in ascending order of the computation resources required by each test Tk of the test suite Uj.
• the Ctj ranking of tests represents the advantage of maximizing the number of detected mutations, when the total test duration of the Mj mutation is limited.
• the method may comprise the determination of a classification Cm of the mutations and / or the determination of a classification Ctj tests of a sequence Uj tests.
• Each classification can be established by taking into account a single corresponding criterion or several corresponding criteria simultaneously.
• the classification Ctj of the tests Tk of a test sequence Uj is reduced to a first test of the classification Ctj of the tests Tk 1 for example by eliminating all the tests which follow the first test of the classification of tests. This makes it possible to identify, in a short-term test step, mutations which are, for example, detected by a simplified test, in particular by a test comprising only the first test of the associated test sequence Uj.
• undetected mutations can be analyzed by performing, for undetected mutations, all the tests of the associated test suites Uj.
• the method comprises, after execution F15 of part of the mutated programs Pj, the determination F17 of additional rankings Ctj of the tests Tk of the test suites Uj by application of additional simplicity criteria to the Tk tests Uj suites of tests.
• the method may also comprise the determination F16 of an additional classification Cm of the mutations Mj by application of an additional simplicity criterion to the sequences Uj of selected tests.
• the execution F7 of the mutated non-executed programs Pj can be continued in order of the additional classification Cm of the mutations Mj and / or respectively by order of the additional rankings Ctj of the tests Tk.
• the execution of the mutated programs Pj may, for example, be interrupted several times, if necessary, to create new additional rankings Cm and / or Ctj. It is also possible to create new rankings additional (Cm and / or C'tj) simultaneously with the analysis of mutations in progress, that is to say simultaneously with the execution of a portion of mutated programs Pj. Then, a next part of mutated programs Pj is executed in order of the new additional classifications Cm and / or C'tj.
• the tests Tk can be grouped according to several second criteria, inside a classification C'tj of tests Tk. For example, when, in the course of the analysis, two tests Tk and Tk * have detected a mutation Mj inserted in a predetermined portion Lj of the program P, these two tests Tk and Tk * can be privileged for the test of mutations Mj which have not yet been tested and which are intended to be inserted in said part Lj of the program P not mutated. These two tests Tk and Tk * are then preferably placed at the beginning of an additional classification C'tj Tk tests for a mutation Mj following intended to be inserted in said part Lj.
• a corresponding simplicity criterion can thus take into account the number of mutations Mj detected by the tests Tk 1 in a predetermined part Lj of the program P. Then, the test Tk having detected the highest number of mutations, in this part Lj predetermined, is placed at the beginning of the additional classification C'tj tests Tk for the Mj mutation test to be inserted in said part Lj.
• the analysis of mutations is improved according to the results obtained previously, during the analysis of mutations.
• the method comprises, after determination of the classification Cm of the mutations Mj (step F6), in a step F18, the selection of a reduced classification Cmr of mutations Mj.
• the reduced rank of mutations Cmr includes mutations intended to be inserted respectively into different parts Lj of the program P not mutated.
• the mutation ranked first among these mutations according to the Cm classification is retained for the reduced classification Cmr and followed, in the reduced classification Cmr, by a mutation intended to be inserted in a different line.
• the execution F19 of the mutated programs Pj, corresponding to the step F7, is performed in order of the reduced classification Cmr. Thus is favored the test of different parts Lj to the multiple test of the same part Lj.
• a predetermined mutated program Pj provides, with a predetermined test Tk, a result RTkj identical to the result RTkD provided by said non-mutated program P executed with said test Tk (step F12)
• the method comprises the elimination, in a step F20, in the classification Cm of the mutations, of all the mutations Mj intended to be inserted in the same part Lj of the non-mutated program P as the mutation Mj corresponding to said mutated program Pj predetermined, so as to obtain a streamlined classification Cmp.
• the mutated programs Pj are then executed according to the refined classification Cmp, in a step F21, corresponding to the step F7.
• the part Lj of the non-mutated program P may, for example, consist of a line, a block or a module of the program P not mutated or by a set of lines, by a set of blocks or by a set of modules.
• the computer program P may be a model of an integrated circuit, expressed in any description language such as, for example, Verilog or VHDL
• a system for evaluating tests of a computer program P by analysis of mutations Mj according to the invention comprises a control system 2 for controlling the various steps of the evaluation method according to the invention, for example for the activation of a programmable circuit 1.
• Computer readable media e.g. a CD-ROM, a hard disk, a floppy disk, etc., includes control instructions from a computer for performing a method of evaluating tests of a computer program P by mutation analysis Mj according to the invention, the method comprising the execution of mutated programs Pj mutated by insertion of Mj mutations and the identification of mutated programs Pj providing, with a predetermined test Tk, a result RTkj identical to a result RTkD provided by said non-mutated program P, executed with said test Tk.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Computer Hardware Design (AREA)
• Quality & Reliability (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Tests Of Electronic Circuits (AREA)
• Debugging And Monitoring (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=34951087

Family Applications (1)

Country Status (4)

Families Citing this family (7)

Family Cites Families (4)

• 2004
  • 2004-07-30 FR FR0408429A patent/FR2873832B1/fr not_active Expired - Lifetime
• 2005
  • 2005-07-12 US US11/631,191 patent/US7574681B2/en active Active
  • 2005-07-12 WO PCT/FR2005/001790 patent/WO2006024723A2/fr active Application Filing
  • 2005-07-12 EP EP05788600A patent/EP1776640A2/de not_active Withdrawn

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events