Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/36—Preventing errors by testing or debugging software
  • G06F11/3668—Software testing
  • G06F11/3696—Methods or tools to render software testable
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/36—Preventing errors by testing or debugging software
  • G06F11/3604—Software analysis for verifying properties of programs

Definitions

• the present invention relates to a method for the evaluation of operational characteristics of a program as well as to an execution system implementing said method.
• An application can for example perform illegal operations (divulge a secret code, connect to unauthorized sites, silently send messages, etc.), perform lawful operations but for which it does not have the appropriate rights, consume more resources than it can claim and thus deprive other applications, alter valuable data, etc.
• an application may use hardware or software functionalities that prove not to be present on the execution platform (for example because they are optional) or that are present unexpectedly or inappropriate (eg because they are limited or because of a version mismatch). In this case, the application does not work or malfunctions, which the end user may wrongly attribute to the service or hardware provider.
• a correction check consists of checks that are specific to a program and its specification, whereas checks of a validity check may be identical for all programs in a class of programs (for example: games , couriers, banking applications, etc.) based in particular on the expected use of system resources for the type of program assumed.
• the same "policy" is applied to a set of programs.
• An example of a correction check is the verification of orders accepted by a smart card application.
• Smartcard applications are programs that receive input byte sequences and output byte sequences.
• An input byte sequence typically represents a command (APDU command, "Application Protocol Data Unit") that the program must execute.
• API command "Application Protocol Data Unit”
• the validity check is governed by the a priori definition of a policy, applicable to a set of programs and defined as a set of criteria to be met or rules to be met.
• security criteria can be derived from a prior risk analysis, which notably assesses the assets to be protected on the platform and in the programs, the threats against these assets, and the measures to be taken for to protect them.
• Interoperability criteria can be drawn from the technical specifications of the different execution platforms, the error cases reported by the test teams and the users, the good programming experience of the developers, etc.
• CS-DESKeyStrength can stipulate that it is forbidden to use "DES” keys that are too weak, for example less than 128 bits long.
• an interoperability criterion could for example stipulate not to use an "RSA” key (for example because the target platform does not allow this type of key to be handled), or no "RSA" key of length greater than 1024 bits (for example because the target platform does not allow to handle keys of this size).
• the "buf”, “off” and “len” values are dynamic arguments provided to the “install” method.
• the “buf” table contains the sequence of bytes transmitted to the map, and “off” and “len” determine the corresponding segment in “buf”.
• “buf [offj” (the byte stored at the “off” index of the table “buf") is equal to "TYPE_DES” and that "getShort ⁇ uf, off + 1)” (the Short integer stored at the "off + 1" and “off + 2" indexes of the "buf” table) is less than 128.
• an application may only run in a particular context.
• the life cycle of a smart card application usually begins with a phase of card loading, installation, and customization, which are typically done under the control of the institution issuing the card. under the control of the application provider.
• the interactions with the application during these phases are not arbitrary.
• the application does not meet the security criterion in the strict sense (that is, for any possible execution, especially for any input provided to the program), the criterion is nevertheless satisfied in practice (ie ie for a given execution class, especially for a given class of inputs).
• the "legal entries" are the entries compatible with said context, and a "legal execution” of the application is an execution on legal entries.
• it is important to define the context of execution considered and in particular the class of the legal entries of the application in this context for example: any entry possible from the application, arguments with a fixed value or in a fixed range of values, etc.
• the situation is the same for the case of the correction check: the verification of the suitability to a specification may relate only to particular execution contexts, including particular input classes, given by the frame (s) ) using the application.
• CS-NoSMSInLoop can prohibit the sending of SMS (text messages) inside a programming loop (iterative or recursive construction) to avoid a number of unintended and unintentional mailings. doubt unjustified.
• CS-NoSMS Surtax can prohibit the use of surcharged SMS, that is to say for which the consumer will be charged in addition to the price of a simple SMS. For example, currently in France, the SMS numbers of the form “3xxxx” (where each "x" is any number) are not surcharged, those of the form “4xxxx” are surcharged by 5 euro cents, those of the form “Ixxxx” are surcharged by 50 euro cents.
• correction and validity checks do not necessarily concern strict Boolean properties that can only be true or false, such as the equality between two values.
• the validity in this case is characterized by the number of bytes allocated. Depending on the context of use, this number may turn out to be lower or higher than certain thresholds, thus determining validity in the particular context.
• a program may correspond more or less well (to different degrees) to what is stipulated in its specification, and it is only given a particular context that the implementation can be considered correct or not, with different degree and level of trust.
• a program may satisfy different validity criteria more or less, and it is also only given a particular context that the implementation can be considered valid or not, to such a degree and with such a level of confidence .
• An operational characteristic of a program is a value (more or less complex) associated with the execution of the program, such as the memory consumption, the execution time, the list of the routines of the system called during the execution of the program as well as the corresponding values of the arguments, etc.
• Operational features also include the status of controls, including correctness and validity, which judge whether the behavior of a program matches that stipulated in a specification or whether the program meets a number of validity criteria.
• the evaluation of an operational characteristic can often be seen as the result of a combination of more basic evaluation fragments.
• the memory consumption can be constructed from a combination operation which is the sum function, and evaluation fragments which are the number of bytes requested during execution, at the various memory allocation sites.
• a control can be expressed using a combination operation that is the logical conjunction as well as evaluation fragments that are the verification of logical conditions that must all be true for the control not to detect. abnormality.
• An operational characteristic of a program, and particularly a validity criterion may or may not relate to a particular location of the program execution.
• the "CS-DESKeyStrength" criterion is uniformly expressed throughout the program and is not tied to a particular code area, nor is a correction check performed on an output of the program.
• the "CS-NoSMSInLoop" criterion controls the type of program area where SMS sending can occur.
• the transgression of a criterion usually occurs in specific places of the program or the execution.
• a program point is an address in the program code, usually an instruction or an executable operation.
• An execution path represents a finite or infinite sequence of successive program points in the same execution of the program. In the case of a program with loops, an execution path may have several times (possibly infinitely many times) the same program point.
• a class of execution paths can be represented by a subgraph of the control flow and / or program flow graph, a grammar, a transition system, a time logic expression with the basic state of the points program, etc.
• An execution point represents a moment in the execution of a program, usually an instant determined by a given current program point and / or a condition on the current state of the program and / or the history of the program. the execution.
• An execution point represents a moment in the execution of a program. At a run-point corresponds a current program point, as well as an execution path from the program start point to the current program point. In addition, the execution of a program is punctuated by the occurrence of events, internal or external to the program, which may or may not be relevant for determining the operational characteristics of the program.
• Hardware events such as pressing a key on the keyboard by the user or the arrival of a call on a mobile phone, result in software events after processing by the execution platform and possible crossing of layers of the runtime system.
• Software events may include simple, localized, qualified and / or composite events:
• Simple events include: executing certain instructions or operations, program calls to runtime routines, runtime system calls to program routines, program interrupts, and / or failover system interrupts. executing, throwing an exception, catching an exception, changes in program state and / or runtime, variations in runtime resource availability, time stamps in the execution of the program (total time and / or detailed time for the various operations performed), etc.
• Localized events are events that occur at specific locations in the program and / or when entering particular places and / or when leaving particular places. A particular place may be for example a program point, an execution path, a routine, a programming loop (iterative and / or recursive), an exception or interrupt catch-up code, etc.
• Qualified events consist of events dealing with particular values or classes of values: parameters of statements or operations, routines called, arguments of routines, type of exception thrown or caught, state of the program and / or system, state of availability of the execution system resources, value of the time spent in the execution of the program (total time and / or detailed time for the various operations performed), etc.
• Compound events consist of combinations of events that can be used to express the history of the execution via the sequence of events, for example the fact that the call to a certain routine occurs only after the call of another.
• a typical qualified event is the call of a given routine with arguments of given value or within given value ranges.
• criteria such as, for example, validity criteria, are expressed as rules.
• a rule is satisfied if its logical value is true. It is transgressed if its logical value is false. It is indeterminate (or unresolved) if its logical value is indeterminate.
• the conditions Cy, ..., CN can relate in particular to the nature of the event E, but also to the values manipulated by the program and / or the execution system at the moment when the event occurs, on the flow control of the program (for example, the presence of the execution point inside a programming loop), at the particular place of the program where the event occurs (for example, given program points or routines) etc.
• the validity criteria given as examples above are special cases of such rules.
• the criterion "CS-DESKeyStrength" of minimum key length DES can be expressed as follows: if the method "buildKey" is called with the first argument "TYPE-DES" then the second argument must be greater than or equal to 128.
• the criterion "CS-NoSMSInLoop" of non-sending SMS in loop can be expressed as follows: if the method "MessageConnection.send” is called, then the calling point must not be inside a programming loop .
• the criterion "CS-NoSMSSurtax" for non-sending overtaxed SMS can be expressed as follows: if the method "Connector.open” is called with an argument starting with "sms: // ", then this argument must be of the form” sms: // 3xx; ⁇ ".
• conditions do not necessarily have to be related to computable elements or properties. For example, they may relate to ethical and ethical principles (no sexual, critic, etc.), which can only be controlled by a human being and not by a machine. The same goes for most display conditions, such as Look and Feel. This type of condition is not automatically verifiable; it requires the advice of a human operator.
• control is static (before running on the platform) or dynamic (during execution);
• control is "black box” (without looking at the code, observing only the reactions to the entries) or "white box” (looking at the code and / or the internal state of the running program);
• the control is "manual” (performed by a human) or automatic (performed by a decision tool).
• a control can be "semi-automatic” (with human intervention and decision tools).
• the control can be "gray box” in the case where only a part of the state of execution of the program is observable.
• the program can also be tested in "black box” or “gray box”, without accessing the code to analyze it, only in response to stimuli. With this approach, the program is executed in a number of supposedly representative circumstances, and its behavior is observed from the outside (or in part from the inside in the case of the gray box) to judge whether it corresponds to the behavior. expected and / or meets the criteria set. •
• the program can also be controlled automatically, using an analysis tool, provided of course that the controls are automatable.
• Some tools check with a static analysis that the code of a program follows general rules of good training and good typing, which provides some guarantees of basic operation, regardless of the platform target. This is the case, for example, of verifiers for virtual machines ("bytecode verifiers") associated with "Java” runtime environments (including “Java Card”). Such type analyzers can work on both source code and object code. However, they do not meet specific criteria, for example regarding security or interoperability.
• French Patent No. 03 15544 filed December 30, 2003 in the name of the Applicant, provides a method for determining the operational characteristics of a program, based on static analysis techniques.
• static analysis techniques are numerous: abstract interpretation, control flow analysis, data flow analysis, model checking, formal proofs, and so on.
• the information resulting from a static analysis may make it possible to deduce certain operational characteristics of a program, for example memory consumption or execution time, and in particular to carry out validity checks. But static analysis also allows in some cases to perform some correction checks.
• the commands that a Java Card application can answer can be determined by exploring all the execution paths that start from the "Applet.process” method, by examining all comparison operations on the first byte (byte class "CLA") and the second byte (instruction byte "INS") of the byte array constituting the "APDU buffer", and observing the status of command returns (normal execution or error).
• the commands accepted by the application correspond to the values of the first and second bytes for which the program does not throw an exception.
• the implementation is correct with respect to the accepted commands if the values determined by the static analysis are equal to the values indicated in the specification.
• Another example is the verification that certain actions do not occur, in particular that certain sensitive data, such as a PIN (Personal Identification Number), are manipulated only in the circumstances provided by the specification.
• PIN Personal Identification Number
• a static analysis can explore all the execution paths that start from the recognition of this command and check that none of them does not affect a PIN object (including its verification status and its successive unsuccessful test counter).
• a static analysis does not always make it possible to precisely determine any operational characteristic of a program. This may be due to a combination of various factors: difficulty or inability to automate the evaluation of the operational characteristic (for example, ergonomics, ethics or ethical issues), variable behavior according to program inputs, lack of precision of the static analysis, intrinsic complexity of the program, etc.
• the specification of a "Java Card” application complying with the ISO 7816 standard may stipulate that the "GET DATA" command (whose INS byte is equal to 202) does not implement a security measure (the octet CLA is equal to 0) and the "PUT DATA" command (whose INS byte is equal to 218) implements a proprietary security measure (the CLA byte is equal to 132).
• an analysis can simply determine that the second argument of the "buildKey” method is between 96 and 192, without further precision. In this case, it is not possible to establish with certainty whether the key length security criterion mentioned above (length at least 128) is satisfied or transgressed.
• a program analysis can provide relevant information to assist in diagnosis: locating the potential offense, the approximate status of the program and / or the program delivery path, etc. This information is also useful in the case where the program is incorrect (respectively invalid) with certainty, in order to know where and possibly how to correct it.
• test is a different approach. This is the most commonly used solution for both the correction check and the validity check.
• a test includes a scenario of interactions with the program as well as evaluations of operational characteristics, based on the observed behavior of the program.
• Interactions include entries to submit to the program (input data provided and actions performed including interrupts, variations of availability of resources, time marks, etc.) and outputs produced (output data and produced actions). Some entries can reach the program via the runtime system (operating system, virtual machine, software interface, etc.) over which the program runs, for example as a call to program routines or under form of return values provided to system routines called by the program. The same goes for program outputs, which can pass through the runtime system.
• runtime system operating system, virtual machine, software interface, etc.
• Operational feature assessments are treatments that focus on inputs to the program and outputs produced. For example, executing on a simulator commands for installing and customizing a smart card application results in the execution of memory allocation operations; several executions of this type, with values representative of a certain context of use, make it possible to estimate approximately the memory consumption of the application.
• the test scenario also includes stopping conditions, for example: exhaustion of the behaviors to be observed, number of errors or transgressions exceeding a certain threshold, time spent in the execution of the test (which has an impact on the cost of the test run, especially in the presence of a test operator), etc.
• a scenario can simply stipulate to browse "as much as possible" the various screens of the application in order to check the texts displayed are written in the target language and not are not truncated. For example, one or more hours can be left for a test operator to perform several such checks at the same time. There is no this case guaranteed result but effort guarantee, the quality of the final status depends a lot on the test operator.
• tests are usually necessary to approach an operational characteristic, or to make a judgment of correction or validity. These tests are grouped in a test suite.
• the program is run using the runtime device of a test platform.
• This execution device can be based on the program's target execution platform or on any equivalent execution platform: simulator, emulator, and so on.
• a test platform may need to be configured with respect to the target execution platforms as well as the intended context and execution environment.
• the test platform may have to offer the program a more or less complete runtime environment. For example, to test a mobile phone application, it may be necessary to have the network of a telephone operator (simulated if necessary), an operational server at the content provider, etc.
• Some execution devices may also have introspection facilities to provide, in addition to regular program outputs, information on the states and internal actions in progress. This information can be provided by the platform in a passive manner (during execution, the platform reports events and state changes) or active (running, the platform may be questioned about his condition).
• Some enforcement devices may also provide a means of acting on enforcement. This is particularly the case when they include a debugger or the features of a debugger. It is then in particular possible to interrupt the execution temporarily, in particular following the setting of breakpoints (at certain program points, at the entry of certain routines, etc.).
• Target platforms in general do not allow introspection or action for various reasons: security, robustness, design, etc.
• test platform does not include introspection or action on execution, it may be possible to have such features by instrumenting the program.
• the program can be instrumented before execution to make this information more easily accessible or accessible with greater precision.
• the program can be instrumented to produce, in addition to its original outputs, additional outputs that provide information on its state and its internal actions.
• the instrumentation may be such that these additional outputs are generated spontaneously during program execution or produced in response to particular inputs of the program.
• Said particular entries may be original inputs of the program and / or additional entries explicitly introduced by the instrumentation in order to query the program and / or the execution system on their state and / or their actions.
• a smart card application can be instrumented to first count the number of bytes allocated since the beginning of the execution and secondly accept an additional command (or an additional parameter to an existing command, for example of type "GET DATA") which returns this value.
• the program can also be instrumented to act on its execution, in particular to interrupt it temporarily (for example while waiting for a particular entry). This allows in particular to provide performance checks when the execution platform does not have them. For example, it is possible to simulate certain debugger functions, such as setting a breakpoint at certain program points, entering certain routines, and so on.
• the program can also be instrumented to, in addition to its original effects, dynamically perform the evaluation of some of the desired operational characteristics and produce the result of this evaluation as an additional output.
• the operational characteristics to be evaluated include checks to be carried out or rules to be verified
• the program may be instrumented to, in addition to its original effects, dynamically perform the check or dynamically check the rule and produce a status of that check or this verification (satisfaction or transgression).
• the instrumentation may precede all occurrences of calls to the method "Connector.open” of a test on the string given as argument of this method to check that if it starts with "sms: //", then it is not followed a premium number, for example of the form "Ixxxx”.
• the instrumentation can be manual and / or automatic.
• Automatic instrumentation may include recognizing particular patterns ("patterns") of code and modifying them systematically. For example, in the case of a "Java Card” application, it is possible to identify all the occurrences of allocation instructions (of the "new" type) as well as all occurrences of calls to allocation methods of objects (of the type "makeTransientByteArray", "buildKey”, etc.) and precede them with the increment a count of the number of bytes allocated by the dynamic value requested at runtime.
• the behavior of the program is monitored by observing the outputs produced by the program in response to the inputs made and / or by interrogating the program on its state and its internal actions if it is instrumented at this time. effect and / or by interrogating the test platform if this allows it (this is usually the case of a test platform based on a simulator or an emulator).
• the information collected on the behavior of the program can be made available via a control device of the test platform, which can in particular provide means for representing the outputs made by the program, means for representing the current state and actions performed by the program and / or the execution system (including an indication of program points and / or execution points achieved since the execution of the test or a set of tests); interrogation of the program and / or the test platform, and storage means and representation of the execution history.
• a control device of the test platform can in particular provide means for representing the outputs made by the program, means for representing the current state and actions performed by the program and / or the execution system (including an indication of program points and / or execution points achieved since the execution of the test or a set of tests); interrogation of the program and / or the test platform, and storage means and representation of the execution history.
• this control device allows a test operator to answer questions asked in the test scripts, including the conditions for stopping a test.
• test scenario is executed by an executable test script that installs and / or launches the program on the test platform and / or links the different interactions stipulated in the scenario.
• a manual test it is a test operator who installs and / or launches the program on the test platform and / or performs the various interactions recorded in the scenario.
• a single test can combine automated operations with manual operations.
• program entries can be provided automatically by the test script and / or manually by a test operator.
• controls can be performed automatically by the test script and / or manually by a test operator.
• test operator In the case where a test operator must intervene, the interactions can be described in a test specification and / or requested by the test script executed, for example via a display on a control station. In the latter case, the test operator is guided in the test scenario by requests for entries and / or checks made by the test script and by the observations he can obtain on the execution of the program. This allows in particular to make entries or controls impossible to achieve by a machine. It also provides information to decide when to stop a test. For example, a test operator may decide to terminate an execution when a group of execution points of a program (for example, all execution points, all execution points associated with a rule, etc.) have been reached by performing one or more tests.
• a group of execution points of a program for example, all execution points, all execution points associated with a rule, etc.
• Providing performance information (execution points so far, called routines, program status and / or runtime system, etc.) also allows the test operator to guess what type Interaction allows to reach such or such point or zone of program and / or to trigger this or that type of event.
• test In the case where a test must evaluate the satisfaction of a rule of the form "if E then Ci and ... and CM", the test must include evaluations of the logical expressions that are the condition of occurrence of the event E and the conditions to be fulfilled Ci, ...,
• test platform is a simulator or emulator of the target platforms
• information for example, the current program point, the status of the program and / or the execution system, etc.
• This additional information can in particular make it possible to automatically determine the logical value of some of the conditions of the rules defined as "if E then Cj and ... and CV".
• test platform In the case where the test platform is the target platform itself, only the external effects are usually visible, not the internal events. Verification of the occurrence of an event may, however, be indirect. For example, the test operator may observe the dialed telephone number when sending an SMS with an authorization request. On the other hand, some information may remain inaccessible, such as the current execution point or the fact that this execution point is in a programming loop. Some information may, however, be made available after instrumentation of the program.
• test When an event associated with a rule occurs, after automatic or manual determination of satisfaction, transgression or irresolution, it is necessary to decide whether or not to continue the test.
• the condition for stopping the execution of a test is stipulated in its scenario. It may depend on various factors relating to rule status, possibly at the point of execution where it is controlled. A test may also be interrupted after a given execution time, for example because it is judged that obtaining additional information would not be profitable, that is to say, would require too much time.
• the quality of a test suite depends in particular on its degree of coverage in different aspects, including:
• control flow coverage a posteriori measurement of the proportion of control flow elements (instructions, execution branches, execution paths, iterative structures, etc.) that are run during the test suite execution for a given implementation. In practice, checking the elements of the control flow that are traversed requires the ability to observe the execution and / or to be able to instrument the code.
• test suite In addition to coverage, other parameters affect the quality of a test suite, including the amount of resources it requires: equivalent tested functionality, better a test that runs in a short time, consuming little memory, etc.
• the degree of coverage and the amount of testing depend on the methodology of test development and the care of the tester. The higher the coverage, the greater the effort required to design and perform the tests. However, in the software industry, productivity can be such an important parameter if it is more than safety or proper functioning criteria.
• All or part of the data of a test suite can also be generated automatically from a formalization of the application specification.
• one of the advantages is to be able to offer guarantees on the degree of coverage of a formal specification.
• the tools capable of producing relevant test data in this way are still rare.
• automatically generated test suites are subject to the same limitations as hand-written test suites.
• test suffers from several limitations. This is a well-adapted approach for correction control (comparing its actual outputs to expected outputs), but less for validity checking.
• test classes are often under-specified, such as, for example, running tests of applications for mobile phones; many of these tests involve running the application in a limited time as an ordinary user and noting some abnormal behavior of the application.
• the human test operator is not sure what to look for or how to find it. At best he has a list of test objectives and he tries to wait for a maximum number of goals.
• test operator does not always have the means to evaluate the quality of his work.
• he remains powerless in front of malicious code, designed to remain hidden and to act only in particular circumstances that do not occur during the first rapid and superficial use of the application.
• the object of the invention is an evaluation of the operational characteristics of a program, which notably allows the development of correction or validity control tools that provide a high level of certainty and / or productivity. . It thus makes it possible to obtain better compromises in efforts to provide and resulting confidence.
• This method proposes a method for the evaluation of operational characteristics of a program, this method implementing a test platform (hardware and / or software) allowing the execution and the control of said program as well as a series of tests for evaluating said operational characteristics.
• This method is characterized in that it comprises an evaluation procedure comprising the following steps:
• a first step where the program is subjected to a static analysis which determines the operational characteristics of said program, with greater or lesser precision and which produces a set of test requirements intended to evaluate the operational characteristics which have not been determined with sufficient precision;
• a third step where the program executed on said test platform is subjected to a test suite parameterized according to the test requirements generated during the first step; a fourth step where the operational characteristics of the program are formed from the operational characteristics determined by said static analysis, supplemented by the operational characteristics evaluated by said test suite.
• the static analysis performed in the first step takes into account the assumption that only the input sequences possible program are sequences of legal entries.
• the operational characteristics determined by the static analysis include one or more of the following information: the control flow, the data flow, the memory consumption, the execution time, the list of routines of the system called during the execution of the program, the corresponding values of the arguments, the status of the correction or validity checks.
• Some tests in the test suite may have parameters that affect the operations performed during test execution and that may be adjusted to vary the accuracy with which the program's operational characteristics are evaluated.
• the static analysis may produce parameter values for some of the selected tests to restrict said tests to only those operations that may improve said accuracy.
• the tests may include:
• test suite may include tests: • whose entries to submit are provided by a test program and / or by a test operator, the test program can indicate to the test operator specific operations to be performed on the test platform, and
• a control device may assist the test operator by recording the observations of the test operator and presenting him with an up-to-date view of the operational characteristics evaluated until then so that he can may consider whether to continue the test or whether the accuracy achieved is sufficient.
• the vision of the evaluated operational characteristics presented to the test operator may include a vision of program execution consisting of a representation indicating whether certain events have already occurred and / or under which circumstances, including:
• Some of the above operational characteristics may be dependent on one or more target (target hardware and / or software) execution environments for the program as well as one or more target configurations of said environments (particularly with respect to resources and / or features available and / or allowed in said environments), and the test platform includes and / or emulates and / or simulates said target execution environments with their target configurations.
• the aforementioned operational characteristics to be evaluated may include the verification of properties and that: In the first step, the static analysis determines properties that the program satisfies with certainty and properties that the program transgress with certainty (the other properties being properties that the analysis does not determine with certainty as being satisfied or transgressed); - In the second step, the program is subjected to the tests of the test suite which concern the properties of the program that the static analysis has not determined with certainty as satisfied or transgressed.
• the static analysis may in particular determine on the one hand what are the execution points in which said event occurs with certainty or can occur and, secondly, the truth value of said conditions at said execution points when said event occurs, and in the second step, when during a test said event occurs or can occur at a PE execution point, only evaluated conditions that were not established by the static analysis as true with certainty at the PE execution point when said event occurs.
• the aforesaid operational features to be evaluated include checking rules that determine whether the program is valid (respectively correct), in the third step, the operational characteristics of the program comprising the rule check being constituted according to the following procedure:
• Test requirements are information that guides the execution of tests. They may include particular interactions with the program and specific assessments to be made. Specific interactions - specific entries to be submitted to the program.
• the guiding of the execution of the tests can result in a parameterization of the execution device of the test platform (including the instrumentation of the platform) and / or an instrumentation of the program including in particular the installation of point systematic shutdowns, the installation of conditional stop points, the production of information enabling the evaluation of operational characteristics.
• the program is subjected to a series of tests intended to evaluate said operational characteristics.
• the static analysis has determined a satisfactory framework (as needed) for the consumption application memory, it is not necessary to test installation and customization commands with various values to try to estimate the memory consumption of the application.
• the static analysis has determined with certainty the status of a control, for example the satisfaction or violation of a rule on the sending of SMS, it is useless to perform tests that try to verify whether this rule is respected or not in various circumstances.
• test fragments capable of performing the evaluation fragments that the static analysis has not determined with sufficient precision are executed; the remaining evaluation fragments are provided by the static analysis.
• operational characteristics are controls to be performed or rules to be verified, only the test fragments capable of performing the evaluation fragments which the static analysis has not determined with sufficient precision for to rule.
• the tests are run on the test platform, configured if necessary against the program's target execution platforms.
• This configuration may be restricted to only those cases which concern the operational characteristics that the static analysis has not determined with sufficient precision.
• the operational characteristics are controls to be carried out or rules to check
• the configuration can be restricted to cases that concern controls or rules on which the static analysis has not determined with certainty.
• the instrumentation of the program may notably be based on information extracted by the static analysis of the first phase and on the particular operational characteristics to be evaluated, in order, for example, to minimize the additional consumption of resources (memory, execution time, etc.) and / or instrumentation limitations inherent in the program and / or the execution platform.
• the instrumentation tool can be parameterized using the information provided by the static analysis.
• the representation (respectively storage) of the information in the control device of the test platform can be made in such a way as to minimize the irrelevant information, in particular by limiting the representation (respectively memorization) to the only information necessary for establishing the data. operational characteristics that the analysis has not determined with sufficient precision. This allows a better comfort and greater productivity of the test operator (respectively an economy of resources in terms of memory and execution time).
• This limitation of the representation (respectively storage) can be obtained by a parameterization of the control platform. This control platform can be directly parameterized by the results of the static analysis.
• the information necessary to establish the occurrence or non-occurrence of an event may concern not only the current state of said execution but also the history of this execution.
• This information can be provided automatically or not by the test platform and / or by a test operator, in particular at the request of the test script and / or with the aid of the control device of the test platform. . It is possible that some information remains indeterminate. When in doubt, the occurrence of the event should be considered as possible. This situation may arise in particular if the test operator can not, does not know or does not want to decide on issues that are his responsibility. There may be several reasons for this, for example: ignorance of the question asked or actions to be taken, lack of additional information, lack of time to provide an answer, etc.
• the program When the question of the occurrence or not of an event arises, the program is in a certain execution point PE (or in a succession of execution points if the program is not interrupted momentarily waiting for a special entry). This execution point may not be immediately known outside the execution platform; it is therefore determined only with more or less precision.
• the test may continue. Otherwise, checks are needed to check the truth value of the conditions in order to try to determine the satisfaction or violation of the rule, before making the decision to continue the test or to interrupt it.
• condition C 7 is true at the PE execution point.
• condition C 7 It is unclear whether condition C 7 is true or false at the PE enforcement point.
• the verification of the condition must be automatable and the information necessary to determine with certainty if the condition is true (respectively false) must be available at this point of execution.
• the information necessary to establish with certainty the truth value of a condition may relate to the execution history as well as the current state of said execution. This information can be provided by the test platform and / or by a test operator, in particular at the request of the test script.
• test operator can not, does not know or does not want to decide on the information necessary to establish the truth value of a condition. This may be due to lack of knowledge of the question, lack of additional information, lack of time, etc. In this case, the condition must be considered unresolved, that is, possibly true or false.
• the R rule is considered to be transgressed with certainty at the PE enforcement point for some program entries.
• the execution of the test can be continued automatically (for example, to find more occurrences of transgressions), to terminate automatically (for example because a single transgression is sufficient to invalidate the rule), or continue only after agreement of the test operator (who is then judge of the relevance of the continuation of the test).
• At least one of the dynamically falsifiable conditions is not established as true with certainty at the PE execution point and none of the dynamically falsifiable conditions is established as definitely false at the PE execution point: In this case, the rule R is considered unresolved at the PE enforcement point for some program entries.
• the execution of the test may continue automatically, terminate automatically, or continue only after agreement of the test operator (who is then judge of the relevance of continuing the test).
• the following particular case of the invention is considered, allowing the assisted certification of applications.
• the execution device of the test platform allows the setting of breakpoints and the performance of introspection functions. If the execution device does not allow these features, the program can be instrumented for this purpose.
• control device of the test platform can be programmed to keep a history of certain information (including past events) and / or to evaluate expressions (in particular by using the functions of introspection and / or history) and / or to ask a test operator to evaluate answering certain questions.
• condition Ki the verification of the occurrence of the event E ⁇ is automatable, but that of the condition K 2 is not; only a human test operator can determine whether the display contains a warning, an authorization request, and so on.
• Step 1 static analysis
• the program is subjected to a static analysis that seeks to determine whether the rules are satisfied or transgressed in said execution context.
• rule R is not satisfied at an execution point.
• R is of indeterminate status
• rule R is transgressed into an execution point.
• execution point PE there exists an execution point PE such that E is determined with certainty as true and at least one of the conditions Q is determined with certainty as false.
• the result of the static analysis includes a set of information about the rules that are satisfied at any point of execution or transgressed at an execution point, as well as a set of test requirements.
• test requirements include in particular the set of indeterminate status rules in at least one execution point and / or, for any rule "if E then Q and ... and C # " of said set, the set PP R program points where the rule may be of indeterminate status and / or, for any PP program point of PPR, event E if it is dynamically falsifiable (ie undetermined) in PP as well as the subset of conditions "Ci, ..., CN" which are dynamically falsifiable in PP.
• Sets of program points belonging to the same class of program points may be grouped together as one program area, such as all program points of a particular routine, a programming loop, and so on.
• the program is subjected to a test suite that seeks to satisfy the test requirements produced by the first step.
• test platform configured as follows. For any rule R of indeterminate status for said test, a breakpoint is set in each of the PP program points of PPR.
• the device evaluates the event E of R if it is dynamically falsifiable and if the event occurs in PP, the control device also evaluates the subsets of R conditions that are dynamically falsifiable. If the verification of a dynamically falsifiable condition or the occurrence of event E is not automated, the test operator is queried, which can answer "true”, "false", or "undetermined".
• the control device continuously records the details and the total number of occurrences of application of each rule to the different program points as well as the different statuses obtained (satisfied, indeterminate, transgressed).
• the execution may continue automatically, stop automatically, or continue only after agreement of the test operator. For example, the execution can be continued even if a rule is satisfied, in order to reach a number of occurrences of satisfied rule deemed convincing. Similarly, one can continue the execution even after a transgression to try to find more cases.
• breakpoints are posed to PP2 program points, PPs and PP 4.
• the condition Kj is evaluated: if the argument of the method "Connector.open” is of the form “sms: // 3xxxx", the number of cases where rule Rj is satisfied is incremented; otherwise the number of application cases where the rule Ri is transgressed is incremented and the test operator is notified, who can decide whether or not to interrupt the test.
• the condition K 2 is evaluated: the question "did the program display a message signaling the sending of an SMS and asked permission of the user? ".
• the number of application cases where the rule R 2 is satisfied, indeterminate, or transgressed is incremented according to whether the operator answered "yes", “no” or "I do not know", respectively.
• the test operator may also decide to interrupt the test; otherwise the test execution continues.
• rule evaluations are only performed at PE enforcement points that are:
• the information from the static analysis and those from the test suite execution are gathered in a single diagnosis.
• the diagnosis can be qualified in terms of confidence depending on the number of remaining indeterminations and the coverage rate of the program points listed among the test requirements (eg, number of program points never achieved).
• the instrumentation of the platform in particular of virtual machine (JVM), can be pushed even further by considering evaluation fragments in execution points, events, conditions, etc.
• the execution device can be coupled to a real debugger allowing to examine in more detail the causes of a problem as soon as it is identified.
• the protocol used to allow the control device to control the execution device may be based on the Java Virtual Machine Debug Interface (JVMDI) used for debugging Java applications.
• JVMDI Java Virtual Machine Debug Interface

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Computer Hardware Design (AREA)
• Quality & Reliability (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Software Systems (AREA)
• Stored Programmes (AREA)
• Inorganic Insulating Materials (AREA)
• Apparatuses And Processes For Manufacturing Resistors (AREA)
• Testing Or Measuring Of Semiconductors Or The Like (AREA)

Applications Claiming Priority (2)

Publications (1)

Family

ID=34955138

Family Applications (1)

Country Status (3)

Citations (2)

Family Cites Families (1)

• 2004
  • 2004-12-22 FR FR0413768A patent/FR2879775B1/fr not_active Expired - Fee Related
• 2005
  • 2005-12-20 EP EP05850582A patent/EP1836580A2/de not_active Ceased
  • 2005-12-20 WO PCT/FR2005/003241 patent/WO2006070113A2/fr active Application Filing

Patent Citations (2)

Also Published As

Similar Documents

Legal Events