US20110271253A1 - Enhancing functional tests coverage using traceability and static analysis - Google Patents

Enhancing functional tests coverage using traceability and static analysis Download PDF

Info

Publication number
US20110271253A1
US20110271253A1 US12/768,764 US76876410A US2011271253A1 US 20110271253 A1 US20110271253 A1 US 20110271253A1 US 76876410 A US76876410 A US 76876410A US 2011271253 A1 US2011271253 A1 US 2011271253A1
Authority
US
United States
Prior art keywords
requirements
code
design specifications
design
computer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12/768,764
Inventor
Jonathan Bnayahu
Maayan Goldstein
Dany Moshkovich
Moti Nisenson
Yahalomit Simionovici
Shmuel Ur
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Priority to US12/768,764 priority Critical patent/US20110271253A1/en
Assigned to INTERNATIONAL BUSINESS MACHINES CORPORATION reassignment INTERNATIONAL BUSINESS MACHINES CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GOLDSTEIN, MAAYAN, UR, SHMUEL, BNAYAHU, JONATHAN, MOSHKOVICH, DANY, NISENSON, MOTI, SIMIONIVICI, YAHALOMIT
Publication of US20110271253A1 publication Critical patent/US20110271253A1/en
Application status is Abandoned legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/3668Software testing
    • G06F11/3672Test management
    • G06F11/3676Test management for coverage analysis
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/10Requirements analysis; Specification techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/20Software design
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/70Software maintenance or management
    • G06F8/73Program documentation
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00Arrangements for software engineering
    • G06F8/70Software maintenance or management
    • G06F8/75Structural analysis for program understanding

Abstract

A method that may include: building a dependencies graph representing dependencies between code elements of a computer code; associating portions of the computer code with corresponding design specifications or requirements derived from a design specifications document or a requirements document respectively which is associated with the computer code, to yield a design specifications or requirements-code tracing map; and analyzing the design specifications or requirements-code tracing map based on the dependencies graph to yield an ordered list of design specifications or requirements respectively, wherein the order is selected such that functional tests written for the computer code and addressing design specifications or requirements of a higher order, will yield a higher level of functional test coverage of the computer code in terms of design specifications or requirements.

Description

    BACKGROUND
  • 1. Technical Field
  • The present invention relates to computer code testing and more particularly, to improving functional test coverage for a given computer code.
  • 2.Discussion of the Related Art
  • Functional tests are used throughout the life cycle of computer code development in order to verify that the computer code being developed meets the functional requirements or the design according to which the computer code has been written. The functional requirements are usually set forth in a specification document, possibly written in a natural language by the client. A design is then written, also in a natural language by a computer system engineer. The design specifications contain high level description, in computer system terms, detailing how to address the requirements.
  • An important quantitative parameter in the code testing domain is the coverage of the design specifications or requirements. The design specifications or requirements coverage usually relates to the amount of the code that a given set of tests covers in terms of the design specifications or the requirements according to which, the code has been written. Existing solutions for assessing the aforementioned coverage require explicit mapping between the design specifications or requirements and the tests, either directly or indirectly in accordance with a specified model. Currently, there are no known methods to achieve such a coverage assessment without an explicit link between the design specifications or requirements and the tests. In addition, an assessment of the coverage associated with each design specifications or requirements may be advantageous in developing new functional tests for a given code in order to improve the code coverage.
  • BRIEF SUMMARY
  • One aspect of the invention provides a method that may include the following steps: building a dependencies graph representing dependencies between code elements of a computer code; associating portions of the computer code with corresponding design specifications or requirements derived from a design document or a requirements document respectively which is associated with the computer code, to yield a design specifications or requirements-code tracing map; and analyzing the design specifications or requirements-code tracing map based at least partially on the dependencies graph to yield an ordered list of design specifications or requirements respectively, wherein the order is selected such that functional tests written for the computer code and addressing design specifications or requirements of a higher order, will yield a higher level of functional test coverage of the computer code in terms of design specifications or requirements, wherein at least one of the building, the associating, and the analyzing is executed by at least one processor , and wherein the ordered list is useable to generate functional tests for enhancing code coverage.
  • Other aspects of the invention may include a system arranged to execute the aforementioned method and a computer readable program configured to execute the aforementioned method. These, additional, and/or other aspects and/or advantages of the embodiments of the present invention are set forth in the detailed description which follows; possibly inferable from the detailed description; and/or learnable by practice of the embodiments of the present invention.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • For a better understanding of embodiments of the invention and to show how the same may be carried into effect, reference will now be made, purely by way of example, to the accompanying drawings in which like numerals designate corresponding elements or sections throughout.
  • In the accompanying drawings:
  • FIG. 1 is a high level schematic block diagram illustrating an environment of a system according to some embodiments of the invention;
  • FIG. 2 is a high level schematic block diagram illustrating a system according to some embodiments of the invention;
  • FIGS. 3A and 3B show a block diagram and a graph diagram respectively illustrating an aspect according to some embodiments of the invention;
  • FIG. 4 shows a high level schematic block diagram illustrating an aspect according to some embodiments of the invention;
  • FIG. 5 is a high level flowchart diagram illustrating a method according to some embodiments of the invention; and
  • FIG. 6 is a high level flowchart diagram illustrating an aspect relating to the method according to some embodiments of the invention.
  • The drawings together with the following detailed description make apparent to those skilled in the art how the invention may be embodied in practice.
  • DETAILED DESCRIPTION
  • Prior to setting forth the detailed description, it may be helpful to set forth definitions of certain terms that will be used hereinafter.
  • The term “requirement” or “functional requirement” as used herein in this application refers to a requirement that defines a function of a software system or its component. A function is described as a set of inputs, the behavior, and outputs. Functional requirements may be calculations, technical details, data manipulation and processing and other specific functionality that define what a system is supposed to accomplish. Behavioral requirements describing all the cases where the system uses the functional requirements are captured in use cases. Functional requirements are supported by non-functional requirements (also known as quality requirements), which impose constraints on the design or implementation.
  • The term “design specifications” as used herein in this application refers to a high level description, usually in computer system terms, detailing how to address the requirements. This document, usually written in natural language is usually written by a computer system engineer after reading the requirements document.
  • The term “system testing” as used herein in this application refers to testing conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. System testing falls within the scope of black box testing, and as such, should require no knowledge of the inner design of the code or logic.
  • The term “test coverage” or simply, “coverage” as used herein in this application refers to a measure used in system testing. It describes the degree to which the source code of a program has been tested. It is a form of testing that inspects the code directly and is therefore a form of white box testing.
  • With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present invention only, and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice.
  • Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and the arrangement of the components set forth in the following description or illustrated in the drawings. The invention is applicable to other embodiments or of being practiced or carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting.
  • FIG. 1 is a high level schematic block diagram illustrating an exemplary environment of a system according to some embodiments of the invention. According to the exemplary embodiment, computer network 10 may include a personal computer 20 connected to a server 30. It is understood however, that any type of computer may be used instead of server 30. Server 30 may be in operative association with computer code 110 and design specifications or requirements document 150. In addition, server 30 may be in communication with functional tests 195.
  • In operation, computer code 110, design specifications or requirements 150, and tests from functional tests 195 may be used by server 30 to generate an ordered list of design specifications or requirements in which the order of the design specifications or the requirements is determined such that design specifications or requirements that are least covered by the tests, and have the higher level of code element associated with them directly or indirectly, will have a higher order of the ordered list of design specifications or requirements.
  • FIG. 2 is a high level schematic block diagram illustrating system 100 according to some embodiments of the invention. System 100 may include a builder component 120 configured to build a dependencies graph 130 representing dependencies between code elements of a computer code 110. System 100 may further include an association component 140 configured to associate portions of computer code 110 with corresponding design specifications or requirements derived from a design document or a requirements document 150 respectively. Computer code 110 has been written based on the design specifications or requirements. The associating of association component 140 may yield a design specifications or requirements-code tracing map 160.
  • System 100 may further include an analyzing component 170 configured to analyze design specifications or requirements-code tracing map 160 based on dependencies graph 130 to yield an ordered list 180 of design specifications or requirements respectively, wherein the order in the list is selected such that functional tests written for the computer code and addressing design specifications or requirements of a higher order, will yield a higher level of functional test coverage of computer code 110 in terms of design specifications or requirements. Optionally, analyzing component 170 may be imputed directly with dependencies graph 130.
  • FIGS. 3A and 3B show a block diagram and a graph diagram respectively illustrating an aspect according to some embodiments of the invention. In FIG. 3A, computer code 110 is shown, containing testable code elements (functions and/or classes) 111-118. FIG. 3B shows a respective graph 300 that shows the dependencies between code elements 111-118. These dependencies may be any form of logical/control flow relationship. For example, a dependent coded element is a code element that another code element may use, access, invoke, and the like.
  • Returning now to FIG. 2, system 100 may further include a testing component 210 configured to run a set of functional tests over computer code 110 and determine, based at least partially on graph 130, for each code element and code element that depends on it, which functional test covered it, to yield a tests-elements coverage map 220. In addition, analyzing component 170 may be further configured to use tests-elements coverage map 220 in the process of generating ordered list 180. Alternatively, the dependency graph may be used by analyzing component 170 independently of the tests, in order to determine which code elements are related to which requirements or design specifications. In other words, the dependency graph is utilized in conjunction with the traceability techniques in order to achieve a more comprehensive mapping of code elements and design specifications or requirements.
  • According to some embodiments, system 100 may further include a scoring component 230 configured to apply a scoring function to design specifications or requirements-code tracing map 160 and the tests-elements coverage map 220, to yield a design specifications or requirements-coverage score 240 for each design portion or requirement, representing a level of code coverage by the functional tests, in design or requirement terms. Additionally, design specifications or requirements presenting lower design specifications or requirements-coverage score 240 are assigned by analyzing component 170 with a higher order in ordered list 180.
  • According to some embodiments, design specifications or requirements which present substantially a common score are reordered by analyzing component 170 in ordered list 180 according to the number of code elements and dependent code elements each design/requirement are addressing in their corresponding functional tests.
  • According to some embodiments, system 100 may further include a test generating component 250 configured to generate functional tests 260 based on the design specifications or requirements in the ordered list and their respective order. Functional tests 260 may be generated automatically by a tests generator, semi-automatically using a wizard, or manually, by a system engineer using ordered list 180 as guidelines for developing new tests.
  • FIG. 4 shows a high level schematic block diagram illustrating an aspect according to some embodiments of the invention. Specifically, it shows the relationship between code, tests and design specifications or requirements. Computer code 110 contains code elements (functions and/or classes) 112, 114, 116, and 118 is in operative association with functional tests 195 which include tests 122, 124, 126, and 128, and further in operative association with design specifications or requirements documents 170 containing design specifications or requirements 172, 174, 176, and 178. The diagram illustrates how each code element may be traced back to a respective design specification or requirement, in addition, how each code element may be associated with a respective test, and finally how the tracing and the association yield a mapping of tests to design specifications or requirements. For example, code element 112 may determined to be tested by test 122. In addition, same code element 112 may be traced back to design specification or requirement 172. Thus, in embodiments of the aforementioned method, test 122 may be mapped into design specification/requirement 172. In embodiments of the invention, test-code element coverage map 140 and design specifications or requirements-code coverage map 160 may be in the form of tables or any other form of data structure that enables a description of bi-directional connection of pairs.
  • FIG. 5 is a high level flowchart diagram illustrating a method 500 according to some embodiments of the invention. The method may include the following steps: building a dependencies graph representing dependencies between code elements of a computer code 510; associating portions of the computer code with corresponding design specifications or requirements derived from a design specifications document or a requirements document respectively which is associated with the computer code, to yield a design specifications or requirements-code tracing map 520; and analyzing the design specifications or requirements-code tracing map based on the dependencies graph to yield an ordered list of design specifications or requirements respectively, wherein the order is selected such that functional tests written for the computer code and addressing design specifications or requirements of a higher order, will yield a higher level of functional test coverage of the computer code in terms of design specifications or requirements 550.
  • In order to implement the aforementioned method, a computer (not shown) may receive instructions and data from a read-only memory or a random access memory or both. At least one of aforementioned steps is performed by at least one processor associated with a computer. The essential elements of a computer are a processor for executing instructions and one or more memories for storing instructions and data. Generally, a computer will also include, or be operatively coupled to communicate with, one or more mass storage devices for storing data files. Storage modules suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices and also magneto-optic storage devices.
  • According to some embodiments of the invention, method 500 may further include the steps of: running a set of functional tests over the computer code; and determining, based on the graph, for each code element and code element that depends on it, which functional test covered it, to yield a tests-elements coverage map 530, wherein in the analyzing, the tests-elements coverage map is further used to yield the ordered list.
  • According to some embodiments of the invention, method 500 may further include the step of applying a scoring function 540 to the design specifications or requirements-code tracing map and the tests-elements coverage map, to yield a design specifications or requirements-coverage score for each design portion or requirement, representing a level of code coverage by the functional tests, in design or requirement terms, wherein design or requirements presenting lower design specifications or requirements-coverage score are assigned with a higher order in the ordered list.
  • The following is a non-limiting example implementing the aforementioned scoring process. The scoring may include the following steps: initializing the coverage score for each design specification/requirement by ri=0, for each i. Then, for each design specification/requirement retrieved, counting the number of links pointing to this design specification/requirement, ci. Then, calculating the coverage score for a design specification/requirement using a normalization of the number of links pointing to that design specification/requirement, for example: ri=ci/Σ{cj}. Finally, the output of the scoring function may show the design specifications or requirements coverage scores {ri}.
  • According to some embodiments of the invention, the computer code has been generated, either by a human developer, automatically, or semi-automatically in order to meet the design specifications or requirements. At least some of the elements of the code that are the subject of the functional tests are functions and at least some of the tested elements are classes. The design specifications or requirements, in turn are usually written in a natural language, possibly by a system engineer.
  • FIG. 6 is a high level flowchart diagram illustrating an aspect relating to method 500 according to some embodiments of the invention. The aforementioned step of associating 520 may further include the following steps: segmenting the design or the requirements document into sections, each section having a single design specification or a single requirement 521; indexing the sections into a search engine 522; creating a query according to words obtained from the code, wherein the words are ranked according to their respective semantic meaning, for each segment of the computer code 523; and retrieving respective requirements or design specifications by applying the search engine to each one of respective queries 524.
  • The following is a more detailed embodiment of the traceability process. The traceability process starts with segmentation of the design requirements document into sections, wherein each section comprises a single design portion or requirement. The segmentation may be followed by pre-processing each section using various techniques. Non-limiting examples of such techniques may include filter stop words (frequent words) and stemming. After pre-processing an indexing of all design specifications or requirements segments into a search engine is performed. Then, given a code segment, the following process is performed in order to find the design specification(s)/requirement(s) from which that section follows from: first, the given code is preprocessed using a pre-processing technique such as the aforementioned techniques. Then, words concatenated, for example, by CamelCase or underlines are separated. Subsequently, a query is created from the code. For example, for a Java class the following information may be used as words in creating the query: class name, public function names, class comments, public function arguments and return types, super class names and implemented interfaces, and public function comments. Then words are being ranked according to the semantic meaning of their respective source. Thus for example, a class name will be ranked higher than a comment. Finally, the query is being entered into the search engine to retrieve the related design specifications or requirements.
  • According to some embodiments of the invention, another sub process is provided for improving the precision of the coverage score of functions. This may be achieved by generating a wider query which combines terms from both the tested elements and the elements that the tested elements depend on according to the dependencies graph.
  • As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
  • Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in base band or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
  • Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire-line, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
  • Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
  • Aspects of the present invention are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
  • The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • The aforementioned flowchart and diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
  • In the above description, an embodiment is an example or implementation of the inventions. The various appearances of “one embodiment,” “an embodiment” or “some embodiments” do not necessarily all refer to the same embodiments.
  • Although various features of the invention may be described in the context of a single embodiment, the features may also be provided separately or in any suitable combination. Conversely, although the invention may be described herein in the context of separate embodiments for clarity, the invention may also be implemented in a single embodiment.
  • Reference in the specification to “some embodiments”, “an embodiment”, “one embodiment” or “other embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments, of the inventions. It is to be understood that the phraseology and terminology employed herein is not to be construed as limiting and are for descriptive purpose only.
  • The principles and uses of the teachings of the present invention may be better understood with reference to the accompanying description, figures and examples.
  • It is to be understood that the details set forth herein do not construe a limitation to an application of the invention.
  • Furthermore, it is to be understood that the invention can be carried out or practiced in various ways and that the invention can be implemented in embodiments other than the ones outlined in the description above.
  • It is to be understood that the terms “including”, “comprising”, “consisting” and grammatical variants thereof do not preclude the addition of one or more components, features, steps, or integers or groups thereof and that the terms are to be construed as specifying components, features, steps or integers.
  • If the specification or claims refer to “an additional” element, that does not preclude there being more than one of the additional element.
  • It is to be understood that where the claims or specification refer to “a” or “an” element, such reference is not be construed that there is only one of that element.
  • It is to be understood that where the specification states that a component, feature, structure, or characteristic “may”, “might”, “can” or “could” be included, that particular component, feature, structure, or characteristic is not required to be included.
  • Where applicable, although state diagrams, flow diagrams or both may be used to describe embodiments, the invention is not limited to those diagrams or to the corresponding descriptions. For example, flow need not move through each illustrated box or state, or in exactly the same order as illustrated and described.
  • Methods of the present invention may be implemented by performing or completing manually, automatically, or a combination thereof, selected steps or tasks.
  • The term “method” may refer to manners, means, techniques and procedures for accomplishing a given task including, but not limited to, those manners, means, techniques and procedures either known to, or readily developed from known manners, means, techniques and procedures by practitioners of the art to which the invention belongs.
  • The descriptions, examples, methods and materials presented in the claims and the specification are not to be construed as limiting but rather as illustrative only.
  • Meanings of technical and scientific terms used herein are to be commonly understood as by one of ordinary skill in the art to which the invention belongs, unless otherwise defined.
  • The present invention may be implemented in the testing or practice with methods and materials equivalent or similar to those described herein.
  • Any publications, including patents, patent applications and articles, referenced or mentioned in this specification are herein incorporated in their entirety into the specification, to the same extent as if each individual publication was specifically and individually indicated to be incorporated herein. In addition, citation or identification of any reference in the description of some embodiments of the invention shall not be construed as an admission that such reference is available as prior art to the present invention.
  • While the invention has been described with respect to a limited number of embodiments, these should not be construed as limitations on the scope of the invention, but rather as exemplifications of some of the preferred embodiments. Other possible variations, modifications, and applications are also within the scope of the invention. Accordingly, the scope of the invention should not be limited by what has thus far been described, but by the appended claims and their legal equivalents.

Claims (21)

1. A method comprising:
building a dependencies graph representing dependencies between code elements of a computer code;
associating portions of the computer code with corresponding design specifications or requirements derived from a design document or a requirements document respectively which is associated with the computer code, to yield a design specifications or requirements-code tracing map; and
analyzing the design specifications or requirements-code tracing map based at least partially on the dependencies graph to yield an ordered list of design specifications or requirements respectively, wherein the order is selected such that functional tests written for the computer code and addressing design specifications or requirements of a higher order, will yield a higher level of functional test coverage of the computer code in terms of design specifications or requirements,
wherein at least one of the building, the associating, and the analyzing is executed by at least one processor.
2. The method according to claim 1, further comprising generating functional tests based on the design specifications or requirements in the ordered list and their respective order.
3. The method according to claim 1, wherein the associating comprises:
segmenting the design or the requirements document into sections, each section having a single design portion or a single requirement;
indexing the sections into a search engine;
creating a query according to words obtained from the code, wherein the words are ranked according to their respective semantic meaning, for each segment of the computer code; and
retrieving respective requirements or design specifications by applying the search engine to each one of respective queries.
4. The method according to claim 1, further comprising:
running a set of functional tests over the computer code;
determining, based on the graph, for each code element and code element that depends on it, which functional test covered it, to yield a tests-elements coverage map,
wherein in the analyzing, the tests-elements coverage map is further used to yield the ordered list.
5. The method according to claim 4, further comprising applying a scoring function to the design specifications or requirements-code tracing map and the tests-elements coverage map, to yield a design specifications or requirements-coverage score for each design portion or requirement, representing a level of code coverage by the functional tests, in design or requirement terms, wherein design or requirements presenting lower design specifications or requirements-coverage score are assigned with a higher order in the ordered list.
6. The method according to claim 5, wherein design specifications or requirements presenting substantially a common score are reordered in the ordered list according to a number of code elements and dependent code elements each design/requirement are addressing in their corresponding functional tests.
7. The method according to claim 5, further comprising generating functional tests based on the order of the ordered list of design specifications or requirements.
8. A system comprising:
a builder component configured to build a dependencies graph representing dependencies between code elements of a computer code;
an association component configured to associate portions of the computer code with corresponding design specifications or requirements derived from a design document or a requirements document respectively which is associated with the computer code, to yield a design specifications or requirements-code tracing map; and
an analyzing component configured to analyze the design specifications or requirements-code tracing map based at least partially on the dependencies graph to yield an ordered list of design specifications or requirements respectively, wherein the order is selected such that functional tests written for the computer code and addressing design specifications or requirements of a higher order, will yield a higher level of functional test coverage of the computer code in terms of design specifications or requirements,
wherein at least one of the components is executed by at least one processor.
9. The system according to claim 8, further comprising a test generating component configured to generate functional tests based on the design specifications or requirements in the ordered list and their respective order.
10. The system according to claim 8, wherein the associating components is further configured to:
segment the design or the requirements document into sections, each section having a single design portion or a single requirement;
index the sections into a search engine;
create a query according to words obtained from the code, wherein the words are ranked according to their respective semantic meaning, for each segment of the computer code; and
retrieve respective requirements or design specifications by applying the search engine to each one of respective queries.
11. The system according to claim 8, further comprising:
A testing component configured to run a set of functional tests over the computer code and determine, based on the graph, for each code element and code element that depends on it, which functional test covered it, to yield a tests-elements coverage map,
wherein the analyzing component is further configured to use the tests-elements coverage map to yield the ordered list.
12. The system according to claim 11, further comprising a scoring component configured to apply a scoring function to the design specifications or requirements-code tracing map and the tests-elements coverage map, to yield a design specifications or requirements-coverage score for each design portion or requirement, representing a level of code coverage by the functional tests, in design or requirement terms, wherein design or requirements presenting lower design specifications or requirements-coverage score are assigned with a higher order in the ordered list.
13. The system according to claim 12, wherein design specifications or requirements presenting substantially a common score are reordered in the ordered list according to a number of code elements and dependent code elements each design/requirement are addressing in their corresponding functional tests.
14. The system according to claim 13, further comprising a test generating component configured to generate functional tests based on the design specifications or requirements in the ordered list and their respective order.
15. A computer program product, the computer program product comprising:
a computer readable storage medium having computer readable program embodied therewith, the computer readable program comprising:
computer readable program configured to build a dependencies graph representing dependencies between code elements of a computer code;
computer readable program configured to associate portions of the computer code with corresponding design specifications or requirements derived from a design document or a requirements document respectively which is associated with the computer code, to yield a design specifications or requirements-code tracing map; and
computer readable program configured to analyze the design specifications or requirements-code tracing map based at least partially on the dependencies graph to yield an ordered list of design specifications or requirements respectively, wherein the order is selected such that functional tests written for the computer code and addressing design specifications or requirements of a higher order, will yield a higher level of functional test coverage of the computer code in terms of design specifications or requirements.
16. The computer program product according to claim 15, further comprising computer readable program configured to generate functional tests based on the design specifications or requirements in the ordered list and their respective order.
17. The computer program product according to claim 15, wherein the computer readable program configure to associate, is further configured to:
segment the design or the requirements document into sections, each section having a single design portion or a single requirement;
index the sections into a search engine;
create a query according to words obtained from the code, wherein the words are ranked according to their respective semantic meaning, for each segment of the computer code; and
retrieve respective requirements or design specifications by applying the search engine to each one of respective queries.
18. The computer program product according to claim 15, further comprising:
computer readable program configured to run a set of functional tests over the computer code and determine, based on the graph, for each code element and code element that depends on it, which functional test covered it, to yield a tests-elements coverage map,
wherein the computer readable program configured to analyze is further configured to use the tests-elements coverage map to yield the ordered list.
19. The computer program product according to claim 18, further comprising computer readable program configured to apply a scoring function to the design specifications or requirements-code tracing map and the tests-elements coverage map, to yield a design specifications or requirements-coverage score for each design portion or requirement, representing a level of code coverage by the functional tests, in design or requirement terms, wherein design specifications or requirements presenting lower design specifications or requirements-coverage score are assigned with a higher order in the ordered list.
20. The computer program product according to claim 19, wherein design specifications or requirements presenting substantially a common score are reordered in the ordered list according to a number of code elements and dependent code elements each design specification/requirement are addressing in their corresponding functional tests.
21. The computer program product according to claim 20, further comprising computer readable program configured to generate functional tests based on the design specifications or requirements in the ordered list and their respective order.
US12/768,764 2010-04-28 2010-04-28 Enhancing functional tests coverage using traceability and static analysis Abandoned US20110271253A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/768,764 US20110271253A1 (en) 2010-04-28 2010-04-28 Enhancing functional tests coverage using traceability and static analysis

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US12/768,764 US20110271253A1 (en) 2010-04-28 2010-04-28 Enhancing functional tests coverage using traceability and static analysis
US13/674,063 US8954936B2 (en) 2010-04-28 2012-11-11 Enhancing functional tests coverage using traceability and static analysis

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US13/674,063 Continuation US8954936B2 (en) 2010-04-28 2012-11-11 Enhancing functional tests coverage using traceability and static analysis

Publications (1)

Publication Number Publication Date
US20110271253A1 true US20110271253A1 (en) 2011-11-03

Family

ID=44859348

Family Applications (2)

Application Number Title Priority Date Filing Date
US12/768,764 Abandoned US20110271253A1 (en) 2010-04-28 2010-04-28 Enhancing functional tests coverage using traceability and static analysis
US13/674,063 Expired - Fee Related US8954936B2 (en) 2010-04-28 2012-11-11 Enhancing functional tests coverage using traceability and static analysis

Family Applications After (1)

Application Number Title Priority Date Filing Date
US13/674,063 Expired - Fee Related US8954936B2 (en) 2010-04-28 2012-11-11 Enhancing functional tests coverage using traceability and static analysis

Country Status (1)

Country Link
US (2) US20110271253A1 (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130232472A1 (en) * 2010-11-09 2013-09-05 Christian Körner Method and Apparatus for the Determination of a Quality Assessment of a Software Code with Determination of the Assessment Coverage
US20140189874A1 (en) * 2012-12-31 2014-07-03 International Business Machines Corporation Hybrid analysis of vulnerable information flows
US9292419B1 (en) * 2013-06-04 2016-03-22 The Mathworks, Inc. Code coverage and confidence determination
US9558101B2 (en) * 2014-08-08 2017-01-31 Raytheon Company Preprocessor directive symbol analyzer devices and methods
US9582270B2 (en) 2015-04-30 2017-02-28 International Business Machines Corporation Effective feature location in large legacy systems
US9606815B2 (en) * 2015-02-26 2017-03-28 International Business Machines Corporation API prioritization and detection
US9747194B2 (en) * 2012-10-30 2017-08-29 Ebay Inc. Systems and methods to maintain consistency between software specification and code

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPWO2013058393A1 (en) * 2011-10-18 2015-04-02 日本電気株式会社 Anomaly detection device, program, and method
US10216620B2 (en) * 2016-07-05 2019-02-26 Synopsys, Inc. Static code testing of active code

Citations (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030028856A1 (en) * 2001-08-01 2003-02-06 Apuzzo Joseph T. Method and apparatus for testing a software component using an abstraction matrix
US6698012B1 (en) * 1999-09-17 2004-02-24 Nortel Networks Limited Method and system for testing behavior of procedures
US6721941B1 (en) * 1996-08-27 2004-04-13 Compuware Corporation Collection of timing and coverage data through a debugging interface
US20040103400A1 (en) * 2002-11-22 2004-05-27 Johnsen John M. Recovery from corruption using event offset format in data trace
US7080358B2 (en) * 2002-04-10 2006-07-18 Sun Microsystems, Inc. Mechanism for generating an execution log and coverage data for a set of computer code
US20070250815A1 (en) * 2006-04-21 2007-10-25 Microsoft Corporation Measuring code coverage
US20080155508A1 (en) * 2006-12-13 2008-06-26 Infosys Technologies Ltd. Evaluating programmer efficiency in maintaining software systems
US20080163178A1 (en) * 2006-12-29 2008-07-03 Ivanova Gorka J System and method for displaying component information of a trace
US7426716B2 (en) * 2003-07-11 2008-09-16 Board Of Regents, The University Of Texas System Recovery and representation of object interaction in an object oriented program
US20080235666A1 (en) * 2004-09-09 2008-09-25 International Business Machines Corporation Generating sequence diagrams using call trees
US20080263505A1 (en) * 2007-04-03 2008-10-23 Ldra Technology, Inc. Automated management of software requirements verification
US20080281841A1 (en) * 2003-09-12 2008-11-13 Kishore Swaminathan Navigating a software project respository
US20090199160A1 (en) * 2008-01-31 2009-08-06 Yahoo! Inc. Centralized system for analyzing software performance metrics
US20090254887A1 (en) * 2008-04-02 2009-10-08 International Business Machines Corporation Testing Software Applications with Progress Tracking
US20090287729A1 (en) * 2008-05-16 2009-11-19 Microsoft Corporation Source code coverage testing
US20090300585A1 (en) * 2004-03-15 2009-12-03 Ramco Systems Limited Method and system for testing a software development activity
US20100005455A1 (en) * 2008-07-03 2010-01-07 International Business Machines Corporation Managing software dependencies during software testing and debugging
US20100037209A1 (en) * 2007-04-09 2010-02-11 Fujitsu Limited Source program review program, source program review method, and source program review device
US20110138358A1 (en) * 2009-12-09 2011-06-09 Stefan Rau Systems and methods for analyzing test coverage at an organizational level
US20110145788A1 (en) * 2009-12-10 2011-06-16 Sap Ag Bridging code changes and testing
US8037453B1 (en) * 2006-09-13 2011-10-11 Urbancode, Inc. System and method for continuous software configuration, test and build management
US8042098B2 (en) * 2000-12-06 2011-10-18 Axiomatic Design Software, Inc. Method and apparatus for producing software
US8250536B2 (en) * 2007-12-24 2012-08-21 Hewlett-Packard Development Company, L.P. Analysis of a legacy source code application
US8490067B2 (en) * 2008-05-22 2013-07-16 National Instruments Corporation Graphical program code coverage

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7114111B2 (en) 1999-06-08 2006-09-26 Cadence Design (Isreal) Ii Ltd. Method and apparatus for maximizing test coverage
US6725399B1 (en) 1999-07-15 2004-04-20 Compuware Corporation Requirements based software testing method
US8104017B2 (en) 2001-10-25 2012-01-24 The Mathworks, Inc. Traceability in a modeling environment
US7299458B2 (en) * 2002-10-31 2007-11-20 Src Computers, Inc. System and method for converting control flow graph representations to control-dataflow graph representations
US7228524B2 (en) 2002-12-20 2007-06-05 The Boeing Company Method and system for analysis of software requirements
US7600221B1 (en) * 2003-10-06 2009-10-06 Sun Microsystems, Inc. Methods and apparatus of an architecture supporting execution of instructions in parallel
US7490319B2 (en) 2003-11-04 2009-02-10 Kimberly-Clark Worldwide, Inc. Testing tool comprising an automated multidimensional traceability matrix for implementing and validating complex software systems
JP2006185211A (en) * 2004-12-28 2006-07-13 Internatl Business Mach Corp <Ibm> Program analysis system, test execution device, and analysis method and program thereof
US20060168566A1 (en) * 2005-01-25 2006-07-27 Microsoft Corporation Pattern matching documentation
US20060225049A1 (en) * 2005-03-17 2006-10-05 Zhiyuan Lv Trace based signal scheduling and compensation code generation
US7603660B2 (en) * 2005-06-01 2009-10-13 Microsoft Corporation Code coverage test selection
US20090019427A1 (en) 2007-07-13 2009-01-15 International Business Machines Corporation Method and Apparatus for Providing Requirement Driven Static Analysis of Test Coverage for Web-Based, Distributed Processes
US8495574B2 (en) * 2008-06-16 2013-07-23 International Business Machines Corporation Code coverage tool
US8806450B1 (en) * 2008-06-26 2014-08-12 Juniper Networks, Inc. Static analysis in selective software regression testing
US8347272B2 (en) * 2008-07-23 2013-01-01 International Business Machines Corporation Call graph dependency extraction by static source code analysis
US8601446B2 (en) * 2008-10-22 2013-12-03 Wipro Limited Complexity dependency change impact system and method
US9015665B2 (en) * 2008-11-11 2015-04-21 International Business Machines Corporation Generating functional artifacts from low level design diagrams
US8307351B2 (en) * 2009-03-18 2012-11-06 Oracle International Corporation System and method for performing code provenance review in a software due diligence system

Patent Citations (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6721941B1 (en) * 1996-08-27 2004-04-13 Compuware Corporation Collection of timing and coverage data through a debugging interface
US6698012B1 (en) * 1999-09-17 2004-02-24 Nortel Networks Limited Method and system for testing behavior of procedures
US8042098B2 (en) * 2000-12-06 2011-10-18 Axiomatic Design Software, Inc. Method and apparatus for producing software
US20030028856A1 (en) * 2001-08-01 2003-02-06 Apuzzo Joseph T. Method and apparatus for testing a software component using an abstraction matrix
US7080358B2 (en) * 2002-04-10 2006-07-18 Sun Microsystems, Inc. Mechanism for generating an execution log and coverage data for a set of computer code
US20040103400A1 (en) * 2002-11-22 2004-05-27 Johnsen John M. Recovery from corruption using event offset format in data trace
US7426716B2 (en) * 2003-07-11 2008-09-16 Board Of Regents, The University Of Texas System Recovery and representation of object interaction in an object oriented program
US20080281841A1 (en) * 2003-09-12 2008-11-13 Kishore Swaminathan Navigating a software project respository
US20090300585A1 (en) * 2004-03-15 2009-12-03 Ramco Systems Limited Method and system for testing a software development activity
US20080235666A1 (en) * 2004-09-09 2008-09-25 International Business Machines Corporation Generating sequence diagrams using call trees
US20070250815A1 (en) * 2006-04-21 2007-10-25 Microsoft Corporation Measuring code coverage
US8037453B1 (en) * 2006-09-13 2011-10-11 Urbancode, Inc. System and method for continuous software configuration, test and build management
US20080155508A1 (en) * 2006-12-13 2008-06-26 Infosys Technologies Ltd. Evaluating programmer efficiency in maintaining software systems
US20080163178A1 (en) * 2006-12-29 2008-07-03 Ivanova Gorka J System and method for displaying component information of a trace
US20080263505A1 (en) * 2007-04-03 2008-10-23 Ldra Technology, Inc. Automated management of software requirements verification
US20100037209A1 (en) * 2007-04-09 2010-02-11 Fujitsu Limited Source program review program, source program review method, and source program review device
US8250536B2 (en) * 2007-12-24 2012-08-21 Hewlett-Packard Development Company, L.P. Analysis of a legacy source code application
US20090199160A1 (en) * 2008-01-31 2009-08-06 Yahoo! Inc. Centralized system for analyzing software performance metrics
US20090254887A1 (en) * 2008-04-02 2009-10-08 International Business Machines Corporation Testing Software Applications with Progress Tracking
US20090287729A1 (en) * 2008-05-16 2009-11-19 Microsoft Corporation Source code coverage testing
US8490067B2 (en) * 2008-05-22 2013-07-16 National Instruments Corporation Graphical program code coverage
US20100005455A1 (en) * 2008-07-03 2010-01-07 International Business Machines Corporation Managing software dependencies during software testing and debugging
US20110138358A1 (en) * 2009-12-09 2011-06-09 Stefan Rau Systems and methods for analyzing test coverage at an organizational level
US20110145788A1 (en) * 2009-12-10 2011-06-16 Sap Ag Bridging code changes and testing

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Constance Heitmeyer, Tools for Formal Specification, Verification, and Validation of Requirements, 1997, pages 2-11 *
H. Ural, Test generation based on control and data dependencies within system specification in SDL, 2000, pages 610-615. *
Vaclav Rajlich, The Role of Concepts in Program Comprehension, 2002, pages 272-276. *

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130232472A1 (en) * 2010-11-09 2013-09-05 Christian Körner Method and Apparatus for the Determination of a Quality Assessment of a Software Code with Determination of the Assessment Coverage
US9311218B2 (en) * 2010-11-09 2016-04-12 Siemens Aktiengesellschaft Method and apparatus for the determination of a quality assessment of a software code with determination of the assessment coverage
US9747194B2 (en) * 2012-10-30 2017-08-29 Ebay Inc. Systems and methods to maintain consistency between software specification and code
US20140189874A1 (en) * 2012-12-31 2014-07-03 International Business Machines Corporation Hybrid analysis of vulnerable information flows
US20140189875A1 (en) * 2012-12-31 2014-07-03 International Business Machines Corporation Hybrid analysis of vulnerable information flows
US8869287B2 (en) * 2012-12-31 2014-10-21 International Business Machines Corporation Hybrid analysis of vulnerable information flows
US9177155B2 (en) * 2012-12-31 2015-11-03 International Business Machines Corporation Hybrid analysis of vulnerable information flows
US9292419B1 (en) * 2013-06-04 2016-03-22 The Mathworks, Inc. Code coverage and confidence determination
US9558101B2 (en) * 2014-08-08 2017-01-31 Raytheon Company Preprocessor directive symbol analyzer devices and methods
US9606815B2 (en) * 2015-02-26 2017-03-28 International Business Machines Corporation API prioritization and detection
US9582270B2 (en) 2015-04-30 2017-02-28 International Business Machines Corporation Effective feature location in large legacy systems

Also Published As

Publication number Publication date
US20130067436A1 (en) 2013-03-14
US8954936B2 (en) 2015-02-10

Similar Documents

Publication Publication Date Title
Gyimothy et al. Empirical validation of object-oriented metrics on open source software for fault prediction
Gethers et al. Using relational topic models to capture coupling among classes in object-oriented software systems
US20140229922A1 (en) Efficient model checking technique for finding software defects
Gligoric et al. Practical regression test selection with dynamic file dependencies
Zhang et al. Automated diagnosis of software configuration errors
Walden et al. Predicting vulnerable components: Software metrics vs text mining
CN102147764B (en) Test code quality assessment
US20080295085A1 (en) Integrated code review tool
JP5652326B2 (en) Software module testing method and system
US9317401B2 (en) Prioritizing test cases using multiple variables
US20070169027A1 (en) Methods and systems for complete static analysis of software for building a system
CN101473301A (en) Iterative static and dynamic software analysis
US9569343B2 (en) Integration of a software content space with test planning and test case generation
US20130111267A1 (en) Optimizing regression testing based on code coverage analysis
Lakhotia et al. An empirical investigation into branch coverage for C programs using CUTE and AUSTIN
Le Goues et al. Specification mining with few false positives
Krogmann et al. Using genetic search for reverse engineering of parametric behavior models for performance prediction
US8381194B2 (en) Methods and apparatuses for selective code coverage
Pate et al. Clone evolution: a systematic review
US9898280B2 (en) Automatic code review and code reviewer recommendation
US20130132933A1 (en) Automated compliance testing during application development
US9298453B2 (en) Source code analytics platform using program analysis and information retrieval
US8943487B2 (en) Optimizing libraries for validating C++ programs using symbolic execution
Calegari et al. Verification of model transformations: A survey of the state-of-the-art
Barmi et al. Alignment of requirements specification and testing: A systematic mapping study

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BNAYAHU, JONATHAN;GOLDSTEIN, MAAYAN;MOSHKOVICH, DANY;AND OTHERS;SIGNING DATES FROM 20100425 TO 20100428;REEL/FRAME:024321/0655

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE