US20150026664A1 - Method and system for automated test case selection - Google Patents

Method and system for automated test case selection Download PDF

Info

Publication number
US20150026664A1
US20150026664A1 US13/944,012 US201313944012A US2015026664A1 US 20150026664 A1 US20150026664 A1 US 20150026664A1 US 201313944012 A US201313944012 A US 201313944012A US 2015026664 A1 US2015026664 A1 US 2015026664A1
Authority
US
United States
Prior art keywords
test
test cases
software
code
metrics
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US13/944,012
Inventor
Timothy S. Bartley
Gavin G. Bray
Elizabeth M. Hughes
Kalvinder P. Singh
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Priority to US13/944,012 priority Critical patent/US20150026664A1/en
Assigned to INTERNATIONAL BUSINESS MACHINES CORPORATION reassignment INTERNATIONAL BUSINESS MACHINES CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: BARTLEY, TIM S, BRAY, GAVIN G, HUGHES, LIZ M, SINGH, KAL P
Publication of US20150026664A1 publication Critical patent/US20150026664A1/en
Application status is Abandoned legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/3668Software testing
    • G06F11/3672Test management
    • G06F11/3676Test management for coverage analysis

Abstract

A computer-implemented method, computer program product, and computer system for intellectually and automatically selecting test cases for testing software that has been changed. In this invention, the automated selection of the subset of the test cases is based on determination of what software under test has been changed, what test cases have exercised these changes, what test data has been used to exercise these changes, what test environment including hardware and software configuration has been used to test these changes, and what pre-requisite test cases have been run prior to having the software under test in the correct state.

Description

    FIELD OF THE INVENTION
  • The present invention relates generally to software testing, and more particularly to a system and method for automatically selecting test cases for testing software that has been changed.
  • BACKGROUND
  • Best practice of software engineering mandates that software should be thoroughly tested before the software is released. Automated software testing is the most cost effective approach. The automated software testing can involve hundreds to thousands of test cases, and each of the test cases includes a combination of test code, test data, and test configuration required to execute the automated software testing. The each of the test cases tests some aspects of software under test.
  • When changes are made to the software, testing the software with the changes is an important but difficult task, especially as the complexity of the software under test and the number of the test cases increase. A first existing solution is to run all the test cases. Running all the test cases to test the software with the changes is not feasible, due to time and resource constraints. In addition, in the first existing solution, feedback to a development team is delayed. A second existing solution is to manually select a subset of the test cases. The second existing solution requires testers to identify some cases for testing the software with the changes. The second existing solution is time consuming for the testers and prone to errors. A third existing solution is to select test cases based on code coverage metrics only and thus is only a partial solution.
  • BRIEF SUMMARY
  • Embodiments of the present invention provide a computer-implemented method, computer program product, and computer system for selecting test cases for testing software that has been changed. The computer system executes one or more test cases with one or more test case input data, one or more test environments, and one or more prerequisite test cases. The computer system generates, for the software, code coverage metrics which describes what code of the software has been executed. The computer system generates, for the software, code change metrics which describes what changes to the software have been made. Based on a correlation between the code coverage metrics and the code change metrics, the computer system determines the changes to the software. From the one or more test cases, the computer system selects test cases corresponding to the changes.
  • BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
  • FIG. 1 is a diagram illustrating an exemplary system for automatically selecting test cases for testing software under test, in accordance with an exemplary embodiment of the present invention.
  • FIG. 2 is a flowchart illustrating operational steps of a test case selection program shown in FIG. 1, in accordance with an exemplary embodiment of the present invention.
  • FIG. 3 is a diagram illustrating components of a computing device hosting the exemplary system shown in FIG. 1, in accordance with an exemplary embodiment of the present invention.
  • DETAILED DESCRIPTION
  • The present invention describes a method and system for intelligently and automatically identifying a subset of test cases for testing software that has been changed. The automated selection of the subset of the test cases is based on determination of what software under test has been changed, what test cases have exercised these changes, what test data has been used to exercise these changes, what test environment including hardware and software configuration has been used to test these changes, and what pre-requisite test cases have been run prior to having the software under test in the correct state. The determination of those mentioned above is automated. The advantages of the present invention include error free, more efficient use of time and resources, fast and more relevant feedback to a development team, and taking account of dependencies in the test data, the test environment, and prerequisite test cases.
  • FIG. 1 is a diagram illustrating exemplary system 100 for automatically selecting test cases for testing software under test 110, in accordance with an exemplary embodiment of the present invention. Exemplary system 100 includes test case selection program 120. Test case selection program 120 identifies test cases for testing software under test 110 that has been changed since the last testing. Test case selection program 120 selects a subset of test cases from test cases 131 on database 130. The subset of test cases are selected to correspond to changes made to software under test 110 and are to be executed for testing software under test 110 that has been changed. Database 130 includes code coverage metrics 135 for software under test 110. To generate code coverage metrics 135 for software under test 110, test case selection program 120 executes test cases chosen from test cases 131 on database 130 against software under test 110. Before executing the test cases chosen from test cases 131, test case selection program 120 runs prerequisite test case(s) chosen from prerequisite test cases 134 on database 130 to establish a correct initial state of software under test 110. Test case selection program 120 executes the test cases chosen from test cases 131 with the test case input data chosen from test case input data 132 on database 130. Test case selection program 120 executes the test cases with different test case input data selected from test case input data 132; execution with the different test case input data may produce quite distinct test case scenarios even though the test case code is the same. Test case selection program 120 executes the test cases within different test environments chosen from test environments 133 on database 130. Each test environment may produce quite distinct test case scenarios even though the test case code is the same. A test environment is a setup of software and hardware on which software under test 110 is to be tested.
  • Referring to FIG. 1, on database 130, code coverage metrics 135 for software under test 110 describes what code of software under test 110 has been executed. The levels of granularity of code coverage metrics 135 include methods, statements, and condition coverage. Code coverage metrics 135 is sometimes used to formulate test cases 131. For example, if code coverage metrics 135 indicates a certain function hasn't been executed, then some test cases to execute the function may be developed in test cases 131. Test case selection program 120 maps code coverage metrics 135 to a combination of test cases 131, test case input data 132, test environments 133, and prerequisite test cases 134. In the exemplary embodiment, based on code coverage metrics 135, test case selection program 120 determines code coverage information. The level of granularity of the code coverage metrics determines the precision of the code coverage information. The code coverage information is listed as follows. (1) For a test case chosen from test cases 131, sections of software under test 110 exercised by the test case are listed. The test case is exercised with its related test case input data chosen from test case input data 132, its test environment(s) chosen from test environments 133, and its related prerequisite test case(s) chosen from prerequisite test cases 134. For example, if the level of granularity is method coverage, then methods in software under test 110 exercised by the test case are listed. (2) For an identified section of software under test 110, an exercise is done by certain test cases chosen from test cases 131 with certain test case input data chosen from test case input data 132, test environments chosen from test environments 133, and certain prerequisite test cases chosen from prerequisite test cases 134. For the identified section, the exercise is listed. For example, if the level of granularity is method coverage, the exercise of the method is listed. (3) Under a test without a test case, sections of software under test 110 are listed for at least one of the following: certain test case input data chosen from test case input data 132, certain test environments chosen from test environments 133, and certain prerequisite test cases chosen from prerequisite test cases 134. (4) For a case in which sections of software under test 110 are exercised by more than one test case chosen from test cases 131, the sections are listed for duplicated or overlapping test cases with their related test case input data chosen from test case input data 132, their test environment(s) chosen from test environments 133, and their related prerequisite test case(s) chosen from prerequisite test cases 134. Having executed the test cases chosen from test cases 131, test case selection program 120 updates, on database 130, code coverage metrics 135 which is mapped to the combination of test cases 131, test case input data 132, test environments 133, and prerequisite test cases 134.
  • Referring to FIG. 1, database 130 includes code change metrics 136. Code change metrics 136 describes what changes to source code, configuration, and other associated information of software under test 110 have been made. The levels of granularity of code change metrics 136 include methods, statements, and condition coverage. Test case selection program 120 generates code change metrics 136 on database 130. Using code change metrics 136, test case selection program 120 determines the following information. The sections of software under test 110 that has been changed within a certain time period are listed. For example, if the level of granularity is method coverage, then methods in software under test 110 that has been changed in the last 24 hours are listed. Based on code change metrics 136, test case selection program 120 determines changes to software under test 110. By correlating code coverage metrics 135 and code change metrics 136, test case selection program 120 determines a subset of test cases 131; the subset includes test cases corresponding to the changes. The test cases of the subset are to be run for testing software under test 110 that has been changed.
  • FIG. 2 is flowchart 200 illustrating operational steps of test case selection program 120 shown in FIG. 1, in accordance with an exemplary embodiment of the present invention. In the exemplary embodiment, test case selection program 120 is run for automatically selecting a subset of test cases 131 (shown in FIG. 1) for software under test 110 (shown in FIG. 1) that has been changed. In the exemplary embodiment, test case selection program 120 is hosted on a computer device shown in FIG. 3.
  • At step 201, test case selection program 120 instruments software under test 110 that has been changed since the last testing. Software under test 110 is instrumented so that code coverage metrics 135 on database 130 (shown in FIG. 1) can be generated. At step 203, test case selection program 120 executes one or more prerequisite test cases chosen from prerequisite test cases 134 on database 130 (shown in FIG. 1). Executing the one or more prerequisite test cases, test case selection program 120 establishes a correct initial state of software under test 110. At step 205, test case selection program 120 executes test cases chosen from test cases 131 on database 130 (shown in FIG. 1). Each of the test cases is executed with its test case input data chosen from test case input data 132 on database 130 (shown in FIG. 1) and under its one or more test environments chosen form test environments 133 on database 130 (shown in FIG. 1).
  • At step 207, test case selection program 120 generates code coverage metrics 135 for software under test 110. Code coverage metrics 135 describes what code of software under test 110 has been executed. The levels of granularity of code coverage metrics 135 include methods, statements, and condition coverage. Numerous tools are available for test case selection program 120 to collect the information of code coverage, such as Rational® Purify® which is a dynamic software analysis tool developed by International Business Machines Corporation (IBM®) and is supported on Windows®, Linux®, Solaris®, and AIX®. In addition, software under test 120 may use static analysis tools to determine dependencies in software under test 110.
  • At step 209, based on code coverage metrics 135 generated at step 207, test case selection program 120 identifies, in software under test 110, sections exercised by the test cases executed at step 205. Through this step, the code coverage information is determined. The code coverage information is listed in a previous paragraph in this document.
  • At step 211, test case selection program 120 maps code coverage metrics 135 to a combination of test cases 131, test case input data 132, test environments 133, and prerequisite test cases 134.
  • At step 213, test case selection program 120 generates code change metrics 136 (shown in FIG. 1) for software under test 110. Code change metrics 136 is on database 130 and describes what changes have been made to source code, configuration, and other associated information of software under test 110. To generate code change metrics 136, test case selection program 120 uses configuration management tools and version control systems. Test case selection program 120 may additionally use the dependencies determined by static analysis tools at step 207 to generate code change metrics 136. For example, if a library method that software under test 110 depends on has been changed, test case selection program 120 includes, in code change metrics 136, all code calling the library method in software under test 110.
  • At step 215, test case selection program 120 updates database 130 including code coverage metrics 135 mapped to the combination of test cases 131, test case input data 132, test environments 133, and prerequisite test cases 134.
  • At step 217, based on code change metrics 136 generated at step 213, test case selection program 120 determines changes to software under test 110. Given a time and date, sections of software under test 110 that has been changed in the intervening time period are listed. For example, if the level of granularity is method coverage, then the method in software under test 110 that has been changed in the last 24 hours is listed.
  • At step 219, test case selection program 120 selects, from database 130 updated at step 215, test cases corresponding to the changes determined at step 217. At this step, selecting the test cases corresponding to the changes is based on correlation between code coverage metrics 135 and code change metrics 136. The test cases corresponding to the changes are selected as a subset of test cases 131 on database 130. The test cases corresponding to the changes are to be run for testing software under test 110 that has been changed.
  • FIG. 3 is a diagram illustrating components of computing device 300 hosting the exemplary system shown in FIG. 1, in accordance with an exemplary embodiment of the present invention. It should be appreciated that FIG. 3 provides only an illustration of one implementation and does not imply any limitations with regard to the environment in which different embodiments may be implemented.
  • Referring to FIG. 3, computing device 300 includes processor(s) 320, memory 310, tangible storage device(s) 330, network interface(s) 340, and I/O (input/output) interface(s) 350. In FIG. 3, communications among the above-mentioned components of computing device 300 are denoted by numeral 390. Memory 310 includes ROM(s) (Read Only Memory) 311, RAM(s) (Random Access Memory) 313, and cache(s) 315.
  • One or more operating systems 331 and one or more computer programs 333 reside on one or more computer-readable tangible storage device(s) 330. In the exemplary embodiment, exemplary system 100, including test case selection program 120 and database 130, resides on one or more computer-readable tangible storage device(s) 330. In other embodiments, test case selection program 120 and database 130 reside respectively on multiple computer devices which are connected by a network. In further other embodiments, different components on database 130, including test cases 131, test case input data 132, test environments 133, prerequisite test cases 134, code coverage metrics 135, and code change metrics 136, reside respectively on multiple computer devices which are connected by a network.
  • Computing device 300 further includes I/O interface(s) 350. I/O interface(s) 350 allow for input and output of data with external device(s) 360 that may be connected to computing device 300. Computing device 300 further includes network interface(s) 340 for communications between computing device 300 and a computer network.
  • As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, and micro-code) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon.
  • Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
  • A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
  • Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF (radio frequency), and any suitable combination of the foregoing.
  • Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java®, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
  • Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
  • The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
  • The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Claims (18)

1. A computer-implemented method for selecting test cases for testing software that has been changed, the method comprising:
executing one or more test cases with one or more test case input data, one or more test environments, and one or more prerequisite test cases;
generating code coverage metrics for the software, code coverage metrics describing what code of the software has been executed;
mapping the code coverage metrics to a combination of the one or more test cases, the one or more test case input data, the one or more test environments, and the one or more prerequisite test cases;
generating code change metrics for the software, code change metrics describing what changes to the software have been made;
updating a database which maps the changes to the one or more test cases, the one or more test case input data, the one or more test environments, and the one or more prerequisite test cases;
determining the changes to the software, based on a correlation between the code coverage metrics and the code change metrics; and
selecting, from the one or more test cases, test cases for testing the software that has been changed.
2. (canceled)
3. (Canceled)
4. The computer-implemented method of claim 3, wherein the database comprises the one or more test cases, the one or more test case input data, the one or more test environments, the one or more prerequisite test cases, the code coverage metrics, and the code change metrics.
5. The computer-implemented method of claim 1, wherein levels of granularity of the code coverage metrics and the code change metrics include methods, statements, and condition coverage.
6. The computer-implemented method of claim 1, further comprising the step of: identifying, in the software, sections executed by the one or more test cases, based on the code coverage metrics
7. A computer program product for selecting test cases for testing software that has been changed, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable to:
execute one or more test cases with one or more test case input data, one or more test environments, and one or more prerequisite test cases;
generate code coverage metrics for the software, code coverage metrics describing what code of the software has been executed;
map the code coverage metrics to a combination of the one or more test cases, the one or more test case input data, the one or more test environments, and the one or more prerequisite test cases;
generate code change metrics for the software, code change metrics describing what changes to the software have been made;
update a database which maps the changes to the one or more test cases, the one or more test case input data, the one or more test environments, and the one or more prerequisite test cases;
determine the changes to the software, based on a correlation between the code coverage metrics and the code change metrics; and
select, from the one or more test cases, test cases for testing the software that has been changed.
8. (canceled)
9. (canceled)
10. The computer program product of claim 9, wherein the database comprises the one or more test cases, the one or more test case input data, the one or more test environments, the one or more prerequisite test cases, the code coverage metrics, and the code change metrics.
11. The computer program product of claim 7, wherein levels of granularity of the code coverage metrics and the code change metrics include methods, statements, and condition coverage.
12. The computer program product of claim 7, further comprising the program code executable to identify, in the software, sections exercised by the one or more test cases, based on the code coverage metrics.
13. A computer system for selecting test cases for testing software that has been changed, the computer system comprising:
one or more processors, one or more computer-readable tangible storage devices, and program instructions stored on at least one of the one or more computer-readable tangible storage devices for execution by at least one of the one or more processors, the program instructions executable to:
execute one or more test cases with one or more test case input data, one or more test environments, and one or more prerequisite test cases;
generate code coverage metrics for the software, code coverage metrics describing what code of the software has been executed;
map the code coverage metrics to a combination of the one or more test cases, the one or more test case input data, the one or more test environments, and the one or more prerequisite test cases;
generate code change metrics for the software, code change metrics describing what changes to the software have been made;
update a database which maps the changes to the one or more test cases, the one or more test case input data, the one or more test environments, and the one or more prerequisite test cases;
determine the changes to the software, based on a correlation between the code coverage metrics and the code change metrics; and
select, from the one or more test cases, test cases for testing the software that has been changed.
14. (canceled)
15. (canceled)
16. The computer system of claim 15, wherein the database comprises the one or more test cases, the one or more test case input data, the one or more test environments, the one or more prerequisite test cases, the code coverage metrics, and the code change metrics.
17. The computer system of claim 13, wherein levels of granularity of the code coverage metrics and the code change metrics include methods, statements, and condition coverage.
18. The computer system of claim 13, further comprising the program instructions executable to identify, in the software, sections exercised by the one or more test cases, based on the code coverage metrics.
US13/944,012 2013-07-17 2013-07-17 Method and system for automated test case selection Abandoned US20150026664A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US13/944,012 US20150026664A1 (en) 2013-07-17 2013-07-17 Method and system for automated test case selection

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US13/944,012 US20150026664A1 (en) 2013-07-17 2013-07-17 Method and system for automated test case selection

Publications (1)

Publication Number Publication Date
US20150026664A1 true US20150026664A1 (en) 2015-01-22

Family

ID=52344681

Family Applications (1)

Application Number Title Priority Date Filing Date
US13/944,012 Abandoned US20150026664A1 (en) 2013-07-17 2013-07-17 Method and system for automated test case selection

Country Status (1)

Country Link
US (1) US20150026664A1 (en)

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150278080A1 (en) * 2011-09-30 2015-10-01 International Business Machines Corporation Processing automation scripts of software
CN105183645A (en) * 2015-08-26 2015-12-23 中国电子科技集团公司第十四研究所 Reuse based design and implementation method for radar software testing
CN105279085A (en) * 2015-10-08 2016-01-27 国网天津市电力公司 Rule self-defining based smart substation configuration file test system and method
CN105930257A (en) * 2015-10-12 2016-09-07 中国银联股份有限公司 Method and apparatus for determining target test cases
US9442830B1 (en) * 2014-06-25 2016-09-13 Emc Corporation Automated test coverage analysis, execution and reporting
CN106201857A (en) * 2015-05-05 2016-12-07 阿里巴巴集团控股有限公司 The choosing method of test case and device
CN106250313A (en) * 2016-07-27 2016-12-21 天津市康凯特软件科技有限公司 Mobile phone terminal VoLte video interconnection automated testing method and device
CN106445810A (en) * 2016-08-30 2017-02-22 福建天晴数码有限公司 Interactive software and device compatibility test method and system
US9582408B1 (en) 2015-09-03 2017-02-28 Wipro Limited System and method for optimizing testing of software production incidents
CN106776268A (en) * 2016-11-04 2017-05-31 中国航空综合技术研究所 A kind of aobvious control software and hardware system reliability test motivational techniques based on section mapping
CN107844423A (en) * 2017-11-10 2018-03-27 郑州云海信息技术有限公司 A kind of appraisal procedure of software test completeness
US10019347B2 (en) * 2014-11-14 2018-07-10 Mastercard International Incorporated Systems and methods for selection of test cases for payment terminals
US20180293160A1 (en) * 2017-04-11 2018-10-11 Semmle Limited Comparing software projects having been analyzed using different criteria
WO2019169760A1 (en) * 2018-03-06 2019-09-12 平安科技(深圳)有限公司 Test case range determining method, device, and storage medium
US10430319B1 (en) 2018-05-04 2019-10-01 Fmr Llc Systems and methods for automatic software testing

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6536036B1 (en) * 1998-08-20 2003-03-18 International Business Machines Corporation Method and apparatus for managing code test coverage data
US20080172655A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Saving Code Coverage Data for Analysis
US20100146340A1 (en) * 2008-12-09 2010-06-10 International Business Machines Corporation Analyzing Coverage of Code Changes
US20120192153A1 (en) * 2011-01-25 2012-07-26 Verizon Patent And Licensing Inc. Method and system for providing a testing framework
US8276123B1 (en) * 2008-07-22 2012-09-25 Juniper Networks, Inc. Adaptive regression test selection within testing environments
US8448141B2 (en) * 2008-03-31 2013-05-21 International Business Machines Corporation Evaluation of software based on change history

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6536036B1 (en) * 1998-08-20 2003-03-18 International Business Machines Corporation Method and apparatus for managing code test coverage data
US20080172655A1 (en) * 2007-01-15 2008-07-17 Microsoft Corporation Saving Code Coverage Data for Analysis
US8448141B2 (en) * 2008-03-31 2013-05-21 International Business Machines Corporation Evaluation of software based on change history
US8276123B1 (en) * 2008-07-22 2012-09-25 Juniper Networks, Inc. Adaptive regression test selection within testing environments
US20100146340A1 (en) * 2008-12-09 2010-06-10 International Business Machines Corporation Analyzing Coverage of Code Changes
US20120192153A1 (en) * 2011-01-25 2012-07-26 Verizon Patent And Licensing Inc. Method and system for providing a testing framework

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Koskela, Lasse. "Introduction to Code Coverage." JavaRanch Journal (2004): n. pag. JavaRanch. Web. 5 Aug. 2014. . *

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150278080A1 (en) * 2011-09-30 2015-10-01 International Business Machines Corporation Processing automation scripts of software
US10387290B2 (en) 2011-09-30 2019-08-20 International Business Machines Corporation Processing automation scripts of software
US9483389B2 (en) * 2011-09-30 2016-11-01 International Business Machines Corporation Processing automation scripts of software
US10324831B1 (en) * 2014-06-25 2019-06-18 EMC IP Holding Company LLC Automated test coverage analysis, execution and reporting
US9442830B1 (en) * 2014-06-25 2016-09-13 Emc Corporation Automated test coverage analysis, execution and reporting
US10019347B2 (en) * 2014-11-14 2018-07-10 Mastercard International Incorporated Systems and methods for selection of test cases for payment terminals
CN106201857A (en) * 2015-05-05 2016-12-07 阿里巴巴集团控股有限公司 The choosing method of test case and device
CN105183645A (en) * 2015-08-26 2015-12-23 中国电子科技集团公司第十四研究所 Reuse based design and implementation method for radar software testing
US9582408B1 (en) 2015-09-03 2017-02-28 Wipro Limited System and method for optimizing testing of software production incidents
CN105279085A (en) * 2015-10-08 2016-01-27 国网天津市电力公司 Rule self-defining based smart substation configuration file test system and method
CN105930257A (en) * 2015-10-12 2016-09-07 中国银联股份有限公司 Method and apparatus for determining target test cases
CN106250313A (en) * 2016-07-27 2016-12-21 天津市康凯特软件科技有限公司 Mobile phone terminal VoLte video interconnection automated testing method and device
CN106445810A (en) * 2016-08-30 2017-02-22 福建天晴数码有限公司 Interactive software and device compatibility test method and system
CN106776268A (en) * 2016-11-04 2017-05-31 中国航空综合技术研究所 A kind of aobvious control software and hardware system reliability test motivational techniques based on section mapping
US20180293160A1 (en) * 2017-04-11 2018-10-11 Semmle Limited Comparing software projects having been analyzed using different criteria
US10346294B2 (en) * 2017-04-11 2019-07-09 Semmle Limited Comparing software projects having been analyzed using different criteria
CN107844423A (en) * 2017-11-10 2018-03-27 郑州云海信息技术有限公司 A kind of appraisal procedure of software test completeness
WO2019169760A1 (en) * 2018-03-06 2019-09-12 平安科技(深圳)有限公司 Test case range determining method, device, and storage medium
US10430319B1 (en) 2018-05-04 2019-10-01 Fmr Llc Systems and methods for automatic software testing

Similar Documents

Publication Publication Date Title
US8752182B2 (en) Pinpointing security vulnerabilities in computer software applications
US20090070734A1 (en) Systems and methods for monitoring software application quality
US9104806B2 (en) Tracking of code base and defect diagnostic coupling with automated triage
US10055338B2 (en) Completing functional testing
US9389849B2 (en) Test case pattern matching
US20120159434A1 (en) Code clone notification and architectural change visualization
Arcuri et al. Black-box system testing of real-time embedded systems using random and search-based testing
US8381184B2 (en) Dynamic test coverage
US9336127B2 (en) Exposing method related data calls during testing in an event driven, multichannel architecture
US20080178154A1 (en) Developing software components and capability testing procedures for testing coded software component
US20100146340A1 (en) Analyzing Coverage of Code Changes
US20140351793A1 (en) Prioritizing test cases using multiple variables
US20130132933A1 (en) Automated compliance testing during application development
US20140245264A1 (en) Identifying Test Cases Based on Changed Test Code
US8856764B2 (en) Distributed static analysis of computer software applications
US9612937B2 (en) Determining relevant events in source code analysis
US20140123107A1 (en) Systems and methods to maintain consistency between software specification and code
US9720811B2 (en) Unified model for visual component testing
US8875104B2 (en) Efficiently developing software using test cases to check the conformity of the software to the requirements
US20140279934A1 (en) Self-analyzing data processing job to determine data quality issues
US10031841B2 (en) Method and system for incrementally updating a test suite utilizing run-time application executions
US9921952B2 (en) Early risk identification in DevOps environments
US9405662B2 (en) Process for displaying test coverage data during code reviews
US8756460B2 (en) Test selection based on an N-wise combinations coverage
US9239773B1 (en) Method and system for debugging a program that includes declarative code and procedural code

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTERNATIONAL BUSINESS MACHINES CORPORATION, NEW Y

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:BARTLEY, TIM S;BRAY, GAVIN G;HUGHES, LIZ M;AND OTHERS;SIGNING DATES FROM 20130712 TO 20130715;REEL/FRAME:030816/0353

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION