US20080162995A1 - In-cycle system test adaptation - Google Patents
In-cycle system test adaptation Download PDFInfo
- Publication number
- US20080162995A1 US20080162995A1 US11/926,063 US92606307A US2008162995A1 US 20080162995 A1 US20080162995 A1 US 20080162995A1 US 92606307 A US92606307 A US 92606307A US 2008162995 A1 US2008162995 A1 US 2008162995A1
- Authority
- US
- United States
- Prior art keywords
- test
- trigger
- plan
- occurrence
- information processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
Definitions
- the present invention generally relates to the field of system testing, and more particularly relates to dynamically adapting system test procedures during the testing cycle.
- System testing is generally used to test a product such as a software application in an integrated system.
- a system can include software, hardware, and the like.
- Current methods for system testing of a product begin with building a test plan.
- the test plan provides a sparse coverage matrix of a particular product based on its documented design and customer usage cases.
- the documented design and customer usage cases are generated prior to the product's creation.
- the system test plan is one of the measurement devices that is used to determine if the quality and function of the product has been verified.
- After the product is created it can be system tested to identify defects and behavior anomalies, which are reported by the testers. Usually a subset of these defects and anomalies are fixed within the current release cycle.
- Design based plans focus on what was thought to be an issue in design, which may not be an issue at all. This causes test resources to be wasted on non-problematic areas leaving other areas not adequately tested.
- a method, information processing system, and computer readable medium for performing a system test on a program.
- the method comprises creating a test plan associated with a system test.
- the system test is for testing a program within an environment.
- At least one test trigger to be monitored for during the system test is defined within the test plan.
- Execution of the system test on a system under test for the at least one test trigger is monitored.
- An occurrence of the at least one test trigger is determined.
- an information processing system for performing a system test on a program.
- the information processing system comprises a memory and a processor that is communicatively coupled to the memory.
- the information processing system also comprises a system test module that is communicatively coupled to the memory and the processor.
- the system test module is for creating a test plan associated with a system test.
- the system test is for testing a program within an environment. At least one test trigger to be monitored for during the system test is defined within the test plan. Execution of the system test on a system under test for the at least one test trigger is monitored. An occurrence of the at least one test trigger is determined.
- the test plan is modified to take into account the occurrence of the at least one test trigger in response to determining the occurrence. Execution of the system test is continued based on the modified test plan.
- a computer readable medium for performing a system test on a program comprises instructions for creating a test plan associated with a system test.
- the system test is for testing a program within an environment.
- At least one test trigger to be monitored for during the system test is defined within the test plan.
- Execution of the system test on a system under test for the at least one test trigger is monitored.
- An occurrence of the at least one test trigger is determined.
- Test triggers can be defined prior to the start of a system test cycle for triggering a test modification analysis. Based on this triggered analysis, the system test can be dynamically updated with new test cases that are executed in the current test cycle. Therefore, the system test is not completely defined by a design specification, but can be adapted based on the actual product's performance.
- the test triggers are self-learning in that once a particular trigger has occurred additional triggers can be created based on the in-cycle learning at that point.
- FIG. 1 is a block diagram illustrating an exemplary testing system according to an embodiment of the present invention
- FIG. 2 a more detailed view of an information processing system for performing systems testing functions according to an embodiment of the present invention.
- FIG. 3 and FIG. 4 are operational flow diagrams illustrating an exemplary process of dynamically modifying an in-cycle system test based on detected triggering events according to an embodiment of the present invention.
- the terms “a” or “an”, as used herein, are defined as one or more than one.
- the term plurality, as used herein, is defined as two or more than two.
- the term another, as used herein, is defined as at least a second or more.
- the terms including and/or having, as used herein, are defined as comprising (i.e., open language).
- the term coupled, as used herein, is defined as connected, although not necessarily directly, and not necessarily mechanically.
- program, software application, and the like as used herein, are defined as a sequence of instructions designed for execution on a computer system.
- a program, computer program, or software application may include a subroutine, a function, a procedure, an object method, an object implementation, an executable application, an applet, a servlet, a source code, an object code, a shared library/dynamic load library and/or other sequence of instructions designed for execution on a computer system.
- FIG. 1 shows a testing information processing system 102 communicatively coupled to information processing system 104 comprising components or software to be tested.
- a network 106 such as a local area network, wide area network, or the like couples the two systems 102 , 104 to one another. Further embodiments of the present invention connect the two systems 102 , 106 together using any suitable communications connection.
- the testing information processing system 102 can be coupled to a plurality of systems to be tested together or separately.
- the testing information processing system 102 controls the testing of the system components or software residing on or coupled to the information processing system 104 .
- the testing information processing system 102 includes a system test module 108 , which comprises a test trigger monitor 110 and a dynamic test updater 112 . These components are discussed in greater detail below.
- Unit testing occurs at the source code level. For example, unit testing confirms that a method produces an expected output when given a known input. A unit test is used to verify that particular source code modules are working properly. Unit tests are generally written from the perspective of the programmer. Functional testing verifies that the system/software is performing how it should. Generally, functional testing confirms that a system or software is performing according to user requirements. System testing tests a product in a system environment to ensure that the product operates satisfactory in the environment it was designed for. Although the following discussion is with respect to system testing, embodiments of the present invention are also applicable to unit testing and functional testing.
- FIG. 2 is a block diagram illustrating a more detailed view of the testing information processing system 102 of FIG. 1 .
- the information processing system 102 is based upon a suitably configured processing system adapted to implement the exemplary embodiment of the present invention. Any suitably configured processing system is similarly able to be used as the information processing system 102 by embodiments of the present invention, for example, a personal computer, workstation, or the like.
- the information processing system 102 includes a computer 202 .
- the computer 202 has a processor 204 that is communicatively connected to a main memory 206 (e.g., volatile memory), mass storage interface 208 , a terminal interface 210 , and network adapter hardware 212 .
- a system bus 214 interconnects these system components.
- the mass storage interface 208 is used to connect mass storage devices, such as data storage device 216 to the central server 102 .
- One specific type of data storage device is a computer readable medium such as a CD drive, which may be used to store data to and read data from a CD or DVD 218 or floppy diskette (not shown).
- Another type of data storage device is a data storage device configured to support, for example, fixed disk type file system operations.
- the main memory 206 includes the system test module 108 .
- the system test module 108 includes the test trigger monitor 110 , dynamic test updater 112 , graphical user interface 220 , and a test plan(s) 222 which includes test triggers 224 . These components are discussed in greater detail below.
- the information processing system 102 utilizes conventional virtual addressing mechanisms to allow programs to behave as if they have access to a large, single storage entity, referred to herein as a computer system memory, instead of access to multiple, smaller storage entities such as the main memory 206 and data storage device 216 .
- computer system memory is used herein to generically refer to the entire virtual memory of the information processing system 102
- Embodiments of the present invention further incorporate interfaces that each includes separate, fully programmed microprocessors that are used to off-load processing from the CPU 204 .
- Terminal interface 210 is used to directly connect one or more terminals 226 to computer 202 to provide a user interface to the computer 202 .
- These terminals 226 which are able to be non-intelligent or fully programmable workstations, are used to allow system administrators and users to communicate with the thin client.
- the terminal 226 is also able to consist of user interface and peripheral devices that are connected to computer 202 and controlled by terminal interface hardware included in the terminal I/F 210 that includes video adapters and interfaces for keyboards, pointing devices, and the like.
- An operating system 228 can be included in the main memory 206 and is a suitable multitasking operating system such as the Linux, UNIX, Windows XP, and Windows Server operating system.
- Embodiments of the present invention are able to use any other suitable operating system, or kernel, or other suitable control software.
- Some embodiments of the present invention utilize architectures, such as an object oriented framework mechanism, that allows instructions of the components of operating system (not shown) to be executed on any processor located within the client.
- the network adapter hardware 212 is used to provide an interface to the network 106 .
- Embodiments of the present invention are able to be adapted to work with any data communications connections including present day analog and/or digital techniques or via a future networking mechanism.
- the system test module 108 controls system testing of products.
- a system test is defined by its test plan 224 .
- a test plan 224 is a set of test cases plus any other additional information that may be required to complete the testing, such as the required environment and context.
- a system test can be performed manually or can be automated. For manual testing, a tester hard codes each test case. Automated testing utilizes a tool that automatically enters a predetermined set of characters or user commands in order to test a system/software according to a created test plan. In some embodiments of the present invention, the test plan is created based upon user input.
- test plans for systems tests are generally created prior to the product being created. Therefore, changes in design process that were not foreseen when the test plans were generated are not taken into account. Therefore, any defects or anomalies in the product caused by these changes usually make their way into the current product release. These problems are not corrected until the product's next release cycle.
- the present invention provides a system test method that adaptively changes its test plans in response to detected events or lack of events. Therefore, the dynamic system test of the present invention can account for changes during the design process.
- test plans 224 of the present invention include test triggers 224 , which allow for a test cycle to be dynamically modified based on the triggering event.
- Test triggers 224 can be defined based on areas of concern within the product. For example, problem areas in the product identified during design can be added as triggers 224 . Environmental events in the development organization of the product can also be used to define a trigger 224 . These are only a few examples on what triggers 224 can be based on.
- a test trigger 224 can be self-learning.
- a self learning trigger is when a predefined trigger causes analysis by a test engineer to determine if the event or events that caused the trigger require new triggers to be designed and deployed. For example, a slow GUI now causes other performance triggers to be defined for other areas that call the same underlying software libraries like associated command line interfaces.
- a minimum self-learning trigger is one that rearms itself when the event no longer occurs after fixes have been applied to the product and the normal predefined testing can resume.
- a test trigger 224 can be generic such that a test modification analysis is performed for each defect found.
- a test trigger 224 can also be narrow such that a test modification analysis is performed for a specific defect or behavior anomaly.
- a lack of defects or anomalies can be set as a test trigger 224 to cause a test modification analysis that may improve testing efficiency.
- a test modification analysis in one embodiment, analyzes the triggering event and its root cause. This allows the tester or an automated testing system to determine whether new triggers or new test cases should be created.
- the test trigger monitor 108 monitors for the triggering events and records each occurrence.
- a system tester is visually presented with a display of test results including any triggering events through the GUI 220 .
- the system tester can then modify the test plan(s) 222 or create new triggers 224 .
- the testing information processing system 102 of one embodiment is able to create at least one additional test trigger based upon user input.
- the requested changes are then applied to the system test via the dynamic test updater 112 .
- the dynamic test updater 112 can automatically define new triggers based on a trigger 224 that is detected by the monitor 110 and in-cycle learning up to that point. Reactive changes to the new triggers 224 or new test plans 222 can cause new in-cycle learning and the re-starting the triggering cycle.
- the test cycle can be configured to end after a predefined period of time or when triggering events are no longer detected.
- test adaptation is as follows.
- a product to detect overheating in a CPU has been designed. When the product detects overheating it can reduce CPU frequencies and/or increase cooling fan speed.
- Test engineers develop a test plan to ensure that applications, data I/O, and network communications are unaffected by the overheating response.
- Test triggers 224 are defined to detect any defect or behavior anomaly outside of the defined applications, data I/O, and network communications. The system test is executed and the applications, I/O, and network communications are not affected by increased fan speed or reduced CPU frequencies.
- a triggering event occurs and is detected by the test trigger monitor 110 .
- This event can be displayed to a system tester who can associate the event with the product or the system test module 108 can automatically record the event with any information needed to identify event, particular test the event was associated with, and the like.
- the test engineers review the test they can recognize that a triggering event occurred and determine context information for the trigger event. In other words, the test engineers recognize that a defect occurred in an area of the system that the test plan was not designed to test. Therefore, the test engineers can modify the test plan to test or accommodate the new graphical interface. Additional triggers can also be added to the test plan.
- the present invention provides an advantageous system testing module 206 .
- Current system testing methods that are static would not have detected that defect in the graphical user interface in the above example. Therefore, this defect would have been included in the release of the product.
- the present invention includes test triggers 224 that allow for defects and behavior anomalies to be detected even though the test plan is not configured to test for them. The test plan can then be updated in response to the detected triggers 224 .
- FIG. 3 and FIG. 4 illustrate an exemplary process for modifying a test plan during a system text cycle based on test triggers 224 .
- the operational flow diagram of FIG. 3 begins at step 302 and flows directly to step 304 .
- a design document, at step 304 is created for a product.
- a test plan, at step 306 is then created based on the design document.
- Exit criteria and test triggers 224 , at step 308 are defined and placed into the test plan.
- An exit criterion defines when a test cycle can exit out of its cycle. Exemplary exit criteria are a given time interval in which test triggers 224 are no longer detected, and the like.
- the product, at step 310 , is created and a system test, at step 312 , is executed for the product based on the test plan.
- the control then flows to entry point A of FIG. 4 .
- the test, at step 402 is monitored for trigger events.
- the system test module 108 determines if a trigger event has occurred. If the result of this determination is negative, the system test module 108 , at step 406 , determines if the exit criteria has been satisfied. If the result of this determination is negative, the system test, at step 408 , is continued and the control flows back to step 402 . If the result of this determination is positive, the test execution, at step 410 , ends and the control flow exits at step 412 .
- the trigger is analyzed.
- the analyzing can be performed manually on data presented by the testing information processing system 102 or by an automated system.
- the triggers are analyzed to identify the defect or behavior anomaly and its underlying cause.
- the portion of the test plan that needs to be modified is identified at step 416 .
- the test plan, at step 418 is modified. For example, new tests and/or existing tests can be combined and inserted into the current test execution. This allows for similar, but different defects to be identified.
- new triggers 224 can be created and inserted into the current test execution cycle. Test case planned attempts may also be modified based on the trigger analysis.
- the system test module 108 determines if new triggers have been created. If the result of this determination is negative, the system test, at step 422 , is continued with the modified test plan and the control flows back to step 402 . If the result of this determination is positive, the test plan, at step 424 , is updated with the new triggers and the system test, at step 426 , is continued with the modified test plan and new triggers. The control flows back to step 402 .
- the present invention as would be known to one of ordinary skill in the art could be produced in hardware or software, or in a combination of hardware and software. However in one embodiment the invention is implemented in software.
- the system, or method, according to the inventive principles as disclosed in connection with the preferred embodiment may be produced in a single computer system having separate elements or means for performing the individual functions or steps described or claimed or one or more elements or means combining the performance of any of the functions or steps disclosed or claimed, or may be arranged in a distributed computer system, interconnected by any suitable means as would be known by one of ordinary skill in the art.
- the invention and the inventive principles are not limited to any particular kind of computer system but may be used with any general purpose computer, as would be known to one of ordinary skill in the art, arranged to perform the functions described and the method steps described.
- the operations of such a computer, as described above, may be according to a computer program contained on a medium for use in the operation or control of the computer, as would be known to one of ordinary skill in the art.
- the computer medium which may be used to hold or contain the computer program product, may be a fixture of the computer such as an embedded memory or may be on a transportable medium such as a disk, as would be known to one of ordinary skill in the art.
- any such computing system can include, inter alia, at least a computer readable medium allowing a computer to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium.
- the computer readable medium may include non-volatile memory, such as ROM, Flash memory, floppy disk, Disk drive memory, CD-ROM, and other permanent storage. Additionally, a computer readable medium may include, for example, volatile storage such as RAM, buffers, cache memory, and network circuits.
- the computer readable medium may include computer readable information in a transitory state medium such as a network link and/or a network interface, including a wired network or a wireless network that allows a computer to read such computer readable information.
- a transitory state medium such as a network link and/or a network interface, including a wired network or a wireless network that allows a computer to read such computer readable information.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
Disclosed are an information processing system and computer readable medium for performing a system test on a program. A test plan associated with a system test is created. The system test is for testing a program within an environment. At least one test trigger to be monitored for during the system test is defined within the test plan. Execution of the system test on a system under test for the at least one test trigger is monitored. An occurrence of the at least one test trigger is determined. The test plan is modified to take into account the occurrence of the at least one test trigger in response to determining the occurrence. Execution of the system test is continued based on the modified test plan.
Description
- The present invention generally relates to the field of system testing, and more particularly relates to dynamically adapting system test procedures during the testing cycle.
- System testing is generally used to test a product such as a software application in an integrated system. A system can include software, hardware, and the like. Current methods for system testing of a product begin with building a test plan. The test plan provides a sparse coverage matrix of a particular product based on its documented design and customer usage cases. The documented design and customer usage cases are generated prior to the product's creation. The system test plan is one of the measurement devices that is used to determine if the quality and function of the product has been verified. After the product is created it can be system tested to identify defects and behavior anomalies, which are reported by the testers. Usually a subset of these defects and anomalies are fixed within the current release cycle.
- One problem with the current state of system testing is that the system test plan is created prior the product being created. Therefore, the abilities of any particular development team at any given point in time cannot be taken into account. In any given development team at any particular point in time (which is a particular release cycle of the product) there are certain abilities and weaknesses. For example, contractors may have little experience with deployment tools, the product design, environment and user cases for the product, and the like. The technology being deployed can be new to the team members. The design process can also change during the design of the product or the product implementation can change. These are only a few examples of events that cannot be taken into account if a system test plan is created prior to the product's creation.
- Accordingly, a design based system test plan is not an adequate method to discover defects in a product. Design based plans focus on what was thought to be an issue in design, which may not be an issue at all. This causes test resources to be wasted on non-problematic areas leaving other areas not adequately tested.
- Therefore a need exists to overcome the problems with the prior art as discussed above.
- Briefly, in accordance with the present invention, disclosed are a method, information processing system, and computer readable medium for performing a system test on a program. The method comprises creating a test plan associated with a system test. The system test is for testing a program within an environment. At least one test trigger to be monitored for during the system test is defined within the test plan. Execution of the system test on a system under test for the at least one test trigger is monitored. An occurrence of the at least one test trigger is determined. The test plan is modified to take into account the occurrence of the at least one test trigger in response to determining the occurrence. Execution of the system test is continued based on the modified test plan.
- In another embodiment an information processing system for performing a system test on a program is disclosed. The information processing system comprises a memory and a processor that is communicatively coupled to the memory. The information processing system also comprises a system test module that is communicatively coupled to the memory and the processor. The system test module is for creating a test plan associated with a system test. The system test is for testing a program within an environment. At least one test trigger to be monitored for during the system test is defined within the test plan. Execution of the system test on a system under test for the at least one test trigger is monitored. An occurrence of the at least one test trigger is determined. The test plan is modified to take into account the occurrence of the at least one test trigger in response to determining the occurrence. Execution of the system test is continued based on the modified test plan.
- In yet another embodiment, a computer readable medium for performing a system test on a program is disclosed. The computer readable medium comprises instructions for creating a test plan associated with a system test. The system test is for testing a program within an environment. At least one test trigger to be monitored for during the system test is defined within the test plan. Execution of the system test on a system under test for the at least one test trigger is monitored. An occurrence of the at least one test trigger is determined. The test plan is modified to take into account the occurrence of the at least one test trigger in response to determining the occurrence. Execution of the system test is continued based on the modified test plan.
- One advantage of the present invention is a new system test process is provided. Test triggers can be defined prior to the start of a system test cycle for triggering a test modification analysis. Based on this triggered analysis, the system test can be dynamically updated with new test cases that are executed in the current test cycle. Therefore, the system test is not completely defined by a design specification, but can be adapted based on the actual product's performance. Another advantage is that the test triggers are self-learning in that once a particular trigger has occurred additional triggers can be created based on the in-cycle learning at that point.
- The accompanying figures where like reference numerals refer to identical or functionally similar elements throughout the separate views, and which together with the detailed description below are incorporated in and form part of the specification, serve to further illustrate various embodiments and to explain various principles and advantages all in accordance with the present invention.
-
FIG. 1 is a block diagram illustrating an exemplary testing system according to an embodiment of the present invention; -
FIG. 2 a more detailed view of an information processing system for performing systems testing functions according to an embodiment of the present invention; and -
FIG. 3 andFIG. 4 are operational flow diagrams illustrating an exemplary process of dynamically modifying an in-cycle system test based on detected triggering events according to an embodiment of the present invention. - As required, detailed embodiments of the present invention are disclosed herein; however, it is to be understood that the disclosed embodiments are merely exemplary of the invention, which can be embodied in various forms. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a basis for the claims and as a representative basis for teaching one skilled in the art to variously employ the present invention in virtually any appropriately detailed structure. Further, the terms and phrases used herein are not intended to be limiting; but rather, to provide an understandable description of the invention.
- The terms “a” or “an”, as used herein, are defined as one or more than one. The term plurality, as used herein, is defined as two or more than two. The term another, as used herein, is defined as at least a second or more. The terms including and/or having, as used herein, are defined as comprising (i.e., open language). The term coupled, as used herein, is defined as connected, although not necessarily directly, and not necessarily mechanically. The terms program, software application, and the like as used herein, are defined as a sequence of instructions designed for execution on a computer system. A program, computer program, or software application may include a subroutine, a function, a procedure, an object method, an object implementation, an executable application, an applet, a servlet, a source code, an object code, a shared library/dynamic load library and/or other sequence of instructions designed for execution on a computer system.
- Exemplary Testing System
- According to an embodiment of the present invention, as shown in
FIG. 1 , anexemplary testing system 100 is shown.FIG. 1 shows a testinginformation processing system 102 communicatively coupled toinformation processing system 104 comprising components or software to be tested. Anetwork 106 such as a local area network, wide area network, or the like couples the twosystems systems information processing system 102 can be coupled to a plurality of systems to be tested together or separately. The testinginformation processing system 102 controls the testing of the system components or software residing on or coupled to theinformation processing system 104. The testinginformation processing system 102 includes asystem test module 108, which comprises a test trigger monitor 110 and adynamic test updater 112. These components are discussed in greater detail below. - Testing of a system or software helps ensure that it performs consistently within its functional specifications and that it is compatible with its environment. There are three main areas of testing: unit testing, functional testing, and system testing. Unit testing occurs at the source code level. For example, unit testing confirms that a method produces an expected output when given a known input. A unit test is used to verify that particular source code modules are working properly. Unit tests are generally written from the perspective of the programmer. Functional testing verifies that the system/software is performing how it should. Generally, functional testing confirms that a system or software is performing according to user requirements. System testing tests a product in a system environment to ensure that the product operates satisfactory in the environment it was designed for. Although the following discussion is with respect to system testing, embodiments of the present invention are also applicable to unit testing and functional testing.
- Exemplary Information Processing System
-
FIG. 2 is a block diagram illustrating a more detailed view of the testinginformation processing system 102 ofFIG. 1 . Theinformation processing system 102 is based upon a suitably configured processing system adapted to implement the exemplary embodiment of the present invention. Any suitably configured processing system is similarly able to be used as theinformation processing system 102 by embodiments of the present invention, for example, a personal computer, workstation, or the like. - The
information processing system 102 includes acomputer 202. Thecomputer 202 has aprocessor 204 that is communicatively connected to a main memory 206 (e.g., volatile memory),mass storage interface 208, aterminal interface 210, andnetwork adapter hardware 212. Asystem bus 214 interconnects these system components. Themass storage interface 208 is used to connect mass storage devices, such asdata storage device 216 to thecentral server 102. One specific type of data storage device is a computer readable medium such as a CD drive, which may be used to store data to and read data from a CD orDVD 218 or floppy diskette (not shown). Another type of data storage device is a data storage device configured to support, for example, fixed disk type file system operations. - The
main memory 206, in one embodiment, includes thesystem test module 108. Thesystem test module 108 includes thetest trigger monitor 110,dynamic test updater 112,graphical user interface 220, and a test plan(s) 222 which includes test triggers 224. These components are discussed in greater detail below. In one embodiment, theinformation processing system 102 utilizes conventional virtual addressing mechanisms to allow programs to behave as if they have access to a large, single storage entity, referred to herein as a computer system memory, instead of access to multiple, smaller storage entities such as themain memory 206 anddata storage device 216. Note that the term “computer system memory” is used herein to generically refer to the entire virtual memory of theinformation processing system 102 - Although only one
CPU 204 is illustrated forcomputer 202, computer systems with multiple CPUs can be used equally effectively. Embodiments of the present invention further incorporate interfaces that each includes separate, fully programmed microprocessors that are used to off-load processing from theCPU 204.Terminal interface 210 is used to directly connect one ormore terminals 226 tocomputer 202 to provide a user interface to thecomputer 202. Theseterminals 226, which are able to be non-intelligent or fully programmable workstations, are used to allow system administrators and users to communicate with the thin client. The terminal 226 is also able to consist of user interface and peripheral devices that are connected tocomputer 202 and controlled by terminal interface hardware included in the terminal I/F 210 that includes video adapters and interfaces for keyboards, pointing devices, and the like. - An
operating system 228 according to an embodiment, can be included in themain memory 206 and is a suitable multitasking operating system such as the Linux, UNIX, Windows XP, and Windows Server operating system. Embodiments of the present invention are able to use any other suitable operating system, or kernel, or other suitable control software. Some embodiments of the present invention utilize architectures, such as an object oriented framework mechanism, that allows instructions of the components of operating system (not shown) to be executed on any processor located within the client. Thenetwork adapter hardware 212 is used to provide an interface to thenetwork 106. Embodiments of the present invention are able to be adapted to work with any data communications connections including present day analog and/or digital techniques or via a future networking mechanism. - Although the exemplary embodiments of the present invention are described in the context of a fully functional computer system, those skilled in the art will appreciate that embodiments are capable of being distributed as a program product via floppy disk, e.g. floppy disk w18, CD ROM, or other form of recordable media, or via any type of electronic transmission mechanism.
- In-Cycle System Test Adaptation
- The
system test module 108 controls system testing of products. A system test is defined by itstest plan 224. In one embodiment, atest plan 224 is a set of test cases plus any other additional information that may be required to complete the testing, such as the required environment and context. A system test can be performed manually or can be automated. For manual testing, a tester hard codes each test case. Automated testing utilizes a tool that automatically enters a predetermined set of characters or user commands in order to test a system/software according to a created test plan. In some embodiments of the present invention, the test plan is created based upon user input. - As discussed above test plans for systems tests are generally created prior to the product being created. Therefore, changes in design process that were not foreseen when the test plans were generated are not taken into account. Therefore, any defects or anomalies in the product caused by these changes usually make their way into the current product release. These problems are not corrected until the product's next release cycle. However, the present invention provides a system test method that adaptively changes its test plans in response to detected events or lack of events. Therefore, the dynamic system test of the present invention can account for changes during the design process.
- The test plans 224 of the present invention include test triggers 224, which allow for a test cycle to be dynamically modified based on the triggering event. Test triggers 224 can be defined based on areas of concern within the product. For example, problem areas in the product identified during design can be added as triggers 224. Environmental events in the development organization of the product can also be used to define a
trigger 224. These are only a few examples on what triggers 224 can be based on. Furthermore, atest trigger 224 can be self-learning. - One example of a self learning trigger is when a predefined trigger causes analysis by a test engineer to determine if the event or events that caused the trigger require new triggers to be designed and deployed. For example, a slow GUI now causes other performance triggers to be defined for other areas that call the same underlying software libraries like associated command line interfaces. A minimum self-learning trigger is one that rearms itself when the event no longer occurs after fixes have been applied to the product and the normal predefined testing can resume.
- A
test trigger 224 can be generic such that a test modification analysis is performed for each defect found. Atest trigger 224 can also be narrow such that a test modification analysis is performed for a specific defect or behavior anomaly. Also, a lack of defects or anomalies can be set as atest trigger 224 to cause a test modification analysis that may improve testing efficiency. A test modification analysis, in one embodiment, analyzes the triggering event and its root cause. This allows the tester or an automated testing system to determine whether new triggers or new test cases should be created. - The test trigger monitor 108 monitors for the triggering events and records each occurrence. A system tester is visually presented with a display of test results including any triggering events through the
GUI 220. The system tester can then modify the test plan(s) 222 or createnew triggers 224. The testinginformation processing system 102 of one embodiment is able to create at least one additional test trigger based upon user input. The requested changes are then applied to the system test via thedynamic test updater 112. Furthermore, thedynamic test updater 112 can automatically define new triggers based on atrigger 224 that is detected by themonitor 110 and in-cycle learning up to that point. Reactive changes to thenew triggers 224 or new test plans 222 can cause new in-cycle learning and the re-starting the triggering cycle. In one embodiment, the test cycle can be configured to end after a predefined period of time or when triggering events are no longer detected. - One example of in-cycle system test adaptation is as follows. A product to detect overheating in a CPU has been designed. When the product detects overheating it can reduce CPU frequencies and/or increase cooling fan speed. Test engineers develop a test plan to ensure that applications, data I/O, and network communications are unaffected by the overheating response. Test triggers 224 are defined to detect any defect or behavior anomaly outside of the defined applications, data I/O, and network communications. The system test is executed and the applications, I/O, and network communications are not affected by increased fan speed or reduced CPU frequencies.
- However, during an upgrade on the user interface to the system, menus appear different and performance is slow. This behavior would normally result in a test failure, but with one embodiment of the present invention, a triggering event occurs and is detected by the
test trigger monitor 110. This event can be displayed to a system tester who can associate the event with the product or thesystem test module 108 can automatically record the event with any information needed to identify event, particular test the event was associated with, and the like. When the test engineers review the test they can recognize that a triggering event occurred and determine context information for the trigger event. In other words, the test engineers recognize that a defect occurred in an area of the system that the test plan was not designed to test. Therefore, the test engineers can modify the test plan to test or accommodate the new graphical interface. Additional triggers can also be added to the test plan. - As can be seen, the present invention provides an advantageous
system testing module 206. Current system testing methods that are static would not have detected that defect in the graphical user interface in the above example. Therefore, this defect would have been included in the release of the product. The present invention, on the other hand, includes test triggers 224 that allow for defects and behavior anomalies to be detected even though the test plan is not configured to test for them. The test plan can then be updated in response to the detected triggers 224. - Exemplary Process For In-Cycle System Test Adaptation
-
FIG. 3 andFIG. 4 illustrate an exemplary process for modifying a test plan during a system text cycle based on test triggers 224. The operational flow diagram ofFIG. 3 begins atstep 302 and flows directly to step 304. A design document, atstep 304, is created for a product. A test plan, atstep 306, is then created based on the design document. Exit criteria and test triggers 224, atstep 308, are defined and placed into the test plan. An exit criterion defines when a test cycle can exit out of its cycle. Exemplary exit criteria are a given time interval in which test triggers 224 are no longer detected, and the like. - The product, at
step 310, is created and a system test, atstep 312, is executed for the product based on the test plan. The control then flows to entry point A ofFIG. 4 . After the system test is executed atstep 312 ofFIG. 3 , the test, atstep 402, is monitored for trigger events. Thesystem test module 108, atstep 404, determines if a trigger event has occurred. If the result of this determination is negative, thesystem test module 108, atstep 406, determines if the exit criteria has been satisfied. If the result of this determination is negative, the system test, atstep 408, is continued and the control flows back tostep 402. If the result of this determination is positive, the test execution, atstep 410, ends and the control flow exits atstep 412. - If the result of the determination at
step 404 is positive, the trigger, atstep 414, is analyzed. The analyzing can be performed manually on data presented by the testinginformation processing system 102 or by an automated system. The triggers are analyzed to identify the defect or behavior anomaly and its underlying cause. Based on the analysis, the portion of the test plan that needs to be modified is identified atstep 416. The test plan, atstep 418, is modified. For example, new tests and/or existing tests can be combined and inserted into the current test execution. This allows for similar, but different defects to be identified. Also,new triggers 224 can be created and inserted into the current test execution cycle. Test case planned attempts may also be modified based on the trigger analysis. - The
system test module 108, atstep 420, determines if new triggers have been created. If the result of this determination is negative, the system test, atstep 422, is continued with the modified test plan and the control flows back tostep 402. If the result of this determination is positive, the test plan, atstep 424, is updated with the new triggers and the system test, atstep 426, is continued with the modified test plan and new triggers. The control flows back tostep 402. - Non-Limiting Examples
- The present invention as would be known to one of ordinary skill in the art could be produced in hardware or software, or in a combination of hardware and software. However in one embodiment the invention is implemented in software. The system, or method, according to the inventive principles as disclosed in connection with the preferred embodiment, may be produced in a single computer system having separate elements or means for performing the individual functions or steps described or claimed or one or more elements or means combining the performance of any of the functions or steps disclosed or claimed, or may be arranged in a distributed computer system, interconnected by any suitable means as would be known by one of ordinary skill in the art.
- According to the inventive principles as disclosed in connection with the preferred embodiment, the invention and the inventive principles are not limited to any particular kind of computer system but may be used with any general purpose computer, as would be known to one of ordinary skill in the art, arranged to perform the functions described and the method steps described. The operations of such a computer, as described above, may be according to a computer program contained on a medium for use in the operation or control of the computer, as would be known to one of ordinary skill in the art. The computer medium, which may be used to hold or contain the computer program product, may be a fixture of the computer such as an embedded memory or may be on a transportable medium such as a disk, as would be known to one of ordinary skill in the art.
- The invention is not limited to any particular computer program or logic or language, or instruction but may be practiced with any such suitable program, logic or language, or instructions as would be known to one of ordinary skill in the art. Without limiting the principles of the disclosed invention any such computing system can include, inter alia, at least a computer readable medium allowing a computer to read data, instructions, messages or message packets, and other computer readable information from the computer readable medium. The computer readable medium may include non-volatile memory, such as ROM, Flash memory, floppy disk, Disk drive memory, CD-ROM, and other permanent storage. Additionally, a computer readable medium may include, for example, volatile storage such as RAM, buffers, cache memory, and network circuits.
- Furthermore, the computer readable medium may include computer readable information in a transitory state medium such as a network link and/or a network interface, including a wired network or a wireless network that allows a computer to read such computer readable information.
- Although specific embodiments of the invention have been disclosed, those having ordinary skill in the art will understand that changes can be made to the specific embodiments without departing from the spirit and scope of the invention. The scope of the invention is not to be restricted, therefore, to the specific embodiments, and it is intended that the appended claims cover any and all such applications, modifications, and embodiments within the scope of the present invention.
Claims (10)
1. An information processing system for performing a system test on a program, the information processing system comprising:
a memory;
a processor communicatively coupled to the memory;
a system test module communicatively coupled to the memory and the processor, wherein the system test module is for:
creating a test plan associated with a system test, wherein the system test is for testing a program within an environment;
defining within the test plan at least one test trigger to be monitored for during the system test, wherein the at least one test trigger is a self-learning test trigger,
wherein the self-learning test trigger creates new test triggers within the test plan based on an event associated with the self-learning test trigger occurring;
monitoring execution of the system test on a system under test for the at least one test trigger;
determining an occurrence of the at least one test trigger;
modifying, in response to determining the occurrence, the test plan to take into account the occurrence of the at least one test trigger; and
continuing execution of the system test based on the modified test plan.
2. The information processing system of claim 1 , wherein the system test module is further for:
creating, in response to the determining the occurrence, at least one additional test trigger based on the at least one test trigger having occurred; and
modifying the test plan to include the at least one additional test trigger.
3. The information processing system of claim 2 , wherein the creating the test plan is performed based upon user input.
4. The information processing system of claim 2 , wherein the creating the at least on additional test trigger is performed by an automated testing tool.
5. The information processing system of claim 2 , wherein the system test module is further for:
monitoring the system test for the at least one test trigger and the at least one additional test trigger.
6. A computer readable medium for performing a system test on a program, the computer readable medium comprising instructions for:
creating a test plan associated with a system test, wherein the system test is for testing a program within an environment;
defining within the test plan at least one test trigger to be monitored for during the system test, wherein the at least one test trigger is a self-learning test trigger, wherein the self-learning test trigger creates new test triggers within the test plan based on an event associated with the self-learning test trigger occurring;
monitoring execution of the system test on a system under test for the at least one test trigger;
determining an occurrence of the at least one test trigger;
modifying, in response to determining the occurrence, the test plan to take into account the occurrence of the at least one test trigger; and
continuing execution of the system test based on the modified test plan.
7. The computer readable medium of claim 6 , further comprising instructions for:
creating, in response to the determining the occurrence, at least one additional test trigger based on the at least one test trigger having occurred; and
modifying the test plan to include the at least one additional test trigger.
8. The computer readable medium of claim 7 , wherein the creating the test plan is performed based upon user input.
9. The computer readable medium of claim 7 , wherein the creating the at least on additional test trigger is performed by an automated testing tool.
10. The computer readable medium of claim 7 , further comprising instructions for:
monitoring the system test for the at least one test trigger and the at least one additional test trigger.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/926,063 US20080162995A1 (en) | 2007-01-03 | 2007-10-28 | In-cycle system test adaptation |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/619,412 US7359820B1 (en) | 2007-01-03 | 2007-01-03 | In-cycle system test adaptation |
US11/926,063 US20080162995A1 (en) | 2007-01-03 | 2007-10-28 | In-cycle system test adaptation |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/619,412 Continuation US7359820B1 (en) | 2007-01-03 | 2007-01-03 | In-cycle system test adaptation |
Publications (1)
Publication Number | Publication Date |
---|---|
US20080162995A1 true US20080162995A1 (en) | 2008-07-03 |
Family
ID=39281674
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/619,412 Active US7359820B1 (en) | 2007-01-03 | 2007-01-03 | In-cycle system test adaptation |
US11/926,063 Abandoned US20080162995A1 (en) | 2007-01-03 | 2007-10-28 | In-cycle system test adaptation |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/619,412 Active US7359820B1 (en) | 2007-01-03 | 2007-01-03 | In-cycle system test adaptation |
Country Status (1)
Country | Link |
---|---|
US (2) | US7359820B1 (en) |
Cited By (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101876942A (en) * | 2009-04-30 | 2010-11-03 | 卓望数码技术(深圳)有限公司 | Terminal software testing method and device |
US20110066887A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to provide continuous calibration estimation and improvement options across a software integration life cycle |
US20110067005A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to determine defect risks in software solutions |
US20110066890A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method for analyzing alternatives in test plans |
US20110066893A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to map defect reduction data to organizational maturity profiles for defect projection modeling |
US20110066558A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to produce business case metrics based on code inspection service results |
US20110066486A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method for efficient creation and reconciliation of macro and micro level test plans |
US20110066557A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to produce business case metrics based on defect analysis starter (das) results |
US20110067006A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to classify automated code inspection services defect output for defect analysis |
US20110066490A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method for resource modeling and simulation in test planning |
US8635056B2 (en) | 2009-09-11 | 2014-01-21 | International Business Machines Corporation | System and method for system integration test (SIT) planning |
CN109977016A (en) * | 2019-03-28 | 2019-07-05 | 恒生电子股份有限公司 | A kind of test triggering method, device and equipment and a kind of test macro |
CN111427802A (en) * | 2020-06-09 | 2020-07-17 | 南京大学 | Test method and system for carrying out test case priority sequencing by utilizing ensemble learning |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070260937A1 (en) * | 2006-04-13 | 2007-11-08 | Carli Connally | Systems and methods for selectively logging test data |
US7359820B1 (en) * | 2007-01-03 | 2008-04-15 | International Business Machines Corporation | In-cycle system test adaptation |
US20110173591A1 (en) * | 2010-01-13 | 2011-07-14 | Target Brands, Inc. | Unit Test Generator |
US8661454B2 (en) * | 2010-01-26 | 2014-02-25 | Target Brands, Inc. | System and method for receiving and transmitting event information |
KR101906823B1 (en) * | 2016-03-07 | 2018-12-05 | 주식회사 럭스로보 | Multi-module compilation system, multi-module compilation method, and non-transitory computer-readable storage medium |
FR3053809B1 (en) * | 2016-07-07 | 2018-08-17 | Alstom Transport Technologies | METHOD FOR TESTING A GRAPHICAL INTERFACE AND ASSOCIATED TESTING SYSTEM |
KR20180073300A (en) * | 2016-12-22 | 2018-07-02 | 삼성전자주식회사 | Scan data control apparatus and electronic system having the same |
US10353804B1 (en) * | 2019-01-22 | 2019-07-16 | Capital One Services, Llc | Performance engineering platform and metric management |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5265101A (en) * | 1987-09-14 | 1993-11-23 | Texas Instruments Incorporated | Function array sequencing for VLSI test system |
US6078189A (en) * | 1996-12-13 | 2000-06-20 | International Business Machines Corporation | Dynamic test reordering |
US20020155628A1 (en) * | 2001-04-20 | 2002-10-24 | International Business Machines Corporation | Method for test optimization using historical and actual fabrication test data |
US20020193892A1 (en) * | 2001-05-29 | 2002-12-19 | International Business Machines Corporation | Method and system for including parametric in-line test data in simulations for improved model to hardware correlation |
US20050102580A1 (en) * | 2001-08-24 | 2005-05-12 | House Richard W. | Test configuration and data management system and associated method for enterprise test operations |
US6915177B2 (en) * | 2002-09-30 | 2005-07-05 | Advanced Micro Devices, Inc. | Comprehensive integrated lithographic process control system based on product design and yield feedback system |
US6929962B1 (en) * | 2004-03-26 | 2005-08-16 | Taiwan Semiconductor Manufacturing Co., Ltd. | System and method for wafer acceptance test configuration |
US7359820B1 (en) * | 2007-01-03 | 2008-04-15 | International Business Machines Corporation | In-cycle system test adaptation |
-
2007
- 2007-01-03 US US11/619,412 patent/US7359820B1/en active Active
- 2007-10-28 US US11/926,063 patent/US20080162995A1/en not_active Abandoned
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5265101A (en) * | 1987-09-14 | 1993-11-23 | Texas Instruments Incorporated | Function array sequencing for VLSI test system |
US6078189A (en) * | 1996-12-13 | 2000-06-20 | International Business Machines Corporation | Dynamic test reordering |
US20020155628A1 (en) * | 2001-04-20 | 2002-10-24 | International Business Machines Corporation | Method for test optimization using historical and actual fabrication test data |
US20020193892A1 (en) * | 2001-05-29 | 2002-12-19 | International Business Machines Corporation | Method and system for including parametric in-line test data in simulations for improved model to hardware correlation |
US20050102580A1 (en) * | 2001-08-24 | 2005-05-12 | House Richard W. | Test configuration and data management system and associated method for enterprise test operations |
US6915177B2 (en) * | 2002-09-30 | 2005-07-05 | Advanced Micro Devices, Inc. | Comprehensive integrated lithographic process control system based on product design and yield feedback system |
US6929962B1 (en) * | 2004-03-26 | 2005-08-16 | Taiwan Semiconductor Manufacturing Co., Ltd. | System and method for wafer acceptance test configuration |
US7359820B1 (en) * | 2007-01-03 | 2008-04-15 | International Business Machines Corporation | In-cycle system test adaptation |
Cited By (34)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101876942A (en) * | 2009-04-30 | 2010-11-03 | 卓望数码技术(深圳)有限公司 | Terminal software testing method and device |
US8667458B2 (en) | 2009-09-11 | 2014-03-04 | International Business Machines Corporation | System and method to produce business case metrics based on code inspection service results |
US20110067006A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to classify automated code inspection services defect output for defect analysis |
US20110066890A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method for analyzing alternatives in test plans |
US8689188B2 (en) | 2009-09-11 | 2014-04-01 | International Business Machines Corporation | System and method for analyzing alternatives in test plans |
US20110066558A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to produce business case metrics based on code inspection service results |
US20110066486A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method for efficient creation and reconciliation of macro and micro level test plans |
US20110066557A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to produce business case metrics based on defect analysis starter (das) results |
US8893086B2 (en) * | 2009-09-11 | 2014-11-18 | International Business Machines Corporation | System and method for resource modeling and simulation in test planning |
US20110066490A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method for resource modeling and simulation in test planning |
US8527955B2 (en) | 2009-09-11 | 2013-09-03 | International Business Machines Corporation | System and method to classify automated code inspection services defect output for defect analysis |
US8539438B2 (en) | 2009-09-11 | 2013-09-17 | International Business Machines Corporation | System and method for efficient creation and reconciliation of macro and micro level test plans |
US8566805B2 (en) | 2009-09-11 | 2013-10-22 | International Business Machines Corporation | System and method to provide continuous calibration estimation and improvement options across a software integration life cycle |
US8578341B2 (en) | 2009-09-11 | 2013-11-05 | International Business Machines Corporation | System and method to map defect reduction data to organizational maturity profiles for defect projection modeling |
US8924936B2 (en) | 2009-09-11 | 2014-12-30 | International Business Machines Corporation | System and method to classify automated code inspection services defect output for defect analysis |
US8645921B2 (en) | 2009-09-11 | 2014-02-04 | International Business Machines Corporation | System and method to determine defect risks in software solutions |
US20110066887A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to provide continuous calibration estimation and improvement options across a software integration life cycle |
US20110066893A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to map defect reduction data to organizational maturity profiles for defect projection modeling |
US20110067005A1 (en) * | 2009-09-11 | 2011-03-17 | International Business Machines Corporation | System and method to determine defect risks in software solutions |
US8635056B2 (en) | 2009-09-11 | 2014-01-21 | International Business Machines Corporation | System and method for system integration test (SIT) planning |
US9052981B2 (en) | 2009-09-11 | 2015-06-09 | International Business Machines Corporation | System and method to map defect reduction data to organizational maturity profiles for defect projection modeling |
US9176844B2 (en) | 2009-09-11 | 2015-11-03 | International Business Machines Corporation | System and method to classify automated code inspection services defect output for defect analysis |
US9262736B2 (en) | 2009-09-11 | 2016-02-16 | International Business Machines Corporation | System and method for efficient creation and reconciliation of macro and micro level test plans |
US9292421B2 (en) | 2009-09-11 | 2016-03-22 | International Business Machines Corporation | System and method for resource modeling and simulation in test planning |
US9442821B2 (en) | 2009-09-11 | 2016-09-13 | International Business Machines Corporation | System and method to classify automated code inspection services defect output for defect analysis |
US9558464B2 (en) | 2009-09-11 | 2017-01-31 | International Business Machines Corporation | System and method to determine defect risks in software solutions |
US9594671B2 (en) | 2009-09-11 | 2017-03-14 | International Business Machines Corporation | System and method for resource modeling and simulation in test planning |
US9710257B2 (en) | 2009-09-11 | 2017-07-18 | International Business Machines Corporation | System and method to map defect reduction data to organizational maturity profiles for defect projection modeling |
US9753838B2 (en) | 2009-09-11 | 2017-09-05 | International Business Machines Corporation | System and method to classify automated code inspection services defect output for defect analysis |
US10185649B2 (en) | 2009-09-11 | 2019-01-22 | International Business Machines Corporation | System and method for efficient creation and reconciliation of macro and micro level test plans |
US10235269B2 (en) | 2009-09-11 | 2019-03-19 | International Business Machines Corporation | System and method to produce business case metrics based on defect analysis starter (DAS) results |
US10372593B2 (en) | 2009-09-11 | 2019-08-06 | International Business Machines Corporation | System and method for resource modeling and simulation in test planning |
CN109977016A (en) * | 2019-03-28 | 2019-07-05 | 恒生电子股份有限公司 | A kind of test triggering method, device and equipment and a kind of test macro |
CN111427802A (en) * | 2020-06-09 | 2020-07-17 | 南京大学 | Test method and system for carrying out test case priority sequencing by utilizing ensemble learning |
Also Published As
Publication number | Publication date |
---|---|
US7359820B1 (en) | 2008-04-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7359820B1 (en) | In-cycle system test adaptation | |
US9703694B2 (en) | Techniques for testing software | |
US10127143B2 (en) | Generating an evolving set of test cases | |
US8561024B2 (en) | Developing software components and capability testing procedures for testing coded software component | |
US20060129892A1 (en) | Scenario based stress testing | |
US10459435B2 (en) | Test manager for industrial automation controllers | |
US20150100829A1 (en) | Method and system for selecting and executing test scripts | |
US20130311827A1 (en) | METHOD and APPARATUS for automatic testing of automation software | |
CN111124919A (en) | User interface testing method, device, equipment and storage medium | |
US10459830B2 (en) | Executable code abnormality detection | |
US20150100831A1 (en) | Method and system for selecting and executing test scripts | |
CN111309570A (en) | Pressure testing method, medium, device and computing equipment | |
US9195562B2 (en) | Recording external processes | |
CN110990289B (en) | Method and device for automatically submitting bug, electronic equipment and storage medium | |
US20080010536A1 (en) | Breakpoints with Separate Conditions | |
CN105653455B (en) | A kind of detection method and detection system of program bug | |
US20210406158A1 (en) | Systems and methods for automated device testing | |
US8855801B2 (en) | Automated integration of feedback from field failure to order configurator for dynamic optimization of manufacturing test processes | |
US10579761B1 (en) | Method and system for reconstructing a graph presentation of a previously executed verification test | |
CN109901831A (en) | The multi-platform compatibility operation method and compatibility operation device of software | |
US10467131B1 (en) | Method and system for performance analysis by test automation frameworks | |
US20180203787A1 (en) | Detection of software errors | |
JP2009104490A (en) | Apparatus for testing program | |
KR101137034B1 (en) | System and method for distributed runtime diagnostics in hierarchical parallel environments | |
US9116875B2 (en) | Test circuit and method for processing a test routine |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO PAY ISSUE FEE |