Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/36—Preventing errors by testing or debugging software
  • G06F11/3668—Software testing
  • G06F11/3672—Test management
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/30—Monitoring
  • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
  • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
  • G06F11/3414—Workload generation, e.g. scripts, playback
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/36—Preventing errors by testing or debugging software
  • G06F11/3668—Software testing
  • G06F11/3672—Test management
  • G06F11/3684—Test management for test design, e.g. generating new test cases
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F11/00—Error detection; Error correction; Monitoring
  • G06F11/36—Preventing errors by testing or debugging software
  • G06F11/3668—Software testing
  • G06F11/3672—Test management
  • G06F11/3688—Test management for test execution, e.g. scheduling of test suites
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F8/00—Arrangements for software engineering
  • G06F8/30—Creation or generation of source code
  • G06F8/38—Creation or generation of source code for implementing user interfaces
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/44—Arrangements for executing specific programs
  • G06F9/451—Execution arrangements for user interfaces
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L43/00—Arrangements for monitoring or testing data switching networks
  • H04L43/50—Testing arrangements

Definitions

• aspects of the present invention relate to computer systems. More particularly, aspects of the present invention relate to testing of computer systems.
• Figure 2 shows a conventional testing system including a server 201 providing a list of test processes to agent 202.
• a test developer at agent 202 selects the specific tests to be run and runs them as shown by processes 1-5 203-207.
• the developer needs to specify each individual testing as well as ensure that the requirements for each test have been met prior to running each test.
• a mistake in not enabling or instantiating a required test element directly translates into a loss of a test run as expected tests cannot be completed.
• Figure 1 shows a general-purpose computing environment in accordance with aspects of the present invention.
• Figure 2 shows a conventional testing system.
• Figure 3 shows processes for assembling and/or deploying test packages in accordance with aspects of the present invention.
• Figure 4 shows an illustrative user interface in accordance with aspects of the present invention.
• Figure 5 shows various alternative processes for assembling and/or deploying test packages in accordance with aspects of the present invention.
• Figure 6 shows yet another process for assembling and/or deploying test packages in accordance with aspects of the present invention.
• Figure 7 shows an illustrative example of relationships between testing packages and related components in accordance with aspects of the present invention.
• aspects of the present invention relate to defining and/or deploying test packages to make testing of computer systems (hardware, software, firmware, and the like) easier for computer developers.
• an exemplary system for implementing the invention includes a computing device, such as computing device 100.
• computing device 100 typically includes at least one processing unit 102 and memory 104.
• memory 104 may be volatile (such as RAM), non- volatile (such as ROM, flash memory, etc.) or some combination of the two.
• This most basic configuration is illustrated in Figure 1 by dashed line 106.
• device 100 may also have additional features/functionality.
• device 100 may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in Figure 1 by removable storage 108 and non-removable storage 110.
• Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
• Memory 104, removable storage 108 and non-removable storage 110 are all examples of computer storage media.
• Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by device 100. Any such computer storage media may be part of device 100.
• Device 100 may also contain communications connection(s) 112 that allow the device to communicate with other devices.
• Communications connection(s) 112 is an example of communication media.
• Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.
• modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
• communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media.
• the term computer readable media as used herein includes both storage media and communication media.
• Device 100 may also have input device(s) 114 such as keyboard, mouse, pen, voice input device, touch input device, etc.
• Output device(s) 116 such as a display, speakers, printer, etc. may also be included. All these devices are well know in the art and need not be discussed at length here.
• testing of computer systems can be a time-consuming and tedious process.
• Automated testing requires the running of an application on a test machine.
• the test application and any dependencies have to be preconfigured on a test machine before the test is executed.
• These dependencies include files, environment variable settings, registry settings, and commands. There can be a significant number of dependencies, of which failing to enable one will jeopardize the validity of a test run.
• Manual testing is another commonly used testing system. Manual testing includes having a user physically control a system to approach a desired condition and then monitoring the condition. For instance, this may include a game developer controlling a game to reach a desired point then evaluate performance or rendering of the game. Consistently being able to reach the same predefined location may be jeopardized by modifications to the environment, thereby making consistent testing difficult.
• a modified version of automated and manual testing may also be used.
• "semi- automated testing” may be used to automate some portion of the testing process (e.g. system configuration) with manual verification and logging by a developer to follow.
• testing packages may be used to help define, organize, and/or pre-configure tests for deployment.
• related testing packages may be combined in which they may inherit properties from one another. Multiple tests may inherit from any one package. Each test may be associated with a package, which includes the inheritable information.
• Computer scripts may be generated ahead of time or as needed.
• One advantage of generating the computer scripts on a just-in-time basis is the reduction in administrative overhead.
• Figure 3 shows a process for creating and deploying the test packages.
• a developer uses a testing packages user interface 301 to perform a number of operations with server 302. The developer may perform the following operations:
• an agent 303 on a test machine queries server 302 for packages associated with a test case.
• the server may then start with a topmost package and walk through a package inheritance chain and return contents to the agent 303.
• the agent 303 then generates script using dependency descriptions as shown in box 306.
• agent 303 executes the generated script or may store or cause the script to be stored for later execution (locally or remotely).
• Script is then invoked by the agent 303 to pre-configure test machines.
• the script may perform a number of other operations including setting environment variables, setting registry entries, copying any required files, monitoring relevant applications, and scheduling any cleanup.
• Package-specific processes 304-306 may then be run on one or more test machines. For instance, package-specific processes 304 may be run on a first test machine while package-specific process 306 is run on a second test machine.
• the script may or may not perform simple cleanup operations after each test has been completed or when the collective set of tests has been completed.
• Figure 4 shows an illustrative user interface for configuring test packages in accordance with aspects of the present invention.
• Alternative user interfaces may include more or less sections or span across multiple interfaces.
• Figure 4 includes user interface 401.
• the user interface provides a developer with the ability to define a package and position it within a package hierarchy.
• the various packages are shown in region 402.
• the top-level package is "Main".
• Subordinate packages include UIMain, UIl (with subordinate packages UIlA and UIlB), UI2, UI3, UI4, and UI5.
• TextBoxMain includes subordinate packages TextBox 1, TextBox 2, and TextBox 3 (with subordinate packages TextBox 3A and TextBox 3B). Other packages may also be included.
• Various add and delete buttons may be used to increase and decrease the files/environment variables, command lines, registry entries, and the like that may be associated with a testing package.
• testing package UIMain has been selected.
• Information relating to testing package UIMain is shown in regions 403-407 of Figure 4.
• Region 403 shows the file and/or files referenced by testing package UIMain (possibly including the file name, source, destination, and whether the file needs to be registered (e.g., a COM/DLL).
• Another file Test.vbs is also shown.
• Region 404 shows environment variables related to testing package UIMain (possibly including a name associated with the environment variable and its setting).
• Region 405 shows a command line or lines associated with Test.vbs (including the command-line as well as an indication whether it should be invoked during cleanup).
• Region 406 shows registry entries (including a root registry entry, a path, registry type, and data associated with the registry entry).
• Region 407 shows a listing of related packages.
• package UIMain is related to and derived from a global package (referenced as _Global) and OS_Main_IP.
• Dependent packages are derived from UIMain. These dependent packages include Workspace, Parsing, Stylus Input, Text Box Automation, Mouse Input and Keyboard Input testing packages.
• Region 407 also includes regions that allow one to specify new dependencies for packages and to re ⁇ order package dependencies (for instance, the order in which they may be executed).
• the following pseudo-code example represents script that may be created by agent 303 running on a test machine, after the agent 303 retrieved packages from server 303.
• the following script example configures the test machine with related dependencies and then runs the test application.
• the list may be sorted from least dependent to most dependent package.
• cleanup of testing procedures is optional in this example. The cleanup may occur after each test is performed, after some tests have been performed, or after all tests have been performed, if at all.
• Figure 5 shows various processes that may be used to deploy testing packages.
• test packages are defined. Before, after, or simultaneously, test cases may be defined in step 502. In some situations, test cases may be defined well ahead of associating tests with test packages.
• a test case or cases may be associated with a package or packages.
• a test pass is defined.
• the testing package or packages are deployed.
• Step 506 shows the automated reporting of results.
• automated testing may be combined with manual testing to ease burdens on developers needing to visually inspect aspects of a computer system that may be tedious to arrive at consistently. This may be referred to as semi-automated testing.
• Step 507 shows the reporting of semi-automated reporting of results.
• Step 508 shows the optional logging of manual results associated with manual verifications made in association with the test pass.
• FIG. 6 shows a manual implementation of testing packages.
• test cases are defined.
• a test pass or test passes are defined. Step 601 or step 602 may occur first. Alternatively, both steps may occur at the same time.
• a user logs manual results from running of the testing system.
• the use of testing packages may allow a developer to more quickly arrive at a testing environment during running of a computer system.
• FIG. 7 shows various relationships between objects, thereby creating a hierarchy of tests and testing packages.
• Testing package 701 includes package identification, a package name, and an indication of the owner of the package.
• Testing package 701 is associated with package dependency 702.
• Package dependency 702 includes host package identification, a guest package identification, and the order of the package dependency. Two references are shown from package dependency 702 to package 701. The first relates to the host package ID (naming the package itself) and the second relates to the guest package ID (the package from which the host package inherits).
• Environment entry 703 includes an environment entry identification, a package identification, an environment variable name, and a setting.
• registry entry 704. includes a registry entry identification, a package identification, a root indication for the registry entry, a path, type, and data fields.
• Package 701 further relates to file 705.
• File 705 includes a file identification, a package identification, a server location, file name, local target, and an indication whether the file needs to be registered (including but not limited to a COM executable or DLL file).
• command-line 706 includes a command-line identification, a package identification, a cleanup operation, and command text.
• TestUI_CMD %RHINO_JOB% ⁇ TestUI.exe /xml:TestCases.xml /a /c /coverage /filter: " "all+ fxcop- setup- monitor+”"
• TestUI_CMD %RHINO_JOB% ⁇ TestUI.exe /xml:TestCases.xml /a /c /filter:" "all+ fxcop- setup- monitor+”"
• SDK15_InstallPoint W serverl ⁇ Builds ⁇ ArchivedBlds ⁇ TPGvl.5-RTM- 2600.xxxx.3027.0 ⁇ Retail ⁇ WispSetup
• PARSER_LOG_FILES W serverlSScriptDataSPlatformSParsingsParserTestSAutomationLog sSM8Logs 7.
• PARSER_ACTIONSCRIPT_FILES W serverlSScriptDataSPlatformSParsingsParserTestSAutomationLog sSM8Logs 7.
• PARSER_METRIC_CONFIG_FILES ⁇ serverl ⁇ ScriptData ⁇ Platform ⁇ Parsing ⁇ ParserTest ⁇ MetricConfig
• PARSER_ENGINE_CONFIG_FILES XX serverlXScriptDataXPlatformXParsingXParserTestXEngineConfig
• PARSER_METRIC_CONFIG_FILES XX server1XScriptDataXPlatform ⁇ Parsing ⁇ ParserTest ⁇ MetricConfig
• VSPATH %programfiles% ⁇ Microsoft Visual Studio .NET 2003 ⁇ Common7XIDE

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Physics & Mathematics (AREA)
• Quality & Reliability (AREA)
• Computer Hardware Design (AREA)
• Software Systems (AREA)
• Computer Networks & Wireless Communication (AREA)
• Signal Processing (AREA)
• Human Computer Interaction (AREA)
• Debugging And Monitoring (AREA)
• User Interface Of Digital Computer (AREA)
• Test And Diagnosis Of Digital Computers (AREA)

Priority Applications (5)

Applications Claiming Priority (1)

Publications (1)

Family

ID=35967774

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (1)

Citations (4)

Family Cites Families (1)

• 2004
  • 2004-08-06 WO PCT/US2004/025184 patent/WO2006022681A1/en active Application Filing
  • 2004-08-06 CN CNA2004800435559A patent/CN101019103A/zh active Pending
  • 2004-08-06 JP JP2007524774A patent/JP2008509472A/ja active Pending
  • 2004-08-06 EP EP04780083A patent/EP1782196A4/en not_active Withdrawn
  • 2004-08-06 KR KR1020077005325A patent/KR101038877B1/ko not_active IP Right Cessation

Patent Citations (4)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events