EP1782196A1

EP1782196A1 - Testing packages

Info

Publication number: EP1782196A1
Application number: EP04780083A
Authority: EP
Inventors: John Wesley Walker
Original assignee: Microsoft Corp
Current assignee: Microsoft Corp
Priority date: 2004-08-06
Filing date: 2004-08-06
Publication date: 2007-05-09
Also published as: WO2006022681A1; KR101038877B1; EP1782196A4; KR20070040412A; CN101019103A; JP2008509472A

Abstract

A system, method, and data structure for testing packages are described. Tests may be combined into testing packages at testing packages user interface (301) to make testing of computer systems easier for developers. Testing packages may be sets of one or more testing procedures that may be applied to computer systems from a testing packages user interface (301). Testing packages may inherit information based on their relationship to other testing packages.

Description

Testing Packages

BACKGROUND OF THE INVENTION

FIELD OF THE INVENTION

[01] Aspects of the present invention relate to computer systems. More particularly, aspects of the present invention relate to testing of computer systems.

DESCRIPTION OF RELATED ART

[02] Computer system developers desire to release bug-free systems and/or applications. Be it hardware, software, or firmware, all computer products undergo some level of testing. Conventional testing systems require test operators to individually specify each test to be run on a system. To run a test, the operator needs to determine the elements required for the test, instantiate the elements or enable the elements, then run a selected test. This combination of steps, while ideally simple, becomes exceedingly complex with complex software and/or hardware.

[03] Figure 2 shows a conventional testing system including a server 201 providing a list of test processes to agent 202. A test developer at agent 202 selects the specific tests to be run and runs them as shown by processes 1-5 203-207. For 1000 or more tests to be run, the developer needs to specify each individual testing as well as ensure that the requirements for each test have been met prior to running each test. A mistake in not enabling or instantiating a required test element directly translates into a loss of a test run as expected tests cannot be completed.

BRIEF SUMMARY OF THE INVENTION

[04] Aspects of the present invention addressed one or more of the issues described above, thereby providing an improved testing method and system for developers.

BRIEF DESCRIPTION OF THE DRAWINGS [05] Aspects of the present invention are illustrated by way of example and not limited in the accompanying figures in which like reference numerals indicate similar elements and in which:

[06] Figure 1 shows a general-purpose computing environment in accordance with aspects of the present invention.

[07] Figure 2 shows a conventional testing system.

[08] Figure 3 shows processes for assembling and/or deploying test packages in accordance with aspects of the present invention.

[09] Figure 4 shows an illustrative user interface in accordance with aspects of the present invention.

[10] Figure 5 shows various alternative processes for assembling and/or deploying test packages in accordance with aspects of the present invention.

[11] Figure 6 shows yet another process for assembling and/or deploying test packages in accordance with aspects of the present invention.

[12] Figure 7 shows an illustrative example of relationships between testing packages and related components in accordance with aspects of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[13] Aspects of the present invention relate to defining and/or deploying test packages to make testing of computer systems (hardware, software, firmware, and the like) easier for computer developers.

[14] The following description is separated into the following sections: general purpose computing environment; automated and manual testing; testing packages; and testing package relationships. General Purpose Computing Environment

[15] With reference to Figure 1, an exemplary system for implementing the invention includes a computing device, such as computing device 100. In its most basic configuration, computing device 100 typically includes at least one processing unit 102 and memory 104. Depending on the exact configuration and type of computing device, memory 104 may be volatile (such as RAM), non- volatile (such as ROM, flash memory, etc.) or some combination of the two. This most basic configuration is illustrated in Figure 1 by dashed line 106. Additionally, device 100 may also have additional features/functionality. For example, device 100 may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in Figure 1 by removable storage 108 and non-removable storage 110. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Memory 104, removable storage 108 and non-removable storage 110 are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by device 100. Any such computer storage media may be part of device 100.

[16] Device 100 may also contain communications connection(s) 112 that allow the device to communicate with other devices. Communications connection(s) 112 is an example of communication media. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer readable media as used herein includes both storage media and communication media.

[17] Device 100 may also have input device(s) 114 such as keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s) 116 such as a display, speakers, printer, etc. may also be included. All these devices are well know in the art and need not be discussed at length here.

Automated And Manual Testing

[18] Testing of computer systems can be a time-consuming and tedious process. Two types of testing exist: automated testing and manual testing. Automated testing requires the running of an application on a test machine. The test application and any dependencies have to be preconfigured on a test machine before the test is executed. These dependencies include files, environment variable settings, registry settings, and commands. There can be a significant number of dependencies, of which failing to enable one will jeopardize the validity of a test run.

[19] Manual testing is another commonly used testing system. Manual testing includes having a user physically control a system to approach a desired condition and then monitoring the condition. For instance, this may include a game developer controlling a game to reach a desired point then evaluate performance or rendering of the game. Consistently being able to reach the same predefined location may be jeopardized by modifications to the environment, thereby making consistent testing difficult.

[20] A modified version of automated and manual testing may also be used. Here, "semi- automated testing" may be used to automate some portion of the testing process (e.g. system configuration) with manual verification and logging by a developer to follow.

Testing Packages

[21] To make testing easier, testing packages may be used to help define, organize, and/or pre-configure tests for deployment. In at least some aspects of the present invention, related testing packages may be combined in which they may inherit properties from one another. Multiple tests may inherit from any one package. Each test may be associated with a package, which includes the inheritable information.

[22] One advantage of packaging tests together is that dependency descriptions may be related between tests. Here, one package may be built upon other packages. A change in one package may then cascade through all dependent packages and associated tests and update them in the process.

[23] Computer scripts may be generated ahead of time or as needed. One advantage of generating the computer scripts on a just-in-time basis is the reduction in administrative overhead.

[24] Figure 3 shows a process for creating and deploying the test packages. A developer uses a testing packages user interface 301 to perform a number of operations with server 302. The developer may perform the following operations:

a. Define testing packages

b. Define test cases

c. Associate test case with package

d. Define test pass and

e. Deploy package.

[25] Next, in a client environment, an agent 303 on a test machine queries server 302 for packages associated with a test case. The server may then start with a topmost package and walk through a package inheritance chain and return contents to the agent 303. The agent 303 then generates script using dependency descriptions as shown in box 306. Next, agent 303 executes the generated script or may store or cause the script to be stored for later execution (locally or remotely).

[26] Script is then invoked by the agent 303 to pre-configure test machines. The script may perform a number of other operations including setting environment variables, setting registry entries, copying any required files, monitoring relevant applications, and scheduling any cleanup.

[27] Package-specific processes 304-306 may then be run on one or more test machines. For instance, package-specific processes 304 may be run on a first test machine while package-specific process 306 is run on a second test machine.

[28] The script may or may not perform simple cleanup operations after each test has been completed or when the collective set of tests has been completed.

[29] Figure 4 shows an illustrative user interface for configuring test packages in accordance with aspects of the present invention. Alternative user interfaces may include more or less sections or span across multiple interfaces.

[30] Figure 4 includes user interface 401. The user interface provides a developer with the ability to define a package and position it within a package hierarchy. The various packages are shown in region 402. Here, the top-level package is "Main". Subordinate packages include UIMain, UIl (with subordinate packages UIlA and UIlB), UI2, UI3, UI4, and UI5. TextBoxMain includes subordinate packages TextBox 1, TextBox 2, and TextBox 3 (with subordinate packages TextBox 3A and TextBox 3B). Other packages may also be included. Various add and delete buttons may be used to increase and decrease the files/environment variables, command lines, registry entries, and the like that may be associated with a testing package.

[31] As shown in region 402, testing package UIMain has been selected. Information relating to testing package UIMain is shown in regions 403-407 of Figure 4. Region 403 shows the file and/or files referenced by testing package UIMain (possibly including the file name, source, destination, and whether the file needs to be registered (e.g., a COM/DLL). Another file Test.vbs is also shown. Region 404 shows environment variables related to testing package UIMain (possibly including a name associated with the environment variable and its setting). Region 405 shows a command line or lines associated with Test.vbs (including the command-line as well as an indication whether it should be invoked during cleanup). Region 406 shows registry entries (including a root registry entry, a path, registry type, and data associated with the registry entry). Region 407 shows a listing of related packages. Here, for instance, package UIMain is related to and derived from a global package (referenced as _Global) and OS_Main_IP. Dependent packages are derived from UIMain. These dependent packages include Workspace, Parsing, Stylus Input, Text Box Automation, Mouse Input and Keyboard Input testing packages. Region 407 also includes regions that allow one to specify new dependencies for packages and to re¬ order package dependencies (for instance, the order in which they may be executed).

[32] The following pseudo-code example represents script that may be created by agent 303 running on a test machine, after the agent 303 retrieved packages from server 303. The following script example configures the test machine with related dependencies and then runs the test application.

Dim p As Package 'an arbitrary package utilized in a test pass

Function GetPackageDependencies (Package x)

^■Returns a list of packages that "x" is dependent on (including "x" itself) .

'The list may be sorted from least dependent to most dependent package.

"The first package in this list is the "_Global" package.

"The last package in this list is "x" itself. End Function

For Each RequiredPackage In GetPackageDependencies (p)

SetEnvironmentVariables(ReguiredPackage)

SetRegistryValues (RequiredPackage)

CopyFiles (RequiredPackage)

InvokeExecutables (RequiredPackage) Next RequiredPackage

For Each RequiredPackage In GetPackageDependencies (p)

CleanupPackage(RequiredPackage) Next RequiredPackage

[33] It is noted that the cleanup of testing procedures is optional in this example. The cleanup may occur after each test is performed, after some tests have been performed, or after all tests have been performed, if at all.

[34] Figure 5 shows various processes that may be used to deploy testing packages. In step 501 test packages are defined. Before, after, or simultaneously, test cases may be defined in step 502. In some situations, test cases may be defined well ahead of associating tests with test packages. In step 503, a test case or cases may be associated with a package or packages. In step 504, a test pass is defined. In step 505, the testing package or packages are deployed. Step 506 shows the automated reporting of results.

[35] In one example, automated testing may be combined with manual testing to ease burdens on developers needing to visually inspect aspects of a computer system that may be tedious to arrive at consistently. This may be referred to as semi-automated testing. Step 507 shows the reporting of semi-automated reporting of results.

[36] Step 508 shows the optional logging of manual results associated with manual verifications made in association with the test pass.

[37] Figure 6 shows a manual implementation of testing packages. In step 601, test cases are defined. In step 602, a test pass or test passes are defined. Step 601 or step 602 may occur first. Alternatively, both steps may occur at the same time. In step 603, a user logs manual results from running of the testing system. The use of testing packages may allow a developer to more quickly arrive at a testing environment during running of a computer system.

Testing Package Relationships

[38] Figure 7 shows various relationships between objects, thereby creating a hierarchy of tests and testing packages. Testing package 701 includes package identification, a package name, and an indication of the owner of the package. Testing package 701 is associated with package dependency 702. Package dependency 702 includes host package identification, a guest package identification, and the order of the package dependency. Two references are shown from package dependency 702 to package 701. The first relates to the host package ID (naming the package itself) and the second relates to the guest package ID (the package from which the host package inherits).

[39] Also related to package 701 is environment entry 703. Environment entry 703 includes an environment entry identification, a package identification, an environment variable name, and a setting. [40] Further related to package 701 is registry entry 704. Registry entry 704 includes a registry entry identification, a package identification, a root indication for the registry entry, a path, type, and data fields.

[41] Package 701 further relates to file 705. File 705 includes a file identification, a package identification, a server location, file name, local target, and an indication whether the file needs to be registered (including but not limited to a COM executable or DLL file).

[42] Finally package 701 includes an indication of a command-line 706. Command-line 706 includes a command-line identification, a package identification, a cleanup operation, and command text.

[43] The following provides sample entries that may be used in conjunction with the package 701 and related objects of Figure 7.

a. Sample environment entries:

1. TestUI_CMD = %RHINO_JOB%\TestUI.exe /xml:TestCases.xml /a /c /coverage /filter: " "all+ fxcop- setup- monitor+""

2. TestUI_CMD = %RHINO_JOB%\TestUI.exe /xml:TestCases.xml /a /c /filter:" "all+ fxcop- setup- monitor+""

3. PARSER_ACTIONSCRIPT_FILES =

SSserverlSScriptDataSPlatformSParsingSParserTestSActionScri ptFilesSAPI-integration

4. SDK15_InstallPoint = W serverl\Builds\ArchivedBlds\TPGvl.5-RTM- 2600.xxxx.3027.0\Retail\WispSetup

5. PARSER_METRIC_CONFIG_FILES = W server1\ScriptData\PlatformSParsing\ParserTest\MetricConfig\ API-integration

6. PARSER_LOG_FILES = W serverlSScriptDataSPlatformSParsingsParserTestSAutomationLog sSM8Logs 7. PARSER_ACTIONSCRIPT_FILES =

WserverlXScriptDataXPlatforiτΛParsinαXParserTestXActionScrip tFiles

8. PARSER_ACTIONSCRIPT-FILES = W serverlXScriptDataXPlatformVParsingXParserTestXActionScriptF iles

9. PARSER_METRIC_CONFIG_FILES = \\ serverl\ScriptData\Platform\Parsing\ParserTest\MetricConfig

10. PARSER_ENGINE_CONFIG_FILES = W serverlXScriptDataXPlatformXParsingXParserTestXEngineConfig

11. PARSER_ENGINE_CONFIG_FILES = XX serverlXScriptDataXPlatformXParsingXParserTestXEngineConfig

12. PARSER_METRIC_CONFIG_FILES = XX server1XScriptDataXPlatform\Parsing\ParserTest\MetricConfig

13. VSPATH = %programfiles%\Microsoft Visual Studio .NET 2003\Common7XIDE

b. Sample Command Lines:

1. %RHINO_JOB%\cpparsertest.exe -logcon -c

Provider=sqloledb;Server=SERVERl;database=InkLogs;Trusted_Co nnection=yes

2. cscript.exe %RHINO_JOB%\MUISetupBVT.vbs x:\Xserverl\osinst\MuirpBVT t:2600.1106.2201.0 w:2600.1106.2201.0

3. xcopy /Y

%TESTSRC%\Test\Personalization\MHC\MHCResources\TestInkFiles X* ,. XMHCResourcesXTestlnkFilesX

4. xcopy /Y

%TESTSRC%\Test\Personalization\MHC\MHCResources\TestScripts\ * . XMHCResourcesXTestScriptsX

5. "%programfiles%\Microsoft Visual Studio\Common\MSDev98\Bin\msdev" . \inkeditautotest.vb6.exe

6. "%programFiles%\Internet Explorer\iexplore.exe" http: //serverl/windowsapplicationδ.exe 7. xcopy

%SERVERl_DATA_FILES%\Systems\ContextE2E\DataUs\*google*.pif

8. cscript.exe %RHINO_JOB%\SetupBVTV2.vbs x:\\serverl\osinst\V2setupBVT t:2600.1081.2201.0

9. cscript.exe %RHINO_JOB%\SetupBVT-Lonestar-removed.vbs x:%RHINO_JOB% t:2600.1081.2201.0

10. cscript.exe %RHINO_JOB%\SetupBVT.vbs x: \\serverl\osinst\VlsetupBVT t:2600.1081.2201.0

11. xcopy %TESTSRC%\TabletPC\Test\ProgrammaticScenarioCases*.xml .

12. xcopy %SERVERl_DATA_FILES%\Systems\ContextE2E\DataUs\*.pif .

13. %RHINOJOB%\signaturesafe -s (signatures.txt) - d(master_sig.txt) -r(sig_cmp_rpt.txt)

14. xcopy %SERVERl_DATA_FILES%\Systems\ContextE2E\DataITA\*.*

15. xcopy %SERVERl_DATA_FILES%\Systems\ContextE2E\DataESP\*.*

16. xcopy %SERVERl_DATA_FILES%\Systems\ContextE2E\DataITA\*.*

17. xcopy %SERVERl_DATA_FILES%\Systems\ContextE2E\DataESP\*.*

18. xcopy %SERVERl_DATA_FILES%\Systems\TIPScenario\ESP\*.pif

19. %RHINOJOB%\idlewriter -s - i (%ENLIST_DRIVE%\\%COM_AUT_DIR%\\* .idl) -o(autsig.txt)

20. %RHINOJOB%\idlewriter -s - i(%ENLIST_DRIVE%\\%COM_DLL_DIR%\\*.idl) -o(dllsig.txt)

21. %RHINO_JOB%\LaunchIt.bat Managedlnheritance.html ManagedInheritance_Control.dll c. Sample File Entries (including a server location, local target and filename if applicable):

1. Wserverl\ScriptData\Platform\Parsing\ParserTest\WritingDraw ingClassification\BVT&Acceptance WritingDrawing classification BVT data_Labeled.xml

2. WServerlXpublicXTabletPCXretailXtestXAPIsControls recoa_vb6.exe

3. %TPG_TESTROOT%\ManifestAPI\Performance\ PerfApp.exe

4. %TPG_TEST_APISCONTROLS% GestureSemiCircleLeft.dat

5. %TPG_TEST_APISCONTROLS% GestureTap.dat

d. Sample Registry Entries (including a root path, a type, and data):

1. HKLM SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\AutoAdminLogon REG_DW0RD 1

2. HKLM SOFTWARENMicrosoft:\Windows NT\CurrentVersion\Winlogon\AutoLogonCount REG_DW0RD 1

3. HKLM SOFTWARE\Microsoft\Windows

NT\CurrentVersion\Winlogon\DefaultDomainName REG_SZ REDMOND

4. HKLM SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\DefaultUserName REG_SZ Guest

[44] The present invention has been described in terms of preferred and exemplary embodiments thereof. Numerous other embodiments, modifications and variations within the scope and spirit of the appended claims will occur to persons of ordinary skill in the art from a review of this disclosure.

Claims

We claim:

1. A system for creating testing packages comprising: a user interface that receives instructions linking two or more testing programs into a testing package; a storage that receives an association between the testing package and said two or more testing programs; a client computer that processes testing package to generate testing procedures for a computer system.

2. The system according to claim 1, wherein said client computer dynamically generates said testing procedures from said testing package.

3. A computer-implemented method for creating testing packages comprising the steps of: defining testing packages; defining test cases; associating test cases with said testing packages; defining a test pass; and deploying said testing packages to a storage.

4. The computer-implemented method according to claim 3, further comprising the step of generating executable code from said packages.

5. The computer-implemented method according to claim 3, further comprising the step of running code generated from said testing packages to test a computer system.

6. The computer-implemented method according to claim 5, further comprising the step of logging manual results.

7. A computer readable medium having a program stored thereon for creating testing packages, said program comprising the steps of: defining testing packages; defining test cases; associating test cases with said testing packages; defining a test pass; and deploying said testing packages to a storage.

8. The computer readable medium according to claim 7, said program further comprising the step of generating executable code from said packages.

9. The computer readable medium according to claim 7, said program further comprising the step of running code generated from said testing packages to test a computer system.

10. The computer readable medium according to claim 9, said program further comprising the step of logging manual results.

11. In a computing system, a user interface for receiving user designations of testing packages comprising: a first region that displays testing packages; a second region that allows a user to modify relationships between packages.

12. The user interface according to claim 11, further comprising: a third region listing files associated with one of the testing packages.

13. The user interface according to claim 11, further comprising: a third region listing environment variables associated with a file that is associated with one of the testing packages.

14. The user interface according to claim 11 , further comprising: a third region listing registry settings associated with a file that is associated with one of the testing packages.

15. The user interface according to claim 11 , further comprising: a third region listing clean up settings associated with a file that is associated with one of the testing packages.

16. The user interface according to claim 11 , further comprising: a third region providing the user with the ability to add a new testing package or file.

17. A computer-readable medium having a data structure stored thereon, said data structure relating a testing package, said data structure comprising: a first data structure storing a testing package identification; a second data structure referencing said testing package identification; and a third data structure referencing a child testing package.

18. The computer-readable medium according to claim 17, wherein said testing package identified in said first data structure inherits from said child testing package.

19. The computer-readable medium according to claim 17, further comprising: a fourth data structure that provides one or more registry entries associated with said testing package.

20. The computer-readable medium according to claim 17, further comprising: a fourth data structure that provides one or more environment variables associated with a file associated with said testing package.

21. The computer-readable medium according to claim 17, further comprising: a fourth data structure that references a file associated with said testing package.

22. The computer-readable medium according to claim 17, further comprising: a fourth data structure that describes a command line usable with a file associated with said testing package.