JP2002535773A - System and method for testing and validating a device having an embedded operating system - Google Patents

Abstract

(57) [Abstract] To improve quality assurance, give many individuals a long time and cost savings, and streamline the product development process of targeting equipment utilizing commercial operating systems such as Windows CE (R). The systems and methods include test equipment that applies a comprehensive set of validation test programs. This system and method, the O / S Validator (1), includes a host graphical user interface harness (12), communication from the host to the target (3), at least one test suit (11), and a result capture method (12). It provides a fully automated design validation package for commercial operating systems (1000a) and Windows CE (R). O / S Validator (1) is a faster, more accurate automated test suite technology that can be used to test target hardware (1000, 1000) to test the ports of commercial operating systems (1000a) such as Windows CE (R). 9) to provide. In addition, the O / S Validator (1) is a comprehensive coat-and-pause specifically developed to intentionally stress operating system O / S, hardware licensing, OEM adaptation tiers (OAL), and hardware wear interactions. including. The testsuit (11) is aimed at identifying three major flaws, including heartware design, heartwear programming (trial / OAL), and operating system interaction. Special diagnostics focus on the operating system that has historically shown the most problems.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[Cross reference to related applications]

This patent application was filed on January 21, 1999, US Provisional Patent Application No. 60 / 116,824 entitled "CE VALIDATOR TEST SUITE", and filed on June 4, 1999,
OTOCOL ACKNOWLEDGEMENT BETWEEN HOMOGENEOUS SYSTEMS "is related to US Provisional Patent Application No. 60 / 137,629.

[0002]

【Technical field】

The present invention relates to product quality assurance and a test system and method for validating an operating system provided to a computerized product. More particularly, the present invention relates to product quality assurance and a test system and method for validating an operating system during the development of a computerized product. More specifically, the present invention relates to product quality assurance and general provision for computerized products, such as Windows CE® manufactured and sold by Microsoft Corporation (Redmond, Wash.). Test systems and methods for validating operating systems.

[0003]

BACKGROUND OF THE INVENTION

Developers are increasingly incorporating operating systems such as Windows CE (R) into many different types of computerized products, including set-top boxes, gaming systems, barcode scanners, and factory automation systems. Just as Windows CE (R) has increased, so has the demand for "generic" software development tools. Many tools and “general purpose” software kits are commercially available to save equipment design time and provide rapid equipment development, but are compatible with these new products, especially in the final stages of equipment development. No quick test system or method for verifying gender existed.

Conventionally, only two options for device testing of an operating system could be employed: (1) write test code in-house, and (2) outsource custom code development to another company. OEM to complete internal test plan
Merchants spend a lot of time training their staff, spending more time developing test code, and getting more prepared before their products can be tested with such code. I have to spend more time. Similarly, outsourced custom code development companies will spend a long time writing code. Thus, both options are time consuming and therefore costly.

[0005] The time-consuming operating system device testing options described above provide an integrated test system and a method to improve product quality, time and cost of many individual-months. There has long been a pressing need to provide savings and streamline the product development process. Among other things, it provides a system and method that overcomes the aforementioned problems and thereby improves product quality, provides time and cost savings for many individuals, streamlines the product development process, and implements equipment design. Systems and methods for testing and validating installed equipment with the embedded Windows CE (R) operating system are required to result in a fully automated design verification package for the consumer. ing.

[0006]

Summary of the Invention

Accordingly, the present invention is an operating system validator (also referred to herein as an O / S validator and referred to in the drawings) under the registered trademark of CEValidator ^™ of Applicants' assignee. This verifier is an automated test suite for testing ports of an operating system, such as Windows CE®, in target device hardware and / or newly developed software. The above-mentioned problem is solved by providing a test system including the method of (1). The O / S Validator contains a comprehensive code base, specifically O / S, device drivers, OEM adaptation hierarchy (OEM a
daptation layer (OAL), and was developed to intentionally stress hardware interactions. The test suite provided has three main flaws: hardware design, hardware programming (driver / OAL
), And identifying operating system interactions. Special diagnostics focus on the Windows CE (R) subsystem, which has historically shown the most problems. The test suite consists of almost 1500 tests, including stress test routines and feature and functional tests, and is a Windows CE (R)
Provides a complete analysis of the port. These tests are grouped by O / S Validator. The O / S Validator includes both test source code and executable programs for all tests.

For host components of the O / S Validator, such as a standard Windows® application that leverages the Microsoft® Windows® user interface to simplify the execution of testsuites and the collection of logging results An intuitive user interface is used. The O / S Validator distributes test suites as client / server applications. A graphical user interface (GUI) interacts with a small application CEHarness.exe running on the target device. This communication can occur over Ethernet, so that at least one host can run the testsuit against at least one target device.

If the target device fails the test, the O / S Validator generates useful error information. While the testsuit is running, the results are displayed in a plurality of dynamically generated log windows, as well as a configuration summary tab. The logging window contains the complete text of any test results. Obstacles are color coded red to facilitate identification. By the navigation button of the logging window, 1
Allows for rapid movement from one obstacle to another. The prologue and epilogue are also generated in their respective result files by the logging API in the test. Information such as processes executed in parallel, battery power level, and date and time of execution are automatically recorded in the results file and displayed in the log window. Summary information, such as program memory loss, storage memory loss, or total test run time, is provided in a log window tab. Also, summary information of any test results is collected and displayed on the summary tab of the configuration window. The Summary tab reports the number of pass (PASS) and fail (FAIL) test cases in real time. A breakdown of the number of passes and rejects for each test suit is also displayed.
The summary tab in the configuration window facilitates quick navigation to individual faults (failures), perhaps in thousands of test results. The exact source file and the line number corresponding to the logged fault are automatically reported by the O / S Validator logging API. Since the O / S Validator provides source code for all of its executables, the ability to go directly to the source code reporting the error is a powerful adjunct to the textual description of the fault It is.

[0009] Other features of the present invention are disclosed or will be apparent in the section entitled "Detailed description of the invention".

[0010] For a better understanding of the present invention, reference is made to the accompanying drawings.

DETAILED DESCRIPTION OF THE INVENTION FIG. 1 shows a computerized product 1000, (9), typical of computer workstations, set-top boxes, gaming systems, barcode scanners, and factory automation systems. A typical computerized product is currently provided with an embedded operating system indicated by reference numeral 1001a. As shown and as described herein, product 1000 includes a typical target device 9 having an operating system 1001a, such as an embedded Windows CE® operating system (O / S). . The computerized product 1000 functions as a stand-alone device, and the product 1000 has its own operating system 1001.
The O / S Validator 1 of the present invention is installed to test a for validity. Independent testing is a new test development and O / S Validato
It facilitates the debugging of reported flaws as an in-depth knowledge of r, and no infrastructure is required. However, in a more likely application, as shown in FIG. 2, the product 1000 is an O / S Validator 1 of the present invention for testing a target device 9 with an operating system 1001a in a manufacturing quality assurance test environment M. Functions as the host computer 4 on which is installed. Referring back to FIG. 1, a computerized product 10
00 comprises, for example, several sub-components, which include a control unit 1020 or a plurality of input / output ports 10
21, keyboard 1009, printer 1010, mouse 1011 and monitor 1
012. The sub-components 1009, 1010, 1011 and 1012 themselves can be testable target devices. The typical control unit 1020 itself consists of several sub-components, including a central processing unit 1001, a storage device such as a hard disk drive 1004, other memory components including a RAM 1002, a ROM 1003, a compact disc 1005, an audio It includes a component 1006, a network / server card 1007, and a modem 1008. Inevitably, to make the product 1000 function as a useful device, the operating system 1001a is included in the control unit.

FIG. 3 shows an O / S Validato including a graphical user interface (GUI) 2, an engine 3, a plurality of test suites 11, and a logging library 12.
Here are the main components of r1. GUI 2 and Engine 3 are O / S Validat
It communicates bidirectionally internally within or1 via a component designated by reference numeral 7 and called Harness.dll, which is described in more detail below.
FIG. 4 shows a host computer 4 including the O / S Validator 1. As shown, a plurality of target devices 9 are O / S1 to be tested in accordance with the present invention.
001a. The O / S Validator 1 is the OS 100 in the target device 9
In order to test a specific function under the control of 1a, a plurality of test configurations 2
It has the ability to generate test configurations like 1a, 21b and 21c. CEHarnes
The device-side component called s8 communicates with the engine 3 of the O / S Validator 1. As shown in FIGS. 5 and 5A, the CEHarness 8 can also communicate with the engine 3 of the O / S Validator 1 via the Ethernet means 4a. FIG.
This indicates that a plurality of hosts 4 can execute a test suite against a plurality of target devices 9 because communication is performed via Ethernet. In yet another alternative, the CEHarness 8 may communicate with the engine 3 of the O / S Validator 1 via a test suit execution connection 1021a, as shown in FIG. , NT host computer, the logging library 12 is also provided to the target device 9, and the test results are provided to the host computer 4 via the socket connection 1021b.

In operation, the O / S Validator 1 tests and validates a target device 9 with an embedded operating system, for example the Windows CE® operating system. Generally speaking, the O / S Validator 1 functions as follows. (1) verify Windows CE (R) port to target device, (2) perform stress and performance tests, (3) log and analyze results, (4) include multiple application program interfaces (APIs) Testing a plurality of functional areas (Tables 1 and 2 in FIGS. 7, 8A, 8B, and 8C)
(5) Perform multiple pass / fail tests on multiple testsuits, (6) Facilitate test customization on automated testsuits, (7) Host-side graphical test harness (8
) Stress evaluation of memory performance, (9) a means for building test automation including multiple APIs (see Table 3 in FIG. 9), and (10) a result analysis tool called CEAnalyzer. I will provide a. As previously mentioned,
Also, as shown in FIG. 10, the O / S Validator 1 includes both a test source code SC and an executable program EP (also referred to as a test execution file) for all tests. The single realization requirement of the test execution file EP is that the test cases “pass” and “fail” have two specific APIs, namely, WRITETESTPASS () and WR.
It is reported using ITETESTFAIL (). These macros are well-known prin
It has signatures similar to the tf () function, but their use creates a standard test case result format in a “results” file that can be applied to automated summarization and integrates reporting with the host user interface. O / S
The method of Validator 1 further includes mediating between GUI 2 and the test cases summarized in the test execution file EP, and providing a means for GUI 2 to distribute and coordinate the execution of tests. Test suite 11 is a language command suite (eg, PUT, RUN) that is a direct representation of the tasks required to distribute and execute tests.
, WAIT, DELETE). Other supported command suites are GET for retrieving files from the target device 9, RUNHOST for running the executable program EP on the host computer 4, useful for client / server testing, client / Waiting for a process to end on the host computer 4, which is also useful for server-based testing
WAITHOST, PUTSYSTEM to put files into the system directory of the device (/ Windows®), SLEEP for basic timing when everything else fails
MSGBOX for displaying a message box on the host machine, SETREG for adding or changing registry settings on the device, and DELREG for removing registry settings from the device. The first file comment suit is provided as a GUI2 description suit in addition to providing an internal documentation suit. The method of the O / S Validator 1 includes organizing the test suit files 5 according to a hierarchical arrangement in the directory structure of the host computer 4. As shown in FIG. 11, the test suit 11 is divided at the top level as being one of an automatic test suit Au, a manual test suit Ma, and a stress test suit SS. The automatic test suit Au is a test that does not require user intervention during execution. In contrast, the manual test suit Ma is a test that requires user intervention (for example, a keyboard and touch panel set). O
The / S Validator1 method pushes the input / output (I / O) processing power to its limit, runs with little or no usable program or object store memory, and runs a custom file system with multi-threaded parallel stress. Includes stressing the system through a stress test suit SS that can be lowered with it. Below this top level of the hierarchy, O / S Va
The method of the lidator 1 involves configuring the test suit 11 by functional areas, as shown generally in FIG.

As described above, FIG. 3 shows a graphical user interface (GUI) 2
, An engine 3, a plurality of test suites 11 and a logging library 12,
/ S Validator1 shows the main components. Visual Basic (VB
) Since the component works well for any level of user,
/ S Validator1 uses GUI2 as a visual basic component. The design of GUI2 is based on the concept of interfacing user input with the underlying components. Engine 3 maintains a core of functionality on host 4. The engine 3 reads out the plurality of suit files 5, analyzes (parses) them, and executes the command. Engine 3 is GUI2
And use the information to set up various execution options. Engine 3 is written in C / C ++ language. Engine 3 is HarnessLin
It is linked to GUI2 via a component called k.dll7, which is ActiveX controlled. HarnessLink.dll 7 is instantiated and called from the GUI 2 by various information passed to the engine 3 before starting execution. DLL link 7 functions to communicate between engine 3 and GUI 2 during execution, relay information, relay error messages, and relay some dynamic runtime commands. The target device 9 includes a device-side (as opposed to the host-side) component called CEHarness8. As shown in FIG. 4, CEHarnes
s is a C / C ++ program existing in the target device 9 and almost exclusively communicates with the engine 3 until the target information is broadcast on the network;
It communicates with the UI 2 to receive such information and passes it to the engine 3 (see FIGS. 5 and 5A). CEHarness 8 is an event-driven application, in which case engine 3 sends an event and CEHarness 8 responds. Since the testsuit 11 is written using a plurality of application program interfaces (APIs) 13 that are part of the logging library 12, the two remaining components, the testsuit 11 and the logging library 12, are involved. (FIG. 8
A, see FIGS. 8B and 8C). These APIs 13 have substantial functionality,
Depends somewhat on the information passed through the component chain. The logging library 12 has a simple functional concept, and stores the logging TCP / IP information 16 in the GUI2.
(See FIG. 5), or by writing the test result 14 and the log file 15 directly to the device 9 (see FIG. 6), the test result 14 is transmitted by generating the log file 15. As shown in FIG. 12, the logging window LW includes the test result 14, the failure F, and the summary tab SumT of the test file 15.
Indicate user access to program memory, pass, fail,
And facilitate timing information. A plurality of test suits 11 are O / S Valida
Contains the essential components of tor1. FIG. 13 shows that the test cycle time CT is
FIG. 4 shows, in the form of a graph, a decrease according to the number of test devices tested in parallel.

[0015] In further detail, GUI 2 is complex code because of the degree of functionality required to process the components of the hierarchy. GUI2 is "Wizard"
And its primary function is to guide the new user through a list of various selectable settings and default settings. As shown in FIG.
The GUI 2 also provides a configuration window CW as a means for performing a test run including a single pass through a selected set of test suits 11 on the target device 9. As shown in FIG. 4, multiple configurations 21a, 21b, 21c can be run to simulate various scenarios. The content of the configuration window CW includes a plurality of user-controlled tabs. For example, the suit tab S provides a tree view of the suit file directory under the O / S Validator directory. This tree view is organized and provides a meaningful distinction between the type of test 11 and user selection. Further, this tree view is generated when the user opens the configuration window CW, and the user adds the new user-entered suit to the suit file directory created by the O / S Validator 1 installation program. / S Validator1 can be extended. Test suit 1
Reference numeral 1 denotes a script composed of commands that the engine 3 reads out and performs an operation corresponding to the script command. Suit file 5 generally begins with a series of comments. These comments appear in the suit file information section of the file. The word “Manual” stops the suit file 5
, The file is considered to require manual execution and is therefore assigned a different icon. In the test suit section, the user can reorder the test suit files 5 in any way, as shown in FIG. Still referring to FIG. 14, the logging tab contains quite valuable information. The user can select three methods of logging: LH for logging to host 4, LTD for logging to target device 9, or LHD for logging to both host 4 and target device 9. next,
Log information is stored in a configurable directory listed in the edit box. All of this information is sent to the engine 3 via the DLL 7 and then to the CEHarness 8 of the target device 9. As a result, information is obtained by the logging library 12 when performing the test 11. Configuration window CW
Other tabs include a stress state setting tab SC, selecting a thread with a higher priority during execution of the test suite by selecting a thread tab T, and selecting program and storage memory by selecting tabs PM and SM. And select the tab SRT to select runtime and stop by selecting the tab STOP. Users can use the infinite loop tab Iloop to find memory leaks in the system. Useful summary information, such as program memory loss, storage memory loss, or total test execution time, is provided in a summary tab SumT. In addition, summary information for any test result is collected and displayed on the summary tab SumT. The Summary tab reports the number of passed and failed test cases in real time. A breakdown of the number of passes and rejects for each test suit is also displayed. The summary tab in the configuration window facilitates quick navigation to individual faults, perhaps in thousands of test results. The exact source file and line number corresponding to the logged fault are
It is automatically reported by the logging API of Validator. Since the O / S Validator provides source code for all of its executables, being able to go directly to the source code reporting the error is a powerful addition to the textual description of the fault.

[0016] Logging options vary dramatically in their implementation and effectiveness. The first option is to “pass” whenever the user runs the testsuit 11
, A set of log files 15, summarizing these test results 14, presumed to result in an automatic publication. Depending on the logging method selected, a summary file 15 is generated by CEHarness 8 or engine 3. Basically, the engine 3 traverses all log files 16 in the logging directory, so that the user may receive a list of log files that does not correspond to the test 11 being executed. To make the summary file 15 more indicative of the execution test 11, the user can delete the log directory before execution, or
A logging option can be selected to select "Return only the newest summary results", whereby engine 3 traverses only one log file 15 for each test 11. This means that if a user runs a file system test 30 times on any given day, there will be only one entry in the summary log for that test corresponding to the most recent run. The summary log has only one entry for each test 11, and the log file 15 for each test 11 still exists in the logged directory. The other two options are handled in GUI2.
When the user logs on to the host 4 via the TCP / IP connection 4a, the entry goes into the log file 15 generated on the user's host 4 while appearing in the log window in the GUI2. This is a great advantage as it allows the user to examine the log file 15 in the context of the O / S Validator 1 and allows the user to immediately monitor the pass and more importantly the fault. However, in certain circumstances, due to the size of the test 11 to be executed, there may be an excessive amount of log files 15 and therefore the log window LW
By closing it without further opening it, the memory of the host 4 is maintained and an empty view area is provided for the O / S Validator 1. It is also advantageous to close all log files 15 without obstruction. The fault indicates to the product design staff that the target device 9 requires further development. If the user wants to keep all log windows F related to the fault open, the F window can be kept open by clicking the logging window option.

When the present invention is operating, as shown in FIG. 5 a, a window called Available Targets indicates active devices on the network that are broadcasting information to GUI 2. Active devices send large amounts of information, some of which are displayed in the Available Targets window. Users can view / view
Information can be viewed by selecting the ilable Targets menu. Another window must be accessed to get the complete set of broadcasted information. This broadcast information is valuable because it is used to initialize the connection from the engine 3 to the specific CEHarness 8 of the test target device 9.

With reference to FIG. 11, the stress test setup SS will now be further described. The stress suit 11 appears in various forms. However, their basic purpose is to
Stressing the target device 9 by performing very long tests, performing multiple iterations of short tests, or selecting a wide range of parameters in one test. These own tests 11
These residuals, limited to regions, for example Database Stress Suites, stress only the functionality of O / S databases such as Windows CE (R). Stress Test Options
Can be distinguished from the stress suit 11. Stress Test Options are different because Stress Test Options are targeted at functionality and provide a broader band of stress scenarios that correspond to real-world handling models. These scenarios run with any set of test suit files 5 selected by the user. Stress Test Options can and should be performed cooperatively and independently, and during such execution a significant increase in the scope of any test plan results. The first two Stress Test Options relate to memory on the target device 9. The first Stress Test Option provides a Low Virtual Meme that significantly reduces the amount of virtual memory on the target device before running the selected test 11.
mory Option (low virtual memory option). This simulates a realistic harsh environment that can occur when a user opens 15 applications and cause malfunctions. The second Stress Test Option is Low Storage Me
mory Option (low storage memory option). If selected, this second S
The tress test option fills the storage memory of the target device 9 up to its maximum capacity, and allows the target device 9 to experience the solution of a small storage memory. In some cases, this second Stress Test Option is advantageous for a test application program contained in the target device 9 because the test application program can rely on non-existent storage memory. Next 3
Two Stress Test Options are execution options. The first viable stress test option is an infinite loop suitable for long test cycles. A common problem in many device drivers is that they malfunction under long, strong stressful situations. This infinite loop stress test provides a test to determine possible failures. This infinite loop test runs the selected test suit 11 until the user manually hits the Stop button.
The next stress execution option is called Data.txt, and the O / S Validator＼Tests
@Configurable CPU cycle strip test available as a text file identified as TestInputFiles. Two examples are given to files that a user can copy, reuse, or modify. The text file, Data.txt, controls the number of threads and their attributes that users include in their test runs. In other words, the user can run his own tests while other processes are consuming CPU time and removing many problems, including timing. Last Stress Test
Option is Random Execution (random execution). If the user selects this option, the GUI 2 reorders the list of testsuits 11 at runtime so that the testsuits 11 are executed in a different order. This option is ideal because it facilitates the diagnosis of interaction problems with various components.

The remaining Test Run Options are general activities that can be controlled by the user. The first option, "Use Selected Target Excl
"usively (exclusively using the selected target)" is important because if the target device 9 is connected via Ethernet, other users of the subnet will be able to use the O / S Validator 1 This is because the target device 9 can be accessed through a simple target device window. This helps to create stress on the target device 9. Should the user want to isolate the problem, no additional stress is applied. In that situation, the user will have exclusive access to the target device 9. Last Test R
un Option is Set Target Time, which prompts the engine 3 to send time from the host computer 4 to the target device 9, thereby setting the system time of the target device 9 to the time of the host computer 4. Synchronize. Synchronization is convenient because the log file returns with a date and time stamp associated with the date and time of the target device 9. To maintain their accuracy, the user must set the date and time on the target device 9. Before running the test, the last tab contains the Environment Sequencing, which contains valuable information about various selectable environment variables.
It is a ttings (environment setting) tab. These preferences are designed to extend and summarize the testsuit file 5 by including environment variables in the testsuit file instead of hard carded information. For example, Seri
al Tests (serial test) is a common port (com-po
rt). If the common port is not entered as an environment variable, the test will fail. This is because the common port cannot be opened. All environmental variables used for the testsuit are provided. However, any additional environment variables may be added to the user's input suit. After running the test, the Test Status is available to get status information about the current test run. The information is being updated dynamically.

The Suites Run (test suit execution) section window displays a list of selected test suits that have been started. The user can open an arbitrary log file from this window by selecting a desired test icon. Other icons in this control provide fault information. For example, a fault icon is shown with a red “X” in the form of a stylized beaker. Test Run Summary Inf
ormation (test execution summary information) is the number of test suit files to be executed,
Keep a record of the selected test suit, the test suit with disability, and the percentage of test suits with disability. At the completion of the test run, the user
e Select the Failed Suites tab to prompt for a new configuration window, select all failed test suits in the current test run, and return to the test. To facilitate.

The remaining two sections are called Test Details. One of these Test Details sections monitors passing and failing individual test cases, and that section is useful for assessing the value of a test run. The remaining Test Details section is the Failed Suites section, in which all selected test suites that failed are listed by suit name, and pass and fail test cases Are shown. All this information gives the user a very good knowledge of the limits of his target device 9 during the test run (ie passing and more importantly failing).

A main object of the present invention is to properly test the port of the O / S 1001a such as Windows CE®. To accomplish this task, hundreds of test suits 11 are required and provided by O / S Validator 1. For example, approximately 150, as grouped by the O / S subsystem to be verified.
Zero test suits 11 are provided. As shown in FIG. 10, O / S Validato
r1 contains both source code and executable code for all tests 11,
It deals with the main O / S 1001a subsystem and common conformance drivers, with particular emphasis on the subsystems that have historically presented the most problems. O / S Valida
OS subsystem components tested by tor1 include Ethernet / NDIS, serial port driver, display driver, touch panel driver, mouse driver, keyboard driver, OEM adaptation layer, and PC card adapter driver. FIGS. 15A, 15B and 15C show a list of test details for these system components.

As described above and shown in FIG. 3, the engine 3 is called HarnessLink.dll 7 and
It is linked to GUI2 via a component that is iveX control. Above all,
HarnessLink.dll 7 calls G to call (ie, affect) Engine 3.
Provide UI2 with a set of functions. Most of HarnessLink.dll 7 makes function calls and sets up some parameters on the engine 3 command line. All of the initial information is passed to the engine 3 via this command line, ensuring readiness of the engine 3 for a given execution. Multiple GUI-related functions provide information on the command line. Information on the command line corresponds to information on GUI2. Another key function is Harnes
What sLink.dll 7 does is to communicate activity during the operation of engine 3 by opening the named pipe. If an error or need arises to convey some memory information, the engine 3 communicates via a named pipe. The named pipes ensure that the communication between the engine 3 and a particular harness link is direct and accurate, even if multiple engines 3 are running, without any duplication problems. When HarnessLink.dll 7 receives the message from the pipe, it signals the appropriate VB event, then causes GUI 2 to get the information and process it accordingly. As described in connection with FIG. 5, the engine 3 communicates with one HarnessLink.dll 7 and then communicates with one CEHarness 8. Execution of Engine 3 is simple, ie, the command line is received and processed,
Establish an execution socket connection to the target device, open a pipe to communicate with the GUI 2, read the test suit file 5, and consequently execute the test in three stages: PreExecution, Execution, and PostExecution. PreExecu
The Action phase establishes an error socket connection between the target device 9 and the host 4. Relevant data such as logging paths and styles, various test run information, and stress scenarios are sent during the PreExecution phase. The Execution phase includes a response to each successive suit command. The suit command is generally sent by the host 4 and processed by the CEHarness 8, which then responds with a socket message when the execution of the command is completed. The PostExecution phase mainly involves reducing log information and generating summary logs. When these summary logs are completed, the engine 3 ends. The CEHarness 8 of the test target device 9 is a significantly more complex component than the engine 3.
This complexity is because each device 9 has one instance of CEHarness 8 at any given time. However, the device can handle multiple simultaneous connections and the O / S
This is a very important feature for the test method provided by Validator1. When the user activates CEHarness 8, it creates two threads that persist throughout the execution time, a broadcast thread and an execution thread. This broadcast thread updates information such as device IP, connection type, and available common port every 10 seconds and sends broadcast messages to the network at the same rate. If device 9 is connected to host computer 4 via the Windows CE® service (ie, on the NT side) and is connected to either remnet or repllog (ie, on the device side) , Connection type is PPP Changed to PEER.
When this occurs, the broadcast message is sent only to the host 4 to which the target device 9 is directly connected. If the user changes the connection at some point during the run, the message is updated. On the other hand, the execution thread waits for a connection attempt from engine 3. When an execution thread receives a connection, it creates another thread, the main execution thread, that performs the various functions needed. The main execution thread launches another socket to send error or memory information. Thus, the execution thread is event driven, receives commands,
Respond appropriately. Each connection attempt creates its own thread of execution, and therefore
A single CEHarness 8 has many active connections and extends the functionality of the test by running multiple configurations simultaneously on one target device 9, thereby creating a more realistic stress situation. .

The logging library 12 shown in FIG. 3 uses an O / S Validator in various modes.
1 is a composite tool incorporated in the system. Primarily, the logging library 12 is incorporated into a test source file. The test makes calls to the library's API, so that the library handles all the details about capturing and recording test results. The logging library 12 also supports various communication options. The recommended option is via TCP and the user can see their display as log files move from the TCP connection. Another connection option is to log directly to a file on the device. This can be advantageous, for example, if the user wants to log on a PCMCIA card, no additional information is required to be broadcast over the network. The logging library 12 acts as a device-side component for TCP communication, while the host 4 acts as a GUI-side component for communication. This aspect of the logging library 12 provides a log window to the host 4 and colors the test results. For example, a fault message is displayed as a red line.
The name and location of the source code file and the line number where the message was generated are included in the message of the logging library 12. More importantly, a detailed error message is provided for the current event describing the program. Each log window has a summary tab that facilitates user access to program memory, pass, fail, and timing information. Another important feature of log files is that they capture large amounts of information at the beginning and end of the test. This information provides a snapshot of memory, system, power, and other valuable information.

The present invention has been particularly shown and described with respect to certain preferred embodiments and features thereof. However, it will be readily apparent to one skilled in the art that various changes and modifications may be made in form, material and design details without departing from the spirit and scope of the invention as set forth in the appended claims. It will be obvious.

[Brief description of the drawings]

FIG. 1 is a schematic representation of a current computerized product with an embedded operating system in a control unit.

FIG. 2 is a manufacturing flowchart illustrating a quality assurance test on a computerized product with an embedded operating system according to the present invention.

FIG. 3 is a block diagram illustrating the major components of the operating system verifier of the present invention, including a graphical user interface, an engine, a plurality of test suites, and a logging library.

FIG. 4 is a block diagram illustrating the present invention executing multiple test suite configurations from a host device to test multiple target devices with an embedded operating system in accordance with the present invention.

FIG. 5 is a block diagram illustrating the present invention essentially the same as that shown in FIG. 4, except that the target device shows communication via the O / S Validator 1 and the Ethernet means in the host.

FIG. 5A is a diagram showing a configuration in which communication between a plurality of hosts and a target device can be performed via Ethernet.

FIG. 6 is a diagram showing still another configuration of the test suit execution status.

FIG. 7 is a table listing functional areas of specific APIs tested in automatic and manual test suit execution.

FIG. 8A is a comprehensive listing of functional areas and APIs that can be tested in automatic or manual mode.

FIG. 8B is a comprehensive listing of functional areas and APIs that can be tested in automatic or manual mode.

FIG. 8C is a comprehensive listing of functional areas and APIs that can be tested in automatic or manual mode.

FIG. 9 is a table listing selected APIs for use in a building automation script.

FIG. 10 is a schematic diagram illustrating the concept of the present invention for providing source code for all executable programs.

FIG. 11 is a block diagram of a window showing test suite selection options, as well as other related summary functions.

FIG. 12 is a block diagram of a logging window showing tabs for test results, test failures, and associated test summary options.

FIG. 13 is a diagram illustrating the relationship between the test cycle time and the number of test devices to be tested in parallel in the form of a graph.

FIG. 14 is a block diagram of a configuration window showing tabs for performing related functions of various configurations.

FIG. 15A is a summary listing of testing details for operating system components in accordance with the present invention.

FIG. 15B is a summary listing of testing details for operating system components in accordance with the present invention.

FIG. 15C is a summary listing of testing details for operating system components in accordance with the present invention.

──────────────────────────────────────────────────続 き Continuation of front page (81) Designated country EP (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, IT, LU, MC, NL, PT, SE ), OA (BF, BJ, CF, CG, CI, CM, GA, GN, GW, ML, MR, NE, SN, TD, TG), AP (GH, GM, KE, LS, MW, SD, SL, SZ, TZ, UG, ZW), EA (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), AE, AL, AM, AT, AU, AZ, BA, BB, BG, BR, BY, CA, CH, CN, CR, CU, CZ, DE, DK, DM, EE, ES, FI, GB, GD, GE, GH, GM, HR, HU, ID , IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, MX, NO, (72) Invention NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, TR, TT, TZ, UA, UG, UZ, VN, YU, ZA, ZW Walters, James, Floyd 98101, Washington, USA, Seattle, Seneca Street No. 404.1101 (72) Inventor: Dicara, Janaldana, Lao 9805, Washington, USA 9980, Belleview, Numbersea 208, One-handed Red Thirty Force Avenue South East 1590 (72) Inventor Sample, Ian 98105, Washington, USA, Seattle, Number 101, University I Woo E Lee North East · 5256 F-term (reference) 5B042 GA21 HH11 [continuation of the summary] another diagnosis, put the historically most often operating Retinku have shown the problem Bu Saff Bu emphasis on the system.

Claims (20)

[Claims] 1. A computer-based system for testing and validating an operating system embedded in a target device, comprising: a host computer; a target device comprising a b operating system; and a c operating system. A test and validation software program, wherein the software program is provided to the host computer, the software program communicates with the target device, graphical user interface program means for interfacing with a user, Engine means for responding to commands from a graphical user interface; and testing and validating at least one component of the operating system. An operating system test and validation software program, comprising: a plurality of test suites including at least one test for testing gender; and d generated by the operating system test and validation software program. A computer-based system comprising: a logging library means for processing and storing relevant information of the test as required. 2. The computer-based system of claim 1, wherein the operating system test and validation software program further comprises a set of functions on the graphical user interface to invoke the engine means. 3. The computer-based system of claim 1, wherein said target device further comprises an event-driven application for communicating with said engine means, wherein said engine means sends an event and said event-driven application responds. . 4. The computer-based system of claim 1, wherein said operating system of said target device comprises a Windows CE® operating system. 5. The computer-based system of claim 1, wherein the test suite includes at least one system stress test routine and at least one feature and functionality test. 6. The system stress test routine includes a code base for stress testing the at least one component of the operating system, wherein the at least one component of the operating system is an Ethernet / NDIS From a group of operating system components, including PCMIA, memory, file systems, serial ports, video systems with multiple application program interfaces, infrared systems, OEM adaptation layers, touch panels, mice, keyboards, and audio / waveform systems. Selected, the test has at least three defects: hardware design, hardware programming, and operating system interaction. The computer-based system of claim 5, wherein the system identifies the action and runs in an automatic mode or a manual mode. 7. The target device includes a stand-alone unit provided in the operating system test and validation software program for performing validation and stress tests independently of the host computer. Clause 1. The computer-based system of Clause 1. 8. The computer-based system of claim 1, further comprising an Ethernet connection coupled to said host computer and to said target device for performing testing and validation tasks. 9. The computer-based system of claim 1, wherein said logging library means includes at least one pass test result file using a WRITETESTPASS application program interface. 10. The computer-based system of claim 9, wherein the at least one pass test result file resides on the target device. 11. The logging library means includes at least one fail test result file using a WRITETESTFAIL application program interface.
The computer-based system of claim 1. 12. The computer-based system of claim 11, wherein the at least one fail test result file resides on the target device. 13. A computer-based method for testing and validating an operating system embedded in a target device, the method comprising: (a) providing a host computer; Providing a target device having: (c) providing a software program for testing and validating an operating system, wherein the software program is provided to the host computer, and wherein the software program interfaces with a user. Graphical user interface program means for communicating with the target device, and engine means for responding to commands from the graphical user interface program means; Preparing an operating system test and validation software program, comprising: a plurality of test suites including at least one test for testing and validating at least one component of the operating system; (D) providing logging library means for processing and storing test-related information as generated by the operating system test and validation software program; and (e) on the target device. Executing the operating system test and validation software program and testing and validating the operating system; and (f) generating pass and fail test results. Tsu comprising a flop, computer-based method for testing the operating system embedded in the target device to check the validity. 14. The target device of claim 13, further comprising the step of providing the operating system test and validation software program with a set of functions on the graphical user interface to invoke the engine means. A computer-based method for testing and validating the operating system embedded in a computer. 15. The target of claim 13, further comprising providing the target device with an event-driven application for communicating with the engine means, wherein the engine means sends an event and the event-driven application responds. A computer-based method for testing and validating an operating system embedded in a device. 16. The computer-based method for testing and validating an operating system embedded in a target device of claim 13, further comprising providing the target device with a Windows CE® operating system. Method. 17. Providing a test suite in the form of a system stress test routine including a code base for stress testing said at least one component of said operating system, said at least one component of said operating system. Operating system components including Ethernet / NDIS, PCMIA, memory, file system, serial port, video system with multiple application program interfaces, infrared system, OEM adaptation hierarchy, touch panel, mouse, keyboard, and audio / waveform system Wherein the test has at least three defects: hardware design, hardware programming, and operating 14. A computer-based method for testing and validating an operating system embedded within a target device of claim 13, further comprising the step of identifying interactions of the operating system and executing in an automatic mode or a manual mode. Method. 18. The target device of claim 13, further comprising the step of providing an Ethernet connection coupled to the host device and to the target device for performing test and validation tasks on the target device. A computer-based method for testing and validating embedded operating systems. 19. The method of claim 13, further comprising: providing the logging library means with at least one pass test result file using a WRITETESTPASS application program interface. A computer-based method for checking validity. 20. The method of claim 13, further comprising providing the logging library means with at least one fail test result file using a WRITETESTFAIL application program interface. A computer-based method for checking validity.