KR20010112250A - A system and method for testing and validating devices having an embedded operating system - Google Patents

Abstract

The present invention is a system and method for efficiently improving the product development process of a target device using a commercially available operating system and improving the quality assurance that saves time and money for many people and months. Testing device. The system and method, the O / S validator 1, uses a host graphical user interface harness 12, a host for the target communication 3, at least one test suite 11 and a result capture method 12. , A fully automated design validation package for commercially available operating system 1000a, Windows CE. The O / S validator 1 provides a high speed and more accurate automated test suite technique for testing ports of commercially available operating system 1000a such as Windows CE for target hardware 1000,9. The O / S validator 1 also includes an extensive code-base developed specifically to intentionally stress operating system O / S, device drivers, OEM adaptation layers (OALs), and hardware interactions. The test suite 11 is focused on identifying three major defects, including hardware design, hardware programming (driver / OAL), and operating system interaction. Certain diagnostic emulations exist in operating systems that historically represent most problems.

Description

A SYSTEM AND METHOD FOR TESTING AND VALIDATING DEVICES HAVING AN EMBEDDED OPERATING SYSTEM}

More and more developers are embedding operating systems such as Windows CE in many different types of computerized products, including set-top boxes, gaming systems, bar code scanners and factory automation systems. As Windows CE grows, the desire for "off-the-shelf" software development tools increases. While many tools and "off-the-shelf" software kits have been introduced to the market to save design time and drive high-speed device development, any high-speed testing system or method exists that specifically checks for new product compatibility at the final stage of device development. Did not do it.

Traditionally, only two operating system device testing options have been used: (1) in-house writing of test code or outsourcing customer code development to another company. To complete the testing project in-house, OEMs spend several months training their staff, and spend more months developing test code. There are many more months of preparation before the product is tested with the test code. Similarly, outsourcing a customer code development house can take months to write code. Therefore, both of these options are time consuming and expensive.

The testing options of the previous time-consuming operating system devices have long been associated with integrated testing systems and methods for improving product quality, saving time and money for multiple people and time periods, and simplifying product development procedures. I felt the need. In particular, it provides a fully automated design check package for designers of devices by overcoming previous challenges, providing systems and methods to improve product quality, save time and money for multiple people and time periods, and streamline product development processes. There is a need for a system and method for testing and validation apparatus that has an embedded Windows CE operating system that is enabled.

FIELD OF THE INVENTION The present invention relates to test systems and methods and product quality assurance for identifying operating systems provided in computerized products. In particular, the present invention relates to test products and methods and product quality assurance for identifying operating systems in the development of computerized products. The invention also relates to test systems and methods and product quality assurance for identifying operating systems, such as Windows CE, typically provided in computerized products and manufactured and marketed by Microsoft Corporation of Redmond, WA.

1.0 is a schematic diagram of a computerized product provided with a built-in operating system in a control unit.

2.0 is a flow diagram of the present invention illustrating quality assurance testing in a computerized product provided with a built-in offrating system.

3.0 is a block diagram illustrating the major parts of an operating system validator of the present invention including a graphical user interface, an engine, multiple test suites, and a logging library.

FIG. 4.0 is a block diagram of the present invention for executing multiple test suite configurations from a host device to test multiple target devices provided with an embedded operating system.

FIG. 5.0 is a block diagram of the invention as shown in FIG. 4 except that the O / S validator 1 communicates with the target device at the host via Ethernet means.

5A illustrates a device in which communication between multiple host and target devices may occur over Ethernet.

6 shows another device for a test suite execution state.

7.0 is a table listing of functional areas of specific APIs tested in automatic and manual test suite execution.

8A, 8B, 8C are tables listing a comprehensive list of APIs and functional areas that can be tested in automatic or manual mode.

9.0 is a table listing of selected APIs for use in creating automated scripts.

10.0 is a schematic diagram of the inventive concept of providing source code for all executable programs.

11.0 is a schematic diagram of a window showing test suite selection options in addition to other related summary functions.

12.0 is a block diagram of a logging window showing tabs for test results, test failures, and associated test summary options.

13 is a graph of the relationship between the number of test devices currently tested versus test cycle time.

14.0 is a block diagram of a configuration window showing tabs for performing various configuration related functions.

15A, 15B, and 15C show table listings testing the details of operating system components in accordance with the present invention.

Commercially available to the applicant's assignee under the trademark CEValidator ^™ , the present invention is an operating system validator (hereinafter referred to as an O / S validator and designated by the number 1 in the drawing), wherein the hardware and / or software of the target device The new problem is solved by providing a test system that includes an automated test suite method that, when newly developed, tests a part of an operating system such as Windows CE. O / S validators include a wide range of code bases, OEM adaptation layers (OALs), and hardware interactions specifically developed to intentionally stress O / S. The test suite provided primarily identifies three major flaws: hardware design, hardware programming (driver / OAL), and operating system interaction. Special emphasis is given to the Windows CE subsystem, which historically shows most problems at the time of diagnosis. The test suite includes nearly 1500 tests, including system stress-testing routines, in addition to characteristic and functional tests that provide a complete analysis of the Windows CE port. These tests are grouped by O / S validators. O / S validators include both test source code and executable programs for all tests.

In order to facilitate the execution of the test suite and the collection of logging results, an intuitive user interface for O / S validator host elements, such as a standard Windows application using the Microsoft Windows user interface, is used. The O / S validator splits the test suite as a client / server application. The graphical user interface (GUI) interacts with a small application running on the target device, CEHarness.exe. Since this communication can occur over Ethernet, at least one host can run the suit for at least one target device.

The O / S validator generates error information that is useful when the target device fails the test. While the suit is running, the results are displayed in a number of dynamically generated log windows in addition to the summary tab of the configuration. The logging window contains the complete text of the given test result. Failures are color-coded red for easy identification. The navigation buttons in the logging window allow you to move quickly from one failure to another. During testing, the logging API also allows prolog and epilog to be generated in each result file. Information such as the currently running process, battery power level and run date and time are automatically recorded in the resulting file and displayed in the log window. Useful summary information such as program memory loss, storage memory loss, or total test execution time is provided in the Log Window tab. Summary information for a given test result is collected and displayed on the Summary tab of the configuration window. The Summary tab reports the number of PASS and FAIL test cases in real time. The breakout PASS and FAIL numbers for the individual suits are also displayed. The Summary tab of the configuration window facilitates quick navigation for each failure among thousands of test results. The exact source file and number of lines corresponding to the logged failure are automatically reported by the logging API of the O / S validator. Because the O / S validator provides the source code for all executables, you have direct access to the source code that reports the error, which is a sure addition to the textual description of the failure.

Other features of the invention are described in detail in the embodiments of the invention.

The invention is described in detail with reference to the drawings.

1.0 shows a typical computerized product 1000, 9 such as a factory automation system, a computer workstation, a set top box, a gaming system and a bar-code scanner provided with a built-in operating system. As shown, product 1000 may include a typical target device 9 provided with an operating system 1001a, such as an embedded Windows CE operating system (O / S). Computerized product 1000 may operate as a standard device equipped with the O / S validator of the present invention for testing and verifying its operating system 1001a. Stand-alone testing facilitates the debugging of reported defects and the development of new tests, as no in-depth knowledge of the O / S validator infrastructure is required. However, as shown in FIG. 2.0, the product 1000 is manufactured as a host computer 4 equipped with the O / S validator 1 of the present invention for testing the target device 9 provided with the operating system 1001a. It will work in quality assurance testing environment M. Referring to FIG. 1.0, the computerized product 1000 includes a control unit 1020, various input / output ports 1021, a keyboard 1009, a printer 1010, a mouse 1011, and a monitor 1012. Will contain multiple sub-parts. The sub-parts 1009, 1010, 1011, 1012 may be testable target devices. A typical control unit 1020 includes a central processing unit 1001, a storage device such as a hard disk drive 1004, a RAM 1001, another memory portion including a ROM 1003, a compact disc 1005, an audio portion ( 1006), network / server card 1007 and modem 100, including various parts. In order to make the product 100 operate as a useful device, an operating system 1001a is included in the control unit.

FIG. 3.0 shows the main part of the O / S validator 1 which includes a user interface (GUI) 2, an engine 3, a plurality of test suites 11 and a logging library 12. The GUI 2 and the engine 3 communicate internally bidirectionally with each other via a portion named HarnessLink.dll designated by the number 7 in the O / S validator 1. 4.0 shows a host computer 4 provided with an O / S validator 1. As shown, multiple target devices 9 are provided with O / S 1001a for testing in accordance with the present invention. The O / S validator 1 has the ability to create a testing configuration, such as multiple test configurations 21a, 21b, 21c, for testing a particular function under the control of the OS 1001a in the target device 9. The side configuration of the device named CEHarness 8 communicates with the engine 3 of the O / S validator 1. As shown in FIGS. 5 and 5A, the CEHarness 8 also communicates with the engine 3 of the O / S validator 1 via the Ethernet means 4a. 5A shows that a number of hosts 4 run the suit against a number of target devices 9 as the communication takes place over Ethernet. In another alternative embodiment, as shown in FIG. 6 for a test suite execution situation, the CEHarness 8 is also in communication with the engine 3 of the O / S validator 1 by the suit execution connection 1021a. The host computer 4 may include an NT host computer, the logging library 12 is provided to the target device 9, and the test results are sent to the host computer 4 by the socket connection 1021b. Is provided.

In operation, the O / S validator 1 tests and verifies the target device 9 provided with a built-in operating system such as a Windows CE operating system. O / S validator (1) identifies Windows CE ports to target devices, (2) provides stress and performance testing, (3) logs and analyzes results, and (4) multiple application programs Testing multiple functional areas including interfaces (APIs), see Tables 1.0 and 2.0 in FIGS. 7,8A, 8B, 8C, (5) performing multiple pass / fail tests in multiple test suites, (6 ) Facilitates test customization in automated suites, (7) provides host-side graphical test harnesses, (8) stresses and evaluates memory performance, and (9) test automation includes multiple APIs. It provides a means to build the system, see Table 3.0 of Figure 9.0, and (10) works by providing a result analysis tool named CEAnalyzer. As described above and as shown in FIG. 10, the O / S validator 1 includes both test source code SC and executable program EP (or referred to as executable test) for all tests. The only execution requirement for the test executable EP is that test cases "pass" and "failure" are reported using two specific APIs, WRITETESTPASS () and WRITETESTFAIL (). These macros have a signature similar to the known printf () function, but their use produces a standard test case result format in a "result" file that follows automated summarization, and integrates the host user interface with its reporting. The O / S validator 1 method also includes mediation between the test cases encapsulated in the test executable EP and the GUI 2, providing a means by which the GUI 2 distributes and integrates test executions. The test suite 11 includes a text file 5 composed of suit language commands (e.g., PUT, RUN, WAIT and DELETE) which are direct representations of the work required to distribute and execute the test. Other supported suit commands are GET to retrieve files from the target device 9, RUNHOST to run an executable program EP on a host computer 4 useful for client / server style testing, and a host computer also useful for client / server style testing. WAITHOST waiting for processing to end on (4), PUTSYSTEM for inputting files in the system directory (/ Windows) of the device, SLEEP for basic timing when all others fail, MSGBOX for displaying a message box on the host device, SETREG to add or change registry settings on the device and DELREG to remove registry settings from the device. In addition to providing an internal suit document, the first suit file command is provided as a suit description of the GUI 2. The O / S validator 1 method involves organizing the test suite file 5 by hierarchical arrangement in a directory structure on a host computer. As shown in FIG. 11, the test suit 11 is divided into an automatic AU test suit, a manual Ma test suit or a stress SS test suit at the top level, respectively. Automatic suit Au is a test that does not require any user intervention during execution. In contrast, the manual suit Ma requires user intervention (eg, keyboard and touch panel suit). The O / S validator (1) method involves stressing the system by the stress suit SS by pressing input / output (I / O) throughput up to the limit line, and operating a program or object storage memory that is rarely used. Bring down the order file system with multi-threaded simultaneous stressing. Under this top level hierarchy, the O / S validator 1 method involves arranging the test suite 11 by functional area.

As noted above, FIG. 3.0 shows a major portion of the O / S validator 1 that includes a graphical user interface (GUI) 2, an engine 3, a plurality of test suites 11, and a logging library 12. Illustrated. The O / S validator 1 uses the GUI 2 as the visual basic part because it works well for any level of user. The GUI 2 design is based on modifications that interface the basic parts with user input. The engine 3 maintains the core of the function on the host 4. The engine 3 reads a number of suit files 5, interprets them and executes the instructions. The engine 3 obtains information from the GUI 2 and uses the information to set up a number of execution options. The engine 3 is written in the C / C ++ language. The engine 3 is linked to the GUI 2 through a part named HarnessLink.dll 7, which is an ActiveX control. HarnessLink.dll 7 is instantiated and called from the GUI 2 by a number of information, which is passed to the engine 3 before execution begins. The Dll link 7 also operates to communicate between the engine 3 and the GUI 2 during execution to relay information, relay error messages, and relay dynamic run-time instructions. The target device 9 comprises a device-side (as opposed to a host-side) named CEHarness 8. CEHarness is a C / C ++ program on the target device 9, which is shown in FIG. 4, and will communicate almost exclusively with the engine 3 if it does not broadcast the target information on the network, where the GUI 2 will The information will be received and passed to the engine 3, which is shown in FIGS. 5 and 5a. CEHarness 8 is an event-driven application in which engine 3 sends an event and a CEHarness 8 response. The two remaining parts, the test suite 11 and the logging library 12, are intertwined because the test suite 11 is recorded using a plurality of application program interfaces (APIs) 13 that are part of the logging library 12, This is illustrated in Figures 8A, 8B and 8C. This API 13 has a substantial function and depends somewhat on the information passed through the component chain. The logging library 12 has a simple concept of functionality, either logging the TCP / IP information 16 back to the GUI 2 (see FIG. 5.0) or outputting the results 14 and log files 15 to the device 9. By writing directly, a log file 15 is created to communicate with the test results 14. As shown in FIG. 12, the logging window LW shows the test results 14 in the test file 15, which results in failure F to facilitate user access to program memory, pass, failure and timing information. ) And the Summary tab (SumT). The plurality of test suite tests 11 comprise an integral part of the O / S validator 1. FIG. 13 shows a graph form in which the test cycle time CT decreases as the test apparatus is tested.

More specifically, the GUI 2 is complex code that follows the degree of functionality needed to handle layers of components. The GUI 2 provides a "wizard" whose primary function is to guide new users by selectively setting and listing default settings. As shown in FIG. 14, the GUI 2 also provides a configuration window CW as a means of executing a test run including a single pass through a series of selected suits 11 on the target device 9. . As shown in FIG. 4, the multiple configurations 21a, 21b, 21c can be operated to simulate multiple scenarios. The content of the configuration window CW includes a number of tabs for user control. As an example, the suit tab S provides a tree structure of the suit file directory under the O / S validator directory. This tree structure is organized to provide meaningful distinction between the types of tests 11 that the user selects. In addition, this tree structure is created when the user opens the configuration window (CW), and the user adds a new user input suite to the suit file directory created by the O / S validator (1) installer. The validator 1 can be extended. The test suites 11 are command scripts that perform an action corresponding to the command of the script after the engine 3 reads it. The suit file 5 generally begins with a series of instructions. The above commands appear in the suit file information section of the file. If the word "Manual" appears at the top of the soot file 5, the file is considered to need manual execution and is thus assigned to different icons. In the test suite section, as shown in FIG. 11, the user can request the test suite file 5 again in any way. Referring to Figure 14, the logging tabs contain valuable information. The user can select three logging methods: LH for logging to host 4, LTD for logging to target device 9 or LHD for logging to both host 4 and target device 9. . The log information is then stored in the configurable directory listed in the edit box. All this information is transmitted to the engine 3 via the DLL 7 and then to the CEHarness 8 of the target device 9. Next, the information is obtained by the logging library 12 when the test 11 is running. The other tabs in the configuration window (CW) select a high priority thread selection during the suite execution by selecting a set of stress state tabs (SC), thread tabs (T), reducing program and storage memory by selecting taps PM and SM, and tap SRT. Runtime selection by selecting, and stopping by selecting tab STOP. You can use an infinite loop tap to find the system's memory leak. Useful summary information such as program memory loss, storage memory loss, or total test execution time is provided in the Summary Tab (SumT). Summary information for a given test result is collected and displayed in the Summary Tab (SumT). The Summary tab reports the number of PASS and FAIL test cases in real time. The breakout PASS and FAIL numbers for the individual suits are also displayed. The Summary tab of the configuration window facilitates quick navigation to individual failures, perhaps among thousands of test results. The exact source file and number of lines corresponding to the logged failure are automatically reported by the logging API of the O / S validator. Since the O / S validator provides source code for all executables, going directly to the source code that reports the error is a powerful addition to the textual description of the failure. Logging options change their execution and effects dramatically. The first option is to assume that whenever a user runs a test suite 11 indicating "pass", a series of log files 15 summarizing these test results 14 are automatically generated. Depending on the logging method selected, the summary file 15 is generated by the CEHarness 8 or the engine 3. By default, the engine 3 traverses all log files 16 in the logging directory, so the user can accept a list of log files that do not correspond to the test 11 being run. In order for the summary file 15 to indicate more of the startup test 11, the user can delete the log directory before startup, and the user can see that the engine 3 has only one log file 15 for each test 11. You can choose the logging option to choose to return to the most recent summary only. This means that if a user runs a file system test 30 times on a given day, there will only be an entry in the summary log for the test corresponding to the most recent run. The summary log will have only one entry for each test 11 in which the log file 15 is still logged. The other two options are handled in the GUI 2. If the user is logged into the host 4 via the TCP / IP connection 4a, the entry will enter the log file 15 created on the user host 4 while appearing in the log window in the GUI 2. This enables the user to examine the log file 15 in the environment of the O / S validator 1 and is more beneficial because the user can immediately monitor the passage and more important failures. However, under certain circumstances there will be an excess of log files 15 that follow the size of the test 11; Therefore, closing the log window LW without opening any more will maintain the memory of the host 4 and will also provide a clear viewing area for the O / S validator 1. Another advantage is to close all log files 15 without fail. The failure indicates to the product design personnel that the target device 9 will require other improvements. The user will want to keep opening all failed log windows F by clicking an option in the logging window to keep the F window open.

In operating the present invention as shown in FIG. 5A, a window named as an available target shows an active device on the network that broadcasts information to the GUI 2. The active device sends large information, some of which are displayed in the available target window. The user can view the information by selecting the View / Enable Target menu. The other window must be accessed to obtain a complete set of broadcasted information. This broadcast information is valuable because it is used to start a connection from the engine 3 to a particular CEHarness 8 in the test target device 9.

The stress test setting SS will be described in detail with reference to FIG. 11. Stress suits 11 come in many forms; However, their main purpose is to stretch the target device 9 by running a very long test, to repeat a short test several times or to select a wide range of parameters in one test. For example, the rest of the specs for your test 11 area, such as the database stress suite, only stress the database capabilities of the O / S, such as Windows CE. The stress test options are distinguished from the stress suit 11. The stress test options are different because they target features to provide more broadband stress scenarios that are identical to real world usage models. This scenario runs in conjunction with any user selected set of test suite files 5. The stress test options can also be run jointly and separately, thereby providing a significant boost over the scope of any test plan. The first two stress test options relate to the memory of the target device 9. The first stress test option is a low virtual memory option that significantly reduces the amount of virtual memory of the target device before the selected tests 11 are run. This simulates a realistic harsh environment that can occur when a user has 15 open applications, and affects malfunctions. The second stress test option fills the storage device of the target device 9 with maximum capacity to experience the target device 9 response to the low storage memory solution. In some cases, this second stress test option is efficient for testing an application program contained in the target device 9 that may follow a non-existent storage memory. The third stress test options that follow are run options. The first viable stress option is an infinite loop, which is the same for long test cycles. A common problem for many device drivers is malfunctioning under long, strong and stressful situations. This infinite loop stress test provides a test to determine possible breakdowns. The infinite loop test above runs the selected suit 11 until the user manually presses the stop button. The next stress run option is a configurable CPU cycle loss test that is available as a text file named Data.txt and separated by the O / S validator Tests TestInputFiles. Two examples are provided for files that users can copy, reuse, or modify. The text file, Data.txt, controls the number of threads and their characteristics that a user can include to run his test. That is, the user can run his or her test while other processing consumes CPU time and eliminates various problems, including timing. The final stress test option is random execution. When the user selects this option, the GUI 2 will re-request the list of test suites 11 at run time to run in different orders. This option is ideal because it facilitates the diagnosis of interaction problems in several parts.

The remaining test run options are common activities that can be controlled by the user. The first option, "Use the selected target exclusively" is important, which means that when the target device 9 is connected over Ethernet, the O / S validator 1 can be used by other users in the subnet to use the target device window. This is because the target device 9 can be accessed through the target device 9. This helps to create stress in the target device 9. In events where a user wants to isolate a problem, extra stress should not be applied. In such a situation, the user must have exclusive access to the target device 9. The last test run option is to set a target time that prompts the engine 3 to transfer time from the host computer 4 to the target device 9, and thus the target device 9 system at the host computer 4 time. You can synchronize the time. Synchronization is advantageous because the log file returns to the date and time stamp associated with the date and time at the target device 9. In order to maintain this accuracy, the user must set the target device 9 date and time. The last tab before running the test is the Preferences tab, which contains valuable information about the various selectable interview variables. For example, a series of tests takes available comfort as a parameter. If the comfort is not entered as a variable environment, the test will fail, because the comfort cannot be opened. All environment variables used in the suite are provided; However, any additional environment variables will be users added to the user-input suite. After running the test, the test state is variable to obtain state information for the current test run. This information is updated dynamically.

The suit run section window lists the selection suits that have started. The user can open any log file from this window by selecting the desired test icon. Other icons under this control provide failure information. The failure icon is shown as a stylized beaker that is crossed with a red "X". The test run summary information maintains the possible number of tracks, selected suits, failed suits, and percentage of failed suits for the test suit file. When the test run is complete, the user can select the Configure Failed Suits tab, which selects all failed suits in the current test run and prompts for the presence of a new configuration window to facilitate return testing.

The other two sections are named Test Details. One of the test details sections monitors individual test cases as to which sections pass or fail critical for measuring the value of the test run. The remaining test details section is the failed suite section where all selected failed suites are listed by suit name, indicating the number of pass and failed test cases. All of the above information provides the user with a very efficient idea of the limitations of his target device 9 during a test run (ie what passed and more importantly what failed).

The main object of the present invention is to properly test the ports of the O / S 1001a, such as Windows CE. To accomplish this task, hundreds of soot tests 11 are needed and provided by the O / S validator 1. Nearly 1500 test suites 11 are provided, grouped by an identified O / S subsystem as an example. As shown in FIG. 10.0, the O / S validator 1 is a source code that covers the main O / S 1001a subsystem and common adaptive drives with specific emulations for the subsystem that historically shows most problems. Contains executable code for all tests 11. O / S subsystems tested by O / S validators include Ethernet / NDIS, serial port drivers, display drivers, touch panel drivers, mouse drivers, keyboard drivers, OEM adaptation layers, and PC card adaptation drivers. 15A, 15B, and 15C are tables listing testing details of these system components.

As described above, in Fig. 3, the engine 3 is linked to the GUI 2 through a portion named HarnessLink.dll 7 which is an ActiveX control. In particular, HarnessLink.dll 7 provides a function for a set of GUIs 2 to invoke (ie, execute) the engine 3. Most HarnessLink.dll (7) function-calls set up parameters for the command line of the engine 3. All initial information is passed to the engine 3 via this command line, ensuring the preparation of the engine 3 for a particular execution. Many GUI-related functions provide information on the command line. The information on the command line corresponds to the GUI 2 information. Another major function assisted by HarnessLink.dll 7 transfers the activity at startup of engine 3 by opening a named pipe. If an error or necessity arises to transmit the memory information, the engine 3 passes through the named pipe. The named pipes ensure that the communication between the engine 3 and the particular harness link is direct and accurate, and there are no radiation problems if multiple engines 3 are running. When HarnessLink.dll 7 receives a message from a pipe, the GUI 2 signals the appropriate VB event to take the information and process it. In FIG. 5.0, the engine 3 then communicates with one HarnessLink.dll 7 in communication with the CEHarness 8. The engine 3 execution is simple: the command line is received and processed, establishes an execution socket connection to the target device, opens a pipe for communicating with the GUI 2, reads the test suite file 5, The test is run in three phases: before, after, and after. The pre-execution step establishes an error socket connection between the target device 9 and the host 4. Relevant data, such as logging paths and styles, various test run information, and stress scenarios, are transmitted during all phases of execution. The execution phase includes a response to each successive suit command. The suit command is generally sent by the host 4 and is handled by the CEHarness 8 which responds with a socket message when the command execution is complete. Post-execution steps mainly include reducing log information and generating a summary log. When this summary log is complete, the engine 3 is present. The CEHarness 8 of the test target device 9 is considerably more complicated than the engine 3. This complexity is because each device 9 has an instance of CEHarness 8 at any time; However, the apparatus can handle a large number of simultaneous connections, and very important characteristics for the testing method are provided by the O / S validator 1. When the user starts CEHarness (8), it creates two threads that endure through the entire execution time, the broadcast thread and the execution thread. It updates information such as device IP, connection type and available comfort every 10 seconds, sending broadcast messages to the network at the same rate. If device 9 is connected to the host computer 4 via a Windows CE service (i.e. on the NT side) and connected to either a ramnet or a repllog (i.e. on the side of the device) , The connection type will be changed to PPP_PEER. If so, the broadcast message is sent only to the host 4 to which the target device 9 is directly connected. If the user changes the connection while running, the message is updated. The execution thread, on the other hand, waits for a connection attempt from the engine. When an execution thread receives a stack, it spawns another thread, the main execution thread, which executes several necessary functions. The main thread of execution starts another socket for sending any error or memory information. The thread of execution is therefore event-driven and receives commands and responses as appropriate. Each connection attempt spawns its own thread of execution; Therefore, one CEHarness 8 can have multiple connections, extending test functionality by running multiple configurations simultaneously on the target device 9, thus creating a realistic realistic stress situation.

The logging library 12 shown in FIG. 3.0 is a complex tool integrated into the O / S validator 1 in various ways. Primarily, the logging library 12 is integrated into a source file for testing. The test calls the library API when the library handles all the details related to capturing and recording the test results. The logging library 12 also supports several connection options. The recommended option is TCP, which allows the user to read the log file when moving from a TCP connection. Another communication option is direct logging of files on the device. This may be advantageous, for example, if the user wants to log onto a PCMCIA card but does not want special information broadcasted on the network. Although the logging library 12 acts as the device side part for TCP communication, the host 4 part acts as the GUI 2 side part for communication. This aspect of the logging library 12 provides color coding for the log window and test results on the host 4. For example, the failure message is displayed as a red line. In addition to the name and location of the sort code file, the number of lines for which the message is generated is included in the logging library 12 message. Also very importantly, detailed error messages are provided to describe the current event in the program. Each log window has a summary tab that facilitates user access to program memory, pass, fail, and timing information. Another important characteristic of log files is the capture of large pieces of information at the beginning and end of the test. This information provides a snapshot of memory, system, power, and other critical information.

The present invention has been described based on the preferred embodiments. However, those skilled in the art can implement the invention in various aspects without departing from the scope of the invention.

Claims (20)

A computer-based system for testing and verifying the embedded operating system in a target device: a. A host computer; b. A target device provided with an operating system; And c. An operating system for testing and verifying a software program provided to said host computer, said program comprising a graphical user interface program for interfacing with a user, engine means for communicating with said target device and responding to commands of said graphical user interface; A plurality of test suites comprising at least one test for testing and verifying at least a portion of the system, d. And logging library means for adjusting and storing test-related information generated by the operating system for testing and verifying software programs. 2. The computer-based system of claim 1 wherein the operating system for testing and verifying a software program further comprises a set of functions on a graphical user interface for invoking engine means. 2. The apparatus of claim 1, wherein the target device comprises an event-driven application for communicating with the engine means, wherein the engine means sends an event and the event driven application response. Computer-based system. The computer-based system of claim 1 wherein the operating system of the target device comprises a Windows CE operating system. The computer-based system of claim 1 wherein the test suites include at least one system stress-testing routine and at least one characteristic and functional test. 6. The system of claim 5, wherein the system stress-testing routine comprises a code base for stress testing at least one portion of the operating system, wherein the at least one portion of the operating system includes Ethernet / NDIS, PCMIA, memory, Selected from the group of operating system portions including file systems, serial ports of video systems with multiple application program interfaces, infrared systems, raw equipment manufacturer adaptation layers, touch panels, mice, keyboards, and audio / wave systems; Is at least three defects, i.e., hardware design, hardware programming and operating system interaction, wherein the computer-based system runs in an automatic or manual mode. 2. The computer-based system of claim 1 wherein the target device comprises a standalone unit provided with the operating system for testing and verifying a software program to perform verification and stress testing independently of the host computer. 2. The computer-based system of claim 1, further comprising an Ethernet connection coupled to the host and the target device to perform testing and verification tasks. 2. The computer based system of claim 1, wherein the logging library means comprises at least one pass test result file using a WRITETESTPASS application program interface. 10. The computer-based system of claim 9 wherein the at least one pass test file is present on the target device. 2. The computer based system of claim 1, wherein the logging library means comprises at least one failure test result using a WRITETESTFAIL application program interface. 12. The computer-based system of claim 11 wherein the at least one failed test file is present on the target device. As a computer-based method for testing and verifying an embedded operating system in a target device: a. Providing a host computer; b. Providing a target device having an operating system; And c. Providing an operating system for testing and verifying a software program provided to said host computer, said program comprising: a graphical user interface program for interfacing with a user, an engine in communication with said target device and responsive to commands of said graphical user interface; A plurality of test suites comprising at least one test for testing and verifying at least a portion of the means and the operating system, d. Providing logging library means for adjusting and storing test related information generated by said operating system for testing and verifying a software program; e. Executing the operating system for testing and verifying a software program on the target device, testing and verifying the operating system; and f. Generating a pass and fail test result. 14. The computer-based method of claim 13, wherein the computer-based method of testing and verifying an embedded operating system further comprises providing the operating system for testing and verifying a software program having a set of functions in a graphical user interface for invoking engine means. Computer-based method comprising a. 14. The computer-based method of claim 13, wherein the computer-based method of testing and verifying an embedded operating system further comprises providing the target device having an event-driven application for communicating with the engine means, Said engine means transmitting an event and said event driven application response. 14. The computer-based method of claim 13, further comprising providing the target device having a Windows CE operating system. 14. The method of claim 13, further comprising providing a test suite in the form of a system stress-testing routine comprising a code base for stress testing at least one portion of the operating system. Parts include Ethernet / NDIS, PCMIA, memory, file systems, serial ports of video systems with multiple application program interfaces, infrared systems, raw equipment manufacturer adaptation layers, touch panels, mice, keyboards, and audio / wave systems Wherein the test identifies at least three defects, i.e., hardware design, hardware programming and operating system interactions, and runs in an automatic or manual mode. that Computer-based method according to claim. 14. The computer-based method of claim 13, further comprising providing an Ethernet connection coupled to the host and the target device for performing testing and verification at the target device. 14. The computer-based method of claim 13, further comprising providing a logging library means having at least one pass test result file using a WRITETESTPASS application program interface. 14. The computer-based method of claim 13, further comprising providing a logging library means having at least one failure test result using a WRITETESTFAIL application program interface.