KR20020012254A - Protocol acknowledgment between homogeneous systems - Google Patents

Abstract

The test and validation software program of the host computer includes a graphical user interface program, an engine in communication with the target device and in communication with a command from the graphical user interface, and at least one test for testing and validating at least one component of the system. The branch includes a plurality of test units, and a protocol aware software package 4000 for use with the target device, which is a protocol related field that distributes execution threads waiting for a recognition message from the target device. Using an operating system-generated event handle, the event handle is located in the header of acknowledgment message packet 3002 and sent back as an acknowledgment message, with the receiving thread waiting for an event handle for acknowledgment message 2003. Transfer execution threads Unblock

Description

Protocol recognition method and apparatus between similar systems {PROTOCOL ACKNOWLEDGMENT BETWEEN HOMOGENEOUS SYSTEMS}

Developers are increasingly embedding operating systems such as Windows CE in different types of computerized products, including set-top boxes, gaming systems, bar code scanners, and factory automation systems. As Windows CE grew, so did the need for "off-the-shelf" software development tools. Several tools and “off-the-shelf” software kits have emerged on the market leading the way in rapid device development to save device design time, but any high-speed system or method of identifying the compatibility of these new products, especially at the end of device development, does not exist.

Traditionally, only two operating system device testing options have been used: (1) in-house recording of test code or (2) outsourcing customer code development to another company. To complete an in-house testing project, OEMs take months to train their staff, more months to develop test code, and even test their products using these codes. It may take more months just to get ready before it is. Similarly, it may take months for an outsourced customer code developer to write code. Therefore, both options are time consuming and expensive.

In the art associated with the QA testing system, several network protocols are used, which suspend program execution while waiting for the acknowledgment of the transmitted message. For example, Transmission Control Protocol (TCP) blocks program execution at a lower level until a particular transmitted message is acknowledged by the remote "on connection". Most protocols are performed in an O / S-independent manner. As such, a sent message requesting approval must be maintained in a list that cross-references the identification of the message sent using the thread of execution, and the thread waits for the approval. Such related art systems follow a long sequence: generating a message ID before sending the initial message; Add an element to the list that associates the thread of execution with the message ID; Message transmission; Blocking execution thread transmissions until the receiving code unblocks it; Acknowledging the transmitted message by sending back an ACK message containing the original message ID by the remote process; Parse the message ID of the first message sent; Reference message IDs in the list; Determine which execution threads have been released from the list; Finally, it releases the first thread of execution to continue executing the program.

These time-consuming operating system device testing options include integrated testing that uses product approvals between similar systems to improve product quality, save time and money for multiple people and time, and improve the flow of the product development process. There is a strong demand for the system. In particular, methods and devices for testing and validating devices with the built-in Windows CE operating system installed have become necessary to overcome the above problems, thereby improving product quality and improving time and cost for different people and time. Methods and devices are provided that save and improve the flow of the product development process, resulting in a fully automated design identification package for device designers. In particular, there is a need for systems that use O / S-supplied events, O / S-internally maintained event lists, and O / S-internally maintained blocking threads (ie, transport threads) lists. Therefore, unlike techniques related to protocol approval methods, several steps need to be eliminated (for example, adding an element that associates a message ID with a thread of execution in a list, cross-referencing a message ID in a list, and some execution). Determining whether a thread is emitted from a list entry).

Therefore, a simpler, shorter, less error-prone code would be helpful through alternative ways of handling ACKs, where multiple actions can be executed when receiving an acknowledgment message (e.g., Cross-referencing and consequent delay elimination, code for receiving ACKs that are very identical to a single threaded case, priority data that does not need to be maintained in the transmitted message if multiple threads have priority, and all threads with proper priority. List of IDs that do not need to be scanned completely to determine thread-emission priority when / S restarts. A useful system can run several waiting execution threads waiting for an ACK message, which is only needed to use the O / S-provided function to wait for event handles that were built by the initial "send" of the protocol header. . Similarly, several advantageous threads of execution should be triggered without further cross-reference processing by an ACK for any of the multiple transmitted messages.

This patent application claims the chapters of U.S. Provisional Application No. 60 / 137,629, filed June 4, 1999 and entitled "PROTOCOL ACKNOWLEDGMENT BETWEEN HOMOGENEOUS SYSTEMS." Partial serial application of US patent application Ser. No. 09 / 489,308, entitled US Patent Application No. 60 / 116,824, filed Jan. 21, 1999, entitled "CE VALIDATOR TEST SUITE."

The present invention relates to product quality assurance and to test systems and methods for validating an operating system provided on a computerized product. In particular, the present invention relates to product quality assurance and to test systems and methods for validating operating systems during development of computerized products. Furthermore, the present invention relates to product quality assurance, and to a test system and method for validating an operating system such as Windows CE, manufactured and sold by Microsoft, Raymond, Washington, which is typically provided for computerized products. It is about.

Reference numerals are used together for the understanding of the present invention.

1.0 is a schematic diagram of a computerized product provided with an operating system embedded in a control unit.

2.0 is a manufacturing flow diagram illustrating QA testing in a computerized product provided with an embedded operating system in accordance with the present invention.

3.0 is a block diagram illustrating the major parts of an operating system validator of the present invention that includes a graphical user interface, an engine, multiple test suites, and a logging library in accordance with the present invention.

4.0 is a block diagram illustrating the execution of multiple test suite configurations from a host device testing multiple target devices provided with an embedded operating system in accordance with the present invention.

FIG. 5.0 is a block diagram illustrating the invention shown in FIG. 4 excluding communication by a target device using an O / S validator at a host by Ethernet means.

5A illustrates an arrangement in which communication between a target device and multiple hosts may occur over Ethernet in accordance with the present invention.

6 illustrates another apparatus for a test suite execution state in accordance with the present invention.

7.0 is a table listing of the functional areas of a particular API tested in automatic and manual test suite execution in accordance with the present invention.

8A, 8B and 8C all contain a comprehensive table listing of their respective APIs and functional areas that can be tested in automatic and manual mode in accordance with the present invention.

9.0 is a table listing of selected APIs for use in building automated scripts in accordance with the present invention.

10.0 is a schematic diagram illustrating the concept of the present invention providing source code for all executable programs in accordance with the present invention.

11.0 is a block diagram representative of a window showing test suite selection options in addition to other related summary functions in accordance with the present invention.

12.0 is a block diagram representing a logging window showing test results, test failures, and associated test summary options in accordance with the present invention.

13.0 is a graphical representation of test cycle time as a function of the number of test devices jointly tested according to the present invention.

14.0 is a block diagram representing a configuration window showing tabs for executing multiple configuration relationship functions in accordance with the present invention.

15A, 15B, and 15C all contain table list testing details of an operating system system portion in accordance with the present invention.

16 is a flow chart illustrating unique concurrent multithreading capability for a similar intersystem authorization protocol in accordance with the present invention.

Accordingly, the invention, which can be used commercially under the Applicant's trademark of CEValidator ^™ , is an operating system validator (hereinafter referred to as an O / S validator), designated by the number " 1 " in the figures, and newly developed target device hardware. And / or providing a test system including an automated test suite method for testing a port of an operating system such as Windows CE in software. The O / S validator provides a unique method of concurrent multithreaded execution of authorization protocols between a comprehensive code base, device driver, OEM adaptation layer (OAL), and similar systems specifically developed to intentionally emphasize O / S. Includes hardware interconnection used. The test suite provided focuses on validating three major flaws: hardware design, hardware programming (driver / OAL), and operating system interaction. The emphasis in particular diagnostics is on the Windows CE subsystem, which historically shows most problems. The test suite provides a complete analysis of the Windows CE port and includes nearly 1500 tests that include system stress-testing routines in addition to feature and functional tests. These tests are grouped by O / S validator. O / S validators include both test source code and executable programs for all tests.

To simplify the execution of the test suite and the collection of logging results, an intuitive user interface for an O / S validator host facility, such as a standard Windows application using the Microsoft Windows user interface, is used. O / S validators distribute test suites, such as client / server applications. The graphical user interface (GUI) interacts with CEHarness.exe, a small application running on the target device. Since this communication can occur over Ethernet, at least one host will run suits for at least one target device.

The O / S validator generates useful error information when the target device fails the test. While the suits are running, the results are displayed in a number of dynamically generated log windows in addition to the Configuration Summary tab. The logging window contains the complete text of the given test result. Failure is color coded red to facilitate identification. Navigation buttons in the logging window allow the user to quickly move from one failure to another. Test logging APIs cause prologs and epilogs to be generated in each result file. Battery power levels, and run dates and times, which are information such as concurrently running processes, are automatically recorded in the result file and displayed in the log window. Useful summary information such as program memory loss, storage memory loss, or total execution time is provided in the Log Window tab. Summary information for a given test result is collected and displayed in the Summary tab of the configuration window. The Summary tab reports the number of pass and fail test cases in real time. The number of passes and failures for the breakout individual suit is also displayed. The Summary tab in the configuration window facilitates quick navigation to individual failures among thousands of test results. The exact source file and line count corresponding to the logged failure are automatically reported by the O / S validator's logging API. Since the O / S validator provides source code for all executables, direct access to the error reporting source code is a powerful appendage to the textual description of the failure.

The present invention uses an O / S-generated event handle as a member field of the protocol that emits an execution thread waiting for an acknowledgment (ACK) message. O / S-generated event handles, such as WIN32 event handles, are used to block the execution of the original transfer thread and are placed in the header and sent back to the ACK. The receiving thread does not need to refer to the transaction validation (ID) in the list. Instead, the receiving thread unblocks any thread waiting for an event. That is, the present invention uses O / S-provided events, O / S-internally-maintained event lists and O / S-internally-maintained lists of blocking threads (ie, transport threads). Therefore, unlike the protocol approval method, which is an association technique, several steps are eliminated: adding an element that associates a thread with a message ID in the list, referring to the message ID in the list, and which execution thread to emit from the list entry. decision.

As a result, the code is further simplified by using an alternative method of the present invention regarding the processing of ACK, which is shorter and less error. Therefore, the present invention provides several advantages of facilitating multiple actions upon receipt of an acknowledgment message: cross-references and accompanying delays are eliminated, and the code for receipt of an ACK is very identical to a single-threaded case, with priority data Does not need to be kept in the sent message list if multiple threads are prioritized, and the ID list does not need to be scanned globally to determine thread-release priority when O / S restarts all threads with the appropriate priority. . In order to perform the execution of a number of suspended threads waiting for an ACK message, the thread is only needed to use the O / S-provided function to wait for the event handle inserted by the last "transmit" in the protocol header. Similarly, multithreaded execution can be triggered by an ACK for any number of messages sent without additional cross-reference processing.

One feature of the invention is described in detail in the embodiments of the invention.

1.0 shows a computerized product 1000, which is a typical computerized product 9, such as a computer workstation, set top box, gaming system, bar code scanner, and factory automation system provided with a built-in operating system represented by numeral 1001a. As shown, the product 1000 may include a typical target device 9 provided with an operating system 1001a, such as an embedded Windows CE operating system (O / S). Computerized product 1000 may function as a standalone device equipped with the present invention, such as an O / S validator, to test and validate its operating system 1001a. Stand-alone testing takes advantage of new test development and debugging of reported defects, as does not require sophisticated awareness of the O / S validator infrastructure. However, in a more similar application as shown in FIG. 2.0, the product 1000 is equipped with an O / S validator for testing the present invention, i.e., the target device 9, in a production quality assurance testing environment M and Operating system 1001a may function as a host computer 4 provided. Referring back to FIG. 1.0, computerized product 1000 has, for example, a number of input / output ports 1021, a keyboard 1009, a printer 1010, a mouse 1011, and a monitor 1012. It may include several subcomponents, including the control unit 1020. The subcomponents 1009, 1010, 1011.1012 can be self-testable target devices. A typical control unit 1020 itself may be a central processing unit 1001, a storage device such as a hard disk drive 1004, other memory components including a RAM 1002, a ROM 1003, a compact disc 1005, audio Various sub-components, including component 1006, network / server card 1007, and modem 1008. The control unit must include an operating system 1001a for the product 1000 to function as a useful device.

FIG. 3.0 shows the main parts of the O / S validator 1 comprising a graphical user interface (GUI) 2, an engine 3, a plurality of test suites 11 and a logging library 12. The GUI 2 and the engine 3 communicate internally in both directions via a part designated by the number 7 in the O / S validator and named HarnessLink.dll, which will be described in detail below. 4.0 shows a host computer 4 provided with an O / S validator 10. As shown, a number of target devices 9 are provided with an O / S 1001a for testing in accordance with the present invention. The O / S validator 1 has the ability to create a testing configuration, such as a number of test configurations 21a, 21b, 21c, for testing a particular function under the control of the OS 1001a in the target device 9. The side part of the device named CEHarness 8 communicates with the engine 3 of the O / S validator 1. As shown in Figs. 5, 5A, the CEHarness 8 can also communicate with the engine 3 of the O / S validator 1 via the Ethernet means 4a. FIG. 5A shows that multiple hosts 4 can run suits for multiple target devices 9 since communication can take place over Ethernet. In another embodiment, as shown in FIG. 6 for the test suite execution state, the CEHarness 8 may also communicate with the engine 3 of the O / S validator 1 via the suit execution connection 1021a. Wherein the host computer 4 may comprise an NT host computer, a logging library 12 may be provided to the target device 9, and a test suite may be provided via the socket connection 1021b. Is provided.

In operation, the O / S validator 1 tests and validates the target device 9 provided with an embedded operating system, such as the Windows CE operating system. Broadly speaking, the O / S validator provides (1) validation of the Windows CE port to the target device, (2) provides stress and performance testing, (3) logging and analysis of results, and (4) multiple application programs. Test multiple functional areas including interfaces (APIs) (see Tables 1.0 and 2.0 in Figures 7,8A, 8B, and 8C), (6) customize tests in automated suites, and (7) host Providing side graphical test facilities, (8) stressing and evaluating memory performance, (9) providing means for automating building tests that include multiple APIs (see Table 3.0 in Figures 3.0 and 9.0), and (10) CEAnalyzer It works by providing a result analysis tool named. As described above and as shown in FIG. 10, the O / S validator 1 includes test source code SC and executable program EP (both referred to as executable tests) for all tests. do. The sole performance requirement for an executable test (EP) is that test cases "pass" and "failure" are reported using two specific APIs: WRITETESTPASS () and WRITETESTFAIL (). Such a macro has a signature similar to the known printf () function, but its use generates a standard test case result format in a "result" file for automated summarization, integrating reporting with the host user interface. The O / S validator 1 method also includes mediation between the test cases encapsulated in the GUI 2 and the executable test (EP), and provides a means for the GUI 2 to distribute and control test execution. . The test suite 11 includes a text file 5 consisting of suit language instructions (eg, PUT, RUN, WAIT and DELETE) which are direct representations of the work required to distribute and execute the test. Other supported suite instructions are GET to retrieve files from target device 9, RUNHOST to run executable program (EP) of host computer 4 useful for client / server style testing, processing on host computer 4 WAITHOST, useful for client / server style tests waiting for termination, PUTSYSTEM for entering files into the system's system directory (/ windows), SLEEP for default timing when everything else fails, and for displaying message boxes on the host machine. MSGBOX, SETREG for adding or changing the device's registry settings, and DELREG for removing registry settings from the device. In addition to providing internal suit documentation, internal suit file comments are provided as the suit description of the GUI 2. The O / S validator 1 method involves organizing the test suite file 5 by hierarchical location in the directory structure of the host computer 4. As shown in FIG. 11, the test suite 11 is divided at a higher level, which is one of the automatic test suit Au, the manual test suit Ma, or the stress test suit SS. Automatic suit (Au) is a test that does not require any user intervention during its execution. In contrast, the manual suit Ma requires user intervention (eg keyboard and touch panel suit). The O / S validator (1) method degrades conventional file systems with multithreaded concurrent stressing by pushing input / output (I / O) processing to the limit and operating without available program or object storage memory. Thereby stretching the system through the stress suite (SS). Hierarchically below the upper level, the O / S validator 1 method involves arranging the test suite 11 by functional area (see Fig. 7).

As mentioned above, FIG. 3.0 shows a major part of the O / S validator 1 which includes a graphical user interface (GUI) 2, an engine 3, a plurality of test suites 11 and a logging library 12. Illustrated. The O / S validator 1 uses the GUI 2 as a visual basic part because it works well at any level of the user. The GUI 2 design is based on the concept of interfacing user input with subparts. The engine 3 holds the functional core on the host 4. The engine 3 reads a number of suit files 5, describes them, and executes the commands. The engine 3 obtains information from the GUI 2 and uses the information to set up a number of execution options. The engine 3 is written in the C / C ++ language. The engine 3 is linked to the GUI 2 through a portion named HarnessLink.dll 7 which is an ActiveX control. HarnessLink.dll 7 is described and called from the GUI 2 by a number of information passed to the engine 3 before execution begins. The Dll link 7 also functions to communicate between the engine 3 and the GUI 2 during execution to relay information, error messages and dynamic run-time instructions. The target device 9 comprises a device side (as opposed to a host side) called CEHarness 8. The CEHarness 8 is a C / C ++ program residing on the target device 9, and as shown in FIG. 4, communicates almost exclusively with the engine 3 unless it broadcasts the target information on the network, and the GUI 2 Receives the information and passes it to the engine 3 (see FIGS. 5, 5A). CEHarness 8 is an event-driven application, engine 3 sends an event and CEHarness 8 responds. The two remaining parts, the test suite 11 and the logging library 12, are intertwined because the test suite 11 is written using multiple application program interfaces (APIs) that are part of the logging library 12 (FIG. 8A). , 8B, 8C). The API 13 has substantial functionality and is somewhat dependent on the information passed through the component chain. The logging library 12 has a simple functional concept, and logs TCP / IP information 16 back to the GUI 2 (see FIG. 5.0), or writes the results (14) and log files (15) to the device (9). ) Directly (see FIG. 6.0) to generate a log file 15 to convey the test results (14). As shown in FIG. 12, the logging window LW uses a summary tab sumT that utilizes user results, passes, failures, and timing information for the test results 14, failures F, and program memory of the test file 15. To show. The plurality of test suites 11 includes the essentials of the O / S validator 1. FIG. 13 shows a graphical form in which the test cycle time CT is reduced as is the case when multiple test devices are jointly tested.

More specifically, the GUI 2 is complex code that follows the degree of functionality needed to handle layers of components. The GUI 2 provides a "wizard" whose primary function is to guide new users by selectively setting and listing default settings. As shown in FIG. 14, the GUI 2 also provides a configuration window CW as a means of executing a test run including a single pass through a series of selected suits 11 on the target device 9. . As shown in FIG. 4, the multiple configurations 21a, 21b, 21c can be operated to simulate multiple scenarios. The content of the configuration window CW includes a number of tabs for user control. As an example, the suit tab S provides a tree structured suit file directory under the O / S validator directory. This tree structure is organized to provide meaningful distinction between the types of tests 11 that the user selects. In addition, this tree structure is created when the user opens the configuration window (CW), and the user adds a new user input suite to the suite file directory created by the O / S validator (1) installer. The validator 1 can be extended. The test suites 11 are command scripts that perform an action corresponding to the command of the script after the engine 3 reads it. The suit file 5 generally begins with a series of instructions. The above commands appear in the suit file information section of the file. If the word "Manual" appears at the top of the soot file 5, the file is considered to need manual execution and is thus assigned to different icons. In the test suite section, as shown in FIG. 11, the user can request the test suite file 5 again in any way. Referring to Figure 14, the logging tabs contain valuable information. The user can select three logging methods: LH for logging to host 4, LTD for logging to target device 9 or LHD for logging to both host 4 and target device 9. . The log information is then stored in the configurable directory listed in the edit box. All this information is transmitted to the engine 3 via the DLL 7 and then to the CEHarness 8 of the target device 9. Next, the information is obtained by the logging library 12 when the test 11 is running. The other tabs in the configuration window (CW) can be used to reduce program and storage memory by selecting high priority thread selection, tap PM and SM during suite execution by a set of stress state tabs (SC), thread tabs (T), and tap SRT. Runtime selection by selecting, and stopping by selecting tab STOP. You can use an infinite loop tap to find the system's memory leak. Useful summary information such as program memory loss, storage memory loss, or total test execution time is provided in the Summary Tab (SumT). Summary information for a given test result is collected and displayed in the Summary Tab (SumT). The Summary tab reports the number of PASS and FAIL test cases in real time. The breakout PASS and FAIL numbers for the individual suits are also displayed. The Summary tab of the configuration window facilitates quick navigation to individual failures, perhaps among thousands of test results. The exact source file and number of lines corresponding to the logged failure are automatically reported by the O / S validator's logging API. Since the O / S validator provides source code for all executables, going directly to the source code that reports the error is a powerful addition to the textual description of the failure. Logging options change their execution and effects dramatically. The first option is to assume that whenever a user runs a test suite 11 indicating "pass", a series of log files 15 summarizing these test results 14 are automatically generated. Depending on the logging method selected, the summary file 15 is generated by the CEHarness 8 or the engine 3. By default, the engine 3 traverses all log files 16 in the logging directory, so the user can accept a list of log files that do not correspond to the test 11 being run. In order for the summary file 15 to indicate more of the startup test 11, the user can delete the log directory before startup, and the user can see that the engine 3 has only one log file 15 for each test 11. You can choose the logging option to choose to return to the most recent summary only. This means that if a user runs a file system test 30 times on a given day, there will only be an entry in the summary log for the test corresponding to the most recent run. The summary log will have only one entry for each test 11 in which the log file 15 is still logged. The other two options are handled in the GUI 2. If the user is logged into the host 4 via the TCP / IP connection 4a, the entry will enter the log file 15 created on the user host 4 while appearing in the log window in the GUI 2. This enables the user to examine the log file 15 in the environment of the O / S validator 1 and is more beneficial because the user can immediately monitor the passage and more important failures. However, under certain circumstances there will be an excess of log files 15 that follow the size of the test 11; Therefore, closing the log window LW without opening it any more will keep the memory of the host 4 and also provide a clean viewing area for the O / S validator 1. Another advantage is to close all log files 15 without fail. The failure indicates to the product design personnel that the target device 9 will require other improvements. The user will want to keep opening all failed log windows F by clicking an option in the logging window to keep the F window open.

In operating the present invention as shown in FIG. 5A, a window named as an available target shows an active device on the network that broadcasts information to the GUI 2. The active device sends large information, some of which are displayed in the available target window. The user can view the information by selecting the View / Enable Target menu. The other window must be accessed to obtain a complete set of broadcasted information. This broadcast information is valuable because it is used to start a connection from the engine 3 to a particular CEHarness 8 in the test target device 9.

The stress test setting SS will be described in detail with reference to FIG. 11. Stress suits 11 come in many forms; However, their main purpose is to stretch the target device 9 by running a very long test, to repeat a short test several times or to select a wide range of parameters in one test. For example, the rest of the specs for your test 11 area, such as the database stress suite, only stress the database capabilities of the O / S, such as Windows CE. The stress test options are distinguished from the stress suit 11. The stress test options are different because they target features to provide more of the same broadband stress scenario as the real world usage model. This scenario runs in conjunction with any user selected set of test suite files 5. The stress test options can also be run jointly and separately, thereby providing a significant boost over the scope of any test plan. The first two stress test options relate to the memory of the target device 9. The first stress test option is a low virtual memory option that significantly reduces the amount of virtual memory of the target device before the selected tests 11 are run. This simulates a realistic harsh environment that can occur when a user has 15 open applications, and affects malfunctions. The second stress test option is a low storage memory option. When the second stress test option is selected, the storage device of the target device 9 is filled with maximum capacity to experience the target device 9 response to the low storage memory solution. In some cases, this second stress test option is efficient for testing an application program contained in the target device 9 that may follow a non-existent storage memory. The third stress test options that follow are run options. The first viable stress option is an infinite loop, which is the same for long test cycles. A common problem for many device drivers is malfunctioning under long, strong and stressful situations. This infinite loop stress test provides a test to determine possible breakdowns. The infinite loop test above runs the selected suit 11 until the user manually presses the stop button. The next stress run option is a configurable CPU cycle loss test that is available as a text file named Data.txt and separated by the O / S validator Tests TestInputFiles. Two examples are provided for files that users can copy, reuse, or modify. The text file, Data.txt, controls the number of threads and their characteristics that a user can include to run his test. That is, the user can run his or her test while other processing consumes CPU time and eliminates various problems, including timing. The final stress test option is random execution. When the user selects this option, the GUI 2 will re-request the list of test suites 11 at run time to run in different orders. This option is ideal because it facilitates the diagnosis of interaction problems in several parts.

The remaining test run options are common activities that can be controlled by the user. The first option, "Use the selected target exclusively", is important, which means that when the target device 9 is connected over Ethernet, the O / S validator 1 can be used by other users on the subnet to use the target device window. This is because the target device 9 can be accessed through the target device 9. This helps to create stress in the target device 9. If the user wants to isolate the problem, no extra stress should be applied. In such a situation, the user must have exclusive access to the target device 9. The last test run option is to set a target time that prompts the engine 3 to transfer time from the host computer 4 to the target device 9, and thus the target device 9 system at the host computer 4 time. You can synchronize the time. Synchronization is advantageous because the log file returns to the date and time stamp associated with the date and time at the target device 9. In order to maintain this accuracy, the user must set the target device 9 date and time. The last tab before running the test is the Preferences tab, which contains valuable information about the various selectable environment variables. The environment setting is designed to extend and refine the test suite by having the test suite include environment variables instead of hardcarded information. For example, a series of tests takes available comfort as a parameter. If the comfort is not entered as a variable environment, the test will fail, because the comfort cannot be opened. All environment variables used in the suite are provided; However, any additional environment variables will be users added to the user-input suite. After running the test, the test state is variable to obtain state information for the current test run. This information is updated dynamically.

The suit run section window lists the selection suits that have started. The user can open any log file from this window by selecting the desired test icon. Other icons under this control provide failure information. The failure icon is shown as a stylized beaker that is crossed with a red "X". The test run summary information maintains the possible number of tracks, selected suits, failed suits, and percentage of failed suits for the test suit file. When the test run is complete, the user can select the Configure Failed Suits tab, which selects all failed suits in the current test run and prompts for the presence of a new configuration window to facilitate return testing.

The other two sections are named Test Details. One of the test details sections monitors individual test cases as to which sections pass or fail critical for measuring the value of the test run. The remaining test details section is the failed suite section where all selected failed suites are listed by suit name, indicating the number of pass and failed test cases. All of the above information provides the user with a very efficient idea of the limitations of his target device 9 during the test run (ie what passed and more importantly what failed).

The main object of the present invention is to properly test the ports of the O / S 1001a, such as Windows CE. To accomplish this task, hundreds of suit tests 11 are needed and provided by the O / S validator 1. As an example there are provided nearly 1500 test suites 11 grouped by validated O / S subsystems. As shown in FIG. 10.0, the O / S validator 1 is a source code that covers the main O / S 1001a subsystem and common adaptive drives with specific emulations for the subsystem that historically shows most problems. Contains executable code for all tests 11. O / S subsystems tested by O / S validators include Ethernet / NDIS, serial port drivers, display drivers, touch panel drivers, mouse drivers, keyboard drivers, OEM adaptation layers, and PC card adaptation drivers. 15A, 15B, and 15C are tables listing testing details of these system components.

As described above, in Fig. 3, the engine 3 is linked to the GUI 2 through a portion named HarnessLink.dll 7 which is an ActiveX control. In particular, HarnessLink.dll 7 provides a function for a set of GUIs 2 to invoke (ie, execute) the engine 3. Most HarnessLink.dll (7) function-calls set up parameters for the command line of the engine 3. All initial information is passed to the engine 3 via this command line, ensuring the preparation of the engine 3 for a particular execution. Many GUI-related functions provide information on the command line. The information on the command line corresponds to the GUI 2 information. Another major function assisted by HarnessLink.dll 7 conveys the activity at startup of the engine 3 by opening a named pipe. If an error or necessity arises to transmit the memory information, the engine 3 passes through the named pipe. The named pipes ensure that the communication between the engine 3 and the particular harness link is direct and accurate, and there are no radiation problems if multiple engines 3 are running. When HarnessLink.dll 7 receives a message from a pipe, the GUI 2 signals the appropriate VB event to take the information and process it. In FIG. 5.0, the engine 3 then communicates with one HarnessLink.dll 7 in communication with the CEHarness 8. The engine 3 execution is simple: the command line is received and processed, establishes an execution socket connection to the target device, opens a pipe for communicating with the GUI 2, reads the test suite file 5, The test is run in three phases: before, after, and after. The pre-execution step establishes an error socket connection between the target device 9 and the host 4. Relevant data, such as logging paths and styles, various test run information, and stress scenarios, are transmitted during all phases of execution. The execution phase includes a response to each successive suit command. The suit command is generally sent by the host 4 and is handled by the CEHarness 8 which responds with a socket message when the command execution is complete. Post-execution steps mainly include reducing log information and generating a summary log. When this summary log is complete, the engine 3 is present. The CEHarness 8 of the test target device 9 is considerably more complicated than the engine 3. This complexity is because each device 9 has an instance of CEHarness 8 at any time; However, the apparatus can handle a large number of simultaneous connections, and very important characteristics for the testing method are provided by the O / S validator 1. When the user starts CEHarness (8), it creates two threads that endure through the entire execution time, the broadcast thread and the execution thread. It updates information such as device IP, connection type and available comfort every 10 seconds, sending broadcast messages to the network at the same rate. If the device 9 is connected to the host computer 4 via the Windows CE service (i.e. on the NT side) and connected to either a ramnet or a repllog (i.e. on the side of the device). , The connection type will be changed to PPP_PEER. If so, the broadcast message is sent only to the host 4 to which the target device 9 is directly connected. If the user changes the connection while running, the message is updated. The execution thread, on the other hand, waits for a connection attempt from the engine. When an execution thread receives a connection, it spawns another thread, the main execution thread, which executes several necessary functions. The main thread of execution starts another socket for sending any error or memory information. The thread of execution is therefore event-driven and receives commands and responses as appropriate. Each connection attempt spawns its own thread of execution; Thus, one CEHarness 8 can have multiple connections, extending test functionality by running multiple configurations simultaneously on the target device 9, thus creating a more realistic stress situation.

16 is a flow diagram illustrating a protocol approval system 4000 of the present invention that includes a message sender 2000 and a message receiver 3000. The message sender has a receive thread 2001 and a send thread 2009, which can be parallelism information. The transport thread 2009 generates a message at block 2008 and places a header around the message to form a packet with a WIN32 event handle in the field of the head, as shown in block 2007. In block 2006 the packet is delivered either to block 2005 blocking the transport thread 2009 or to block 3001 receiving the packet by the message receiver 3000, which is then shown in block 3002. As described above, a packet having a WIN32 event handle in a field of a header may be transmitted to the receiving thread 2001 through the receiving block 2002, or the processing may be continuously performed according to the message receiver 3000 function. Receive thread 2001 with a received acknowledgment packet then uses the WIN32 event handle in the header portion of the acknowledgment packet for the transmit thread at block 2003, and subsequently transmits the transmit thread 2009 at block 2004. Retry or continue processing the receive thread 2001 function.

Therefore, the logging library 12 shown in FIG. 3.0 is a composite tool integrated into the O / S validator 1 in many ways. Primarily, the logging library 12 is integrated into a source file for testing. The test calls the library API when the library handles all the details related to capturing and recording the test results. The logging library 12 also supports several connection options. The recommended option is TCP, which allows the user to read the log file when moving from a TCP connection. Another communication option is direct logging of files on the device. This may be advantageous, for example, if the user wants to log onto a PCMCIA card but does not want special information broadcast on the network. Although the logging library 12 acts as the device side part for TCP communication, the host 4 part acts as the GUI 2 side part for communication. This aspect of the logging library 12 provides color coding for the log window and test results on the host 4. For example, the failure message is displayed in red line. In addition to the name and location of the sort code file, the number of lines for which a message is generated is included in the logging library 12 message. Also very importantly, detailed error messages are provided to describe the current event in the program. Each log window has a summary tab that facilitates user access to program memory, pass, fail, and timing information. Another important characteristic of log files is the capture of large pieces of information at the beginning and end of the test. This information provides a snapshot of memory, system, power, and other critical information.

The information described herein can sufficiently achieve the above object of the present invention, and the preferred embodiments of the present invention are representative of the main issues of the present invention and are most widely considered by the present invention. The scope of the invention includes other embodiments that are apparent to those skilled in the art and are limited only by the claims. The same structures and functions as the above-described preferred embodiments are known to those skilled in the art and are cross-referenced herein. Moreover, there is no need for a method or apparatus for describing a problem solved by the present invention to be covered by the claims. Furthermore, no elements, components or method steps of the present invention are intended to be used in the common sense regardless of the elements, components or method steps described explicitly in the claims. No claim is made under the assumption of the sixth paragraph of 35 U.S.C. 112, unless an element is cited using the phrase "means for."

Claims (20)

A computer-based system for testing and validating embedded operating systems in target devices: a. A host computer; b. A target device provided with an operating system; c. An operating system for testing and validating a software program provided to the host computer, the program comprising: graphical user interface program means for interfacing with a user, engine means for communicating with the target device and responsive to commands of the graphical user interface; A plurality of test suites including at least one test for testing and validating at least a portion of the operating system and a protocol approval software package executed to use the target device, wherein the protocol approval software package is from the target device. It uses an operating system-generated event handle as a member field of the protocol that emits an execution thread waiting for an acknowledgment message, which is located in the header of the message packet and forwards back to the acknowledgment message. Sent, and the receiving thread unblocks any transmitting thread execution waiting for the event handle of the acknowledgment message; And d. And a logging library means for adjusting and storing test-related information generated by the operating system for testing and validating software programs. 2. The computer-based system of claim 1, wherein the operating system for testing and validating a software program further comprises a set of functions in a graphical user interface to invoke the engine means. The apparatus of claim 1, wherein the target device further comprises an event-driven application for communicating with the engine means, wherein the engine means sends an event and the event driven application response. Computer-based system. The computer-based system of claim 1 wherein the operating system of the target device comprises a Windows CE operating system and the event handle comprises a WIN32 event handle. The computer-based system of claim 1 wherein the test suites include at least one system stress-testing routine and at least one characteristic and functional test. 6. The system of claim 5, wherein the system stress-testing routine comprises a code base for stress testing at least one portion of the operating system, wherein the at least one portion of the operating system includes Ethernet / NDIS, PCMIA, memory, Selected from the group of operating system parts including file systems, serial ports of video systems with multiple application program interfaces, infrared systems, raw equipment manufacturer adaptation layers, touch panels, mice, keyboards, and audio / wave systems; Is at least three defects, i.e., hardware design, hardware programming and system interactions, wherein the computer-based system runs in an automatic or manual mode. The computer-based system of claim 1, wherein the target device includes a standalone unit provided with the operating system testing and validation software program for performing validation and stress testing independently of the host computer. The computer-based system of claim 1 further comprising an Ethernet connection coupled to the host and the target device to perform testing and validation tasks. 2. The computer based system of claim 1, wherein said logging library means comprises at least one pass test result file utilizing a WRITETESTPASS application program interface. 10. The computer-based system of claim 9 wherein the at least one pass test file is present on the target device. 2. The computer based system of claim 1, wherein the logging library means comprises at least one failure test result using a WRITETESTFAIL application program interface. 12. The computer-based system of claim 11 wherein the at least one failed test file is present on the target device. As a computer-based method of testing and validating an embedded operating system in a target device: a. Providing a host computer; b. Providing a target device having an operating system; c. Providing an operating system for testing and validating a software program provided to said host computer, said program comprising graphical user interface program means for interfacing with a user, communicating with said target device, and responding to commands of said graphical user interface; A plurality of test suites comprising engine means, at least one test suite for testing and validating at least a portion of the operating system, and a protocol approval software package executed to use the target device. Uses an operating system-generated event handle as a member field of a protocol that emits an execution thread waiting for an acknowledgment message from the target device, the event handle being located in the header portion of the message packet. Is sent back to the message, the recipient thread is frozen and blocking any transmitting thread running to wait for the event handler in the grant message; d. Providing a logging library means for adjusting and storing test related information generated by the operating system for testing and validating a software program; e. Executing the operating system for testing and validating a software program on the target device and testing and validating the operating system, wherein the executing step executes a transport thread execution in response to the event handle of the acknowledgment message. Unblocking; And f. Generating a pass and fail test result. 14. The computer-based method of claim 13, wherein the computer-based method of testing and validating an embedded operating system provides the operating system for testing and validating a software program having a set of functions in a graphical user interface to invoke engine means. Computer-based method further comprising. 14. The computer-based method of claim 13, wherein the computer-based method of testing and validating an embedded operating system further comprises providing the target device having an event-driven application for communicating with the engine means. The engine means transmitting an event and the event driven application response. 14. The computer-based method of claim 13, further comprising providing (a) the target device having a Windows CE operating system and (b) providing the event handle having a WIN32 event handle. 14. The method of claim 13, further comprising providing a test suite in the form of a system stress-testing routine comprising a code base for stress testing at least a portion of the operating system. Portions include operations including Ethernet / NDIS, PCMIA, memory, file systems, serial ports of video systems with multiple application program interfaces, infrared systems, original equipment manufacturer adaptation layers, touch panels, mice, keyboards, and audio / wave systems Selected from the group of system parts, wherein the test identifies at least three defects, namely hardware design, hardware programming and operating system interaction, and is executed in an automatic or manual mode. 14. The computer-based method of claim 13, further comprising providing an Ethernet connection coupled to the host and the target device to perform testing and validation tasks on the target device. 14. The computer-based method of claim 13, further comprising providing a logging library means having at least one pass test result file using a WRITETESTPASS application program interface. 14. The computer-based method of claim 13, further comprising providing a logging library means having at least one failure test result using a WRITETESTFAIL application program interface.