JP2001243089A - Device and method for verifying software - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a verifying device for easily searching the cause of the abnormality of software to be tested. SOLUTION: A software verifying device is provided with a test program storing part 110 for storing a test program for testing software by using the API of software to be tested and a test automatic executing part 120 for starting a debugger by designating the test program stored in the test program storing part 110 as a program to be operated by using the debugger, and for setting a break point at the part of the designated test program for allowing the test program to grasp an error by the API of the software by using the started debugger, and for executing the set test program by using the started debugger.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a computer operating system (OS).
The present invention relates to an apparatus and a method for testing an operation system. In particular, the present invention provides an automatic test execution method for a verification device that checks the service contents of an OS using an application programming interface provided by the OS, which facilitates pursuit of a cause when a test fails.

[0002]

2. Description of the Related Art Conventional methods for automating a program test include, for example, a test method and a program test apparatus disclosed in Japanese Patent Application Laid-Open No. 8-305609. FIG. 31 shows an example of the configuration of a conventional system.
FIG. 31 shows a relationship between the program test apparatus 10, a target system 20 as a means for executing a test target program, and an inspector 30, for example, an operator. The program test device includes a test script management unit 10a, a test automation control unit 10b, a test script execution unit 10c, and a test pass / fail determination unit 10d.
And a program fault specifying means 10e. Further, the program test apparatus 10 includes test script input means 10f, test result output means 10g, mode switching / executing means 10h, and debug command input means 10i as interface means with the inspector 30.
And These interface means 10f to 10i
Are respectively connected to the test automation control means 10b.

In the prior art, the program test operation is as follows. Prepare a test script including at least a statement that outputs an execution result at a predetermined step in the test target program suitable for determining whether or not the test target program has a failure and a test item for the execution result. In addition to executing the program, it is determined whether or not the test target program has a failure by processing the test script in a predetermined step.

Further, in the prior art, there is a fault identification operation. This is performed as follows. The inspector causes the display device of the program test apparatus 10 to display a test script for determining the presence or absence of a failure with respect to the program rejected in the program test operation because the failure includes the failure, using the test script input unit 10f. Next, a test script for fault identification suitable for identifying a fault location of a program including the fault is registered in a file in the storage unit of the test script management means 10a using the test script input means 10f.

[0005] When a failure is determined in the test mode (cited from the above publication), the failure identification test script to be registered is suitable for identifying the failure based on the obtained test results. Then, modify and create the test script for failure determination. For example, breakpoints and data names to be dumped are changed. Thereafter, part or all of the test program is re-executed. As a result, the fault location can be specified by the processing of the fault identification test script.

[0006]

However, as in the prior art, a test method for once detecting a failure and then registering and re-executing a script for specifying the location of the failure is as follows.
There is a problem that it is difficult to find the cause of a failure with low reproducibility. A first object of the present invention is to set a breakpoint on a test program in an apparatus for automatically verifying a test target program using the test program. And stops the test operation.

A second object is to set a breakpoint for the first object at a place where the test program detects a failure, and to obtain the internal state of the test program after stopping the test program so that reproduction can be performed. This is to obtain information for analyzing the cause of a difficult failure when the failure is detected.

A third object is to reproduce a breakpoint set for the first object by obtaining an internal state of the test target program after the test program is stopped in a situation where the test program detects a failure. This is to obtain information for analyzing the cause of the failure when the failure is detected.

A fourth object is to use a resource used by a test program in a situation where a breakpoint set for the first object is targeted at a place where the test program detects a failure.
Depending on the state, the operating system (hereinafter referred to as OS
The information about the execution environment of the test program that affects the execution result of the test program due to the change of the processing described in (1) is obtained.

A fifth object is to define a time-out determination time, determine that the test program has hanged up if the test program does not end within this time, and analyze data for analyzing the cause of the hang-up. After the automatic collection of the test program, the test program is terminated to enable continuous processing of the test operation.

A sixth object is to provide a method for setting a time-out determination time efficiently in the fifth object.

A seventh object is to prevent the automatic execution of the test from being stopped by the abnormal stop of the test program, and to collect data for analyzing the cause of the abnormal stop.

An eighth object is to set a target machine on which the OS to be tested operates in terms of hardware in a situation where the OS to be tested causes a serious failure due to execution of the test program and the test fails due to a system down. After resetting and restarting the target machine, the test operation can be continuously performed.

[0014]

A software verification device according to the present invention is a software verification device capable of activating a debugger for testing software, and includes an application programming program for software to be tested.
Using an interface (hereinafter referred to as “API”),
A test program storage unit that stores a test program for performing the software test, and the test program stored in the test program storage unit is designated as a program that operates using a debugger, and the debugger is started up. A test automatic execution unit that uses a debugger to set a breakpoint at a location of a test program in which a designated test program grasps an error by using an API of the software, and executes the set test program using the started debugger. And characterized in that:

The test program includes an error detection point for grasping that the operation of the software is abnormal based on an execution result defined by an API of the software. The test program storage unit further includes an error detection point for the test program. The test automatic execution unit stores a definition relating to execution of a test program including points, and sets a breakpoint in the test program based on an error detection point of the test program stored in the test program storage unit. Features.

[0016] The software verification apparatus further includes a failure information collecting unit that collects, when the test program stops at the set breakpoint, failure information at the time when the test program stops.

The test program includes variable information as information for specifying variables used during the execution of the test program. The fault information collecting unit extracts the variable information from the test program using a debugger. And collecting variable values of the test program based on the extracted variable information.

The software includes test target variable information as information for specifying a variable used during execution of the software, and the fault information collecting unit extracts the test target variable information from the software using a debugger. Then, values of the variables of the test program are collected based on the extracted test target variable information.

[0019] The failure information collecting unit changes an execution environment of the software, causes the software to execute on the changed execution environment, and acquires an execution result of the executed software.

If the test program does not end after a lapse of a predetermined time, the test automatic execution section stops the test program using the debugger, and the fault information collecting section determines the stop position of the stopped test program. It is characterized by acquiring and acquiring failure information.

[0021] The automatic test execution section changes the predetermined time based on the execution result of the test program.

The test automatic execution section detects that the test program has stopped abnormally, and forcibly terminates the test program detected using the debugger. The fault information collection section forcibly terminates the test program that has been forcibly terminated. In addition to acquiring the end position, failure information is collected.

The test automatic execution unit sends a signal for instructing reset of a computer on which the software operates after the test program abnormally stops, and executes the test program after the computer is restarted by the sent signal. It is characterized by making it.

[0024] The software is characterized in that it includes either an operating system or an application program.

The software verification method according to the present invention comprises:
A software verification method capable of starting a debugger for testing software, wherein a test program for testing the software is stored using an application programming interface (hereinafter, referred to as “API”) of the software to be tested. The test program storage step and the test program stored in the test program storage step are designated as a program operated using a debugger, the debugger is started, and the specified test program is used by using the started debugger. A test automatic execution step of setting a breakpoint at a location of a test program for grasping an error by using an API of the software and executing the set test program by using a started debugger.

[0026]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 In the first embodiment, in a situation where the OS is verified by executing the test program, a mechanism is provided in which the test program stops the test program immediately after the failure is detected and keeps the state at the time of the failure detection. In this embodiment, an OS verification device that verifies an OS will be described as an example of a software verification device. Similarly, in Embodiments 2 to 8, an OS verification device will be described as an example. Therefore, the test target program to be tested is
OS.

FIG. 1 shows a configuration diagram of an example of a system for implementing the first embodiment. Reference numeral 100 denotes an OS verification device as an example of a software verification device. In the present invention, the OS verification device (100) supports an OS test performed on a target machine. In the present embodiment, the OS verification device (100) includes a storage unit (test program storage unit) (110), a test automatic execution unit (12).
0), a virtual terminal (130), and an interface (101) with a communication path connected to the target machine.

Reference numeral 110 denotes a storage unit (secondary storage device) as a test program storage unit. Here, a test program group (1) to be executed on the target machine and a file (simply, a file representing path information of the test program representing the position information (path) of the test program on the secondary storage device)
(Also referred to as "test program path information") (4). Details are shown in FIG. 2 and FIG. Reference numeral 120 denotes an automatic test execution unit in the OS verification device, which performs the operation shown in FIG. Reference numeral 130 denotes a virtual terminal,
A group of test programs (23) as a session for operating a test program that runs on the OS under test.
0). The virtual terminal (130) notifies the instruction from the test automatic execution unit (120) to the test program group (230).

Reference numeral 200 denotes a target machine. Then, the OS (220) to be tested operates.
Reference numeral 201 denotes an interface with a communication path connected to the OS verification device. 210 is a secondary storage device on the target machine. Reference numeral 220 denotes an OS (OS program) to be tested. 230 is a group of test programs operating on the OS (220) to be tested. 231
Is a command interpreter that operates on the OS under test, and is a leader that is first activated in the test program group (230). The command interpreter
After the test program ends, it stays on the target machine as long as the connection with the virtual terminal (130) is maintained.

Reference numeral 232 denotes a user level debugger which is activated on the command interpreter (231). 233
Is a test program, a user-level debugger (23
Invoked via 2). A communication path 300 connects the OS verification device (100) and the target machine (200). As the medium, for example, RS232C or Ethernet is used. Using this communication path, a test program is transferred from the OS verification device to the target machine,
It is possible to direct the standard input / output of the program operating on the target machine to the OS verification device.

FIG. 2 shows a state where the test program is managed by a folder for each test item. The folder (1a) for each test item has a hierarchical structure. The folder (1a) hierarchically lower than the folder has an src folder (2) for storing a source file for generating a test program, and the execution of the test program. Sometimes there is a bin (3) folder for storing files. In the src folder (2), source files (2
1) There is a file (22) that describes a procedure for generating a test program from a source file. The bin folder (3) contains a load module (31) of the test program generated from the test program source file (21) and a definition (5) relating to the execution of the test program.

Definition of Test Program Execution (5)
Is ".ru" in the load module name of the test program.
By defining with the name “n”, the OS verification device can grasp the definition (5) relating to the execution of the test program on the load module (31) of the test program.

FIG. 3 shows a file (4) for describing path information of the test program and its format. In the file describing the path information of the test program, the path (41) of the test program is described one line at a time. In the present embodiment, since the test program is executed through the user-level debugger, a source file is required when the test program is executed. As shown in FIG. 2, the test program is managed in a folder having a hierarchical structure. In the path (41) of the test program, the last is the file name of the load module of the test program, the last is the bin folder, and the last is the bin folder. The previous item is the test item name. Example of test item name "/ kernel / syscal
1 / read / ”is shown in Fig. 3. With this, the test automatic execution unit (120) performs each test item necessary for the test program execution according to the operation mode of the user-level debugger on the target machine. The contents of the folder (1a) can be transferred to the target machine.

Alternatively, the contents of the bin folder (3) under this folder may be transferred. In the process of transferring only the contents of the bin folder (3), the OS verification device (100) is a cross development host of the target machine (200), and the user level debugger main body on the OS verification device (100) is an OS verification device. It operates on the verification device (100), and corresponds to a test environment in which a part of the user level debugger and the test program operate on the target machine (200). OS
The verification device processes each test program entry in the file (4) describing the path information of the test program. Thus, the OS verification device knows the test program being executed on the target machine.

FIG. 4 shows the definition (5) relating to the execution of the test program and its format. The definition (5) relating to the execution of the test program exists in the folder (1a) for each test item at a ratio of one to the test program. In principle, the extension (.ru) for the test program name
By n), the test automatic execution unit (120) can grasp the correspondence. The definition (5) relating to the execution of the test program includes the following information required when executing the test program on the OS verification system.

Reference numeral 51 denotes argument information when the test program is executed. In the present embodiment, it is described in the first row. 52
Is information on a breakpoint set when the test program is executed. In the present embodiment, one line is used for one breakpoint after the second line. Information about the breakpoint (52)
It consists of a function name (53) which is a keyword when a list of files constituting the test program is displayed on the debugger, and a line number (54) in the file constituting the test program. In addition, the place where the break point is set in the test program is selected by the test implementer in principle as a place where the test program detects an error.

FIG. 5 shows the flow of processing of the automatic test execution unit in the OS verification device that implements the first embodiment. The processing in each step will be described below.

In step 900, the OS verification device (10
0), a virtual terminal (130) is prepared on the target machine (20) using the communication function of the virtual terminal.
0) requesting the command interpreter (231) to start, and connecting to the generated command interpreter (231). Through the operation on the virtual terminal (130), various commands of the user level debugger (232) and the test program (233) are activated on the target machine (200) to test the OS (220) to be tested.

At step 1000, path information (4) of the test program is read. For example, the automatic test execution unit (120) can handle the file by passing the file name describing the path information (4) of the test program as an argument at the time of startup or setting it to a fixed path. I have.
The test automatic execution unit (120) handles one test program at a time in one loop process from step 1000 to step 1700. Therefore, step 1000
Then, in the path information (4) of the test program, the entries of the test program are read sequentially from the beginning.

In step 1100, step 1000
Based on the path information read in the processing in step 2, the processing is changed depending on whether there is a test to be performed. No description of the test to be executed means that the test program has reached the last line in the path information (4) file. If there is a description of the test to be performed, proceed to step 1200,
If there is no description of the test to be executed, the process of the test automatic execution unit ends.

In step 1200, means (hereinafter, also referred to as "automated means") for automatically providing the virtual terminal (130) prepared in step 900 with an effect equivalent to a key type by an operator is provided. It is realized using. The character string passed in the processing corresponding to the key type performed on the virtual terminal (130) is interpreted by the command interpreter since the virtual terminal (130) and the command interpreter on the target machine (200) are connected. Is done. The purpose here is to launch the test program (233) on the target machine (200) via the user level debugger (232), which is performed as follows.

First, a test set required to execute a test program on the target machine is transferred to the target machine. For example, the command interpreter interprets the execution of the remote copy command, and transfers the entire folder (1) for each test item to the secondary storage device (210) on the target machine. From the path information (4) of the test program, the test program name and the bin
It can be obtained as a path excluding the folder. Specifically, in the OS verification device, "rcp-r OS
Verification device name: Path name of the folder to be transferred. (Line feed) ”on the virtual terminal (130). After the transfer of the test program to the target machine is completed, the path of the user-level debugger to the virtual terminal (130) is followed by the test program. Sends a string consisting of a name, which is interpreted by the command interpreter on the target machine and invokes the test program on the target machine via the user-level debugger, at which point the test program still starts executing I haven't.

Step 1300 is processing for reading a definition (5) file relating to the execution of the test program. In this process, the argument (51) when the test program is executed
Then, information (52) on the breakpoint set during the test program is obtained. In the present embodiment, the file of the definition (5) relating to the execution of the test program is defined as the path of the currently executed test program with the extension “.run” as shown in FIG. Exists in the folder where the test program exists.

In step 1400, step 1200
A breakpoint is set in the test program (233) on the user-level debugger started on the target machine (200) in the processing of step 1200 using the automation means described in (1). Here, the user-level debugger is caused to read the source file constituting the test program using 53 out of the information (52) relating to the breakpoints set in the test program read in step 1300, and then the test program is loaded. A breakpoint is set at the line number (54) in the constituent file. If you need to set multiple breakpoints,
Repeat this operation on the user-level debugger.

In step 1500, step 1200
The test program started by the user-level debugger is started with the argument at the time of execution of the test program obtained in step 1300 using the automation means described in (1).
Specifically, for the virtual terminal (130), a character string interpreted by a user-level debugger on the target machine to start the test program with desired arguments,
In the example of FIG. 4, "runarg1 arg2 arg3
(Return) ”.

Step 1600 is a process of monitoring the end of the test program (233) by an automatic interactive operation (screen capture) on the virtual terminal (130). In this process, the screen of the virtual terminal (130) is captured at regular intervals (for example, one second), and character data is obtained in a character string array corresponding to the size of the virtual terminal.
For example, the virtual terminal (13
0) Perform processing to capture the characters output above. Specifically, in a situation where the test program and the standard output of the user-level debugger are displayed on the virtual terminal (130), and the displayed content is scrolled upward, the examiner may select the lowest line (24 lines) on the screen. The prompt of the user-level debugger displayed in (E) and a character string indicating the information indicating the state of the test program by the user-level debugger that should have been output immediately before and displayed one line above (23rd line) Obtain and monitor as data. The execution status of the test is grasped by comparing the character string data of the two lines with the character strings. Incidentally, since the prompt of the debugger is not displayed until the test program is completed, the character string data corresponding to the 24th line on the screen captured by the screen capture is not a character string representing the prompt of the debugger.

In step 1700, step 1600
When the test program (233) stops at the breakpoint based on the processing in (1), the process ends. If the test program has not stopped at the breakpoint, the process returns to step 1000. As an example showing a state where the virtual terminal (130) is stopped at a breakpoint, in the data obtained by screen-capturing the virtual terminal (130) in the process of step 1600, the character string data corresponding to the 24th line on the screen is displayed by the user-level debugger. This is the same as the prompt, and furthermore, the character string data corresponding to the 23rd line indicates the break location, and the character string data corresponding to the 22nd line indicates that the recording has stopped at the break point.

If no breakpoint is detected during the execution of the test program, the test program is automatically executed sequentially based on the path information (4) of the test program. On the other hand, when a breakpoint is detected, the test program remains stopped due to the breakpoint set in the process of step 1400. In this way, the execution of the test program is stopped at a desired position in the test program.

As described above, the operating system of the computer to be tested is changed to the OS using the application programming interface provided by the OS.
In a software verification device for verifying with a test program for confirming service contents of the above, the test program is started via a user-level debugger realized on an OS, and execution is started after setting a breakpoint at a place where the test program causes an error. By doing so, the test program is stopped after the failure is detected, and the state in which the failure is detected is maintained.

Embodiment 2 In the second embodiment, as in the first embodiment, when the test program stops at a set breakpoint, failure information collection processing for acquiring values (argument information) of variables used by the test program The case of realizing the mechanism will be described. In this embodiment, the test program obtains the internal state of the test target program when a failure is detected. The problem to be determined can be avoided. In this embodiment, as in the first embodiment, an OS verification device will be described as an example.

FIG. 6 is a configuration diagram of an example of a system for implementing the second embodiment. A failure information collection unit 140 in the OS verification device performs the operation shown in FIG. 111
Is failure information. The fault information in the present embodiment is data of a test program stopped on the target machine, which is collected by the fault information collecting unit (140). The failure information is associated with the name of the test program stopped on the target machine, and
It is recorded in the next storage device. In FIG. 6, the failure information (11
1) is represented as a storage device different from the storage unit (110), but may be the same storage device.

FIG. 7 shows a test program symbol information file (6) representing symbol information inside the test program and its format. The symbol information indicates an address to which the head (referred to as a symbol) of data and a function used inside the program when the program is expanded on the memory is allocated. Usually, the assigned address is determined when a program source file is compiled and then linked. The symbol information is included in the load module. In this embodiment, the failure information collection unit (140) acquires symbol information as needed. This will be described later (FIG. 9, step 1722). The file (6) representing the symbol information inside the test program is, for example, a standard U
In the case of NIX (“UNIX” is a registered trademark), it is obtained by specifying the path of the load module of the test program in the argument of the standard nm command in a state where the load module of the test program includes the symbol table. .

Reference numeral 61 denotes a symbol address. 6
2 represents a symbol identifier. The symbol identifier indicates whether the symbol is data or a head of a function. The information is used to determine a symbol that is data. 63 indicates a symbol name. If the symbol type is data, it is the name of variable data. When the test program stops at a breakpoint, the variable data obtained from the symbol information is a target for collecting the value at that time as fault information.

FIG. 8 is a diagram showing an example of a processing flow of the automatic test execution section (120) for realizing the second embodiment. This processing corresponds to step 17 in FIG.
The processing of step 00 is changed to step 1710, and the processing of step 1720 is added. The processing in these changed steps will be described below.

In step 1710, the processing is changed depending on whether the test program (233) has stopped at the break point set in the processing in step 1400. If stopped at a breakpoint, go to step 172
Move to the process of 0. If the processing is terminated without stopping at the break point, the process returns to step 1000. As an example showing a state where the virtual terminal (130) is stopped at a breakpoint, in the data obtained by screen-capturing the virtual terminal (130) in the process of step 1600, the character string data corresponding to the 24th line on the screen is displayed by the user-level debugger. This is the same as the prompt, and furthermore, the character string data corresponding to the 23rd line indicates the break location, and the character string data corresponding to the 22nd line indicates that the recording has stopped at the break point.

In step 1720, the processing of the fault information collection unit (140) is performed. Details of the processing of step 1720 are shown in the flow of FIG.

Next, FIG. 9 will be described. Step 1
At 721, the test program (23) is executed by an automatic interactive operation (screen capture) on the virtual terminal (130).
This is the process of acquiring the stop line number of 3) and leaving the failure information (111). In the case where the program has stopped at a breakpoint set on the user level debugger, for example, in the screen captured data, the user level debugger prompt,
The line indicates the stop line number, and the line 22 indicates the stop at the break point and stop position information (the source file of the test program and the stop line number).

In this step, the failure information (111) is left in a file. A file having a test program name with an extension of “.log” is generated as this file name, and the file name of the data obtained by the above capture processing is generated. Write the information obtained in lines 22 and 23. An example of this content is shown in FIG. 10 as the stop point (12) of the test program.

In step 1722, a symbol table is obtained from the load module of the test program (233) being executed, and the data name in the test program (233) is obtained. In the present embodiment, the storage unit (11
0) In the load module of the test program above,
Using the nm command, information on the symbols used inside the test program in the format shown in FIG. 7 is obtained in a file. Next, in this file, those whose symbol identifiers represent data are extracted, and finally, the variable names inside the test program are obtained.

In step 1723, the automation procedure described in step 1200 and screen capture are performed on the virtual terminal (130), and the test program detects an error and stops the variables in the test program (233). To obtain the failure information (11
Record as 1). Specifically, in the variable name in the test program obtained in the process of step 1722, a character string is transmitted to the virtual terminal to display the data value on the user-level debugger, and then the output of the user-level debugger is displayed on the screen. The captured information is recorded in a file that records the failure information created in step 1721.

For example, what the user-level debugger outputs as desired variable display procedure (line feed), desired variable value (line feed), and prompt (line feed) is scrolled on a virtual terminal of 24 lines × 80 columns. Then, the display procedure of the desired variable on the 22nd line, the value of the desired variable on the 23rd line, 24
A prompt appears on the line. The screen capture function enables these to be obtained as character strings and processed by a program. The failure information obtained in this step is shown in FIG. 10 by a desired variable display procedure (113).
And the value of the desired variable (114).

In step 1724, information useful for analyzing the flow of the execution of the test program, such as the back trace of the stack of the stopped test program (233), by the automation procedure described in step 1200 on the virtual terminal (130). To the user-level debugger. This processing depends on the function of the user level debugger used.

For example, when performing a bag trace, a character string “bt (return)” is transmitted to the virtual terminal to give an instruction to the user-level debugger. Thus, the user-level debugger analyzes the stack frame of the stopped test program. As a result, a function name including the stopping point indicating the history at which the current stopping point was called, and a source file name including the function are obtained.

Further, the result obtained by the user-level debugger is acquired by screen capture performed on the virtual terminal, and the failure information (1
Leave as 11). These are shown as an instruction (115) to the user level debugger and an analysis result (116) by the user level debugger in FIG.

In the present embodiment, the variable values of the test program obtained from the user-level debugger are obtained by the screen capture function. However, if the virtual terminal has a log output function to a file, this variable is used at the start of the processing in step 1721. By turning on the logging function and associating it with the file storing the fault information (111) on the secondary storage device and turning off the logging function at the end of the processing in step 1724, the automatic interactive operation to the user-level debugger can be performed with a key type. It only needs to be done.

As described above, after stopping at the breakpoint set by the test program, the software verification apparatus obtains the variable names used by the test program from the symbol table included in the load module of the test program, and executes the test program. A fault information collecting process for acquiring a variable value of a test program being stopped through a running user level debugger is provided.

Embodiment 3 In the third embodiment, after the test program stops at the set break point,
A failure information collection processing mechanism that acquires the value of a variable used by the OS under test is realized.

FIG. 11 shows a configuration diagram of a system for implementing the third embodiment. In the present embodiment, the target machine (200) is connected to a virtual terminal (150) having a communication function that operates on the OS verification device (100).
The above OS can be debugged.

Reference numeral 150 denotes a virtual terminal that operates on the OS verification device. The virtual terminal is connected to a kernel level debugger on the target machine in order to perform a debugging operation of the OS to be tested on the target machine from the OS verification device (100) by using the communication function.
The input / output to / from the virtual terminal (150) on the OS verification device (100) is performed by the failure information collection unit (140).

Reference numeral 221 denotes a kernel level debugger which is implemented in the OS to be tested. The kernel-level debugger is associated with a specific communication port on the target machine, and by performing input / output using this communication port, the OS under test can be debugged.

Reference numeral 7 denotes a load module of the OS to be tested. Reference numeral 8 denotes a file representing symbol information used inside the OS under test (also simply referred to as “symbol information used inside the OS under test”). The contents are shown in FIG.

FIG. 12 shows a file (8) representing symbol information used inside the OS under test and its format. The symbol information is information indicating an address to which data and a function name (symbol) used inside the OS when the OS is expanded on the memory are assigned.
Usually, the address is determined when a program source file is compiled and linked. The symbol information is included in the load module. The symbol information can be output to a file or the like at the time of link processing.

For example, in UNIX, the OS symbol information can be obtained by setting the path of the OS load module in the argument of the nm command and executing it with the symbol table included in the OS load module. In addition, this can be requested to be created at the time of OS generation. In the present embodiment, a method of obtaining from the load module of the OS will be described.

Reference numeral 81 denotes a symbol address. 8
2 represents a symbol identifier. The information is used to determine a symbol that is data. 83 indicates a symbol name. If the symbol type is data, it is the name of variable data. When the test program stops at a breakpoint, the variable data obtained from the symbol information is a target for collecting the value at that time as fault information.

FIG. 13 shows the flow of the process of the fault information collecting unit in this embodiment. FIG. 3 is an extension of step 1725 and step 1726 in addition to the processing flow of the failure information collection unit shown in FIG. The processing in the extended steps will be described below. Step 17
In step 25, the data name in the OS and the address to which the data name is assigned are obtained from the symbol information (8) used inside the load module of the OS to be tested. FIG.
In the information shown in FIG. 2, a symbol name (83) whose address (81) is included in the data space is obtained by the following method.
In the load module (7) of the OS to be tested on the storage unit (110), information of symbols used inside the OS to be tested in the format shown in FIG.

In step 1726, the test program (233) detects an error and stops the data used by the OS (220) to be tested by the automation procedure described in step 1200 performed on the virtual terminal (150). Display the value in the state in which the error has occurred, obtain the output result by screen capture, and obtain the fault information (111).
Record as

Specifically, the information of the symbol inside the OS to be tested obtained in the processing of step 1725 is
After appending to the file describing the failure information (111) created in 21 (8 in FIG. 14), subsequently, the operation of dumping the kernel data space on the kernel level debugger by the key type to the virtual terminal (150) After that, the screen is captured, and the captured data is added to the file describing the failure information (111) (request (17) to the kernel level debugger in FIG. 14 and output (18) by the kernel level debugger). . With the information obtained here, the contents of the memory at the specific address are actually
It becomes possible to know which variable of S is assigned. If the test program is running on the OS
You can know the internal variable values.

In this embodiment, the data value in the OS to be tested obtained from the kernel level debugger is acquired by the screen capture function. However, if the virtual terminal has a log output function to a file, the processing in step 1725 is started. At this point, the logging function is turned on, the failure information (111) on the secondary storage device is associated with the file to be stored, and the logging function is turned off at the end of the processing in step 1726, so that the automatic transmission to the virtual terminal (150) is performed. Only the key type should be used for the interactive operation.

As described above, when the software verification apparatus stops at the breakpoint set by the test program, the software verification apparatus obtains the variable names used by the OS under test from the symbol table included in the load module of the OS, and executes the test program. And a failure information collecting process for acquiring a value held by a variable inside the OS under a situation tested by a kernel level debugger.

Embodiment 4 In the fourth embodiment, after the test program stops at the set break point,
A failure information collection processing mechanism that acquires the status of resources managed by the OS used by the test program is realized.

FIG. 15 shows a configuration diagram of a system for implementing the fourth embodiment. Reference numeral 241 denotes a failure command interpreter used when collecting failure information operating on the target machine (200). At the start of the test, the command interpreter for the failure uses the OS verification device (10
0) and resides on the target machine during the test. Reference numeral 160 denotes a virtual terminal that operates on the OS verification device, and a failure command interpreter (2) that operates on the target machine by a communication function.
Group of programs started from (41) (240)
Is connected to On the OS verification device (100), input / output to / from the virtual terminal is performed by a failure information collecting unit (140).
Wins. 240 is a group of programs started by the failure command interpreter (241) operating on the OS (220) to be tested on the target machine (200). Various programs (242) are generated on the target machine (200) in response to a request for the failure command interpreter (241) made by the failure information collection unit (140) via the virtual terminal (160).
The standard output of these programs is the virtual terminal (16
0) above. 140 is a failure information collecting unit. The processing of the failure information collection unit (140) is extended in this embodiment. Details are shown in FIG.

FIG. 16 shows an example of a definition file (5) relating to the execution of the test program and its format. Elements extended as compared with the definition file (5) shown in FIG. 4 are as follows. Reference numeral 55 denotes an automatic dialog procedure (contents of a process corresponding to a key type of an operator) performed on a virtual terminal connected to a command interpreter on a target machine. In this embodiment, a character string (such as ps-Al in FIG. 16) and a return code written after the identifier “CMD:” are sent to the virtual terminal.

FIG. 17 shows the processing of the automatic test execution section (120) of this embodiment. FIG. 17 is an extension of the process of step 900 to the process of the automatic test execution section (120) shown in FIG. The following describes the extended steps. In step 910, a virtual terminal (160) is prepared on the OS verification device (100), and a command prompt is generated on the target machine (200) using the communication function of the virtual terminal. In the OS verification device, various commands of various programs (242) are activated on the target machine (200) through the operation on the virtual terminal (160), and the results can be obtained.

FIG. 18 shows the processing of the fault information collecting unit (140) of this embodiment. FIG.
Steps 1727 and 1728 are extended to the processing of the failure information collection unit (140) shown in FIG. Hereinafter, processing in the extended steps will be described.

In step 1727, a failure command interpreter (24) on the target machine is used to obtain information on the environment that affects the execution of the test program, such as information on the secondary storage device (210) on the target machine.
1) Obtain the procedure to be performed above. In the present embodiment, the command interpreter for failure on the target machine (241)
The procedure performed above is described by a character string starting with the identifier "CMD:" in the definition (5) relating to the execution of the test program shown in FIG.

In step 1728, a character string representing an operation for obtaining the status of resources managed by the OS used by the test program obtained by the processing in step 1727 by the automation procedure described in step 1200 performed on the virtual terminal (160) To the command interpreter for failure (2
Send to 41). The command interpreter interprets the character string and performs the specified operation.

The resulting output is obtained by capturing the screen of the virtual terminal (160), converting it into character string data (24 rows × 80 columns) corresponding to the virtual terminal size, and storing it in a file as failure information (111). Record.
The failure information obtained in this step is as shown in FIG.
This is shown as an instruction to the command interpreter (117) and the status of resources managed by the OS (118).

In the present embodiment, the information about the test environment obtained by the operation performed on the failure command interpreter (241) is obtained by the screen capture performed on the virtual terminal (160), but the log output to the virtual terminal is output to a file. If there is a function, the logging function is turned on at the start of the processing of step 1728, and the log file is associated with the file storing the failure information (111) on the secondary storage device. Is turned off, the automatic interactive operation to the virtual terminal (160) only needs to be performed by the key type.

As described above, this software verification device is a resource used by the test program after stopping at the breakpoint set by the test program, and the processing of the OS changes depending on the state, thereby affecting the execution result of the test program. An operation for obtaining information on the execution environment of the test program is performed on a command interpreter operating on the OS to be tested, and a failure information collecting process for obtaining the result is provided.

Embodiment 5 In the fifth embodiment, it is possible to set a time-out period enabling a test program to determine a hang-up and to define a measure to be performed after the test program hangs up. If the program has not ended, it is determined that the test program has hung up, and a mechanism for automating a series of processes such as stopping the test program, collecting fault information, terminating the test program, and coping with the hang-up is realized.

FIG. 20 shows a definition (5) relating to the execution of a test program whose members have been expanded to implement the fifth embodiment. The expanded member will be described below. 56 is a timeout time. If the test program does not end even after the time shown here, it is determined that the test program has hung up. In the present embodiment, the number represented by a number after the identifier “TOUT:” (exemplified as 10 in FIG. 20) is treated as the number of seconds.

Step 57 is a countermeasure after determining that the test program has hung up. In the present embodiment, the character string described after the identifier "HUP:" and the return code are transmitted to the virtual terminal (160) by the automation procedure described in step 1200. here,
The process of releasing the resources managed by the OS used by the test program is registered (in the example of FIG. 20, the temporary file (* .tmp) is deleted by the rm command). As a result, the test program does not end to the end,
It solves the problem caused by not releasing resources managed by the OS.

FIG. 21 shows the flow of processing of the automatic test execution section (120). FIG. 21 is a flowchart showing the processing of the test automatic execution section (120) shown in FIG.
0, the processing of steps 1730 and 1740 has been extended, and the processing of step 1600 has been changed to step 1601.

At step 1550, the current time is obtained as the test program start time, and at step 130
The alarm is set so that a timeout occurs after the timeout period of the test program acquired in process 0.

In step 1601, even if the test program has not been completed, if the alarm set in step 1550 has occurred (timeout has occurred), the process proceeds to step 1710.

In step 1730, the process is changed depending on whether a timeout has occurred. If a timeout has occurred, the process proceeds to step 1740; otherwise, the process proceeds to step 1700.

Step 1740 is a process in the case where a timeout has occurred. Details are shown in FIG. FIG. 22 shows the processing after the test program times out, which is extended to the automatic test execution unit in the fifth embodiment. The details of the extended processing will be described below.

Step 1741 is processing for stopping the test program. For example, if the OS under test is UN
In the case of IX, a SIGINT signal is sent to the test program being executed on the user level debugger to stop the test program. For example, the virtual terminal (13
On 0), type the interrupt code (ctrl-c).

At step 1720, the fault information collection unit (1620) performs processing. The contents of the processing are shown in FIG.

In step 1742, the test program (233) is terminated by the user-level debugger (232) by the automation procedure described in step 1200 on the virtual terminal (130). For example, a character string “quit (return)” is transmitted to the virtual terminal (130).

In step 1743, a method (57) for handling after the test program hangs up is read from the definition (5) relating to the execution of the test program. As shown in FIG. 20, in the present embodiment, a definition (5) relating to the execution of a test program includes an action (57) after the test program hangs up. In the example of FIG. 20, “HU
P: ”, followed by“ rm *. tmp "is read.

At step 1744, step 1743 is executed.
Is transmitted to the virtual terminal (160) by the automation procedure described in step 1200, and the failure information (111) indicates that the test program hangs up and execution is interrupted. Record.

As described above, the software verification device can define the timeout period of the test program and the action to be taken after detecting the hang-up, so that the test program does not end or stop even after the timeout period of the test program has elapsed. In this case, the test program determines that a hang-up has occurred, sends a signal to the test program on the user-level debugger, stops the program, obtains a stop position, collects fault information, and further performs user-level debugger. A test automation method for terminating the test program and executing a measure after the detection of the hang-up is provided.

Embodiment 6 FIG. In the sixth embodiment, in the function of preventing the automatic execution of the test from being stopped due to the hang-up of the test program and automatically collecting data in order to analyze the cause of the hang-up, the time-out determination time is set efficiently. To realize a mechanism.

FIG. 23 shows the flow of processing of the test automatic execution section (120). In order to realize the sixth embodiment, the processing of step 1350 and step 1750 is extended to FIG. The processing in the extended steps will be described below.

In step 1350, the definition file (5) relating to the execution of the test program is reopened in the writable mode. In the sixth embodiment, the timeout time (55) in the definition file (5) (FIG. 20) relating to the execution of the test program extended in the fifth embodiment is changed to a value based on the time actually required for executing the test program. Rewrite to optimize the hang-up determination time for the next test.

At step 1750, the current time is obtained as the test program end time, and at step 155
The elapsed time from the test program start time obtained in the process 0 + α is written as the timeout time (55) in the definition file (5) relating to the execution of the test program. + Α is the time to allow for the hang-up determination, and may be, for example, the elapsed time from the test program start time calculated in step 1750.

As described above, in the software verification apparatus and method for automating software testing, the timeout period of the test program that determines the timeout period of the test program is determined based on the actual value when the test program is executed.

Embodiment 7 FIG. In the seventh embodiment, a mechanism that prevents the automatic execution of the test from being stopped due to the abnormal stop of the test program and realizes the automatic collection of data to analyze the cause of the abnormal stop is realized.

FIG. 24 is a definition (5) relating to the execution of a test program which is an extension of FIG. 20 to realize the seventh embodiment. The expanded part will be described below. The extension is for performing a continuation operation on the user-level debugger when the test program stops upon receiving a signal assumed by the test program. Reference numeral 58 denotes information on a signal assumed to be received by the test program. This format is composed of: signal type: file name of test program: line number receiving expected signal (start): line number receiving expected signal (end).

In the present embodiment, the OS to be tested is UN
In the situation where 32 types of signals are defined in IX, the notation of the signal type is the same as the format displayed by the user-level debugger when the running test program receives a signal and stops.

More specifically, the signal (58) assumed by the test program in FIG.
SR1 is the same as the notation obtained on the user-level debugger. In the definition of the information regarding the signal (58) assumed to be received by the test program shown in FIG. 24, the SIGUSR1 signal is test_s
rc1. It is stated that there is a possibility that the file will be received within the first 100 lines of the c file.

FIG. 25 shows the process of the automatic test execution section (120). In order to realize the seventh embodiment,
The processing of step 1760, step 1761, and step 1765 has been extended to FIG. Hereinafter, processing in the extended steps will be described.

In step 1760, if the test program has stopped abnormally, the flow proceeds to step 1761; otherwise, the flow proceeds to step 1765. The abnormal termination of the test program on the user-level debugger is caused by the test program receiving a signal. This signal is sent from the OS due to illegal processing due to an error in the description of the test program. Sent by On the other hand, the stop that can be assumed by the test program on the user-level debugger is caused by executing a breakpoint or receiving a signal that can be assumed in the program processing.

From this, it can be determined that the test program has stopped abnormally. To determine whether the test program has stopped, the virtual terminal (130) is screen-captured and characters of a terminal size (for example, 24 lines × 80 columns) are captured. After the information is obtained in a column array, by comparing the information obtained as a character string,
The cause of the stop (which signal was received and which line of the file was stopped at the time of execution) was grasped, and it was determined that the cause of the stop was neither a breakpoint nor a signal (58) expected by the test program. I do.

However, there is a possibility that the location stopped by receiving the signal is in a library function not included in the source file of the test program.
Perform a backtrace operation as described in 724,
Judge at the final call location in the test program.

Specifically, after capturing the back trace information on the virtual terminal screen and obtaining a character string array of a terminal size (for example, 24 lines × 80 columns), the test program which is the analysis result of the stack frame is used. The file name that called the stop location and the line number in that file are included in the range of the signal (58) assumed by the test program, and the signal received due to the stop factor and the test program If the signals are the same, it is determined that the operation is not abnormal stop.

In step 1765, step 1760
If the test program has stopped in response to the signal in the normal processing in the determination processing of (1), the process proceeds to step 1766. Otherwise, the process proceeds to step 1750.

At step 1766, the test program is continuously operated on the user level debugger. Specifically, an automatic interactive operation for continuously executing the test program on the user-level debugger is performed on the virtual terminal (130). Thereafter, the process returns to step 1601.

Step 1761 is a process performed when the test program is abnormally stopped. Details are shown in FIG.

Next, a description will be given of a process after the test program abnormally stops, in which the automatic test execution section (120) is expanded in the seventh embodiment shown in FIG. The processing in the steps extended in the present embodiment will be described below. In step 1720, the failure information collection unit (140)
Performs the processing. The contents have been shown in FIG.

In step 1742, the automation procedure described in step 1200 is performed in the virtual terminal (130), and the test program (233) is terminated on the user level debugger (232). For example, a character string “quit (return)” is transmitted to the virtual terminal (130).

In step 1743, as in the fifth embodiment, since the test program has not been executed to the end similarly to the fifth embodiment, the test program is executed in order to release resources managed by the OS used by the test program. From the definition (5) relating to execution, a measure (57) after the test program hangs up is read.

In step 1762, the character string obtained in the processing in step 1743 is transmitted to the virtual terminal (160) by the automation procedure described in step 1200, and after the test program abnormally stops on the failure command interpreter (241). Is performed. In addition, the information that the test program stopped abnormally and the failure information (11
Record in 1).

As described above, by making it possible to define the action to be taken after a test program abnormally stops, in the test verification method, after the test program abnormally stops, the stop position is acquired, and the fault information collecting unit obtains the fault information. The collection is performed, the test program is terminated on the user-level debugger, and a test for executing a measure after the abnormal stop of the test program is automated.

Embodiment 8 FIG. In the eighth embodiment, an automatic test execution mechanism that enables recovery after a failure of a test due to a system failure due to a serious failure of an OS to be tested is realized.

FIG. 27 shows an OS verification device (100) and a target machine (200) in which the function of resetting the target system is extended to FIG. 15 in order to realize the eighth embodiment. Hereinafter, the extended portion will be described.

Reference numeral 250 denotes a hardware reset circuit having a function of resetting the target machine by inputting a signal from the outside (contact input port). 17
0 is output signal control software, which outputs a signal for resetting the target machine using the contact output port. Note that 170 and 250 are electrically connected at a low level.

FIG. 28 is a definition (5) relating to the execution of a test program which is an extension of FIG. 24 for realizing the eighth embodiment. The expanded part will be described below.

Reference numeral 59 denotes a reset delay time. After determining that the test program has hung up, this is the time to wait when the test program is brought to a halt. If the hang-up state continues even after the elapse of this time, hardware reset processing of the target machine is performed. In the present embodiment,
The number represented by the numeral after the identifier “RST:” is treated as the number of seconds.

FIG. 29 shows the processing of the failure information collection unit (140) shown in FIG. 22 in order to realize the eighth embodiment. The processing in the extended step will be described below.

The step 1770 changes the processing depending on whether or not the test program has been stopped by the signal sent in the processing of the step 1741. Step 1 if stopped
The process moves to step 720, and if not stopped, the process moves to extended step 1771.

In step 1771, the processing here in the test program determined to have caused a hang-up is different between the first case and the second case. In the case of the first time, the process moves from step 1772 to step 1775. In the case of the second time, step 1780 including a process of resetting the target machine halfway
Then, the process proceeds to step 900. Step 1 below
The processing when 771 is the first time will be described.

At the step 1772, the kernel level debugger (221) is called. Specifically, the virtual terminal (150) performs a data transmission process that has the same effect as a specific key type performed when the operator calls the kernel-level debugger.

At step 1773, step 1772
The process is changed depending on whether the kernel level debugger (221) can be called in the process (1). This determination is made as follows. The screen of the virtual terminal (150) is captured, and character data is obtained in a character string array corresponding to the size of the virtual terminal. For example, the process is to take in characters output on the virtual terminal (130) into an array of 80 columns × 24 rows. When the output of the kernel-level debugger is displayed on the virtual terminal (130) and the displayed content is scrolled upward, the kernel-level debugger prompt displayed on the lowest line (line 24) on the screen is obtained. It can be determined whether or not the kernel level debugger can be called based on whether or not it has been called.

If the kernel level debugger (221) can be called, the process proceeds to step 1774. If the kernel level debugger cannot be called, the test target O
Judge that S is hanging up and step 1
Move on to 780.

In step 1774, the kernel level debugger (221) is terminated, and the operation of the OS is continued.
Specifically, data transmission processing is performed on the virtual terminal (150), which has the same effect as a specific key type performed when the operator exits the kernel level debugger.

In step 1775, a wait for the reset delay time in the definition (5) relating to the execution of the test program is performed. Then, the process returns to step 1670.
The next time it passes through step 1671, it is determined to be the second time.

The processing in the case where the step 1771 is the second time, that is, the steps 1680 to 1616 after the state that the test program does not stop is assured is described below.
The process of 82 will be described.

In step 1780, the output signal control software (170) is operated to send a reset signal to the hardware reset circuit (250) of the target machine to reset the target machine.

At step 1781, since the test program does not end normally, the fact that the target machine has been reset is recorded in the failure record (111).

At step 1782, the process waits until the target machine starts.

The processing of the fault information collection unit described above
Processing after Test Program Timeout After Test Program Timeout in Embodiment 7 (Step 1)
740), the next test indicated by the test program path (4) is continuously executed even if the target machine itself is hung up and the target machine is restarted. Is possible.

As described above, in the software test automation method, when the test target OS hangs up or crashes, the hardware reset signal is transmitted to the target machine on which the test target OS operates, and the target machine is restarted. A test automation method for continuously executing a test after a test in which a failure has occurred after the test has been performed.

Embodiment 9 FIG. In the first to eighth embodiments, the OS verification device has been described as an example of the software verification device. However, the above software verification device
Needless to say, the case where the application program is verified as software other than the OS is also included. In addition to the above, a software program that operates on a computer and can be started by a test program is also included.

Embodiment 10 FIG. Embodiments 1 to 8 above
Then, the software verification device and the target machine (20
(0) is realized on another computer. However, as shown in FIG. 30 as an example, a function provided in the software verification device, for example, a software program group is provided on the target machine (200). (1) and path information (4) of the test program may be set.

[0147]

As described above, according to the software verification apparatus or method according to the present invention, when a breakpoint set on the test program is passed, the test program remains stopped. Is set at the location where is detected, it is possible to keep the state where the failure has occurred.

According to the present invention, the internal state of the test program can be obtained while the test program remains stopped after the test program detects a failure. As a result, for a fault that is difficult to reproduce, information for identifying the cause of the fault is detected at the time of fault detection, the procedure for causing the fault in the test target program is grasped, and the test target program is determined to be faulty due to an error in the test program. Problem can be avoided.

Further, according to the present invention, it is possible to obtain the internal state of the OS such as a variable value related to the internal resources of the OS used by the test program while the test program remains stopped after the test program detects an error. Can be.
For this reason, it is possible to obtain information for analyzing the cause of a failure that is difficult to reproduce when the failure is detected.

Further, according to the present invention, there is an effect that the state of resources managed by the OS used by the test program can be obtained in a state where the test program remains stopped after the test program detects an error. .

According to the present invention, when the test program hangs up, the test program is stopped, data for analyzing the cause of the hang-up is automatically collected, the test program is terminated, and the test program is used. Free up resources As a result, even if the test program hangs up, data necessary for factor analysis can be obtained, and the automatic execution of the test can be continued.

Further, according to the present invention, it is possible to prevent the automatic execution of the test from being stopped by the hang-up of the test program, and to efficiently collect data for analyzing the cause of the hang-up. There is an effect that the timeout determination time can be set.

Further, according to the present invention, it is possible to prevent the automatic execution of the test from being stopped due to the abnormal stop of the test program, and to automatically collect data for analyzing the cause of the abnormal stop. This has the effect.

Further, according to the present invention, in a situation where the OS to be tested causes a serious failure due to execution of the test program and the test fails due to a system down,
There is an effect that the target machine on which the OS to be tested runs is reset in terms of hardware, and after the target machine is restarted, the test can be automatically executed to continuously execute the test following the test in which the failure occurred.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating an example of a system that implements a first embodiment.

FIG. 2 is a diagram showing an example of a folder for each test item for managing a test program.

FIG. 3 is a diagram illustrating an example of a path of a test program.

FIG. 4 is a diagram illustrating an example of a definition related to execution of a test program.

FIG. 5 is a diagram illustrating an example of a processing flow of an automatic test execution section that implements the first embodiment.

FIG. 6 is a diagram illustrating an example of a system that implements the second embodiment.

FIG. 7 is a diagram showing an example of symbol information used in a test program.

FIG. 8 is a diagram illustrating an example of a processing flow of an automatic test execution section that implements a second embodiment.

FIG. 9 is a diagram illustrating an example of processing of a failure information collection unit according to the second embodiment.

FIG. 10 is a diagram illustrating an example of fault information according to the second embodiment.

FIG. 11 is a diagram illustrating an example of a system that implements the third embodiment.

FIG. 12 is a diagram illustrating an example of a symbol used inside an OS under test.

FIG. 13 is a diagram illustrating an example of processing of a failure information collection unit according to the third embodiment.

FIG. 14 is a diagram illustrating an example of fault information according to the third embodiment.

FIG. 15 is a diagram illustrating an example of a system that implements the fourth embodiment.

FIG. 16 is a diagram illustrating an example of a definition related to execution of a test program extended in the fourth embodiment.

FIG. 17 is a diagram illustrating an example of a process of a test automatic execution unit extended in the fourth embodiment.

FIG. 18 is a diagram illustrating an example of a process of a failure information collection unit that implements the fourth embodiment.

FIG. 19 is a diagram illustrating an example of failure information according to the fourth embodiment.

FIG. 20 is a diagram illustrating an example of a definition related to execution of a test program for implementing the fifth embodiment.

FIG. 21 is a diagram illustrating an example of a process performed by an automatic test execution unit that implements the fifth embodiment.

FIG. 22 is a diagram illustrating an example of a process after a test program timeout in an automatic test execution section.

FIG. 23 is a diagram illustrating an example of a process performed by an automatic test execution unit that implements the sixth embodiment.

FIG. 24 is a diagram showing an example of a definition related to execution of a test program for implementing the seventh embodiment.

FIG. 25 is a diagram illustrating an example of a process performed by an automatic test execution unit that implements the seventh embodiment.

FIG. 26 is a diagram illustrating an example of processing after abnormal stop of a test program for implementing the seventh embodiment.

FIG. 27 is a diagram illustrating an example of an OS verification device that implements Embodiment 8;

FIG. 28 is a diagram illustrating an example of a definition related to execution of a test program for implementing the eighth embodiment.

FIG. 29 is a diagram illustrating an example of processing of a failure information collection unit that realizes the eighth embodiment.

FIG. 30 is a diagram illustrating an example of a system that implements the tenth embodiment.

FIG. 31 is a diagram showing an example of the configuration of a conventional system.

[Explanation of symbols]

1 test program group, 1a folder for each test item,
2 src folder, 3 bin, 4 path information of test program, 5 definition of test program execution,
6 File representing symbol information, 7 O to be tested
S load module, 8 symbol information used inside the OS under test, 10 program test device, 1
0a test script management means, 10b test automation control means, 10c test script execution means, 10c
d Test pass / fail determination means, 10e Program fault identification means, 10f Test script input means, 10g Test result output means, 10h Mode switching / execution means, 1
0i debug command input means, 20 target system, 21 test program source file, 22
File describing the procedure for generating the test program, 30 inspectors, 31 test program load module, 41 test program path, 51 argument information,
52 Breakpoint information, 53 Function name,
54 line number, 56 timeout time, 57 hang-up handling, 59 reset grace time, 61 symbol address, 62 symbol identifier, 63 symbol name, 100 OS verification device, 101 interface,
110 storage unit (test program storage unit), 111 failure information, 120 automatic test execution unit, 130 virtual terminal, 140 failure information collection unit, 150 virtual terminal, 160 virtual terminal, 200 target machine, 201 interface, 210 secondary storage Device, 220 OS to be tested, 221 kernel level debugger, 230 test program group, 231
Command interpreter, 232 user-level debugger, 233 test program, group of 240 programs, 241 command interpreter, 242 various programs, 300 communication paths.

Claims (12)

[Claims] 1. A software verification device capable of starting a debugger for testing software, wherein the software is tested using an application programming interface (hereinafter, referred to as "API") of the software to be tested. A test program storage unit that stores a test program, and a test program stored in the test program storage unit is specified as a program that operates using a debugger, the debugger is started, and the specified debugger is started using the started debugger. The tested test program is A
A software verification device, comprising: a test automatic execution unit that sets a breakpoint at a location of a test program that grasps an error by using a PI and executes the set test program using a started debugger. 2. The test program includes an error detection point for grasping that the operation of the software is abnormal based on an execution result defined by an API of the software. The test program storage unit further stores the test program. A definition regarding execution of a test program including an error detection point is stored, and the test automatic execution unit sets a breakpoint in the test program based on an error detection point of the test program stored in the test program storage unit. The software verification device according to claim 1, wherein: 3. The software verification device according to claim 2, further comprising:
2. The software verification apparatus according to claim 1, further comprising: a failure information collection unit that collects failure information at a time when the test program stops at the set breakpoint. 4. The test program includes variable information as information for specifying a variable used during execution of the test program, and the fault information collection unit uses a debugger to extract the variable information from the test program. 4. The software verification apparatus according to claim 3, wherein the extracted values of the variables of the test program are collected based on the extracted variable information. 5. The software includes test target variable information as information for specifying a variable to be used during execution of the software. The fault information collecting unit uses a debugger to convert the test target variable information from the software. 4. The software verification apparatus according to claim 3, wherein the software verification device extracts the values of the variables of the test program based on the extracted test target variable information. 6. The failure information collecting unit changes an execution environment of the software, causes the software to execute on the changed execution environment, and acquires an execution result of the executed software. Item 3. The software verification device according to Item 3. 7. The test automatic execution unit, if the test program does not end after a predetermined time has elapsed, stops the test program using the debugger, and the failure information collection unit stops the test program that has been stopped. 7. The software verification device according to claim 3, wherein the software verification device acquires a position and collects fault information. 8. The software verification device according to claim 7, wherein the automatic test execution unit changes the predetermined time based on an execution result of a test program. 9. The test automatic execution unit detects that the test program has stopped abnormally, and forcibly terminates the test program detected using a debugger. The fault information collection unit includes a test program that is forcibly terminated. The software verification device according to claim 3, wherein the software verification device acquires a forced termination position and collects failure information. 10. The test automatic execution section sends a signal for instructing reset of a computer on which software runs after the test program has stopped abnormally, and restarts the computer by the sent signal. 2. The software verification device according to claim 1, wherein 11. The software verification device according to claim 1, wherein the software includes one of an operating system and an application program. 12. A software verification method capable of starting a debugger for testing software, wherein the software is tested using an application programming interface (hereinafter, referred to as “API”) of the software to be tested. A test program storing step of storing a test program, and the test program stored in the test program storing step is designated as a program to be operated using a debugger, the debugger is started, and the specified program is designated by using the started debugger. A test automatic setting step of setting a breakpoint at a position of the test program in which an error is grasped by an API of the software, and executing the set test program by using the activated debugger. Characteristic software A verification method.