JPH07121409A - Performance evaluation device - Google Patents

Abstract

PURPOSE:To sufficiently verify the delay of a response owing to the deterioration of performance or a device fault on performance, the hardware fault of the deterioration of an element and the state of frequent occurrence of retry, for example. CONSTITUTION:A test execution part 10 causing a tested device to execute an operation becoming a test object by a test program and measuring the execution time, a standard execution time storage part 12 storing and preserving the execution time of the test program, which is previously measured, as standard execution time and an evaluation part 11 evaluating a measured result based on data which the test execution part 10 measures are provided. When the test execution part 10 measures execution time, the evaluation part 11 compares measured execution time with standard execution time, evaluates the three stages of 'GOOD', 'WARNING' and 'ERROR' and displays an evaluated result.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to performance evaluation of an input / output device (DASD, magnetic tape device, LAN control device, printer, etc.) connected to a computer, such as design verification and manufacturing confirmation before factory shipment. Alternatively, the present invention relates to a performance evaluation device used for various verifications performed at the user's site during maintenance.

[0002]

2. Description of the Related Art Conventionally, a performance evaluation device for evaluating the performance of an input / output device connected to a computer, for example, a DASD, a magnetic tape device, a LAN control device, a printer, etc. has been known.

This performance evaluation device evaluates the performance of the input / output device in, for example, design verification before manufacturing the product from the factory, production confirmation, or various verifications performed during maintenance at the user's site. The performance evaluation processing by the performance evaluation device will be described below.

Generally, an input / output device connected to a computer is evaluated by using a test program loaded in the performance evaluation device. This test program obtains the execution result by executing the instruction set and compares the expected value data and the expected state prepared by the test program to verify that the function is working normally and evaluate it. Do (evaluation of consequentialism).

In this case, it takes a certain time from the execution of the instruction set to the end of the operation of the input / output device to be evaluated and the response of the result information. Therefore, in the test program, for example, the time is monitored by interrupt timeout, and if there is no response within the fixed time defined in the test program, an error occurs.

It has also been practiced to forcibly set a pseudo fault in an input / output device by executing a diagnostic command to verify the operation of the input / output device.

[0007]

SUMMARY OF THE INVENTION The above-mentioned conventional devices have the following problems. : In the evaluation processing by the time monitoring, a large value is often set for the time width of the monitoring time (because of the result-based evaluation). For this reason, there is a problem that even if a response delay occurs due to performance degradation of the device (for example, element deterioration), it is not judged as an error.

[0008]: Time monitoring in the conventional performance evaluation processing is
This is due to interrupt timeout. Therefore, even if a hardware failure such as a retry frequent occurrence state or an element deterioration state occurs, if an interrupt occurs as a result of the retry, it is not determined as an error.

In the evaluation process based on the setting of the pseudo fault, it is sufficient that the response is returned within the defined constant time, so that the fault of the input / output device related to the performance cannot be detected.

The present invention solves such a conventional problem, and can sufficiently verify a response delay due to performance deterioration, a hardware failure such as element deterioration, and a performance-related apparatus failure such as a retry frequent state. The purpose is to do.

[0011]

FIG. 1 is a diagram for explaining the principle of the present invention, FIG. 1A is a device configuration diagram, and FIG. 1B is a diagram showing an allowable execution time range. In FIG. 1, 1 is a performance evaluation device, and 8
Is an input / output device (device under test), 10 is a test execution unit,
11 is an evaluation unit, 12 is a standard execution time storage unit, 13 is an execution time storage unit, 14 is an input unit, and 15 is an output unit.

The present invention has the following structure to solve the above problems. A test is performed by using the input / output device 8 of the computer as a device under test, and the performance evaluation device for evaluating the performance is caused to execute the operation (input / output operation) to be tested by the test program in the device under test. , The test execution unit 1 that measures the "execution time"
0, the test execution unit 10 stores the “execution time” of the test program measured in advance as the “standard execution time”, and stores the standard execution time storage unit 12 and the data measured by the test execution unit 10. On the basis of this, an evaluation unit 11 for evaluating the measurement result is provided, and the test execution unit 10 causes the device under test to be evaluated to execute the operation to be tested by the test program, and the “execution time” thereof. A performance evaluation device that evaluates the measured “execution time” by comparing the measured “execution time” with the stored “standard execution time” when the measurement is performed.

In the configuration, in the evaluation unit 11, when the "execution time" is within the allowable safe value from the parameters input in advance (GOOD),
If it is within the allowable range but exceeds the safe value (WAR
NING), and when the allowable range is exceeded (ERRO
R) The three levels of "execution time allowable range" are set, and when the execution time is evaluated, the "execution time allowable range" of the three steps is set.
An evaluation device that evaluates according to and outputs each evaluation result.

In the configuration, the above-mentioned "execution time allowable range" of the three stages is an evaluation device which is set by a parameter instructed by the value of% on the plus side and the value of% on the minus side, respectively.

[0015]

The operation of the present invention based on the above construction will be described with reference to FIG. (1): In the present invention, a performance evaluation test of the input / output device 8 (device under test) connected to the computer by the performance evaluation device 1 is performed. In this case, the test execution unit 10 measures the “execution time” of the test program in advance, and the measurement result data is stored as “standard execution time” in the standard execution time storage unit 12 (magnetic tape or the like) and saved. I'll do it.

(2): For the setting of the "permissible range of execution time" by the evaluation unit 11, the parameters input in advance from the input unit 14 by the operator or the like are used to set what percentage is on the plus side and what percentage is on the minus side. .

[0017] Then, there are three levels: if the value is within the allowable safety value (GOOD), if it is within the allowable range but exceeds the safe value (WARNING), and if it exceeds the allowable range (ERROR). "Execution time tolerance"
Set and evaluate the execution time.

(3): When the performance evaluation is performed on the input / output device 8 to be evaluated, the test execution unit 10 causes the test program to operate the input / output device 8 which is the device under test by the test program. (I / O operation) is executed.

At this time, the test execution unit 10 measures the "execution time" of the test program, and the execution time storage unit 13
To store. After that, the evaluation unit 11 determines that the execution time storage unit 1
By taking out the "execution time" stored in No. 3 and comparing it with the "standard execution time" held in the standard execution time storage unit 12, performance degradation occurs according to the setting of the "execution time allowable range". Evaluate whether or not.

Then, in the evaluation section 11, the "GOO
Evaluation is performed by classifying into three stages of “D”, “WARNING”, and “ERROR”, and the evaluation result is sent to the output unit 15 and output (for example, displayed).

By performing the evaluation based on the "execution time" as described above, the response delay due to the performance deterioration, the hardware failure such as element deterioration, and the apparatus failure related to the performance such as the retry frequent state are sufficiently performed. It becomes possible to verify.

[0022]

Embodiments of the present invention will be described below with reference to the drawings. 2 to 7 are views showing an embodiment of the present invention. In FIGS. 2 to 7, the same parts as those in FIG. 1 are designated by the same reference numerals.

Further, 2 is a CPU (central processing unit), 3 is a channel (CH), 4 is an operator console device, 5
Is a magnetic tape device (MT device), 6 is a memory, 7 is a timer, and 8 is an input / output device.

§1: Description of the configuration of the performance evaluation device--see FIG. 2 FIG. 2 is a device configuration diagram of the embodiment. The configuration of the performance evaluation device will be described below with reference to FIG.

As shown in the figure, the performance evaluation device 1 includes a CPU
2, a channel (CH) 3, an operator console device 4, a magnetic tape device 5, etc. The CPU 2 is provided with a memory 6 and a timer 7. Functions and the like of the above-mentioned respective parts are as follows.

The memory 6 is a memory for loading a test program or setting the time table 1 and the time table 2, and is accessed by the CPU 2.

The timer 7 is for measuring time at the time of testing (time mechanism). The operator console device 4 is equipped with a display device, a keyboard and the like, and is used for inputting data or displaying output data through conversation with the operator.

The magnetic tape device 5 is a device for driving a magnetic tape (medium). The magnetic tape stores a test program, standard execution time data, and the like. When performing the performance evaluation of the device under test, the input / output device 8 as the device under test is connected to the performance evaluation device 1.
Then, the performance evaluation test of the input / output device is performed by executing the test program loaded in the memory 6.

The input / output device is a DASD, a magnetic tape device, a printer, a LAN controller or the like. §2: Outline of performance evaluation test and description of allowable execution time range ... See FIG. 3. FIG. 3 is an explanatory diagram of the allowable execution time range. The outline of the performance evaluation test and the allowable execution time range will be described below with reference to FIG.

(1): A performance evaluation test of the input / output device 8 connected to the computer by the performance evaluation device 1 is performed. First, the input / output device 8 (standard device) is used and tested. Measure the "execution time" of a program.
Then, the "standard execution time" is set by holding the measurement result data on, for example, a magnetic tape.

Thereafter, a test is performed on the input / output device 8 (device under test) to be evaluated, and the "execution time" measured at that time is compared with the "standard execution time" to evaluate the performance. To do.

(2): In this case, the "execution time allowable range" described below is set and evaluated. The "execution time allowable range" is set to what percentage (+ a%) on the plus side and what percentage (-b%) to the minus side.

Then, according to the following three criteria,
The allowable range of the "GOOD" condition and the allowable range of the "WARNING" condition are set. This setting is made on the operator console device 4 by the conversation between the program and the operator.
Enter the required parameters.

The judgment criteria are as follows. That is: If the value is within the safe value within the allowable range, "G
The allowable range for the “ODD” condition. : If it is within the allowable range but exceeds the safe value, it is within the allowable range of the "WARNING" condition. : If the tolerance is exceeded,
"ERROR".

(3): As described above, when the performance of the device under test is evaluated, the input / output device 8 to be evaluated is
Measure the "execution time" of the test program. And
It compares with the held "standard execution time" and determines whether or not performance degradation has occurred according to the "execution time allowable range" setting.

As described above, the "permissible range of execution time" is
The plus side and the minus side are set and it is determined whether or not there is an execution time in this range. Judgment is based on the "GOO
It is shown by the result information of "D", "WARNING", and "ERROR".

In this case, the execution time of a series of input / output operations is measured and the performance is determined by setting a plurality of measurement points. The device to be evaluated is a "normal operation system."
And execute the test program separately for "abnormal operation system".

In the case of a normal system, the operation of the device is verified without any modification. Further, in the case of an abnormal system, by executing a diagnostic command, a pseudo fault is forcibly set in the device and the operation of the device is verified.

(4): For example, when the measurement value of the "execution time" is 500 milliseconds, the "execution time allowable range" when the necessary parameters are set as follows is as shown in FIG. Is.

In this example, the allowable range of the "GOOD" condition is plus 2% and minus 1%.
The allowable range of the "RNING" condition is up to plus 5% and minus 2% beyond the "GOOD" condition. Then, the area other than the above range is set to "ERROR".

§3: Description of memory and magnetic tape ...
See FIG. 4. FIG. 4 is an explanatory diagram of the memory and the magnetic tape, FIG. 4A shows the relationship between the time table and the pointer, and FIG. 4B shows the storage format of the magnetic tape.

: Explanation of relationship between time table and pointer
.. See FIG. 4A. The "standard execution time" is stored in the "time table 1" set in the memory 6, and the "time table 2" is stored in the "time table 2".
The "execution time" when the test program is executed for the input / output device to be evaluated is stored.

In this case, a test is carried out for a plurality of test items, and data is stored for each test item. For example, as shown in the figure, "execution time of test item 1", "execution time of test item 2", "execution time of test item 3"
-Store data in the order of testing, such as "Execution time of test item n".

When reading the data from each time table, the pointers P1 and P2 are designated at the head to make them initial values, and then the pointers P1 and P2 are stepped forward to read the data sequentially.

The pointer P1 is the time table 1
Pointer (address pointer) of the pointer P2
Is a pointer (address pointer) of the time table 2. These pointers sequentially move under the control of the CPU 2 to sequentially read data as described above.

Description of storage format of magnetic tape--FIG. 4
See B. On the magnetic tape (medium) of the magnetic tape device 5, the "IPL program" is stored at the beginning, and subsequently the "test program" is stored. Finally, "standard execution time" is stored.

§4: Description of Standard Execution Time Measurement and Saving Processing--See FIG. 5 FIG. 5 is a flowchart of standard execution time measurement and saving processing. The standard execution time measurement and storage processing will be described below with reference to FIG. Note that S1 to S12 indicate processing steps.

S1: First, under the control of the CPU 2 (execution of the IPL program), the "test program" stored in the magnetic tape (medium) in the magnetic tape device 5 is read and loaded into a predetermined area of the memory 6.

S2: Next, the operator console device 4
The parameters necessary for the test execution are input through a conversation with the operator via the. S3: After the process of S2 is completed, the CPU 2 resets the timer 7 before executing the operation (input / output operation) to be tested. After resetting, the timer 7 starts time-up from the timer value 0.

S4: After the processing of S3 is completed, the CPU 2 executes the test target operation (input / output operation) by the test program with the input / output device 8 (standard device) as the target device.

In this case, the input / output device 8 reports the processing end status to the CPU 2 by an end interrupt when the above operation is completed. S5: After starting operation execution in the process of S4, the CPU
2 monitors the passage of time by the timer 7 and determines whether or not there is an end interrupt from the input / output device 8.

S10: When it is determined that the end interrupt does not occur in the process of S5, the CPU 2 monitors the time by the timer 7 and determines whether or not the time is out. As a result, if there is no timeout, the process of S5 is performed.

S11: In the process of S10, when it is determined that the end interrupt does not occur and the predetermined time has passed and the time-out occurs, the CPU 2 terminates the process. S6: In the process of S5, when it is determined that the end interrupt is generated from the input / output device 8 and the end status is received, the CPU 2 reads the timer value of the timer 7 and measures the “execution time”.

S7: The CPU 2 writes the "execution time" measured in S6 in the time table 1 of the memory 6 as the "standard execution time". S8: After the processing of S7 is completed, the CPU 2 determines whether or not the execution of all test items has been completed.

S12: In the process of S8, when it is determined that all the test items have not been completed, the CPU 2 updates the test item and repeats the process from the process of S3. S9: In the processing of S8, when it is determined that the test is completed for all the test items, the CPU 2 causes the memory 6
The "standard execution time" stored in the magnetic tape device 5 is stored and saved in the magnetic tape of the magnetic tape device 5, and the process ends.

§5: Description of execution time evaluation process ... See FIGS. 6 and 7. FIG. 6 is an execution time evaluation process flowchart 1 and FIG. 7 is an execution time evaluation process flowchart 2. Hereinafter, the evaluation processing of the execution time for the evaluation target device will be described with reference to FIGS. 6 and 7. Note that S31 to S53 represent processing steps.

S31: First, under the control of the CPU 2, the "test program" stored in the magnetic tape (medium) in the magnetic tape device 5 is read and loaded into a predetermined area of the memory 6.

S32: Next, the CPU 2 reads the "standard execution time" stored in the magnetic tape and writes it in the "time table 1". S33: After the processing of S32 is completed, the parameters necessary for executing the test and the parameters for setting the “execution time allowable range” are input through a conversation with the operator via the operator console device 4.

S34: Next, prior to execution of the operation (input / output operation) to be tested, the CPU 2
Reset the timer 7. After resetting, the timer 7 starts time-up from the timer value 0.

S35: After completion of the processing of S34, the CPU 2
Targets the input / output device 8 which is the evaluation target device and executes the test target operation (input / output operation) (test execution) by the test program.

In this case, the input / output device 8 reports the processing end status to the CPU 2 by an end interrupt when the above operation is completed. S36: After starting the test execution in S35, the CPU 2 monitors the elapse of time by the timer 7 and determines whether or not there is an end interrupt from the input / output device 8.

S40: When it is determined in the process of S36 that the end interrupt does not occur, the CPU 2 determines whether or not the time is monitored by the timer 7 and the time is out. As a result, if the time is not out, the process of S36 is performed.

S41: When it is determined in the processing of S40 that the time-out has occurred, the CPU 2 writes the occurrence of the time-out in the "time table 2" of the memory 6. S37: The end interrupt is generated in the process of S36, and CP
When U2 determines that the end status is received from the input / output device 8, the CPU 2 reads the timer value of the timer 7 and measures the execution time.

S38: The CPU 2 writes the "execution time" measured in the process of S37 in the "time table 2". S39: CPU after the processing of S38 and S41 is completed
2 determines whether or not all the test items have been completed.

S42: If it is determined in the processing of S39 that all the test items have not been completed, the CPU
In step 2, the test item is updated, and the process is repeated from step S34.

S43: In the process of S39, when it is determined that all the test items have been completed, the CPU 2 indicates the "time table 1" and the pointer P of the "time table 2".
Return 1 and P2 to the beginning and set the initial value.

S44: The CPU 2 reads the "standard execution time" from the "time table 1" of the memory 6. S45: The CPU 2 reads the “execution time” from the “time table 2” of the memory 6.

S46: The CPU 2 calculates the "permissible range of execution time" from the input parameters, "GOOD",
Set the range of "WARNING" and "ERROR". The CPU 2 then executes the "standard execution time" of the "time table 1" and the "execution time" of the "time table 2".
"GOOD", "WARNING", "ER"
ROR ”.

S47: As a result of the processing in S46, first, CP
U2 determines whether the condition is "GOOD". S48: When the condition of "GOOD" is determined in the process of S47, the CPU 2 sends the information to the operator console device 4, and displays "GOOD" on the display screen.

S50: When it is determined in the process of S47 that the condition is not "GOOD", the CPU 2 outputs "WARNING".
Determine if it is a condition. S51: When it is determined in the process of S50 that the condition is "WARNING", the CPU 2 sends the information to the operator console device 4, and displays "WARNING" on the display screen.

S52: In the process of S50, "WARNIN
When it is determined that the condition is not “G”, it is determined that the condition is “ERROR”, and the CPU 2 sends the information to the operator console device 4 and displays “ERROR” on the display screen.

S49: When the above display is completed, the CPU
2 determines whether or not comparison of all time tables has been completed. As a result, the test ends when the comparison of all time tables is completed.

S53: However, in the processing of S49, when it is determined that the comparison of all time tables is not completed,
The CPU2 updates the pointers P1 and P2, and the S4
Repeat from the process of 4.

(Other Embodiments) The embodiments have been described above, but the present invention can also be implemented as follows. : The evaluation result data may not only be displayed but also printed out by a printer and output.

"GOOD", "WARNING",
The display of "ERROR" can be performed by replacing it with other similar display information. : The program stored in the magnetic tape or the data such as the execution time can be stored in another medium such as a magnetic disk.

[0076]

As described above, the present invention has the following effects. : In the performance evaluation processing of the present invention, performance evaluation is performed based on the measured execution time of the test program. Therefore,
In the factory shipment test of a product, by comparing and verifying the difference with the standard hardware, it is possible to check out the variation of the product and the element with high accuracy.

In the field maintenance, it is possible to verify the logically correct performance of the hardware and the system that barely operates due to frequent retries, and to knock out and evaluate these states. It is possible.

The standard execution time, which is a comparison value at the time of evaluation, is a value measured as a result of actually executing the operation of the test target on the device under test, and therefore the performance evaluation with extremely high accuracy is possible.

"GOOD", "WARNING",
Since the evaluation is performed in three stages of "ERROR", it is possible to accurately verify various states such as a state immediately before a failure and a state where the performance of the element is deteriorated.

[Brief description of drawings]

FIG. 1 is a diagram illustrating the principle of the present invention.

FIG. 2 is a device configuration diagram of an embodiment.

FIG. 3 is an explanatory diagram of an allowable execution time range according to the embodiment.

FIG. 4 is an explanatory diagram of a memory and a magnetic tape according to an embodiment.

FIG. 5 is a flowchart of standard execution time measurement and storage processing according to an embodiment.

FIG. 6 is a flowchart 1 for evaluating the execution time of the embodiment.
Is.

FIG. 7 is a flowchart 2 for evaluating the execution time of the embodiment.
Is.

[Explanation of symbols]

 1 Performance Evaluation Device 8 Input / Output Device (Device Under Test) 10 Test Execution Unit 11 Evaluation Unit 12 Standard Execution Time Storage Unit 13 Execution Time Storage Unit 14 Input Unit 15 Output Unit

Claims (3)

[Claims] 1. A performance evaluation device for performing a test by using an input / output device (8) of a computer as a device under test and evaluating the performance of the device under test by an operation (input / output) by a test program. Motion)
The test execution unit (10) that measures the “execution time”, and the standard execution that stores and saves the “execution time” of the test program measured in advance by the test execution unit (10) as the “standard execution time”. The time storage unit (12) and the evaluation unit (11) that evaluates the measurement result based on the data measured by the test execution unit (10) are provided, and the test execution unit (10) is the device under test to be evaluated. On the other hand, when the operation to be tested is executed by the test program and the "execution time" is measured, the evaluation unit (11) stores the measured "execution time" in the stored "standard". A performance evaluation device characterized by performing evaluation by comparing with "execution time". 2. The performance evaluation device according to claim 1, wherein the evaluation unit (11) is within an allowable safe value from the parameters input in advance for the “execution time” (GOOD). , When the execution time is evaluated by setting the "execution time allowance range" of three levels, if it is within the allowable range but exceeds the safe value (WARNING) and if it exceeds the allowable range (ERROR). A performance evaluation device characterized in that evaluation is performed according to the “execution time allowable range” of the three stages and each evaluation result is output. 3. The performance evaluation apparatus according to claim 2, wherein the "execution time allowable range" of each of the three stages is set by a parameter instructed by a plus side% value and a minus side% value. A performance evaluation device characterized by: