JP2013175065A - Performance evaluation method, information processing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To evaluate accuracy of a prediction of calculation time required by a computer system.SOLUTION: A storage unit 11 stores measurement information 11a indicating multiple calculation times measured by causing a computer system 20 to execute a sample program, with resource quantity changed. On the basis of the measurement information 11a, an arithmetic unit 12 determines a value of a parameter used for an evaluation model to calculate the relation between the resource quantity and the calculation time. The arithmetic unit 12 calculates variation of the value of the parameter when the difference between the calculation times indicated by the measurement information 11a and calculation time calculated according to the evaluation model is allowed to be widened to a threshold value. The arithmetic unit 12 outputs information indicating influence of the variation of the value of the parameter on the relation between the resource quantity and the calculation time.

Description

  The present invention relates to a performance evaluation method, an information processing apparatus, and a program.

  Currently, when processing a large amount of data or performing complicated calculations, a computer system having a large resource with a large number of processors may be used. A user who uses such a large-scale computer system predicts the relationship between the amount of resources (for example, the number of processors) allocated to a program to be input and the calculation time until the execution of the program is completed. Reserve an amount of resources. If there is an error in prediction, there will be a problem that the calculation will not be completed by the intended date or time, or the calculation will be completed early and the reserved resources will be wasted. For this reason, how to predict the calculation time is important.

  In this regard, a model formula has been proposed which shows that the calculation time τ depends on the number of processors p, the problem size x, and the parameter c corresponding to the computer system to be used. According to this model formula, the parallelization overhead increases as the number of processors p increases. When the model formula is used, for example, the calculation time τ of the sample problem is measured while changing the number of processors p, and the parameter c is determined by fitting a set of the number of processors p and the measured calculation time τ into the model formula. . When the parameter c is determined and a model corresponding to the computer system is generated, the relationship between the number of processors p and the calculation time τ for the program to be input can be predicted.

Shigeo Orii, “Performance Evaluation of Parallel Processing Using Temporal Model -Parallel Overhead Hidden in Parallel Unit-”, Research Report of Information Processing Society of Japan High Performance Computing (HPC), 2011-HPC-130 July 2011

  However, it is rare to generate a model in which the measured value and the predicted value completely match (for example, all measured values are on the curve indicated by the model in the graph). There is often a residual between the predicted values. For this reason, in the process of determining the parameter value by fitting the measurement value to the model equation, the prediction accuracy of the generated model may be lowered depending on how the measurement value varies. On the other hand, when a model generated by solving a small sample problem is applied to a large-scale problem, a slight decrease in accuracy appears as a large calculation time error. For example, a 1% decrease in accuracy may result in a calculation time error of several tens of hours. For this reason, in predicting the calculation time from the model, it is preferable that the prediction accuracy (for example, how much error can occur) can be evaluated.

  In one aspect, a computer-implemented performance evaluation method is provided that evaluates the relationship between the amount of resources used by a computer system and the computation time. In the performance evaluation method, the relationship between the resource amount and the calculation time is calculated based on a plurality of first calculation times measured by changing the amount of resources to be used and causing the computer system to execute the sample program a plurality of times. The value of the parameter used for the evaluation model is determined. When the difference between each of the plurality of measured first calculation times and the second calculation time corresponding to the first calculation time calculated according to the evaluation model is allowed to extend to the threshold value, Calculate the amount of value fluctuation. Information indicating the influence that the relationship between the resource amount indicated by applying the determined parameter value to the evaluation model and the calculation time varies by the calculated variation amount of the determined parameter value is output.

  In one aspect, the storage unit stores measurement information indicating a plurality of first calculation times measured by causing the computer system to execute the sample program a plurality of times while changing the amount of resources to be used, and the measurement information. An information processing apparatus is provided that includes a calculation unit that uses and evaluates the relationship between the resource amount and the calculation time. The calculation unit determines a parameter value used in an evaluation model for calculating the relationship between the resource amount and the calculation time based on the measurement information. The calculation unit allows a difference between each of the plurality of first calculation times indicated by the measurement information and the second calculation time corresponding to the first calculation time calculated according to the evaluation model to extend to a threshold value. In this case, the fluctuation amount of the parameter value is calculated. The calculation unit displays information indicating an influence that the relationship between the resource amount indicated by applying the determined parameter value to the evaluation model and the calculation time is changed by the calculated variation amount of the determined parameter value. Output.

  In one aspect, a program for evaluating the relationship between the amount of resources used by the computer system and the calculation time is provided. The computer executing the program determines the relationship between the resource amount and the calculation time based on a plurality of first calculation times measured by changing the amount of resources to be used and causing the computer system to execute the sample program multiple times. The value of the parameter used for the evaluation model for calculation is determined. When the difference between each of the plurality of measured first calculation times and the second calculation time corresponding to the first calculation time calculated according to the evaluation model is allowed to extend to the threshold value, Calculate the amount of value fluctuation. Information indicating the influence that the relationship between the resource amount indicated by applying the determined parameter value to the evaluation model and the calculation time varies by the calculated variation amount of the determined parameter value is output.

  In one aspect, it is possible to evaluate the accuracy of calculation time required by the computer system.

It is a figure which shows the information processing apparatus of 1st Embodiment. It is a figure which shows the information processing system of 2nd Embodiment. It is a block diagram which shows the hardware example of a simulation apparatus. It is a block diagram which shows the function example of a simulation apparatus. It is a flowchart which shows the example of a procedure of calculation time prediction. It is a graph which shows the example of the value which a parameter can take. It is a 1st graph which shows the example of the prediction result of calculation time. It is a 2nd graph which shows the example of the prediction result of calculation time.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating the information processing apparatus according to the first embodiment. The information processing apparatus 10 evaluates the relationship between the amount of resources used by the computer system 20 and the calculation time. The resource amount is, for example, the number of processors, and the computer system 20 can execute a program using a plurality of processors in parallel. The calculation time is, for example, the time from the start of execution of a program input by any one of a plurality of processors used until the end of execution of the program input by all of the plurality of processors. The information processing apparatus 10 may be a client computer as a terminal device operated by a user, or a server computer accessible from the terminal device. The information processing apparatus 10 can instruct the computer system 20 to execute a program by specifying a resource amount via a network.

  The information processing apparatus 10 includes a storage unit 11 and a calculation unit 12. The storage unit 11 may be a volatile memory such as a RAM (Random Access Memory) or a non-volatile storage device such as an HDD (Hard Disk Drive) or a flash memory. The arithmetic unit 12 may be a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor), or an electronic circuit other than a processor such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The processor executes, for example, a program stored in the storage unit 11 or another memory. The processor may include other electronic circuits in addition to an arithmetic unit and a register for executing program instructions.

  The storage unit 11 stores measurement information 11a in which a resource amount (for example, the number of processors) is associated with a measured calculation time. The plurality of calculation times indicated by the measurement information 11a are measured by causing the computer system 20 to execute the sample program a plurality of times while changing the resource amount. The execution instruction of the sample program, measurement of the calculation time, generation of the measurement information 11a, and the like may be performed by the information processing apparatus 10 or another information processing apparatus. The measurement information 11a may be generated every time performance evaluation is performed, or information generated in advance may be used.

  The problem to be solved by the sample program is preferably sufficiently smaller than the problem to be solved by the production program executed by the computer system 20 after performing the performance evaluation described later. However, the problem solved by the sample program and the problem solved by the production program are preferably the same kind of problems (for example, those having similar calculation algorithms). The sample program and the production program may be the same, and the scale of the problem may be changed by changing the amount of data to be processed. This makes it easier to apply the model of the computer system 20 generated through a small problem to a large problem.

  The calculation unit 12 uses the model formula for calculating the relationship between the resource amount and the calculation time and the measurement information 11 a stored in the storage unit 11 to determine the relationship between the resource amount and the calculation time for the computer system 20. evaluate. The model formula is prepared in advance, and includes a parameter C that depends on the computer system that executes the program (may include two or more parameters). The parameter C depends on, for example, the architecture related to the processor and the connection between the processors, and may depend on the type of problem to be solved. The computing unit 12 determines the value of the parameter C included in the model formula using the measurement information 11a. For example, the calculation unit 12 uses the least square method to reduce the residual sum (difference between the measured calculation time and the calculation time predicted from the model) for each resource amount so that the sum of squares of the parameter C is minimized. Determine the value.

  In addition, the calculation unit 12 calculates the value of the variation amount ΔC of the parameter C in consideration of the error, using the model formula, the determined value of the parameter C, and the measurement information 11a. The variation amount ΔC is such that the value of the parameter C is determined as described above when the difference between the measured calculation time and the calculation time predicted from the model is allowed to spread to a threshold value for each resource amount indicated by the measurement information 11a. It shows how much it can be changed from the measured value. The value of the fluctuation amount ΔC can be calculated by mathematical expression processing.

  For example, when the calculation unit 12 measures n (n is an integer of 2 or more) calculation times, the difference between the measured calculation time and the calculation time on the model considering the error is equal to or less than the threshold value. Generate n inequalities that indicate The calculation time on the model considering the error with respect to each resource amount is obtained by substituting the parameter C of the model equation with C + ΔC and substituting the value of the resource amount and the determined parameter C into the replaced equation. Thereby, n inequalities including the variation ΔC as a free variable are generated. The threshold value is, for example, the maximum value of the residual calculated when determining the value of the parameter C, or the minimum value under the condition that n inequalities have solutions (usually larger than the minimum value of the residual). . However, the threshold value may be an arbitrary value between the two.

  For the generated n inequalities, the calculation unit 12 calculates a possible value of the variation ΔC, which is a free variable, by applying a quantifier elimination (QE) method, for example. When the measured calculation time varies greatly and the residual is large on average, or when there is a singular point in the measured calculation time and only a part of the residual is large, the value of the variation ΔC can be large. . Quantum erasure is described in, for example, the following book: Hirokazu Anai, Kazuhiro Yokoyama, “QE calculation algorithm and its application—optimization by mathematical processing”, University of Tokyo Press, August 25, 2011. A specific example of limited quantum erasure will also be described in the second embodiment.

  Then, the calculation unit 12 determines that the relationship between the resource amount calculated by substituting the value of the parameter C determined above into the model formula and the calculation time is such that the value of the parameter C varies by the amount of variation ΔC. Outputs information indicating how it is affected. For example, the calculation unit 12 displays information indicating the influence of the variation amount ΔC on a display connected to the information processing apparatus 10 or another information processing apparatus. As information indicating the influence of the fluctuation amount ΔC, for example, a curve indicating a model in which the determined value is substituted for the parameter C of the model formula and a curve indicating a model in which the value corresponding to C + ΔC is substituted for the parameter C are displayed. Information to do is conceivable. Further, as information indicating the influence of the fluctuation amount ΔC, for example, information indicating how much the calculation time corresponding to a certain resource amount changes by a maximum of% when the fluctuation amount ΔC is taken into consideration is also conceivable.

  According to the information processing apparatus 10 of the first embodiment, it is quantified how accurately the model generated by determining the value of the parameter C from the measurement information 11a can predict the calculation time of the computer system 20. Can be evaluated. For this reason, even when the computer system 20 is caused to execute a program that solves a problem larger than the sample problem by using a large resource, the prediction result of the calculation time by the model can be effectively used. For example, when the prediction accuracy is acceptable, the user can reserve resources of the computer system 20 in consideration of the prediction error. On the other hand, when the prediction accuracy is unacceptable, in order to generate a model with higher accuracy, an abnormal measurement value is removed from the measurement information 11a by using the information output from the calculation unit 12 as a clue. And measures such as redoing the measurement.

[Second Embodiment]
FIG. 2 illustrates an information processing system according to the second embodiment. The information processing system according to the second embodiment includes a simulation apparatus 100 and server apparatuses 210 to 240. The simulation apparatus 100 and the server apparatuses 210 to 240 are connected to the network 31. The simulation apparatus 100 is an example of the information processing apparatus 10 described above, and the set of server apparatuses 210 to 240 is an example of the computer system 20 described above.

  The simulation device 100 is a computer as a terminal device operated by a user. The simulation apparatus 100 causes the server apparatuses 210 to 240 to execute a sample program, measures the calculation time, and generates a model that evaluates the performance of the server apparatuses 210 to 240 from the measurement result. Further, the simulation apparatus 100 reserves a part or all of the resources included in the server apparatuses 210 to 240 by designating the resource amount and the use period in accordance with an instruction from the user, and stores a program in the server apparatuses 210 to 240. Let it run.

  The server apparatuses 210 to 240 are server computers that execute a program acquired from a user using resources reserved by the user. It can be said that the set of server apparatuses 210 to 240 is a parallel processing system capable of performing calculations using a plurality of processors in parallel. The amount of resources specified by the user includes the number of processors. For example, a server device that controls a job receives a program execution instruction from the simulation device 100, selects a specified number of processors from free processors to which no job is assigned, and assigns a job to the selected processor. To start the calculation.

  FIG. 3 is a block diagram illustrating a hardware example of the simulation apparatus. The simulation apparatus 100 includes a CPU 101, a RAM 102, an HDD 103, an image signal processing unit 104, an input signal processing unit 105, a disk drive 106, and a communication interface 107. The CPU 101 is an example of the calculation unit 12 according to the first embodiment, and the RAM 102 and the HDD 103 are examples of the storage unit 11 according to the first embodiment.

  The CPU 101 is a processor including an arithmetic unit that executes program instructions. The CPU 101 loads at least a part of the program and data stored in the HDD 103 into the RAM 102 and executes the program. The CPU 101 may include a plurality of processor cores, the simulation apparatus 100 may include a plurality of processors, and the processes described below may be executed in parallel using a plurality of processors or processor cores.

  The RAM 102 is a volatile memory that temporarily stores programs executed by the CPU 101 and data used for calculation. Note that the simulation apparatus 100 may include a type of memory other than the RAM, or may include a plurality of memories.

  The HDD 103 is a nonvolatile storage device that stores an OS (Operating System), software programs such as firmware and application software, and data. Note that the simulation apparatus 100 may include other types of nonvolatile storage devices such as a flash memory, and may include a plurality of nonvolatile storage devices.

  The image signal processing unit 104 outputs an image to the display 32 connected to the simulation apparatus 100 in accordance with a command from the CPU 101. As the display 32, a CRT (Cathode Ray Tube) display, a liquid crystal display, or the like can be used.

  The input signal processing unit 105 acquires an input signal from the input device 33 connected to the simulation apparatus 100 and notifies the CPU 101 of the input signal. As the input device 33, a pointing device such as a mouse or a touch panel, a keyboard, or the like can be used.

  The disk drive 106 is a drive device that reads programs and data recorded on the recording medium 34. As the recording medium 34, for example, a magnetic disk such as a flexible disk (FD) or HDD, an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk (MO). Can be used. For example, the disk drive 106 stores the program and data read from the recording medium 34 in the RAM 102 or the HDD 103 in accordance with an instruction from the CPU 101.

  The communication interface 107 is an interface that can communicate with the server apparatuses 210 to 240 via the network 31. The communication interface 107 may be a wired interface connected to a wired network or a wireless interface connected to a wireless network.

  The server apparatuses 210 to 240 can also be realized by the same hardware configuration as that of the simulation apparatus 100. However, the server apparatuses 210 to 240 may not be connected to a display or an input device, and may not include an image signal processing unit, an input signal processing unit, and a disk drive. Each of the server devices 210 to 240 may include a plurality of CPUs, RAMs, and HDDs.

  FIG. 4 is a block diagram illustrating an example of functions of the simulation apparatus. The simulation apparatus 100 includes a measurement unit 111, a measurement information storage unit 112, a model generation unit 113, an error determination unit 114, and an evaluation result display unit 115. The measurement information storage unit 112 can be realized as a storage area secured in the RAM 102 or the HDD 103, for example. The measurement unit 111, the model generation unit 113, the error determination unit 114, and the evaluation result display unit 115 can be realized as a module of a program executed by the CPU 101, for example.

  The measurement unit 111 causes the parallel processing system including the server apparatuses 210 to 240 to execute the sample program a plurality of times while changing the number of processors, and measures the calculation time required for the parallel processing system. For example, the measurement unit 111 designates twelve ways of 10, 20, 30, 40, 50, 60, 70, 80.90, 100, 110, and 120 as the number of processors, and measures the calculation time in each case. . As the number of processors increases, the overhead of parallelization also increases. Therefore, the calculation time does not necessarily decrease as the number of processors increases.

  Note that the problem to be solved by the sample program is the same type as the large-scale problem to be solved using the server apparatuses 210 to 240 (for example, a calculation algorithm is similar) and is small. The calculation time is a time from when any one of a plurality of processors to be used starts calculation until all the processors finish calculation. The calculation time may be calculated based on the logs generated by the server devices 210 to 240, or may be calculated by measuring the time from when the measurement unit 111 transmits a command until the response is received. .

The measurement information storage unit 112 stores measurement information indicating the measurement result of the measurement unit 111. The measurement information includes n entries (n is an integer of 2 or more) indicating a set of the number of processors p _i and the measured calculation time d _i (i = 1, 2,..., N). For example, when the measurement unit 111 specifies twelve processors and measures the calculation time, the measurement information includes 12 entries.

The model generation unit 113 generates a model corresponding to the server apparatuses 210 to 240 by fitting the measurement information stored in the measurement information storage unit 112 to a model formula. The model formula used in the second embodiment is a function T (C, p, x) for calculating the calculation time from the parameter set C, the number of processors p, and the problem size x. The parameter set C includes one or more parameters. Each parameter takes a value corresponding to the architecture of the server apparatuses 210 to 240 and the type of problem to be solved. The model generation unit 113 fixes the problem size x and applies the least squares method to the set of the processor number p _i and the calculation time d _i indicated by the measurement information, whereby each parameter c _j (j = 1, 2, ...) are determined.

The error determination unit 114 evaluates the accuracy of the model generated by the model generation unit 113 determining the value of the parameter c _j . Specifically, the error determination unit 114 estimates an error that occurs in the value of the parameter c _{j in} the process of fitting the measured calculation time d _i to the model formula, and the error of the parameter c _j is used to predict the calculation time. Quantitatively assess the degree of impact.

First, the error determining unit 114 uses the variation amount .DELTA.c _j values of the parameters c _j as free variables, generating n inequalities corresponding to the processor number p _i of the n different illustrated measurement information. Each inequality indicates that the difference between the measured calculation time d _i and the calculation time T (C + ΔC, p _i , x _i ) calculated according to the model equation is allowed up to a threshold value e. ΔC is a set of fluctuation amounts Δc _j . By substituting the value determined by the model generation unit 113 for the parameter c _j and determining the threshold value e by a predetermined method, the n inequalities include only the variation Δc _j as a free variable. Then, the error determination unit 114 calculates the range of possible values of the variation set ΔC by solving the generated n inequalities using the quantifier elimination (QE) method described later. Thereby, it is possible to calculate a range in which the calculation time predicted from the model varies.

The evaluation result display unit 115 includes a model curve using the value of the parameter c _j calculated by the model generation unit 113 and a model curve using the value of the parameter c _j + Δc _j considering the error calculated by the error determination unit 114. The curve is combined and displayed on the display 32. Thereby, the user of the simulation apparatus 100 can visually confirm the accuracy of the model. In addition, when the number of processors is designated by the user, the evaluation result display unit 115 indicates how much the calculation time calculated using the model generated by the least square method may increase or decrease by a maximum. The calculated calculation time and its error are displayed on the display 32. Thereby, the user can confirm the prediction error of calculation time quantitatively.

  Here, the quantified erasure (QE) will be described. Quantum erasure is mathematical expression processing that converts a logical expression including a quantifier described as first-order predicate logic into an equivalent expression that does not include a quantifier. An input logical expression can be described using an algebraic equation, an algebraic inequality, a quantifier (generic quantifier and existence quantifier), and a logical operator (logical product or logical sum). If the input logical expression includes a free variable not having a quantifier in addition to a bound variable having a quantifier, the output expression indicates a range of values that the free variable can take.

For example, the left side of Equation (1) shown below requires that the logical expression in parentheses is satisfied for all values of x. When the bound variable x is eliminated from the logical expression by the quantifier elimination method, as shown on the right side of Equation (1), an expression indicating the conditions of the free variables a, b, and c for establishing the left-side logical expression is calculated. The In addition, the left side of the following formula (2) requires that there is at least one set of x ₁ and x ₂ values such that the logical expression in parentheses is established. When the bound variables x ₁ and x ₂ are eliminated from the logical expression by the quantifier elimination method, as shown on the right side of Equation (2), an expression indicating the condition of the free variable y for establishing the logical expression on the left side is calculated. The Such quantifier elimination may be implemented in application software that performs mathematical expression processing.

FIG. 5 is a flowchart illustrating a procedure example of calculation time prediction.
(Step S1) The measurement unit 111 causes the server apparatuses 210 to 240 to execute the sample program while changing the number of processors p, with the problem size x fixed (for example, using the same data as the same sample program). , N calculation times d are measured. For example, the measurement unit 111 measures twelve calculation times d of p = 10, 20,. The measurement unit 111 generates measurement information that associates the number of processors p _i with the measured calculation time d _i , and stores the generated measurement information in the measurement information storage unit 112.

(Step S2) The model generation unit 113 fits n calculation times d _i indicated by the measurement information stored in the measurement information storage unit 112 to a function T (C, p, x) that is a model formula. The value of each parameter c _j belonging to the parameter set C is determined. In the fitting, the value of each parameter c _j is determined by the least square method so that the sum of squares of the residuals e _i (see the following formula (3)) for each number of processors p _i is minimized.

(Step S3) The error determination unit 114 generates n inequalities corresponding to the n calculation times d _i indicated by the measurement information, as shown in the following formula (4). Each inequality is such that the difference between the measured calculation time d _i and the equation T (C + ΔC, p _i , x _i ) representing the calculation time after the parameter change using the variation ΔC as a free variable is less than or equal to the threshold e. Indicates that there is. The expression representing the calculation time after the parameter change is generated by replacing the parameter set C in the model expression with C + ΔC and substituting the value of the parameter set C determined in step S2 and the number of processors p _i .

Further, the error determination unit 114 determines a threshold value e. The threshold value e is equal to or less than the maximum value Max (e _i ) among the _n residuals e _{1 to} en calculated by Expression (3), and the n inequalities have a solution. It is desirable that the lower limit value e _m or more. Lower limit e _m is typically a minimum value Min (e _i) is greater than value in the residual e ₁ to e _n. If you select Max (e _i) as a threshold value e, it will be pessimistic evaluate the error of the parameter set C, if you select the e _m as the threshold value e, optimistically evaluation to the error parameter set C become. From the standpoint of model accuracy evaluation, the error determining unit 114, preferably, for both the cases of the threshold e = e _m threshold e = Max (e _i), the following processing is executed. A method of calculating the lower limit value e _m threshold e of n inequalities will be described later.

  (Step S4) The error determination unit 114 applies the quantifier elimination (QE) method to the set of expressions defining the n inequalities and the threshold e generated in Step S3, and eliminates the threshold e that is a bound variable. Then, the range of possible values of the fluctuation amount ΔC, which is a free variable, is calculated.

(Step S5) The error determination unit 114 increases the calculation time calculated according to the model the most from the range of possible values of the fluctuation amount ΔC calculated in Step S4 (the positive error becomes the largest) Δc _j Extract combinations. In addition, the error determination unit 114 extracts a combination of the fluctuation amounts Δc _j that reduces the calculation time calculated according to the model (the negative error becomes the largest) from the range of values that the fluctuation amount ΔC can take.

(Step S6) The evaluation result display unit 115 displays on the display 32 a graph generated by substituting the value of each parameter c _j determined by the model generation unit 113 in step S2 into the model formula. The evaluation result display unit 115 generates the parameter c _j by changing the value of the parameter c _j by the variation Δc _j extracted by the error determination unit 114 in step S5 (substituting the value of c _j + Δc _j into the model formula). The graph to be displayed is displayed on the display 32.

As an example, consider the following equation (5) with the problem size x fixed (not including the problem size x as a variable) as a model equation used for fitting. In Equation (5), the two parameters c ₁ and c ₂ belong to the parameter set C, and the constant a is a fixed value. Equation (5) includes a term that decreases as the number of processors p increases, and a term that increases as the number of processors p increases. The former reflects a decrease in the amount of calculation per processor due to load balancing, and the latter reflects the overhead due to communication between processors. According to Equation (5), when the number of processors p is gradually increased from 1, the calculation time is shortened at the beginning. However, after a certain number of processors, the calculation time becomes longer due to the overhead.

The following formula (6) is an example of a logical expression of the first order predicate logic generated in step S3 when the model formula of the formula (5) is used. Here, the measurement unit 111 has twelve calculation times d ₁ corresponding to twelve different processor numbers p = 10, 20, 30, 40, 50, 60, 70, 80, 90, ₁₀₀ , 110, 120. consider the case where the ~d ₁₂ was measured.

In this logical expression, twelve inequalities corresponding to twelve different processor numbers are connected by logical product. In Equation (6), the constant a is a fixed value, and the values of the parameters c ₁ and c ₂ are those determined in step S2. The maximum value Max (e _i ) of the residual is substituted for the threshold value e. It is also possible to substitute another value (for example, the lower limit value e _m ) for the threshold value e. Variation .DELTA.c ₂ variation amount .DELTA.c ₁ and the parameter c ₂ parameters c ₁ is a free variable, threshold e has a presence quantifier is attached bound variables. This logical expression indicates that the absolute value of each of the 12 residuals is allowed to vary within a range equal to or less than the threshold value e.

When the quantifier elimination method is applied to the equation (6), the equation (7) is calculated as a solution of the set of inequalities. Equation (7) shows a range of possible values of the fluctuation amounts Δc ₁ and Δc ₂ that were free variables in Equation (6). Since the logical expression as input includes two free variables, the expression as output shows the relationship between the two free variables. The relationship between the fluctuation amount Δc ₁ and the fluctuation amount Δc ₂ expressed by the equation and the inequality can be visualized on a two-dimensional plane.

FIG. 6 is a graph illustrating an example of values that can be taken by the parameter. When Expression (7) is mapped onto a two-dimensional plane of variation Δc ₁ × variation Δc ₂ , a graph as shown in FIG. 6 is generated. The regions indicating the movable ranges of the fluctuation amounts Δc ₁ and Δc ₂ are (Δc ₁ , Δc ₂ ) = (0.000946051, −7.996273 × 10 ⁻⁸ ), (−0.000115456, 4.14635 × 10 ^{− 8} ) The two points are the end points. These two end points correspond to the solutions of the first two lines of Equation (7). One of the two end points, variation .DELTA.c ₁ to maximize the positive error, it shows a combination of .DELTA.c _2, variation .DELTA.c ₁ the other is to maximize the negative error indicates the combination of .DELTA.c _2.

As described above, by generating a graph from the result of the quantified erasure and detecting the end point, it is possible to determine the combination of the variation Δc _j that maximizes the model error. However, such a combination of fluctuation amounts Δc _j can also be confirmed by the quantifier elimination method from the n inequalities generated in step S3. As shown in Equation (8), only one variation amount is selected as a free variable among the variables included in the n inequalities, and the other variation amount and the threshold value e are attached to the bound variable with existence quantifiers. To do. When the bound variable is eliminated by applying the quantum elimination method to the set of inequalities, only the range of values that can be taken by one selected variation is calculated. Such calculation of Expression (8) is performed for each variation amount Δc _j .

Quantum erasure is performed by changing the bound variable in Equation (6) to the variation Δc ₁ and threshold e, and the quantifier erasure is performed by changing the bound variable in Equation (6) to variation Δc ₂ and threshold e. By doing so, the following formula (9) is calculated. The combination of the upper limit value of the fluctuation amount Δc _{1 and} the lower limit value of the fluctuation amount Δc ₂ corresponds to one end point of the graph shown in FIG. 6, and the combination of the lower limit value of the fluctuation amount Δc _{1 and} the upper limit value of the fluctuation amount Δc ₂ . Corresponds to the other end point of the graph shown in FIG.

As described above, the threshold e in Equation (6), instead of the maximum value Max (e _i) of the residual may be assigned a lower value e _m of the extent that there is a solution to the inequality. Lower limit e _m can be calculated according to equation (10) shown below. As shown in Equation (10), among the variables included in the n inequalities, only the threshold value e is set as a free variable, and all variables are attached with existence qualifiers as bound variables. When the bound variable is erased by applying the quantifier elimination method to the set of inequalities, all the fluctuation amounts are erased and only the range of values that the threshold e can take is calculated. The minimum value of the threshold e indicated by the calculated expression, is employed as the lower limit value e _m.

FIG. 7 is a first graph illustrating an example of a calculation time prediction result. The points (x marks) plotted in FIG. 7 indicate the calculation time d _i measured by the measurement unit 111. Also, depicted in Figure 7, a curve showing the model generated by the method of least squares, and the curve showing the upper and lower limits of error, and a curve representing the optimistic error calculated using the lower limit value e _m as a threshold e ing. As shown in FIG. 7, when the size of the problem and the number of processors used are the same as when the sample program is executed (small enough), the error in the calculation time predicted from the model seems to be sufficiently small. Looks like. However, as the size of the problem and the number of processors used increase, the computation time predicted from this model can include large errors.

FIG. 8 is a second graph illustrating an example of calculation time prediction results. A graph as shown in FIG. 8 can be displayed on the display 32 by the evaluation result display unit 115. The graph of FIG. 8 is obtained by extending the horizontal axis of the graph of FIG. 7, a curve indicating a generated model, a curve indicating an upper limit and a lower limit of an error, and a lower limit value as a threshold e in a section where the number of processors is large. and a curve indicating an error calculated using e _m. As shown in FIG. 8, when a model generated based on a small-scale problem is applied to a large-scale problem, a small error generated in the parameter set C during the fitting process greatly affects the predicted calculation time. give. For example, the user can quantitatively confirm how much the predicted calculation time corresponding to a certain number of processors may vary by looking at the graph as shown in FIG. As described above, the evaluation result display unit 115 may calculate and present to the user what percentage error occurs.

  According to the information processing system of the second embodiment, a model corresponding to the server apparatuses 210 to 240 is generated by determining the value of the parameter set C of the model formula from the measured calculation time, and calculation for an arbitrary program is performed. Time can be predicted. Further, by estimating an error that occurs in the parameter set C, it is possible to quantitatively evaluate to what degree the generated model can predict the calculation time. For this reason, even when a parallel processing system uses a large resource to execute a program that solves a problem larger than the sample problem, the prediction of the calculation time by the model can be effectively used.

  For example, when the prediction accuracy is acceptable, the user can reserve resources of the parallel processing system in consideration of the prediction error. On the other hand, when the prediction accuracy is unacceptable, in order to generate a model with higher prediction accuracy, the measurement information is searched for abnormal measurement data, the abnormal measurement data is removed, and the model is regenerated. be able to.

  As described above, the information processing of the first embodiment can be realized by causing the information processing apparatus 10 to execute a program, and the information processing of the second embodiment executes a program to the simulation apparatus 100. This can be achieved. The program can be recorded on a computer-readable recording medium (for example, the recording medium 34). As the recording medium, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like can be used. Magnetic disks include FD and HDD. Optical discs include CD, CD-R (Recordable) / RW (Rewritable), DVD, and DVD-R / RW.

  When distributing the program, for example, a portable recording medium in which the program is recorded is provided. It is also possible to store the program in a storage device of another computer and distribute the program via a network. The computer stores, for example, a program recorded on a portable recording medium or a program received from another computer in a storage device (for example, HDD 103), and reads and executes the program from the storage device. However, a program read from a portable recording medium may be directly executed, or a program received from another computer via a network may be directly executed.

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 11 Memory | storage part 11a Measurement information 12 Calculation part 20 Computer system

Claims (6)

A computer-implemented performance evaluation method for evaluating the relationship between the amount of resources used by a computer system and calculation time,
An evaluation model for calculating the relationship between the resource amount and the calculation time based on a plurality of first calculation times measured by causing the computer system to execute the sample program a plurality of times while changing the amount of resources to be used Determine the value of the parameter used for
When the difference between each of the measured first calculation times and the second calculation time corresponding to the first calculation time calculated according to the evaluation model is allowed to spread to a threshold value, Calculate the amount of change in the parameter value,
Information indicating the influence of the relationship between the resource amount indicated by applying the determined parameter value to the evaluation model and the calculation time due to the determined parameter value varying by the calculated variation amount Output performance evaluation method.   In the calculation of the fluctuation amount, the difference between the measured first calculation time and the second calculation time corresponding to the first calculation time is equal to or less than the threshold value, and the fluctuation amount of the parameter value is calculated. The performance evaluation method according to claim 1, wherein a plurality of inequalities expressed using the indicated free variable is generated, and a value that the free variable can take is calculated by mathematically processing the set of the plurality of inequalities.   The performance evaluation method according to claim 2, wherein the mathematical expression processing includes applying a quantifier elimination method to the set of the plurality of inequalities.   As the threshold, a difference between each measured first calculation time and a second calculation time corresponding to the first calculation time calculated by applying the value of the determined parameter to the evaluation model The performance evaluation method according to claim 1, wherein the maximum value is used. A storage unit for storing measurement information indicating a plurality of first calculation times measured by changing the amount of resources to be used and causing the computer system to execute the sample program a plurality of times;
A calculation unit that evaluates a relationship between the resource amount and the calculation time using the measurement information;
The calculation unit includes:
Based on the measurement information, determine the value of the parameter used in the evaluation model for calculating the relationship between the resource amount and the calculation time,
When the difference between each of the plurality of first calculation times indicated by the measurement information and the second calculation time corresponding to the first calculation time calculated according to the evaluation model is allowed to spread to a threshold value , Calculate the amount of change in the parameter value,
Information indicating the influence of the relationship between the resource amount indicated by applying the determined parameter value to the evaluation model and the calculation time due to the determined parameter value varying by the calculated variation amount An information processing apparatus for outputting. A program for evaluating the relationship between the amount of resources used by a computer system and calculation time.
An evaluation model for calculating the relationship between the resource amount and the calculation time based on a plurality of first calculation times measured by causing the computer system to execute the sample program a plurality of times while changing the amount of resources to be used Determine the value of the parameter used for
When the difference between each of the measured first calculation times and the second calculation time corresponding to the first calculation time calculated according to the evaluation model is allowed to spread to a threshold value, Calculate the amount of change in the parameter value,
Information indicating the influence of the relationship between the resource amount indicated by applying the determined parameter value to the evaluation model and the calculation time due to the determined parameter value varying by the calculated variation amount A program that outputs and executes processing.

Images