CN115587014A

CN115587014A - Performance calculation method, system and medium for high-performance computer workflow scheduling

Info

Publication number: CN115587014A
Application number: CN202211166770.4A
Authority: CN
Inventors: 董勇; 戴屹钦; 王睿伯; 卢凯; 张伟; 张文喆; 谢旻; 周恩强; 迟万庆; 邬会军; 李佳鑫; 吴振伟; 雷斐
Original assignee: National University of Defense Technology
Current assignee: National University of Defense Technology
Priority date: 2022-09-23
Filing date: 2022-09-23
Publication date: 2023-01-10

Abstract

The invention discloses a performance calculation method, a system and a medium for high-performance computer workflow scheduling, wherein the performance calculation method comprises the steps of initializing a set X and a set Y which can start to run tasks, and then iterating: task v if X _i If the ending time of (b) is equal to the variable k, updating X and Y; if Y is not empty, selecting a set Z of tasks, updating the starting time of the tasks to be k, adding the starting time to X, calculating the estimated ending time of the kth stage aiming at the tasks in X, and taking the shortest estimated ending time as the finishing time of the kth stage; and repeating the iteration until X and Y are both empty, and outputting the sum of the completion time of each stage as the total completion time of the workflow. The invention can realize the performance quantitative calculation of the high-performance computer workflow scheduling so as to quickly determine the influence of each variable on the workflow scheduling performance in the workflow scheduling operation process, thereby conveniently determining the minimum resource quantity required by the optimal workflow scheduling performance.

Description

Performance calculation method, system and medium for high-performance computer workflow scheduling

Technical Field

The invention relates to a workflow scheduling technology of a high-performance computer, in particular to a performance calculation method, a system and a medium for workflow scheduling of the high-performance computer.

Background

Workflows (scientific workflows) are a sequence of tasks defined to achieve various scientific research goals. Driven by the development of service-oriented architectures and their loosely-coupled nature, workflow has become a key technology in the current distributed and dynamic environments. Workflow has significant advantages in describing complex scientific problems, making it commonly used to solve large-scale scientific problems in the fields of bioinformatics, astronomy, and physics. In particular, a workflow is typically composed of multiple independent computing tasks with strict dependencies. Directed acyclic graphs are an efficient tool for representing workflows. As shown in FIG. 1, nodes in the graph represent independent tasks in the workflow, and directed edges represent dependencies between the tasks. The weight of a node represents the amount of computing resources (core number or node number) required by the task, and the weight of a directed edge represents the data dependency between two tasks. See, for example, FIG. 1, any of whichAffair v ₁ Weight 2 of represents task v ₁ Requiring the use of 2 computing resources (2 cores or 2 compute nodes) from task v ₁ To task v ₃ Represents the task v with directed edges ₃ Need to be at task v ₁ Thereafter, operation is started and a read task v is required ₁ Generated 10GB of data.

The goal of workflow scheduling is to maintain good overall performance or throughput of the computing system while meeting user needs and resource provider management metrics. For a single workflow, minimizing the completion time of the workflow is a common scheduling objective. For a given workflow, the shorter its completion time, the higher the workflow scheduling performance. The amount of computational and I/O resources allocated to a workflow and the scheduling policy all affect the scheduling performance of the workflow. In recent years, with the increasing parallel performance of large-scale high-performance computers, the high-performance computers are becoming important operation platforms of workflows. The scenario of scheduling workflows on a high performance computer is complex. FIG. 2 illustrates a scenario for scheduling a workflow on a high performance computer. Firstly, a temporary independent resource partition is opened on a high-performance computer, all tasks in a workflow run in the partition, wherein the total number of resources in the partition should be larger than the resource requirement of any one task. Each workflow task is submitted to the high-performance computer to run at an appropriate point in time as a separate batch task. The shared file system is a storage medium for data transmission among tasks, and each workflow task reads data from the shared file system and writes data into the shared file system. During the operation of the workflow, there is I/O interference between different tasks that are running simultaneously, and the read-write rate of the resource partition to the file system may change over time. Burst buffers are a storage technique proposed to meet user requirements for better I/O performance. The use of burst buffers may increase the total bandwidth available to an application. Therefore, for a high-performance computer with a certain burst buffer capacity, according to the burst buffer capacity, some or all of the tasks may be allowed to use the burst buffer to improve the I/O efficiency of the application. When a task is allowed to use the burst buffer, all of its outputs are directed to the burst buffer. But for each task where (shared file system or burst buffer) the task reads data from depends on whether its predecessor uses a burst buffer. Therefore, the scene of dispatching the workflow on the high-performance computer is very complex, and an effective tool for researching the influence of each variable on the dispatching performance of the workflow in the dispatching operation process of the workflow, particularly the influence of each dispatching strategy on the dispatching performance of the workflow, is lacked at present. Furthermore, for a given workflow, it is difficult to determine the minimum amount of resources needed to achieve optimal workflow scheduling performance. Therefore, how to implement performance calculation of high-performance computer workflow scheduling becomes a key technical problem to be solved urgently.

Disclosure of Invention

The technical problems to be solved by the invention are as follows: aiming at the problems in the prior art, the invention provides a performance calculation method, a system and a medium for high-performance computer workflow scheduling, which can realize the performance quantitative calculation of the high-performance computer workflow scheduling so as to quickly determine the influence of each variable on the workflow scheduling performance in the workflow scheduling operation process, thereby conveniently determining the minimum resource quantity required by the optimal workflow scheduling performance.

In order to solve the technical problems, the invention adopts the technical scheme that:

a performance calculation method for high performance computer workflow scheduling, comprising:

s1, initializing a variable k to be 1, and finishing time T of the kth stage _k Is 0; setting the starting time and the ending time of all tasks in the workflow to be 0; initializing a set Y which is used for recording that a set X of the current running task is empty and the running task can be started;

s2, aiming at all the tasks in the set X, if a certain task v _i If the termination time is equal to the variable k, updating the set X and the set Y;

s3, if the set Y is not empty, selecting a set Z which starts to run the task from the set Y, and if the set Z is not empty, skipping to S4; otherwise, skipping S5;

s4, aiming at each task in the set Z, updating the starting time of the task to be a variable k, and adding the task into the set X:

s5, aiming at each task v in the set X _i Calculation task v _i Read-write bandwidth to the shared file system, and calculate task v according to the read-write bandwidth to the shared file system of the task _i Estimated end time at kth stage

S6, determining the estimated ending time at the k stage according to all tasks in the set X

Shortest task v _n Will task v _n Corresponding estimated end time

Assigning a completion time T to the kth stage _k And sets up task v _n The end time of (d) is k +1;

s7, if the set X and the set Y are both empty, the completion time T of all the kth stages is determined _k Taking the sum of the total completion time of the workflow as the total completion time of the workflow, ending and exiting; otherwise, the variable k is added with 1, and the step S2 is skipped to enter the next stage.

Optionally, the function expression of the set X and the set Y updated in step S2 is:

X＝X-v _i ，

in the above formula, v _i For tasks whose termination time is equal to the variable k, v _j For task v _i Set of successor tasks succ (v) _i ) Task of (1), v _m For task v _j Set of predecessor tasks of pred (v) _j ) Task of (1), e _m For task v _m The end time of (c).

OptionallyIn step S5, the calculation task v _i Estimated end time at kth stage

The functional expression of (a) is:

in the above-mentioned formula, the compound has the following structure,

for task v _i Estimated end time, T, at stage k +1 _k Is the completion time of the kth stage, C _i For task v _i The total amount of the calculation of (c),

for task v _i Quantity calculated in m-th stage, p _i For task v _i The number of the used computing nodes, s is the computing speed of the computing nodes,

and

are respectively tasks v _i Read and write Bandwidth to shared File System, FI, at stage k _i And FO _i Respectively representing tasks v _i Total read and write data size, BI, to shared file system _i And BO _i Respectively represent tasks v _i The total size of read and write data to the burst buffer, F the shared file system size, B the burst buffer size, I _i And O _i Are respectively task v _i The total size of the read and write data of (c),

and W _i ^(k) Are respectively tasks v _i Read and write data size at the kth stage.

Optionally, step SAlso included in 5 are for each task v in the set X _i Calculation task v _i For the calculated amount in the k stage, and calculate task v _i The functional expression for the calculated amount at the k-th stage is:

in the above formula, C _i ^(k) For task v _i Quantity of calculation in the k-th stage, T _k+1 Is the completion time of the k +1 th stage, min represents the minimum value, R _B And W _B The read bandwidth and the write bandwidth of the burst buffer area are respectively.

Optionally, the task v _i Read bandwidth at stage k, task v _i The computational function expression of the write bandwidth at the kth stage is:

in the above formula, the first and second carbon atoms are,

and

are respectively tasks v _i Read and write Bandwidth at stage k, task v _j Set X of running tasks for the k-th stage ^(k) In the task (1), min represents taking the minimum value,

and

respectively representing tasks v _i For the density of reading and writing to the shared file system,R _s read bandwidth, W, for computing nodes to a parallel file system _s Write Bandwidth, R, for a compute node to a parallel File System _f Read bandwidth, W, for a parallel file system for a node partition _f Write Bandwidth, p, to parallel File System for node partitioning _i For task v _i The number of compute nodes used.

Optionally, task v _i The computational function expression for read-write density to a shared file system is:

in the above-mentioned formula, the compound has the following structure,

and

respectively representing tasks v _i For read-write density to a shared file system, FI _i And FO _i Respectively representing tasks v _i Total read and write data size, t, for a shared file system _i For task v _i The time required for operation.

Optionally, task v _i The calculation function expression of the total size of the read data and the write data of the shared file system is as follows:

task v _i The total read and write data size of the burst buffer is calculated by the following functional expression：

Wherein, FI _i And FO _i Respectively representing tasks v _i Total read and write data size, BI, to shared file system _i And BO _i Respectively representing tasks v _i Total read and write data size, v, to burst buffer _j For task v _i Succ (v) successor task set of _i ) The task (2) of (1) is,

scheduling policy for employing preset storage resources

To schedule task v _j Whether or not burst buffers are allowed to be used, e _ji Representing a task v _j And task v _i Amount of data transferred between, O _i For task v _i Has a total write data size of v and has a task of v _i The calculation function expression of the total calculation amount of (a) is:

C _i ＝p _i *s，

task v _i The calculation function expression of the total size of the read data and the write data is as follows:

in the above formula, e _ij For task v _i And task v _j Amount of data to be transmitted between, succ (v) _i ) For task v _i Is the set of successor tasks of, pred (v) _i ) For task v _i The set of predecessor tasks.

In addition, the invention also provides a performance calculation method for high-performance computer workflow scheduling, which comprises the following steps:

s101, aiming at four influencing factors of system computing resources, system storage resources, computing resource scheduling strategies and storage resource scheduling strategies in a high-performance computer on the scheduling performance of a workflow, generating a plurality of resource allocation schemes by fixing three influencing factors and changing the resource allocation of the remaining influencing factor, and aiming at a given workflow, calling a performance computing method for scheduling the workflow of the high-performance computer aiming at each resource allocation scheme to obtain the total completion time of the corresponding workflow, thereby obtaining a relation curve between each influencing factor and the total completion time of the workflow;

s102, based on a relation curve between each influence factor and the total completion time of the workflow, respectively selecting the optimal resource allocation from the four influence factors to obtain the optimal resource allocation scheme of the high-performance computer for the given workflow.

In addition, the invention also provides a performance computing system for high-performance computer workflow scheduling, which comprises a microprocessor and a memory which are connected with each other, wherein the microprocessor is programmed or configured to execute the performance computing method for the high-performance computer workflow scheduling.

Furthermore, the present invention also provides a computer-readable storage medium having stored therein a computer program for being programmed or configured by a microprocessor to perform the performance calculation method of the high-performance computer workflow schedule.

Compared with the prior art, the invention mainly has the following advantages: the invention can realize the performance quantitative calculation of the high-performance computer workflow scheduling so as to quickly determine the influence of each variable on the workflow scheduling performance in the workflow scheduling operation process, thereby conveniently determining the minimum resource quantity required by the optimal workflow scheduling performance.

Drawings

Fig. 1 is a schematic diagram of a conventional workflow.

Fig. 2 is a schematic diagram of a scenario of scheduling a workflow on a conventional high-performance computer.

FIG. 3 is a schematic diagram of a basic flow of a method according to an embodiment of the present invention.

FIG. 4 is a multi-stage schematic of a workflow process according to an embodiment of the invention.

Detailed Description

As shown in fig. 3, the performance calculating method for high-performance computer workflow scheduling in this embodiment includes:

s1, initializing a variable k to be 1, and finishing time T of the kth stage _k Is 0; setting the starting time and the ending time of all tasks in the workflow to be 0; initializing a set Y for recording that a set X of the currently running tasks is empty and the tasks can be started to run;

s2, aiming at all the tasks in the set X, if a certain task v _i If the ending time of the set X is equal to the variable k, updating the set X and the set Y;

s4, aiming at each task in the set Z, updating the starting time of the task to be a variable k, and adding the task into the set X;

Shortest task v _n Will task v _n Corresponding estimated end time

Assigning a completion time T to the kth stage _k And sets up task v _n The end time of (2) is k +1;

In step S1 of this embodiment, setting the start time and the end time of all tasks in the workflow to 0 may be represented as:

in the above formula, G represents a workflow, v _i For tasks in workflow G, s _i And e _i Respectively represent v _i The start time and the end time of (c). Specifically, the workflow is represented by a directed acyclic graph in the present embodiment. In a directed acyclic graph G = (V, E). Set of nodes V = { V = ₁ ，v ₂ ，...，v _n Are independent tasks in the workflow. Each task v _i There are two attributes: one is task v _i Number of compute nodes p used _i Second is task v _i Time t required for operation _i . Thus, the total computation of the definable tasks is:

C _i ＝p _i *s，

where s is a constant representing the speed of operation of a single compute node in the system. Edge set

Representing dependencies between tasks in the workflow. If (v) ₁ ，v ₂ ) If it belongs to the edge set E, it indicates v ₁ And v ₂ Have a dependency relationship between them, i.e. v ₂ Must be at v ₁ And starting operation after the operation is finished. The weight of each edge represents the data dependency between two tasks with dependenciesIs, for example e _ij =10 denotes task v _i Writing 10GB data into file system or burst buffer, and reading task vj from file system or burst buffer by task v _i 10GB of data are written. Task v _i Is pred (v) as a set of predecessor tasks _i ) The set of successor tasks is succ (v) _i ). Adding an empty task v ₀ If a task has no successor, the null task v ₀ As a continuation of this task to constitute the initial task. Adding an empty task v _n+1 If a task has no successor tasks, the task v is executed _n+1 As a successor to the task (end task v) _n+1 ). Initial task v ₀ And end task v _n+1 The number of compute nodes used is 0, as is the run time. In step S1 of this embodiment, initializing a set X for recording that a currently running task is empty and a set Y for starting to run a task is represented as:

X＝φ，

Y＝{v _i s.t.pred(v _i )＝v ₀ }，

in the above formula,. Phi.represents the null set, pred (v) _i ) For task v _i V. set of predecessors of ₀ Is an initial task, i.e. a task that can be executed independently, independent of other task data.

In step S2 of this embodiment, the function expressions of the set X and the set Y are updated as follows:

X＝X-v _i ，

If the set Y is not empty (Y ≠ Φ) in step S3 of this embodiment, selecting the set Z that starts to run the task from the set Y is implemented by invoking a preset computing resource scheduling policy η, which is generally expressed as Z = η (G, P, Y) according to the workflow G, the number of computing nodes P and the set Y, and the implementation of the computing resource scheduling policy η is not the content of interest of the method of this embodiment, such as random or round robin, and the detailed implementation details thereof are not described herein.

In step S4 of this embodiment, each task in the set Z is targeted

Updating the start time of the task to a variable k, and adding the task to the set X can be expressed as: s is _i ＝k，X＝X+v _i 。

The present embodiment calculates task v in step S5 _i Estimated end time F at the k-th stage _i ^(k) The functional expression of (a) is:

in the above formula, the first and second carbon atoms are,

for task v _i Calculated quantity in m-th stage, p _i For task v _i The number of the used computing nodes, s is the computing speed of the computing nodes,

and

are respectively tasks v _i Read and write bandwidth to a shared file system in the kth stage, FI _i And FO _i Respectively representing tasks v _i Total read and write data size, BI, to shared file system _i And BO _i Respectively represent tasks v _i The total size of read and write data to the burst buffer, F the shared file system size, B the burst buffer size, I _i And O _i Are respectively tasks v _i The total size of the read and write data of,

and W _i ^(k) Are respectively task v _i Read and write data size at the kth stage. The parameters involved in the calculation of the function may be calculated in this step, or may be calculated in an appropriate step as described above, as necessary.

In this embodiment, step S5 further includes, for each task v in the set X _i Calculation task v _i For the calculated amount in the k stage, and calculate task v _i The functional expression for the calculated amount at the k-th stage is:

in the above formula, the first and second carbon atoms are,

for task v _i Amount of calculation in the k-th stage, T _k+1 Is the completion time of the k +1 th stage, min represents the minimum value, R _B And W _B Respectively the read and write bandwidths of the burst buffer. The burst buffer typically has a communication link and storage device that is independent of the shared file system, so the read and write bandwidth of the burst buffer is not affected by the shared file system. The probability of I/O interference in a burst buffer is lower than in a shared file system for two reasons. One is that the read-write bandwidth of the burst buffer is usually much larger than that of the shared file system, and the read-write requirement of the task can be completed in a short time by using the burst buffer, so that I/O contention is avoided. The second is due to the capacity of the burst bufferVolume constraints, usually only a portion of discrete tasks may use the burst buffer. In conjunction with the above analysis, for model simplicity, task v will be described _i Read bandwidth for burst buffer at any stage is fixed to R _B Write bandwidth to burst buffer is fixed as W _B . When modeling the workflow operation process, the start or the end of each task is defined as an event, and the task v _i Has an initial running time of s _i The termination running time is e _i . The time period between two consecutive events is defined as a phase. Obviously, the operation cycle of the workflow is composed of a plurality of phases, and the completion time of the workflow is the time when the last event occurs. As shown in FIG. 4, event p is task v _i And v _j Start running for a time T _p Event p +1 is task v _i Terminating running simultaneous tasks v _j Start running for a time T _p+1 The time period between the two is the phase p. In a certain phase, a running task is stable, so the read-write bandwidth of the file system is constant, which is the key property of the phase and is the original intention of the concept of defining the phase. The task set consisting of tasks running in phase k is X ^(k) . For task v _i The amount of computation it performs in stage k is

Analyzing an already started stage k and analyzing any task v _i ∈X ^(k) The following five constraints need to be met:

of the five constraints mentioned above, the first constraint describes that the task must complete all the computing tasks in phase k, and the second and third constraints describe that the task must complete the read and write tasks to the shared file system in phase k. Note that task v is based on the previous assumptions _i Read-write demand on shared file system in phase k occupies task v _i The proportion of the total read-write demand on the shared file system is equal to task v _i The amount of computation in phase k accounts for task v _i Ratio of the total calculated amount. Similarly, the fourth and fifth limits describe that this task must complete the read and write tasks to the burst buffer during phase k. According to the five limits, the task v can be obtained _i The functional expression for the calculated amount at the k-th stage is shown as the above expression. According to task v _i For the functional expression of the calculated quantities in the k-th phase, for each task v running in phase k _i Assuming that no event has occurred all the time after phase k begins, task v may be determined _i Is estimated to be the end time

Taking the minimum task end time as the occurrence time of the next event, namely:

the read-write operation of the task is assumed to be uniformly distributed in the task running process, and under the condition that no I/O interference exists, the read-write data volume of the task to the file system is in direct proportion to the task execution progress. To describe I/O interference in a shared file system generated by multiple parallel tasks running simultaneously within the same resource partition, task v is thus defined _i Read bandwidth at stage k, task v _i Write bandwidth at phase k. In this embodiment, task v _i Read bandwidth, task v at stage k _i The computational function expression of the write bandwidth at the kth stage is:

in the above formula, the first and second carbon atoms are,

and

are respectively tasks v _i Read and write Bandwidth at stage k, task v _j Set X of running tasks for the k-th stage ^(k) In (1), min represents taking the minimum value,

and

respectively representing tasks v _i For read and write density to a shared file system, R _s Read bandwidth, W, of a parallel file system for a compute node _s Write Bandwidth, R, for computing nodes on a parallel File System _f Read bandwidth, W, for a parallel file system for a node partition _f Write Bandwidth, p, for node partitions to parallel File System _i For task v _i The number of compute nodes used.

The read bandwidth of a single computing node to the parallel file system is R _s Write bandwidth of W _s . Node partition pair parallelismThe reading bandwidth of the file system is R _f The writing bandwidth of the node partition to the parallel file system is W _f . The read bandwidth of a single computing node to the burst buffer is R _b Write bandwidth of W _b . In a practical scenario, the shared file system usually has a large enough capacity, and in this embodiment, the task v _i The computational function expression for read-write density to a shared file system is:

in the above formula, the first and second carbon atoms are,

and

respectively representing tasks v _i For read-write density to a shared file system, FI _i And FO _i Respectively representing tasks v _i Total read and write data size, t, for a shared file system _i For task v _i The time required for operation. According to the functional expression, the task v _i The occupation amount of the read-write bandwidth in the stage k is in positive correlation with the proportion of the read-write density of the task in the total read-write density, and the maximum read-write bandwidth corresponding to the used computing resource cannot be exceeded.

This embodiment, task v _i The calculation function expression of the total size of the read data and the write data of the shared file system is as follows:

task v _i The calculation function expression of the total size of the read data and the write data of the burst buffer is as follows:

wherein, FI _i And FO _i Respectively representing tasks v _i Total read and write data size, BI, to shared file system _i And BO _i Respectively represent tasks v _i Total read and write data size, v, to burst buffer _j For task v _i Succ (v) successor task set of _i ) The task (2) of (1),

for adopting preset storage resource scheduling strategy

To schedule task v _j Whether or not the burst buffer is allowed to be used (a value of 1 indicates permission, a value of 0 indicates non-permission, and only the shared file system can be used if not), e _ji Representing a task v _j And task v _i Amount of data transferred between, O _i For task v _i Has a total write data size of v and has a task of v _i The total computation of (a) is expressed as:

C _i ＝p _i *s，

in the above formula, e _ij For task v _i And task v _j Amount of inter-transmission data (task v) _i And task v _j Edges between corresponding nodes in the directed acyclic graph G), succ (v) _i ) For task v _i Is the set of successor tasks of, pred (v) _i ) For task v _i The set of predecessor tasks. And has the following components: i is _i ＝FI _i +BI _i ，O _i ＝FO _i +BO _i 。

In this embodiment, step S6 determines the estimated ending time at the kth stage according to all tasks in the set X

Shortest task v _n Can be expressed as:

will task v _n Corresponding estimated end time

Assigning a completion time T to the kth stage _k And sets up task v _n Is k +1, can be expressed as:

e _n ＝k+1。

finally, in step S7, the set X and the set Y are determined, and if the set X and the set Y are both empty, the completion time T of all the kth stages is determined _k The sum of (b) is used as the total completion time of the workflow, and the operation is finished and quitted; otherwise, the variable k is added by 1 (k = k + 1), and the step S2 is skipped to enter the next stage.

In addition, on the basis of the foregoing method, this embodiment further provides a performance calculating method for high-performance computer workflow scheduling, including:

Through step S101, the performance calculation model mentioned above in this embodiment may be utilized to study the influence of the system computing resources, the system storage resources, the computing resource scheduling policy, and the storage resource scheduling policy on the scheduling performance of the workflow. For example, the fixed system storage resources, the computing resource scheduling policy and the storage resource scheduling policy are unchanged, the system computing resources are changed, and the scheduling performance of the workflow under different system computing resources can be obtained by calling the performance model, so that the influence of the number of the system computing resources on the scheduling performance of the workflow is obtained. Using the performance computation model described above, the minimum resource configuration combination that can achieve the best scheduling performance for a given workflow, i.e., the minimum resource partition size and the minimum amount of storage resources allocated to the workflow, can be computed. Specifically, the value ranges of two variables, namely the size of the resource partition and the number of the storage resources, can be set, the combination of the two variables is used as a search space, the minimum completion time of a given workflow is searched in the search space by using a performance model, and the size of the resource partition and the number of the storage resources corresponding to the minimum completion time are the optimal resource configuration.

In addition, the embodiment also provides a performance computing system for high-performance computer workflow scheduling, which comprises a microprocessor and a memory which are connected with each other, wherein the microprocessor is programmed or configured to execute the performance computing method for high-performance computer workflow scheduling. Furthermore, the present embodiment also provides a computer-readable storage medium, in which a computer program is stored, the computer program being programmed or configured by a microprocessor to execute the performance calculation method of the aforementioned high-performance computer workflow scheduling.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein. The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments, and all technical solutions belonging to the idea of the present invention belong to the protection scope of the present invention. It should be noted that modifications and adaptations to those skilled in the art without departing from the principles of the present invention should also be considered as within the scope of the present invention.

Claims

1. A performance calculation method for high-performance computer workflow scheduling is characterized by comprising the following steps:

Shortest task v _n Will task v _n Corresponding estimated end time

s7, if the set X and the set Y are both empty, finishing time T of all the kth stages _k Taking the sum of the total completion time of the workflow as the total completion time of the workflow, ending and exiting; otherwise, the variable k is added with 1, and the step S2 is skipped to enter the next stage.

2. The method of claim 1, wherein the step S2 of updating the functional expressions of the set X and the set Y is as follows:

X＝X-v _i ，

in the above formula, v _i For tasks whose termination time is equal to the variable k, v _j For task v _i Succ (v) successor task set of _i ) Task of (1), v _m For task v _j Set of predecessor tasks of pred (v) _j ) Task of (1), e _m For task v _m The end time of (c).

3. The method of claim 1, wherein the computing task v in step S5 is a computing task _i Estimated end time at kth stage

The functional expression of (a) is:

in the above formula, the first and second carbon atoms are,

and

are respectively tasks v _i Read and write Bandwidth to shared File System, FI, at stage k _i And FO _i Respectively representing tasks v _i Total read and write data size, BI, to shared file system _i And BO _i Respectively representing tasks v _i The total size of read and write data to the burst buffer, F the shared file system size, B the burst buffer size, I _i And O _i Are respectively task v _i The total size of the read and write data of (c),

and

are respectively tasks v _i Read and write data size at the kth stage.

4. The method of claim 3, wherein step S5 further comprises for each task v in the set X _i Calculation task v _i For the calculated amount in the k stage, and calculate task v _i The functional expression for the calculated amount at the k-th stage is:

in the above formula, the first and second carbon atoms are,

for task v _i Amount of calculation in the k-th stage, T _k+1 Is the completion time of the k +1 stage, min represents taking the minimum value, R _B And W _B Respectively the read and write bandwidths of the burst buffer.

5. The method of claim 4, wherein said task v is a task of a high-performance computer workflow schedule _i Read bandwidth, task v at stage k _i The computational function expression of the write bandwidth at the kth stage is:

in the above formula, the first and second carbon atoms are,

and

and

respectively represent tasks v _i For read and write density to a shared file system, R _s Read bandwidth, W, for computing nodes to a parallel file system _s Write Bandwidth, R, for computing nodes on a parallel File System _f Read bandwidth, W, for a parallel file system for a node partition _f Write Bandwidth, p, for node partitions to parallel File System _i For task v _i The number of compute nodes used.

6. The method of claim 5, wherein task v is a task of a high-performance computer workflow schedule _i The computational function expression for read-write density to a shared file system is:

in the above formula, the first and second carbon atoms are,

and

respectively representing tasks v _i For read-write density to a shared file system, FI _i And FO _i Respectively representing tasks v _i Read and write data to shared file systemTotal size, t _i For task v _i The time required for operation.

7. The method of claim 3, wherein the task v is a task _i The calculation function expression of the total size of the read data and the write data of the shared file system is as follows:

wherein, FI _i And FO _i Respectively representing tasks v _i Total read and write data size, BI, to shared file system _i And BO _i Respectively representing tasks v _i Total read and write data size, v, to burst buffer _j For task v _i Succ (v) successor task set of _i ) The task (2) of (1),

scheduling policy for employing preset storage resources

To schedule task v _j Whether or not it is allowedUsing a burst buffer, e _ji Representing a task v _j And task v _i Amount of data transferred therebetween, O _i For task v _i Has a total write data size of v and has a task of v _i The calculation function expression of the total calculation amount of (a) is:

C _i ＝p _i *s，

in the above formula, e _ij For task v _i And task v _j Amount of data to be transmitted between, succ (v) _i ) For task v _i Is the successor task set of pred (v) _i ) For task v _i The set of predecessor tasks.

8. A performance calculation method for high-performance computer workflow scheduling, comprising:

s101, aiming at four influence factors of system computing resources, system storage resources, computing resource scheduling strategies and storage resource scheduling strategies in a high-performance computer on workflow scheduling performance, generating a plurality of resource configuration schemes by fixing three influence factors and changing resource configuration of the remaining influence factor, and aiming at a given workflow, calling the performance computing method for workflow scheduling of the high-performance computer according to any one of claims 1 to 7 aiming at each resource configuration scheme to obtain total completion time of the corresponding workflow, so as to obtain a relation curve between each influence factor and the total completion time of the workflow;

9. A performance computing system for high performance computer workflow scheduling comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to perform the performance computing method for high performance computer workflow scheduling according to any one of claims 1 to 8.

10. A computer-readable storage medium, in which a computer program is stored, the computer program being adapted to be programmed or configured by a microprocessor to perform a method of performance computation of a high performance computer workflow schedule according to any of the claims 1-8.