CN111950835B - Deadline constraint workflow resource scheduling method based on bidding type example - Google Patents

Deadline constraint workflow resource scheduling method based on bidding type example Download PDF

Info

Publication number
CN111950835B
CN111950835B CN201911243645.7A CN201911243645A CN111950835B CN 111950835 B CN111950835 B CN 111950835B CN 201911243645 A CN201911243645 A CN 201911243645A CN 111950835 B CN111950835 B CN 111950835B
Authority
CN
China
Prior art keywords
task
scheduling
segment
instance
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911243645.7A
Other languages
Chinese (zh)
Other versions
CN111950835A (en
Inventor
邓科峰
曹书锦
宋君强
任开军
李小勇
任小丽
周翱隆
张家灏
杨云天
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
National University of Defense Technology
Original Assignee
National University of Defense Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National University of Defense Technology filed Critical National University of Defense Technology
Priority to CN201911243645.7A priority Critical patent/CN111950835B/en
Publication of CN111950835A publication Critical patent/CN111950835A/en
Application granted granted Critical
Publication of CN111950835B publication Critical patent/CN111950835B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G06Q10/06313—Resource planning in a project environment
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
    • G06Q10/06316—Sequencing of tasks or work
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0637—Strategic management or analysis, e.g. setting a goal or target of an organisation; Planning actions based on goals; Analysis or evaluation of effectiveness of goals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00—Commerce
    • G06Q30/06—Buying, selling or leasing transactions
    • G06Q30/08—Auctions

Abstract

The invention discloses a deadline constraint workflow resource scheduling method based on a bidding type example, which comprises the following steps: acquiring an initial scheduling scheme of a workflow on resources; traversing the initial scheduling scheme to obtain a scheduling task set on each instance; segmenting the scheduling task, and adopting different rescheduling strategies according to the task types contained in the segmentation; and scheduling according to the rescheduled workflow resource scheduling scheme. Compared with the traditional scheduling scheme, the method and the device have the advantages that the initial scheduling is optimized in a segmented mode by combining with the bidding type examples, different scheduling strategies are adopted according to task characteristics in different segments, so that the scheduling cost is optimized, the cost can be further optimized on the basis of the overall scheduling scheme by adopting the workflow scheduling method, and the use efficiency is maximized.

Description

Deadline constraint workflow resource scheduling method based on bidding type example
Technical Field
The invention relates to a resource scheduling method for a workflow, in particular to a deadline constraint workflow resource scheduling method based on a bidding type example.
Background
Today, famous cloud service providers at home and abroad, such as Amazon, google, microsoft, etc., have established a plurality of large cloud data centers around the world. The cloud provider reconfigures its computing resources (e.g., CPU, memory, disk storage, network, etc.) and provides them to the tenants in the form of services. Due to its flexible scalability, high availability, and "pay-as-you-go" billing model, more and more researchers are using cloud resources to perform scientific applications, such as large-scale scientific workflows. However, scientific workflows typically have complex structures, cover massive data transfers, and require significant computing resources. Therefore, for large-scale scientific workflows, it is important to establish an efficient scheduling solution to benefit from the potential advantages of cloud services.
In fact, conventional cloud service providers often over-configure cloud resources to meet peak demands of users, such that the computing resources of the cloud computing center are underutilized most of the time. To improve resource utilization, cloud providers provide cost-effective resource models for users. For example, since 2009 Amazon proposed an instance type based auction, namely a bidding type instance. In such a resource configuration model, a user may run an instance at the actual price per hour when a bid exceeds the current spot price. Specifically, when the user's price is not less than the market price, the instance can be successfully started; when the market price is higher than the user's bid, the instance will be revoked. Although such resources are unreliable, the low price makes it possible to save up to 90% of the use cost, since the competitive example is Amazon's idle resource.
The advantage of the competitive example is that when there is available resource, the user can access the resource with the same performance at a lower price (typically 50% to 90% cheaper). This brings a reciprocal situation for cloud providers and users, as it not only reduces the operating and maintenance costs caused by idle resources, but also helps tenants run large-scale workloads with significant cost savings. Nevertheless, the biggest challenge is that bidding type instances may be interrupted at any time. When such instances are used, suddenly dropping a running instance will severely degrade the performance of the application. For example, if a long running task is interrupted, the entire workflow completion will be severely deferred.
It is difficult to schedule scientific workflows with deadline constraints using competitive instances compared to commonly used on-demand instances. On the one hand, due to the uncertain time-to-live, in order to guarantee continuous availability of bidding type instances, it is necessary to reduce the probability of instance revocation through a reasonable bidding strategy. On the other hand, fault tolerance mechanisms, such as task Retry (Retry), task replication (replay), and Checkpoint mechanism (Checkpoint), need to be introduced in the workflow scheduling process. The fault tolerance technique enhances the robustness of the scheduling algorithm and the flexibility of workflow execution. That is, the fault tolerance mechanism makes scheduling with bidding-type instances reliable, recoverable, and safe in the event of an unexpected failure. One possible solution is to checkpoint and perform a failover after a bidding-type instance is interrupted. However, the effect of these techniques is twofold. For example, setting a large number of checkpoints will inevitably introduce additional overhead to execute the application; replicating a task across multiple instances inevitably introduces additional execution costs; increasing bids can reduce the chance of interruption of live instances, but higher bid prices also increase the execution overhead of scientific workflows. Therefore, how to design a scheduling method with cost effectiveness and fault tolerance to adapt to the uncertain survival time in the auction mode is an urgent technical problem to be solved.
Disclosure of Invention
The invention aims to provide a deadline constraint workflow resource scheduling method based on a bidding type example, which is used for optimizing the resource scheduling cost of a scientific workflow under the condition of ensuring stable execution of a job.
In order to achieve the purpose, the invention adopts the following technical scheme, and the deadline constraint workflow resource scheduling method based on the bidding type example comprises the following steps:
step 1, obtaining an initial scheduling scheme of a workflow on resources;
step 2, traversing the initial scheduling scheme to obtain a scheduling task set on each instance;
step 3, segmenting the scheduling task, and adopting different rescheduling strategies according to the task types contained in the segmentation;
and 4, scheduling according to the rescheduled workflow resource scheduling scheme.
Specifically, the step 3 of segmenting the scheduling task includes the following steps: for each task tiIf the task is a critical path task, the critical task is segmented independently; if the task is not a critical path task, adding the task to the temporary subset of tasks
Figure BDA0002306941370000037
Simultaneous recording of temporary task subsets
Figure BDA0002306941370000032
The start time EST (t) of the first task in (1)start) For the segment start time
Figure BDA0002306941370000038
Updating end times of subsections simultaneously
Figure BDA0002306941370000034
For the current task tiEnd time EFT (t)i) If the duration time of the temporary task subset is above a preset time threshold, the temporary task subset is used
Figure BDA0002306941370000035
And dividing into a section, if the duration of the temporary task subset does not reach a preset time threshold, continuously traversing the tasks and adding the temporary task subset to obtain the section.
Specifically, the adoption of different rescheduling strategies according to the task types contained in the segments specifically includes the following steps:
if the segments are key tasks, selecting an on-demand instance to schedule the tasks;
if segmentation is not a critical task, then formulas are utilized
Figure BDA0002306941370000036
Calculating the utilization of the segments, wherein the numerator sigma taskeexecutiontime isThe execution time of all tasks in the segment, denominator is the instance lease time of the segment, and the lease start time VMLeaseStartTime is subtracted from the lease end time VMLeaseFinishTime;
if the utilization rate theta of the segment is lower than a set threshold value, the segment is defined as a low-utilization-rate segment and is scheduled by using a bidding type instance;
if the utilization rate theta of the segment is higher than a set threshold value, the segment is a high-utilization-rate segment, a parameter maxexeetime is introduced to record the maximum task execution time in the current sub-segment, and if the maximum task execution time maxeetime of the segment is smaller than a time parameter gamma, the segment is defined as a fine-grained task segment and is scheduled by using a competitive example;
if the maximum task execution time maxexectime of the high-utilization segment is greater than the time parameter gamma, the segment contains a large task, and the scheduling is carried out by adopting an on-demand instance.
Preferably, the preset time threshold is a charging period, i.e. a minimum charging time.
Specifically, using bidding-type instances for scheduled segmentation, withdrawal of instances is handled using the following steps:
recording an index of a current scheduling task;
re-enabling a type-k bidding-type instance VMnew；
If it is to be segmented
Figure BDA0002306941370000041
Scheduling of remaining tasks to the New Bid instance VMnewIf the sub deadline constraint condition is violated, the bidding type instance VM is usednewThe type of (2) is upgraded from k to k + 1;
if the current instance type is already the optimal type, then continue with this type of instance scheduling, calculating the violated deadline Δ into the next segment.
Further, the initial scheduling scheme in step 1 is generated based on the ProLiS scheduling method.
Further, the initial scheduling scheme in step 1 is generated based on the ICPCP scheduling method.
The main ideas of the invention are as follows: on the basis of a heuristic initial scheduling scheme, by segmenting a scheduling sub-scheme on a single instance, different scheduling strategies are adopted according to task types contained in different segments: and the key tasks are segmented and scheduled to the on-demand instances, and segments containing fine-grained tasks and low utilization rate are introduced into the bidding type instances for scheduling. And simultaneously, an ingenious bidding strategy is adopted for the sub-section using the bidding type example, so that free example hour using time is won for the user.
The invention can achieve the following beneficial effects:
compared with the traditional scheduling scheme, the method and the device have the advantages that the initial scheduling is optimized in a segmented mode by combining with the bidding type examples, and different scheduling strategies are adopted according to task characteristics in different segments, so that the scheduling overhead is optimized. By adopting the workflow scheduling method, the expenditure can be further optimized on the basis of a global scheduling scheme, and the use efficiency is maximized.
Drawings
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is an example of a scientific workflow scheduled by an embodiment of the present invention;
FIG. 3 is an exemplary diagram of a piecewise-optimal scheduling scheme based on a bidding-type example;
FIG. 4 illustrates a cost optimization effect based on a ProLiS algorithm segment scheduling policy under a constraint of a tightening deadline;
FIG. 5 illustrates the cost optimization effect of a piecewise scheduling strategy based on the ICPCP algorithm under the constraint of a tightening deadline;
FIG. 6 shows a ProLiS algorithm-based piecewise scheduling policy cost optimization under relaxed deadline constraints;
FIG. 7 illustrates the cost optimization effect of the strategy based on the ICPCP algorithm segmented scheduling under the constraint of the loose deadline.
Detailed Description
Let the science workflow be denoted as G ═ V, E, i.e., the DAG graph contains | V | nodes and | E | edges. Where V represents tasks and E represents data dependencies between tasks.
Definition 1.data (t)i,tj) Namely dataij: representing slave task tiTo task tjAnd task t, andiis task tjThe preceding task of (1).
Definitions 2.pred (t)i)&succ(ti)：pred(ti) Represents tiAll the preceding tasks (parent nodes); succ (t)i) Represents tiAll successor tasks (child nodes) of (1).
EST (t)i)&EFT(ti)：EST(ti) Representing a task tiEarliest executable time, task tiIs the earliest executable time and its predecessor task pred (t)i) The end time of (2) is related to the data transmission time, and the calculation mode is as follows: EST (t)i)＝max{EFT(tj)+dji,tj∈pred(ti)}；EFT(ti) Representing a task tiThe earliest execution end time, denoted EFT (t)i)＝EST(ti)+ωik。
djiIs task tjTo task tiThe data transmission time of (1); omegaikIs task tiIn a virtual machine VMkThe execution time of (1); if task tjAnd task tiAre all allocated to the same virtual machine, have dji＝0。
The specific process of the scientific workflow scheduling sectional optimization method based on the bidding type example is as follows: firstly, each instance in the scheduling scheme is traversed to obtain a scheduling sub-scheme on each instance, and each scheduling sub-scheme is divided according to a segmentation strategy. In the segmentation process, capturing the characteristics of each divided sub-segment, and rescheduling the sub-segments according to a corresponding scheduling strategy. The cost of the sectional scheduling scheme is better than that of the original scheduling scheme on the premise of meeting the deadline constraint.
As shown in fig. 1, the deadline constraint workflow resource scheduling method based on the bidding type example includes the following steps:
step 1, obtaining an initial scheduling scheme of a workflow on resources;
step 2, traversing the initial scheduling scheme to obtain a scheduling task set on each instance;
step 3, segmenting the scheduling task, and adopting different rescheduling strategies according to the task types contained in the segmentation;
and 4, scheduling according to the rescheduled workflow resource scheduling scheme.
Specifically, the step 3 of segmenting the scheduling task includes the following steps: for each task tiIf the task is a critical path task, the critical task is segmented independently; if the task is not a critical path task, adding the task to the temporary subset of tasks
Figure BDA0002306941370000065
Simultaneous recording of temporary task subsets
Figure BDA0002306941370000062
The start time EST (t) of the first task in (1)start) For the segment start time
Figure BDA0002306941370000066
Updating end times of subsections simultaneously
Figure BDA0002306941370000064
For the current task tiEnd time EFT (t)i) If the duration time of the temporary task subset is above a preset time threshold, the temporary task subset is used
Figure BDA0002306941370000071
And dividing into a section, if the duration of the temporary task subset does not reach a preset time threshold, continuously traversing the tasks and adding the temporary task subset to obtain the section.
Specifically, the adoption of different rescheduling strategies according to the task types contained in the segments specifically includes the following steps:
if the segments are key tasks, selecting an on-demand instance to schedule the tasks;
if segmentation is not a critical task, then formulas are utilized
Figure BDA0002306941370000072
Calculating the utilization rate of the segment, wherein the numerator sigma taskeexecutiontime is the execution time of all tasks in the segment, the denominator is the instance lease time of the segment, and the lease start time VMLeaseStartTime is subtracted from the lease end time vmleasefinishitime;
if the utilization rate theta of the segment is lower than a set threshold value, the segment is defined as a low-utilization-rate segment and is scheduled by using a bidding type instance;
if the utilization rate theta of the segment is higher than a set threshold value, the segment is a high-utilization-rate segment, a parameter maxexeetime is introduced to record the maximum task execution time in the current sub-segment, and if the maximum task execution time maxeetime of the segment is smaller than a time parameter gamma, the segment is defined as a fine-grained task segment and is scheduled by using a competitive example;
if high utilization fragmentation
Figure BDA0002306941370000073
When the maximum task execution time maxexectime is greater than the time parameter gamma, the segment contains a large task, and the scheduling is performed by adopting an on-demand instance.
Preferably, the preset time threshold is a charging period, i.e. a minimum charging time.
Specifically, using bidding-type instances for scheduled segmentation, withdrawal of instances is handled using the following steps:
recording an index of a current scheduling task;
re-enabling a type-k bidding-type instance VMnew；
If it is to be segmented
Figure BDA0002306941370000074
Scheduling of remaining tasks to the New Bid instance VMnewIf the sub deadline constraint condition is violated, the bidding type instance VM is usednewThe type of (2) is upgraded from k to k + 1;
if the current instance type is already the optimal type, then continue with this type of instance scheduling, calculating the violated deadline Δ into the next segment.
Further, the initial scheduling scheme in step 1 is generated based on the ProLiS scheduling method.
Further, the initial scheduling scheme in step 1 is generated based on the ICPCP scheduling method.
IC-PCP and ProLiS are both common and effective heuristic algorithms in scientific workflow scheduling.
The input of the more detailed calculation method is an initial scheduling scheme and a deadline D; the output is a segmented scheduling scheme SegSchedule, and the specific steps are as follows:
(1) traversing the leased instance set in the initial scheduling scheme Schedule to obtain the scheduling task set S on each instancek，i；
(2) Traversal task set Sk，iFor each task
Figure BDA0002306941370000081
If task tiSelecting an on-demand instance to schedule the task t for the critical path taskiThen, continuously traversing the next task;
(3) if task tiIf the task is not a critical path task, the task t is processediJoining a temporary subset of tasks
Figure BDA0002306941370000082
Recording current subsets simultaneously
Figure BDA0002306941370000083
The start time EST (t) of the first task in (1)start) For the segment start time
Figure BDA00023069413700000813
Updating end times of subsections simultaneously
Figure BDA0002306941370000085
For the current task tiEnd time EFT (t)i)；
(4) If it is the current childSegment of
Figure BDA0002306941370000086
Schedule time of
Figure BDA00023069413700000812
If the time does not exceed one example hour, executing the step (3) according to the requirement; if the scheduling time of the current sub-section exceeds one example hour, allocating the virtual machine according to a corresponding scheduling strategy;
(5) reference formula
Figure BDA0002306941370000088
Computing segmentation
Figure BDA0002306941370000089
Where the numerator is the execution time of all tasks in the segment and the denominator is the instance lease time of the segment.
5.1 if segmentation
Figure BDA00023069413700000810
When the utilization rate theta is lower than a set threshold value, defining the section as a low-utilization-rate section, and scheduling by using a bidding type example;
5.2 if segmentation
Figure BDA00023069413700000811
When the utilization rate theta is higher than a set threshold value, segmenting into high-utilization-rate segments, and scheduling by using an on-demand instance;
5.3 executing the step (2) and continuously traversing the task set Sk，i；
(6) Traversal segmentation
Figure BDA0002306941370000091
All tasks in (1), introducing a parameter maxexeltime to record the current sub-section
Figure BDA0002306941370000092
Maximum task execution time of (2).
6.1 if segmentation
Figure BDA0002306941370000093
When the maximum task execution time maxexestime is smaller than the time parameter gamma, the segment is defined as a fine-grained task segment and is scheduled by using a bidding type instance;
6.2 if segmentation
Figure BDA0002306941370000094
When the maximum task execution time maxexectime is greater than the time parameter gamma, the segment contains a large task and is scheduled by adopting an on-demand instance;
6.3 executing step (2) and continuously traversing the task set Sk，i；
(7) For using bidding type instance VMs[k]Scheduling segments
Figure BDA0002306941370000097
Revocation of instances is handled using the following steps:
7.1 recording the index of the current scheduling task;
7.2 Re-enabling a type k bidding-type instance VMnew；
7.3 if the segmentation is to be carried out
Figure BDA0002306941370000096
Scheduling of remaining tasks to the New Bid instance VMnewIf the sub deadline constraint condition is violated, the bidding type instance VM is usednewIs upgraded from k to k + 1. If the current instance type is the optimal type, continuing to use the instance scheduling of the type, and calculating the violated deadline delta into the next segment;
(8) and obtaining a new scheduling scheme SegSchedule.
FIG. 2 is a diagram of five common scientific workflows, where Montage is commonly used in astronomy, Cybershake is primarily used in seismology, LIGO is commonly used in quantum physics, and Epigenomics is commonly used in biology. The scheduling algorithm in the invention utilizes a high-performance computing application research center of the national defense science and technology university as an experimental environment, and carries out real experiments by imitating various types and charging modes of Amazon EC2 virtual machines. The following describes an embodiment of the present invention with reference to fig. 3, taking the workflow in fig. 2 as input data.
When we compute task t with the slowest instanceiThe earliest start time EST (t)i) And earliest completion time and EFT (t)i) In time, the instance upgrade may reserve more free time to undertake the withdrawal of the bidding-type instance. FIG. 3(a) depicts a case where tasks are executed using only on-demand instances using an initial scheduling scheme. Obviously, the overall execution time is 315 minutes. According to Amazon's charging policy, the charging time in this case is 6 instance hours. In fig. 3(b), we divide this scheduling process into three phases and take a bidding type example in the latter two phases, which greatly reduces the cost overhead. Therefore, the temperature of the molten metal is controlled,
Figure BDA0002306941370000103
by
Figure BDA0002306941370000104
And
Figure BDA0002306941370000105
three segments are composed of
Figure BDA0002306941370000101
And
Figure BDA0002306941370000102
scene 1: the critical path tasks are performed using on-demand instances. The Critical Path (Critical Path) is the longest Path of the workflow from the start task to the end task, and the Critical Path tasks are tasks distributed on the Critical Path. We describe the tasks on the critical path uniformly as tcpBecause they are very sensitive to deadline constraints and play an irreplaceable role in the execution of the entire workflow, less reliable execution of the bidding instance is not recommended. Critical task t as in fig. 3(b)1Success or failure of its execution may have a large impact on the execution of subsequent tasks. In other words, the critical task t1In the implementation ofAn outage may result in the entire workflow failing to complete within the deadline. Therefore, we will be tasked with (e.g., t)1) An on-demand instance is assigned to ensure it will complete on time. For task t on task critical pathcpWe assign it to an on-demand instance VMd[k]。
Scene 2: highly concurrent fine-grained tasks are performed using bidding-type instances. In real-world scientific workflows, there are many concurrent tasks with low execution overhead, which we call fine-grained tasks. We introduce an empirical factor y to resolve the exact execution time of the fine-grained task. Because of the shorter execution time, such tasks have a relatively lower retry cost after the withdrawal of the bidding instance. For example, a task is executed for 40 seconds, so the retry cost after failure is less than or equal to 40 seconds, and the influence on the whole workflow execution is negligible.
In FIG. 3(b), such as t4,t8,t10,t16And t20Like tasks are fine-grained tasks that are executed in sequence and are segmented
Figure BDA0002306941370000106
Thus leaving a considerable idle time for task retries. On this basis, we can consider moving them to bidding-type instances, using relatively aggressive bidding, while ensuring that the task can be completed on time. Furthermore, if a bidded instance is withdrawn, the current task and subsequent tasks will immediately be transferred to other bidded instances for re-execution.
Scene 3: low utilization segmentation is performed using a bidding-type instance and a pick-and-happen auction (auction-auction) strategy is employed. According to Amazon EC 2's bidding rules, the user does not have to pay any fee when a running instance is withdrawn by Amazon within the first hour; if the user manually terminates the instance, the user makes a payment in terms of minutes used. By utilizing the rule, the Amazon adopts aggressive bidding to withdraw the instance in time after the task is completed, and if the instance is not more than one instance hour, the task execution does not generate cost. In addition, we introduce a factor of θ to quantify the instance usage, the denominator is the time to charge for the instance, the lease on time (vmleasefixishtime) is subtracted by the lease on time (VMLeaseStartTime); and the numerator is the sum of the execution times of all tasks allocated on that instance.
In FIG. 3(b), segmentation
Figure BDA0002306941370000111
Two example hours of payment are required, but task t30And t100With 30 minutes of idle time in between, which results in a low utilization of the whole execution process. At this time, if we are for task t30Adopting a smart bidding strategy, when the task is executed, the scheduling strategy withdraws the instance by making the spot price higher than the bidding price through accurate bidding, and the task t is at the moment30Is performed without any fee. And since the instance is revoked, the subsequent task t100Needs to be scheduled to a new bidding instance. At this point, t can be completed within one example hour with conservative bidding100Is performed. Thus, segmenting
Figure BDA0002306941370000112
The charging time of (2) is reduced from the initial two instance hours to one instance hour, and the economic cost can be greatly reduced.
And verifying the performance of the scheduling algorithm by utilizing five classic type workflows and data sets with different task sizes. As shown in fig. 4, 5, 6, 7, the proposed optimization strategy can significantly reduce the cost overhead of the initial scheduling solution while satisfying the user-specified deadline constraints.

Claims (6)

1. The deadline constraint workflow resource scheduling method based on the bidding type example is characterized by comprising the following steps of:
step 1, obtaining an initial scheduling scheme of a workflow on resources;
step 2, traversing the initial scheduling scheme to obtain a scheduling task set on each instance;
step 3, segmenting the scheduling task, and adopting different rescheduling strategies according to the task types contained in the segmentation;
step 4, scheduling according to the rescheduled workflow resource scheduling scheme;
segmenting the scheduling task in step 3, comprising the steps of: for each task tiIf the task is a critical path task, the critical task is segmented independently; if the task is not a critical path task, adding the task to the temporary subset of tasks
Figure FDA0002895236700000011
Simultaneous recording of temporary task subsets
Figure FDA0002895236700000012
The start time EST (t) of the first task in (1)start) For the segment start time
Figure FDA0002895236700000013
Updating end times of subsections simultaneously
Figure FDA0002895236700000014
For the current task tiEnd time EFT (t)i) If the duration time of the temporary task subset is above a preset time threshold, the temporary task subset is used
Figure FDA0002895236700000015
And dividing into a section, if the duration of the temporary task subset does not reach a preset time threshold, continuously traversing the tasks and adding the temporary task subset to obtain the section.
2. The method for deadline constraint workflow resource scheduling based on bidded instances according to claim 1, wherein said adopting different rescheduling strategies according to the task types contained in the segments specifically comprises the following steps:
if the segments are key tasks, selecting an on-demand instance to schedule the tasks;
if segmentation is not a critical task, then formulas are utilized
Figure FDA0002895236700000016
Calculating the utilization rate of the segment, wherein the numerator sigma taskeexecutiontime is the execution time of all tasks in the segment, the denominator is the instance lease time of the segment, and the lease start time VMLeaseStartTime is subtracted from the lease end time vmleasefinishitime;
if the utilization rate theta of the segment is lower than a set threshold value, the segment is defined as a low-utilization-rate segment and is scheduled by using a bidding type instance;
if the utilization rate theta of the segment is higher than a set threshold value, the segment is a high-utilization-rate segment, a parameter maxexeetime is introduced to record the maximum task execution time in the current sub-segment, and if the maximum task execution time maxeetime of the segment is smaller than a time parameter gamma, the segment is defined as a fine-grained task segment and is scheduled by using a competitive example;
if the maximum task execution time maxexectime of the high-utilization segment is greater than the time parameter gamma, the segment contains a large task, and the scheduling is carried out by adopting an on-demand instance.
3. The method for deadline constraint workflow resource scheduling based on bidding type instances according to claim 1, wherein the preset time threshold is a charging period, i.e. a minimum charging time.
4. The method for deadline constrained workflow resource scheduling based on bidded instances according to claim 2 or 3 wherein the bidded instances are used for the segmentation of the schedule, the withdrawal of instances being handled by the following steps:
recording an index of a current scheduling task;
re-enabling a type-k bidding-type instance VMnew；
If it is to be segmented
Figure FDA0002895236700000021
Is left overScheduling of remaining tasks to the New Bid instance VMnewIf the sub deadline constraint condition is violated, the bidding type instance VM is usednewThe type of (2) is upgraded from k to k + 1;
if the current instance type is already the optimal type, then the scheduling continues with that instance type, and the violated deadline Δ is computed into the next segment.
5. The method for deadline constrained workflow resource scheduling based on bidded instances according to claim 4 wherein the initial scheduling scheme in step 1 is generated based on ProLiS scheduling method.
6. The deadline constraint workflow resource scheduling method based on bidded instances according to claim 4 wherein the initial scheduling scheme in step 1 is generated based on an ICPCP scheduling method.
CN201911243645.7A 2019-12-06 2019-12-06 Deadline constraint workflow resource scheduling method based on bidding type example Active CN111950835B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911243645.7A CN111950835B (en) 2019-12-06 2019-12-06 Deadline constraint workflow resource scheduling method based on bidding type example

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911243645.7A CN111950835B (en) 2019-12-06 2019-12-06 Deadline constraint workflow resource scheduling method based on bidding type example

Publications (2)

Publication Number Publication Date
CN111950835A CN111950835A (en) 2020-11-17
CN111950835B true CN111950835B (en) 2021-03-12

Family

ID=73336219

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911243645.7A Active CN111950835B (en) 2019-12-06 2019-12-06 Deadline constraint workflow resource scheduling method based on bidding type example

Country Status (1)

Country Link
CN (1) CN111950835B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106447173A (en) * 2016-09-05 2017-02-22 北京理工大学 Cloud workflow scheduling method supporting any flow structure
CN108762899A (en) * 2018-05-16 2018-11-06 武汉轻工大学 A kind of cloud task rescheduling method and apparatus
CN109784663A (en) * 2018-12-20 2019-05-21 西北大学 A kind of workflow schedule method and apparatus

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140257906A1 (en) * 2013-03-08 2014-09-11 Trimble Navigation Limited Workflow Management Method and System
US9594478B2 (en) * 2013-08-30 2017-03-14 Oracle International Corporation Flow pipe totals
US10318761B2 (en) * 2016-06-10 2019-06-11 OneTrust, LLC Data processing systems and methods for auditing data request compliance
CN106339924A (en) * 2016-08-29 2017-01-18 东南大学 Workflow based mixed renting method of cloud computing resources
CN109426550B (en) * 2017-08-23 2022-12-06 阿里巴巴集团控股有限公司 Resource scheduling method and equipment
US20190180218A1 (en) * 2017-12-12 2019-06-13 Einthusan Vigneswaran Methods and systems for automated multi-user task scheduling
CN110503534A (en) * 2019-08-27 2019-11-26 山东大学 Cloud computing service resource dynamic dispatching method and system based on price expectation

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106447173A (en) * 2016-09-05 2017-02-22 北京理工大学 Cloud workflow scheduling method supporting any flow structure
CN108762899A (en) * 2018-05-16 2018-11-06 武汉轻工大学 A kind of cloud task rescheduling method and apparatus
CN109784663A (en) * 2018-12-20 2019-05-21 西北大学 A kind of workflow schedule method and apparatus

Also Published As

Publication number Publication date
CN111950835A (en) 2020-11-17

Similar Documents

Publication Publication Date Title
Chung et al. Stratus: Cost-aware container scheduling in the public cloud
US20210255986A1 (en) Managing use of program execution capacity
US10114668B2 (en) Managing private use of program execution capacity
Calheiros et al. Energy-efficient scheduling of urgent bag-of-tasks applications in clouds through DVFS
Zhou et al. Transformation-based monetary costoptimizations for workflows in the cloud
US9645848B2 (en) Scheduling homogeneous and heterogeneous workloads with runtime elasticity in a parallel processing environment
Joseph et al. IntMA: Dynamic interaction-aware resource allocation for containerized microservices in cloud environments
Voorsluys et al. Provisioning spot market cloud resources to create cost-effective virtual clusters
Huang et al. Cap3: A cloud auto-provisioning framework for parallel processing using on-demand and spot instances
Chen et al. Uncertainty-aware real-time workflow scheduling in the cloud
CN106775932B (en) Real-time workflow scheduling method triggered by random event in cloud computing system
Shin et al. Deadline-guaranteed scheduling algorithm with improved resource utilization for cloud computing
Arabnejad et al. Multi-QoS constrained and profit-aware scheduling approach for concurrent workflows on heterogeneous systems
Kang et al. A SLA driven VM auto-scaling method in hybrid cloud environment
Henzinger et al. A marketplace for cloud resources
Kumbhare et al. Exploiting application dynamism and cloud elasticity for continuous dataflows
Monge et al. Autoscaling Scientific Workflows on the Cloud by Combining On-demand and Spot Instances.
Zhou et al. Fault tolerant stencil computation on cloud-based GPU spot instances
Marathe et al. Exploiting redundancy and application scalability for cost-effective, time-constrained execution of hpc applications on amazon ec2
US20160350143A1 (en) Methods and systems to assess efficient usage of data-center resources by virtual machines
CN111950835B (en) Deadline constraint workflow resource scheduling method based on bidding type example
Meng et al. An uncertainty-aware evolutionary scheduling method for cloud service provisioning
Genez et al. A flexible scheduler for workflow ensembles
Cao et al. An optimizing algorithm for deadline constrained scheduling of scientific workflows in iaas clouds using spot instances
Galchonkov et al. Reducing cloud infrastructure costs through task management

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant